💾 Archived View for gmi.bacardi55.io › gemlog › 2022 › 12 › 15 › announcing-discogem captured on 2024-08-18 at 17:35:11. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Announcing DiscoGem, a way to discover new capsules every days

Posted on 2022-12-15

TLDR;

In a nutshell, I've created a small capsule called [DiscoGem]. Every day at 00:05am, it will pick 5 random capsule URLs from a list of known hosts (thanks to Lupa crawler). It then creates a small gemlog entry and an [atom feed].

NOTA: There is no manual curation (and not planned tbh), and capsules might be empty or not updated for a very long time. Goal is to discover capsules, not find "fresh content" (that you can already get via Antenna or Cosmos).

DiscoGem

atom feed

Introduction

I've been thinking lately about improving discovery of capsules in the gemini space. Usually, aggregator like [antenna], [cosmos] or selfhosted solution like [comitium] simplify finding and reading new entries, but that is only solving part of the issue. Don't get me wrong, these tools are great and my gemini browsing usually starts on cosmos and/or antenna! But they mainly allow to find new entry from people submitting their entries to antenna.

But I'm sure that the great gemini space has more to offer, either via capsules that aren't updated often, or have particular content not in a form of a gemlog. How to discover these capsules then? Of course following station, the gemini hastag on the fediverse or using gemini search engines help, but I'm sure there is more to find! The smolzine project is a good example of a project helping discovering cool / different capsules in the gemini space.

I looked at [TLGS] and [Kennedy] search engines that both provide a list of known hosts, as well as the [Lupa crawler]. Lupa's list has a big advantage as it only shows hosts that have been active in the last 30 days, making this list more reliable then the 2 others (at least, it means less tests from my end). On Lupa, there are more than 2300 known hosts listed, which is great! It definitely confirmed that there was a lot more going own in the gemini space than what meets the eyes (or in this case, meets the aggregators).

Of course, I didn't want to go through 2k+ links manually and certainly not at once :). I started thinking about a small code that would just take X random links from the known host lists and send it to me via email or something I would launch manually. Thinking longer about it made me realise maybe I wasn't the only one that wanted to discover new stuff in the gemini space, so I decided to create a dedicated capsule for it. That's how the idea of DiscoGem appeared.

Concepts

The whole thing is very easy and half of what I needed already existed:

Retrieve the list of known hosts from Lupa
Select 5 random links from this list
Save the previously used to avoid duplicates
Generate a new gemlog entry on the dedicated capsule based with the 5 random links.

For now, the database is updated daily just before picking up the 5 links. It should avoid having (too many) broken links, but that's not sure yet :). As said in the TLDR; there is no manual review. I'm thinking about adding an extra check to avoid having empty pages (or responsive server but without any capsule at this domain anymore), but that will be about it in term of pre publication checks.

I'm planning to do a soft reset every X months by reseting the list of used links. I'm not yet sure of when yet, because with 2.3k known hosts, even with 5 links a day, "only" ±900 links would be shown over a period of 3 months. Reseting too often might lead to too many duplicates. I may just not reset until all links have been displayed… The code to reset is done already, I just don't know when to use it yet.

I have small ideas to improve this service but I want it to stay simple: just discover a small amount of capsules a day, making it bearable even if I check it only once a week.

Technical "stack"

I wanted to reuse things as much as possible, so I decided to:

Create a small CLI tool to retrieve the list of known hosts, select X random capsules and generate a gemtext output based on a given template.
Use Kiln, a static capsule generator I'm already using for my gemlog, to generate the capsule files (including the atom feed).
Golang as all my gemini related tools are built in golang.
No database, I simply save stuff in files.
Capsule is hosted on the same gemserv instance hosting houston and the tinylog aggregator.
A simple cron task launching a shell script running the different command and rsync-ing the capsule files to the right place.

This will happen every day at 5 past midnight.

The CLI tool to get the known hosts and give the random links is called [discogem and is available on sourcehut]. Be careful, this project has been developed in a quick and dirty mindset just to make it work in one evening :). There is no README or documentation either, I also need to work on that.

discogem and is available on sourcehut

Conclusion

I'm happy at this stage about having a basic working solution that I don't intend to complexify much. I just want to try browsing more capsules I'm not aware of and maybe update my commitium instance with new cool things. Maybe not… But in any cases, opening 5 links per day won't take too much time^^.

Hope it is useful for others, if not, well… It is at least for me :).

/gemlog/

Send me a gemini mention

send me an email!