💾 Archived View for auragem.letz.dev › devlog › 20240513.gmi captured on 2024-06-16 at 12:44:38. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-05-26)
-=-=-=-=-=-=-
I have updated AuraGem search so that you can search all of the smallnet, or just search Geminispace or just Scrollspace. Smallnet search includes links from Gemini, Nex, and Scroll.
I created this a while ago, but I never had the time to make it usable until now. It is an "aggregator" in that it presents recently published gemini pages from within the past year based on the Search Engine database. Publication dates come from dates in page titles, filenames, or internal links to the page.
Because the Search Engine checks upon crawling whether a page should be considered a feed, the aggregator is able to check every feed page from the search engine's db every 13 hours to crawl 1-level deep for new pages. In this way, the aggregator crawls more frequently, but with significantly less pages, whereas the search engine crawler will run less frequently but try to crawl all of geminispace.
I have updated the code for my new aggregator service, which is titled "Publications from the Past Year". Before, it would query the search database on-the-fly, which was very slow, but now every 13 hours it will query the db and create gemini files so that going to the service will present these gemini files instead. The experience is much faster.
Publications from the Past Year
Note that robots.txt files are followed by both the aggregator crawler and the normal crawler.