At last the crawler can finish without performance degradation and attendence. From now on crawling will happen automatically every Wednesday and Sunday 00:00 UTC.
Also security update. Due to some weird firewall logs. Which I suspect are simply bots poking around, but anyways. TLGS server now runs on GrapheneOS's hardened_malloc and with much less privilege. I think my code is secure. But better safe than sorry. (Man, I'd love to have unveil and pledge on Linux).
I switched the ranking algorithm from HITS to SALSA after some optimization and bug fixes. It is faster than HITS and surfaces more relevant pages. Also very pround that gemini.circumlunar.space is finally the first link in the search result when I search for "gemini". Followed by other very popular capsules like medusae.space!
Happy new year! The search engine can decuplicate search results. Hopefully it improes the user experience. Let me know if it hilds important results for you. It shouldn't. But who knows. maybe a bug is hiding.
The crawler is updated to handle common wildcard patterns and server stablity is improved. Hopefully redusing trobles people have with TLGS.
TLGS has a common-web interface! You can search content on Gemini from common web. It does not proxy content from outside of TLGS's site. You still need a Gemini browser to browse searched result.
Running well. However it cannot finish crawling without attendence. Likely some bug in the crawler. If you see random downtime of TLGS recently, that's likely me fixing stuff.
Going public! Hope this goes well
Tried a full-crawl of the Geminispace. Currently only crawls when I want to. If this goes well will move to a VPS and open it for public use.
Test deployment in my homelab. Running with 40K pages in index. Response time is acceptable with a bloody slow CPU.