💾 Archived View for auragem.letz.dev › devlog › 20240521.gmi captured on 2024-06-16 at 12:43:36. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

2024-05-21 More AuraGem Search Updates: Detection of Duplicates and Mirrors

AuraGem search is now able to detect duplicate documents across all of the supported smallnet protocols (scroll, gemini, nex, and spartan). The search engine has already been storing hashes of documents from the very beginning, since 2021, so detecting duplicates is as simple as comparing hashes. The following rules determine whether duplicate content is shown in search results:

If duplicates are hosted over the same protocol scheme, then only one will be shown in search results.
If a duplicate is hosted over a different protocol scheme, then it will only appear in search results if: 1.) It is hosted over gemini and the user is searching all of the smallnet OR 2.) If the user is searching only that specific protocol using the respective AuraGem search links for each protocol.

As an example, all of AuraGem is hosted over Gemini, Spartan, Nex, and Scroll. When a user searches all of the smallnet, only links from the Gemini server of AuraGem will be shown in search results. However, if a user searches the scroll protocol only, then only links from the Scroll server of AuraGem will be shown in search results.

As an additional example, if a page is mirrored on two different domains, but they are both on the same protocol, then only one set will be shown in search results, and the other will be hidden.

This was added to both hide mirrors from search results and to allow crawling a capsule over multiple protocols without having duplicates in search results.

Other Updates

I am preparing an additional update that will allow capsules to request a full crawl of the capsule on-demand, following internal links only. While it is ready and working right now, I want to add a couple of security checks first. This will be out tomorrow.