2023-09-12 More Updates to AuraGem

Over the past two days I have made some more updates to AuraGem, particularly the Search Engine. These are mostly stability and QoL updates.

Search Engine:

• Fragments are stripped in the crawler
• The total indexed pages is now at over 340,000 pages and 2,178 domains.
• Initial Backlinks support - you can now search a url and see a list of all backlinks to it
• Pagination has been added to some of the "Indexed ..." pages. This vastly improved the stability of the gemini capsule.
• Mimetypes pages have been limited to 300 results for now. In the future pagination will be added to these pages as well.

AuraGem Ask:

• "Register Page" has been renamed to "Register Cert" so that it is more understandable
• The "Register Cert" link shows up on more pages when a user has no client certificate active or when the current cert has not been registered with a username

Robots.txt:

• The Sefaria proxy has been added to the robots.txt of pages that should not be crawled.