This post is a description of the current state of Search Engines on geminispace and their accuracy of results toward user expectations. This is going to be a fairly brief post that will be further expanded upon in Part II, which probably won't be published for at least a few days.
The four main Gemini Search Engines that we will be looking at are:
Note that all of the statistics below are only representative of results from 2022-08-07. Changes that affect these statistics could happen in the future. Additionally, more work could be done to get more accurate study results, because the sample sizes provided here are quite low. They are meant to be preliminary, and are not conclusive of which Search Engine is better or worse.
Note also that this article only talks about what is expected by the user and categorizes types of searches based on different expectations. Each general attribute listed below depends on the type of search a user is doing, because the type of search informs what the user expects out of that search. A user searching for a specific capsule wants back an exact url for that capsule. But a user searching for a category does not want an exact url for a capsule, but rather any results that are relevant to that category. How to *achieve* these results, or whether achieving them is viable, is not the goal of this article, but an article in the future.
The goal is to first categorize types of searches, and their expectations, that a user might enact, and see how well each Search Engine does with each type. The specific means of how Search Engines are able to effectively deduce what is expected from the user, or what type of search a user is choosing to do, is very important, but not relevant for this post. The user is not a programmer. They don't necessarily have any knowledge on how to solve these hard problems. They just know that they are doing a type of search, and the Search Engine either gives back what they wanted, or it doesn't. We must gauge user expectations before we can even consider algorithm improvements, and you do that by looking at different searches that users make, what they expect from those searches, and how well the Search Engine lends toward those expectations.
I wanted to analyze how the four main Gemini Search Engines are doing, so I came up with 6 general attributes of search results. Not all of these are appropriate or expected for every type of search. Rather, each type of search creates an expectation in one or more of these general attributes. This is a way to organize user expectation.
1. Exactness - whether the Search Engine got the exact match that you were looking for. This is relevant for search types where the user tries searching for a specific page or capsule. If a user was searching for something broad, or categories of capsules, then exactness is not desired for that type of search.
2. Relevance - whether the Search Engine gave back results that matched the words within your query (textual relevance), or the pages were related to your query in terms of categorization (category relevance)
3. Popularity - whether the Search Engine prioritized well-known capsules or pages
4. Balance - whether the results are skewed toward one opinion, group of interconnected sites, one site, etc. This was one of the attributes considered for the SALSA algorithm.
5. Broadness - the number of relevant matches the Search Engine was able to give back. Search Engines that cache full page contents to search would naturally be higher in this area.
6. Recency - whether the results prioritized recent posts vs. older posts. This interacts with the type of search one is doing. Generally, when searching for topics related to news, you might want recent news. But, if searching for a well-known document that you want to get back to, you want pages that are older.
For this article, I will be focusing on Exactness and Category Relevance, because these are very relevant for four types of searches chosen for this post, outlined below. Other types of searches will be covered in Part II.
1. Name of Capsule/Service - a user might want to look up the exact name of a service or capsule and get the capsule as the very first search result. This is particularly useful if the capsule or service has a long or hard-to-remember url. In this case, Exactness is desired, and possibly Popularity, especially if one is searching the name of a popular capsule/service.
2. Capsule by Username - a user might want to look up the capsule or profile(s) of a specific user. This should bring up their capsule, social network profiles, etc. Exactness is also desired here, because one is looking for exact/specific capsules and pages.
3. Capsule Categories - a user might want to look for capsules that would be categorized as a certain type of capsule. This might include proxies, aggregators, dictionaries, podcasts, etc. In this case, one wants Category Relevance, which might not be the same as Textual Relevance, because a capsule does not necessarily contain text categorizing itself. In this post we will only being considering Relevancy for this type, but a future post might also take into account the other attributes.
4. Specific Texts (books, well-known documents, pdfs, etc.) - a user might want to get to a specific text by looking up the title of that text. In this case, Exactness and Popularity might be prioritized, as well as Textual Relevance.
It should be noted that sometimes a query is ambiguous in what type of search the user is choosing to do. This is a problem for how Search Engines interpret queries or how they balance assumptions about what the user wants. But, again, The goal of this post is not how to achieve correct interpretation of queries, or how a user can effectively create a search query that makes the expectations unambiguous, but rather, how often the Search Engine gives back what the user expects for the type of search they are intending to do, given a natural search query.
Now, let us compare each of the four Search Engines, based on the four types of searches described above:
For this category, we are looking for specific URLs that are given back when we input the name of a capsule or service. The study results below show the first position, within the search results, of the URL that was desired. Unless otherwise specified, the desired result is the root URL of the capsule/service. "300+" means the desired URL was not found within the first 300 results. For the purpose of averages, they are counted as 300.
| TLGS | AuraGem Search | Kennedy | geminispace.info |
--------------------|------|----------------|---------|------------------|
Antenna (root) | 1 | 2 | 81 | 11 |
Antenna (/about.gmi)| 3 | 8 | 1 | 1 |
Cosmos (root) | 1 | 3 | 268 | 120 |
Cosmos (/view.gmi) | 6 | 1 | 10 | 7 |
Haiku (root) | 88 | 82 | 1 | 3 |
Haiku (/get-haiku) | 9 | 3 | 15 | 716 |
Station | 1 | 1 | 272 | 221 |
AuraGem Music | 1 | 2 | 3 | 1 |
Project Gemini | 1 | 1 | 1 | 300+ |
Geddit | 1 | 2 | 2 | 2 |
Astrobotany | 1 | 1 | 1 | 2 |
Rocketcaster | 1 | 1 | 2 | 26 |
Gemini Radio | 1 | 5 | 300+ | 300+ |
sggs | 5 | 4 | 1 | 5 |
Konpeito | 1 | 86 | 1 | 1 |
Gemipedia | 2 | 34 | 1 | 21 |
Capcom | 1 | 4 | 1 | 1 |
--------------------|------|----------------|---------|------------------|
Averages | 7.29 | 14.12 | 56.53 | 102.24 |
For this category, we are looking for specific URLs that are given back when we input a username/nickname. The desired URLs are profiles on social networks (particularly Station) as well as the person's own capsule. The study results below show the first position of the URL, within the search results, that was desired.
| TLGS | AuraGem Search | Kennedy | geminispace.info |
--------------------|------|----------------|---------|------------------|
krixano | 5 | 1 | 300+ | 3 |
nytpu | 2 | 1 | 5 | 9 |
benk (tilde.team) | 1 | 3 | 207 | 46 |
benk (station) | 3 | 2 | 1 | 1 |
solderpunk | 1 | 2 | 1 | 1 |
skyjake | 1 | 1 | 15 | 1 |
skyjake (station) | 4 | 27 | 26 | 3 |
--------------------|------|----------------|---------|------------------|
Averages | 2.43 | 5.29 | 79.29 | 9.14 |
For this cateogry, we are looking at the *number* of URLs within the top 30 results that adhere to the given category. For example, a category of Aggregator means we are looking for how many aggregators are listed within the top 30 results. A category of Podcast and Dictionary were taken to mean capsules that provide podcasts or dictionaries. None of the results below include pages that just *discuss* aggregators, podcasts, dictionaries, or proxies. Lastly, proxies include any capsules that proxy from gemini to http or vice versa.
| TLGS | AuraGem Search | Kennedy | geminispace.info |
------------|------|----------------|---------|------------------|
Aggregator | 6 | 4 | 5 | 0 |
Podcast | 4 | 3 | 4 | 3 |
Dictionary | 8 | 9 | 5 | 1 |
Proxy | 4 | 3 | 4 | 1 |
------------|------|----------------|---------|------------------|
Averages | 5.5 | 4.75 | 4.5 | 1.25 |
For the final type of Part 1, we are looking at the earliest position of any URL that contains the desired text. However, these results are not as useful because of the very low sample size; I only analyzed two texts because I could not think of that many texts that are both served on geminispace and searchable. However, what is useful is using these terms to see how the FTS and ranking of each search engine works with regard to descendant pages (/tanakh/chapter1 being a descendant of /tanakh, for example). It was often the case that the "root" (or, ancestor) page of the text was well under many descendant pages in the search results for TLGS and Kennedy, whereas AuraGem prioritized the root over its descendants for these particular search queries. Geminispace.info is not included because these particular pages were not crawled by its crawler.
Futher investigation in this area is needed.
| TLGS | AuraGem Search | Kennedy |
--------|------|----------------|---------|
Tanakh | 101 | 1 | 1 |
Talmud | 300+ | 1 | 300+ |
Disclaimer: While both of these texts are provided by AuraGem, AuraGem Search in no way artificially bosts any content from the AuraGem capsule.
Before we make any definitive conclusions, we must remember that the four types of searches above prioritize Exactness, Textual and Category Relevancy, and Popularity. Popularity can affect Exactness if one is searching for a specific capsule or document that is popular. The chances of someone searching for an exact capsule that is popular is much higher than one that is not popular. Additionally, the sample sizes used here are very low and are not necessarily representative of most search queries that one will use.
For the results above:
In Part II, I plan to discuss Recency, Balance, and Broadness, and what types of searches might lend more toward those attributes. These results are far from perfect, but hopefully they are a start that can help developers improve their search engines.
I made a few other observations while working on this: