4 upvotes, 1 direct replies (showing 1)
View submission: Anyone notice that question megathreads aren't picked up by google?
Google is going to index threads, not individual comments.
And not even the entire thread, but it'll go to the base URL and index whatever shows up -- and if anything is hidden behind a '"load more comments" or a "continue thread", well, it won't get indexed.
So small threads would get fully indexed, but large threads would only get partially indexed.
Now, this assumes that reddit isn't giving google specially formatted pages that include everything -- and they could -- but if I'm right, it would probably explain what you're referring to.
Comment by Ajreil at 04/11/2024 at 20:17 UTC*
1 upvotes, 1 direct replies
The "load more comments" button is a hyperlink to a different page, so webcrawlers should be able to navigate it just fine.
Webcrawlers are designed to be explorative rather than thorough, so unless Google is specifically trying to archive all of Reddit it it will miss a lot of content.
(Google did buy all of Reddit's data but that's for LLM training, not search)