💾 Archived View for jb55.com › ward.asia.wiki.org › over-the-horizon-later captured on 2022-01-08 at 14:03:30. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-12-04)
-=-=-=-=-=-=-
We've created and run a federation wide scrape without instrumentation to tell us what it is doing. We've run some preliminary tallies and now repeat them to understand what the program has done overnight.
We report the page count and domain name of sites found to be online and reporting pages in their sitemaps. page
We fetch once per second so we can predict runtime.
Search Results 51 217 / ((3600 units) / hour) = 14.2269444 hours
From file system creation dates we see we started at Feb 21 20:49, about 3 hours to midnight. we now estimate completion at 14-3 = 11 am today.
So now numbers at 6 am, 9/14 = 64% done.
We refresh our counts of sites visited and welcome pages found to compute 126/213 = 59% of sites edit welcome.
We count unique slugs over all sites.
We count unique sites named by fork or reference.
Is this program still working as expected? Let's look at what it is doing right now and see if it is producing one new file every second. Yes, five new files in five seconds.