An inital scrape can take the better part of a day. Press 'start' to begin. Shift-'start' to do one cycle at a time.
We start scrapes hourly skipping hours if they run long.
We fetch sitemaps for one site and then discover more.
We fetch page json to index and inspect for more sites.
See Queue Stats while running.
See also Active Sites, Failed Sites, Crazy Business