Kennedy Changelog

2023-03-19

Massive improvement to Delorean, making it store a history of cached versions of content, and not just the copy found in the most recent crawl

Redesign of crawler code which improved speed of the crawler. Robots.txt files are downloaded ondemand instead of requiring a pre-flight step, ensuring that all capsules with Robots.txt are respected

Updated "Page Info" view to support image meta data (dimensions, format, text used in index)
Updated Delorean to work show cached images and other cached, non-text content

Added image search! Images are indexed based on the text in their file path, as well as the text in all their inbound links

Updated searched Also include snippet for Gemipedia about the search query and link to Gemipedia entry

Added a "Page Info" view that shows title, language, # lines, size of response, and incoming/outbound links to a page
Improved Delorean by adding a "View Cached" link for each page in the "Page Info" view.
Streamlined the meta data shown on the search results page into a single line and made it a link to "Page Info" view.
Improved "title" extraction code to use the first header encountered, regardless of level, or alt text from the first pre-formatted section.

Added Delorean which lets you view cached content from most recent scan by providing a URL