💾 Archived View for freeshell.de › tldr › scrapy.gmi captured on 2023-07-10 at 14:56:00. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2022-04-28)

-=-=-=-=-=-=-

scrapy

Web-crawling framework.

More information.

• Create a project:

scrapy startproject {project_name}

• Create a spider (in project directory):

scrapy genspider {spider_name} {website_domain}

• Edit spider (in project directory):

scrapy edit {spider_name}

• Run spider (in project directory):

scrapy crawl {spider_name}

• Fetch a webpage as Scrapy sees it and print the source to stdout:

scrapy fetch {url}

• Open a webpage in the default browser as Scrapy sees it (disable JavaScript for extra fidelity):

scrapy view {url}

• Open Scrapy shell for URL, which allows interaction with the page source in a Python shell (or IPython if available):

scrapy shell {url}

Copyright © 2014—present the tldr-pages team and contributors.

This work is licensed under the Creative Commons Attribution 4.0 International License (CC-BY).

CC-BY