This cookiecutter sets up a new scrapy project for you. It defers in a few ways from what you would get from scrapy startproject
to align
with DataMade's scraping practices.
scrapy crawl
will exit after the first exception.- if an error is encountered,
scrapy crawl
will exit with a non-zero exit code. - the project contains a
setup.py
file wich will allow you to install the scraper withpip
like a normal package. when installed this way, you can runscrapy crawl your_spider_name
without having to be in a scrapy project.