Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark that Scraping has been attempted #91

Open
tarrow opened this issue Sep 23, 2016 · 0 comments
Open

Mark that Scraping has been attempted #91

tarrow opened this issue Sep 23, 2016 · 0 comments

Comments

@tarrow
Copy link
Contributor

tarrow commented Sep 23, 2016

PMR has been writing scripts to restart scrapes if they fails for one reason or another. For example if phantom crashes or the machine runs out of memory or we hit a bug in quick scrape.

He would like to be able to know if a scrape has been started but was killed before it succeeded so that it can be removed from the list of input URLs after he restarts.

This might mean placing an empty quickscrape_result.json for example

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant