New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Instead of parsing the WARC files, process the WAT files #5

Open

laxika opened this issue Oct 26, 2021 · 0 comments

Labels

Milestone

Member

laxika commented Oct 26, 2021

Instead of downloading and parsing the WARC files, we could simply get the links from the WAT files. A lot less network and CPU intensive.

https://commoncrawl.org/the-data/get-started/#WAT-Format
https://gist.github.com/Smerity/e750f0ef0ab9aa366558#file-bbc-pretty-wat

laxika added the enhancement label

laxika added this to the 1.1.0 milestone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment