Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download only whosonfirst-data-latest.db #437

Closed
tadjik1 opened this issue Apr 17, 2019 · 2 comments · Fixed by #487
Closed

download only whosonfirst-data-latest.db #437

tadjik1 opened this issue Apr 17, 2019 · 2 comments · Fixed by #487

Comments

@tadjik1
Copy link

tadjik1 commented Apr 17, 2019

Hi there,
I'm working currently on infrastructure for pip-service and it requires sqlite dump to be downloaded first. It turns out that this module downloads more than 150 dbs from dist.whosonfirst, even though later during pip-service start only 1 main file whosonfirst-data-latest.db is used. Is there a way to download only 1 file contains all data and not all of them?

ref: https://github.com/pelias/wof-admin-lookup/blob/master/src/pip/readStream.js#L21

@orangejulius
Copy link
Member

orangejulius commented Apr 17, 2019

Hey @tadjik1,

The download script in the pip-service is configured to only download administrative data, so it should download only the file you need.

Have you given that one a try? If it downloads extra data, we want to know about it, since that's the whole point of that script.

@missinglink
Copy link
Member

missinglink commented Apr 17, 2019

Are the other 149 databases for postcodes?

If the config setting imports.whosonfirst.importPostalcodes is set to true (defaults to false) then I believe they will be included in the list of files to download.

Postcodes are not loaded in to PIP, although there is a long discussion about why that it the case, I won't go in to detail here.

The postcodes are imported in to elasticsearch under the postalcode layer.

orangejulius added a commit that referenced this issue Jan 9, 2020
The country-specific SQLite DBs currently available for download from
https://dist.whosonfirst.org have a few integrity issues. This causes
problems in Pelias such as addresses in NYC or Berlin not having proper
admin information.

The planet-wide SQLite DB, while a little out of date, is overall more
valid.

While in the long term we probably want to utilize the country-specific
DBs, for now it makes sense to filter them out.

Fixes #469
Connects #460
Fixes #437
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants