Skip to content

Commit

Permalink
feat(download): Filter out country-specific SQLite DBs
Browse files Browse the repository at this point in the history
The country-specific SQLite DBs currently available for download from
https://dist.whosonfirst.org have a few integrity issues. This causes
problems in Pelias such as addresses in NYC or Berlin not having proper
admin information.

The planet-wide SQLite DB, while a little out of date, is overall more
valid.

While in the long term we probably want to utilize the country-specific
DBs, for now it makes sense to filter them out.

Fixes #469
Connects #460
Fixes #437
  • Loading branch information
orangejulius committed Jan 9, 2020
1 parent 31d55e8 commit 6449dcb
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions utils/download_sqlite_all.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ function download(callback) {
const content = JSON.parse(downloadFileSync(`${wofDataHost}/sqlite/inventory.json`))
// Only latest compressed files
.filter(e => e.name_compressed.indexOf('latest') >= 0)
// country-specific SQLite bundles are broken, filter them out for now
.filter(e => !e.name.match(/admin-..-/))
// Postalcodes only when importPostalcodes is ture and without --admin-only arg
.filter(e => e.name_compressed.indexOf('postalcode') < 0 ||
(config.imports.whosonfirst.importPostalcodes && process.argv[2] !== '--admin-only'))
Expand Down

0 comments on commit 6449dcb

Please sign in to comment.