Unable to start pip after download with sqlite=true #454
Comments
orangejulius
added a commit
that referenced
this issue
Aug 2, 2019
The creation of the `spr_obsolete` index is problematic for the PIP service and any of the admin lookup workers running in the importers. Because there are always multiple processes doing admin lookup, and each one tries to create the index if it doesn't exist, most end up failing. SQLite only allows a single write lock on the DB, so all but one will fail to achieve it. However, in my testing, all it takes to solve this gracefully is to wrap the index creation in a try/catch block. The index will be created by one of the worker processes, and all subsequent queries appear to have the performance improvements of the index (about 30-50% faster time to load admin data). Supercedes #431 Fixes #453 Connects #454 (we should not actually remove the index creation quite yet)
orangejulius
added a commit
that referenced
this issue
Aug 2, 2019
The creation of the `spr_obsolete` index is problematic for the PIP service and any of the admin lookup workers running in the importers. Because there are always multiple processes doing admin lookup, and each one tries to create the index if it doesn't exist, most end up failing. SQLite only allows a single write lock on the DB, so all but one will fail to achieve it. However, in my testing, all it takes to solve this gracefully is to wrap the index creation in a try/catch block. The index will be created by one of the worker processes, and all subsequent queries appear to have the performance improvements of the index (about 30-50% faster time to load admin data). Supersedes #431 Fixes #453 Connects #454 (we should not actually remove the index creation quite yet)
orangejulius
added a commit
that referenced
this issue
Aug 2, 2019
The creation of the `spr_obsolete` index is problematic for the PIP service and any of the admin lookup workers running in the importers. Because there are always multiple processes doing admin lookup, and each one tries to create the index if it doesn't exist, most end up failing. SQLite only allows a single write lock on the DB, so all but one will fail to achieve it. However, in my testing, all it takes to solve this gracefully is to wrap the index creation in a try/catch block. The index will be created by one of the worker processes, and all subsequent queries appear to have the performance improvements of the index (about 30-50% faster time to load admin data). Supersedes #431 Fixes #454 Connects #453 (we should not actually remove the index creation quite yet)
|
Hi @otbutz, Let us know if there are still any problems. |
|
Thanks, will give it a spin ;) |
|
It works flawlessly now 👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Executing the following snippet with
sqliteandimportPostalcodesenabled:The download finishes properly:
But the sqlite DBs seem to be locked:
I tried it multiple times even with a
ps -auxbetweendownloadandstartbut it seems that all processes start bydownloadhave properly finished.Are there any lock files i'm missing?
Edit: noticed that i posted parts the log of my first run in which i used the pip-service download instead of the whosonfirst one.
The text was updated successfully, but these errors were encountered: