Move importer code to psycopg3 #3487
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is only used in one place.
Also changes the drop function so that it can drop multiple tables at once.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This resolves the rather odd situation that frontend and backend were depending on different versions of the psycopg library.
The port turned out to be a bigger undertaking than initially planned for. The biggest challenge was moving the indexing code from a select-based approach of waiting for the parallel queries to asyncio-based code. Coroutines are really a beast that is hard to control and getting the code to switch between coroutines at the optimal point in time can be a challenge. But it was worth the effort. The final result is a bit faster than the original code.
The DB helper functions have been moved out of the custom Connection and Cursor classes into freestanding functions. Subclassing Connection and Cursor is not so well supported in psycopg3 anymore.
Indexing now runs multiple times on import if necessary to make sure that all objects are really done.
The other changes are adaptions for the breaking changes in psycopg3. Altogether the version 3 interface feels a lot cleaner. And that is well reflected in the fact that this PR gets rid of a good 500 lines of code.