Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance optimization in precomp_annotations.subdivide #819

Merged
merged 3 commits into from
Dec 20, 2024

Conversation

JoeStrout
Copy link
Contributor

(substantially speeds up finalization of large annotation files)

Copy link

codecov bot commented Oct 28, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (c40fc57) to head (a93f53d).
Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #819   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          142       142           
  Lines         5958      6011   +53     
=========================================
+ Hits          5958      6011   +53     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JoeStrout JoeStrout force-pushed the joe/opt-precomp_annotations branch 4 times, most recently from 37be3ee to ce48c45 Compare November 7, 2024 20:40
@nkemnitz
Copy link
Collaborator

Is this one ready for review?

@JoeStrout
Copy link
Contributor Author

Yep!

@JoeStrout JoeStrout force-pushed the joe/opt-precomp_annotations branch from 1392623 to 619b7e1 Compare December 13, 2024 16:42
@JoeStrout
Copy link
Contributor Author

Hey @nkemnitz , I pushed some additional changes, but I now hope to quit mucking with this file for a while (and if I do, it'll be in a new branch). Could you please review?

Copy link
Collaborator

@nkemnitz nkemnitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could simplify the code and get a speed boost by using batch requests for fetching size and data using CloudFiles.

Other than that, the single chunk case doesn't seem to be covered by tests, yet.

zetta_utils/db_annotations/precomp_annotations.py Outdated Show resolved Hide resolved
zetta_utils/db_annotations/precomp_annotations.py Outdated Show resolved Hide resolved
zetta_utils/db_annotations/precomp_annotations.py Outdated Show resolved Hide resolved
@JoeStrout JoeStrout force-pushed the joe/opt-precomp_annotations branch from fb681f4 to a93f53d Compare December 19, 2024 15:01
@nkemnitz nkemnitz merged commit 793a75d into main Dec 20, 2024
10 checks passed
@nkemnitz nkemnitz deleted the joe/opt-precomp_annotations branch December 20, 2024 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants