Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connect to_hierarchical_clusters into the process with new Results and insert_results #34

Merged
merged 23 commits into from
Jan 9, 2025
Merged

Connect to_hierarchical_clusters into the process with new Results and insert_results #34

merged 23 commits into from
Jan 9, 2025

Conversation

wpfl-dbt
Copy link
Collaborator

@wpfl-dbt wpfl-dbt commented Dec 20, 2024 •
edited
Loading

Context

It's time to hook up all the different parts. This PR brings all our new client-side changes together.

This isn't intended to be super efficient yet -- just to hook things up.

Changes proposed in this pull request

  • Results replaces ProbabilityResults and ClusterResults
    • to_clusters now just runs client-side and uses IntMap to produce its pseudo-hashes
  • insert_results refactored to deal with the new structure
  • Probabilities.probability changes to SMALLINT to match pa.uint8() on the frontend
  • Adds matchbox.common.logging for console logging with rich, but doesn't move logger instantiation there (yet)
  • Replaces all pa.large_binary() with pa.binary(). Arrow advises against using the large version for our use case

Guidance to review

  • Review Results
  • Review insert_results
  • Review everything else

Relevant links

Checklist:

  • My code follows the style guidelines of this project
  • New and existing unit tests pass locally with my changes

@wpfl-dbt wpfl-dbt changed the title Initial refactor to show rough new structure Connect to_hierarchical_clusters into the process with new Results and insert_results Dec 20, 2024
@wpfl-dbt wpfl-dbt marked this pull request as ready for review January 7, 2025 12:56
@wpfl-dbt wpfl-dbt requested a review from leo-mazzone January 7, 2025 12:57
@wpfl-dbt wpfl-dbt merged commit 2061f9d into feature/new-ingest-process Jan 9, 2025
3 checks passed
@wpfl-dbt wpfl-dbt deleted the feature/new-results branch January 9, 2025 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leo-mazzone leo-mazzone leo-mazzone approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wpfl-dbt @leo-mazzone