Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export failure from retriction graph whith multiple restrictions to the same leaf #1170

Closed
samuelbray32 opened this issue Oct 24, 2024 · 2 comments
Assignees
Labels
bug Something isn't working common

Comments

@samuelbray32
Copy link
Collaborator

Describe the bug

  • The export process depends on tracing table entries upstream using a restriction graph here.
  • The set of leaves come from the unique entries in the ExportSelection.Table
    • HOWEVER: There can be multiple leaves for the same table but different restrictions
  • When the restriction graph is made, the leaves are added one at a time without cascading, and then cascaded
  • When each leaf of the same table is added it replaces the restriction on that leaf
  • This means only the last restriction added from ExportSelection matters and is traced, resulting in missing entries in the export.

To Reproduce
In the franklab database:

from spyglass.common.common_usage import unique_dicts
paper_key = {"paper_id": "oct2024_testing_v4"}
table_restr = {"table_name":"`lfp_electrode`.`l_f_p_electrode_group__l_f_p_electrode`"}
table_key = {**paper_key, **table_restr}
ExportSelection() * ExportSelection.Table & table_key # contains 3 unique entries, with 3 values in `lfp_electrode`.`l_f_p_electrode_group__l_f_p_electrode`
ExportSelection().get_restr_graph(table_key).leaf_ft # has only a single entry in the table

Solution ideas

  1. In ExportSelection.get_restr_graph combine the uniques restrictions for the same table using a dj.OrList()
@samuelbray32 samuelbray32 added bug Something isn't working common labels Oct 24, 2024
@samuelbray32
Copy link
Collaborator Author

Additional note: the restriction strings placed in the ExportSelection.Table() entries do not all have balanced parentheses (saw multiple with dangling parentheses on the end. This causes issues whith the OR logic when applying them together as a list.

Code to sanitize these:

cleaned_restriction = (
        "("*restriction.count( "(" )
       + restriction
       + ")"*restriction.count( ")" )
)

@CBroz1 CBroz1 self-assigned this Oct 28, 2024
@CBroz1
Copy link
Member

CBroz1 commented Oct 28, 2024

I ran update on the restrictions captured here to remove unbalanced parens, and found the leaf node updated appropriately. I've pushed an update to #1164 that should fix the unbalanced parens issue, and I've also rerun the Export.populate/resulting shell script. The _Populate.sql is updated in the relevant export folder

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working common
Projects
None yet
Development

No branches or pull requests

2 participants