-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Export] REGEXP
restriction fails on empty list
#1183
Labels
bug
Something isn't working
Comments
Fix for future PR class ExportSelection(SpyglassMixin, dj.Manual):
...
def _add_externals_to_restr_graph(
self, restr_graph: RestrGraph, key: dict
) -> RestrGraph:
"""Add external tables to a RestrGraph for a given restriction/key.
Tables added as nodes with restrictions based on file paths. Names
added to visited set to appear in restr_ft obj bassed to SQLDumpHelper.
Parameters
----------
restr_graph : RestrGraph
A RestrGraph object to add external tables to.
key : dict
Any valid restriction key for ExportSelection.Table
Returns
-------
restr_graph : RestrGraph
The updated RestrGraph
"""
if raw_files := self._list_raw_files(key):
raw_tbl = self._externals["raw"]
raw_name = raw_tbl.full_table_name
raw_restr = "filepath in ('" + "','".join(raw_files) + "')"
restr_graph.graph.add_node(raw_name, ft=raw_tbl, restr=raw_restr)
restr_graph.visited.add(raw_name)
if analysis_files := self._list_analysis_files(key):
analysis_tbl = self._externals["analysis"]
analysis_name = analysis_tbl.full_table_name
# to avoid issues with analysis subdir, we use REGEXP
# this is slow, but we're only doing this once, and future-proof
analysis_restr = "filepath REGEXP '" + "|".join(analysis_files) + "'"
restr_graph.graph.add_node(
analysis_name, ft=analysis_tbl, restr=analysis_restr
)
restr_graph.visited.add(analysis_name)
return restr_graph |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A late addition to the export process was the inclusion of externals tables using a
REGEXP
restrictionspyglass/src/spyglass/common/common_usage.py
Lines 222 to 224 in 239f2a5
If there are no such files, the empty regex will cause an error
The text was updated successfully, but these errors were encountered: