Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Export] REGEXP restriction fails on empty list #1183

Closed
CBroz1 opened this issue Nov 13, 2024 · 1 comment · Fixed by #1192
Closed

[Export] REGEXP restriction fails on empty list #1183

CBroz1 opened this issue Nov 13, 2024 · 1 comment · Fixed by #1192
Assignees
Labels
bug Something isn't working

Comments

@CBroz1
Copy link
Member

CBroz1 commented Nov 13, 2024

A late addition to the export process was the inclusion of externals tables using a REGEXP restriction

analysis_restr = ( # filepaths have analysis subdir. regexp substrings
"filepath REGEXP '" + "|".join(self._list_analysis_files(key)) + "'"
) # regexp is slow, but we're only doing this once, and future-proof

If there are no such files, the empty regex will cause an error

@CBroz1
Copy link
Member Author

CBroz1 commented Nov 13, 2024

Fix for future PR

class ExportSelection(SpyglassMixin, dj.Manual):
    ...

    def _add_externals_to_restr_graph(
        self, restr_graph: RestrGraph, key: dict
    ) -> RestrGraph:
        """Add external tables to a RestrGraph for a given restriction/key.

        Tables added as nodes with restrictions based on file paths. Names
        added to visited set to appear in restr_ft obj bassed to SQLDumpHelper.

        Parameters
        ----------
        restr_graph : RestrGraph
            A RestrGraph object to add external tables to.
        key : dict
            Any valid restriction key for ExportSelection.Table

        Returns
        -------
        restr_graph : RestrGraph
            The updated RestrGraph
        """
        if raw_files := self._list_raw_files(key):
            raw_tbl = self._externals["raw"]
            raw_name = raw_tbl.full_table_name
            raw_restr = "filepath in ('" + "','".join(raw_files) + "')"
            restr_graph.graph.add_node(raw_name, ft=raw_tbl, restr=raw_restr)
            restr_graph.visited.add(raw_name)

        if analysis_files := self._list_analysis_files(key):
            analysis_tbl = self._externals["analysis"]
            analysis_name = analysis_tbl.full_table_name
            # to avoid issues with analysis subdir, we use REGEXP
            # this is slow, but we're only doing this once, and future-proof
            analysis_restr = "filepath REGEXP '" + "|".join(analysis_files) + "'"
            restr_graph.graph.add_node(
                analysis_name, ft=analysis_tbl, restr=analysis_restr
            )
            restr_graph.visited.add(analysis_name)

        return restr_graph

@CBroz1 CBroz1 self-assigned this Nov 13, 2024
@CBroz1 CBroz1 added the bug Something isn't working label Nov 13, 2024
CBroz1 added a commit to CBroz1/spyglass that referenced this issue Nov 20, 2024
@CBroz1 CBroz1 linked a pull request Nov 21, 2024 that will close this issue
7 tasks
@CBroz1 CBroz1 mentioned this issue Nov 27, 2024
7 tasks
edeno pushed a commit that referenced this issue Dec 5, 2024
* #1175

* #1185

* #1183

* Fix circular import

* #1163

* #1105

* Fix failing tests, close download subprocesses

* WIP: fix decode changes spikesort tests

* Fix fickle test

* Revert typo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant