Add pytests to position (LorenFrankLab#966)

* WIP: Add vid downloader, start position pytests

* WIP: Full coverage on common_position.py

* WIP: Start add tests for DLC

* WIP: pytests for position 2

* WIP: Align yml files

* WIP: Add DLC Centroid tests

* WIP: tests for centoid, model, orietation

* WIP: Subpackage coverage 72%

* WIP: Run tests on main

* WIP: Add prints to debug data download

* WIP: remove redundant slash

* WIP: download full directory

* WIP: let pytests down dlc items

* WIP: gh-action managed container

* WIP: use mysql as service

* WIP: fix container ID

* WIP: revise no-docker arg name

* WIP: Box vars available to test step

* WIP: Only first test

* WIP: remove duplicate installs from conda deps -> pyproject

* WIP: Add cuda driver

* WIP: Cuda method selective

* WIP: unpin DLC

* WIP: parameterize skipping dlc tests

* WIP: Add back Checkout

* WIP: All download in action

* WIP: custom wget func

* WIP: revise data downloader

* WIP: Run all

* WIP: no docker container id in test database_settings

* WIP: revise utils tests for --no-docker

* WIP: add password to DatabaseSettings no-docker run

* WIP: Spellcheck

* WIP: debug mysql add role command

* WIP: fix typo

* 🎉 : Update changelog

* Edit PR template to remind of local tests

* Revert conda-installed dependencies

* Return mountainsort to conda pip list

* Upgrade action versions per node.js 16 deprecation

.github/pull_request_template.md

@@ -33,5 +33,6 @@ Table.alter() # Comment regarding the change
 - [ ] If release, I have updated the `CITATION.cff`
 - [ ] This PR makes edits to table definitions: (yes/no)
 - [ ] If table edits, I have included an `alter` snippet for release notes.
+- [ ] If this PR makes changes to positon, I ran the relevant tests locally.
 - [ ] I have updated the `CHANGELOG.md` with PR number and description.
 - [ ] I have added/edited docs/notebooks to reflect the changes

.github/workflows/publish-docs.yml

@@ -5,6 +5,7 @@ on:
       - "*.*.*" # For docs bump, use X.X.XaX
     branches:
       - test_branch
+  workflow_dispatch: # Manually trigger with 'Run workflow' button
 
 permissions:
   contents: write

.github/workflows/test-conda.yml

@@ -1,4 +1,4 @@
-name: Test conda env and run tests
+name: Tests
 
 on:
   push:
@@ -7,52 +7,74 @@ on:
       - '!documentation'
   schedule:  # once a day at midnight UTC
     - cron: '0 0 * * *'
+  workflow_dispatch: # Manually trigger with 'Run workflow' button
+
+concurrency: # Replace Cancel Workflow Action
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
 
 jobs:
   run-tests:
-    runs-on: ${{ matrix.os }}
+    runs-on: ubuntu-latest
     defaults:
      run:
        shell: bash -l {0}
-    strategy:
-      matrix:
-        os: [ubuntu-latest]  #, macos-latest, windows-latest]
     env:
-      OS: ${{ matrix.os }}
-      PYTHON: '3.8'
+      OS: ubuntu-latest
+      PYTHON: '3.9'
+      UCSF_BOX_TOKEN: ${{ secrets.UCSF_BOX_TOKEN }} # for download and testing
+      UCSF_BOX_USER: ${{ secrets.UCSF_BOX_USER }}
+    services:
+      mysql:
+        image: datajoint/mysql:8.0
+        env: # args: mysql -h 127.0.0.1 -P 3308 -uroot -ptutorial -e "CMD;"
+          MYSQL_DATABASE: localhost
+          MYSQL_ROOT_PASSWORD: tutorial
+        ports:
+          - 3308:3306
+        options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3
     steps:
-    - name: Cancel Workflow Action
-      uses: styfle/[email protected]
-      with:
-        access_token: ${{ github.token }}
-        all_but_latest: true
     - name: Checkout
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ env.PYTHON }}
-    - name: Set up conda environment
-      uses: conda-incubator/setup-miniconda@v2
+    - name: Set up conda
+      uses: conda-incubator/setup-miniconda@v3
       with:
         activate-environment: spyglass
         environment-file: environment.yml
         miniforge-variant: Mambaforge
         miniforge-version: latest
-    - name: Install spyglass
+        use-mamba: true
+    - name: Install apt dependencies
       run: |
-        pip install -e .[test]
+        sudo apt-get update # First mysql options
+        sudo apt-get install mysql-client libmysqlclient-dev libgirepository1.0-dev -y
+        sudo apt-get install ffmpeg libsm6 libxext6 -y # non-dlc position deps
+    - name: Run pip install for test deps
+      run: | 
+        pip install --quiet .[test]
     - name: Download data
       env:
-        UCSF_BOX_TOKEN: ${{ secrets.UCSF_BOX_TOKEN }}
-        UCSF_BOX_USER: ${{ secrets.UCSF_BOX_USER }}
-        WEBSITE: ftps://ftp.box.com/trodes_to_nwb_test_data/minirec20230622.nwb
+        BASEURL: ftps://ftp.box.com/trodes_to_nwb_test_data/
+        NWBFILE: minirec20230622.nwb # Relative to Base URL
+        VID_ONE: 20230622_sample_01_a1/20230622_sample_01_a1.1.h264
+        VID_TWO: 20230622_sample_02_a1/20230622_sample_02_a1.1.h264
         RAW_DIR: /home/runner/work/spyglass/spyglass/tests/_data/raw/ 
+        VID_DIR: /home/runner/work/spyglass/spyglass/tests/_data/video/ 
       run: |
-        mkdir -p $RAW_DIR
-        wget --recursive --no-verbose --no-host-directories --no-directories \
-        --user $UCSF_BOX_USER --password $UCSF_BOX_TOKEN \
-        -P $RAW_DIR $WEBSITE
+        mkdir -p $RAW_DIR $VID_DIR
+        wget_opts() { # Declare func with download options
+          wget \
+            --recursive --no-verbose --no-host-directories --no-directories \
+            --user "$UCSF_BOX_USER" --password "$UCSF_BOX_TOKEN" \
+            -P "$1" "$BASEURL""$2"
+        }
+        wget_opts $RAW_DIR $NWBFILE
+        wget_opts $VID_DIR $VID_ONE
+        wget_opts $VID_DIR $VID_TWO
     - name: Run tests
       run: |
-        pytest -rP  # env vars are set within certain tests
+        pytest --no-docker --no-dlc 

.github/workflows/test-package-build.yml

@@ -13,32 +13,33 @@ on:
     branches:
       - master
       - maint/*
+  workflow_dispatch: # Manually trigger with 'Run workflow' button
 defaults:
   run:
     shell: bash
 jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: 3.9
       - run: pip install --upgrade build twine
       - name: Build sdist and wheel
         run: python -m build
       - run: twine check dist/*
       - name: Upload sdist and wheel artifacts
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: dist
           path: dist/
       - name: Build git archive
         run: mkdir archive && git archive -v -o archive/archive.tgz HEAD
       - name: Upload git archive artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: archive
           path: archive/
@@ -51,13 +52,13 @@ jobs:
     steps:
       - name: Download sdist and wheel artifacts
         if: matrix.package != 'archive'
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
           name: dist
           path: dist/
       - name: Download git archive artifact
         if: matrix.package == 'archive'
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
           name: archive
           path: archive/
@@ -74,13 +75,9 @@ jobs:
       - name: Install sdist
         if: matrix.package == 'sdist'
         run: pip install dist/*.tar.gz
-      - name: Install archive
-        if: matrix.package == 'archive'
+      - name: Install archive # requires tag
+        if: matrix.package == 'archive' && startsWith(github.ref, 'refs/tags/')
         run: pip install archive/archive.tgz
-      # - name: Install test extras
-      #   run: pip install spyglass[test]
-      # - name: Run tests
-      #   run: pytest --doctest-modules -v --pyargs spyglass
   publish:
     name: Upload release to PyPI
     runs-on: ubuntu-latest
@@ -92,7 +89,7 @@ jobs:
       id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
     if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
     steps:
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: dist
           path: dist/

.gitignore

@@ -59,6 +59,7 @@ coverage.xml
 *.cover
 .hypothesis/
 .pytest_cache/
+tests/_data/*
 
 # Translations
 *.mo

CHANGELOG.md

@@ -14,11 +14,18 @@
 - Add long-distance restrictions via `<<` and `>>` operators. #943, #969
 - Fix relative pathing for `mkdocstring-python=>1.9.1`. #967, #968
 - Clean up old `TableChain.join` call in mixin delete. #982
+- Add pytests for position pipeline, various `test_mode` exceptions #966
+- Migrate `pip` dependencies from `environment.yml`s to `pyproject.toml` #966
 
 ### Pipelines
 
+- Common
+    - `PositionVideo` table now inserts into self after `make` #966
+- Decoding: Default values for classes on `ImportError` #966
 - DLC
     - Allow dlc without pre-existing tracking data #973, #975
+    - Raise `KeyError` for missing input parameters across helper funcs #966
+    - `DLCPosVideo` table now inserts into self after `make` #966
 
 ## [0.5.2] (April 22, 2024)
 

CODE_OF_CONDUCT.md

@@ -1,12 +1,11 @@
-
 # Contributor Covenant Code of Conduct
 
 ## Our Pledge
 
 We as members, contributors, and leaders pledge to make participation in our
 community a harassment-free experience for everyone, regardless of age, body
 size, visible or invisible disability, ethnicity, sex characteristics, gender
-identity and expression, level of experience, education, socio-economic status,
+identity and expression, level of experience, education, socioeconomic status,
 nationality, personal appearance, race, caste, color, religion, or sexual
 identity and orientation.
 
@@ -18,24 +17,24 @@ diverse, inclusive, and healthy community.
 Examples of behavior that contributes to a positive environment for our
 community include:
 
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
-  and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the overall
-  community
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes,
+    and learning from the experience
+- Focusing on what is best not just for us as individuals, but for the overall
+    community
 
 Examples of unacceptable behavior include:
 
-* The use of sexualized language or imagery, and sexual attention or advances of
-  any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email address,
-  without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
+- The use of sexualized language or imagery, and sexual attention or advances of
+    any kind
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email address,
+    without their explicit permission
+- Other conduct which could reasonably be considered inappropriate in a
+    professional setting
 
 ## Enforcement Responsibilities
 
@@ -61,8 +60,8 @@ representative at an online or offline event.
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
 reported to the community leaders responsible for enforcement at
-[email protected].
-All complaints will be reviewed and investigated promptly and fairly.
+[email protected]. All complaints will be reviewed and investigated
+promptly and fairly.
 
 All community leaders are obligated to respect the privacy and security of the
 reporter of any incident.
@@ -120,14 +119,14 @@ version 2.1, available at
 [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
 
 Community Impact Guidelines were inspired by
-[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+[Mozilla's code of conduct enforcement ladder][mozilla coc].
 
 For answers to common questions about this code of conduct, see the FAQ at
-[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/faq][faq]. Translations are available at
 [https://www.contributor-covenant.org/translations][translations].
 
+[faq]: https://www.contributor-covenant.org/faq
 [homepage]: https://www.contributor-covenant.org
-[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
-[Mozilla CoC]: https://github.com/mozilla/diversity
-[FAQ]: https://www.contributor-covenant.org/faq
+[mozilla coc]: https://github.com/mozilla/diversity
 [translations]: https://www.contributor-covenant.org/translations
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html

docs/src/misc/mixin.md

@@ -21,6 +21,11 @@ schema = dj.schema("my_schema")
 @schema
 class MyOldTable(dj.Manual):
     pass
+
+
+@schema
+class MyNewTable(SpyglassMixin, dj.Manual):
+    pass
 ```
 
 **NOTE**: The mixin must be the first class inherited from in order to override
@@ -60,10 +65,10 @@ key and `>>` as a shorthand for `restrict_by` a downstream key.
 from spyglass.example import AnyTable
 
 AnyTable() << 'upstream_attribute="value"'
-AnyTable() >> 'downsteam_attribute="value"'
+AnyTable() >> 'downstream_attribute="value"'
 
 # Equivalent to
-AnyTable().restrict_by('downsteam_attribute="value"', direction="down")
+AnyTable().restrict_by('downstream_attribute="value"', direction="down")
 AnyTable().restrict_by('upstream_attribute="value"', direction="up")
 ```
 
@@ -136,7 +141,7 @@ function, `delete_downstream_merge`, to handle this, which is run by default
 when calling `delete`.
 
 `delete_downstream_merge`, also aliased as `ddm`, identifies all Merge tables
-downsteam of where it is called. If `dry_run=True`, it will return a list of
+downstream of where it is called. If `dry_run=True`, it will return a list of
 entries that would be deleted, otherwise it will delete them.
 
 Importantly, `delete_downstream_merge` cannot properly interact with tables that
@@ -156,7 +161,7 @@ from spyglass.example import MyMerge
 restricted_nwbfile.delete_downstream_merge(reload_cache=True, dry_run=False)
 ```
 
-Because each table keeps a cache of downsteam merge tables, it is important to
+Because each table keeps a cache of downstream merge tables, it is important to
 reload the cache if the table has been imported after the cache was created.
 Speed gains can also be achieved by avoiding re-instancing the table each time.
 

environment.yml

@@ -10,9 +10,10 @@ name: spyglass
 channels:
   - conda-forge
   - defaults
-  # - pytorch # dlc-only
   - franklab
   - edeno
+  # - pytorch # dlc-only
+  # - anaconda  # dlc-only, for cudatoolkit
 dependencies:
   - bottleneck
   # - cudatoolkit=11.3 # dlc-only
@@ -36,15 +37,6 @@ dependencies:
   # - torchvision # dlc-only
   - track_linearization>=2.3
   - pip:
-      - "black[jupyter]"
-      - datajoint>=0.13.6
-      # - deeplabcut<2.3.0 # dlc-only
       - ghostipy # for common_filter
-      - ndx-franklab-novela>=0.1.0
       - mountainsort4
-      - panel<=1.3.5 # See panel #6325
-      - pubnub<=6.4.0
-      - pynwb>=2.2.0,<3
-      - sortingview>=0.11
-      - spikeinterface>=0.99.1,<0.100
       - .