Change dependencies. See details.

Across `pyproject.toml` and both `environment.yml`:
- Alphabetical sort to allow for easier comparison across toml/yaml files.
- Remove dependencies required by other dependencies to reduce build time.
- Move `black` to `test` dependencies. Could be `dev` instead?
- Move `click` to `test` dependencies. CLI should be its own package.
- Add ruff config to `pyproject.toml`. Flake8 replacement to allow removal of
  `setup.cfg`.
- Remove `tqdm`, as it is required by `datajoint`

Standard environment:
- Add comments for all dependencies specific to position environment to allow
  for easier comparison across yaml files.
- Remove `dask`, as it is required by `ghostipy`
- Remove `hdmf`, as it is required by `pynwb`
- Remove `pymysql`, as it is required by `datajoint`
- Remove `pyyaml`, as it is required by `sortingview`
- Remove `pydotplus` from `pip` section, as it is already in `conda` section

Position environment:
- Rename `_position` or `-position` to `_dlc` or `-dlc` to be more precise.
- Remove `ipython`, `numba`, and `tensorflow`, as they are all required by
  `deeplabcut`
- Add items present in standard environment but not in position environment.

CHANGELOG.md

@@ -21,18 +21,14 @@
 - Update docs to reflect new notebooks. #776
 - Add overview of Spyglass to docs. #779
 - Update linting for Black 24. #808
+- Steamline dependency management. #822 (?)
 
 ### Pipelines
 
 - Spike sorting:
     - Add SpikeSorting V1 pipeline. #651
     - Move modules into spikesorting.v0 #807
     - Add MUA analysis to spike sorting pipeline
-- LFP: Minor fixes to LFPBandV1 populator and `make`. #706, #795
-
-### Pipelines
-
-- Spike sorting: Add SpikeSorting V1 pipeline. #651
 - LFP:
     - Minor fixes to LFPBandV1 populator and `make`. #706, #795
     - LFPV1: Fix error for multiple lfp settings on same data #775

environment.yml

@@ -1,38 +1,53 @@
+# 1. Install a conda distribution.
+#    https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html
+# 2. Run: `mamba env create -f environment.yml`
+# 3. Activate: `conda activate spyglass`
+#
+# (lines intentionally left blank)
+#
+#
 name: spyglass
 channels:
   - conda-forge
   - defaults
+  # - pytorch # dlc-only
   - franklab
   - edeno
 dependencies:
-  - python>=3.9,<3.10
+  - bottleneck
+  # - cudatoolkit=11.3 # dlc-only
+  # - cupy # dlc-only
+  # - dask-cuda # dlc-only
+  # - ffmpeg # dlc-only
+  - ipympl
   - jupyterlab>=3.*
-  - pydotplus
-  - dask
+  # - libgcc # dlc-only
+  - matplotlib
+  - non_local_detector
+  - numpy<1.24
   - pip
   - position_tools
-  - numpy<1.24
-  - track_linearization>=2.3
+  # - pybind11 # dlc-only
+  - pydotplus
+  - pyfftw<=0.12.0  # ghostipy req. install from conda-forge for Mac ARM
+  - python>=3.9,<3.10
+  - pytorch<1.12.0
+  - replay_trajectory_classification
   - ripple_detection
-  - non_local_detector
-  - matplotlib
   - seaborn
-  - bottleneck
-  - ipympl
-  - tqdm
-  - pyfftw<=0.12.0  # used by ghostipy. install from conda-forge so that it works on Mac ARM processors
+  # - torchaudio # dlc-only
+  # - torchvision # dlc-only
+  - track_linearization>=2.3
+  # - trajectory_analysis_tools # dlc-only
   - pip:
-      - pubnub<6.4.0
-      - spikeinterface>=0.98.2,<0.99
-      - pynwb>=2.2.0,<3
-      - hdmf>=3.4.6
       - datajoint>=0.13.6
-      - ghostipy
-      - pymysql
-      - sortingview>=0.11
+      # - deeplabcut<2.3.0 # dlc-only
       - figurl-jupyter
-      - git+https://github.com/LorenFrankLab/ndx-franklab-novela.git
-      - pyyaml
-      - click
-      - "black[jupyter]"
+      - ghostipy # for common_filter
+      - ndx-franklab-novela>=0.1.0
+      - mountainsort4
+      - pubnub<=6.4.0
+      - pynwb>=2.2.0,<3
+      - sortingview>=0.11
+      - spikeinterface>=0.98.2,<0.99
       - .

environment_dlc.yml

@@ -0,0 +1,53 @@
+# 1. INSTALL CORRECT DRIVER for your GPU-equipped machine.
+#    see https://stackoverflow.com/questions/30820513/what-is-the-correct-version-of-cuda-for-my-nvidia-driver/30820690
+# 2. install: mamba env create -f environment_dlc.yml
+#    update existing install:  mamba env update -f environment_dlc.yml
+# 2. After installing...
+#    run: conda activate spyglass-dlc
+#    run: mamba env config vars set LD_LIBRARY_PATH=~/path/to/<anaconda3 or mambaforge>/envs/spyglass-position/lib/
+#    run: mamba install -c conda-forge wxpython
+name: spyglass-dlc
+channels:
+  - conda-forge
+  - defaults
+  - pytorch  # dlc-only
+  - franklab
+  - edeno
+dependencies:
+  - bottleneck
+  - cudatoolkit=11.3  # dlc-only
+  - cupy  # dlc-only
+  - dask-cuda  # dlc-only
+  - ffmpeg  # dlc-only
+  - ipympl
+  - jupyterlab>=3.*
+  - libgcc  # dlc-only
+  - matplotlib
+  - non_local_detector
+  - numpy<1.24
+  - pip>=20.2.*
+  - position_tools
+  - pybind11 #To avoid isosplit5 build error. Req by mountainsort4
+  - pydotplus>=2.0.*
+  - pyfftw<=0.12.0  # ghostipy req. install from conda-forge for Mac ARM
+  - python>=3.9,<3.10
+  - pytorch<1.12.0
+  - replay_trajectory_classification
+  - ripple_detection
+  - seaborn
+  - torchaudio  # dlc-only
+  - torchvision  # dlc-only
+  - track_linearization>=2.3
+  - trajectory_analysis_tools  # dlc-only
+  - pip:
+      - datajoint>=0.13.6
+      - deeplabcut<2.3.0
+      - figurl-jupyter
+      - ghostipy # for common_filter
+      - ndx-franklab-novela>=0.1.0
+      - mountainsort4
+      - pubnub<=6.4.0
+      - pynwb>=2.2.0,<3
+      - sortingview>=0.11
+      - spikeinterface>=0.98.2,<0.99
+      - .[dlc]

pyproject.toml

@@ -35,28 +35,26 @@ keywords = [
     "sortingview",
 ]
 dependencies = [
-    "pydotplus",
+    "bottleneck",
     "dask",
-    "position_tools",
-    "track_linearization>=2.3",
-    "non_local_detector",
-    "ripple_detection",
+    "datajoint>=0.13.6",
+    # "ghostipy", # removed from list bc M1 users need to install pyfftw first
+    "hdmf>=3.4.6",
+    "ipympl",
     "matplotlib",
-    "seaborn",
-    "bottleneck",
+    "ndx_franklab_novela>=0.1.0",
+    "non_local_detector",
     "numpy<1.24",
-    "ipympl",
-    "tqdm",
-    "pubnub<6.4.0",                 # TODO: remove this when sortingview is updated
+    "opencv-python",
+    "position_tools",
+    "pubnub<6.4.0",            # TODO: remove this when sortingview is updated
+    "pydotplus",
     "pynwb>=2.2.0,<3",
-    "hdmf>=3.4.6",
-    "datajoint>=0.13.6",
-    "pymysql",
+    "ripple_detection",
+    "seaborn",
     "sortingview>=0.11",
-    "pyyaml",
-    "click",
     "spikeinterface>=0.98.2,<0.99",
-    "ndx_franklab_novela>=0.1.0",
+    "track_linearization>=2.3",
 ]
 dynamic = ["version"]
 
@@ -68,14 +66,16 @@ spyglass_cli = "spyglass.cli:cli"
 "Bug Tracker" = "https://github.com/LorenFrankLab/spyglass/issues"
 
 [project.optional-dependencies]
-position = ["ffmpeg", "numba>=0.54", "deeplabcut<2.3.0"]
+dlc = ["ffmpeg", "numba>=0.54", "deeplabcut<2.3.0"]
 test = [
+    "black[jupyter]", # code formatting, rather than a 'dev' install
+    "click",          # for CLI subpackage only
     "docker",         # for tests in a container
     "pytest",         # unit testing
     "pytest-cov",     # code coverage
     "kachery",        # database access
     "kachery-client",
-    "kachery-cloud",
+    "kachery-cloud>=0.4.0",
 ]
 docs = [
     "hatch",                 # Get version from env
@@ -121,7 +121,7 @@ addopts = [
     # "--pdb", # drop into debugger on failure
     "-p no:warnings",
     # "--no-teardown", # don't teardown the database after tests
-    "--quiet-spy",                                    # don't show logging from spyglass
+    "--quiet-spy",                  # don't show logging from spyglass
     "--show-capture=no",
     "--pdbcls=IPython.terminal.debugger:TerminalPdb", # use ipython debugger
     "--cov=spyglass",
@@ -150,3 +150,7 @@ omit = [ # which submodules have no tests
     "*/spikesorting/*",
     # "*/utils/*",
 ]
+
+[tool.ruff] # CB: Propose replacing flake8 with ruff to delete setup.cfg
+line-length = 80
+ignore = ["F401" , "E402", "E501"]

setup.cfg

@@ -1,4 +1,4 @@
-[flake8]
+[flake8] # Planned for removal in favor of Ruff
 max-line-length = 80
 max-complexity = 17
 exclude =

src/spyglass/cli/cli.py

@@ -1,8 +1,15 @@
 from typing import Union
 
-import click
 import yaml
 
+try:
+    import click
+except ImportError:
+    raise ImportError(
+        "spyglass.cli.cli requires the 'click' package. "
+        "You can install it with 'pip install click'."
+    )
+
 
 @click.group(help="Spyglass command-line client")
 def cli():

src/spyglass/utils/dj_mixin.py

@@ -3,10 +3,12 @@
 from typing import Dict, List, Union
 
 import datajoint as dj
+from datajoint.errors import DataJointError
 from datajoint.expression import QueryExpression
 from datajoint.logging import logger as dj_logger
 from datajoint.table import Table
 from datajoint.utils import get_master, user_choice
+from pymysql.err import DataError
 
 from spyglass.utils.dj_chains import TableChain, TableChains
 from spyglass.utils.dj_helper_fn import fetch_nwb