WIP: fetch upstream

CHANGELOG.md

@@ -24,7 +24,9 @@ SpikeSortingRecording().update_ids()
 
 - Ensure merge tables are declared during file insertion #1205
 - Update URL for DANDI Docs #1210
+- Add common method `get_position_interval_epoch` #1056
 - Improve cron job documentation and script #1226, #1241
+- Update export process to include `~external` tables #1239
 - Only add merge parts to `source_class_dict` if present in codebase #1237
     - Add recompute ability for `SpikeSortingRecording` #1093
 
@@ -39,6 +41,9 @@ SpikeSortingRecording().update_ids()
 - Spikesorting
     - Fix compatibility bug between v1 pipeline and `SortedSpikesGroup` unit
         filtering #1238
+    - Speedup `get_sorting` on `CurationV1` #1246
+- Behavior
+    - Implement pipeline for keypoint-moseq extraction of behavior syllables #1056
 
 ## [0.5.4] (December 20, 2024)
 

dj_local_conf_example.json

@@ -51,6 +51,11 @@
       "video": "/your/base/path/deeplabcut/video",
       "output": "/your/base/path/deeplabcut/output"
     },
+    "moseq_dirs": {
+      "base": "/your/base/path/moseq",
+      "project": "/your/base/path/moseq/projects",
+      "video": "/your/base/path/moseq/video",
+    },
     "kachery_zone": "franklab.default"
   }
 }

docs/src/Features/Export.md

@@ -165,3 +165,42 @@ Or, optionally, you can use the `RestrGraph` class to cascade hand-picked tables
 and restrictions without the background logging of `ExportMixin`. The assembled
 list of restricted free tables, `RestrGraph.all_ft`, can be passed to
 `Export.write_export` to generate a shell script for exporting the data.
+
+## Backwards Compatibility
+
+Spyglass databases that were declared before varchars were reduced to
+accommodate MySQL key length restrictions (see
+[#630](https://github.com/LorenFrankLab/spyglass/issues/630)) will have trouble
+importing exported data. Specifically, varchar mismatches will throw foreign key
+errors. To fix this, you run the following bash script on the generated `sql`
+files before importing them into the new database.
+
+<details><summary>Script</summary>
+
+```bash
+#!/bin/bash
+
+for file in ./_Pop*sql; do \
+    echo $file
+    sed -i 's/ DEFAULT CHARSET=[^ ]\w*//g' "$file"
+    sed -i 's/ DEFAULT COLLATE [^ ]\w*//g' "$file"
+    sed -i 's/ `nwb_file_name` varchar(255)/ `nwb_file_name` varchar(64)/g' "$file"
+    sed -i 's/ `analysis_file_name` varchar(255)/ `analysis_file_name` varchar(64)/g' "$file"
+    sed -i 's/ `interval_list_name` varchar(200)/ `interval_list_name` varchar(170)/g' "$file"
+    sed -i 's/ `position_info_param_name` varchar(80)/ `position_info_param_name` varchar(32)/g' "$file"
+    sed -i 's/ `mark_param_name` varchar(80)/ `mark_param_name` varchar(32)/g' "$file"
+    sed -i 's/ `artifact_removed_interval_list_name` varchar(200)/ `artifact_removed_interval_list_name` varchar(128)/g' "$file"
+    sed -i 's/ `metric_params_name` varchar(200)/ `metric_params_name` varchar(64)/g' "$file"
+    sed -i 's/ `auto_curation_params_name` varchar(200)/ `auto_curation_params_name` varchar(36)/g' "$file"
+    sed -i 's/ `sort_interval_name` varchar(200)/ `sort_interval_name` varchar(64)/g' "$file"
+    sed -i 's/ `preproc_params_name` varchar(200)/ `preproc_params_name` varchar(32)/g' "$file"
+    sed -i 's/ `sorter` varchar(200)/ `sorter` varchar(32)/g' "$file"
+    sed -i 's/ `sorter_params_name` varchar(200)/ `sorter_params_name` varchar(64)/g' "$file"
+done
+```
+
+</details>
+
+This is essentially a series of `sed` commands that adjust varchar lengths to
+their updated values. This script should be run in the directory containing the
+`_Populate*.sql` files generated by the export process.

environment_moseq_cpu.yml

@@ -0,0 +1,42 @@
+# 1. Install a conda distribution.
+#    https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html
+# 2. Run: `mamba env create -f environment.yml`
+# 3. Activate: `conda activate spyglass`
+#
+# (lines intentionally left blank)
+#
+#
+name: spyglass-moseq-cpu
+channels:
+  - conda-forge
+  # - defaults # deprecated
+  - franklab
+  - edeno
+  # - pytorch # dlc-only
+  # - anaconda  # dlc-only, for cudatoolkit
+dependencies:
+  - bottleneck
+  # - cudatoolkit=11.3 # dlc-only
+  # - ffmpeg # dlc-only
+  - ipympl
+  - jupyterlab>=3.*
+  # - libgcc # dlc-only
+  - matplotlib
+  - non_local_detector
+  - numpy
+  - pip
+  - position_tools
+  - pybind11  # req by mountainsort4 -> isosplit5
+  - pydotplus
+  - pyfftw<=0.12.0  # ghostipy req. install from conda-forge for Mac ARM
+  - python>=3.9,<3.13
+  - pytorch<1.12.0
+  - ripple_detection
+  - seaborn
+  # - torchaudio # dlc-only
+  # - torchvision # dlc-only
+  - track_linearization>=2.3
+  - pip:
+      - ghostipy # for common_filter
+      - mountainsort4
+      - .[moseq-cpu]

environment_moseq_gpu.yml

@@ -0,0 +1,42 @@
+# 1. Install a conda distribution.
+#    https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html
+# 2. Run: `mamba env create -f environment.yml`
+# 3. Activate: `conda activate spyglass`
+#
+# (lines intentionally left blank)
+#
+#
+name: spyglass-moseq-gpu
+channels:
+  - conda-forge
+  # - defaults # deprecated
+  - franklab
+  - edeno
+  # - pytorch # dlc-only
+  # - anaconda  # dlc-only, for cudatoolkit
+dependencies:
+  - bottleneck
+  # - cudatoolkit=11.3 # dlc-only
+  # - ffmpeg # dlc-only
+  - ipympl
+  - jupyterlab>=3.*
+  # - libgcc # dlc-only
+  - matplotlib
+  - non_local_detector
+  - numpy
+  - pip
+  - position_tools
+  - pybind11  # req by mountainsort4 -> isosplit5
+  - pydotplus
+  - pyfftw<=0.12.0  # ghostipy req. install from conda-forge for Mac ARM
+  - python>=3.9,<3.13
+  - pytorch<1.12.0
+  - ripple_detection
+  - seaborn
+  # - torchaudio # dlc-only
+  # - torchvision # dlc-only
+  - track_linearization>=2.3
+  - pip:
+      - ghostipy # for common_filter
+      - mountainsort4
+      - .[moseq-gpu]

notebooks/00_Setup.ipynb

@@ -285,7 +285,47 @@
     "The Decoding pipeline relies on `jax` to process data with GPUs. Please see\n",
     "their conda installation steps\n",
     "[here](https://jax.readthedocs.io/en/latest/installation.html#conda-installation).\n",
-    "\n"
+    "\n",
+    "#### Deep Lab Cut (DLC)\n",
+    "\n",
+    "Spyglass provides an environment build for using the DLC pipeline. To create an \n",
+    "environment with these features, please:\n",
+    "1. navigate to your cloned spyglass repo.\n",
+    "2. build the environment from the dlc version\n",
+    "3. activate the environment to use\n",
+    "\n",
+    "```bash\n",
+    "cd /path/to/spyglass # 1\n",
+    "mamba env create -f environment_dlc.yml # 2\n",
+    "mamba activate spyglass-dlc # 3\n",
+    "```\n",
+    "\n",
+    "Alternatively, you can pip install using\n",
+    "```bash\n",
+    "pip install spyglass[dlc]\n",
+    "```\n",
+    "\n",
+    "#### Keypoint-Moseq\n",
+    "\n",
+    "Spyglass provides an environment build for using the Moseq pipeline. To create an \n",
+    "environment with these features, please:\n",
+    "1. navigate to your cloned spyglass repo.\n",
+    "2. build the environment from one of the moseq versions\n",
+    "3. activate the environment to use\n",
+    "\n",
+    "```bash\n",
+    "cd /path/to/spyglass # 1\n",
+    "mamba env create -f environment_moseq_cpu.yml # 2\n",
+    "mamba activate spyglass-moseq-cpu # 3\n",
+    "```\n",
+    "\n",
+    "Alternatively, you can pip install using\n",
+    "```bash\n",
+    "pip install spyglass[moseq-cpu]\n",
+    "```\n",
+    "\n",
+    "To use a GPU enabled version of the package, replace `cpu` with `gpu` in the above \n",
+    "commands\n"
    ]
   },
   {
@@ -593,7 +633,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "spyglass",
    "language": "python",
    "name": "python3"
   },
@@ -607,7 +647,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.19"
+   "version": "3.9.18"
   }
  },
  "nbformat": 4,