Update environment creation (#1366)

* First 'working' env for 1.3.1, no GPU... * First working 1.3.1 w/gpu * Sorry, this is the first gpu working env * Fix `imgstore` version-dependent bug * Build and entry points work, but pip packages not installed * Add default channels in condarc.yaml * Rename environment.yaml to .yml * Working build no gpu (documented) * Env working w/gpu using tensorflow from pypi * Working build with GPU! And pip dep * Attempt to fix cannot find wheel ~= 0.35 issue * Run constrain everything * Use minimal conda req for install_requires, inc build number to get it working * Ubuntu build and environments working * Env creation working on M2 * Build and env working on windows (again) * Apple silicon (M2) build working * Pip package working (M2) * Exclude tests and docs from pypi wheel * Add comments to requirements * Get ready for manual build * Update os images, trigger dev build * Retry manual build win and mac * Retry mac manual build (to upload) * Require latest pynwb, remove setuptools * Remove old mac environment * Add twine as a build dependency * Add comments to manual build * Update build.yml * Rename "apple_silicon" to "mac" * Update installation docs (#1365) * Update installation docs * Trigger website build * Update quick install and build docs * Add Mambaforge links * Remove comment about experimental apple silicon support Co-authored-by: Talmo Pereira <[email protected]> * Fix minor typo * Change installation docs to reference universal mac env --------- Co-authored-by: Talmo Pereira <[email protected]> * More flexible with python version ~=3.7 * Pin py3.7.12 for host and run (good on win) * Add comments for why python ~=3.7 * Update ci workflow (#1371) * Update CI workflow to include mac * Trigger CI workflow * Remove verbose? python version * Migrate to micromamba * Explicitly state environment name * Remove environment chaching (for now) * Use different environment file for mac * Use correct syntax * Add `env_hash` and `bash -l {0}` * Remove env_file * Try different nested variable call via format * Use correct syntax * Update env file reference AS -> mac * Different path to environment file * Checkout repo * Use default shells * Remove unused comments * Fix caching attempt by changing caching key * Remove environment caching * Increase python in no_cuda to 3.7.17 * Less restrictive with python ~3.7 version * More direct installation docs 🤞 * Increase build numbers for develop branch build --------- Co-authored-by: Talmo Pereira <[email protected]>
talmolab · Jun 30, 2023 · 8c6bdb4 · 8c6bdb4
1 parent 1a0404c
commit 8c6bdb4
Show file tree

Hide file tree

Showing 31 changed files with 723 additions and 599 deletions.
diff --git a/.conda/README.md b/.conda/README.md
@@ -0,0 +1,21 @@
+This folder defines the conda package build for Linux and Windows. There are runners for both Linux and Windows on GitHub Actions, but it is faster to experiment with builds locally first.
+
+To build, first go to the base repo directory and install the build environment:
+
+```
+conda env create -f environment_build.yml -n sleap_build && conda activate sleap_build
+```
+
+And finally, run the build command pointing to this directory:
+
+```
+conda build .conda --output-folder build -c conda-forge -c nvidia -c https://conda.anaconda.org/sleap/ -c anaconda
+```
+
+To install the local package:
+
+```
+conda create -n sleap_0 -c conda-forge -c nvidia -c ./build -c https://conda.anaconda.org/sleap/ -c anaconda sleap=x.x.x
+```
+
+replacing x.x.x with the version of SLEAP that you just built.
diff --git a/.conda/bld.bat b/.conda/bld.bat
@@ -1,62 +1,14 @@
-@echo off
+@REM Install anything that didn't get conda installed via pip.
 
-rem # Install anything that didn't get conda installed via pip.
-rem # We need to turn pip index back on because Anaconda turns
-rem # it off for some reason. Just pip install -r requirements.txt
-rem # doesn't seem to work, tensorflow-gpu, jsonpickle, networkx,
-rem # all get installed twice if we do this. pip doesn't see the
-rem # conda install of the packages.
-
-rem # Install the pip dependencies and their dependencies. Conda turns of
-rem # pip index and dependencies by default so re-enable them. Had to figure
-rem # this out myself, ughhh.
+@REM We need to turn pip index back on because Anaconda turns it off for some reason.
 set PIP_NO_INDEX=False
 set PIP_NO_DEPENDENCIES=False
 set PIP_IGNORE_INSTALLED=False
 
-pip install numpy==1.19.5
-pip install six==1.15.0
-pip install imageio==2.15.0
-pip install attrs==21.2.0
-pip install cattrs==1.1.1
-pip install jsonpickle==1.2
-pip install networkx
-pip install nixio>=1.5.3
-@REM pip install tensorflow>=2.6.3,<=2.7.1
-@REM pip install h5py>=3.1.0,<=3.6.0
-pip install python-rapidjson
-@REM pip install opencv-python-headless>=4.2.0.34,<=4.5.5.62
-@REM pip install opencv-python @ git+https://github.com/talmolab/wrap_opencv-python-headless.git@ede49f6a23a73033216339f29515e59d594ba921
-@REM pip install pandas
-pip install psutil
-@REM pip install PySide2>=5.13.2,<=5.14.1
-pip install pyzmq
-pip install pyyaml
-pip install imgaug==0.4.0
-@REM pip install scipy>=1.4.1,<=1.7.3
-pip install scikit-image
-pip install scikit-learn==1.0.*
-pip install scikit-video
-pip install tensorflow-hub
-pip install imgstore==0.2.9
-pip install qimage2ndarray==1.9.0
-pip install jsmin
-pip install seaborn
-pip install pykalman==0.9.5
-pip install segmentation-models==1.0.1
-pip install rich==10.16.1
-pip install certifi==2021.10.8
-pip install pynwb
-pip install ndx-pose
-
-rem # Use and update environment.yml call to install pip dependencies. This is slick.
-rem # While environment.yml contains the non pip dependencies, the only thing left
-rem # uninstalled should be the pip stuff because that is left out of meta.yml
-rem conda env update -f=environment.yml
-
-rem # Install requires setuptools-scm
-pip install setuptools-scm
+@REM Install the pip dependencies. Note: Using urls to wheels might be better: 
+@REM https://docs.conda.io/projects/conda-build/en/stable/user-guide/wheel-files.html)
+pip install -r .\requirements.txt
 
-rem # Install sleap itself.
-rem # NOTE: This is the recommended way to install packages
+@REM Install sleap itself. This does not install the requirements, but will list which 
+@REM requirements are missing (see "install_requires") when user attempts to install.
 python setup.py install --single-version-externally-managed --record=record.txt
diff --git a/.conda/build.sh b/.conda/build.sh
@@ -1,51 +1,15 @@
-#!/usr/bin/env bash
-
 # Install anything that didn't get conda installed via pip.
-# We need to turn pip index back on because Anaconda turns
-# it off for some reason. Just pip install -r requirements.txt
-# doesn't seem to work, tensorflow-gpu, jsonpickle, networkx,
-# all get installed twice if we do this. pip doesn't see the
-# conda install of the packages.
 
+# We need to turn pip index back on because Anaconda turns it off for some reason.
 export PIP_NO_INDEX=False
 export PIP_NO_DEPENDENCIES=False
 export PIP_IGNORE_INSTALLED=False
 
-pip install numpy==1.19.5
-pip install six==1.15.0
-pip install imageio==2.15.0
-pip install attrs==21.2.0
-pip install cattrs==1.1.1
-pip install jsonpickle==1.2
-pip install networkx
-pip install nixio>=1.5.3
-# pip install tensorflow>=2.6.3,<=2.7.1
-# pip install h5py>=3.1.0,<=3.6.0
-pip install python-rapidjson
-# pip install opencv-python-headless==4.5.5.62
-# pip install git+https://github.com/talmolab/wrap_opencv-python-headless.git@ede49f6a23a73033216339f29515e59d594ba921
-# pip install pandas
-pip install psutil
-# pip install PySide2>=5.13.2,<=5.14.1
-pip install pyzmq
-pip install pyyaml
-pip install imgaug==0.4.0
-# pip install scipy>=1.4.1,<=1.7.3
-pip install scikit-image
-pip install scikit-learn==1.0.*
-pip install scikit-video
-pip install tensorflow-hub
-pip install imgstore==0.2.9
-pip install qimage2ndarray==1.9.0
-pip install jsmin
-pip install seaborn
-pip install pykalman==0.9.5
-pip install segmentation-models==1.0.1
-pip install rich==10.16.1
-pip install certifi==2021.10.8
-pip install pynwb
-pip install ndx-pose
+# Install the pip dependencies. Note: Using urls to wheels might be better: 
+# https://docs.conda.io/projects/conda-build/en/stable/user-guide/wheel-files.html)
+pip install -r ./requirements.txt
 
-pip install setuptools-scm
 
+# Install sleap itself. This does not install the requirements, but will list which 
+# requirements are missing (see "install_requires") when user attempts to install.
 python setup.py install --single-version-externally-managed --record=record.txt
diff --git a/.conda/conda_build_config.yaml b/.conda/conda_build_config.yaml
diff --git a/.conda/condarc.yaml b/.conda/condarc.yaml
@@ -0,0 +1,5 @@
+channels:
+  - conda-forge
+  - nvidia
+  - sleap
+  - anaconda
diff --git a/.conda/meta.yaml b/.conda/meta.yaml
@@ -1,6 +1,3 @@
-# Ref: https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
-
-# Jinja template: Process setup.py to obtain version and metadata
 {% set data = load_setup_py_data() %}
 
 
@@ -15,50 +12,79 @@ about:
   license: {{ data.get('license') }}
   summary: {{ data.get('description') }}
 
-build:
-  number: 0
-
 source:
   path: ../
 
+build:
+  number: 9
+
 requirements:
   host:
-    - python=3.7
-    # - sleap::pyside2=5.14.1
-    - conda-forge::numpy=1.19.5
-    - sleap::tensorflow=2.6.3
-    - conda-forge::pyside2=5.13.2
-    - conda-forge::h5py=3.1.0
-    - conda-forge::scipy=1.7.3
-    - conda-forge::six=1.15.0
-    - pillow=8.4.0
-    - shapely=1.7.1
-    - conda-forge::pandas
-    - ffmpeg
-    - qtpy>=2.0.1
-    - conda-forge::pip!=22.0.4
+    - conda-forge::python ==3.7.12  # Run into _MAX_WINDOWS_WORKERS not found if <
+    - numpy >=1.19.5,<1.23.0  # Linux likes anaconda, windows likes conda-forge
+    - conda-forge::cudatoolkit ==11.3.1
+    - conda-forge::cudnn=8.2.1
+    - nvidia::cuda-nvcc=11.3
+    - conda-forge::setuptools
+    - conda-forge::pip
 
-  run:
-    - python=3.7
-    - conda-forge::numpy~=1.19.5
-    - sleap::tensorflow=2.6.3
-    - conda-forge::pyside2>=5.13.2,<=5.14.1
-    - conda-forge::h5py~=3.1.0
-    - conda-forge::scipy>=1.4.1,<=1.7.3
-    - conda-forge::six~=1.15.0
-    - pillow=8.4.0
-    - shapely=1.7.1
+    # Only the packages above are required to build, but listing them all ensures no 
+    # unnecessary pypi packages are installed via the build script (bld.bat, build.sh)
+    - conda-forge::attrs ==21.4.0
+    - conda-forge::cattrs ==1.1.1
+    - conda-forge::h5py ==3.1  # [not win]
+    - conda-forge::imgaug ==0.4.0
+    - conda-forge::jsmin
+    - conda-forge::jsonpickle ==1.2
+    - conda-forge::networkx
+    - conda-forge::opencv
     - conda-forge::pandas
-    - ffmpeg
-    - qtpy>=2.0.1
-    - cudatoolkit=11.3.1
-    - cudnn=8.2.1
+    - conda-forge::pillow >=8.3.2
+    - conda-forge::psutil
+    - conda-forge::pykalman
+    - conda-forge::pyside2 >=5.12,<5.14
+    - conda-forge::python-rapidjson
+    - conda-forge::pyyaml
+    - conda-forge::pyzmq
+    - conda-forge::qtpy >=2.0.1
+    - conda-forge::rich
+    - conda-forge::scipy >=1.4.1,<=1.9.0
+    - conda-forge::scikit-image
+    - conda-forge::scikit-learn ==1.0
+    - conda-forge::scikit-video
+    - conda-forge::seaborn
+  run:
+    - conda-forge::python ==3.7.12  # Run into _MAX_WINDOWS_WORKERS not found if <
+    - conda-forge::attrs ==21.4.0
+    - conda-forge::cattrs ==1.1.1
+    - conda-forge::cudatoolkit ==11.3.1
+    - conda-forge::cudnn=8.2.1
     - nvidia::cuda-nvcc=11.3
-    - conda-forge::pip!=22.0.4
-
-  run_constrained:
-    - pyqt==9999999999
+    - conda-forge::h5py ==3.1  # [not win]
+    - conda-forge::imgaug ==0.4.0
+    - conda-forge::jsmin
+    - conda-forge::jsonpickle ==1.2
+    - conda-forge::networkx
+    - numpy >=1.19.5,<1.23.0  # Linux likes anaconda, windows likes conda-forge
+    - conda-forge::opencv
+    - conda-forge::pandas
+    - conda-forge::pillow >=8.3.2
+    - conda-forge::psutil
+    - conda-forge::pykalman
+    - conda-forge::pyside2 >=5.12,<5.14  # To ensure works correctly with QtPy.
+    - conda-forge::python-rapidjson
+    - conda-forge::pyyaml
+    - conda-forge::pyzmq
+    - conda-forge::qtpy >=2.0.1
+    - conda-forge::rich
+    - conda-forge::scipy >=1.4.1,<=1.9.0
+    - conda-forge::scikit-image
+    - conda-forge::scikit-learn ==1.0
+    - conda-forge::scikit-video
+    - conda-forge::seaborn
+    - sleap::tensorflow >=2.6.3,<2.11  # No windows GPU support for >2.10, sleap channel has 2.6.3
+    - conda-forge::tensorflow-hub
 
 test:
   imports:
-    - sleap
+    - sleap
diff --git a/.conda_apple_silicon/README.md b/.conda_apple_silicon/README.md
diff --git a/.conda_apple_silicon/build.sh b/.conda_apple_silicon/build.sh
diff --git a/.conda_apple_silicon/conda_build_config.yaml b/.conda_apple_silicon/conda_build_config.yaml