Add tensorflow-macos conda package (#2)

* update .gitignore

* add windows/linux tensorflow conda package files from conda_packages repo

* update environment name

* add workflow to build tensorflow conda package

* update environment file which is not building, possible because of comments

* fix workflow: python version needs to be in quotes and each step should have a name, remove cache comments

* try different minor version

* add channel arg to upload

* try to enable long path support in the windows runner

* try 2.8.0

* try different github action checkout version

* go back to 2.9.0

* Try shorter env name

* enabling windows long path does not work locally or on the runner

* try to shorten path using substitution

* need drive letter

* create shortened link on windows

* try substitution for input and output

* tf 2.9.2

* try to shorten environment path with `CONDA_PREFIX`

* correct conda prefix

* debug build

* add prefix to build command

* try to config environment path

* setup-miniconda supports paths as environment names

* fix ubuntu environment path

* fix version and try to test import

* setup Miniconda depends on os

* try to upload using token

* try to upload without a password

* `ANACONDA_API_TOKEN` may automatically authenticate

* remove windows long path support that doesn't work

* turnoff upload for now

* add macos initial files

* login with `ANACONDA_PASSWORD instead of API token and upload Windows package

* add branch to action to test

* user needs to enter "yes" for anaconda login to work

* fix build path and try `ANACONDA_LOGIN`

* troubleshoot login

* change password and troubleshoot login again

* don't need echo yes when testing locally

* build conda packages

* make build script for macos

* make minimal meta.yaml for tensorflow-macos to test

* add macos to workflow

* add all packages installed via pip for requirements in macos meta.yaml to install via conda-forge

* remove tensorflow and tensorboard packages from meta.yaml for macos

* can't find mac requirements

* numpy error since tensorflow is compiled with numpy 1.x

* add comment for source path after can't find requirements in action

* change package name to tensorflow since sleap imports tensorflow

* remove packages were conda-forge not found and install markdown via anaconda

* clean up macos meta.yaml

* see if we can upload to anaconda using API token on macos runner

* fix build path for macos

* ANACONDA_API_TOKEN results in 401 error for macos as well as windows so use password to login

* Build a working tensorflow==2.12.0 for macos

* Do use runner to build (already built manually)

* Remove all out-of-scope code

* Add instructions for manually building/installing tensorflow on Mac

* Increase build number and trigger action

---------

Co-authored-by: roomrys <[email protected]>

.conda.tensorflow_macos/build.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+export PIP_NO_INDEX=False
+export PIP_NO_DEPENDENCIES=False
+export PIP_IGNORE_INSTALLED=False
+
+pip install --no-cache-dir -r ./requirements.tensorflow_macos.txt

.conda.tensorflow_macos/meta.yaml

@@ -0,0 +1,101 @@
+# Ref: https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
+
+package:
+  name: tensorflow
+  version: 2.12.0
+
+about:
+  home: https://tensorflow.org
+  license: Apache License
+  summary: "TensorFlow conda package based on the PyPI wheels for Apple Silicon Macs."
+
+build:
+  number: 4
+
+source:
+  # can't find requirements.tensorflow_macos.txt without this
+  path: ../
+
+requirements:
+  host:
+    - conda-forge::absl-py >=1.0.0
+    - conda-forge::astunparse >=1.6.0
+    - conda-forge::cachetools <6.0,>=2.0.0
+    - conda-forge::certifi >=2017.4.17
+    - conda-forge::charset-normalizer <4,>=2
+    - conda-forge::gast <=0.4.0,>=0.2.1
+    - conda-forge::google-auth <3,>=1.6.3
+    - conda-forge::google-auth-oauthlib <0.5,>=0.4.1
+    - conda-forge::google-pasta >=0.1.1
+    - conda-forge::grpcio <2.0,>=1.24.3
+    - conda-forge::h5py >=2.9.0
+    - conda-forge::idna <4,>=2.5
+    - conda-forge::keras <2.13.0,>=2.12.0 # Required for tensorflow-macos
+    - conda-forge::keras-preprocessing >=1.1.1
+    - anaconda::markdown >=2.6.8
+    - conda-forge::MarkupSafe >=2.1.1
+    - conda-forge::numpy >=1.16.5,<2.0 # tensorflow compiled with numpy 1.x
+    - conda-forge::oauthlib >=3.0.0
+    - conda-forge::opt-einsum >=2.3.2
+    - conda-forge::packaging
+    - conda-forge::pip
+    - conda-forge::protobuf >3.20 # Needed for pyside6, opencv, and tf compatibility
+    - conda-forge::pyasn1 <0.7.0,>=0.4.6
+    - conda-forge::pyasn1-modules >=0.2.1
+    - conda-forge::python >=3.10.0,<3.11.0
+    - conda-forge::requests <3,>=2.21.0
+    - conda-forge::requests-oauthlib >=0.7.0
+    - conda-forge::rsa <5,>=3.1.4
+    - conda-forge::termcolor >=1.1.0
+    - conda-forge::typing-extensions >=3.6.6
+    - conda-forge::urllib3 <3,>=1.21.1
+    - conda-forge::werkzeug >=1.0.1
+    - conda-forge::wrapt >=1.11.0,<1.15.0 # https://github.com/tensorflow/tensorflow/issues/59869
+
+  run:
+    - conda-forge::absl-py >=1.0.0
+    - conda-forge::astunparse >=1.6.0
+    - conda-forge::cachetools <6.0,>=2.0.0
+    - conda-forge::certifi >=2017.4.17
+    - conda-forge::charset-normalizer <4,>=2
+    - conda-forge::gast <=0.4.0,>=0.2.1
+    - conda-forge::google-auth <3,>=1.6.3
+    - conda-forge::google-auth-oauthlib <0.5,>=0.4.1
+    - conda-forge::google-pasta >=0.1.1
+    - conda-forge::grpcio <2.0,>=1.24.3
+    - conda-forge::h5py >=2.9.0
+    - conda-forge::idna <4,>=2.5
+    - conda-forge::keras >=2.12.0,<2.13.0 # Required for tensorflow-macos==2.12.0
+    - conda-forge::keras-preprocessing >=1.1.1
+    - anaconda::markdown >=2.6.8
+    - conda-forge::MarkupSafe >=2.1.1
+    - conda-forge::numpy >=1.16.5,<2.0 # tensorflow compiled with numpy 1.x
+    - conda-forge::oauthlib >=3.0.0
+    - conda-forge::opt-einsum >=2.3.2
+    - conda-forge::packaging
+    - conda-forge::pip
+    - conda-forge::protobuf >3.20 # Needed for pyside6, opencv, and tf compatibility
+    - conda-forge::pyasn1 <0.7.0,>=0.4.6
+    - conda-forge::pyasn1-modules >=0.2.1
+    - conda-forge::python >=3.10.0,<3.11.0
+    - conda-forge::requests <3,>=2.21.0
+    - conda-forge::requests-oauthlib >=0.7.0
+    - conda-forge::rsa <5,>=3.1.4
+    - conda-forge::termcolor >=1.1.0
+    - conda-forge::typing-extensions >=3.6.6
+    - conda-forge::urllib3 <3,>=1.21.1
+    - conda-forge::werkzeug >=1.0.1
+    - conda-forge::wrapt >=1.11.0,<1.15.0 # https://github.com/tensorflow/tensorflow/issues/59869
+
+    # the following are pip installed, even when conda-forge versions are available
+    # - conda-forge::flatbuffers <2,>=1.12
+    # - conda-forge::libclang >=13.0.0
+    # - conda-forge::six >=1.12.0
+    # - conda-forge::tensorboard <2.10,>=2.9
+    # - conda-forge::tensorboard-data-server <0.7.0,>=0.6.0
+    # - conda-forge::tensorboard-plugin-wit >=1.6.0
+    # - conda-forge::tensorflow-estimator <2.10.0,>=2.9.0rc0
+
+test:
+  imports:
+    - tensorflow

.github/workflows/build_tensorflow.yml

@@ -0,0 +1,57 @@
+name: Build TensorFlow
+
+on:
+  push:
+    branches:
+      - main
+      - elizabeth/add-tensorflow-macos-conda-package
+
+    paths:
+      - ".github/workflows/build_tensorflow.yml"
+      - ".conda.tensorflow_macos/**"
+      - "environment.tensorflow.yml"
+      - "requirements.tensorflow_macos.txt"
+
+jobs:
+  build:
+    name: Build package (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: ["macos-14"]
+    steps:
+      # Checkout the repository
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Miniconda (Ubuntu and MacOS)
+        if: matrix.os == 'macos-14'
+        uses: conda-incubator/[email protected]
+        with:
+          python-version: "3.10"
+          environment-file: environment.tensorflow.yml
+          activate-environment: tf
+
+      - name: Print environment info
+        shell: bash -l {0}
+        run: |
+          which python
+          conda info
+          conda list
+
+      - name: Build conda package (MacOS)
+        if: matrix.os == 'macos-14'
+        shell: bash -l {0}
+        run: |
+          conda build --debug .conda.tensorflow_macos --output-folder build.tensorflow_macos -c conda-forge
+
+      - name: Upload to Anaconda (MacOS)
+        if: matrix.os == 'macos-14'
+        env:
+          ANACONDA_PASSWORD: ${{ secrets.ANACONDA_PASSWORD }}
+        shell: bash -l {0}
+        run: |
+          anaconda login --username sleap-deps --password "$ANACONDA_PASSWORD"
+          anaconda -v upload build.tensorflow_macos/osx-arm64/*.tar.bz2 --label dev --user sleap-deps
+          anaconda logout

.gitignore

@@ -20,6 +20,7 @@ parts/
 sdist/
 var/
 wheels/
+pip-wheel-metadata/
 share/python-wheels/
 *.egg-info/
 .installed.cfg
@@ -49,7 +50,6 @@ coverage.xml
 *.py,cover
 .hypothesis/
 .pytest_cache/
-cover/
 
 # Translations
 *.mo
@@ -72,7 +72,6 @@ instance/
 docs/_build/
 
 # PyBuilder
-.pybuilder/
 target/
 
 # Jupyter Notebook
@@ -83,9 +82,7 @@ profile_default/
 ipython_config.py
 
 # pyenv
-#   For a library or package, you might want to ignore these files since the code is
-#   intended to run in multiple environments; otherwise, check them in:
-# .python-version
+.python-version
 
 # pipenv
 #   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
@@ -94,24 +91,7 @@ ipython_config.py
 #   install all needed dependencies.
 #Pipfile.lock
 
-# poetry
-#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
-#   This is especially recommended for binary packages to ensure reproducibility, and is more
-#   commonly ignored for libraries.
-#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
-#poetry.lock
-
-# pdm
-#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
-#pdm.lock
-#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
-#   in version control.
-#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
-.pdm.toml
-.pdm-python
-.pdm-build/
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
 __pypackages__/
 
 # Celery stuff
@@ -148,15 +128,5 @@ dmypy.json
 # Pyre type checker
 .pyre/
 
-# pytype static type analyzer
-.pytype/
 
-# Cython debug symbols
-cython_debug/
-
-# PyCharm
-#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
-#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
-#  and can be added to the global gitignore or merged into this file.  For a more nuclear
-#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
+build.*/

README.md

@@ -1,2 +1,34 @@
 # sleap-deps
 Packages necessary for sleap build maintained by the SLEAP team.
+
+## tensorflow
+
+There is a Github Actions workflow `build_tensorflow.yml` that uses Github runners to build packages for us (and upload them to the `sleap-deps/label/dev` channel). But, if you would like to locally build and test the tensorflow package:
+
+1. Create the tensorflow build environment:
+
+```bash
+mamba env create -f environment.tensorflow.yml -n tf
+```
+
+2. Activate the tensorflow build environment:
+
+```bash
+mamba activate tf
+```
+
+3. Build the tensorflow conda package:
+
+#### Mac
+
+```bash
+conda build --debug.conda.tensorflow_macos --output-folder build.tensorflow_macos -c conda-forge
+```
+
+4. Test the conda package:
+
+#### Mac
+
+```bash
+mamba create -n tf_macos -c ./build.tensorflow_macos -c conda-forge tensorflow
+```

environment.tensorflow.yml

@@ -0,0 +1,12 @@
+name: tf
+
+channels:
+  - conda-forge
+  - anaconda
+  - defaults
+
+dependencies:
+- conda-build
+- anaconda-client
+- conda-verify
+- pip

requirements.tensorflow_macos.txt

@@ -0,0 +1,4 @@
+tensorflow-macos>=2.12.0,<2.13.0; sys_platform == 'darwin' and platform_machine == 'arm64'  # lowest version with protobuf>3.20
+tensorflow-metal; sys_platform == 'darwin' and platform_machine == 'arm64'
+tensorflow-hub; sys_platform == 'darwin' and platform_machine == 'arm64'
+protobuf>3.20; sys_platform == 'darwin' and platform_machine == 'arm64'