Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add step to enable conda envs #78

Open
bschilder opened this issue Sep 19, 2023 · 9 comments
Open

Add step to enable conda envs #78

bschilder opened this issue Sep 19, 2023 · 9 comments
Assignees
Labels
conda Issues related to conda and/or python enhancement New feature or request GitHub Actions Issue occurring on GitHub Actions

Comments

@bschilder
Copy link
Collaborator

Adding a step to setup conda/miniconda/mamba automatically within rworkflows could be quite helpful (came up here).

Possible parameters:

@bschilder bschilder self-assigned this Sep 19, 2023
@bschilder bschilder added enhancement New feature or request GitHub Actions Issue occurring on GitHub Actions labels Sep 19, 2023
@bschilder bschilder moved this from Todo to In Progress in Project Management Sep 19, 2023
@bschilder
Copy link
Collaborator Author

bschilder commented Sep 19, 2023

GHA offers runners with python (and miniconda?) pre-installed:
https://github.com/marketplace/actions/setup-miniconda#use-a-different-environment-name-or-path

This will be create a named env in $CONDA/envs/whatever, where $CONDA is the path to the infrequently-updated, but very fast to start, "bundled" Miniconda installation.

@bschilder
Copy link
Collaborator Author

bschilder commented Sep 19, 2023

rworkflows::use_workfow should offer options to read in different kinds of environment-specification files:

  • conda env .yaml: Extract and use env name, channels, and deps
    Done.
  • python requirements.txt: Extract deps
    Can export with new func conda_export.
    Conda should be able to use ".yml" or "requirements.txt", but i get an error each time I go to build with the latter:
+ /Users/bms20/anaconda3/bin/conda env create --quiet --name 'testenv2' -f '/var/folders/rd/rbc_wrdj4k3djf3brk6z0_dc0000gp/T//Rtmpd0gX2x/file3d7eae45b8b_requirements.txt'
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

ResolvePackageNotFound: 
  - tzdata==2023c=h71feb2d_0
  - bzip2==1.0.8=h3422bc3_4
  - urllib3==2.0.6=pypi_0
  - libzlib==1.2.13=h53f4e23_5
  - setuptools==68.2.2=pyhd8ed1ab_0
  - ncurses==6.4=h7ea286d_0
  - ca-certificates==2023.7.22=hf0a4a13_0
  - certifi==2023.7.22=pypi_0
  - idna==3.4=pypi_0
  - charset-normalizer==3.3.0=pypi_0
  - pip==23.2.1=pyhd8ed1ab_0
  - wheel==0.41.2=pyhd8ed1ab_0
  - libsqlite==3.43.2=h091b4b1_0
  - tk==8.6.13=hb31c410_0
  - requests==2.31.0=pypi_0
  - libexpat==2.5.0=hb7217d7_1
  - libffi==3.4.2=h3422bc3_5
  - openssl==3.1.3=h53f4e23_0
  - python==3.12.0=h47c9636_0_cpython
  - readline==8.2=h92ec313_1
  - xz==5.2.6=h57fd34a_0

Error: Error creating conda environment [exit code 1]
  • poetry .toml: Extract and use env name and deps
    Probably possible, but gets a bit complicated. May return to this later.

Though these may be less necessary as the setup-miniconda action provides the environment-file: arg.

@bschilder bschilder added the conda Issues related to conda and/or python label Sep 23, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in Project Management Oct 11, 2023
@bschilder
Copy link
Collaborator Author

rworkflows is now working well and passing GHA when creating/exporting conda envs via yaml files.

@bschilder
Copy link
Collaborator Author

Expose these new conda args to the use_workflows R function

@bschilder
Copy link
Collaborator Author

bschilder commented Oct 31, 2023

Conda installation isn't being recognized by rworkflows, and thus tests aren't being run:

Just set up a Codespace #103 and found these potentially useful env variables:
Screenshot 2023-10-31 at 18 47 42

Reticulate registers conda as installed too (tho not sure if this is true for the setup-miniconda action too:
Screenshot 2023-10-31 at 18 52 32

@bschilder
Copy link
Collaborator Author

bschilder commented Oct 31, 2023

Was able to get a bit further this time: one of the conda env variables was recognized, but reticulate::conda_create failed to find the conda executable:
https://github.com/neurogenomics/rworkflows/actions/runs/6711171547/job/18237958236#step:4:2738
Screenshot 2023-10-31 at 18 59 46

@bschilder
Copy link
Collaborator Author

bschilder commented Oct 31, 2023

It appears that it's the CONDA env variable that's being detected:
https://github.com/neurogenomics/rworkflows/actions/runs/6711316713/job/18238415871#step:4:2563
Screenshot 2023-10-31 at 19 15 03

Potentially related issues:

Just asked the reticulate maintainers for a update on this ☝️ , but in the meantime a user suggested that miniforge_variant: "Mambaforge" is a useful workaround.

@bschilder
Copy link
Collaborator Author

but in the meantime a user suggested that miniforge_variant: "Mambaforge" is a useful workaround.
Setting miniforge_variant: "Mambaforge" didn't seem to work for me:

Screenshot 2023-10-31 at 19 34 40

Nor did setting the additional parameter use-mamba: true at the same time (same error):
https://github.com/neurogenomics/rworkflows/actions/runs/6711618082/job/18239368376#step:4:2741

@bschilder bschilder removed this from the Publish rworkflows manuscript milestone Jan 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
conda Issues related to conda and/or python enhancement New feature or request GitHub Actions Issue occurring on GitHub Actions
Projects
None yet
Development

No branches or pull requests

1 participant