loading python package via reticulate fails with rmarkdown::render whereas Rscript succeeds

[bobermayer]

Hi,

I'm trying to run python packages via reticulate, but imports fail when using rmarkdown::render("script.R") whereasRscript script.R succeeds.

here's my conda env

mamba create -c conda-forge -n test scanpy pandoc

here's my script test_env.R

library(methods)
library(futile.logger)
conda_home <- '/home/obermayb_c/miniconda3'
conda_env <- 'test'
knitr::opts_chunk$set(echo = TRUE, cache=TRUE, cache.lazy=FALSE, message=FALSE, warning=FALSE,
                      engine.path = list(python = file.path(conda_home,"envs",conda_env,"bin","python")))
Sys.setenv(RETICULATE_PYTHON=file.path(conda_home,'envs',conda_env,'bin','python'))
Sys.setenv(LD_LIBRARY_PATH=file.path(conda_home,'envs',conda_env,'lib'))
library(reticulate)
use_python(file.path(conda_home,"envs",conda_env,'bin','python'))
use_condaenv(file.path(conda_home,"envs",conda_env), conda = file.path(conda_home,"condabin/conda"))
flog.info(.libPaths())
flog.info(paste("LD_LIBRARY_PATH: ",Sys.getenv("LD_LIBRARY_PATH")))
flog.info(py_config())
scanpy <- reticulate::import('scanpy', convert=TRUE)

here's the output of Rscript test_env.R:

INFO [2024-02-26 13:36:13] /home/obermayb_c/R/x86_64-pc-linux-gnu-library/4.0
INFO [2024-02-26 13:36:13] /opt/R/4.0.3/lib/R/library
INFO [2024-02-26 13:36:13] LD_LIBRARY_PATH:  /home/obermayb_c/miniconda3/envs/test/lib
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/bin/python
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/lib/libpython3.12.so
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test:/home/obermayb_c/miniconda3/envs/test
INFO [2024-02-26 13:36:13] /home/obermayb_c/R/x86_64-pc-linux-gnu-library/4.0/reticulate/config:/home/obermayb_c/miniconda3/envs/test/lib/python312.zip:/home/obermayb_c/miniconda3/envs/test/lib/python3.12:/home/obermayb_c/miniconda3/envs/test/lib/python3.12/lib-dynload:/home/obermayb_c/miniconda3/envs/test/lib/python3.12/site-packages
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test
INFO [2024-02-26 13:36:13] 
INFO [2024-02-26 13:36:13] 
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/bin/python
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/bin/python
INFO [2024-02-26 13:36:13] 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:50:58) [GCC 12.3.0]
INFO [2024-02-26 13:36:13] 3.12
INFO [2024-02-26 13:36:13] 64bit
INFO [2024-02-26 13:36:13] FALSE
INFO [2024-02-26 13:36:13] True
INFO [2024-02-26 13:36:13] list(path = "/home/obermayb_c/miniconda3/envs/test/lib/python3.12/site-packages/numpy", version = list(c(1, 26, 4)))
INFO [2024-02-26 13:36:13] NULL
INFO [2024-02-26 13:36:13] NULL
INFO [2024-02-26 13:36:13] TRUE
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/bin/python
INFO [2024-02-26 13:36:13] RETICULATE_PYTHON

and here's the result of Rscript -e "library(rmarkdown); rmarkdown::render('test_env.R','html_document')" :

processing file: test_env.spin.Rmd
  |...................................                 |  67% [unnamed-chunk-1]
Quitting from lines 3-19 [unnamed-chunk-1] (test_env.spin.Rmd)
Error in `py_module_import()`:
! ImportError: /home/obermayb_c/miniconda3/envs/test/lib/python3.12/site-packages/PIL/../../../libtiff.so.6: undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0
Run `reticulate::py_last_error()` for details.
Backtrace:
 1. reticulate::import("scanpy", convert = TRUE)
 2. reticulate:::py_module_import(module, convert = convert)
                                                                                                            
Execution halted

uncommenting the reticulate::import line gives the exact same logs, so library paths etc. should all be identical as far as I can see

it's somehow related to libjpeg-turbo, but I can't figure out why there's a difference between the Rscript and the rmarkdown::render calls.

sessionInfo

> xfun::session_info("rmarkdown")
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.4 LTS, RStudio 2023.12.1.402

Locale:
  LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
  LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

Package version:
  base64enc_0.1.3   bslib_0.6.1       cachem_1.0.8      cli_3.6.2         digest_0.6.34     ellipsis_0.3.2   
  evaluate_0.23     fastmap_1.1.1     fontawesome_0.5.2 fs_1.6.3          glue_1.7.0        graphics_4.0.3   
  grDevices_4.0.3   highr_0.10        htmltools_0.5.7   jquerylib_0.1.4   jsonlite_1.8.8    knitr_1.45       
  lifecycle_1.0.4   magrittr_2.0.3    memoise_2.0.1     methods_4.0.3     mime_0.12         R6_2.5.1         
  rappdirs_0.3.3    rlang_1.1.3       rmarkdown_2.25    sass_0.4.8        stats_4.0.3       stringi_1.8.3    
  stringr_1.5.1     tinytex_0.49      tools_4.0.3       utils_4.0.3       vctrs_0.6.5       xfun_0.42        
  yaml_2.3.8       

Pandoc version: 3.1.12.1

Checklist

When filing a bug report, please check the boxes below to confirm that you have provided us with the information we need. Have you:

• formatted your issue so it is easier for us to read?

• included a minimal, self-contained, and reproducible example?

• pasted the output from xfun::session_info('rmarkdown') in your issue?

• upgraded all your packages to their latest versions (including your versions of R, the RStudio IDE, and relevant R packages)?

• installed and tested your bug with the development version of the rmarkdown package using remotes::install_github("rstudio/rmarkdown")?

[cderv]

Quitting from lines 3-19 [unnamed-chunk-1] (test_env.spin.Rmd)

Do you have access to the intermediate Rmd created ?

Are you able to use the import inside a Rmd document using Reticulate ? or more generally the conda environment ?

Either using the python engine (https://rstudio.github.io/reticulate/articles/r_markdown.html) or regular R chunk with reticulate code ?

RScript will run the R script directly. rmarkdown::render() on a .R file will first transformed it to a Rmd document, and render this Rmd file. So I am wondering if there is an issue while running the code in R Markdown context specifically.

I don't have mamba, nor conda, so I would need to setup an environment to reproduce.

Installing scanpy on my python setup (using pyenv and virtualenv, and then running a simple .Rmd works

---
title: "Test"
date: "`r Sys.Date()`"
output: html_document
---

```{r}
scanpy <- reticulate::import('scanpy', convert=TRUE)
```

Running also this simple R scripts works for me (with rmarkdown::render("test.R", "html_document")

scanpy <- reticulate::import('scanpy', convert=TRUE)

So it seems to be related to some reticulate configuration, or interaction in R Markdown context, when using Conda env possibly. Not easy to debug...

If you are able to build something like a Docker Image or dev container config that reproduce this, this will help us. Or even a GH action.

Otherwise, I'll see if I can setup an environment to reproduce.

[bobermayer]

hi,
thanks a lot for looking into this. I don't think it's related to the markdown (I had a markdown in the beginning and switched to an R script so I could more easily compare Rscript vs. rmarkdown::render).
And yes, import works within an R chunk of a markdown inside the Rstudio IDE. but hitting knit or using rmarkdown::render doesn't.
I see that this is really hard to debug. maybe it's somewhat related to this issue with openssl in rstudio vs. conda?
are there any other env variables that I could check, or some kind of verbose / debug mode that I could activate?
and thanks for the Docker suggestion, I'll try that as well.

[cderv]

maybe it's somewhat related to this issue with openssl in rstudio vs. conda?

Is the issue only happening inside RStudio IDE ? What happens if you run the rmarkdown::render() command in a R console outside of RStudio IDE ?

are there any other env variables that I could check, or some kind of verbose / debug mode that I could activate?

Error seems to happen in Error in py_module_import(): so you could look into reticulate to see which debug or verbose option they have. Or look into Conda for such flag too.

thanks for the Docker suggestion, I'll try that as well.

If you manage to do one, then I'll be able to look on my side too. This would help a lot!

[bobermayer]

hi thanks again!
the issue happens with knit in the Rstudio IDE and when running rmarkdown::render() from the commandline, but not when executing chunks in Rstudio or using Rscript. so I wonder if rmarkdown loads some system libs for image processing / output that override the ones from conda (in that way it could be related to the openSSL issue). but setting LD_LIBRARY_PATH in the console, in an .Renviron file, or via Rscript -e "Sys.setenv(...); library(rmarkdown); rmarkdown::render()" doesn't help.

So far I didn't find any good debug / verbose options for reticulate, and I'm still struggling to set up a Docker image with conda env installed ...

[cderv]

Thanks for the additional information.

I don't think knitr or rmarkdown is doing anything with LD_LIBRARY_PATH, and I don't know if reticulate does anything specific when loaded in knitr context (it could, though).

This is some type of annoying bug that require to be reproduced so that several people can look into it.

If docker itself is not easy, maybe setting up a devcontainer configuration will be easier. I see there is specification for Anaconda available (https://github.com/devcontainers/images/tree/main/src/anaconda)

Dev container works well locally in VSCODE and also only in CODESPACE. Maybe this starter template can help ?
https://github.com/crazy4pi314/conda-devcontainer-demo

The Rocker project also has some devcontainer recipes specific for R and R package

• https://rocker-project.org/images/devcontainer/features.html
• https://github.com/rocker-org/devcontainer-features

Maybe with this, it can be a little bit easier than doing DOCKERFILE from scratch.

Hope it helps