Weird interaction with Jupyter notebooks resulting in "forgetting" the Julia dependencies

[MilesCranmer]

Affects: JuliaCall

Describe the bug
Just flagging this in case others have seen it. I will:

1. Install my environment from the command line, using pip.
2. Use, e.g., python -c 'import pysr' to set up the Julia environment.
3. Then, I will open a Jupyter notebook, set the kernel to the one in my virtualenv, and import my juliacall-enabled package (in this case, PySR).

Sometimes I do this, and it seems to... "forget" what Julia packages are in my environment. In other words, when I do

import pysr

It will start re-installing the environment, with only PythonCall.jl. It seems like the first time it hits a from juliacall import Main, it just "resets" the Julia project and forgets the juliapkg.json previously used.

I wonder if this might be because the ipykernel itself is installed via a different python virtualenv? However it still seems to import the Python packages correctly; it just ends up uninstalling the Julia dependencies...

Reproducing. You can try to reproduce this as follows (not positive what seems to cause this; it might work or might not work for you):

1. Create a virtualenv at /path/to/my/.venv and install pysr, ipython, ipykernel, and jupyter with pip.
2. Start ipython, and import pysr. This will trigger the juliapkg set up of the Julia project with the correct dependencies.
3. Now, close ipython, and open Jupyter. Set the Jupyter kernel to be /path/to/my/.venv. Now, in Jupyter, import pysr.

You might see that it will start re-installing the Julia project, but this time, only with PythonCall.jl, and no other dependencies. It will fail halfway through

Any idea @cjdoris? How are juliapkg.json used at import time to generate the right installation commands? Maybe I can provide more debugging info in some way.

Maybe I just have weird ipykernel settings and nobody can reproduce this?? Or perhaps the juliapkg.json isn't getting packaged correctly in the wheel?

Your system
Please provide detailed information about your system:

• The operating system: macOS
• The version of:
  • Julia: 1.10.3
  • Python: 3.12.2
  • PythonCall: 0.9.20 (also see this on 0.9.19)

Additional context

IPython always seems safe. Never had an issue. It's exclusively Jupyter notebooks (both in VSCode, and in regular Jupyter notebooks in the browser) where the juliapkg "forgetfulness" happens.

[MilesCranmer]

I wonder if it could also be something about the way the package gets installed? Like perhaps it is dropping the juliapkg.json file somewhere...

Edit: Maybe I should just use juliapkg directly rather than writing a json.

[MilesCranmer]

Update: Seems like I was able to fix this by using the Python interface of juliapkg rather than storing a juliapkg.json file! It seems like that json file is not getting picked up for some reason. I think the explicit Python strategy might be more robust anyways (?) so I will go with that.

Edit: although... I'm not sure why, the Jupyter notebook now seems to try to re-install every time it starts?

[cjdoris]

Can you be more explicit in how to reproduce? I.e. give a sequence of commands specifying exactly how you set up the environment and are invoking python/jupyter.

My immediate thought is just that you're not in the venv you think you are.

I don't understand your juliapkg.json comments - you never mentioned interacting with that directly in the OP.

[MilesCranmer]

My immediate thought is just that you're not in the venv you think you are.

This was my first thought too, but it’s actually writing the Project.toml to the correct path. (And is able to import the correct Python libraries)

I don't understand your juliapkg.json comments - you never mentioned interacting with that directly in the OP.

Sorry. PySR has shipped a manually-edited juliapkg.json so far in its releases. But perhaps I should have been using the juliapkg package API calls instead as part of __init__.py (?).

I.e. give a sequence of commands specifying exactly how you set up the environment and are invoking python/jupyter

It’s difficult to do this because the way an environment/kernel is selected in VSCode/Jupyter, which is all GUI based.

However, here’s what I do. To set up the environment, I just do

python -m virtualenv .venv
source activate .venv/bin/activate
pip install pysr jupyter ipython ipykernel
python -c "import pysr"
jupyter notebook

Then, inside a new notebook, I select this environment for the kernel and run import pysr, which is when I see this re-installation happening (which correctly goes to the same .venv environment, but just seems to ignore the juliapkg.json from the earlier pysr installation).

[cjdoris]

I cannot reproduce your issue by following those steps. Can you compare juliapkg.state.STATE and juliapkg.deps.deps_files() between python and jupyter?

[ma-sadeghi]

@MilesCranmer Not sure but could be related: pyjuliapkg uses a non-standard way to detect if it's run in a virtualenv. That's patched up (JuliaPy/pyjuliapkg/pull/31) now, but not yet released.

The side effect of the non-standard way was that depending on whether VIRTUAL_ENV environment variable was set, pyjuliapkg would install the dependencies in different places (i.e., either .venv/julia_env or ~/.julia/environments/pyjuliapkg.

[cjdoris]

It's now released.