Julia 1.0 Error : julia.core.JuliaError: Exception 'UndefVarError' occurred while calling julia code:

[albeiroep]

Hello. I'm trying to use PyJulia with Julia 1.0. I already cloned PyJulia and i configured all the Julia environment (I can execute julia code without problems). But when i try to use pyjulia i get this error :

Traceback (most recent call last):
File "/home/spiritusspei/PycharmProjects/taf_analytics/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2961, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
runfile('/home/spiritusspei/PycharmProjects/taf_analytics/tafs/Testing/Testing.py', wdir='/home/spiritusspei/PycharmProjects/taf_analytics/org/imt')
File "/snap/pycharm-professional/83/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/snap/pycharm-professional/83/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/spiritusspei/PycharmProjects/taf_analytics/tafs/Testing/Testing.py", line 13, in
j = julia.Julia(jl_runtime_path="/home/spiritusspei/Programs/julia-1.0.0/bin/julia", debug=True)
I tried too with j = julia.Julia()
File "/home/spiritusspei/PycharmProjects/taf_analytics/venv/lib/python3.6/site-packages/julia/core.py", line 560, in init
self.__julia = Julia(*args, **kwargs)
File "/home/spiritusspei/PycharmProjects/taf_analytics/venv/lib/python3.6/site-packages/julia/core.py", line 443, in init
""lib", "pyjulia%s-v$(VERSION.major).$(VERSION.minor)"))" % sys.version_info[0])
File "/home/spiritusspei/PycharmProjects/taf_analytics/venv/lib/python3.6/site-packages/julia/core.py", line 488, in _call
self.check_exception(src)
File "/home/spiritusspei/PycharmProjects/taf_analytics/venv/lib/python3.6/site-packages/julia/core.py", line 515, in check_exception
.format(exception, src))

julia.core.JuliaError: Exception 'UndefVarError' occurred while calling julia code:
unshift!(Base.LOAD_CACHE_PATH, abspath(Pkg.Dir._pkgroot(),"lib", "pyjulia3-v$(VERSION.major).$(VERSION.minor)"))

I went to the julia console and when i execute Base.LOAD_CACHE_PATH i get the error
ERROR: UndefVarError: LOAD_CACHE_PATH not defined

What can i do in order to use PyJulia from Python? Thank you in advance.

[tkf]

Full 1.0 support still is a work in progress. #185 explains the current situation. Though using my branch #190 may solve the issue.

BTW I'm just curious: is it a Linux machine? What is the output of the following code?

import sys
from subprocess import call
call(["ldd", sys.executable])

(I'm just trying to see what is libpython used by python in pycharm)

[albeiroep]

Thank you TKF :) Yes, i'm using Ubuntu 18.04. This is the result of executing your commands :

linux-vdso.so.1 (0x00007fff377eb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3615a08000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3615804000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3615601000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f36153cf000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f36151b2000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3614e14000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3614a23000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3615c27000)

Still not working with your branch. So sad :(

[albeiroep]

Is it normal that i can't find Base.LOAD_CACHE_PATH in the Julia console? Because i see that everyone on internet talk about this variable but i receive an ERROR: UndefVarError: LOAD_CACHE_PATH not defined when i try to access it with Julia 1.0 and Julia 0.7

[tkf]

Yes, it's normal. LOAD_CACHE_PATH stuff is only for Julia 0.6. From the output of ldd, #190 does not help in your case. Please read #185 (and #185 (comment)) for more info.

[ExpandingMan]

For some strange reason this happens to me on one machine, but not another (where I am also running Julia 1.0). How is that possible?

[tkf]

If you are using different Python executables and/or different OSes/Linux distributions then that's possible. See: #185

Hopefully once #186 is merged the error message explains why you can't use PyJulia in your machine.

[tkf]

@ExpandingMan #186 is merged so you can try the master and see if it can tell the reason why it doesn't work.

[ExpandingMan]

Ah, I see

Julia interpreter:
    julia
Python interpreter and libpython used by PyCall.jl:
    /usr/bin/python3
    None
Python interpreter used to import PyJulia and its libpython.
    /usr/bin/python3
    None

Your Python interpreter "/usr/bin/python3"
is statically linked to libpython.  Currently, PyJulia does not support
such Python interpreter.  For available workarounds, see:
    https://github.com/JuliaPy/pyjulia/issues/185

For more information, see:
    https://github.com/JuliaPy/pyjulia
    https://github.com/JuliaPy/PyCall.jl

I suppose this shouldn't be too surprising as that machine is running a really old (16.04) version of Ubuntu. Is it odd that I do indeed have the shared libraries installed but that the interpreter doesn't seem to use them?

[tkf]

@ExpandingMan Thanks for testing it out! It looks like it doesn't matter how old Ubuntu/Debian is. Even Ubuntu 18.04 has python statically linked to libpython (see above #196 (comment)). I guess that's Debian's packaging policy or something. When the Python interpreter is statically linked, whether there is a libpython file or not doesn't matter for PyJulia with Julia 1.0 because it can't use the normal precompilation cache of PyCall.jl.

[ExpandingMan]

Hm. That's rather annoying. I guess the statically linked Python is that way for performance reasons? Side note: Arch linux indeed has the dynamically linked version.

This seems like a long term problem for this package. We really need this to be simple to setup on debian, everybody uses it. Are there any options other than using a different Python? I guess probably not right? What's the simplest alternative to using the system Python?

I don't know. Let me know if you have any ideas, I'm willing to help.

[tkf]

Yeah it's annoying... I was surprised that Debian does it. Do you have a performance improvement going to static linking? I didn't know that.

I'll be posting the solutions in #185 as I find. The best long-term solution would be to get the support from Julia core JuliaLang/julia#28518

A workaround ATM (other than using different Python) is https://github.com/tkf/julia-venv (a similar hack can be implemented in PyJulia too #173). It works by having a private DEPOT_PATH[1] (which is normally ~/.julia). I also have to monkey-patch some Base functions to make it work. It is a tricky solution so that I started feel like building Python from source is the nicest solution...

[ExpandingMan]

Any performance linking from static linking would be small, but I don't know, the thinking might be that the system binaries should be ultra optimized, I don't know. For all I know it might even be possible to get around any performance hit from dynamic linking with the right compile options, so I could be wrong.

[tkf]

Thanks. I'd be interesting to know the reason why Debian does it but at the same time I guess it's just impossible to expect any change (the project is too big).

[homocomputeris]

I have a similar issue:

Python 3.7.0
[GCC 8.2.1 20180831] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> from subprocess import call
>>> call(["ldd", sys.executable])
	linux-vdso.so.1 (0x00007ffe54959000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f22ad8d5000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f22ad711000)
	libpython3.7m.so.1.0 => /usr/lib/libpython3.7m.so.1.0 (0x00007f22ad398000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f22ad92b000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f22ad393000)
	libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f22ad38e000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f22ad209000)
0
>>> import diffeqpy
>>> diffeqpy.install()
ERROR: LoadError: UndefVarError: Pkg not defined
Stacktrace:
 [1] top-level scope at none:0
in expression starting at /usr/lib/python3.7/site-packages/diffeqpy/install.jl:1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/site-packages/diffeqpy/__init__.py", line 33, in install
    subprocess.check_call(['julia', os.path.join(script_dir, 'install.jl')])
  File "/usr/lib/python3.7/subprocess.py", line 328, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['julia', '/usr/lib/python3.7/site-packages/diffeqpy/install.jl']' returned non-zero exit status 1.
>>> 

[tkf]

This is an issue in diffeqpy SciML/diffeqpy#30 (comment)

[homocomputeris]

I see. From the error msg it was not exactly clear which package does not work.

[tkf]

If you look at the traceback, you can see that there is no mention of PyJulia (which would be installed under site-packages/julia).

[homocomputeris]

True, thanks. My first though was if Python module cannot find Julia's default package Pkg and then I tried the example on a reinstall

julia> Pkg.installed()
Dict{String,Union{Nothing, VersionNumber}} with 2 entries:
  "DifferentialEquations" => v"5.3.1"
  "PyCall"                => v"1.18.4"

pip install --user julia  
Collecting julia
  Using cached https://files.pythonhosted.org/packages/2e/2a/04e16c1b1cde76c669328e09da9e833ee50a9aed4fad2b1fddf137d2d199/julia-0.1.5-py2.py3-none-any.whl
Installing collected packages: julia
Successfully installed julia-0.1.5

$ python
Python 3.7.0 (default, Sep 15 2018, 19:13:07) 
[GCC 8.2.1 20180831] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia import Base
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Base' from 'julia' (/home/nik/.local/lib/python3.7/site-packages/julia/__init__.py)

and assumed it might be the problem in the Julia/Python interface, and then I found this issue.

[tkf]

If you are using Julia 1.0, you need PyJulia master at the moment. Use something like this (untested):

pip install --user 'https://github.com/JuliaPy/pyjulia/archive/master.zip#egg=julia'

[tkf]

PyJulia 0.2 is out now. So I consider this to be resolved.