Could not load python library when run from compiled julia app on another machine #981

sairus7 · 2022-04-08T09:16:41Z

When I compile my app that has a PyCall dependency, it could not find python library, because it is compiled with absolute path to that library found on dev machine. So when I run my app on another machine, it can't find it:

fatal: error thrown and no exception handler available.
InitError(mod=:PyCall, error=ErrorException("could not load library "C:\absolute\path\from\another\machine\.julia\conda\3\python39.dll"
The specified module could not be found. "))
jl_errorf at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:77
jl_load_dynamic_library at /cygdrive/c/buildbot/worker/package_win64/build/src\dlload.c:284
#dlopen#3 at .\libdl.jl:117
dlopen at .\libdl.jl:117 [inlined]
init at C:\absolute\path\from\another\machine\.julia\packages\PyCall\7a7w0\src\pyinit.jl:149
jfptr___init___58920.clone_1 at C:\_KTAuto\backend\MainBackendCompiled\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_module_run_initializer at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:73

So its this line pyinit.jl:149:

PyCall.jl/src/pyinit.jl

Line 150 in 6b18f38

Libdl.dlopen(libpython, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL)

The text was updated successfully, but these errors were encountered:

stevengj · 2022-04-08T23:06:29Z

Yes, compiling in the libpython into PyCall was something we did to improve load times (#169). Nowadays, it might be possible to revisit this; see e.g. the PythonCall.jl package for a more dynamic approach.

sairus7 · 2022-04-12T00:05:21Z

There are some other problems compiling app with PythonCall - see here: JuliaPy/PythonCall.jl#146

@stevengj Can you please explain a little bit more, what should be fixed in PyCall to exclude fixed library paths from compiled binaries? Or any other ways to use it with compiled app?

efJerryYang · 2022-04-12T12:34:25Z

I also encountered this problem today, almost the exact same error message as @sairus7 . At first I used create_sysimage it crashed in a very similar way, so I chose create_app but got this error.

stevengj · 2022-04-13T11:49:25Z

I wonder if it's as simple as inserting the following line here:

libpython = relpath.(libpython, @__DIR__)

Want to give it a try?

efJerryYang · 2022-04-13T13:35:44Z

Oh, things seem to crash in the same way with fixed absolute path... Any other operations needed before building the app?

And I wonder if this is the right absolute path for build.jl? I build my app in default julia environment. @stevengj

C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl

sairus7 · 2022-04-13T13:41:45Z

And I wonder if this is the right absolute path for build.jl?

Make sure you dev PyCall in your project environment and make changes into .julia/dev/PyCall/deps/build.jl

efJerryYang · 2022-04-13T13:46:09Z

Thanks, but this error was thrown when building the pkgs in my project environment

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
ERROR: Error building `PyCall`:
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

┌ Info: Using the Python distribution in the Conda package by default.
└ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
[ Info: Running `conda install -y numpy` in root environment
ERROR: LoadError: MethodError: no method matching relpath(::Ptr{Nothing}, ::String)
Closest candidates are:
  relpath(!Matched::String, ::String) at D:\ProgLangToolkit\julia\julia-1.7.2\share\julia\base\path.jl:536
  relpath(!Matched::AbstractString, ::AbstractString) at D:\ProgLangToolkit\julia\julia-1.7.2\share\julia\base\path.jl:573
Stacktrace:
 [1] _broadcast_getindex_evalf
   @ .\broadcast.jl:670 [inlined]
 [2] _broadcast_getindex
   @ .\broadcast.jl:643 [inlined]
 [3] getindex
   @ .\broadcast.jl:597 [inlined]
 [4] copy
   @ .\broadcast.jl:875 [inlined]
 [5] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0}, Nothing, typeof(relpath), Tuple{Base.RefValue{Ptr{Nothing}}, Base.RefValue{String}}})
   @ Base.Broadcast .\broadcast.jl:860
 [6] top-level scope
   @ C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl:84
 [7] include(fname::String)
   @ Base.MainInclude .\client.jl:451
 [8] top-level scope
   @ none:5
in expression starting at C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\deps\build.jl:43

sairus7 · 2022-04-13T13:46:22Z

I build my app in default julia environment.

To compile the project into an app I use project own environment, not the default one.
This is typical script compile_app.sh, notice --project=@. argument:

julia -e 'using Pkg; Pkg.add("PackageCompiler")'

julia --project=@. --startup-file=no -e 'using Pkg; Pkg.instantiate()'

julia --project=@. --startup-file=no -e '
using PackageCompiler;
PackageCompiler.create_app(pwd(), "MyProjectCompiled";
    cpu_target="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)",
    include_transitive_dependencies=false,
    filter_stdlibs=true,
    precompile_execution_file=["test/runtests.jl"])
'

sairus7 · 2022-04-13T13:50:44Z

ERROR: LoadError: MethodError: no method matching relpath(::Ptr{Nothing}, ::String)

Maybe that's because libpython === nothing?

efJerryYang · 2022-04-13T13:52:23Z

Sorry, is this solution works for you? And that may be my problem...

sairus7 · 2022-04-13T13:55:50Z

I will check this in a couple of hours

efJerryYang · 2022-04-13T14:02:01Z

Great, but problem is that the path actually should not be nothing, and after I commented that line pkgs could be built without error (in the Demo environment)

efJerryYang · 2022-04-13T14:06:44Z

I confirmed that the error was caused by that line of code.

after commenting that line:

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
Precompiling project...
  5 dependencies successfully precompiled in 43 seconds (235 already precompiled)

cancel commenting that line:

(Demo) pkg> build
    Building GR ────→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\af237c08bda486b74318c8070adb96efa6952530\build.log`
    Building Conda ─→ `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6e47d11ea2776bc5627421d59cdcc1296c058071\build.log`
    Building PyCall → `C:\Users\JerryYang\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\1fc929f47d7c151c839c5fc1375929766fb8edcc\build.log`
ERROR: Error building `PyCall`:
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
...

efJerryYang · 2022-04-13T14:25:33Z

I got the correct build result after replace libpython with libpy_name like this, which is really strange to assign a pointer type with string value.

libpython = relpath.(libpy_name, @__DIR__)

The value of libpy_name is just the fixed absolute path C:\absolute\path\from\another\machine\.julia\conda\3\python39.dll

sairus7 · 2022-04-13T15:07:45Z

libpython = relpath.(libpython, @__DIR__)

This fix didn't work for me - it runs through tests and compilation stages, but trying to run the app gives the same error as in the first message.

efJerryYang · 2022-04-13T15:08:28Z

Hi, I was wrong in the last comment. I stuck here when replaced both libpython with libpy_name, but encountered an error when loading directory in startup.jl.

julia> import Pkg; Pkg.precompile()
Precompiling project...
  ✗ PyCall
  0 dependencies successfully precompiled in 2 seconds (239 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]

Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to C:\Users\JerryYang\.julia\compiled\v1.7\PyCall\jl_DD83.tmp.
ERROR: LoadError: could not load library "..\..\..\..\conda\3\python39.dll"
The specified module could not be found. . Please run `Pkg.build("PyCall")` if your Python build has changed
Stacktrace:
  [1] error(::String, ::String)
    @ Base .\error.jl:42
  [2] top-level scope
    @ C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\startup.jl:51
  [3] include(mod::Module, _path::String)
    @ Base .\Base.jl:418
  [4] include(x::String)
    @ PyCall C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\PyCall.jl:1
  [5] top-level scope
    @ C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\PyCall.jl:38
  [6] include
    @ .\Base.jl:418 [inlined]
  [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1318
  [8] top-level scope
    @ none:1
  [9] eval
    @ .\boot.jl:373 [inlined]
 [10] eval(x::Expr)
    @ Base.MainInclude .\client.jl:453
 [11] top-level scope
    @ none:1
in expression starting at C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\startup.jl:41
in expression starting at C:\Users\JerryYang\.julia\packages\PyCall\7a7w0\src\PyCall.jl:1

I look into it, and it is this line startup.jl:48 just the same as you found at the very first:

PyCall.jl/src/startup.jl

Line 48 in 6b18f38

Libdl.dlopen(libpython, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL)

efJerryYang · 2022-04-13T15:10:03Z

it runs through tests and compilation stages, but trying to run the app gives the same error as in the first message.

It actually did not run through tests, with a skipped error message like this:

Precompiling project...
  3 dependencies successfully precompiled in 40 seconds (236 already precompiled, 1 skipped during auto due to previous errors)

efJerryYang · 2022-04-13T15:18:47Z

I think just replace the absolute libpy_name with its related path representation is not enough, because it is libpython that caused those errors. There may need some more changes to be made in related procedure, even not just the file build.jl.

efJerryYang · 2022-04-13T15:22:58Z

Such modification only changed the variable libpython stored in deps.jl, you can see here:

const python = "C:\\Users\\JerryYang\\.julia\\conda\\3\\python.exe"
const libpython = "..\\..\\..\\..\\conda\\3\\python39.dll"
const pyprogramname = "C:\\Users\\JerryYang\\.julia\\conda\\3\\python.exe"
const pyversion_build = v"3.9.10"
const PYTHONHOME = "C:\\Users\\JerryYang\\.julia\\conda\\3"

"True if we are using the Python distribution in the Conda package."
const conda = true

sairus7 mentioned this issue Apr 8, 2022

Does it work from a compiled app run on another machine / from another user? JuliaPy/CondaPkg.jl#35

Closed

sairus7 mentioned this issue Jun 14, 2022

Allow any Python choice at runtime like PythonCall? #988

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Could not load python library when run from compiled julia app on another machine #981

Could not load python library when run from compiled julia app on another machine #981

sairus7 commented Apr 8, 2022 •

edited

Loading

stevengj commented Apr 8, 2022

sairus7 commented Apr 12, 2022

efJerryYang commented Apr 12, 2022

stevengj commented Apr 13, 2022 •

edited

Loading

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 13, 2022

sairus7 commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

Could not load python library when run from compiled julia app on another machine #981

Could not load python library when run from compiled julia app on another machine #981

Comments

sairus7 commented Apr 8, 2022 • edited Loading

stevengj commented Apr 8, 2022

sairus7 commented Apr 12, 2022

efJerryYang commented Apr 12, 2022

stevengj commented Apr 13, 2022 • edited Loading

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 13, 2022

sairus7 commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

efJerryYang commented Apr 13, 2022

sairus7 commented Apr 8, 2022 •

edited

Loading

stevengj commented Apr 13, 2022 •

edited

Loading