Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Precompilation failure (InitError: failed process) #179

Open
pedromxavier opened this issue May 27, 2022 · 7 comments
Open

Precompilation failure (InitError: failed process) #179

pedromxavier opened this issue May 27, 2022 · 7 comments

Comments

@pedromxavier
Copy link
Contributor

When precompiling Anneal.jl (docs branch) there seems to be some process lock problem with libmamba.

The complete shell session is here: (.CondaPkg is created but left empty)

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Try the new cross-platform PowerShell https://aka.ms/pscore6

PS D:\gits\Anneal.jl> julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.    
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0 (2021-11-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release  
|__/                   |

(Anneal) pkg> precompile
Precompiling project...
  ✗ Anneal
  0 dependencies successfully precompiled in 19 seconds (32 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

Anneal [e4d9eb7f-b088-426e-aeb5-1c0dae3d8abb]

Failed to precompile Anneal [e4d9eb7f-b088-426e-aeb5-1c0dae3d8abb] to C:\Users\pedroxavier\.julia\compiled\v1.7\Anneal\jl_E2.tmp.
    CondaPkg Found dependencies: D:\gits\Anneal.jl\CondaPkg.toml
    CondaPkg Found dependencies: C:\Users\pedroxavier\.julia\packages\PythonCall\4eoCM\CondaPkg.toml
    CondaPkg Resolving changes
             + dwave-neal (pip)
             + pip
             + python
    CondaPkg Installing packages
warning  libmamba Cannot lock 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
    Waiting for other mamba process to finish
error    libmamba Path already locked by the same PID: 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
warning  libmamba LockFile creation for path 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache' failed, continuing without it
error    libmamba Path already locked by the same PID: 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
warning  libmamba LockFile creation for path 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache' failed, continuing without it
error    libmamba Path already locked by the same PID: 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
warning  libmamba LockFile creation for path 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache' failed, continuing without it
error    libmamba Path already locked by the same PID: 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
ERROR: LoadError: InitError: failed process: Process(`'C:\Users\pedroxavier\.julia\artifacts\d628738c351341856bc6e8abe94cd6c8d8a7bf66\Library\bin\micromamba.exe' -r 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -q -y -p 'D:\gits\Anneal.jl\.CondaPkg\env' --override-channels --no-channel-priority pip "python[version='>=3.7,<4']" -c anaconda -c conda-forge`, ProcessExited(3221226505)) [3221226505]

Stacktrace:
  [1] pipeline_error
    @ .\process.jl:531 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base .\process.jl:446
  [3] run(::Cmd)
    @ Base .\process.jl:444
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:283
  [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
    @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:191
  [6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:395
  [7] resolve()
    @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:288
  [8] envdir()
    @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\env.jl:70
  [9] init_context()
    @ PythonCall.C C:\Users\pedroxavier\.julia\packages\PythonCall\4eoCM\src\cpython\context.jl:56
 [10] __init__()
    @ PythonCall.C C:\Users\pedroxavier\.julia\packages\PythonCall\4eoCM\src\cpython\CPython.jl:21
 [11] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base .\loading.jl:768
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base .\loading.jl:854
 [13] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1097
 [14] require(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1013
 [15] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:997
 [16] include(mod::Module, _path::String)
    @ Base .\Base.jl:418
 [17] include(x::String)
    @ Anneal D:\gits\Anneal.jl\src\Anneal.jl:1
 [18] top-level scope
    @ D:\gits\Anneal.jl\src\Anneal.jl:40
 [19] include
    @ .\Base.jl:418 [inlined]
 [20] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1318
 [21] top-level scope
    @ none:1
 [22] eval
    @ .\boot.jl:373 [inlined]
 [23] eval(x::Expr)
    @ Base.MainInclude .\client.jl:453
 [24] top-level scope
    @ none:1
during initialization of module C
in expression starting at D:\gits\Anneal.jl\src\annealers\simulated\simulated.jl:1
in expression starting at D:\gits\Anneal.jl\src\Anneal.jl:1

julia> versioninfo()
Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

julia> using Pkg; Pkg.status()
     Project Anneal v0.1.0
      Status `D:\gits\Anneal.jl\Project.toml`
  [992eb4ea] CondaPkg v0.2.10
  [b8f27783] MathOptInterface v1.2.0
  [6099a3de] PythonCall v0.9.0
  [9a3f8284] Random

julia> using CondaPkg; CondaPkg.status()
CondaPkg Status D:\gits\Anneal.jl\CondaPkg.toml
Not Resolved (resolve first for more information)
Packages
  python (>=3.7,<4)
Channels
  anaconda
  conda-forge
Pip packages
  dwave-neal (>=0.5.6)

julia> CondaPkg.re
read_deps read_meta  resolve
julia> CondaPkg.resolve()
    CondaPkg Found dependencies: D:\gits\Anneal.jl\CondaPkg.toml
    CondaPkg Found dependencies: C:\Users\pedroxavier\.julia\packages\PythonCall\4eoCM\CondaPkg.toml
    CondaPkg Resolving changes
             + dwave-neal (pip)
             + pip
             + python
    CondaPkg Installing packages
warning  libmamba Cannot lock 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
    Waiting for other mamba process to finish
error    libmamba Path already locked by the same PID: 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
warning  libmamba LockFile creation for path 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache' failed, continuing without it
error    libmamba Path already locked by the same PID: 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
warning  libmamba LockFile creation for path 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache' failed, continuing without it
error    libmamba Path already locked by the same PID: 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
warning  libmamba LockFile creation for path 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache' failed, continuing without it
error    libmamba Path already locked by the same PID: 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
ERROR: failed process: Process(`'C:\Users\pedroxavier\.julia\artifacts\d628738c351341856bc6e8abe94cd6c8d8a7bf66\Library\bin\micromamba.exe' -r 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -q -y -p 'D:\gits\Anneal.jl\.CondaPkg\env' --override-channels --no-channel-priority pip "python[version='>=3.7,<4']" -c anaconda -c conda-forge`, ProcessExited(3221226505)) [3221226505]

Stacktrace:
 [1] pipeline_error
   @ .\process.jl:531 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base .\process.jl:446
 [3] run(::Cmd)
   @ Base .\process.jl:444
 [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
   @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:283
 [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
   @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:191
 [6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
   @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:395
 [7] resolve()
   @ CondaPkg C:\Users\pedroxavier\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:288
 [8] top-level scope
   @ REPL[5]:1

julia>

PS: I have some experience with the Python C API . If you could provide some insights about your thought process on this topic, so I can learn more about the package, I'd be happy to make pull requests for PythonCall in the future!
@cjdoris
Copy link
Collaborator

cjdoris commented Jun 4, 2022

I'm totally stumped, I cannot reproduce this issue on Windows with the same package versions as you. I tried deleting ~/.julia/compiled and .CondaPkg. I also tried working in an entirely new Julia Depot, making a fresh clone of your repo and precompiling that.

Are you able to reliably reproduce the problem? That is, is there a sequence of steps that you can run which always/usually gets to this problem? If not, I'm not sure I can help.

I have just released a new version of MicroMamba.jl with a newer version of micromamba (0.24.0 instead of 0.22.0). Maybe that helps?

@cjdoris
Copy link
Collaborator

cjdoris commented Jun 4, 2022

I have just tried and failed to reproduce the issue in a Julia 1.7.0 Docker container too.

Given the error message is about lock files, maybe reproducing the problem in a clean environment won't work, we probably need some stale lock files lying around.

@pedromxavier
Copy link
Contributor Author

pedromxavier commented Jun 4, 2022 •
edited
Loading

I've just deleted ~/.julia and installed 1.7.3 (Windows) and 1.7.0 (WSL). It works fine now on both with PythonCall 0.9 + MicroMamba 0.24. I've updateded the packages before reinstall but only complete julia reset made the error disappear.
I think you can close this issue, but there should be some check to erase old files/artifacts/locks under updates or package cleanups after ]rm. I'm not sure if it is possible or it is something for Pkg to figure out.
Thanks for your help!

@cjdoris
Copy link
Collaborator

cjdoris commented Jun 5, 2022

I'll leave it open for now. If it happens again, please make a copy of the pkgs/cache directory that is mentioned in the logs. Also please record the modification times on all the files in there (e.g. the output of dir).

I suspect the problem occurs if you interrupt MicroMamba while resolving dependencies, leaving lock files lying around. It may be possible to get CondaPkg to manually remove them.

@Roger-luo
Copy link

Happens to me on M1 Mac as well from Rossetta, I'm not sure if this is a M1 issue or not

julia> using Bloqade
[ Info: Precompiling Bloqade [bd27d05e-4ce1-5e79-84dd-c5d7d508bbe1]
┌ Warning: Python library "/Users/roger/Code/Julia/Bloqade/build/2.adiabatic/.CondaPkg/env/lib/libpython3.10.dylib" could not be opened.
└ @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/context.jl:109
ERROR: LoadError: InitError: ArgumentError: NULL library handle
Stacktrace:
  [1] #dlsym#1
    @ ./libdl.jl:57 [inlined]
  [2] dlsym
    @ ./libdl.jl:57 [inlined]
  [3] init_pointers(p::PythonCall.C.CAPIPointers, lib::Ptr{Nothing}) (repeats 2 times)
    @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/pointers.jl:283
  [4] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/context.jl:129
  [5] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/CPython.jl:21
  [6] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:768
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:854
  [8] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1097
  [9] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1013
 [10] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:997
 [11] include
    @ ./Base.jl:418 [inlined]
 [12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1318
 [13] top-level scope
    @ none:1
 [14] eval
    @ ./boot.jl:373 [inlined]
 [15] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [16] top-level scope
    @ none:1
during initialization of module C
in expression starting at /Users/roger/.julia/packages/Bloqade/c5pbN/src/Bloqade.jl:1
ERROR: Failed to precompile Bloqade [bd27d05e-4ce1-5e79-84dd-c5d7d508bbe1] to /Users/roger/.julia/compiled/v1.7/Bloqade/jl_UrbrnH.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
   @ Base ./loading.jl:1466
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:1410
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1120
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997

@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2023

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

@github-actions github-actions bot added the stale Issues about to be auto-closed label Sep 9, 2023
@github-actions
Copy link
Contributor

This issue has been closed because it has been stale for 7 days. If it is still
relevant, please re-open it.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 16, 2023
@cjdoris cjdoris reopened this Sep 22, 2023
@cjdoris cjdoris removed the stale Issues about to be auto-closed label Sep 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cjdoris @Roger-luo @pedromxavier