make init fails if pip is somehow configured in unexpected way

[devsnek]

just trying to set this up and test it out, getting some cryptic errors from make init:

python3 -m venv jsparagus_build_venv &&\
./jsparagus_build_venv/bin/pip install --upgrade pip &&\
./jsparagus_build_venv/bin/pip install -r requirements.txt
Collecting pip
  Using cached https://files.pythonhosted.org/packages/8e/76/66066b7bc71817238924c7e4b448abdb17eb0c92d645769c223f9ace478f/pip-20.0.2.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Skipping bdist_wheel for pip, due to binaries being disabled for it.
Installing collected packages: pip
  Running setup.py install for pip: started
    Running setup.py install for pip: finished with status 'done'
Successfully installed pip-20.0.2
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting entrypoints==0.3 (from -r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/b4/ef/063484f1f9ba3081e920ec9972c96664e2edb9fdc3d8669b0e3b8fc0ad7c/entrypoints-0.3.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /jsparagus/jsparagus_build_venv/bin/python3 /jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-w53v2n6u/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit
       cwd: None
  Complete output (39 lines):
  Collecting flit
    Using cached https://files.pythonhosted.org/packages/35/87/a92625dc5c81cf1608553b27b5c216bdc1e678ea6738a3a8a69ee2e5fb51/flit-2.2.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /jsparagus/jsparagus_build_venv/bin/python3 /jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-kpdrr8sc/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core >=2.2.0,<3'
         cwd: None
    Complete output (27 lines):
    Collecting flit_core<3,>=2.2.0
      Using cached https://files.pythonhosted.org/packages/77/72/5dda5dc417a4e702e0d7e4a77e9802792a0e4a2daec2aeed915ead7db477/flit_core-2.2.0.tar.gz
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Collecting pytoml (from flit_core<3,>=2.2.0)
      Using cached https://files.pythonhosted.org/packages/f4/ba/98ee2054a2d7b8bebd367d442e089489250b6dc2aee558b000e961467212/pytoml-0.1.21.tar.gz
    Skipping bdist_wheel for flit-core, due to binaries being disabled for it.
    Installing collected packages: pytoml, flit-core
      Running setup.py install for pytoml: started
        Running setup.py install for pytoml: finished with status 'done'
      Running setup.py install for flit-core: started
        Running setup.py install for flit-core: finished with status 'error'
        ERROR: Command errored out with exit status 1:
         command: /jsparagus/jsparagus_build_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8_dkx0yf/flit-core/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8_dkx0yf/flit-core/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-veb7s8mg/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-kpdrr8sc/overlay --compile --install-headers /jsparagus/jsparagus_build_venv/include/site/python3.8/flit-core
             cwd: /tmp/pip-install-8_dkx0yf/flit-core/
        Complete output (3 lines):
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
        ModuleNotFoundError: No module named 'setuptools'
        ----------------------------------------
    ERROR: Command errored out with exit status 1: /jsparagus/jsparagus_build_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8_dkx0yf/flit-core/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8_dkx0yf/flit-core/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-veb7s8mg/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-kpdrr8sc/overlay --compile --install-headers /jsparagus/jsparagus_build_venv/include/site/python3.8/flit-core Check the logs for full command output.
    WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    ----------------------------------------
  ERROR: Command errored out with exit status 1: /jsparagus/jsparagus_build_venv/bin/python3 /jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-kpdrr8sc/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core >=2.2.0,<3' Check the logs for full command output.
  WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /jsparagus/jsparagus_build_venv/bin/python3 /jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-w53v2n6u/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit Check the logs for full command output.
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
make: *** [Makefile:20: init-venv] Error 1

[arai-a]

sounds like issue in python or pip.
what happens if you upgrade them?

[devsnek]

@arai-a they're already at their latest versions, i think those version warnings are coming from the versions jsparagus bundles in the virtual env.

[arai-a]

Doesn't reproduce with:

• Python 3.7.6
• pip 20.0.2

on macOS

after setting up, pip list shows the following:

$ ./jsparagus_build_venv/bin/pip list
Package           Version
----------------- -------
entrypoints       0.3    
flake8            3.7.9  
html5lib          1.0.1  
mccabe            0.6.1  
mypy              0.770  
mypy-extensions   0.4.3  
pip               20.0.2 
pycodestyle       2.5.0  
pyflakes          2.1.1  
setuptools        41.2.0 
six               1.11.0 
typed-ast         1.4.1  
typing-extensions 3.7.4.1
webencodings      0.5.1  

which OS and python version are you using?

also, what happens if you create temporary virtualenv manually and installing each item in requirements.txt ?

[devsnek]

$ ./jsparagus_build_venv/bin/pip list
Package    Version
---------- -------
pip        19.2.3 
setuptools 41.2.0 
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ pip --version
pip 20.0.2 from /home/snek/.local/lib/python3.8/site-packages/pip (python 3.8)

also, what happens if you create temporary virtualenv manually and installing each item in requirements.txt ?

how do i do this

[arai-a]

what does python3 --version say ?

Then, try running the following commands one by one, until any error happens:
(this is basically what make init does)

python3 -m venv jsparagus_build_venv
./jsparagus_build_venv/bin/pip install --upgrade pip
./jsparagus_build_venv/bin/pip install entrypoints==0.3
./jsparagus_build_venv/bin/pip install flake8==3.7.9
./jsparagus_build_venv/bin/pip install html5lib==1.0.1
./jsparagus_build_venv/bin/pip install mccabe==0.6.1
./jsparagus_build_venv/bin/pip install mypy==0.770
./jsparagus_build_venv/bin/pip install mypy-extensions==0.4.3
./jsparagus_build_venv/bin/pip install pycodestyle==2.5.0
./jsparagus_build_venv/bin/pip install pyflakes==2.1.1
./jsparagus_build_venv/bin/pip install six==1.11.0
./jsparagus_build_venv/bin/pip install typed-ast==1.4.1
./jsparagus_build_venv/bin/pip install typing-extensions==3.7.4.1
./jsparagus_build_venv/bin/pip install webencodings==0.5.1

[arai-a]

sorry, before doing the above, please remove jsparagus_build_venv directory, to test on clean state

[devsnek]

$ python3 --version
Python 3.8.2

$ python3 -m venv jsparagus_build_venv
$ ./jsparagus_build_venv/bin/pip install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/8e/76/66066b7bc71817238924c7e4b448abdb17eb0c92d645769c223f9ace478f/pip-20.0.2.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Skipping bdist_wheel for pip, due to binaries being disabled for it.
Installing collected packages: pip
  Running setup.py install for pip ... done
Successfully installed pip-20.0.2
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
 ~/D/m/jsparagus ./jsparagus_build_venv/bin/pip install entrypoints==0.3
Collecting entrypoints==0.3
  Using cached https://files.pythonhosted.org/packages/b4/ef/063484f1f9ba3081e920ec9972c96664e2edb9fdc3d8669b0e3b8fc0ad7c/entrypoints-0.3.tar.gz
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/bin/python3 /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yqny98em/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit
       cwd: None
  Complete output (39 lines):
  Collecting flit
    Using cached https://files.pythonhosted.org/packages/35/87/a92625dc5c81cf1608553b27b5c216bdc1e678ea6738a3a8a69ee2e5fb51/flit-2.2.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/bin/python3 /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-1qs38kv_/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core >=2.2.0,<3'
         cwd: None
    Complete output (27 lines):
    Collecting flit_core<3,>=2.2.0
      Using cached https://files.pythonhosted.org/packages/77/72/5dda5dc417a4e702e0d7e4a77e9802792a0e4a2daec2aeed915ead7db477/flit_core-2.2.0.tar.gz
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Collecting pytoml (from flit_core<3,>=2.2.0)
      Using cached https://files.pythonhosted.org/packages/f4/ba/98ee2054a2d7b8bebd367d442e089489250b6dc2aee558b000e961467212/pytoml-0.1.21.tar.gz
    Skipping bdist_wheel for flit-core, due to binaries being disabled for it.
    Installing collected packages: pytoml, flit-core
      Running setup.py install for pytoml: started
        Running setup.py install for pytoml: finished with status 'done'
      Running setup.py install for flit-core: started
        Running setup.py install for flit-core: finished with status 'error'
        ERROR: Command errored out with exit status 1:
         command: /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7lcydly5/flit-core/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7lcydly5/flit-core/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-mse8o079/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-1qs38kv_/overlay --compile --install-headers /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/include/site/python3.8/flit-core
             cwd: /tmp/pip-install-7lcydly5/flit-core/
        Complete output (3 lines):
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
        ModuleNotFoundError: No module named 'setuptools'
        ----------------------------------------
    ERROR: Command errored out with exit status 1: /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7lcydly5/flit-core/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7lcydly5/flit-core/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-mse8o079/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-1qs38kv_/overlay --compile --install-headers /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/include/site/python3.8/flit-core Check the logs for full command output.
    WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    ----------------------------------------
  ERROR: Command errored out with exit status 1: /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/bin/python3 /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-1qs38kv_/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core >=2.2.0,<3' Check the logs for full command output.
  WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/bin/python3 /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yqny98em/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- flit Check the logs for full command output.
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[arai-a]

Are you using debian-based distribution?
If so, I think you have to upgrade python3-pip package.
as the message says, there's new pip version available.

if not, can you tell me which OS and distro and versions you're using?

let me know if upgrading pip doesn't solve the issue.

[devsnek]

@arai-a i'm using arch, and all my packages are up to date (rolling release)

additionally, my pip is up to date:

$ pip --version
pip 20.0.2 from /home/snek/.local/lib/python3.8/site-packages/pip (python 3.8)

The out of date pip appears to be coming from virtualenv:

$ ./jsparagus_build_venv/bin/pip --version
pip 19.2.3 from /home/snek/Desktop/misc/jsparagus/jsparagus_build_venv/lib/python3.8/site-packages/pip (python 3.8)

[arai-a]

what does virtualenv --version say ?

[devsnek]

@arai-a 16.6.2

[arai-a]

I don't know much about arch tho, there seems to be python-virtualenv package
https://www.archlinux.org/packages/?sort=&q=virtualenv&maintainer=&flagged=
and its version is 20.0.15-1.

did you install virtualenv with that package? or somewhere else?

[devsnek]

virtualenv seems to have been some python2 package (i never installed it though so idk where it came from). I removed it and installed python-virtualenv but i'm still having the same problem. same pip version warning, same errors.

[arai-a]

what does virtualenv --version say now?

[devsnek]

virtualenv 20.0.15 from /usr/lib/python3.8/site-packages/virtualenv/__init__.py

[arai-a]

does the issue persist if you remove jsparagus_build_venv directory and run make init again?

[devsnek]

yep

[arai-a]

what happens if you run the following commands, in some temporary empty directory outside of jsparagus?

python3 -m venv venv
./venv/bin/pip --version
./venv/bin/pip install --upgrade pip

[devsnek]

I don't think the init commands are correctly entering the virtualenv. That being said, I also don't know what the correct way is.

$ python3 -m venv venv
$ ./venv/bin/pip --version
pip 19.2.3 from /home/snek/venv/lib/python3.8/site-packages/pip (python 3.8)
$ ./venv/bin/pip install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/8e/76/66066b7bc71817238924c7e4b448abdb17eb0c92d645769c223f9ace478f/pip-20.0.2.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Skipping bdist_wheel for pip, due to binaries being disabled for it.
Installing collected packages: pip
  Running setup.py install for pip ... done
Successfully installed pip-20.0.2
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ ./venv/bin/pip --version
pip 19.2.3 from /home/snek/venv/lib/python3.8/site-packages/pip (python 3.8)

[arai-a]

pip 19.2.3 gets installed into the virtual env sounds very suspicious.
(that would mean, you have multiple pip versions somewhere, or virtualenv package is configured incorrectly)
also, given that it happens outside of jsparagus directory, it's your system-wide issue.

it would be nice to look into it.

then, about entering virtualenv, in most case running command without activating the venv works,
but might be nice to test with activating it.
try the following in jsparagus directory.

rm -rf jsparagus_build_venv
python3 -m venv jsparagus_build_venv
. jsparagus_build_venv/bin/activate
pip --version
pip install --upgrade pip
pip --version
pip install entrypoints==0.3
pip install flake8==3.7.9
pip install html5lib==1.0.1
pip install mccabe==0.6.1
pip install mypy==0.770
pip install mypy-extensions==0.4.3
pip install pycodestyle==2.5.0
pip install pyflakes==2.1.1
pip install six==1.11.0
pip install typed-ast==1.4.1
pip install typing-extensions==3.7.4.1
pip install webencodings==0.5.1

[devsnek]

@arai-a same issue while activated.

[arai-a]

can you provide the exact output?
in which command it fails?

[devsnek]

Ok I am learning some things:

https://unix.stackexchange.com/questions/464671/why-does-pythons-pip-reset-to-version-10-0-1-in-every-new-virtual-environment

diff --git a/Makefile b/Makefile
index b94b5b7..032f5ab 100644
--- a/Makefile
+++ b/Makefile
@@ -17,8 +17,9 @@ PIP := $(VENV_BIN_DIR)/pip
 all: $(PY_OUT) rust
 
 init-venv:
-       python3 -m venv jsparagus_build_venv &&\
-       $(PIP) install --upgrade pip &&\
+       python3 -m virtualenv jsparagus_build_venv && \
+       $(PIP) config set install.user no && \
+       $(PIP) install --upgrade pip && \
        $(PIP) install -r requirements.txt
 
 init: init-venv

It's all working now

[arai-a]

$(PIP) config set install.user no

this will be written into your user configuration (somewhere around ~/.config/pip/pip.conf) I think,
so the command shouldn't be here.
also, if this solves the issue, I guess your user configuration was affecting the pip command.

I'll check if there's a way to avoid getting affected from it.

[devsnek]

the real fix was changing venv to virtualenv, as they apparently are not the same thing. I guess the config line isn't needed.

[arai-a]

Just to make sure, can you test the following?

1. remove jsparagus_build_venv directory
2. revert all changes, run make init again, to see if it still fails
3. remove jsparagus_build_venv directory
4. replace venv with virtualenv in Makefile, and run make init again, to see if it succeeds

[devsnek]

seems like unmodified makefile works now... idk what changed 🤷‍♂️

[arai-a]

okay, then it should be that your user configuration was affecting pip behavior.
I'll try to reproduce the issue with modified pip.conf, and check if there's a workaround.