Porting to Python 3 #6062
Comments
|
Hi @stuhood - thanks for the interest. Foursquare will work with Eric on the patch but technical advice and review from the Pants community makes a tremendous difference. Stu and I chatted about this before opening the issue - he suggested that @benjyw, @CMLivingston and @kwlzn would be interested in the discussion. |
|
Also pinging @jsirois on this, because he should be able to answer questions about some of the CI infrastructure changes needed. The first few steps look reasonable to me though. |
|
awesome to have you working on this @Eric-Arellano! based on what I've seen floating around for Python 3.7.x perf improvements (with 3.7.0 just released last week and 3.7.1 expected in July), its possible that this effort could lead to a significant runtime speedup once pants can run seamlessly on 3.7.x. so it'd be super interesting to post some early benchmark numbers for various pants invocations (e.g. one quick thought on the path forward for both 2.7 + 3.x testing is that running the entirety of CI against both interpreter versions will effectively ~double the CI runtime (which is already quite lengthy at around an ~hour and a half). given that added cost, it could make sense to hasten a jump to 3.x as the "recommended" python version for pants with a strategy that could look something like:
from then on, we could then begin to utilize 3.x features (like e.g. data classes) in pants' codebase with the need for 2.7 compatibility behind us. we could also peel away the In terms of blockers, I believe there is at least one known issue to solve related to pants' execution interpreter vs target interpreters for working with python targets (which is currently biting us for 2.x -> 3.x IIRC) - so it'd be good to sync up with @CMLivingston on that so we can all be working strategically to unblock the initial dogfooding. |
|
@kwlzn I am completely on board with that - great strategy. Python and most key libraries are dropping official support by January 1 2020 (new versions of Numpy, Scikit, Django, Pylint, etc already dropped support in newest versions). Hence, I think a lot of organizations are this year finally dealing with Python3 migration, and it is much more tractable to urge them to upgrade now. Once we drop Python2, removing |
|
Ah, this is awesome! We've been circling this for a while, glad someone's finally going in for a bite! I agree with @kwlzn that there is no need for long-term support for running Pants on 2.7. The interpreter Pants runs on is sort-of an implementation detail, not parts of its API. Although, as you've discovered, the fact that your own code (specifically, custom pants plugins) can use Pants as a library muddies those waters... So an eager switch is 💯. |
…ty. (#6063) ### Problem First stage of [Python 3 port](#6062) ### Solution Stage 1 of the futurize tool only makes safe changes, and it keeps the code base at Python 2. See http://python-future.org/futurize.html#forwards-conversion-stage1. Stage 2 will later convert this to actual Python 3 code. This command was ran on - src/python - test/python - contrib - examples - pants-plugins - build-support/bin/check-header-helper.py (not ran on testprojects)
### Description Implements stage 2 of [porting plan](#6062), which is adding the future library. Also ports the `src/pants/base` and `tests/python/pants_test/base` as a demo of how this tool works. At this stage, we aren't testing things yet with the Python 3 interpreter, even though technically it's Python 3 code. The goal for now is to make sure things still work on Python 2. #### FYI - how the backports work The `future` library adds several modules: `future`, `builtins`, `past`, and individual ports like `queue` (instead of `Queue`). These added names all correspond exactly to Python 3 code. When running Python 3, it will simply use the native Python 3 version. When using Python 2, it will use the library implementation, which is semantically equivalent to Python 3. #### FYI - using old `moves` interface Typically `futurize` will try to use this idiom for certain backports ```python from future import standard_library standard_library.install_aliases() import [backports] ``` This however does not work in our codebase, because all imports must appear above code with iSort, so the `install_aliases()` line gets moved below the relevant imports. Instead, we are using the legacy `future.moves` library. Once we remove Python 2 support we'll want to convert this type of import `from future.moves.itertools import x` to `from itertools import x`.
Part of #6062 ## Problem `ABCMeta` expects `unicode` on Python3, but `bytes` on Python2. ## Solution We use feature detection to try with idiomatic Py3 and fall back on Py2. This is [preferred over version detection](https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection) (i.e. checking which Python interpreter is used). Note that the source of this file, [`twitter.commons.lang`](https://github.com/twitter/commons/blob/master/src/python/twitter/common/lang/__init__.py#L179), did not have to do this because they don't use `from __future__ import unicode_literals`.
Part of porting project #6062 CI fails for me on mac (we discussed this in slack), so hopefully this works since I haven't tested it..
Part of #6062 P.S. Let me know if these PRs are too small - I've been keeping it one directory per PR so that we can use `git bisect` / debug easier and make it easier to do code review. This means we'll eventually have 54 PRs in total ([spreadsheet tracking this all](https://docs.google.com/spreadsheets/d/1i4xFUrgejVFp6NSwT9aGBQWyjbiDIuqMqGEIxcSMCC0/edit#gid=0))
### Problem In order to avoid backsliding as changes are made in service of pantsbuild#6062, we should begin running a travis shard that uses python 3 even before it has gone completely green. ### Solution Repurpose one of the unit test shards to run with python 3, with abbreviated logging and `|| exit 0`. This should allow us to track the total failure count under python 3, and attempt to push it down.
|
Update: all unit tests are now passing on both Python 2 and Python 3! The next stage is getting all integration tests passing for Python 3. As discussed in Slack, our overall CI strategy will be to run Python 3 for all integration shards, and then run a nightly Python 2 job to check for regressions. Since some integration tests still fail on Python 3, here is the path forward:
|
|
Update: all tests—integration and unit—are now passing Python 3! The next step is getting Pants to work with Python 3 under-the-hood. To test this out, apply the following diff: diff --git a/build-support/virtualenv b/build-support/virtualenv
index d34e285fa..0f8d617a8 100755
--- a/build-support/virtualenv
+++ b/build-support/virtualenv
@@ -14,10 +14,10 @@ fi
source ${REPO_ROOT}/build-support/common.sh
if [[ -z "${PY}" ]]; then
- if which python2.7 >/dev/null; then
- PY=`which python2.7`
+ if which python3 >/dev/null; then
+ PY=`which python3`
else
- die 'No python2.7 interpreter found on the path. Python will not work!'
+ die 'No python3 interpreter found on the path. Python will not work!'
fi
fi
diff --git a/pants.ini b/pants.ini
index ba51a2631..a60b2948b 100644
--- a/pants.ini
+++ b/pants.ini
@@ -372,7 +372,7 @@ verify_commit: False
# We only support pants running under 2.7 for now with 3.3+ support to be added later.
# Any example or test targets that are meant to test interpreters outside pants own
# acceptable set should specify an explicit compatibility constraint.
-interpreter_constraints: ["CPython>=2.7,<3"]
+interpreter_constraints: ["CPython>=2.7,<3", "CPython>=3.6"]
interpreter_cache_dir: %(pants_bootstrapdir)s/python_cache/interpreters
resolver_cache_dir: %(pants_bootstrapdir)s/python_cache/requirementsThere are several issues newly surfaced, which will hopefully be closed out soon. |
### Problem The final step in #6062 is to add support for releasing Pants and its contrib packages as compatible with Python 2.7 _and_ 3.6 and 3.7. Specifically, universal wheels should be marked as `py27.py36.py37` and `pantsbuild.pants` should be marked as `cp27-cp27{m,mu}` or as `cp36-abi3` because it uses native code (see [comment in packages.py](https://github.com/pantsbuild/pants/pull/7197/files#diff-cccea5c0944f25138aa83f5815236ea2R95) for an explanation on `abi3`). `packages.py` and `release.sh` had several issues preventing building Python 3 wheels. Further, we needed to add shards to build Python 3 wheels in CI. This resolves the wheels portion of #6450. ### Solution * Add a `-3` CLI flag to `release.sh` to indicate it should build wheels compatible with Py3. Also add `--py3` to `packages.py`. * Update the default `Package` config to setup support for `py27.py36.py37`, meaning any Python implementation that is one of those 3 versions. * Update the `pantsbuild.pants` flags to indicate either `cp27` or `cp36-abi3`. The former is only compatible with CPython2.7, and the latter is compatible with any CPython implementation >= 3.6 due to the ABI constraint (see https://docs.python.org/3/c-api/stable.html). * Add Python 3 wheels shards to CI. * Pull down either a pex27 or pex36 release, rather than alway using pex27. * Modernize Python code, such as fixing bad `subprocess32` import. * Remove the compatibility tags from `PantsRequirement`. John explained it was wrong to put here, I think because it should be marked in` packages.py` when we actually build the wheels? This results in also tweaking the corresponding unit tests. ### Result `./build-support/bin/release.sh -3n` builds Python 3 compatible wheels ready for release. Note we still cannot release a Python 3 compatible PEX, which is blocked by pex-tool/pex#654.
### Problem Now that we're releasing Pants with Python 3 wheels (pantsbuild/pants#7197, overall migration tracked by pantsbuild/pants#6062), we need to add a way for every day Pants users to consume the upgrade. See #30 for more context, including the design goals. ### Solution Parse the `pants_runtime_python_version` from `pants.ini` to determine which Python interpreter to use. If it's missing, default to Python 2.7. This requires modifying the venv folders to support multiple Python versions. Rather than using the naming scheme `${pants_version}`, we now consistently use `${pants_version}_py{py_major_minor}`, regardless of if the user is using `pants_runtime_python_version`. It also requires upgrading virtualenv to avoid a deprecation warning when running with Python 3. Finally, we add tests to ensure this all works. ### Result Running `./pants` like normal for the first time will bootstrap Pants again and create the new folder `1.15.dev3_py{27,36,37}`. Changing the `$PYTHON` env var or the `pants_runtime_python_version` in `pants.ini` will change which Python version is used to run Pants. When changing interpreters for the first time, it will bootstrap Pants again. After that, you can change without issue.
|
Update: the migration is now mostly complete, as we now release Pants as Python 3 wheel and Python 3 Pex. 1.15.0 will be the first stable version to support Python 3, and 1.16.0 will be the last to support Python 2. I've added remaining tasks to the original issue description, which involve fixing known issues and getting Python 3.7 added to CI. |
Python 3 is now completely supported. Part of #6062.
|
We now fully support Python 3! Our first stable release with Python 3 is now public. If you've been using Pants already, refer to https://groups.google.com/forum/#!topic/pants-devel/ii-kA73qyxk for instructions. The next part of this project is to make these code improvements once we drop Python 2.7 in Pants 1.17.0.dev0: https://github.com/pantsbuild/pants/projects/12. |
|
🥇 🎉 🎉 |
Hello, I'm an intern from Foursquare working with @mateor and @iantabolt to migrate internally to Python 3, and one of our blockers is not being able to import Pants within Python 3.
I'm hoping to be able to spearhead migrating Pants to Python 3 that is backwards compatible with 2.7.
Scope
Porting guides
Official Python guide
Python-future library, tools and imports to maintain interopable codebase
Tasks
futurize --stage1for automatic and safe fixes; code still Python 2futurelibrary, which provides cross-compatibility functions and tools (expandedsix)futurize --stage2for automatic yet unsafe changes. Still Python 2, although using Python 3 semantics in most places../pantsis invoked Fixed overhead for python3 vs python2 in pantsbuild/pants #7373The text was updated successfully, but these errors were encountered: