Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PYTHON-4976 Replace hatch with uv as our python environment and workfow tool #2068

Merged
merged 45 commits into from
Jan 22, 2025
Merged
Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
e8cb249
PYTHON-4976 Replace hatch with uv as our python environment and workf…
blink1073 Jan 18, 2025
81daf54
remove todos
blink1073 Jan 18, 2025
aee4e49
clean up setup dev script
blink1073 Jan 18, 2025
e180e92
restore missing logic
blink1073 Jan 18, 2025
5e492b4
cleanup
blink1073 Jan 18, 2025
e50f30c
cleanup
blink1073 Jan 18, 2025
a116b18
drop pypy 3.9 support and fix run-tests
blink1073 Jan 18, 2025
7e34658
fix doctest
blink1073 Jan 18, 2025
18c3411
fix windows and kms
blink1073 Jan 18, 2025
fb2e91d
clean up dependency handling
blink1073 Jan 18, 2025
d72e246
cleanup
blink1073 Jan 18, 2025
225b419
fix handling of uv
blink1073 Jan 18, 2025
0f75704
fix handling of just
blink1073 Jan 18, 2025
056d02c
add files
blink1073 Jan 19, 2025
6595302
add files
blink1073 Jan 19, 2025
f75de9a
add files
blink1073 Jan 19, 2025
2e81e3a
add files
blink1073 Jan 19, 2025
a8dbd8a
add files
blink1073 Jan 19, 2025
8746ba5
add files
blink1073 Jan 19, 2025
821c1bf
add files
blink1073 Jan 19, 2025
4034422
add files
blink1073 Jan 19, 2025
8e22999
add files
blink1073 Jan 19, 2025
f24128e
add files
blink1073 Jan 19, 2025
b647621
add files
blink1073 Jan 19, 2025
fa153e5
add files
blink1073 Jan 19, 2025
8f1519a
add files
blink1073 Jan 19, 2025
b2fa299
add files
blink1073 Jan 19, 2025
a2b01c6
add files
blink1073 Jan 19, 2025
4502ada
fix handling of files
blink1073 Jan 19, 2025
9eb9700
clean up install
blink1073 Jan 20, 2025
468a788
clean up install
blink1073 Jan 20, 2025
84d8cdf
see if certs are causing problems
blink1073 Jan 20, 2025
de92978
try another aws fix
blink1073 Jan 20, 2025
8a621f2
clean up install
blink1073 Jan 20, 2025
8b6f224
clean up install
blink1073 Jan 20, 2025
2ca87ce
fix handling of DRIVERS_TOOLS_BINARIES
blink1073 Jan 21, 2025
47bb41f
fix gha errors
blink1073 Jan 21, 2025
6b6884b
lint
blink1073 Jan 21, 2025
9ff8bad
fix link
blink1073 Jan 21, 2025
bae10df
Address review
blink1073 Jan 21, 2025
c19673f
Merge branch 'master' of github.com:mongodb/mongo-python-driver into …
blink1073 Jan 21, 2025
1a05fd4
fix merge
blink1073 Jan 21, 2025
ba43e87
cleanup
blink1073 Jan 21, 2025
08d1290
clean up typing
blink1073 Jan 21, 2025
8b9d90a
remove files
blink1073 Jan 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 12 additions & 52 deletions .evergreen/generated_configs/variants.yml
Original file line number Diff line number Diff line change
Expand Up @@ -256,15 +256,15 @@ buildvariants:
expansions:
COMPRESSORS: zstd
PYTHON_BINARY: /opt/python/3.9/bin/python3
- name: compression-snappy-rhel8-pypy3.9
- name: compression-snappy-rhel8-pypy3.10
tasks:
- name: .standalone .noauth .nossl .sync_async
display_name: Compression snappy RHEL8 PyPy3.9
display_name: Compression snappy RHEL8 PyPy3.10
run_on:
- rhel87-small
expansions:
COMPRESSORS: snappy
PYTHON_BINARY: /opt/python/pypy3.9/bin/python3
PYTHON_BINARY: /opt/python/pypy3.10/bin/python3
- name: compression-zlib-rhel8-pypy3.10
tasks:
- name: .standalone .noauth .nossl .sync_async
Expand All @@ -274,15 +274,15 @@ buildvariants:
expansions:
COMPRESSORS: zlib
PYTHON_BINARY: /opt/python/pypy3.10/bin/python3
- name: compression-zstd-rhel8-pypy3.9
- name: compression-zstd-rhel8-pypy3.10
tasks:
- name: .standalone .noauth .nossl .sync_async !.4.0
display_name: Compression zstd RHEL8 PyPy3.9
display_name: Compression zstd RHEL8 PyPy3.10
run_on:
- rhel87-small
expansions:
COMPRESSORS: zstd
PYTHON_BINARY: /opt/python/pypy3.9/bin/python3
PYTHON_BINARY: /opt/python/pypy3.10/bin/python3

# Disable test commands tests
- name: disable-test-commands-rhel8-python3.9
Expand Down Expand Up @@ -460,15 +460,6 @@ buildvariants:
test_encryption: "true"
test_encryption_pyopenssl: "true"
PYTHON_BINARY: /opt/python/3.12/bin/python3
- name: encryption-rhel8-pypy3.9
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep this variant with PyPy3.10 instead?

tasks:
- name: .sharded_cluster .auth .ssl .sync_async
display_name: Encryption RHEL8 PyPy3.9
run_on:
- rhel87-small
expansions:
test_encryption: "true"
PYTHON_BINARY: /opt/python/pypy3.9/bin/python3
- name: encryption-macos-python3.9
tasks:
- name: .latest .replica_set .sync_async
Expand Down Expand Up @@ -608,15 +599,6 @@ buildvariants:
expansions:
AUTH: auth
PYTHON_BINARY: C:/python/Python313/python.exe
- name: auth-enterprise-rhel8-pypy3.9-auth
tasks:
- name: test-enterprise-auth
display_name: Auth Enterprise RHEL8 PyPy3.9 Auth
run_on:
- rhel87-small
expansions:
AUTH: auth
PYTHON_BINARY: /opt/python/pypy3.9/bin/python3
- name: auth-enterprise-rhel8-pypy3.10-auth
tasks:
- name: test-enterprise-auth
Expand Down Expand Up @@ -900,10 +882,10 @@ buildvariants:
TOPOLOGY: server
VERSION: "8.0"
PYTHON_BINARY: /opt/python/3.13/bin/python3
- name: ocsp-rhel8-rapid-pypy3.9
- name: ocsp-rhel8-rapid-pypy3.10
tasks:
- name: .ocsp
display_name: OCSP RHEL8 rapid PyPy3.9
display_name: OCSP RHEL8 rapid PyPy3.10
run_on:
- rhel87-small
batchtime: 20160
Expand All @@ -912,11 +894,11 @@ buildvariants:
SSL: ssl
TOPOLOGY: server
VERSION: rapid
PYTHON_BINARY: /opt/python/pypy3.9/bin/python3
- name: ocsp-rhel8-latest-pypy3.10
PYTHON_BINARY: /opt/python/pypy3.10/bin/python3
- name: ocsp-rhel8-latest-python3.9
tasks:
- name: .ocsp
display_name: OCSP RHEL8 latest PyPy3.10
display_name: OCSP RHEL8 latest Python3.9
run_on:
- rhel87-small
batchtime: 20160
Expand All @@ -925,7 +907,7 @@ buildvariants:
SSL: ssl
TOPOLOGY: server
VERSION: latest
PYTHON_BINARY: /opt/python/pypy3.10/bin/python3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we split the PyPy3.9 deprecation out into a separate PR? It would be cleaner to separate that from switching to uv.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: #2070

PYTHON_BINARY: /opt/python/3.9/bin/python3
- name: ocsp-win64-v4.4-python3.9
tasks:
- name: .ocsp-rsa !.ocsp-staple
Expand Down Expand Up @@ -1061,17 +1043,6 @@ buildvariants:
expansions:
test_pyopenssl: "true"
PYTHON_BINARY: C:/python/Python313/python.exe
- name: pyopenssl-rhel8-pypy3.9
tasks:
- name: .replica_set .auth .ssl .sync_async
- name: .7.0 .auth .ssl .sync_async
display_name: PyOpenSSL RHEL8 PyPy3.9
run_on:
- rhel87-small
batchtime: 10080
expansions:
test_pyopenssl: "true"
PYTHON_BINARY: /opt/python/pypy3.9/bin/python3
- name: pyopenssl-rhel8-pypy3.10
tasks:
- name: .replica_set .auth .ssl .sync_async
Expand Down Expand Up @@ -1164,17 +1135,6 @@ buildvariants:
expansions:
COVERAGE: coverage
PYTHON_BINARY: /opt/python/3.12/bin/python3
- name: test-rhel8-pypy3.9
tasks:
- name: .sharded_cluster .auth .ssl .sync_async
- name: .replica_set .noauth .ssl .sync_async
- name: .standalone .noauth .nossl .sync_async
display_name: "* Test RHEL8 PyPy3.9"
run_on:
- rhel87-small
expansions:
COVERAGE: coverage
PYTHON_BINARY: /opt/python/pypy3.9/bin/python3
- name: test-macos-python3.9
tasks:
- name: .sharded_cluster .auth .ssl !.sync_async
Expand Down
81 changes: 48 additions & 33 deletions .evergreen/install-dependencies.sh
Original file line number Diff line number Diff line change
@@ -1,45 +1,60 @@
#!/bin/bash
set -eu

# Copy PyMongo's test certificates over driver-evergreen-tools'
cp ${PROJECT_DIRECTORY}/test/certificates/* ${DRIVERS_TOOLS}/.evergreen/x509gen/

# Replace MongoOrchestration's client certificate.
cp ${PROJECT_DIRECTORY}/test/certificates/client.pem ${MONGO_ORCHESTRATION_HOME}/lib/client.pem

if [ -w /etc/hosts ]; then
SUDO=""
else
SUDO="sudo"
fi

# Install just.
# Set where binaries are expected to be.
# On Evergreen jobs, "CI" will be set, and we don't want to write to $HOME.
if [ "${CI:-}" == "true" ]; then
BIN_DIR=$DRIVERS_TOOLS_BINARIES
_BIN_DIR=${DRIVERS_TOOLS_BINARIES:-}
# On Windows spawn hosts, $HOME/cli_bin is on the PATH.
elif [ "Windows_NT" = "${OS:-}" ]; then
_BIN_DIR=$HOME/cli_bin
# On local machines and Linux spawn hosts, we expect $HOME/.local/bin to be on the PATH.
else
BIN_DIR=$HOME/.local/bin
_BIN_DIR=$HOME/.local/bin
fi
if [ ! -f $BIN_DIR/just ]; then
mkdir -p $_BIN_DIR 2>/dev/null || true

function _pip_install() {
_HERE=$(dirname ${BASH_SOURCE:-$0})
. $_HERE/utils.sh
_VENV_PATH=$(mktemp -d)
echo "Installing $2 using pip..."
createvirtualenv "$(find_python3)" $_VENV_PATH
python -m pip install $1
ln -s "$(which $2)" $_BIN_DIR/$2
echo "Installing $2 using pip... done."
}

function _curl() {
curl --tlsv1.2 -LsSf $1
}

# Install just.
if ! command -v just 2>/dev/null; then
_TARGET=""
if [ "Windows_NT" = "${OS:-}" ]; then
TARGET="--target x86_64-pc-windows-msvc"
else
TARGET=""
_TARGET="--target x86_64-pc-windows-msvc"
fi
curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- $TARGET --to "$BIN_DIR" || {
# CARGO_HOME is defined in configure-env.sh
export CARGO_HOME=${CARGO_HOME:-$HOME/.cargo/}
export RUSTUP_HOME="${CARGO_HOME}/.rustup"
. ${DRIVERS_TOOLS}/.evergreen/install-rust.sh
cargo install just
if [ "Windows_NT" = "${OS:-}" ]; then
mv $CARGO_HOME/just.exe $BIN_DIR/just
else
mv $CARGO_HOME/just $BIN_DIR
fi
# On most systems we can install directly.
echo "Installing just..."
_curl https://just.systems/install.sh | bash -s -- $_TARGET --to "$_BIN_DIR" || {
_pip_install rust-just just
}
if ! command -v just 2>/dev/null; then
export PATH="$PATH:$_BIN_DIR"
fi
echo "Installing just... done."
fi

# Add 'server' and 'hostname_not_in_cert' as a hostnames
echo "127.0.0.1 server" | $SUDO tee -a /etc/hosts
echo "127.0.0.1 hostname_not_in_cert" | $SUDO tee -a /etc/hosts
# Install uv.
if ! command -v uv 2>/dev/null; then
echo "Installing uv..."
# On most systems we can install directly.
_curl https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="$_BIN_DIR" INSTALLER_NO_MODIFY_PATH=1 sh || {
_pip_install uv uv
}
if ! command -v uv 2>/dev/null; then
export PATH="$PATH:$_BIN_DIR"
fi
echo "Installing uv... done."
fi
13 changes: 9 additions & 4 deletions .evergreen/run-azurekms-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,22 @@ export AZUREKMS_VMNAME=${AZUREKMS_VMNAME}
export AZUREKMS_PRIVATEKEYPATH=/tmp/testazurekms_privatekey
export LIBMONGOCRYPT_URL=https://s3.amazonaws.com/mciuploads/libmongocrypt/debian11/master/latest/libmongocrypt.tar.gz
SKIP_SERVERS=1 bash $HERE/setup-encryption.sh
tar czf /tmp/mongo-python-driver.tgz .
# Set up the remote files to test.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these changes related to the uv conversion?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

git add .
git commit -m "add files" || true
git archive -o /tmp/mongo-python-driver.tar HEAD
tar -rf /tmp/mongo-python-driver.tar libmongocrypt
gzip -f /tmp/mongo-python-driver.tar
# shellcheck disable=SC2088
AZUREKMS_SRC="/tmp/mongo-python-driver.tgz" AZUREKMS_DST="~/" \
AZUREKMS_SRC="/tmp/mongo-python-driver.tar.gz" AZUREKMS_DST="~/" \
$DRIVERS_TOOLS/.evergreen/csfle/azurekms/copy-file.sh
echo "Copying files ... end"
echo "Untarring file ... begin"
AZUREKMS_CMD="tar xf mongo-python-driver.tgz" \
AZUREKMS_CMD="tar xf mongo-python-driver.tar.gz" \
$DRIVERS_TOOLS/.evergreen/csfle/azurekms/run-command.sh
echo "Untarring file ... end"
echo "Running test ... begin"
AZUREKMS_CMD="KEY_NAME=\"$AZUREKMS_KEYNAME\" KEY_VAULT_ENDPOINT=\"$AZUREKMS_KEYVAULTENDPOINT\" SUCCESS=true TEST_FLE_AZURE_AUTO=1 ./.evergreen/just.sh test-eg" \
AZUREKMS_CMD="pwd && ls && KEY_NAME=\"$AZUREKMS_KEYNAME\" KEY_VAULT_ENDPOINT=\"$AZUREKMS_KEYVAULTENDPOINT\" SUCCESS=true TEST_FLE_AZURE_AUTO=1 bash ./.evergreen/just.sh test-eg" \
$DRIVERS_TOOLS/.evergreen/csfle/azurekms/run-command.sh
echo "Running test ... end"
bash $HERE/teardown-encryption.sh
11 changes: 8 additions & 3 deletions .evergreen/run-gcpkms-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,16 @@ export GCPKMS_ZONE=${GCPKMS_ZONE}
export GCPKMS_INSTANCENAME=${GCPKMS_INSTANCENAME}
export LIBMONGOCRYPT_URL=https://s3.amazonaws.com/mciuploads/libmongocrypt/debian11/master/latest/libmongocrypt.tar.gz
SKIP_SERVERS=1 bash $HERE/setup-encryption.sh
tar czf /tmp/mongo-python-driver.tgz .
GCPKMS_SRC=/tmp/mongo-python-driver.tgz GCPKMS_DST=$GCPKMS_INSTANCENAME: $DRIVERS_TOOLS/.evergreen/csfle/gcpkms/copy-file.sh
# Set up the remote files to test.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here as above.

git add .
git commit -m "add files" || true
git archive -o /tmp/mongo-python-driver.tar HEAD
tar -rf /tmp/mongo-python-driver.tar libmongocrypt
gzip -f /tmp/mongo-python-driver.tar
GCPKMS_SRC=/tmp/mongo-python-driver.tar.gz GCPKMS_DST=$GCPKMS_INSTANCENAME: $DRIVERS_TOOLS/.evergreen/csfle/gcpkms/copy-file.sh
echo "Copying files ... end"
echo "Untarring file ... begin"
GCPKMS_CMD="tar xf mongo-python-driver.tgz" $DRIVERS_TOOLS/.evergreen/csfle/gcpkms/run-command.sh
GCPKMS_CMD="tar xf mongo-python-driver.tar.gz" $DRIVERS_TOOLS/.evergreen/csfle/gcpkms/run-command.sh
echo "Untarring file ... end"
echo "Running test ... begin"
GCPKMS_CMD="SUCCESS=true TEST_FLE_GCP_AUTO=1 ./.evergreen/just.sh test-eg" $DRIVERS_TOOLS/.evergreen/csfle/gcpkms/run-command.sh
Expand Down
Loading
Loading