Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRIVERS-3032 Add Python CLI for drivers-evergreen-tools orchestration #578

Merged
merged 91 commits into from
Jan 27, 2025
Merged
Changes from 1 commit
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
64d9c4c
DRIVERS-3032 Add Python CLI for drivers-evergreen-tools orchestration
blink1073 Dec 18, 2024
0826dea
fix requireApiVersion handling
blink1073 Dec 18, 2024
c700e99
pass args
blink1073 Dec 18, 2024
d4e28f7
add missing dep
blink1073 Dec 18, 2024
05a8d87
use git branch
blink1073 Dec 18, 2024
cdfe02e
use git branch
blink1073 Dec 18, 2024
42a91c7
use release
blink1073 Dec 18, 2024
af0fe48
debug
blink1073 Dec 18, 2024
5c2e77f
debug
blink1073 Dec 18, 2024
973733c
debug
blink1073 Dec 18, 2024
7cdbaee
add to PATH
blink1073 Dec 18, 2024
e09db95
debug
blink1073 Dec 18, 2024
5b9dce8
debug
blink1073 Dec 18, 2024
6653a4e
debug
blink1073 Dec 18, 2024
974055b
use python to run
blink1073 Dec 18, 2024
55ebffc
cleanup
blink1073 Dec 18, 2024
339e8ee
cleanup
blink1073 Dec 18, 2024
d789481
cleanup
blink1073 Dec 18, 2024
3048008
cleanup
blink1073 Dec 18, 2024
fa5ba06
cleanup
blink1073 Dec 18, 2024
159a307
fix results
blink1073 Dec 18, 2024
aa8da58
fix setup
blink1073 Dec 18, 2024
61996fa
cleanup
blink1073 Dec 18, 2024
9c99620
try node 20
blink1073 Dec 18, 2024
92a4c77
fix docker
blink1073 Dec 19, 2024
602305c
fix oidc
blink1073 Dec 19, 2024
8d674df
fix docker
blink1073 Dec 19, 2024
8feee45
fix docker
blink1073 Dec 19, 2024
c06aa7f
debug docker
blink1073 Dec 19, 2024
de61002
debug docker
blink1073 Dec 19, 2024
5b53228
fix handling of mo-expansion.sh
blink1073 Dec 19, 2024
c8a8063
fix handling of auth and ssl
blink1073 Dec 19, 2024
d1fec45
cleanup
blink1073 Dec 19, 2024
c05e483
Merge branch 'master' of github.com:mongodb-labs/drivers-evergreen-to…
blink1073 Dec 20, 2024
eb34155
refactor
blink1073 Dec 20, 2024
905472c
fix option handling
blink1073 Dec 20, 2024
ea2e311
debug windows
blink1073 Dec 20, 2024
c02506d
debug windows
blink1073 Dec 21, 2024
0d44675
debug windows
blink1073 Dec 21, 2024
c8ce36d
fix handling of crypt shared
blink1073 Dec 21, 2024
8dd01a3
more verbose
blink1073 Dec 21, 2024
07f5833
another check
blink1073 Dec 21, 2024
b1cc565
use posixpath
blink1073 Dec 21, 2024
9e47b61
suppress stderr by default
blink1073 Dec 21, 2024
5867f06
fix handling of paths
blink1073 Dec 21, 2024
dac9bf1
wip
blink1073 Dec 21, 2024
c038bae
Merge branch 'master' of github.com:mongodb-labs/drivers-evergreen-to…
blink1073 Jan 3, 2025
9daeb7e
use libs
blink1073 Jan 3, 2025
ee883b1
cleanup
blink1073 Jan 4, 2025
8992328
fix path handling
blink1073 Jan 4, 2025
54dee1f
fix path handling
blink1073 Jan 4, 2025
a631c30
fix path handling
blink1073 Jan 4, 2025
745e9f9
fix path handling
blink1073 Jan 4, 2025
8432ee7
try as a subprocess
blink1073 Jan 4, 2025
3f1d45b
fix subprocess
blink1073 Jan 4, 2025
360c7ac
fix subprocess
blink1073 Jan 4, 2025
59cbd0f
fix subprocess
blink1073 Jan 4, 2025
b7a1995
try with pipes
blink1073 Jan 4, 2025
7272c68
fix subprocess
blink1073 Jan 4, 2025
f687f0f
try again
blink1073 Jan 4, 2025
45b91bc
fix subprocess
blink1073 Jan 4, 2025
5fdf93c
debug
blink1073 Jan 4, 2025
793dd16
debug
blink1073 Jan 4, 2025
336338d
debug
blink1073 Jan 4, 2025
9f45d9e
more debug
blink1073 Jan 4, 2025
dc05a7a
try again
blink1073 Jan 4, 2025
5db57a3
fixups
blink1073 Jan 4, 2025
a4cef9f
add cli to path
blink1073 Jan 4, 2025
3072257
Merge branch 'master' of github.com:mongodb-labs/drivers-evergreen-to…
blink1073 Jan 6, 2025
4643825
propagate logging levels
blink1073 Jan 6, 2025
4ac66b4
propagate logging levels
blink1073 Jan 6, 2025
dc58b22
fix handling of var name
blink1073 Jan 6, 2025
641e57b
use branch
blink1073 Jan 7, 2025
4b54160
try with direct invocation
blink1073 Jan 7, 2025
69ab2a7
fix path on windows
blink1073 Jan 7, 2025
406faaf
more debug
blink1073 Jan 8, 2025
01d71b4
try windows again
blink1073 Jan 8, 2025
01dad7d
cleanup
blink1073 Jan 8, 2025
d7d28f2
more debug
blink1073 Jan 8, 2025
cfaf056
more debug
blink1073 Jan 8, 2025
10bec70
cleanup
blink1073 Jan 8, 2025
d6b23a8
Merge branch 'master' of github.com:mongodb-labs/drivers-evergreen-to…
blink1073 Jan 11, 2025
b328ae5
fix handling of macos x86
blink1073 Jan 11, 2025
1968e17
clean up output
blink1073 Jan 13, 2025
bbef5be
Merge branch 'master' of github.com:mongodb-labs/drivers-evergreen-to…
blink1073 Jan 14, 2025
088ef50
fix file copy handling
blink1073 Jan 14, 2025
5be28a2
fix for docker
blink1073 Jan 14, 2025
610bec5
fix azure oidc
blink1073 Jan 15, 2025
73186d4
Update .evergreen/orchestration/drivers_orchestration.py
blink1073 Jan 24, 2025
071d42e
Handle PermissionError
blink1073 Jan 24, 2025
b036dc7
Merge branch 'DRIVERS-3032-dec' of github.com:blink1073/drivers-everg…
blink1073 Jan 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cleanup
blink1073 committed Jan 8, 2025
commit 10bec70a4ec73e69cbf44013ce1c18b5d7cd43fc
43 changes: 15 additions & 28 deletions .evergreen/orchestration/drivers_orchestration.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""
Run mongo-orchestration.
Run mongo-orchestration and launch a deployment.

Use '--help' for more information.
"""
@@ -101,26 +101,21 @@ def get_options():

# Get the options, and then allow environment variable overrides.
opts = parser.parse_args()
LOGGER.info("opts were %s", opts)
LOGGER.info("keys: %s", list(os.environ))
LOGGER.info("auth: %s", os.environ.get("AUTH"))
for key in vars(opts).keys():
env_var = key.upper()
if env_var == "VERSION":
env_var = "MONGODB_VERSION"
if env_var in os.environ:
if key == "auth":
LOGGER.info("hi auth %s", os.environ.get("AUTH"))
opts.auth = os.environ.get("AUTH") == "auth"
elif key == "ssl":
LOGGER.info("hi ssl %s", os.environ.get("SSL"))
opts.ssl = os.environ.get("SSL") == "ssl"
elif isinstance(getattr(opts, key), bool):
if os.environ[env_var]:
setattr(opts, key, True)
else:
setattr(opts, key, os.environ[env_var])
LOGGER.info("opts are %s", opts)

if opts.mongo_orchestration_home is None:
opts.mongo_orchestration_home = DRIVERS_TOOLS / ".evergreen/orchestration"
if opts.mongodb_binaries is None:
@@ -202,7 +197,7 @@ def run(opts):
mongodl(shlex.split(args))
LOGGER.info(f"Downloading mongodb {version}... done.")

# Download legacy shell
# Download legacy shell.
if opts.install_legacy_shell:
args = f"{default_args} --version 5.0"
args += " --strip-path-components 2 --component shell"
@@ -213,14 +208,7 @@ def run(opts):
# Download crypt shared.
if not opts.skip_crypt_shared:
# Get the download URL for crypt_shared.
# The crypt_shared package is available on server 6.0 and newer.
# Try to download a version of crypt_shared matching the server version.
# If no matching version is available, try to download the latest Major release of crypt_shared.
if version in ["3.6", "4.0", "4.2", "4.4", "5.0"]:
crypt_shared_version = "latest"
else:
crypt_shared_version = version
args = f"{default_args} --version {crypt_shared_version}"
args = f"{default_args} --version {version}"
args += " --strip-path-components 1 --component crypt_shared"
LOGGER.info("Downloading crypt_shared...")
mongodl(shlex.split(args))
@@ -249,19 +237,18 @@ def run(opts):
# Handle orchestration file - explicit or implicit.
orchestration_file = opts.orchestration_file
if not orchestration_file:
prefix = "basic"
fname = "basic"
if opts.auth:
prefix = "auth"
fname = "auth"
if opts.ssl:
prefix += "-ssl"
fname += "-ssl"
if opts.load_balancer:
prefix += "-load-balancer"
fname += "-load-balancer"
elif opts.disable_test_commands:
prefix = "disableTestCommands"
fname = "disableTestCommands"
elif opts.storage_engine:
prefix = opts.storage_engine

orchestration_file = f"{prefix}.json"
fname = opts.storage_engine
orchestration_file = f"{fname}.json"

# Get the orchestration config data.
topology = opts.topology
@@ -358,14 +345,14 @@ def start(opts):
mo_config_str = mo_config.as_posix()
command = f"{sys.executable} -m mongo_orchestration.server"

# Handle windows-specific concerns.
# Handle Windows-specific concerns.
if os.name == "nt":
# Copy client certificates.
src = DRIVERS_TOOLS / ".evergreen/x509gen/client.pem"
dst = mo_home / "lib/client.pem"
shutil.copy2(src, dst)

# We need to use the executable, and add it to our path.
# We need to use the CLI executable, and add it to our path.
os.environ["PATH"] = (
f"{Path(sys.executable).parent}{os.pathsep}{os.environ['PATH']}"
)
@@ -381,7 +368,7 @@ def start(opts):
output_file = mo_home / "out.log"
server_file = mo_home / "server.log"

# NOTE: we need to use a separate file id and close it so Evergreen does not hang.
# NOTE: we need to use a separate file id for stdout and close it so Evergreen does not hang.
output_fid = output_file.open("w")
try:
subprocess.run(
@@ -416,7 +403,7 @@ def start(opts):

def stop():
LOGGER.info("Stopping mongo-orchestration...")
py_exe = sys.executable.replace(os.sep, "/")
py_exe = Path(sys.executable).as_posix()
args = f"{py_exe} -m mongo_orchestration.server stop"
proc = subprocess.run(
shlex.split(args), check=True, stderr=subprocess.STDOUT, stdout=subprocess.PIPE
2 changes: 1 addition & 1 deletion .evergreen/orchestration/pyproject.toml
Original file line number Diff line number Diff line change
@@ -17,7 +17,7 @@ classifiers = [
"Programming Language :: Python",
"Programming Language :: Python :: Implementation :: CPython",
]
dependencies = ["drivers-evergreen-tools @ {root:parent:uri}", "mongo-orchestration @ https://github.com/blink1073/mongo-orchestration/archive/test-DRIVERS-3032-dec.tar.gz"]
dependencies = ["drivers-evergreen-tools @ {root:parent:uri}", "mongo-orchestration @ https://github.com/mongodb/mongo-orchestration/archive/master.tar.gz"]

[project.scripts]
drivers-orchestration = "drivers_orchestration:main"