Upgrade dependencies 2024-12-09 (#6750)

[achave11-ucsc]

Connected issue: #6750

Checklist

Author

• Target branch is develop
• Name of PR branch matches upgrades/yyyy-mm-dd
• On ZenHub, PR is connected to the upgrade issue it resolves
• PR title matches Upgrade dependencies yyyy-mm-dd
• PR title references the connected issue

Author (upgrading deployments)

• Ran make docker_images.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
• Added u tag to commit title _{or this PR does not require upgrading deployments}
• This PR is labeled upgrade _{or does not require upgrading deployments}
• This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
• This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
• This PR is labeled backup:gitlab
• This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

• Rebased PR branch on develop, squashed old fixups
• Ran make requirements_update _{or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}
• make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}

System administrator (after approval)

• Actually approved the PR
• Labeled connected issue as no demo
• A comment to this PR details the completed security design review
• PR title is appropriate as title of merge commit
• Moved connected issue to Approved column
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Squashed PR branch and rebased onto develop
• Sanity-checked history
• Pushed PR branch to GitHub
• Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select dev.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
• Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
• Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Checked the items in the next section _{or this PR is labeled deploy:gitlab}
• PR is assigned to only the system administrator _{or this PR is not labeled deploy:gitlab}

System administrator

• Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Ran _select dev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Ran _select anvildev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Added sandbox label
• Pushed PR branch to GitLab dev
• Pushed PR branch to GitLab anvildev
• Build passes in sandbox deployment
• Build passes in anvilbox deployment
• Reviewed build logs for anomalies in sandbox deployment
• Reviewed build logs for anomalies in anvilbox deployment
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but excluded any p tags}
• Moved connected issue to Merged lower column in ZenHub
• Moved blocked issues to Triage _{or no issues are blocked on the connected issue}
• Closed related Dependabot PRs with a comment referencing the corresponding commit in this PR _{or this PR does not include any such commits}
• Pushed merge commit to GitHub

Operator (after pushing the merge commit)

• Pushed merge commit to GitLab dev
• Pushed merge commit to GitLab anvildev
• Build passes on GitLab dev
• Reviewed build logs for anomalies on GitLab dev
• Build passes on GitLab anvildev
• Reviewed build logs for anomalies on GitLab anvildev
• Ran _select dev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Deleted PR branch from GitHub
• Deleted PR branch from GitLab dev
• Deleted PR branch from GitLab anvildev

Operator

• At least 24 hours have passed since anvildev.shared was last deployed
• Ran scripts/export_inspector_findings.py against anvildev, imported results to Google Sheet and posted screenshot of relevant¹ findings as a comment on the connected issue.
• Propagated the deploy:shared, deploy:gitlab, deploy:runner and backup:gitlab labels to the next promotion PRs _{or this PR carries none of these labels}
• Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner and backup:gitlab labels, from the description of this PR to that of the next promotion PRs _{or this PR carries none of these labels}
• PR is assigned to only the system administrator

¹A relevant finding is a high or critical vulnerability in an image
that is used within the security boundary. Images not used within the boundary
are tracked in azul.docker_images under a key starting with _.

System administrator

• No currently reported vulnerability requires immediate attention
• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[coveralls]

coverage: 85.543%. remained the same
when pulling e39203c on upgrades/2024-12-09
into db53009 on develop.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.52%. Comparing base (db53009) to head (e39203c).
Report is 11 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #6769   +/-   ##
========================================
  Coverage    85.52%   85.52%           
========================================
  Files          153      153           
  Lines        21190    21190           
========================================
  Hits         18123    18123           
  Misses        3067     3067           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hannes-ucsc]

Subject: [PATCH] Document need to redeploy Data Browser after index (#6657)
---
Index: requirements.txt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/requirements.txt b/requirements.txt
--- a/requirements.txt	(revision d0a624f69fae49baa871d7fc223b826a7626af7c)
+++ b/requirements.txt	(date 1734502901084)
@@ -1,24 +1,24 @@
 attrs==24.3.0
 aws-requests-auth==0.4.3
 bdbag==1.7.3
-boto3==1.35.82  # match this with the version of the `boto3-stubs` dev dependency
+boto3==1.35.82  # Match version of the `boto3-stubs` dev dependency
 botocore==1.35.82
 chevron==0.14.0
 deprecated==1.2.15
-elasticsearch==7.17.12
-elasticsearch-dsl==7.4.1
+elasticsearch==7.17.12  # 7.x to match server-side
+elasticsearch-dsl==7.4.1  # 7.x to match server-side
 fastavro==1.9.7
 furl==2.1.3
 google-api-core==2.24.0
 google-auth[pyopenssl]==2.37.0
-google-cloud-bigquery==3.13.0  # Can't go higher, due to https://github.com/DataBiosphere/azul/issues/6709
+google-cloud-bigquery==3.13.0  # < 3.14.0, see https://github.com/DataBiosphere/azul/issues/6709
 http-message-signatures==0.5.0
 jmespath==1.0.1
 more-itertools==10.5.0
 msgpack==1.1.0
 requests==2.32.3
-rsa==4.7.2  # resolve ambiguity with build-time dependency
-setuptools==70.3.0  # Keep consistent with requirements.pip.txt. Python 3.12 removed distutils, which we depended on transitively. Luckily, setuptools includes a vendored copy.
+rsa==4.7.2  # Resolve ambiguity with build-time dependency
+setuptools==70.3.0  # Match requirements.pip.txt. Python 3.12 removed distutils, which we depended on transitively. Luckily, setuptools includes a vendored copy.
 urllib3==1.26.20
 werkzeug==3.1.3
 -r requirements.trans.txt
Index: requirements.pip.txt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/requirements.pip.txt b/requirements.pip.txt
--- a/requirements.pip.txt	(revision d0a624f69fae49baa871d7fc223b826a7626af7c)
+++ b/requirements.pip.txt	(date 1734502901092)
@@ -1,3 +1,3 @@
 pip==24.0
-setuptools==70.3.0  # keep consistent with requirements.txt
+setuptools==70.3.0  # Match requirements.txt
 wheel==0.38.4

Did you consider upgrading the pip dependencies?

[hannes-ucsc]

Did you try updating this? What are the version constraints?

[achave11-ucsc]

Didn't 😅, but now did: #6774.

[achave11-ucsc]

Did you consider upgrading the pip dependencies?

Negative. Is fixed now.

[hannes-ucsc]

Security design review

• Security design review completed; this PR does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with Terra
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below