Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨Computational backend: passing labels and envs 🗃️ #4429

Merged

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Jun 27, 2023

What do these changes do?

  • sets user_id, study_id, uuid (node_id), product_name, simcore_user_agent on computational service containers (allows to filter puppeteer runs from real users for monitoring purpose @elisabettai , allows resource tracking)
  • virtually allow to pass any kind of run_metadata on the computational service containers
  • passes environment variables to computational service containers, they are now also contained in comp_tasks table
  • 🗃️ new metadata column on comp_runs table, necessary to pass container labels
  • substitutes osparc variables thanks to @pcrespov

@matusdrobuliak66 the limits will come in a different PR: #4453
I need to do some changes to the naming of these labels because _ are not valid docker label keys. I will have to change these.

Related issue/s

How to test

DevOps Checklist

@sanderegg sanderegg added a:dask-service Any of the dask services: dask-scheduler/sidecar or worker changelog:🎨enhancement labels Jun 27, 2023
@sanderegg sanderegg added this to the Watermelon milestone Jun 27, 2023
@sanderegg sanderegg self-assigned this Jun 27, 2023
@codecov
Copy link

codecov bot commented Jun 27, 2023

Codecov Report

Merging #4429 (47dd9dc) into master (1a71293) will increase coverage by 1.3%.
The diff coverage is 84.0%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #4429     +/-   ##
========================================
+ Coverage    84.6%   86.0%   +1.3%     
========================================
  Files        1010     899    -111     
  Lines       43115   39642   -3473     
  Branches     1016     587    -429     
========================================
- Hits        36493   34095   -2398     
+ Misses       6401    5409    -992     
+ Partials      221     138     -83     
Flag Coverage Δ
integrationtests 61.1% <83.8%> (-15.1%) ⬇️
unittests 84.4% <84.0%> (+0.5%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...sk_task_models_library/container_tasks/protocol.py 0.0% <0.0%> (ø)
...rary/src/models_library/service_settings_labels.py 97.5% <ø> (ø)
...decar/src/simcore_service_dask_sidecar/settings.py 100.0% <ø> (ø)
...ore_service_director_v2/api/routes/computations.py 93.1% <ø> (+22.4%) ⬆️
...ce_director_v2/modules/db/repositories/clusters.py 94.7% <ø> (ø)
...-sidecar/src/simcore_service_dask_sidecar/utils.py 59.6% <66.6%> (ø)
...rector_v2/modules/comp_scheduler/dask_scheduler.py 94.0% <66.6%> (ø)
...car/src/simcore_service_dask_sidecar/file_utils.py 96.3% <73.3%> (-1.9%) ⬇️
...simcore_service_director_v2/modules/dask_client.py 92.9% <83.3%> (+<0.1%) ⬆️
...r-v2/src/simcore_service_director_v2/utils/dask.py 89.7% <96.7%> (+0.8%) ⬆️
... and 15 more

... and 173 files with indirect coverage changes

@sanderegg sanderegg force-pushed the comp_backend/add-labels-env branch 2 times, most recently from fc00543 to dd65c83 Compare June 30, 2023 14:17
@sonarqubecloud
Copy link

Please retry analysis of this Pull-Request directly on SonarCloud.

@sanderegg sanderegg force-pushed the comp_backend/add-labels-env branch 3 times, most recently from 6d7184d to e12bb89 Compare July 3, 2023 17:16
@sanderegg sanderegg marked this pull request as ready for review July 3, 2023 17:24
@sanderegg sanderegg requested review from elisabettai and mguidon July 3, 2023 17:25
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 !

Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 nice

@sanderegg sanderegg force-pushed the comp_backend/add-labels-env branch from 9d46c5d to e436cdd Compare July 4, 2023 07:15
@sanderegg sanderegg changed the title ✨Computational backend: passing labels and envs ✨Computational backend: passing labels and envs 🗃️ Jul 4, 2023
@sanderegg sanderegg force-pushed the comp_backend/add-labels-env branch from 4843f6c to f9fdf13 Compare July 4, 2023 07:55
@mrnicegyu11
Copy link
Member

ack for the DB migration :)

Copy link
Contributor

@matusdrobuliak66 matusdrobuliak66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks 👍 I merged a DB migration few minutes ago, probably you need to change the DB migration id

@sanderegg sanderegg force-pushed the comp_backend/add-labels-env branch from 4e2dd5a to 47dd9dc Compare July 4, 2023 08:33
@codeclimate
Copy link

codeclimate bot commented Jul 4, 2023

Code Climate has analyzed commit 47dd9dc and detected 0 issues on this pull request.

View more on Code Climate.

@sonarqubecloud
Copy link

sonarqubecloud bot commented Jul 4, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@sanderegg sanderegg enabled auto-merge (squash) July 4, 2023 09:50
@sanderegg sanderegg merged commit 86567d7 into ITISFoundation:master Jul 4, 2023
@sanderegg sanderegg deleted the comp_backend/add-labels-env branch July 4, 2023 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:dask-service Any of the dask services: dask-scheduler/sidecar or worker
Projects
None yet
5 participants