add new concurrency config to consolidate concurrency settings #26931
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
prha
force-pushed
the
prha/concurrency_config
branch
from
12b7103 to
f71c7ea
Compare
This was referenced Jan 8, 2025
prha
changed the title
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
13f2c3c to
4710c64
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
f71c7ea to
f572156
Compare
This was referenced Jan 9, 2025
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
4710c64 to
2ce911a
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
f572156 to
8cfee9c
Compare
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
2ce911a to
e5e8022
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
8cfee9c to
152beca
Compare
This was referenced Jan 9, 2025
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
e5e8022 to
32eec84
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
152beca to
068a084
Compare
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
32eec84 to
3ba911e
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
068a084 to
6927f22
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
bf19552 to
17ed4d9
Compare
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
253141f to
24f23d7
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
17ed4d9 to
6a116d5
Compare
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
24f23d7 to
16acc11
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
6a116d5 to
7309b15
Compare
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
16acc11 to
7dae80e
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
7309b15 to
256d726
Compare
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
7dae80e to
8649c64
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
256d726 to
3efd17a
Compare
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
8649c64 to
f1b7934
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
3efd17a to
3d2b288
Compare
prha
changed the base branch from
prha/hoist_run_queue_config
to
prha/run_pools_graphql
|
Deploy preview for dagit-core-storybook ready! ✅ Preview Built with commit 3d2b288. |
OwenKephart
approved these changes
Jan 17, 2025
Comment on lines
236
to
249
| def verify_config_match(config: Mapping[str, Any], path_a: Sequence[str], path_b: Sequence[str]): | ||
| value_a = pluck_config_value(config, path_a) | ||
| value_b = pluck_config_value(config, path_b) | ||
| if value_a is None or value_b is None: | ||
| return | ||
|
|
||
| if value_a != value_b: | ||
| path_a_str = " > ".join(path_a) | ||
| path_b_str = " > ".join(path_b) | ||
| raise DagsterInvalidConfigError( | ||
| f"Found `{value_a}` for `{path_a_str}` that conflicts with `{value_b}` for `{path_b_str}`.", | ||
| [], | ||
| None, | ||
| ) |
There was a problem hiding this comment.
this can be deleted now
prha
force-pushed
the
prha/concurrency_config
branch
from
3d2b288 to
e63a776
Compare
prha
changed the base branch from
prha/run_pools_graphql
to
prha/hoist_run_queue_config
prha
force-pushed
the
prha/hoist_run_queue_config
branch
from
f1b7934 to
be7ec05
Compare
prha
force-pushed
the
prha/concurrency_config
branch
from
e63a776 to
6dd295b
Compare
prha
added a commit
that referenced
this pull request
Jan 21, 2025
…ce (#26932) ## Summary & Motivation For ease of use, #26931 consolidates concurrency settings (including queued run coordinator settings) into a single deployment settings block. This PR makes it so that run queue settings are now accessed off of the instance, instead of off of the run coordinator. The run coordinator is a standalone-configured module. This makes it difficult to incorporate instance settings from other places. ## How I Tested These Changes BK
prha
force-pushed
the
prha/concurrency_config
branch
from
6dd295b to
9f65155
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR enables setting run concurrency settings from the run_queue configuration block into the
concurrencydeployment settings block.This is to simplify concepts from a configuration standpoint.
The hypothesis is that most people will not need to even know about the run coordinator, and instead will only interact with this concurrency section.
How I Tested These Changes
BK
Changelog
Moves run queue configuration from its standalone deployment setting into the
concurrencydeployment setting, along with new settings for pools.