Improve worker load threshold detection #6010
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
perlpunk
reviewed
Oct 15, 2024
perlpunk
reviewed
Oct 15, 2024
okurz
force-pushed
the
feature/worker_load_limit
branch
from
7d9d9ea to
519058c
Compare
okurz
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6010 +/- ##
=======================================
Coverage 98.96% 98.96%
=======================================
Files 396 396
Lines 39364 39381 +17
=======================================
+ Hits 38958 38975 +17
Misses 406 406 ☔ View full report in Codecov by Sentry. |
Martchus
requested changes
Oct 17, 2024
okurz
force-pushed
the
feature/worker_load_limit
branch
from
519058c to
30896d5
Compare
done |
nicksinger
reviewed
Oct 21, 2024
With 03b301d we have a worker load limit which helps but there can still be cases like happened on a PowerNV machine with the load going way above the configured load limit. The reason was that when the worker was idle within a short time frame of roughly one minute multiple jobs were assigned to individual worker instances on the machine. As we only looked at load15 which was still low at that time all jobs were picked up by the machine leading to overload only about one minute later. Further reducing the load limit would not realistically prevent this situation but only delay until load15 decays sufficiently enough so that new jobs will be picked up again. Instead this commit changes the evaluation to look at all three system load values, load1, load5 and load15, but considering the load evolution over time to react quickly enough if the load rises but still accept a falling edge to allow to pick up jobs again when the load decays. Related progress issue: https://progress.opensuse.org/issues/168244
okurz
force-pushed
the
feature/worker_load_limit
branch
from
30896d5 to
f8b5424
Compare
nicksinger
requested changes
Oct 21, 2024
Martchus
approved these changes
Oct 21, 2024
asdil12
approved these changes
Oct 22, 2024
nicksinger
approved these changes
Oct 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With 03b301d we have a worker load limit which helps but there can still
be cases like happened on a PowerNV machine with the load going way above
the configured load limit. The reason was that when the worker was idle
within a short time frame of roughly one minute multiple jobs were
assigned to individual worker instances on the machine. As we only
looked at load15 which was still low at that time all jobs were picked
up by the machine leading to overload only about one minute later.
Further reducing the load limit would not realistically prevent this
situation but only delay until load15 decays sufficiently enough so that
new jobs will be picked up again. Instead this commit changes the
evaluation to look at all three system load values, load1, load5 and
load15, but considering the load evolution over time to react quickly
enough if the load rises but still accept a falling edge to allow to
pick up jobs again when the load decays.
Related progress issue: https://progress.opensuse.org/issues/168244