[jobs] fix possible managed job status issues #4637
+85
−25
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CANCELLING is no longer a terminal status - we shouldn't set end_at.
It's possible that the job status erroneously gets set to some other value (e.g. RUNNING) by another process after getting set to CANCELLING - we should avoid this case, but if we end up here, we should still transition to CANCELLED.
I saw a bug where a job that was CANCELLING got updated by RUNNING by a different process. Avoid state transitions like this that obviously violate the expected state machine.
|
/quicktest-core |
|
/smoke-test --managed-jobs |
|
/smoke-test --managed-jobs |
cg505
added a commit
to cg505/skypilot
that referenced
this pull request
Feb 4, 2025
Usually, a job should always transition to a terminal status before it is set to DONE. However, if a bug makes that fail for some reason (e.g. the issues fixed in skypilot-org#4637), we should still catch this state.
4 tasks
Is it possible to just avoid the job from being set to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Descriptions from the commits:
[jobs] don't set end_at for CANCELLING
CANCELLING is no longer a terminal status - we shouldn't set end_at.
[jobs] don't require job to be CANCELLING to become CANCELLED
It's possible that the job status erroneously gets set to some other value (e.g. RUNNING) by another process after getting set to CANCELLING - we should avoid this case, but if we end up here, we should still transition to CANCELLED.
[jobs] avoid status transitions that don't make sense
I saw a bug where a job that was CANCELLING got updated by RUNNING by a different process. Avoid state transitions like this that obviously violate the expected state machine.
Tested (run the relevant ones):
bash format.sh