fix: improve remote runtime reliability on large-scale evaluation #4869
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tofarr
approved these changes
Nov 9, 2024
rbren
reviewed
Nov 9, 2024
Comment on lines
290
to
292
| # Wait for pending status | ||
| while pod_status in ('Pending', 'Running'): | ||
| time.sleep(2) |
There was a problem hiding this comment.
while X: sleep() is definitely not a good practice 😬 we need to bail out eventually
rbren
requested changes
Nov 9, 2024
|
@rbren maybe smth like this? |
rbren
reviewed
Nov 9, 2024
| @@ -89,6 +89,7 @@ def __init__( | |||
| ) | |||
| self.runtime_id: str | None = None | |||
| self.runtime_url: str | None = None | |||
| self.runtime_init_timeout = self.config.sandbox.remote_runtime_init_timeout | |||
There was a problem hiding this comment.
probably don't need to set this as a member var--can just use self.config.sandbox.remote_runtime_init_timeout
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
End-user friendly description of the problem this fixes or functionality that this introduces
Give a summary of what the PR does, explaining any non-trivial design decisions
When running large-scale evaluation, it could take more than 180 seconds for a pod to switch from Pending to Running - and the ceiling of the time taken is actually unknown (e.g., some images are very large that take a while to pull, maybe the cluster is full and we need to create new node to put the pod, etc).
This PR try to add back the while logic for the "Pending" and "Running" states - we no longer throw
RuntimeNotReadyErrorfor these two states and instead wait patiently for it. Without this, evaluation break pretty easily (eval 300 instances need 5-8 restarts which is a little bit unacceptable).Link of any specific issues this addresses
To run this PR locally, use the following command: