Skip to content
This repository was archived by the owner on Oct 4, 2023. It is now read-only.

Implement retry with exponential backoff on all gcloud service calls #369

Closed
nhammond opened this issue May 4, 2017 · 4 comments
Closed

Comments

@nhammond
Copy link
Collaborator

nhammond commented May 4, 2017

Google recommends doing this for all 5xx and 429 errors.
https://cloud.google.com/storage/docs/exponential-backoff

@nhammond
Copy link
Collaborator Author

nhammond commented May 4, 2017

Consider also handling "socket.error: [Errno 104] Connection reset by peer", as seen in #361

@nhammond
Copy link
Collaborator Author

nhammond commented May 4, 2017

Also this: ssl.SSLError: ('The read operation timed out',)

#364

@nhammond
Copy link
Collaborator Author

Retries added for everything in filemanager and task_monitor. There may be more work to do in the playbooks. 1a2ada9

@nhammond
Copy link
Collaborator Author

I searched playbooks and found retries on everything except Docker pull commands. Added retries to docker pull in 7dc5b79.

Retries in ansible playbooks are still fixed interval, since ansible doesn't yet enable exponential backoff. #388 is open for changing to exponential backoff when avaialble.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant