Protect with retry logic when updating acl

[dojeda]

This happened today:

Traceback (most recent call last):
  File "/code/quetzal/app/api/data/file.py", line 137, in create
    storage.set_permissions(obj, workspace.owner)
  File "/code/quetzal/app/api/data/storage/__init__.py", line 73, in set_permissions
    return gcp.set_permissions(file_obj, owner)
  File "/code/quetzal/app/api/data/storage/gcp.py", line 60, in set_permissions
    acl.save()
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/acl.py", line 504, in save
    self._save(acl, None, client)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/acl.py", line 476, in _save
    query_params=query_params,
  File "/usr/local/lib/python3.6/site-packages/google/cloud/_http.py", line 319, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.ServiceUnavailable: 503 PATCH https://www.googleapis.com/storage/v1/b/some-bucket/o/some_file.bin?projection=full: Service Unavailable

There was a short hiccup on the google api service, which we could catch and retry a couple of times before failing.

[dojeda]

Not only on acl operations, today I had a problem at file download:

google.api_core.exceptions.ServiceUnavailable: 503 GET https://www.googleapis.com/storage/v1/b/somebucket?projection=noAcl: Backend Error
at api_request (/usr/local/lib/python3.6/site-packages/google/cloud/_http.py:319)
at reload (/usr/local/lib/python3.6/site-packages/google/cloud/storage/_helpers.py:106)
at get_bucket (/usr/local/lib/python3.6/site-packages/google/cloud/storage/client.py:227)
at get_bucket (/code/quetzal/app/helpers/google_api.py:44)
at get_object (/code/quetzal/app/helpers/google_api.py:51)
at _download_file_gcp (/code/quetzal/app/api/data/file.py:602)
at _download_file (/code/quetzal/app/api/data/file.py:592)
at details_w (/code/quetzal/app/api/data/file.py:440)
at wrapper (/usr/local/lib/python3.6/site-packages/connexion/decorators/parameter.py:126)
at wrapper (/code/quetzal/app/hacks.py:56)
at wrapper (/usr/local/lib/python3.6/site-packages/connexion/decorators/validation.py:347)
at wrapper (/usr/local/lib/python3.6/site-packages/connexion/decorators/uri_parsing.py:143)
at wrapper (/usr/local/lib/python3.6/site-packages/connexion/decorators/security.py:300)
at wrapper (/usr/local/lib/python3.6/site-packages/connexion/decorators/decorator.py:48)
at dispatch_request (/usr/local/lib/python3.6/site-packages/flask/app.py:1799)
at full_dispatch_request (/usr/local/lib/python3.6/site-packages/flask/app.py:1813)
at reraise (/usr/local/lib/python3.6/site-packages/flask/_compat.py:35)
at handle_user_exception (/usr/local/lib/python3.6/site-packages/flask/app.py:1718)
at full_dispatch_request (/usr/local/lib/python3.6/site-packages/flask/app.py:1815)
at wsgi_app (/usr/local/lib/python3.6/site-packages/flask/app.py:2292)
at reraise (/usr/local/lib/python3.6/site-packages/flask/_compat.py:35)
at handle_exception (/usr/local/lib/python3.6/site-packages/flask/app.py:1741)
at wsgi_app (/usr/local/lib/python3.6/site-packages/flask/app.py:2295)
at __call__ (/code/quetzal/app/middleware/headers.py:18)
at __call__ (/usr/local/lib/python3.6/site-packages/flask/app.py:2309)
at handle_request (/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py:176)
at handle (/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py:135)