Workspace initialization does not retry when DB fails

[dojeda]

Got this earlier today:

[aztec-quetzal-worker-f7f96dd5c-cxqfg] INFO quetzal.app.api.data.tasks 2019-10-24 08:01:07 Waiting for creation of workspace 96 ...
[aztec-quetzal-worker-f7f96dd5c-cxqfg] ERROR celery.app.trace 2019-10-24 08:01:08 Task quetzal.app.api.data.tasks.wait_for_workspace[ef234090-f9c4-49c9-8cf8-a6b5535050db] raised unexpected: OperationalError('(psycopg2.OperationalError) server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n',)
[aztec-quetzal-worker-f7f96dd5c-cxqfg] Traceback (most recent call last):
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     cursor, statement, parameters, context
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     cursor.execute(statement, parameters)
[aztec-quetzal-worker-f7f96dd5c-cxqfg] psycopg2.OperationalError: server closed the connection unexpectedly
[aztec-quetzal-worker-f7f96dd5c-cxqfg]  This probably means the server terminated abnormally
[aztec-quetzal-worker-f7f96dd5c-cxqfg]  before or while processing the request.
[aztec-quetzal-worker-f7f96dd5c-cxqfg]
[aztec-quetzal-worker-f7f96dd5c-cxqfg]
[aztec-quetzal-worker-f7f96dd5c-cxqfg] The above exception was the direct cause of the following exception:
[aztec-quetzal-worker-f7f96dd5c-cxqfg]
[aztec-quetzal-worker-f7f96dd5c-cxqfg] Traceback (most recent call last):
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     R = retval = fun(*args, **kwargs)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/code/quetzal/app/helpers/celery.py", line 52, in __call__
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     return task_base.__call__(self, *_args, **_kwargs)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 641, in __protected_call__
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     return self.run(*args, **kwargs)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/code/quetzal/app/api/data/tasks.py", line 40, in wait_for_workspace
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     workspace = Workspace.query.get(wid)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 959, in get
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     return self._get_impl(ident, loading.load_on_pk_identity)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 1069, in _get_impl
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     return db_load_fn(self, primary_key_identity)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 282, in load_on_pk_identity
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     return q.one()
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3275, in one
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     ret = self.one_or_none()
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3244, in one_or_none
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     ret = list(self)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     return self._execute_and_instances(context)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     result = conn.execute(querycontext.statement, self._params)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     return meth(self, multiparams, params)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     return connection._execute_clauseelement(self, multiparams, params)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     distilled_params,
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     e, statement, parameters, cursor, context
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     util.raise_from_cause(sqlalchemy_exception, exc_info)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     reraise(type(exception), exception, tb=exc_tb, cause=cause)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     raise value.with_traceback(tb)
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     cursor, statement, parameters, context
[aztec-quetzal-worker-f7f96dd5c-cxqfg]   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
[aztec-quetzal-worker-f7f96dd5c-cxqfg]     cursor.execute(statement, parameters)
[aztec-quetzal-worker-f7f96dd5c-cxqfg] sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
[aztec-quetzal-worker-f7f96dd5c-cxqfg]  This probably means the server terminated abnormally
[aztec-quetzal-worker-f7f96dd5c-cxqfg]  before or while processing the request.
[aztec-quetzal-worker-f7f96dd5c-cxqfg]
[aztec-quetzal-worker-f7f96dd5c-cxqfg] [SQL: SELECT workspace.id AS workspace_id, workspace.name AS workspace_name, workspace._state AS workspace__state, workspace.description AS workspace_description, workspace.creation_date AS workspace_creation_date, workspace.temporary AS workspace_temporary, workspace.data_url AS workspace_data_url, workspace.pg_schema_name AS workspace_pg_schema_name, workspace.fk_user_id AS workspace_fk_user_id, workspace.fk_last_metadata_id AS workspace_fk_last_metadata_id
[aztec-quetzal-worker-f7f96dd5c-cxqfg] FROM workspace
[aztec-quetzal-worker-f7f96dd5c-cxqfg] WHERE workspace.id = %(param_1)s]
[aztec-quetzal-worker-f7f96dd5c-cxqfg] [parameters: {'param_1': 96}]
[aztec-quetzal-worker-f7f96dd5c-cxqfg] (Background on this error at: http://sqlalche.me/e/e3q8)

So, in other words, retrieving the workspace object failed due to a communication hiccup with the database. Should all these operations be protected for this case?