user log /callback log

[adrianbr]

We need user-readable logs

• what loaded (source) where (hostname/destination/project/dataset)
• ideally nr of
• schema changes

and callback parameters

• Pipeline/ task name

• env name - In airflow you have a base url that tells you which instance is failing.

• logs url

• Schema changes

• error message

• success message:

  • how many rows were loaded to what table
  • if runs metadata is logged persistently then give a summary if the vol of rows is normal (compare to last runs)

Partial success:

• unless intentionally configured, this is a non-atomic load and should be avoided.
• report failures - offer way to retry? offer way to

Retry:

• keep track of retries to be able to report them on final fail
• optionally report failures of retries. In some patterns you will want to turn off retry notification and only notify failure.

[adrianbr]

possibly offer a way for an automated "lateness" alert based on past runs time, for example alert anything that is >3 std from avg as a simple toggle

reason: sometimes the download might take much longer due to more data produced (could be 100x more in db migration/reconciliation cases), sometimes, the download might be stuck. In a different case (airflow separating scheduler from runner) the SLA check is done by the scheduler because the runner might be dead (zombie - dead but did not report failure, or running but disconnected from reporting success)

[adrianbr]

It would be nice to also collect any variables that were sent to the resource when data was requested, so we can report back which instance/request/date/segment of the task failed.