TFX on Cloud AI Platform Pipelines: TFX pipeline run cannot write into defined bucket (403 Insufficient Permission)

[jpatokal]

In Lab 02, when the TFX Tuner SA is configured as documented:

CUSTOM_SERVICE_ACCOUNT = 'tfx-tuner-caip-service-account@qwiklabs-gcp-01-1057c4de4b13.iam.gserviceaccount.com'

Runs of the TFX pipeline fail because Pipelines can't write into the bucket (which is missing by default, see issue #124).

tensorflow.python.framework.errors_impl.PermissionDeniedError: Error executing an HTTP request: HTTP response code 403 with body '{
  "error": {
    "code": 403,
    "message": "Insufficient Permission",
    "errors": [
      {
        "message": "Insufficient Permission",
        "domain": "global",
        "reason": "insufficientPermissions"
      }
    ]
  }
}
'
	 when initiating an upload to gs://my-missing-bucket/tfx_covertype_continuous_training/

I'm somewhat baffled as to why, since the Tuner SA and a few more all have Object Storage Admin privs on the bucket:

qwiklabs-gcp-01-1057c4de4b13@qwiklabs-gcp-01-1057c4de4b13.iam.gserviceaccount.com | Qwiklabs User Service Account | Storage Admin | qwiklabs-gcp-01-1057c4de4b13 |   |  

[email protected] | Google Cloud ML Engine Service Agent | AI Platform Service AgentStorage Object Admin | qwiklabs-gcp-01-1057c4de4b13 qwiklabs-gcp-01-1057c4de4b13

tfx-tuner-caip-service-account@qwiklabs-gcp-01-1057c4de4b13.iam.gserviceaccount.com | TFX Tuner CAIP Vizier | Storage Object Admin

Unfortunately you can't really tell from the logs which SA it's using.