Github Actions, Workload Identity Federation and Impersonation

[robrankin]

Github Actions Workflow using Workload Identity Federation and Impersonation is failing at the terraform apply step.

Not sure if this is a bug or if I've broken something, so posting here rather than bugs/issues.

I did find the following issue which suggest not using impersonation in a GH Actions Workflow, however there doesn't seem to be any other reference to this suggestion anywhere I can find, so unsure if that's the correct approach.

#1872 (comment)

Environment

Terraform v1.9.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v6.6.0
+ provider registry.terraform.io/hashicorp/google-beta v6.6.0
+ provider registry.terraform.io/hashicorp/local v2.5.2

To Reproduce
Attempting to setup CI/CD for the bootstrap stage. GH workflow being triggered by a push to main.

The terraform plan step works, terraform apply step however fails with an impersonation related error.

The WIF provider attribute mapping is setup with attribute.fast_sub = "repo:" + assertion.repository + ":ref:" + assertion.ref, see below.

The repository_owner is correct as compared to the OIDC token claims.

The connected service accounts appears correct far as I can tell, having this for the -1 and -1r SAs

123456-prod-bootstrap-1 -> attribute.fast_sub="repo:companyx/tf-gcp-00-bootstrap:ref:refs/heads/main"

123456-prod-bootstrap-1r -> attribute.repository="companyx/tf-gcp-00-bootstrap"

Based on the GCP log included below, it appears to me the -1 SA is attempting to impersonate the -0r SA, which of course will not have read/wrote access to the terraform state file, and therefore terraform fails.

I've checked the providers being used, and they are correctly specifying impersonation:

terraform {
  backend "gcs" {
    bucket                      = "123456-prod-iac-core-bootstrap-0"
    impersonate_service_account = "123456-prod-bootstrap-0@123456-prod-iac-core-0.iam.gserviceaccount.com"
  }
}
provider "google" {
  impersonate_service_account = "123456-prod-bootstrap-0@123456-prod-iac-core-0.iam.gserviceaccount.com"
}
provider "google-beta" {
  impersonate_service_account = "123456-prod-bootstrap-0@123456-prod-iac-core-0.iam.gserviceaccount.com"
}

Expected behavior
Terraform apply succeeds

Result

Terraform Apply Error

terraform apply -input=false -no-color tfplan
Error: error loading state: Failed to open state file at gs://123456-prod-iac-core-bootstrap-0/default.tfstate: Get "https://storage.googleapis.com/123456-prod-iac-core-bootstrap-0/default.tfstate": impersonate: status code 403: {
  "error": {
    "code": 403,
    "message": "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "IAM_PERMISSION_DENIED",
        "domain": "iam.googleapis.com",
        "metadata": {
          "permission": "iam.serviceAccounts.getAccessToken"
        }
      }
    ]
  }
}

GCP Logs error

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 7,
      "message": "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist)."
    },
    "authenticationInfo": {
      "principalEmail": "something-bootstrap-1@123456-prod-iac-core-0.iam.gserviceaccount.com",
      "serviceAccountDelegationInfo": [
        {
          "principalSubject": "principal://iam.googleapis.com/projects/1053903256637/locations/global/workloadIdentityPools/123456/subject/repo:companyx/tf-gcp-00-bootstrap:ref:refs/heads/main"
        }
      ],
      "principalSubject": "serviceAccount:123456-prod-bootstrap-1@123456-prod-iac-core-0.iam.gserviceaccount.com"
    },
    "requestMetadata": {
      "callerIp": "13.73.34.227",
      "callerSuppliedUserAgent": "Go-http-client/2.0,gzip(gfe)",
      "requestAttributes": {
        "time": "2024-10-12T09:11:13.932285720Z",
        "auth": {}
      },
      "destinationAttributes": {}
    },
    "serviceName": "iamcredentials.googleapis.com",
    "methodName": "GenerateAccessToken",
    "authorizationInfo": [
      {
        "permission": "iam.serviceAccounts.getAccessToken",
        "resourceAttributes": {},
        "permissionType": "ADMIN_READ"
      }
    ],
    "resourceName": "projects/-/serviceAccounts/204425173640036832231",
    "request": {
      "@type": "type.googleapis.com/google.iam.credentials.v1.GenerateAccessTokenRequest",
      "name": "projects/-/serviceAccounts/123456-prod-bootstrap-0r@123456-prod-iac-core-0.iam.gserviceaccount.com"
    },
    "metadata": {
      "identityDelegationChain": [
        "projects/-/serviceAccounts/123456-prod-bootstrap-0r@123456-prod-iac-core-0.iam.gserviceaccount.com"
      ]
    }
  },
  "insertId": "13g66gwdf5r6",
  "resource": {
    "type": "service_account",
    "labels": {
      "email_id": "123456-prod-bootstrap-0r@123456-prod-iac-core-0.iam.gserviceaccount.com",
      "project_id": "123456-prod-iac-core-0",
      "unique_id": "204425173640036832231"
    }
  },
  "timestamp": "2024-10-12T09:11:13.920147822Z",
  "severity": "ERROR",
  "logName": "projects/123456-prod-iac-core-0/logs/cloudaudit.googleapis.com%2Fdata_access",
  "operation": {
    "id": "8876006798909250090",
    "producer": "iamcredentials.googleapis.com",
    "first": true,
    "last": true
  },
  "receiveTimestamp": "2024-10-12T09:11:14.252236489Z"
}

Additional context
Add any other context about the problem here

[robrankin]

Appears TF is re-using the backend configuration, including impersonation, for the plan step in the apply step. Pretty sure its not a problem with FAST though, so closing this.