AWS cost attribution (2a/4): deploy to nasa-cryo

[consideRatio]

By following the docs from #4873, deploy the cost attribution system to the nasa-cryo cluster.

Definition of done

• https://grafana.cryointhecloud.2i2c.cloud/ has a dashboard available
• Any relevant documentation updates have been made, or an issue capturing that has been opened and refined to be scheduled as a followup

[sgibson91]

https://infrastructure.2i2c.org/howto/cost-attribution/

[sgibson91]

Should I be creating hub-specific node groups for nasa-cryo?

[sgibson91]

I ran into issues because the nasa-cryo account is a linked account to our AWS SSO sign-on, and cost allocation tags are only available at the payer account level.

Visit this URL to see: https://us-east-1.console.aws.amazon.com/billing/home?region=us-west-2#/tags

Terraform error:

│ Error: updating Cost Explorer Cost Allocation Tag (kubernetes.io/cluster/nasa-cryo): operation error Cost Explorer: UpdateCostAllocationTagsStatus, https response error StatusCode: 400, RequestID: 9e32aa56-2236-44a7-9fef-282910a934ea, api error AccessDeniedException: Failed to update Cost Allocation Tag: Linked account doesn't have access to cost allocation tags.
│ 
│   with aws_ce_cost_allocation_tag.cost_allocation_tags["kubernetes.io/cluster/nasa-cryo"],
│   on cost-allocation.tf line 2, in resource "aws_ce_cost_allocation_tag" "cost_allocation_tags":
│    2: resource "aws_ce_cost_allocation_tag" "cost_allocation_tags" {
│ 
╵
╷
│ Error: updating Cost Explorer Cost Allocation Tag (kubernetes.io/created-for/pvc/namespace): operation error Cost Explorer: UpdateCostAllocationTagsStatus, https response error StatusCode: 400, RequestID: 376d02cb-4606-4abe-9f22-ab55491d50b3, api error AccessDeniedException: Failed to update Cost Allocation Tag: Linked account doesn't have access to cost allocation tags.
│ 
│   with aws_ce_cost_allocation_tag.cost_allocation_tags["kubernetes.io/created-for/pvc/namespace"],
│   on cost-allocation.tf line 2, in resource "aws_ce_cost_allocation_tag" "cost_allocation_tags":
│    2: resource "aws_ce_cost_allocation_tag" "cost_allocation_tags" {
│ 
╵
╷
│ Error: updating Cost Explorer Cost Allocation Tag (2i2c.org/cluster-name): operation error Cost Explorer: UpdateCostAllocationTagsStatus, https response error StatusCode: 400, RequestID: ad097fbc-767e-4c9f-8f5f-a536082cbb97, api error AccessDeniedException: Failed to update Cost Allocation Tag: Linked account doesn't have access to cost allocation tags.
│ 
│   with aws_ce_cost_allocation_tag.cost_allocation_tags["2i2c.org/cluster-name"],
│   on cost-allocation.tf line 2, in resource "aws_ce_cost_allocation_tag" "cost_allocation_tags":
│    2: resource "aws_ce_cost_allocation_tag" "cost_allocation_tags" {
│ 
╵
╷
│ Error: updating Cost Explorer Cost Allocation Tag (alpha.eksctl.io/cluster-name): operation error Cost Explorer: UpdateCostAllocationTagsStatus, https response error StatusCode: 400, RequestID: b894e903-a5ea-48b1-98b5-7234f2c997e1, api error AccessDeniedException: Failed to update Cost Allocation Tag: Linked account doesn't have access to cost allocation tags.
│ 
│   with aws_ce_cost_allocation_tag.cost_allocation_tags["alpha.eksctl.io/cluster-name"],
│   on cost-allocation.tf line 2, in resource "aws_ce_cost_allocation_tag" "cost_allocation_tags":
│    2: resource "aws_ce_cost_allocation_tag" "cost_allocation_tags" {
│ 
╵
╷
│ Error: updating Cost Explorer Cost Allocation Tag (2i2c:hub-name): operation error Cost Explorer: UpdateCostAllocationTagsStatus, https response error StatusCode: 400, RequestID: 31e55d03-c7bf-4c86-a1ed-4b650ccb19c8, api error AccessDeniedException: Failed to update Cost Allocation Tag: Linked account doesn't have access to cost allocation tags.
│ 
│   with aws_ce_cost_allocation_tag.cost_allocation_tags["2i2c:hub-name"],
│   on cost-allocation.tf line 2, in resource "aws_ce_cost_allocation_tag" "cost_allocation_tags":
│    2: resource "aws_ce_cost_allocation_tag" "cost_allocation_tags" {

We should maybe pick a different cluster to enable this on first, one that isn't a linked account? While we figure out this bug.

[sgibson91]

This is now unblocked as we'll just deploy to another cluster, as decided on slack. Let's go for nasa-veda.

[sgibson91]

VEDA has the same issue. On Monday, I'll try with earthscope.

[sgibson91]

I'm getting a warning about a deprecated argument which I'll ignore for now in favour of just getting another deployment rolled out.

│ Warning: Argument is deprecated
│ 
│   with aws_iam_role.aws_ce_grafana_backend_iam_role,
│   on aws-ce-grafana-backend-iam.tf line 2, in resource "aws_iam_role" "aws_ce_grafana_backend_iam_role":
│    2: resource "aws_iam_role" "aws_ce_grafana_backend_iam_role" {
│ 
│ The inline_policy argument is deprecated. Use the aws_iam_role_policy resource instead. If Terraform should exclusively manage all inline policy associations (the current behavior of
│ this argument), use the aws_iam_role_policies_exclusive resource as well.

[sgibson91]

Nope, earthscope is linked too. I'm officially calling this blocked.

│ Error: updating Cost Explorer Cost Allocation Tag (alpha.eksctl.io/cluster-name): operation error Cost Explorer: UpdateCostAllocationTagsStatus, https response error StatusCode: 400, RequestID: bfd27b73-b16a-4040-9771-dec05e88b6d5, api error AccessDeniedException: Failed to update Cost Allocation Tag: Linked account doesn't have access to cost allocation tags.