Enable budget alerts on azure clusters

[GeorgianaElena]

Fixes https://github.com/2i2c-org/meta/issues/1278

Important notes

1. dynamic budget
   I don't think there's a way to automatically adjust the amount on azure like it was done for GCP or AWS because the amount field must be a number and that's it. Based on my reading of the documentation there isn't a way to automatically adjust this number
2. recent hub
   Because the hub has only been deployed at the end of May, the budget is calculated based on the cost of one month-June, which is the only full month that the hub has been operating for

Todos before merging

• verify that an alert is triggered
• document this

[sgibson91]

2. Because the hub has only been deployed at the end of May, the budget is calculated based on the cost of one month-June, which is the only full month that the hub has been operating for

In another issue somewhere, and I remember adding it to docs (but Erik probably overwrote it when he made everything automatically adjustable), we agreed to use Kitware to set a threshold for new hubs.

ETA: https://github.com/2i2c-org/meta/issues/1258

[GeorgianaElena]

terraform plan output:

Terraform will perform the following actions:

  # azurerm_consumption_budget_resource_group.budget will be created
  + resource "azurerm_consumption_budget_resource_group" "budget" {
      + amount            = 600
      + etag              = (known after apply)
      + id                = (known after apply)
      + name              = "BudgetResourceGroup"
      + resource_group_id = "/subscriptions/4ca0b08a-26e1-482f-bca6-f4eb0926124a/resourceGroups/2i2c-pchub"
      + time_grain        = "Monthly"

      + notification {
          + contact_emails = [
              + "[email protected]",
            ]
          + contact_groups = []
          + contact_roles  = []
          + enabled        = true
          + operator       = "GreaterThanOrEqualTo"
          + threshold      = 120
          + threshold_type = "Forecasted"
        }

      + time_period {
          + end_date   = "2029-07-01T00:00:00Z"
          + start_date = "2024-07-01T00:00:00Z"
        }
    }

  # azurerm_kubernetes_cluster.jupyterhub will be updated in-place
  ~ resource "azurerm_kubernetes_cluster" "jupyterhub" {
        id                                  = "/subscriptions/4ca0b08a-26e1-482f-bca6-f4eb0926124a/resourceGroups/2i2c-pchub/providers/Microsoft.ContainerService/managedClusters/hub-cluster"
        name                                = "hub-cluster"
        tags                                = {}
        # (35 unchanged attributes hidden)

      ~ default_node_pool {
            name                          = "core"
            tags                          = {}
            # (33 unchanged attributes hidden)

          - upgrade_settings {
              - drain_timeout_in_minutes      = 0 -> null
              - max_surge                     = "10%" -> null
              - node_soak_duration_in_minutes = 0 -> null
            }
        }

        # (5 unchanged blocks hidden)
    }

Plan: 1 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

[GeorgianaElena]

In another issue somewhere, and I remember adding it to docs (but Erik probably overwrote it when he made everything automatically adjustable), we agreed to use Kitware to set a threshold for new hubs

Thank you @sgibson91! I will add back that piece of info to the docs. However, for this hub, as it's not a new one and has been operating for a full month, I think it's best to take its own cost rather than one from another hub. Does this sound ok to you?

[GeorgianaElena]

Update

I had to replace consumption_budget_resource_group mentioned in the issue for consumption_budget_subscription because there were more than one resource group and we only define one through terraform, the others get automatically created.

@sgibson91 has provided more context about this and possible solution in #4478. Thank you @sgibson91!

[GeorgianaElena]

I've set this to a lower budget limit to see if it triggers an email. The docs say it should take for an email to come through after the evaluation, but not sure how often the evaluation happens.

If this takes a lot of time, I will merge it and check again tomorow for further fixes if it hadn't triggered still.

[GeorgianaElena]

🎉 Yuhoo, it triggered