Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core key vault firewall should not be set to "Allow public access from all networks" #4260

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open
1 change: 1 addition & 0 deletions CHANGELOG.md
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have time to go over this the next few days and guess @marrobi is the same. Just wanted to point out we now have 2 vaults being used from the deployer point of view.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When CMK is enabled another vault is created in the mgmt resource group

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a pending ask to enable the deployer to access resources such as these keyvaults over private network only.

  1. Will this make this approach obsolete?
  2. If not, this means we will need to do all of this in a conditional way. Right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. That would mean all TRE deployers would need to switch to private self hosted runners right? If so, yes this would be obsolete.
  2. If we want to support both deployment patterns - deployment from GitHub hosted runners + deployment from private self hosted runners with KV set to private networking - then yes we'd need to do it conditionally (in order to prevent the KV from being fully public).

I guess it depends on whether implementing keyvaults private networking only means switching off the ability to use Github hosted runners - is that the plan?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal I'm referring to that resources will be accessed via private network only which means private runners.

The question for you, is weather this PR comes from a similar place but doesn't go as far yet and just limits which public IPs can access. Or, in a situation where private agents / network is done, will you still need this method of limiting public IPs

Copy link
Collaborator Author

@jonnyry jonnyry Jan 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep when the codebase switches to deployment from private runners ONLY, then this change won't be needed anymore.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, in your deployment/usecase you would also like to use private runners?

Copy link
Collaborator

@tamirkamara tamirkamara Jan 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just considering most orgs that are concerned with this might want to go all the way. This solution might not be enough in those cases.
Also considering the number of changes you had to do just for this resource... next up will be the ACR, storage account and yet another keyvault in the mgmt resource group. All in all it might not be worth the investment and complexities if the end goal is anyway to go private agents...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree private runners should be the objective, but when are they scheduled to be implemented? Is this a reasonable stop gap until then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to ensure the solution can still be deployed without private runners. Most orgs start out testing out the solution etc, and don't have the infrastructure for private runners.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point is that we might not want to offer this middle way of opening the deployer IP in all currently defined public resources due to the complexities of supporting all the resources I mentioned above.
It might be we offer public like today and secured via private runners. Testing out could be done via the current way and if you want to be secure then it would mean you need private agents.

Copy link
Collaborator Author

@jonnyry jonnyry Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Supporting all resources might be too much (e.g. the ACR) but the KV is one that stands out as worthy of tightening network access on; though I understand that private runners will supersede this once implemented.

Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ ENHANCEMENTS:
* Upgrade Python version from 3.8 to 3.12 ([#3949](https://github.com/microsoft/AzureTRE/issues/3949))Upgrade Python version from 3.8 to 3.12 (#3949)
* Disable storage account key usage ([[#4227](https://github.com/microsoft/AzureTRE/issues/4227)])
* Update Guacamole dependencies ([[#4232](https://github.com/microsoft/AzureTRE/issues/4232)])
* Core key vault firewall should not be set to "Allow public access from all networks" ([#4250](https://github.com/microsoft/AzureTRE/issues/4250))
* Add option to force tunnel TRE's Firewall ([#4237](https://github.com/microsoft/AzureTRE/issues/4237))
* Add EventGrid diagnostics to identify airlock issues ([#4258](https://github.com/microsoft/AzureTRE/issues/4258))

Expand Down
19 changes: 19 additions & 0 deletions core/terraform/.terraform.lock.hcl

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions core/terraform/deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ set -o pipefail
set -o nounset
# set -o xtrace

# add trap to remove kv network exception
# shellcheck disable=SC1091
trap 'source "../../devops/scripts/kv_remove_network_exception.sh"' EXIT

# now add kv network exception
# shellcheck disable=SC1091
source "../../devops/scripts/kv_add_network_exception.sh"

# This is where we can migrate any Terraform before we plan and apply
# For instance deprecated Terraform resources
# shellcheck disable=SC1091
Expand Down
8 changes: 8 additions & 0 deletions core/terraform/destroy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ set -o pipefail
set -o nounset
# set -o xtrace

# add trap to remove kv network exception
# shellcheck disable=SC1091
trap 'source "../../devops/scripts/kv_remove_network_exception.sh"' EXIT

# now add kv network exception
# shellcheck disable=SC1091
source "../../devops/scripts/kv_add_network_exception.sh"

# These variables are loaded in for us
# shellcheck disable=SC2154
../../devops/scripts/terraform_wrapper.sh -g "${TF_VAR_mgmt_resource_group_name}" \
Expand Down
24 changes: 22 additions & 2 deletions core/terraform/keyvault.tf
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
resource "azurerm_key_vault" "kv" {
name = "kv-${var.tre_id}"
name = local.kv_name
tenant_id = data.azurerm_client_config.current.tenant_id
location = azurerm_resource_group.core.location
resource_group_name = azurerm_resource_group.core.name
Expand All @@ -8,7 +8,27 @@ resource "azurerm_key_vault" "kv" {
purge_protection_enabled = var.kv_purge_protection_enabled
tags = local.tre_core_tags

lifecycle { ignore_changes = [access_policy, tags] }
public_network_access_enabled = local.kv_public_network_access_enabled

network_acls {
default_action = local.kv_network_default_action
bypass = local.kv_network_bypass
ip_rules = [local.myip] # exception for deployment IP, this is removed in kv_remove_network_exception.sh
}

lifecycle {
ignore_changes = [access_policy, tags]
}

# create provisioner required due to https://github.com/hashicorp/terraform-provider-azurerm/issues/18970
#
provisioner "local-exec" {
when = create
command = <<EOT
az keyvault update --name ${local.kv_name} --public-network-access ${local.kv_public_network_access_enabled ? "Enabled" : "Disabled"} --default-action ${local.kv_network_default_action} --bypass "${local.kv_network_bypass}" --output none
az keyvault network-rule add --name ${local.kv_name} --ip-address ${local.myip} --output none
EOT
}
}

resource "azurerm_role_assignment" "keyvault_deployer_role" {
Expand Down
6 changes: 6 additions & 0 deletions core/terraform/locals.tf
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,10 @@ locals {

cmk_name = "tre-encryption-${var.tre_id}"
encryption_identity_name = "id-encryption-${var.tre_id}"

# key vault variables
kv_name = "kv-${var.tre_id}"
kv_public_network_access_enabled = true
kv_network_default_action = var.enable_local_debugging ? "Allow" : "Deny"
kv_network_bypass = "AzureServices"
}
9 changes: 9 additions & 0 deletions core/terraform/scripts/letsencrypt.sh
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the Storage Account rules in this script be handles the same way?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes certainly - planning to have a look at storage accounts after this.

Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@ set -e

script_dir=$(realpath "$(dirname "${BASH_SOURCE[0]}")")

# add trap to remove kv network exception
# shellcheck disable=SC1091
trap 'source "$script_dir/../../../devops/scripts/kv_remove_network_exception.sh"' EXIT

# now add kv network exception
# shellcheck disable=SC1091
source "$script_dir/../../../devops/scripts/kv_add_network_exception.sh"


if [[ -z ${STORAGE_ACCOUNT} ]]; then
echo "STORAGE_ACCOUNT not set"
exit 1
Expand Down
2 changes: 1 addition & 1 deletion core/version.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.11.17"
__version__ = "0.11.18"
10 changes: 10 additions & 0 deletions devops/scripts/destroy_env_no_terraform.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,16 @@ then
no_wait_option="--no-wait"
fi

script_dir=$(realpath "$(dirname "${BASH_SOURCE[0]}")")

# add trap to remove kv network exception
# shellcheck disable=SC1091
trap 'source "$script_dir/kv_remove_network_exception.sh"' EXIT

# now add kv network exception
# shellcheck disable=SC1091
source "$script_dir/kv_add_network_exception.sh"

group_show_result=$(az group show --name "${core_tre_rg}" > /dev/null 2>&1; echo $?)
if [[ "$group_show_result" != "0" ]]; then
echo "Resource group ${core_tre_rg} not found - skipping destroy"
Expand Down
16 changes: 13 additions & 3 deletions devops/scripts/key_vault_list.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,21 @@ fi

echo "DEBUG: Check keyvault and secrets exist"

script_dir=$(realpath "$(dirname "${BASH_SOURCE[0]}")")

# add trap to remove kv network exception
# shellcheck disable=SC1091
trap 'source "$script_dir/kv_remove_network_exception.sh"' EXIT

# now add kv network exception
# shellcheck disable=SC1091
source "$script_dir/kv_add_network_exception.sh"

echo "az keyvault show"
az keyvault show --name kv-${TRE_ID}
az keyvault show --name "kv-${TRE_ID}"

echo "az keyvault secret list"
az keyvault secret list --vault-name kv-${TRE_ID}
az keyvault secret list --vault-name "kv-${TRE_ID}"

echo "az keyvault secret list-deleted"
az keyvault secret list-deleted --vault-name kv-${TRE_ID}
az keyvault secret list-deleted --vault-name "kv-${TRE_ID}"
66 changes: 66 additions & 0 deletions devops/scripts/kv_add_network_exception.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
#!/bin/bash

function main() {

set -o errexit
set -o pipefail

# parse params/set up inputs
#
if [[ -z "$TRE_ID" ]]; then
echo -e "Could not add keyvault deployment network exception: TRE_ID is not set\nExiting...\n"
exit 1
fi

local RG_NAME="rg-${TRE_ID}"
local KV_NAME="kv-${TRE_ID}"
local MY_IP="${PUBLIC_DEPLOYMENT_IP_ADDRESS:-}"

if [[ -z "$MY_IP" ]]; then
MY_IP=$(curl -s "ipecho.net/plain"; echo)
fi


# add keyvault network exception
#
echo -e "\nAdding deployment network exception to key vault $KV_NAME..."

if [[ -z "$(az group list --query "[?name=='$RG_NAME']" --output tsv)" ]]; then
echo -e " Core resource group $RG_NAME not found\n"
return 0
fi

if [[ -z "$(az keyvault list --resource-group "$RG_NAME" --query "[?name=='$KV_NAME'].id" --output tsv)" ]]; then
echo -e " Core key vault $KV_NAME not found\n"
return 0
fi

az keyvault network-rule add --resource-group "$RG_NAME" --name "$KV_NAME" --ip-address "$MY_IP" --output none

local ATTEMPT=1
local MAX_ATTEMPTS=10

while true; do

if KV_OUTPUT=$(az keyvault secret list --vault-name "$KV_NAME" --query '[].name' --output tsv 2>&1); then
echo -e " Keyvault $KV_NAME is now accessible\n"
break
fi

if [ $ATTEMPT -eq $MAX_ATTEMPTS ]; then
echo -e "Could not add deployment network exception for $KV_NAME"
echo -e "Unable to access keyvault $KV_NAME after $ATTEMPT/$MAX_ATTEMPTS.\n"
echo -e "$KV_OUTPUT\n"

exit 1
fi

echo " Unable to access keyvault $KV_NAME after $ATTEMPT/$MAX_ATTEMPTS. Waiting for network rules to take effect."
sleep 5
((ATTEMPT++))

done

}

main "$@"
43 changes: 43 additions & 0 deletions devops/scripts/kv_remove_network_exception.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
#!/bin/bash

function main() {

set -o errexit
set -o pipefail

# parse params/set up inputs
#
if [[ -z "$TRE_ID" ]]; then
echo -e "Could not remove keyvault deployment network exception: TRE_ID is not set\nExiting...\n"
exit 1
fi

local RG_NAME="rg-${TRE_ID}"
local KV_NAME="kv-${TRE_ID}"
local MY_IP="${PUBLIC_DEPLOYMENT_IP_ADDRESS:-}"

if [[ -z "$MY_IP" ]]; then
MY_IP=$(curl -s "ipecho.net/plain"; echo)
fi


# remove keyvault network exception
#
echo -e "\nRemoving deployment network exception to key vault $KV_NAME..."

if [[ -z "$(az group list --query "[?name=='$RG_NAME']" --output tsv)" ]]; then
echo -e " Core resource group $RG_NAME not found\n"
return 0
fi

if [[ -z "$(az keyvault list --resource-group "$RG_NAME" --query "[?name=='$KV_NAME'].id" --output tsv)" ]]; then
echo -e " Core key vault $KV_NAME not found\n"
return 0
fi

az keyvault network-rule remove --name "$KV_NAME" --ip-address "$MY_IP" --output none
echo -e " Deployment network exception removed\n"

}

main "$@"
18 changes: 15 additions & 3 deletions devops/scripts/set_contributor_sp_secrets.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,19 @@ set -e
#

echo -e "\n\e[34m»»» 🤖 \e[96mCreating (or updating) service principal ID and secret to Key Vault\e[0m..."

script_dir=$(realpath "$(dirname "${BASH_SOURCE[0]}")")

# add trap to remove kv network exception
# shellcheck disable=SC1091
trap 'source "$script_dir/kv_remove_network_exception.sh"' EXIT

# now add kv network exception
# shellcheck disable=SC1091
source "$script_dir/kv_add_network_exception.sh"


key_vault_name="kv-$TRE_ID"
az account set --subscription $ARM_SUBSCRIPTION_ID
az keyvault secret set --name deployment-processor-azure-client-id --vault-name $key_vault_name --value $RESOURCE_PROCESSOR_CLIENT_ID
az keyvault secret set --name deployment-processor-azure-client-secret --vault-name $key_vault_name --value $RESOURCE_PROCESSOR_CLIENT_SECRET > /dev/null
az account set --subscription "$ARM_SUBSCRIPTION_ID"
az keyvault secret set --name deployment-processor-azure-client-id --vault-name "$key_vault_name" --value "$RESOURCE_PROCESSOR_CLIENT_ID"
az keyvault secret set --name deployment-processor-azure-client-secret --vault-name "$key_vault_name" --value "$RESOURCE_PROCESSOR_CLIENT_SECRET" > /dev/null
Loading