Skip to content

Commit

Permalink
Clarify Azure storage permissions (#763)
Browse files Browse the repository at this point in the history
Co-authored-by: Joyce Fee <[email protected]>
  • Loading branch information
kbatuigas and Feediver1 authored Sep 18, 2024
1 parent 2ace6a5 commit c943f90
Show file tree
Hide file tree
Showing 3 changed files with 49 additions and 15 deletions.
23 changes: 22 additions & 1 deletion modules/manage/pages/security/iam-roles.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,28 @@ If you are using Microsoft Azure as your cloud provider, you must satisfy the fo
. https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp#create-a-user-assigned-managed-identity[Create a user-assigned managed identity]^.
. https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal#create-a-storage-account-1[Create an Azure storage account]^.
. https://learn.microsoft.com/en-us/azure/storage/blobs/blob-containers-portal#create-a-container[Create a container]^ in the storage account.
. Assign the identity, with the https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/storage#storage-blob-data-contributor[`Storage Blob Data Contributor`]^ role, either during the creation of the storage account, or for an existing storage account. See the official https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/qs-configure-portal-windows-vm#user-assigned-managed-identity[Azure Managed Identities] documentation for more guidance.
. Create a https://learn.microsoft.com/en-us/azure/role-based-access-control/custom-roles[custom role^] that only has the minimum permissions required for Tiered Storage, and assign the role to the identity. This helps prevent unauthorized actions on your data and minimize security risks. The custom role should have the following set of permissions:
+
```
"permissions": [
{
"actions": [
],
"notActions": [],
"dataActions": [
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action",
"Microsoft.Storage/storageAccounts/fileServices/fileShares/files/write",
"Microsoft.Storage/storageAccounts/fileServices/writeFileBackupSemantics/action"
],
"notDataActions": []
}
]
```
+
Assign the identity either during the creation of the storage account, or for an existing storage account. See the https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/qs-configure-portal-windows-vm#user-assigned-managed-identity[Azure Managed Identities] documentation for more guidance.

== Configuring IAM roles

Expand Down
24 changes: 17 additions & 7 deletions modules/manage/partials/tiered-storage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,7 @@ CAUTION: Do not set an object storage property to an empty string `""` or to `nu

==== Microsoft ABS/ADLS

You can configure access to Azure Blob Storage with either shared keys or Azure's managed identities system to securely interact with Azure Blob Storage. Shared keys, as static credentials, are simple to use but require manual management and vigilant security practices to prevent breaches due to their unchanging nature. In contrast, managed identities provide a more secure and maintenance-free solution by automating credential management and rotation, though they are exclusive to the Azure ecosystem.
You can configure access to Azure Blob Storage with either account access keys or Azure's managed identities system to securely interact with Azure Blob Storage. Account access keys, as static credentials, require manual management and vigilant security practices to prevent breaches due to their unchanging nature. In contrast, managed identities provide a more secure and maintenance-free solution by automating credential management and rotation, though they are exclusive to the Azure ecosystem.

include::manage:partial$azure-blob-limitations.adoc[]

Expand Down Expand Up @@ -582,7 +582,7 @@ NOTE: The `serviceAccount` annotations and the `statefulset` Pod labels are esse

CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.

To configure access to ABS/ADLS with shared keys:
To configure access to ABS/ADLS with account access keys:

. Get an account access key for the Azure container that Redpanda will run on. For information on how to view your account access keys, see the https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#view-account-access-keys[Azure documentation^].
. Override the following required cluster properties in the Helm chart:
Expand Down Expand Up @@ -695,7 +695,7 @@ CAUTION: Do not set an object storage property to an empty string `""` or to `nu

- For information about how to grant access from an internet IP range (if you need to open additional routes/ports between your broker nodes and Azure Blob Storage; for example, in a hybrid cloud deployment), see the https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#grant-access-from-an-internet-ip-range[Microsoft documentation^].

- For more information about shared key authentication, see the https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key[Microsoft documentation^].
- For more information about Shared Key authentication, see the https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key[Microsoft documentation^].

endif::[]

Expand Down Expand Up @@ -801,7 +801,17 @@ include::manage:partial$azure-blob-limitations.adoc[]
To configure access to an Azure container with a managed identity:
. Configure an xref:manage:security/iam-roles.adoc#configuring-iam-roles[Azure managed identity].
. Configure an xref:manage:security/iam-roles.adoc#azure-prerequisites[Azure managed identity].
+
Note the minimum set of permissions required for Tiered Storage:
+
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action
- Microsoft.Storage/storageAccounts/fileServices/fileShares/files/write
- Microsoft.Storage/storageAccounts/fileServices/writeFileBackupSemantics/action
. Run the `rpk cluster config edit` command, then edit the following required properties:
+
[,properties]
Expand All @@ -815,7 +825,7 @@ cloud_storage_azure_container: <container-name>
+
Replace `<placeholders>` with your own values.
To configure access to Azure Blob Storage with shared keys:
To configure access to Azure Blob Storage with account access keys:
. Copy an account access key for the Azure container you want Redpanda to use and enter it in the `cloud_storage_azure_shared_key` property. For information on how to view your account access keys, see the https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#view-account-access-keys[Azure documentation^].
. Run the `rpk cluster config edit` command, then edit the following required properties:
Expand All @@ -832,7 +842,7 @@ Replace `<placeholders>` with your own values.
+
For information about how to grant access from an internet IP range (if you need to open additional routes/ports between your broker nodes and ABS/ADLS; for example, in a hybrid cloud deployment), see the https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#grant-access-from-an-internet-ip-range[Microsoft documentation^].
+
For information about shared key authentication, see the https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key[Microsoft documentation^].
For information about Shared Key authentication, see the https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key[Microsoft documentation^].
+
CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.
Expand Down Expand Up @@ -1717,7 +1727,7 @@ Required for ABS/ADLS.
Required for ABS/ADLS.

| config_ref:cloud_storage_azure_shared_key,true,properties/object-storage-properties[]
| Azure shared key.
| Azure storage account access key.

Required for ABS/ADLS.

Expand Down
17 changes: 10 additions & 7 deletions modules/reference/pages/properties/object-storage-properties.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,15 @@ To authenticate using IAM roles, see <<cloud_storage_credentials_source>>.

=== cloud_storage_api_endpoint

Optional API endpoint.
Optional API endpoint. The only instance in which you must set this value is when using a custom domain with your object storage service.

- AWS: When blank, this is automatically generated using <<cloud_storage_region,region>> and <<cloud_storage_bucket,bucket>>. Otherwise, this uses the value assigned.
- GCP: Uses `storage.googleapis.com`.
- AWS: If not set, this is automatically generated using <<cloud_storage_region,region>> and <<cloud_storage_bucket,bucket>>. Otherwise, this uses the value assigned.
- GCP: If not set, this is automatically generated using `storage.googleapis.com` and <<cloud_storage_bucket,bucket>>.
- Azure: If not set, this is automatically generated using `blob.core.windows.net` and <<cloud_storage_azure_storage_account,`cloud_storage_azure_storage_account`>>. If you have enabled hierarchical namespaces for your storage account and use a custom endpoint, use <<cloud_storage_azure_adls_endpoint,`cloud_storage_azure_adls_endpoint`>>.

*Requires restart:* No

*Optional:* No
*Optional:* Yes (if not using a custom domain)

*Visibility:* `user`

Expand Down Expand Up @@ -354,9 +355,11 @@ When set to `true`, Redpanda automatically retrieves cluster metadata from a spe

Azure Data Lake Storage v2 endpoint override. Use when hierarchical namespaces are enabled on your storage account and you have set up a custom endpoint.

If not set, this is automatically generated using `dfs.core.windows.net` and <<cloud_storage_azure_storage_account,`cloud_storage_azure_storage_account`>>.

*Requires restart:* Yes

*Optional:* No
*Optional:* Yes (if not using a custom domain)

*Visibility:* `user`

Expand All @@ -368,7 +371,7 @@ Azure Data Lake Storage v2 endpoint override. Use when hierarchical namespaces a

=== cloud_storage_azure_adls_port

Azure Data Lake Storage v2 port override. See also: <<cloud_storage_azure_adls_endpoint,`cloud_storage_azure_adls_endpoint`>>. Use when Hierarchical Namespaces are enabled on your storage account and you have set up a custom endpoint.
Azure Data Lake Storage v2 port override. See also: <<cloud_storage_azure_adls_endpoint,`cloud_storage_azure_adls_endpoint`>>. Use when hierarchical namespaces are enabled on your storage account and you have set up a custom endpoint.

*Requires restart:* Yes

Expand Down Expand Up @@ -440,7 +443,7 @@ The managed identity ID to use for access to the Azure storage account. To use A

=== cloud_storage_azure_shared_key

The shared key to be used for Azure Shared Key authentication with the Azure storage account configured by <<cloud_storage_azure_storage_account,`cloud_storage_azure_storage_account`>>. If `null`, the property is disabled.
The account access key to be used for Azure Shared Key authentication with the Azure storage account configured by <<cloud_storage_azure_storage_account,`cloud_storage_azure_storage_account`>>. If `null`, the property is disabled.

NOTE: Redpanda expects this key string to be Base64 encoded.

Expand Down

0 comments on commit c943f90

Please sign in to comment.