Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify Azure storage permissions #763

Merged
merged 7 commits into from
Sep 18, 2024
Merged

Clarify Azure storage permissions #763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
574939e
Clarify Azure storage permissions
kbatuigas Sep 12, 2024
e8d68db
Use 'endpoint'
kbatuigas Sep 12, 2024
7c29c23
Clarify usage for custom endpoints
kbatuigas Sep 13, 2024
11f2e3c
Use 'account access key' for clarity
kbatuigas Sep 13, 2024
44e5ba5
Use stronger language for using Azure custom role
kbatuigas Sep 17, 2024
0fefeca
Apply suggestions from code review
kbatuigas Sep 18, 2024
ec8ddbd
Apply suggestions from code review
kbatuigas Sep 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion modules/manage/pages/security/iam-roles.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,28 @@ If you are using Microsoft Azure as your cloud provider, you must satisfy the fo
. https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp#create-a-user-assigned-managed-identity[Create a user-assigned managed identity]^.
. https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal#create-a-storage-account-1[Create an Azure storage account]^.
. https://learn.microsoft.com/en-us/azure/storage/blobs/blob-containers-portal#create-a-container[Create a container]^ in the storage account.
. Assign the identity, with the https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/storage#storage-blob-data-contributor[`Storage Blob Data Contributor`]^ role, either during the creation of the storage account, or for an existing storage account. See the official https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/qs-configure-portal-windows-vm#user-assigned-managed-identity[Azure Managed Identities] documentation for more guidance.
. Create a https://learn.microsoft.com/en-us/azure/role-based-access-control/custom-roles[custom role^] that only has the minimum permissions required for Tiered Storage, and assign the role to the identity. This helps prevent unauthorized actions on your data and minimize security risks. The custom role should have the following set of permissions:
+
```
"permissions": [
{
"actions": [
],
"notActions": [],
"dataActions": [
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
"Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action",
"Microsoft.Storage/storageAccounts/fileServices/fileShares/files/write",
"Microsoft.Storage/storageAccounts/fileServices/writeFileBackupSemantics/action"
],
"notDataActions": []
}
]
```
+
Assign the identity either during the creation of the storage account, or for an existing storage account. See the https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/qs-configure-portal-windows-vm#user-assigned-managed-identity[Azure Managed Identities] documentation for more guidance.

== Configuring IAM roles

Expand Down
24 changes: 17 additions & 7 deletions modules/manage/partials/tiered-storage.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,7 @@ CAUTION: Do not set an object storage property to an empty string `""` or to `nu

==== Microsoft ABS/ADLS

You can configure access to Azure Blob Storage with either shared keys or Azure's managed identities system to securely interact with Azure Blob Storage. Shared keys, as static credentials, are simple to use but require manual management and vigilant security practices to prevent breaches due to their unchanging nature. In contrast, managed identities provide a more secure and maintenance-free solution by automating credential management and rotation, though they are exclusive to the Azure ecosystem.
You can configure access to Azure Blob Storage with either account access keys or Azure's managed identities system to securely interact with Azure Blob Storage. Account access keys, as static credentials, require manual management and vigilant security practices to prevent breaches due to their unchanging nature. In contrast, managed identities provide a more secure and maintenance-free solution by automating credential management and rotation, though they are exclusive to the Azure ecosystem.

include::manage:partial$azure-blob-limitations.adoc[]

Expand Down Expand Up @@ -582,7 +582,7 @@ NOTE: The `serviceAccount` annotations and the `statefulset` Pod labels are esse

CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value.

To configure access to ABS/ADLS with shared keys:
To configure access to ABS/ADLS with account access keys:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to verify: as far as I can tell, Azure account access keys grant full access to storage account data.

If so, then these instructions we provide for AWS S3 shared keys don't have an equivalent for Azure, correct?

To configure access to an S3 bucket with access keys instead of an IAM role:

  1. Grant a user the following permissions to read and create objects on the bucket to be used with the cluster (or on all buckets): GetObject, DeleteObject, PutObject, PutObjectTagging, ListBucket.

Meaning there is no way to modify permissions if using account access keys, and if they want to go by "least privilege," they must use managed identities instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SMEs--please provide this guidance ( @deniscoady @andijcr or @WillemKauf ) Thx.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meaning there is no way to modify permissions if using account access keys, and if they want to go by "least privilege," they must use managed identities instead?

Good catch. yes, the granular permissions are only for azure managed identities


. Get an account access key for the Azure container that Redpanda will run on. For information on how to view your account access keys, see the https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#view-account-access-keys[Azure documentation^].
. Override the following required cluster properties in the Helm chart:
Expand Down Expand Up @@ -695,7 +695,7 @@ CAUTION: Do not set an object storage property to an empty string `""` or to `nu

- For information about how to grant access from an internet IP range (if you need to open additional routes/ports between your broker nodes and Azure Blob Storage; for example, in a hybrid cloud deployment), see the https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#grant-access-from-an-internet-ip-range[Microsoft documentation^].

- For more information about shared key authentication, see the https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key[Microsoft documentation^].
Copy link
Contributor Author

@kbatuigas kbatuigas Sep 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have gotten some feedback that "Shared Key" might be more of an implementation detail and is not all that helpful for our end users. Is it better to remove this entirely? I'm not sure that "account access key" and "shared key" are interchangeable in the context of ABS.

Copy link

@WillemKauf WillemKauf Sep 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Microsoft docs seem to use "access key" as the term for the generated tokens allowing users access to ABS, while "Shared Key" authorization is the process by which they are used- at least, that's how I'm interpreting the docs.

I think the changes you have made here (using Shared Key authenication to refer to the process and account access keys as the term for the keys themselves) are in line with their messaging.

- For more information about Shared Key authentication, see the https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key[Microsoft documentation^].

endif::[]

Expand Down Expand Up @@ -801,7 +801,17 @@ include::manage:partial$azure-blob-limitations.adoc[]

To configure access to an Azure container with a managed identity:

. Configure an xref:manage:security/iam-roles.adoc#configuring-iam-roles[Azure managed identity].
. Configure an xref:manage:security/iam-roles.adoc#azure-prerequisites[Azure managed identity].
+
Note the minimum set of permissions required for Tiered Storage:
+
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write
- Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action
- Microsoft.Storage/storageAccounts/fileServices/fileShares/files/write
- Microsoft.Storage/storageAccounts/fileServices/writeFileBackupSemantics/action

. Run the `rpk cluster config edit` command, then edit the following required properties:
+
[,properties]
Expand All @@ -815,7 +825,7 @@ cloud_storage_azure_container: <container-name>
+
Replace `<placeholders>` with your own values.

To configure access to Azure Blob Storage with shared keys:
To configure access to Azure Blob Storage with account access keys:

. Copy an account access key for the Azure container you want Redpanda to use and enter it in the `cloud_storage_azure_shared_key` property. For information on how to view your account access keys, see the https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#view-account-access-keys[Azure documentation^].
. Run the `rpk cluster config edit` command, then edit the following required properties:
Expand All @@ -832,7 +842,7 @@ Replace `<placeholders>` with your own values.
+
For information about how to grant access from an internet IP range (if you need to open additional routes/ports between your broker nodes and ABS/ADLS; for example, in a hybrid cloud deployment), see the https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal#grant-access-from-an-internet-ip-range[Microsoft documentation^].
+
For information about shared key authentication, see the https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key[Microsoft documentation^].
For information about Shared Key authentication, see the https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key[Microsoft documentation^].
+
CAUTION: Do not set an object storage property to an empty string `""` or to `null` as a way to reset it to its default value. To reset a property to its default value, run `rpk cluster config force-reset <config-name>` or remove that line from the cluster configuration with `rpk cluster config edit`.

Expand Down Expand Up @@ -1717,7 +1727,7 @@ Required for ABS/ADLS.
Required for ABS/ADLS.

| config_ref:cloud_storage_azure_shared_key,true,properties/object-storage-properties[]
| Azure shared key.
| Azure storage account access key.

Required for ABS/ADLS.

Expand Down
17 changes: 10 additions & 7 deletions modules/reference/pages/properties/object-storage-properties.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,15 @@ To authenticate using IAM roles, see <<cloud_storage_credentials_source>>.

=== cloud_storage_api_endpoint

Optional API endpoint.
Optional API endpoint. The only instance in which you must set this value is when using a custom domain with your object storage service.

- AWS: When blank, this is automatically generated using <<cloud_storage_region,region>> and <<cloud_storage_bucket,bucket>>. Otherwise, this uses the value assigned.
- GCP: Uses `storage.googleapis.com`.
- AWS: If not set, this is automatically generated using <<cloud_storage_region,region>> and <<cloud_storage_bucket,bucket>>. Otherwise, this uses the value assigned.
- GCP: If not set, this is automatically generated using `storage.googleapis.com` and <<cloud_storage_bucket,bucket>>.
- Azure: If not set, this is automatically generated using `blob.core.windows.net` and <<cloud_storage_azure_storage_account,`cloud_storage_azure_storage_account`>>. If you have enabled hierarchical namespaces for your storage account and use a custom endpoint, use <<cloud_storage_azure_adls_endpoint,`cloud_storage_azure_adls_endpoint`>>.

*Requires restart:* No

*Optional:* No
*Optional:* Yes (if not using a custom domain)

*Visibility:* `user`

Expand Down Expand Up @@ -354,9 +355,11 @@ When set to `true`, Redpanda automatically retrieves cluster metadata from a spe

Azure Data Lake Storage v2 endpoint override. Use when hierarchical namespaces are enabled on your storage account and you have set up a custom endpoint.

If not set, this is automatically generated using `dfs.core.windows.net` and <<cloud_storage_azure_storage_account,`cloud_storage_azure_storage_account`>>.

*Requires restart:* Yes

*Optional:* No
*Optional:* Yes (if not using a custom domain)

*Visibility:* `user`

Expand All @@ -368,7 +371,7 @@ Azure Data Lake Storage v2 endpoint override. Use when hierarchical namespaces a

=== cloud_storage_azure_adls_port

Azure Data Lake Storage v2 port override. See also: <<cloud_storage_azure_adls_endpoint,`cloud_storage_azure_adls_endpoint`>>. Use when Hierarchical Namespaces are enabled on your storage account and you have set up a custom endpoint.
Azure Data Lake Storage v2 port override. See also: <<cloud_storage_azure_adls_endpoint,`cloud_storage_azure_adls_endpoint`>>. Use when hierarchical namespaces are enabled on your storage account and you have set up a custom endpoint.

*Requires restart:* Yes

Expand Down Expand Up @@ -440,7 +443,7 @@ The managed identity ID to use for access to the Azure storage account. To use A

=== cloud_storage_azure_shared_key

The shared key to be used for Azure Shared Key authentication with the Azure storage account configured by <<cloud_storage_azure_storage_account,`cloud_storage_azure_storage_account`>>. If `null`, the property is disabled.
The account access key to be used for Azure Shared Key authentication with the Azure storage account configured by <<cloud_storage_azure_storage_account,`cloud_storage_azure_storage_account`>>. If `null`, the property is disabled.

NOTE: Redpanda expects this key string to be Base64 encoded.

Expand Down