-
Notifications
You must be signed in to change notification settings - Fork 8.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
HADOOP-19404: [ABFS][FNSOverBlob] Update documentation for FNS Blob O…
…nboard (#7341) Contributed by Anmol Asrani Reviewed and Signed off by Anuj Modi
- Loading branch information
1 parent
f1ee35b
commit f900339
Showing
2 changed files
with
77 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,8 +14,6 @@ | |
|
||
# ABFS Driver for Namespace Disabled Accounts (FNS: Flat Namespace) | ||
|
||
### Note: FNS-BLOB Support is being built and not yet ready for usage. | ||
|
||
## Background | ||
The ABFS driver is recommended to be used only with HNS Enabled ADLS Gen-2 accounts | ||
for big data analytics because of being more performant and scalable. | ||
|
@@ -79,4 +77,80 @@ configured service type. | |
<name>fs.azure.fns.account.service.type</name> | ||
<value>BLOB</value> | ||
</property> | ||
``` | ||
``` | ||
|
||
5. Auth type supported over FNS Blob: SharedKey, OAuth and fixed SAS. | ||
```xml | ||
<property> | ||
<name>fs.azure.account.auth.type</name> | ||
<value>SharedKey</value> | ||
</property> | ||
``` | ||
- How to configure Shared Key | ||
auth: [Shared Key](./abfs.md#a-nameshared-key-autha-default-shared-key) | ||
- How to configure | ||
OAuth: [OAuth](./abfs.md#a-nameoauth-client-credentialsa-oauth-20-client-credentials) | ||
- How to configure fixed | ||
SAS: [Fixed SAS](./abfs.md#using-accountservice-sas-with-abfs) | ||
|
||
OAuth is recommended auth type as it is more secure and flexible. | ||
|
||
### <a name="renamedeleteoptions"></a> Rename delete configs | ||
|
||
The following configs are related to rename and delete operations. | ||
|
||
- `fs.azure.blob.copy.progress.wait.millis`: Blob copy API is an async API, this | ||
configuration defines polling duration for checking copy status. The default | ||
value is 1 sec i.e. 1000 ms. | ||
|
||
- `fs.azure.blob.copy.max.wait.millis`: Maximum time to wait for a blob copy | ||
operation to complete. The default value is 5 minutes. | ||
|
||
- `fs.azure.blob.atomic.rename.lease.refresh.duration`: Blob rename lease | ||
refresh | ||
duration in milliseconds. This setting ensures that the lease on the blob is | ||
periodically refreshed during a rename operation to prevent other operations | ||
from interfering. | ||
The default value is 60 seconds. | ||
|
||
- `fs.azure.blob.dir.list.producer.queue.max.size`: Maximum number of blob | ||
entries | ||
enqueued in memory for rename or delete orchestration. The default value is 2 | ||
times the default value of list max results, which is 5000, making the current | ||
value 10000. | ||
|
||
- `fs.azure.blob.dir.list.consumer.max.lag`: It sets a limit on how much blob | ||
information can be waiting to be processed (consumer lag) during a blob | ||
listing | ||
operation. If the amount of unprocessed blob information exceeds this limit, | ||
the | ||
producer will pause until the consumer catches up and the lag becomes | ||
manageable. The default value is equal to the value of default value of list | ||
max | ||
results which is 5000 currently. | ||
|
||
- `fs.azure.blob.dir.rename.max.thread`: Maximum number of threads per blob | ||
rename | ||
orchestration. The default value is 5. | ||
|
||
- `fs.azure.blob.dir.delete.max.thread`: Maximum number of thread per | ||
blob-delete | ||
orchestration. The default value currently is 5. | ||
|
||
## Features currently not supported | ||
|
||
1. **User Delegation SAS** feature is currently not supported but we | ||
plan to bring support for it in the future. | ||
Jira to track this | ||
workitem : https://issues.apache.org/jira/browse/HADOOP-19406. | ||
|
||
|
||
2. **Context Provider Key (CPK)** support is currently not available. It refers to the ability to use a | ||
customer-provided encryption key to encrypt and decrypt data in Azure Blob | ||
Storage. This feature allows users to manage their own encryption keys, | ||
providing an additional layer of security and control over their data. | ||
|
||
## Ask all about ABFS Driver | ||
|
||
For any queries related to onboard to FNS Blob or anything related to ABFS | ||
Driver in general, kindly reach out to us at **[email protected]**. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters