Skip to content

Commit

Permalink
HADOOP-19404: [ABFS][FNSOverBlob] Update documentation for FNS Blob O…
Browse files Browse the repository at this point in the history
…nboard (#7341)

Contributed by Anmol Asrani
Reviewed and Signed off by Anuj Modi
  • Loading branch information
anmolanmol1234 authored Feb 5, 2025
1 parent f1ee35b commit f900339
Show file tree
Hide file tree
Showing 2 changed files with 77 additions and 5 deletions.
80 changes: 77 additions & 3 deletions hadoop-tools/hadoop-azure/src/site/markdown/fns_blob.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@

# ABFS Driver for Namespace Disabled Accounts (FNS: Flat Namespace)

### Note: FNS-BLOB Support is being built and not yet ready for usage.

## Background
The ABFS driver is recommended to be used only with HNS Enabled ADLS Gen-2 accounts
for big data analytics because of being more performant and scalable.
Expand Down Expand Up @@ -79,4 +77,80 @@ configured service type.
<name>fs.azure.fns.account.service.type</name>
<value>BLOB</value>
</property>
```
```

5. Auth type supported over FNS Blob: SharedKey, OAuth and fixed SAS.
```xml
<property>
<name>fs.azure.account.auth.type</name>
<value>SharedKey</value>
</property>
```
- How to configure Shared Key
auth: [Shared Key](./abfs.md#a-nameshared-key-autha-default-shared-key)
- How to configure
OAuth: [OAuth](./abfs.md#a-nameoauth-client-credentialsa-oauth-20-client-credentials)
- How to configure fixed
SAS: [Fixed SAS](./abfs.md#using-accountservice-sas-with-abfs)

OAuth is recommended auth type as it is more secure and flexible.

### <a name="renamedeleteoptions"></a> Rename delete configs

The following configs are related to rename and delete operations.

- `fs.azure.blob.copy.progress.wait.millis`: Blob copy API is an async API, this
configuration defines polling duration for checking copy status. The default
value is 1 sec i.e. 1000 ms.

- `fs.azure.blob.copy.max.wait.millis`: Maximum time to wait for a blob copy
operation to complete. The default value is 5 minutes.

- `fs.azure.blob.atomic.rename.lease.refresh.duration`: Blob rename lease
refresh
duration in milliseconds. This setting ensures that the lease on the blob is
periodically refreshed during a rename operation to prevent other operations
from interfering.
The default value is 60 seconds.

- `fs.azure.blob.dir.list.producer.queue.max.size`: Maximum number of blob
entries
enqueued in memory for rename or delete orchestration. The default value is 2
times the default value of list max results, which is 5000, making the current
value 10000.

- `fs.azure.blob.dir.list.consumer.max.lag`: It sets a limit on how much blob
information can be waiting to be processed (consumer lag) during a blob
listing
operation. If the amount of unprocessed blob information exceeds this limit,
the
producer will pause until the consumer catches up and the lag becomes
manageable. The default value is equal to the value of default value of list
max
results which is 5000 currently.

- `fs.azure.blob.dir.rename.max.thread`: Maximum number of threads per blob
rename
orchestration. The default value is 5.

- `fs.azure.blob.dir.delete.max.thread`: Maximum number of thread per
blob-delete
orchestration. The default value currently is 5.

## Features currently not supported

1. **User Delegation SAS** feature is currently not supported but we
plan to bring support for it in the future.
Jira to track this
workitem : https://issues.apache.org/jira/browse/HADOOP-19406.


2. **Context Provider Key (CPK)** support is currently not available. It refers to the ability to use a
customer-provided encryption key to encrypt and decrypt data in Azure Blob
Storage. This feature allows users to manage their own encryption keys,
providing an additional layer of security and control over their data.

## Ask all about ABFS Driver

For any queries related to onboard to FNS Blob or anything related to ABFS
Driver in general, kindly reach out to us at **[email protected]**.
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

import org.junit.Assume;
import org.junit.Test;
import org.mockito.Mockito;
import org.mockito.stubbing.Stubber;
Expand All @@ -39,7 +38,6 @@
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.azurebfs.constants.AbfsServiceType;
import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
import org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations;
import org.apache.hadoop.fs.azurebfs.contracts.services.DfsListResultEntrySchema;
Expand Down

0 comments on commit f900339

Please sign in to comment.