Skip to content

Commit

Permalink
Clarify properties in durable storage docs. (#17659)
Browse files Browse the repository at this point in the history
Three changes:

- Add configs for Google Storage, which was previously missing.
- Separate out the configs for the various durable storage types, to
  make it more clear how each one should be configured.
- Replace the blockquote at the top with an infobox, which is more
  appropriate styling for that particular case.
  • Loading branch information
gianm authored Feb 4, 2025
1 parent 31ede5c commit f830a3b
Showing 1 changed file with 37 additions and 8 deletions.
45 changes: 37 additions & 8 deletions docs/operations/durable-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ sidebar_label: "Durable storage"

You can use durable storage to improve querying from deep storage and SQL-based ingestion.

> Note that S3, Azure and Google are all supported as durable storage locations.
:::info
S3, Google Storage, and Azure Blob Storage are supported as durable storage locations.
:::

Durable storage for queries from deep storage provides a location where you can write the results of deep storage queries to. Durable storage for SQL-based ingestion is used to temporarily house intermediate files, which can improve reliability.

Expand All @@ -35,21 +37,48 @@ If the configured limit is too low, Druid may throw the error, `NotEnoughTempora

## Enable durable storage

To enable durable storage, you need to set the following common service properties:
To enable durable storage, you need to set the following common service properties.

For S3:

```
druid.msq.intermediate.storage.enable=true
druid.msq.intermediate.storage.type=s3
# Remote storage location
druid.msq.intermediate.storage.bucket=YOUR_BUCKET
druid.msq.intermediate.storage.prefix=YOUR_PREFIX
# Local temporary directory (on each Druid server)
druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir
```

For Google Storage:

```
druid.msq.intermediate.storage.enable=true
druid.msq.intermediate.storage.type=google
# Remote storage location
druid.msq.intermediate.storage.bucket=YOUR_BUCKET
druid.msq.intermediate.storage.prefix=YOUR_PREFIX
# Local temporary directory (on each Druid server)
druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir
```

# Include these configs if you're using S3
# druid.msq.intermediate.storage.type=s3
# druid.msq.intermediate.storage.bucket=YOUR_BUCKET
For Azure Blob Storage:

# Include these configs if you're using Azure Blob Storage
# druid.msq.intermediate.storage.type=azure
# druid.sq.intermediate.storage.container=YOUR_CONTAINER
```
druid.msq.intermediate.storage.enable=true
druid.msq.intermediate.storage.type=azure
# Remote storage location
druid.msq.intermediate.storage.container=YOUR_CONTAINER
druid.msq.intermediate.storage.prefix=YOUR_PREFIX
# Local temporary directory (on each Druid server)
druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir
```

For detailed information about these and additional settings related to durable storage, see [Durable storage configurations](../multi-stage-query/reference.md#durable-storage-configurations).
Expand Down

0 comments on commit f830a3b

Please sign in to comment.