Component #1823
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,281 @@ | ||
| --- | ||
| canonical: https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.remote.queue/ | ||
| description: Learn about prometheus.remote.queue | ||
| title: prometheus.remote.queue | ||
| --- | ||
|
|
||
|
|
||
| <span class="badge docs-labels__stage docs-labels__item">Experimental</span> | ||
|
|
||
| # prometheus.remote.queue | ||
mattdurham marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| `prometheus.remote.queue` collects metrics sent from other components into a | ||
| Write-Ahead Log (WAL) and forwards them over the network to a series of | ||
| user-supplied endpoints. Metrics are sent over the network using the | ||
| [Prometheus Remote Write protocol][remote_write-spec]. | ||
|
|
||
| You can specify multiple `prometheus.remote.queue` components by giving them different labels. | ||
|
|
||
| You should consider everything here extremely experimental and highly subject to change. | ||
| [emote_write-spec]: https://prometheus.io/docs/specs/remote_write_spec/ | ||
|
|
||
|
|
||
|
|
||
| ## Usage | ||
|
|
||
| ```alloy | ||
| prometheus.remote.queue "LABEL" { | ||
| endpoint "default "{ | ||
| url = REMOTE_WRITE_URL | ||
|
|
||
| ... | ||
| } | ||
|
|
||
| ... | ||
| } | ||
| ``` | ||
|
|
||
| ## Arguments | ||
|
|
||
| The following arguments are supported: | ||
|
|
||
| Name | Type | Description | Default | Required | ||
| ---- | ---- | ----------- | ------- | -------- | ||
| `ttl` | `time` | `duration` | How long the samples can be queued for before they are discarded. | `2h` | no | ||
|
|
||
| ## Blocks | ||
|
|
||
| The following blocks are supported inside the definition of | ||
| `prometheus.remote.queue`: | ||
|
|
||
| Hierarchy | Block | Description | Required | ||
| --------- | ----- | ----------- | -------- | ||
| persistence | [persistence][] | Configuration for persistence | no | ||
| endpoint | [endpoint][] | Location to send metrics to. | no | ||
| endpoint > basic_auth | [basic_auth][] | Configure basic_auth for authenticating to the endpoint. | no | ||
|
|
||
| The `>` symbol indicates deeper levels of nesting. For example, `endpoint > | ||
| basic_auth` refers to a `basic_auth` block defined inside an | ||
| `endpoint` block. | ||
|
|
||
| [endpoint]: #endpoint-block | ||
| [basic_auth]: #basic_auth-block | ||
| [persistence]: #persistence-block | ||
|
|
||
| ### persistence block | ||
|
|
||
| The `persistence` block describes how often and at what limits to write to disk. Persistence settings | ||
| are shared for each `endpoint`. | ||
|
|
||
| The following arguments are supported: | ||
|
|
||
| Name | Type | Description | Default | Required | ||
| ---- | ---- |-------------------------------------------------------------------------------|---------| -------- | ||
| `max_signals_to_batch` | `uint` | The maximum number of signals before they are batched to disk. | `10000` | no | ||
| `batch_frequency` | `duration` | How often to batch signals to disk if `max_signals_to_batch` is not reached. | `5s` | no | ||
|
|
||
|
|
||
| ### endpoint block | ||
|
|
||
| The `endpoint` block describes a single location to send metrics to. Multiple | ||
| `endpoint` blocks can be provided to send metrics to multiple locations. Each | ||
| `endpoint` will have its own WAL folder. | ||
|
|
||
| The following arguments are supported: | ||
|
|
||
| Name | Type | Description | Default | Required | ||
| ---- | ---- |------------------------------------------------------------------| ------ | -------- | ||
| `url` | `string` | Full URL to send metrics to. | | yes | ||
| `write_timeout` | `duration` | Timeout for requests made to the URL. | `"30s"` | no | ||
| `retry_backoff` | `duration` | How often to wait between retries. | `1s` | no | ||
| `max_retry_attempts` | Maximum number of retries before dropping the batch. | `0` | no | ||
| `batch_count` | `uint` | How many series to queue in each queue. | `1000` | no | ||
| `flush_frequency` | `duration` | How often to wait until sending if `batch_count` is not trigger. | `1s` | no | ||
mattdurham marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| `parallelism` | `uint` | How many parallel batches to write. | 10 | no | ||
| `external_labels` | `map(string)` | Labels to add to metrics sent over the network. | | no | ||
|
|
||
| ### basic_auth block | ||
|
|
||
| {{< docs/shared lookup="reference/components/basic-auth-block.md" source="alloy" version="<ALLOY_VERSION>" >}} | ||
|
|
||
|
|
||
| ## Exported fields | ||
|
|
||
| The following fields are exported and can be referenced by other components: | ||
|
|
||
| Name | Type | Description | ||
| ---- | ---- | ----------- | ||
| `receiver` | `MetricsReceiver` | A value that other components can use to send metrics to. | ||
|
|
||
| ## Component health | ||
|
|
||
| `prometheus.remote.queue` is only reported as unhealthy if given an invalid | ||
| configuration. In those cases, exported fields are kept at their last healthy | ||
| values. | ||
|
|
||
| ## Debug information | ||
|
|
||
| `prometheus.remote.queue` does not expose any component-specific debug | ||
| information. | ||
|
|
||
| ## Debug metrics | ||
|
|
||
| The following metrics are provided for backward compatibility. | ||
| They generally behave the same, but there are likely edge cases where they differ. | ||
|
|
||
| * `prometheus_remote_write_wal_storage_created_series_total` (counter): Total number of created | ||
| series appended to the WAL. | ||
| * `prometheus_remote_write_wal_storage_removed_series_total` (counter): Total number of series | ||
| removed from the WAL. | ||
| * `prometheus_remote_write_wal_samples_appended_total` (counter): Total number of samples | ||
| appended to the WAL. | ||
| * `prometheus_remote_write_wal_exemplars_appended_total` (counter): Total number of exemplars | ||
| appended to the WAL. | ||
| * `prometheus_remote_storage_samples_total` (counter): Total number of samples | ||
| sent to remote storage. | ||
| * `prometheus_remote_storage_exemplars_total` (counter): Total number of | ||
| exemplars sent to remote storage. | ||
| * `prometheus_remote_storage_metadata_total` (counter): Total number of | ||
| metadata entries sent to remote storage. | ||
| * `prometheus_remote_storage_samples_failed_total` (counter): Total number of | ||
| samples that failed to send to remote storage due to non-recoverable errors. | ||
| * `prometheus_remote_storage_exemplars_failed_total` (counter): Total number of | ||
| exemplars that failed to send to remote storage due to non-recoverable errors. | ||
| * `prometheus_remote_storage_metadata_failed_total` (counter): Total number of | ||
| metadata entries that failed to send to remote storage due to | ||
| non-recoverable errors. | ||
| * `prometheus_remote_storage_samples_retries_total` (counter): Total number of | ||
| samples that failed to send to remote storage but were retried due to | ||
| recoverable errors. | ||
| * `prometheus_remote_storage_exemplars_retried_total` (counter): Total number of | ||
| exemplars that failed to send to remote storage but were retried due to | ||
| recoverable errors. | ||
| * `prometheus_remote_storage_metadata_retried_total` (counter): Total number of | ||
| metadata entries that failed to send to remote storage but were retried due | ||
| to recoverable errors. | ||
| * `prometheus_remote_storage_samples_dropped_total` (counter): Total number of | ||
| samples which were dropped after being read from the WAL before being sent to | ||
| remote_write because of an unknown reference ID. | ||
| * `prometheus_remote_storage_exemplars_dropped_total` (counter): Total number | ||
| of exemplars that were dropped after being read from the WAL before being | ||
| sent to remote_write because of an unknown reference ID. | ||
| * `prometheus_remote_storage_enqueue_retries_total` (counter): Total number of | ||
| times enqueue has failed because a shard's queue was full. | ||
| * `prometheus_remote_storage_sent_batch_duration_seconds` (histogram): Duration | ||
| of send calls to remote storage. | ||
| * `prometheus_remote_storage_queue_highest_sent_timestamp_seconds` (gauge): | ||
| Unix timestamp of the latest WAL sample successfully sent by a queue. | ||
| * `prometheus_remote_storage_samples_pending` (gauge): The number of samples | ||
| pending in shards to be sent to remote storage. | ||
| * `prometheus_remote_storage_exemplars_pending` (gauge): The number of | ||
| exemplars pending in shards to be sent to remote storage. | ||
| * `prometheus_remote_storage_samples_in_total` (counter): Samples read into | ||
| remote storage. | ||
| * `prometheus_remote_storage_exemplars_in_total` (counter): Exemplars read into | ||
| remote storage. | ||
|
|
||
| Metrics that are new to `prometheus.remote.write`. These are highly subject to change. | ||
|
|
||
| * `alloy_queue_series_serializer_incoming_signals` (counter): Total number of series written to serialization. | ||
| * `alloy_queue_metadata_serializer_incoming_signals` (counter): Total number of metadata written to serialization. | ||
| * `alloy_queue_series_serializer_incoming_timestamp_seconds` (gauge): Highest timestamp of incoming series. | ||
| * `alloy_queue_series_serializer_errors` (gauge): Number of errors for series written to serializer. | ||
| * `alloy_queue_metadata_serializer_errors` (gauge): Number of errors for metadata written to serializer. | ||
| * `alloy_queue_series_network_timestamp_seconds` (gauge): Highest timestamp written to an endpoint. | ||
| * `alloy_queue_series_network_sent` (counter): Number of series sent successfully. | ||
| * `alloy_queue_metadata_network_sent` (counter): Number of metadata sent successfully. | ||
| * `alloy_queue_network_series_failed` (counter): Number of series failed. | ||
| * `alloy_queue_network_metadata_failed` (counter): Number of metadata failed. | ||
| * `alloy_queue_network_series_retried` (counter): Number of series retried due to network issues. | ||
| * `alloy_queue_network_metadata_retried` (counter): Number of metadata retried due to network issues. | ||
| * `alloy_queue_network_series_retried_429` (counter): Number of series retried due to status code 429. | ||
| * `alloy_queue_network_metadata_retried_429` (counter): Number of metadata retried due to status code 429. | ||
| * `alloy_queue_network_series_retried_5xx` (counter): Number of series retried due to status code 5xx. | ||
| * `alloy_queue_network_metadata_retried_5xx` (counter): Number of metadata retried due to status code 5xx. | ||
| * `alloy_queue_network_series_network_duration_seconds` (histogram): Duration writing series to endpoint. | ||
| * `alloy_queue_network_metadata_network_duration_seconds` (histogram): Duration writing metadata to endpoint. | ||
| * `alloy_queue_network_series_network_errors` (counter): Number of errors writing series to network. | ||
| * `alloy_queue_network_metadata_network_errors` (counter): Number of errors writing metadata to network. | ||
|
|
||
| ## Examples | ||
|
|
||
| The following examples show you how to create `prometheus.remote.queue` components that send metrics to different destinations. | ||
|
|
||
| ### Send metrics to a local Mimir instance | ||
|
|
||
| You can create a `prometheus.remote.queue` component that sends your metrics to a local Mimir instance: | ||
|
|
||
| ```alloy | ||
| prometheus.remote.queue "staging" { | ||
| // Send metrics to a locally running Mimir. | ||
| endpoint "mimir" { | ||
| url = "http://mimir:9009/api/v1/push" | ||
|
|
||
| basic_auth { | ||
| username = "example-user" | ||
| password = "example-password" | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Configure a prometheus.scrape component to send metrics to | ||
| // prometheus.remote.queue component. | ||
| prometheus.scrape "demo" { | ||
| targets = [ | ||
| // Collect metrics from the default HTTP listen address. | ||
| {"__address__" = "127.0.0.1:12345"}, | ||
| ] | ||
| forward_to = [prometheus.remote.queue.staging.receiver] | ||
| } | ||
|
|
||
| ``` | ||
|
|
||
| ## Technical details | ||
|
|
||
| `prometheus.remote.queue` uses [snappy][] for compression. | ||
| `prometheus.remote.queue` sends native histograms by default. | ||
| Any labels that start with `__` will be removed before sending to the endpoint. | ||
|
|
||
| ### Data retention | ||
|
|
||
| Data is written to disk in blocks utilizing [snappy][] compression. These blocks are read on startup and resent if they are still within the TTL. | ||
| Any data that has not been written to disk, or that is in the network queues is lost if {{< param "PRODUCT_NAME" >}} is restarted. | ||
|
|
||
| ### Retries | ||
|
|
||
| `prometheus.remote.queue` will retry sending data if the following errors or HTTP status codes are returned: | ||
|
|
||
| * Network errors. | ||
| * HTTP 429 errors. | ||
| * HTTP 5XX errors. | ||
|
|
||
| `prometheus.remote.queue` will not retry sending data if any other unsuccessful status codes are returned. | ||
|
|
||
| ### Memory | ||
|
|
||
| `prometheus.remote.queue` is meant to be memory efficient. | ||
| You can adjust the `max_signals_to_batch`, `queue_count`, and `batch_size` to control how much memory is used. | ||
| A higher `max_signals_to_batch` allows for more efficient disk compression. | ||
| A higher `queue_count` allows more concurrent writes, and `batch_size` allows more data sent at one time. | ||
| This can allow greater throughput at the cost of more memory on both {{< param "PRODUCT_NAME" >}} and the endpoint. | ||
| The defaults are suitable for most common usages. | ||
|
|
||
| <!-- START GENERATED COMPATIBLE COMPONENTS --> | ||
|
|
||
| ## Compatible components | ||
|
There was a problem hiding this comment. This section looks a bit weird. Usually the comments are before the section heading. Also, the same section is listed twice :) It'd be nice to regenerate it. |
||
|
|
||
| `prometheus.remote.queue` has exports that can be consumed by the following components: | ||
|
|
||
| - Components that consume [Prometheus `MetricsReceiver`](../../../compatibility/#prometheus-metricsreceiver-consumers) | ||
|
|
||
| {{< admonition type="note" >}} | ||
| Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. | ||
| Refer to the linked documentation for more details. | ||
| {{< /admonition >}} | ||
|
|
||
| <!-- END GENERATED COMPATIBLE COMPONENTS --> | ||
|
|
||
| [snappy]: https://en.wikipedia.org/wiki/Snappy_(compression) | ||
| [WAL block]: #wal-block | ||
| [Stop]: ../../../../set-up/run/ | ||
| [run]: ../../../cli/run/ | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It'd be nice to add a bit more info on how
prometheus.remote.queueis different fromprometheus.remote_write, and when to use what component.