Merge branch 'main' into update-make-docs

CHANGELOG.md

@@ -20,6 +20,10 @@ Main (unreleased)
 
 - Have `loki.echo` log the `entry_timestamp` and `structured_metadata` for any loki entries received (@dehaansa)
 
+- Bump snmp_exporter and embedded modules in `prometheus.exporter.snmp` to v0.28.0 (@v-zhuravlev)
+
+- Update `prometheus.write.queue` to reduce memory fragmentation and increase sent throughput. (@mattdurham)
+
 - Update mysqld_exporter to v0.17.2, most notable changes: (@cristiangreco)
   - [0.17.1] Add perf_schema quantile columns to collector
   - [0.17.1] Fix database quoting problem in collector 'info_schema.tables'
@@ -33,6 +37,26 @@ Main (unreleased)
 
 - Add support for proxy and headers in `prometheus.write.queue`. (@mattdurham)
 
+- Update `jfr-parser` dependency to v0.9.3 to fix jfr parsing issues in `pyroscope.java`. (@korniltsev)
+
+### Bugfixes
+
+- Fixed an issue where indexing targets as maps (e.g. `target["foo"]`) or objects (e.g. `target.foo`) didn't work in
+  some circumstances resulting in `expected object or array, got capsule` error. This could also lead to
+  `foreach evaluation failed` errors when using the `foreach` configuration block. (@thampiotr)
+
+- Fixed an issue where passing targets from some standard library functions was failing with `target::ConvertFrom` error. (@thampiotr)
+
+- Fixed `expected object or array, got capsule` errors that could be encountered when using targets with `coalesce` and
+  `array.combine_maps` functions. (@thampiotr)
+
+### Other changes
+
+- Upgrading to Prometheus v2.55.1. (@ptodev)
+  - Added a new `http_headers` argument to many `discovery` and `prometheus` components.
+  - Added a new `scrape_failure_log_file` argument to `prometheus.scrape`.
+
+
 v1.7.1
 -----------------
 

Makefile

@@ -48,6 +48,7 @@
 ##   generate-ui               Generate the UI assets.
 ##   generate-versioned-files  Generate versioned files.
 ##   generate-winmanifest      Generate the Windows application manifest.
+##   generate-snmp             Generate SNMP modules from prometheus/snmp_exporter for prometheus.exporter.snmp and bumps SNMP version in _index.md.t.
 ##
 ## Other targets:
 ##
@@ -211,8 +212,8 @@ alloy-image-windows:
 # Targets for generating assets
 #
 
-.PHONY: generate generate-drone generate-helm-docs generate-helm-tests generate-ui generate-versioned-files generate-winmanifest
-generate: generate-drone generate-helm-docs generate-helm-tests generate-ui generate-versioned-files generate-docs generate-winmanifest
+.PHONY: generate generate-drone generate-helm-docs generate-helm-tests generate-ui generate-versioned-files generate-winmanifest generate-snmp
+generate: generate-drone generate-helm-docs generate-helm-tests generate-ui generate-versioned-files generate-docs generate-winmanifest generate-snmp
 
 generate-drone:
 	drone jsonnet -V BUILD_IMAGE_VERSION=$(BUILD_IMAGE_VERSION) --stream --format --source .drone/drone.jsonnet --target .drone/drone.yml
@@ -258,6 +259,18 @@ ifeq ($(USE_CONTAINER),1)
 else
 	go generate ./internal/winmanifest
 endif
+
+generate-snmp:
+ifeq ($(USE_CONTAINER),1)
+	$(RERUN_IN_CONTAINER)
+else
+# Fetch snmp.yml file of the same version as the snmp_exporter go module, use sed to update the file we need to fetch in common.go:
+	@LATEST_SNMP_VERSION=$$(go list -f '{{ .Version }}' -m github.com/prometheus/snmp_exporter); \
+	sed -i "s|snmp_exporter/[^/]*/snmp.yml|snmp_exporter/$$LATEST_SNMP_VERSION/snmp.yml|" internal/static/integrations/snmp_exporter/common/common.go; \
+	go generate ./internal/static/integrations/snmp_exporter/common; \
+	sed -i "s/SNMP_VERSION: v[0-9]\+\.[0-9]\+\.[0-9]\+/SNMP_VERSION: $$LATEST_SNMP_VERSION/" docs/sources/_index.md.t
+endif
+
 #
 # Other targets
 #

docs/sources/_index.md

@@ -7,7 +7,7 @@ cascade:
   ALLOY_RELEASE: v1.8.0
   OTEL_VERSION: v0.119.0
   PROM_WIN_EXP_VERSION: v0.27.3
-  SNMP_VERSION: v0.27.0
+  SNMP_VERSION: v0.28.0
   FULL_PRODUCT_NAME: Grafana Alloy
   PRODUCT_NAME: Alloy
 hero:
@@ -53,7 +53,7 @@ cards:
 
 # Overview
 
-{{< figure src="/media/docs/alloy/flow-diagram-small-alloy.png" alt="Alloy flow diagram" >}}
+{{< figure src="/media/docs/alloy/flow-diagram-alloy_03-2025.png" alt="Alloy flow diagram" >}}
 
 Collect all your telemetry with one product
 

docs/sources/_index.md.t

@@ -7,7 +7,7 @@ cascade:
   ALLOY_RELEASE: $ALLOY_VERSION
   OTEL_VERSION: v0.119.0
   PROM_WIN_EXP_VERSION: v0.27.3
-  SNMP_VERSION: v0.27.0
+  SNMP_VERSION: v0.28.0
   FULL_PRODUCT_NAME: Grafana Alloy
   PRODUCT_NAME: Alloy
 hero:
@@ -53,7 +53,7 @@ cards:
 
 # Overview
 
-{{< figure src="/media/docs/alloy/flow-diagram-small-alloy.png" alt="Alloy flow diagram" >}}
+{{< figure src="/media/docs/alloy/flow-diagram-alloy_03-2025.png" alt="Alloy flow diagram" >}}
 
 Collect all your telemetry with one product
 

docs/sources/collect/ecs-opentelemetry-data.md

@@ -10,13 +10,15 @@ weight: 500
 
 # Collect Amazon Elastic Container Service or AWS Fargate OpenTelemetry data
 
-You can configure {{< param "FULL_PRODUCT_NAME" >}} to collect OpenTelemetry-compatible data from Amazon Elastic Container Service (ECS) or AWS Fargate and forward it to any OpenTelemetry-compatible endpoint.
+You can configure {{< param "FULL_PRODUCT_NAME" >}} or AWS ADOT to collect OpenTelemetry-compatible data from Amazon Elastic Container Service (ECS) or AWS Fargate and forward it to any OpenTelemetry-compatible endpoint.
 
-There are three different ways you can use {{< param "PRODUCT_NAME" >}} to collect Amazon ECS or AWS Fargate telemetry data.
+Metrics are available from various sources including ECS itself, the ECS instances when using EC2, X-Ray, and your own application.
+You can also collect logs and traces from your applications instrumented for Prometheus or OTLP.
 
-1. [Use a custom OpenTelemetry configuration file from the SSM Parameter store](#use-a-custom-opentelemetry-configuration-file-from-the-ssm-parameter-store).
-1. [Create an ECS task definition](#create-an-ecs-task-definition).
-1. [Run {{< param "PRODUCT_NAME" >}} directly in your instance, or as a Kubernetes sidecar](#run-alloy-directly-in-your-instance-or-as-a-kubernetes-sidecar)
+1. [Collect task and container metrics](#collect-task-and-container-metrics)
+1. [Collect application telemetry](#collect-application-telemetry)
+1. [Collect EC2 instance metrics](#collect-ec2-instance-metrics)
+1. [Collect application logs](#collect-logs)
 
 ## Before you begin
 
@@ -25,77 +27,167 @@ There are three different ways you can use {{< param "PRODUCT_NAME" >}} to colle
 * Identify where {{< param "PRODUCT_NAME" >}} writes received telemetry data.
 * Be familiar with the concept of [Components][] in {{< param "PRODUCT_NAME" >}}.
 
-## Use a custom OpenTelemetry configuration file from the SSM Parameter store
+## Collect task and container metrics
 
-You can upload a custom OpenTelemetry configuration file to the SSM Parameter store and use {{< param "PRODUCT_NAME" >}} as a telemetry data collector.
+In this configuration, you add an OTEL collector to the task running your application, and it uses the ECS Metadata Endpoint to gather task and container metrics in your cluster.
 
-You can configure the AWS Distro for OpenTelemetry Collector with the `AOT_CONFIG_CONTENT` environment variable.
-This environment variable contains a full collector configuration file and it overrides the configuration file used in the collector entry point command.
-In ECS, you can set the values of environment variables from AWS Systems Manager Parameters.
+You can choose between two collector implementations:
 
-### Update the task definition
+- You can use ADOT, the AWS OpenTelemetry collector. ADOT has native support for scraping task and container metrics. ADOT comes with default configurations that can be selected in the task definition.
 
-1. Select the task definition.
+- Alternatively you can use {{< param "PRODUCT_NAME" >}} as a collector alongside the [Prometheus ECS exporter](https://github.com/prometheus-community/ecs_exporter) which exposes the ECS metadatada endpoint metrics in Prometheus format.
 
-   1. Open the AWS Systems Manager console.
-   1. Select Elastic Container Service.
-   1. In the navigation pane, choose *Task definition*.
-   1. Choose *Create new revision*.
+### Configure ADOT
 
-1. Add an environment variable.
+If you use ADOT as a collector, add a new container to your task definition and use a custom configuration you define in your AWS SSM Parameter Store.
 
-   1. Select the AWS Distro for OpenTelemetry Collector container and navigate to the Environment variables section.
-   1. Add an environment variable named `AOT_CONFIG_CONTENT`.
-   1. Select `ValueFrom` to tell ECS to get the value from the SSM Parameter, and set the value to `otel-collector-config`.
+You can find sample OTEL configuration files in the [AWS Observability repo][otel-templates].
+You can use these samples as a starting point and add the appropriate exporter configuration to send metrics to a Prometheus or Otel endpoint.
 
-1. Finish updating the task definition and create your revision.
+* Use [`ecs-default-config`][ecs-default-config] to consume StatsD metrics, OTLP metrics and traces, and AWS X-Ray SDK traces.
+* Use [`otel-task-metrics-config`][otel-task-metrics-config] to consume StatsD, OTLP, AWS X-Ray, and Container Resource utilization metrics.
 
-### Create the SSM parameter
+Read [`otel-prometheus`][otel-prometheus] to find out how to set the Prometheus remote write (AWS managed Prometheus in the example).
 
-1. Open the AWS Systems Manager console.
-1. In the navigation pane, choose *Parameter Store*.
-1. Choose *Create parameter*.
-1. Create a parameter with the following values:
+Complete the following steps to create a sample task. Refer to the [ADOT doc][adot-doc] for more information.
 
-   * Name: `otel-collector-config`
-   * Tier: `Standard`
-   * Type: `String`
-   * Data type: `Text`
-   * Value: Copy and paste your custom OpenTelemetry configuration file or [{{< param "PRODUCT_NAME" >}} configuration file][configure].
+1. Create an SSM Parameter Store entry to hold the collector configuration file.
+
+   1. Open the AWS Console.
+   1. In the AWS Console, choose Parameter Store.
+   1. Choose *Create parameter*.
+   1. Create a parameter with the following values:
+
+      * Name: `collector-config`
+      * Tier: Standard
+      * Type: String
+      * Data type: Text
+      * Value: Copy and paste your custom OpenTelemetry configuration file.
 
-### Run your task
-
-When you run a task with this Task Definition, it uses your custom OpenTelemetry configuration file from the SSM Parameter store.
-
-Refer to [Running an application as an Amazon ECS task][run] for more information about running your application as a task.
-
-## Create an ECS Task definition
+1. Download the [ECS Fargate][fargate-template] or [ECS EC2][ec2-template] task definition template from GitHub.
+1. Edit the task definition template and add the following parameters.
 
-To create an ECS Task Definition for AWS Fargate with an ADOT collector, complete the following steps.
+   * `{{region}}`: The region to send the data to.
+   * `{{ecsTaskRoleArn}}`: The AWSOTTaskRole ARN.
+   * `{{ecsExecutionRoleArn}}`: The AWSOTTaskExcutionRole ARN.
+   * Add an environment variable named AOT_CONFIG_CONTENT.
+   Select ValueFrom to tell ECS to get the value from the SSM Parameter, and set the value to `collector-config`.
+
+1. Follow the ECS Fargate setup instructions to [create a task definition][task] using the template.
 
-1. Download the [ECS Fargate task definition template][template] from GitHub.
+### Configure {{% param "PRODUCT_NAME" %}}
+
+Use the following as a starting point for your {{< param "PRODUCT_NAME" >}} configuration:
+
+```
+prometheus.scrape "stats" {
+  targets    = [
+    { "__address__" = "localhost:9779" },
+  ]
+  metrics_path = "/metrics"
+  scheme       = "http"
+  forward_to   = [prometheus.remote_write.default.receiver]
+}
+
+// additional OTEL config as in [ecs-default-config]
+// OTLP receiver
+// statsd
+// Use the alloy convert command to use one of the AWS ADOT files
+// https://grafana.com/docs/alloy/latest/reference/cli/convert/
+...
+
+prometheus.remote_write "default" {
+  endpoint {
+    url = sys.env("PROMETHEUS_REMOTE_WRITE_URL")
+      basic_auth {
+        username = sys.env("PROMETHEUS_USERNAME")
+        password = sys.env("PROMETHEUS_PASSWORD")
+      }
+  }
+}
+```
+
+This configuration sets up a scrape job for the container metrics and export them to a Prometheus endpoint.
+
+Complete the following steps to create a sample task.
+
+1. Create an SSM Parameter Store entry to hold the collector configuration file.
+
+   1. Open the AWS Console.
+   1. In the AWS Console, choose Parameter Store.
+   1. Choose *Create parameter*.
+   1. Create a parameter with the following values:
+
+      * Name: `collector-config`
+      * Tier: Standard
+      * Type: String
+      * Data type: Text
+      * Value: Copy and paste your custom {{< param "PRODUCT_NAME" >}} configuration file.
+
+1. Download the [ECS Fargate][fargate-template] or [ECS EC2][ec2-template] task definition template from GitHub.
 1. Edit the task definition template and add the following parameters.
+
    * `{{region}}`: The region to send the data to.
    * `{{ecsTaskRoleArn}}`: The AWSOTTaskRole ARN.
    * `{{ecsExecutionRoleArn}}`: The AWSOTTaskExcutionRole ARN.
-   * `command` - Assign a value to the command variable to select the path to the configuration file.
-     The AWS Collector comes with two configurations. Select one of them based on your environment:
-     * Use `--config=/etc/ecs/ecs-default-config.yaml` to consume StatsD metrics, OTLP metrics and traces, and AWS X-Ray SDK traces.
-     * Use `--config=/etc/ecs/container-insights/otel-task-metrics-config.yaml` to use StatsD, OTLP, AWS X-Ray, and Container Resource utilization metrics.
+   * Add an environment variable named ALLOY_CONFIG_CONTENT.
+      * Select ValueFrom to tell ECS to get the value from the SSM Parameter, and set the value to `collector-config`.
+   * Add environment variables for Prometheus remote write:
+      * PROMETHEUS_REMOTE_WRITE_URL
+      * PROMETHEUS_USERNAME
+      * PROMETHEUS_PASSWORD *- For increased security, create a password in AWS Secret Manager and reference the ARN of the secret in the ValueFrom field.*
+   * In the docker configuration, change the Entrypoint to `bash,-c`
+   * `{{command}}`: `"echo \"$ALLOY_CONFIG_CONTENT\" > /tmp/config_file && exec alloy run --server.http.listen-addr=0.0.0.0:12345 /tmp/config_file"` *Make sure you don't omit the double quotes around the command.*
+   * Alloy doesn't currently support collecting container metrics from the ECS metadata endpoint directly, so you need to add a second container for the [prometheus exporter](https://github.com/prometheus-community/ecs_exporter) if needed:
+
+      1. Add a new container to the task.
+      1. Set the container name to `"ecs-exporter"`.
+      1. Set the image to `"quay.io/prometheuscommunity/ecs-exporter:latest"`.
+      1. Add `tcp/9779` as a port mapping.
+
 1. Follow the ECS Fargate setup instructions to [create a task definition][task] using the template.
 
-## Run {{% param "PRODUCT_NAME" %}} directly in your instance, or as a Kubernetes sidecar
+## Collect EC2 instance metrics
+
+For ECS Clusters running on EC2, you can collect instance metrics by using AWS ADOT or {{< param "PRODUCT_NAME" >}} in a separate ECS task deployed as a daemon.
+
+### {{% param "PRODUCT_NAME" %}}
+
+You can follow the steps described in [Configure {{< param "PRODUCT_NAME" >}}](#configure-alloy) to create another task, with the following changes:
+
+* Only add the {{< param "PRODUCT_NAME" >}} container, not the Prometheus exporter, and run the task as daemon, so it will automatically run one instance per node in your cluster.
+* Update your {{< param "PRODUCT_NAME" >}} configuration to collect metrics from the instance.
+   The configuration varies depending on the type of EC2 node. Refer to the [`collect`](https://grafana.com/docs/alloy/latest/collect/) documentation for details.
+
+### ADOT
+
+The approach described in [the AWS Otel documentation](https://aws-otel.github.io/docs/setup/ecs#3-setup-the-adot-collector-for-ecs-ec2-instance-metrics) uses the `awscontainerinsightreceiver` receiver from OTel. This receiver is included in ADOT.
+
+You need to use a custom configuration SSM Parameter based on the [sample](https://github.com/aws-observability/aws-otel-collector/blob/main/config/ecs/otel-instance-metrics-config.yaml) configuration file to route the telemetry to your final destination.
+
+## Collect application telemetry
 
-SSH or connect to the Amazon ECS or AWS Fargate-managed container. Refer to [9 steps to SSH into an AWS Fargate managed container][steps] for more information about using SSH with Amazon ECS or AWS Fargate.
+To collect metrics and traces emitted by your application, use the OTLP endpoints exposed by the collector side car container regardless of the collector implementation.
+Specify `localhost` as the host name, which is default for most instrumentation library and agents.
 
-You can also use your own method to connect to the Amazon ECS or AWS Fargate-managed container as long as you can pass the parameters needed to install and configure {{< param "PRODUCT_NAME" >}}.
+For Prometheus endpoints, add a scrape job to the ADOT or {{< param "PRODUCT_NAME" >}} configuration, and use `localhost` , service port, and endpoint path.
 
-### Install {{% param "PRODUCT_NAME" %}}
+## Collect Logs
 
-After connecting to your instance, follow the {{< param "PRODUCT_NAME" >}} [installation][install], [configuration][configure] and [deployment][deploy] instructions.
+The easiest way to collect application logs in ECS is to leverage the [AWS firelens log driver][firelens-doc].
+Depending on your use case, you can forward your logs to the collector container in your task using the [FluentBit plugin for OpenTelemetry][fluentbit-otel-plugin] or using the FluentBit Loki plugin.
+You can also send everything directly to your final destination.
 
 [Components]: https://grafana.com/docs/alloy/<ALLOY_VERSION>/get-started/components
-[template]: https://github.com/aws-observability/aws-otel-collector/blob/master/examples/ecs/aws-cloudwatch/ecs-fargate-sidecar.json
+[firelens-doc]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-taskdef.html
+[fluentbit-otel-plugin]: https://docs.fluentbit.io/manual/pipeline/outputs/opentelemetry
+[otel-templates]: https://github.com/aws-observability/aws-otel-collector/tree/main/config/ecs
+[otel-prometheus]: https://github.com/aws-observability/aws-otel-collector/blob/357f9c7b8896dba6ee0e03b8efd7ca7117024d2e/config/ecs/ecs-amp-xray-prometheus.yaml
+[adot-doc]: https://aws-otel.github.io/docs/setup/ecs
+[otel-task-metrics-config]: https://github.com/aws-observability/aws-otel-collector/blob/main/config/ecs/container-insights/otel-task-metrics-config.yaml
+[ecs-default-config]: https://github.com/aws-observability/aws-otel-collector/blob/main/config/ecs/ecs-default-config.yaml
+[fargate-template]: https://github.com/aws-observability/aws-otel-collector/blob/master/examples/ecs/aws-cloudwatch/ecs-fargate-sidecar.json
+[ec2-template]: https://github.com/aws-observability/aws-otel-collector/blob/main/examples/ecs/aws-prometheus/ecs-fargate-task-def.json
 [configure]: https://grafana.com/docs/alloy/<ALLOY_VERSION>/configure/
 [steps]: https://medium.com/ci-t/9-steps-to-ssh-into-an-aws-fargate-managed-container-46c1d5f834e2
 [install]: https://grafana.com/docs/alloy/<ALLOY_VERSION>/set-up/install/linux/