Collect ECS or Fargate OpenTelemetry data

Collect Amazon Elastic Container Service or AWS Fargate OpenTelemetry data

You can configure Grafana Alloy or AWS ADOT to collect OpenTelemetry-compatible data from Amazon Elastic Container Service (ECS) or AWS Fargate and forward it to any OpenTelemetry-compatible endpoint.

Metrics are available from various sources including ECS itself, the ECS instances when using EC2, X-Ray, and your own application. You can also collect logs and traces from your applications instrumented for Prometheus or OTLP.

Collect task and container metrics
Collect application telemetry
Collect EC2 instance metrics
Collect application logs

Before you begin

Ensure that you have basic familiarity with instrumenting applications with OpenTelemetry.
Have an available Amazon ECS or AWS Fargate deployment.
Identify where Alloy writes received telemetry data.
Familiarize yourself with the concept of Components in Alloy.

Collect task and container metrics

In this configuration, you add an OpenTelemetry Collector to the task running your application, and it uses the ECS Metadata Endpoint to gather task and container metrics in your cluster.

You can choose between two collector implementations:

You can use ADOT, the AWS OpenTelemetry collector. ADOT has native support for scraping task and container metrics. ADOT comes with default configurations that you can select in the task definition.
You can use Alloy with either the otelcol.receiver.awsecscontainermetrics component or the Prometheus ECS exporter sidecar.

Configure ADOT

If you use ADOT as a collector, add a container to your task definition and use a custom configuration you define in your AWS Systems Manager Parameter Store.

You can find sample OpenTelemetry configuration files in the AWS Observability repository. You can use these samples as a starting point and add the appropriate exporter configuration to send metrics to a Prometheus or OpenTelemetry endpoint.

Use ecs-default-config to consume StatsD metrics, OTLP metrics and traces, and AWS X-Ray SDK traces.
Use otel-task-metrics-config to consume StatsD, OTLP, AWS X-Ray, and Container Resource utilization metrics.

Read otel-prometheus to find out how to set the Prometheus remote write. The example uses AWS managed Prometheus.

Complete the following steps to create a sample task. Refer to the ADOT doc for more information.

Create a Parameter Store entry to hold the collector configuration file.
1. Open the AWS Console.
2. In the AWS Console, choose Parameter Store.
3. Choose Create parameter.
4. Create a parameter with the following values:
  - Name: collector-config
  - Tier: Standard
  - Type: String
  - Data type: Text
  - Value: Copy and paste your custom OpenTelemetry configuration file.
Download the ECS Fargate or ECS EC2 task definition template from GitHub.
Edit the task definition template and add the following parameters.
- {{region}}: The region to send the data to.
- {{ecsTaskRoleArn}}: The AWSOTTaskRole ARN.
- {{ecsTaskExecutionRoleArn}}: The AWSOTTaskExecutionRole ARN.
- Add an environment variable named AOT_CONFIG_CONTENT. Select ValueFrom to tell ECS to get the value from the Parameter Store, and set the value to collector-config.
Follow the ECS Fargate setup instructions to create a task definition using the template.

Configure Alloy

Alloy offers two approaches for collecting ECS task and container metrics:

Native receiver: Use the otelcol.receiver.awsecscontainermetrics component to read the ECS Task Metadata Endpoint V4 directly. This approach is simpler but requires enabling experimental features.
Prometheus exporter: Use the Prometheus ECS exporter as a sidecar container and scrape metrics with prometheus.scrape. This approach requires an additional container.

Native receiver

The otelcol.receiver.awsecscontainermetrics component reads the reads AWS ECS task- and container-level metadata, and resource usage metrics such as CPU, memory, network, and disk, and forwards them to other otelcol.* components.

Note
The otelcol.receiver.awsecscontainermetrics component is experimental. You must set the --stability.level=experimental flag to use it. Refer to Permitted stability levels for more information.

Use the following as a starting point for your Alloy configuration:

otelcol.receiver.awsecscontainermetrics "default" {
  collection_interval = "60s"

  output {
    metrics = [otelcol.exporter.otlphttp.default.input]
  }
}

otelcol.exporter.otlphttp "default" {
  client {
    endpoint = sys.env("OTLP_ENDPOINT")

    auth = otelcol.auth.basic.default.handler
  }
}

otelcol.auth.basic "default" {
  client_auth {
    username = sys.env("OTLP_USERNAME")
    password = sys.env("OTLP_PASSWORD")
  }
}

This configuration collects ECS task and container metrics every 60 seconds and exports them to an OTLP endpoint.

Prometheus exporter

If you prefer a stable approach, use the Prometheus ECS exporter as a sidecar container.

Use the following as a starting point for your Alloy configuration:

prometheus.scrape "ecs" {
  targets    = [{"__address__" = "localhost:9779"}]
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = sys.env("PROMETHEUS_REMOTE_WRITE_URL")

    basic_auth {
      username = sys.env("PROMETHEUS_USERNAME")
      password = sys.env("PROMETHEUS_PASSWORD")
    }
  }
}

This configuration scrapes metrics from the ECS exporter running on port 9779 and exports them to a Prometheus endpoint.

Deploy the task

Complete the following steps to create a sample task.

Create a Parameter Store entry to hold the collector configuration file.
1. Open the AWS Console.
2. In the AWS Console, choose Parameter Store.
3. Choose Create parameter.
4. Create a parameter with the following values:
  - Name: collector-config
  - Tier: Standard
  - Type: String
  - Data type: Text
  - Value: Copy and paste your custom Alloy configuration file.
Download the ECS Fargate or ECS EC2 task definition template from GitHub.
Edit the task definition template and add the following parameters.
- {{region}}: The region to send the data to.
- {{ecsTaskRoleArn}}: The AWSOTTaskRole ARN.
- {{ecsTaskExecutionRoleArn}}: The AWSOTTaskExecutionRole ARN.
- Set the container image to grafana/alloy:<VERSION>, for example grafana/alloy:latest or a specific version such as grafana/alloy:v1.8.0.
- Add a custom environment variable named ALLOY_CONFIG_CONTENT. Select ValueFrom to tell ECS to get the value from the Parameter Store, and set the value to collector-config. Alloy doesn’t read this variable directly, but you use it with the command below to pass the configuration.
- Add environment variables for your endpoint.
  - For the native receiver:
    - OTLP_ENDPOINT
    - OTLP_USERNAME
    - OTLP_PASSWORD - For increased security, create a password in AWS Secrets Manager and reference the ARN of the secret in the ValueFrom field.
  - For the Prometheus exporter:
    - PROMETHEUS_REMOTE_WRITE_URL
    - PROMETHEUS_USERNAME
    - PROMETHEUS_PASSWORD - For increased security, create a password in AWS Secrets Manager and reference the ARN of the secret in the ValueFrom field.
- In the Docker configuration, change the Entrypoint to /bin/sh,-c. This allows the command to use shell features like redirection and chaining.
- {{command}}: For the native receiver, use "printenv ALLOY_CONFIG_CONTENT > /tmp/config_file && exec /bin/alloy run --stability.level=experimental --server.http.listen-addr=0.0.0.0:12345 /tmp/config_file". The --stability.level=experimental flag enables the use of the otelcol.receiver.awsecscontainermetrics component.
  For the Prometheus exporter, use "printenv ALLOY_CONFIG_CONTENT > /tmp/config_file && exec /bin/alloy run --server.http.listen-addr=0.0.0.0:12345 /tmp/config_file".
  Make sure you don’t omit the double quotes around the command.
- Prometheus exporter only: Add the Prometheus ECS exporter as a sidecar container:
  1. Add a container to the task.
  2. Set the container name to ecs-exporter.
  3. Set the image to a pinned version of the exporter, for example quay.io/prometheuscommunity/ecs-exporter:v0.1.1. Check the ecs_exporter releases for the latest stable version. You should avoid using the latest tag in production.
  4. Add tcp/9779 as a port mapping.
Follow the ECS Fargate setup instructions to create a task definition using the template.

Collect EC2 instance metrics

For ECS Clusters running on EC2, you can collect instance metrics by using AWS ADOT or Alloy in a separate ECS task deployed as a daemon.

Collect metrics with Alloy

You can follow the steps described in Configure Alloy to create another task, with the following changes:

If you’re using the Prometheus exporter, you don’t need to add the ecs-exporter sidecar for EC2 instance metrics.
Run the task as a daemon so it automatically runs one instance per node in your cluster.
Update your Alloy configuration to collect metrics from the instance. The configuration varies depending on the type of EC2 node. Refer to the collect documentation for details.

Collect metrics with ADOT

The approach described in the AWS OpenTelemetry documentation uses the awscontainerinsightreceiver receiver from OpenTelemetry. ADOT includes this receiver.

You need to use a custom configuration Parameter Store entry based on the sample configuration file to route the telemetry to your final destination.

Collect application telemetry

To collect metrics and traces emitted by your application, use the OTLP endpoints exposed by the collector sidecar container regardless of the collector implementation. Specify localhost as the host name, which is the default for most instrumentation libraries and agents.

For Prometheus endpoints, add a scrape job to the ADOT or Alloy configuration, and use localhost, service port, and endpoint path.

Collect Logs

The easiest way to collect application logs in ECS is to use the AWS FireLens log driver. Depending on your use case, you can forward your logs to the collector container in your task using the Fluent Bit plugin for OpenTelemetry or using the Fluent Bit Loki plugin. You can also send everything directly to your final destination.

Was this page helpful?

Email docs@grafana.com

Help and support

Community