Feature request: Add multiple dimensets to the same Metrics instance

[north-star-saj]

Use case

Overview

Enabling the AWS Powertools Python package to support multiple dimension sets for the same Metrics instance would significantly enhance the packages monitoring capabilities. This feature would allow users to gain more granular insights and comprehensive views of their applications by creating aggregating metrics across various dimensions.

Reference Code

This feature request is akin to the aws_embedded_metrics (code link) put_dimensions method which adds a dimension set to a common MetricsContext. The metrics then get serialized into Embedded Metric Format (EMF) with multiple dimension sets (code link).

Example Use Case

This is a simplified example that demonstrates my use case. In my usecase, one of these dimensions's values is not known in advance. Instead, it is dynamically retrieved.

I am monitoring a lambda that gets deployed to two environments (beta and production) across three regions (us-east-1, us-west-1, and eu-west-1). My lambda produces application-specific metrics such as SuccessfulRequests, FailedRequests, and RetryCount. By creating multiple dimension sets across these three dimensions, I can create aggregate metrics which enable a comprehensive view of my application.

The generated EMF log may look something like this:

{
    "_aws": {
        "Timestamp": 946684800000,
        "CloudWatchMetrics": [
            {
                "Namespace": "MyApplication",
                "Dimensions": [["Environment", "Region"]],
                "Metrics": [
                    {"Name": "SuccessfulRequests", "Unit": "Count"},
                    {"Name": "FailedRequests", "Unit": "Count"},
                    {"Name": "RetryCount", "Unit": "Count"}
                ]
            },
            {
                "Namespace": "MyApplication",
                "Dimensions": [["Environment"]],
                "Metrics": [
                    {"Name": "SuccessfulRequests", "Unit": "Count"},
                    {"Name": "FailedRequests", "Unit": "Count"},
                    {"Name": "RetryCount", "Unit": "Count"}
                ]
            },
            {
                "Namespace": "MyApplication",
                "Dimensions": [["Region"]],
                "Metrics": [
                    {"Name": "SuccessfulRequests", "Unit": "Count"},
                    {"Name": "FailedRequests", "Unit": "Count"},
                    {"Name": "RetryCount", "Unit": "Count"}
                ]
            }
        ]
    },
    "Environment": "Production",
    "Region": "us-west-2",
    "SuccessfulRequests": 20,
    "FailedRequests": 0,
    "RetryCount": 1
}

Benefits

• Improved granularity in monitoring and alerting: Supports better visibility into metrics for environments, regions, and other dynamic factors.
• Easier aggregation of metrics across various dimensions, making it simpler to analyze and report on complex data sets.
• Automation of dimension management reduces manual errors and maintenance overhead.

Solution/User Experience

The following is an example uses a new add_dimension_set method defined in the AmazonCloudWatchEMFProvider class.

# logic which would produce the example EMF log from the ticket
@metrics.log_metrics
def lambda_handler(event: dict, context: LambdaContext):
    metrics.add_dimension_set({"environment": STAGE})
    metrics.add_dimension_set({"environment:" STAGE, "region": REGION})
    metrics.add_metric(name="SuccessfulRequests", unit=MetricUnit.Count, value=20)
    metrics.add_metric(name="FailedRequests", unit=MetricUnit.Count, value=0)
    metrics.add_metric(name="RetryCount", unit=MetricUnit.Count, value=1)

Considerations:

From my point of view, these are a few considerations that will need to be accounted for as part of this feature request.

• Does add_dimension add the dimension to all dimension sets? How does it work when invoked before or after the add_dimension_set method?
• Does add_dimension_set handle duplicate dimensions? What about duplicate dimension keys, but differing values?
• Is the new method name clear to existing and future customers (e.g. add_dimension_set) ?

Alternatives

I've considered the following alternatives. Each of these solutions come up short compared to an easy-to-use method in powertools that lets me add multiple dimension set to the same metric value.

Alternative	Benefits	Weaknesses
Publish metrics using aws-embedded-metrics-python	Supports multi-dimension set metrics	Limited to EMF logging; adds unnecessary dependencies like async which is not required in Lambda.
Create multiple instances of EphemeralMetrics for each dimensionset	Enables aggregated metric logging	Creates excess EMF logs, error prone and repetitive
Create custom metric without using dimension sets	Simple to implement	Lacks granularity and flexibility; requires manual aggregation of metrics and fails to provide the same level of dynamic insight across multiple dimensions.
Use CloudWatch Metric Math	Allows mathematical operations across metrics	Complex to set up, especially for a large number of dimensions. As the number of dimensions increases, maintaining and scaling Metric Math queries becomes harder
Use CloudWatch SEARCH functionality	Enables querying and analyzing metrics data	Requires advanced querying skills, which may not be familiar to all users.Additionally, this approach may require additional resources for query optimization, performance tuning, and potentially more API calls to fetch and process large volumes of data.

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hey @north-star-saj! Thanks so much for opening this issue here! I'm working on finishing up a few items for the release we have on Friday, but I'll give you some feedback here by the end of the week.