Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs on severity #1686

Merged
merged 2 commits into from
Jan 13, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 45 additions & 9 deletions docs/setup-robusta/additional-settings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,15 @@ The following labels determine which Kubernetes resource relates to an alert:
* - HorizontalPodAutoscaler
- horizontalpodautoscaler, namespace
* - Node
- node
- node or instance (used as a fallback if node doesn't exist)

If your alerts have different labels, you can change the mapping with the ``alertRelabel`` helm value.

A relabeling has 3 attributes:

* ``source``: Use the value from this label
* ``target``: This label will contain the value from ``source``
* ``operation``: Operation can be ``add`` (default) or ``replace``.
* ``source``: The label's name on your alerts (which differs from the expected value in the above table)
* ``target``: The standard label name that Robusta expects (a value from the table above)
* ``operation``: Either ``add`` (default) or ``replace``. If ``add``, your custom mapping will be recognized *in addition* to Robusta's default mapping.

For example:

Expand All @@ -79,18 +79,54 @@ For example:
Mapping Custom Alert Severity
------------------------------------

To correctly map your custom alert severity, you need to add ``custom_severity_map``. The values for each alert should be: high, medium, low, info, and debug.
To help you prioritize alerts from different sources, Robusta maps alert severity to five standard levels:

For example:
* HIGH - requires your immediate attention - may indicate a service outage
* MEDIUM - likely not a current outage, but could be a warning sign beforehand - should be investigated within a reasonable timeframe (hours to days)
* LOW - minor problems and areas for improvement (e.g. performance) - to be reviewed periodically on a weekly or bi-weekly cadence
* INFO - you probably want to be aware of these, but do not necessarily need to take action
* DEBUG - debug only - can be ignored unless you're actively debugging an issue

You are free to interpret these levels differently, but the above is a good starting point for most companies.

Prometheus alerts are normalized to the above levels as follows:

.. list-table::
:header-rows: 1

* - Prometheus Severity
- Robusta Severity
* - critical
- HIGH
* - high
- HIGH
* - medium
- MEDIUM
* - error
- MEDIUM
* - warning
- LOW
* - low
- LOW
* - info
- INFO
* - debug
- DEBUG

Prometheus alerts with a severity **not in the above list** are mapped to Robusta's INFO level.

You can map your own Prometheus severities, using the ``custom_severity_map`` Helm value. For example:

.. code-block:: yaml

globalConfig:
custom_severity_map:
devs_high: high
ops_high: high
devs_low: low
# maps a p1 value on your own alerts to Robusta's HIGH value
p1: high
# maps a p2 value on your own alerts to Robusta's HIGH value
p2: medium

The mapped values must be one of: high, medium, low, info, and debug.

Two-way Interactivity
------------------------
Expand Down
Loading