Labels docs (#3762)
Co-authored-by: Vadim Stepanov <vadimkerr@gmail.com> Co-authored-by: Joey Orlando <joseph.t.orlando@gmail.com>
This commit is contained in:
Innokentii Konstantinov
GitHub
parent
e477394b9c
commit
7d30a253ef
5 changed files with 310 additions and 8 deletions
|
|
@ -34,6 +34,17 @@ escalation chain and chatops channels.
|
|||
8. Click **Arrow Up** and **Arrow Down** on the right to change the order of routes
|
||||
9. Click **Three dots** and **Delete Route** to delete the route
|
||||
|
||||
### Routing based on labels
|
||||
|
||||
> **Note:** Labels are currently available only in cloud.
|
||||
|
||||
In addition, there is a `labels` variable available to your routing templates, which contains all of the labels assigned
|
||||
to the Alert Group, as a `dict`. This allows you to route based on labels (or a mix of labels and/or payload based data):
|
||||
|
||||
> **Example:**
|
||||
>
|
||||
> * `{{ labels.foo == "bar" or "hello" in labels.keys() or payload.severity == "critical" }}`
|
||||
|
||||
## Escalation Chains
|
||||
|
||||
Once an alert group is created and assigned to the route with escalation chain, the
|
||||
|
|
|
|||
|
|
@ -137,8 +137,195 @@ available in other templates as a variable `{{ source_link }}`.
|
|||
To edit the name of an integration:
|
||||
|
||||
1. Navigate to the **Integrations** tab, select an integration from the list of enabled integrations.
|
||||
1. Click the **three dots** next to the integration name and select **Integration settings**.
|
||||
1. Provide a new name, description, and team, and click **Save**.
|
||||
2. Click the **three dots** next to the integration name and select **Integration settings**.
|
||||
3. Provide a new name, description, and team, and click **Save**.
|
||||
|
||||
#### Labels
|
||||
|
||||
> **Note:** Labels are currently available only in cloud.
|
||||
|
||||
Integration labels allows to manage and filter integrations based on specific criteria
|
||||
and pass these labels down to Alert Groups.
|
||||
It could be useful to organize integrations by service, region or other custom attribute.
|
||||
|
||||
To assign labels to the integration:
|
||||
|
||||
1. Navigate to the **Integrations** tab, select an integration from the list of enabled integrations.
|
||||
2. Click the **three dots** next to the integration name and select **Integration settings**.
|
||||
3. Define a Key and Value of the label, either by:
|
||||
- Selecting existing key and values from the dropdown list
|
||||
- Typing new keys and values into the fields and accepting with enter/return key
|
||||
4. If you want to add more labels click on **Add** button. You can also remove the label using X button next to the key-value pair.
|
||||
5. Click **Save**.
|
||||
|
||||
To filter integrations by labels:
|
||||
|
||||
1. Navigate to the **Integrations** tab
|
||||
2. Find the **Search or filter results…** dropdown and select **Label**
|
||||
3. Start typing to find suggestions and select the key-value pair you'd like to filter by.
|
||||
|
||||
#### Alert Group Labels
|
||||
|
||||
The Alert Group Labeling feature allows users to:
|
||||
|
||||
- Assign labels to alert groups
|
||||
- Filter alert groups by labels
|
||||
- Customize the Alert Group table
|
||||
- Pass labels in [webhooks][webhook-labels]
|
||||
|
||||
##### Label Assignment Limits
|
||||
|
||||
Up to 15 Labels: OnCall allows the assignment of up to 15 labels to an alert group.
|
||||
If there are more than 15 labels to be assigned, only the first 15 labels (sorted alphabetically)
|
||||
from the first alert in the group will be assigned.
|
||||
|
||||
##### Label Persistence
|
||||
|
||||
Once a label is assigned to an alert group, it remains unchanged, even if the label is edited.
|
||||
This approach considers the label as historical data.
|
||||
|
||||
##### Configuration
|
||||
|
||||
Alert Group Labeling is configured per-integration, and the settings are accessible in the Alert Group Labeling tab.
|
||||
|
||||
To find Alert Group Labeling Settings:
|
||||
|
||||
1. Navigate to the **Integrations** tab.
|
||||
2. Select an integration from the list of enabled integrations.
|
||||
3. Click the three dots next to the integration name.
|
||||
4. Choose **Alert Group Labeling**
|
||||
|
||||
##### Assign Labels to Alert Groups
|
||||
|
||||
###### Pass Down Integration Labels
|
||||
|
||||
These labels are automatically assigned to each alert group coming to the integration,
|
||||
based on the labels assigned to the [integration][integration-labels].
|
||||
|
||||
1. Navigate to the Integration Labels section in the Alert Group Labeling tab.
|
||||
2. Enable/disable passing down specific labels using the toggler.
|
||||
|
||||
###### Dynamic & Static Labels
|
||||
|
||||
This feature allows you to assign arbitrary labels to alert groups, either by deriving them from the payload or by specifying static values.
|
||||
Dynamic: label values are extracted from the alert payload using Jinja. Keys remain static.
|
||||
Static: these are not derived from the payload; both key and value are static.
|
||||
These labels will not be attached to the integration.
|
||||
|
||||
1. In the Alert Group Labeling tab, navigate to Dynamic & Static Labels.
|
||||
2. Press the **Add Label** button and choose between dynamic or static labels.
|
||||
|
||||
For Static Labels:
|
||||
|
||||
1. Choose or create key and value from the dropdown list.
|
||||
2. These labels will be assigned to all alert groups received by this integration.
|
||||
|
||||
For Dynamic Labels:
|
||||
|
||||
1. Choose or create a key from the dropdown list.
|
||||
2. Enter a template to parse the value for the given key from the alert payload.
|
||||
|
||||
To illustrate the Dynamic Labeling feature, let's consider an example where a dynamic label is created with a `severity` key
|
||||
and a template to parse values for that key:
|
||||
|
||||
```jinja2
|
||||
{{ payload.get("severity) }}
|
||||
```
|
||||
|
||||
Created dynamic label:
|
||||
<img src="/static/img/oncall/dynamic-label-example.png" width="700px">
|
||||
|
||||
Two alerts were received and grouped to two different alert groups:
|
||||
|
||||
Alert 1:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "critical alert",
|
||||
"severity": "critical"
|
||||
}
|
||||
```
|
||||
|
||||
Alert 2:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "warning alert",
|
||||
"severity": "warning"
|
||||
}
|
||||
```
|
||||
|
||||
As a result:
|
||||
|
||||
- The first alert group will have a label: `severity: critical`.
|
||||
- The second alert group will have a label: `severity: warning`.
|
||||
|
||||
###### Multi-label extraction template
|
||||
|
||||
The Multi-label Extraction Template enables users to extract and modify multiple labels from the alert payload using a single template.
|
||||
This functionality not only supports dynamic values but also accommodates dynamic keys, with the Jinja template required to result in a valid JSON object.
|
||||
|
||||
Consider the following example demonstrating the extraction of labels from a Grafana Alerting payload:
|
||||
|
||||
Incoming Payload (trimmed for readability):
|
||||
|
||||
```json
|
||||
{
|
||||
...
|
||||
"commonLabels": {
|
||||
"job": "node",
|
||||
"severity": "critical",
|
||||
"alertname": "InstanceDown"
|
||||
},
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Template to parse it:
|
||||
|
||||
```jinja2
|
||||
{{ payload.commonLabels | tojson }}
|
||||
```
|
||||
|
||||
As a result `job`, `severity` and `alertname` labels will be assigned to the alert group:
|
||||
|
||||
<img src="/static/img/oncall/mutli-label-extraction-result.png" width="700px">
|
||||
|
||||
An advanced example showcases the extraction of labels from various fields of the alert payload, utilizing the Grafana Alerting payload:
|
||||
|
||||
```jinja2
|
||||
{%- set labels = {} -%}
|
||||
{# add several labels #}
|
||||
{%- set labels = dict(labels, **payload.commonLabels) -%}
|
||||
{# add one label #}
|
||||
{%- set labels = dict(labels, **{"status": payload.status}) -%}
|
||||
{# add label not from payload #}
|
||||
{%- set labels = dict(labels, **{"service": "oncall"}) -%}
|
||||
{# dump labels dict to json string, so OnCall can parse it #}
|
||||
{{ labels | tojson }}
|
||||
```
|
||||
|
||||
#### Alert Group table customization
|
||||
|
||||
Grafana OnCall provides users with the flexibility to customize their Alert Group table to suit individual preferences.
|
||||
This feature allows users to select and manage the columns displayed in the table, including the option to add custom columns based on labels.
|
||||
Customizations made to the Alert Group table are user-specific. Each user can personalize their view according to their preferences.
|
||||
To access customization Navigate to the **Alert Groups** tab and Locate the **Columns** dropdown.
|
||||
|
||||
##### Managing default columns
|
||||
|
||||
By default, the Columns dropdown provides a list of predefined columns that users can enable or disable based on their preferences.
|
||||
|
||||
To manage default columns use the toggler next to each column name to enable or disable its visibility in the table.
|
||||
|
||||
##### Adding Custom Columns
|
||||
|
||||
Users with admin permissions have the ability to add custom columns based on labels. Follow these steps to add a custom column:
|
||||
|
||||
1. Press the Add button in the Columns dropdown. A modal will appear.
|
||||
2. In the modal, begin typing the name of the labels key you want to create a column for.
|
||||
3. Select the desired label from the options presented and press the Add button.
|
||||
4. The new custom column, titled with the label's key, will now be available as an option in the Column Settings for all users.
|
||||
|
||||
## List of available integrations
|
||||
|
||||
|
|
@ -159,4 +346,10 @@ To edit the name of an integration:
|
|||
|
||||
[routing-template]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/jinja2-templating#routing-template"
|
||||
[routing-template]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/jinja2-templating#routing-template"
|
||||
|
||||
[webhook-labels]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/outgoing-webhooks/#labels"
|
||||
[webhook-labels]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/outgoing-webhooks/#labels"
|
||||
|
||||
[integration-labels]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/integrations/#labels"
|
||||
[integration-labels]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations/#labels"
|
||||
{{% /docs/reference %}}
|
||||
|
|
|
|||
|
|
@ -75,6 +75,17 @@ not in the Grafana OnCall integrations list, you can create a generic `webhook`
|
|||
integration, send an alert, and configure
|
||||
your templates.
|
||||
|
||||
### Labels
|
||||
|
||||
> **Note:** Labels are currently available only in cloud.
|
||||
|
||||
In addition, there is a `labels` variable available to your templates, which contains all of the labels assigned
|
||||
to the Alert Group, as a `dict`. This allows you to template based on labels (or a mix of labels and/or payload based data):
|
||||
|
||||
> **Example:**
|
||||
>
|
||||
> - `{{ labels.foo or payload.bar }}`
|
||||
|
||||
## Types of templates
|
||||
|
||||
Alert templates allow you to format any alert fields recognized by Grafana OnCall. You can
|
||||
|
|
@ -88,6 +99,8 @@ customization, use Jinja templates.
|
|||
|
||||
> **Note:** For conditional templates, the output should be `True` to be applied, for example `{{ True if payload.state == 'OK' else False }}`
|
||||
|
||||
See more details in the [Routes][routes] section.
|
||||
|
||||
#### Appearance templates
|
||||
|
||||
How alerts are displayed in the UI, messengers, and notifications
|
||||
|
|
@ -214,3 +227,8 @@ Built-in functions:
|
|||
- Usage example: `{{ payload.data | b64decode }}`
|
||||
|
||||
{{< section >}}
|
||||
|
||||
{{% docs/reference %}}
|
||||
[routes]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/escalation-chains-and-routes#routes"
|
||||
[routes]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes#routes"
|
||||
{{% /docs/reference %}}
|
||||
|
|
|
|||
|
|
@ -176,6 +176,34 @@ Toggle to send the entire webhook payload instead of using the values in the **D
|
|||
| :------: | :----------------------------------------------: | :-----------: |
|
||||
| ❌ | ❌ | _False_ |
|
||||
|
||||
## Labels
|
||||
|
||||
> **Note:** Labels are currently available only in cloud.
|
||||
|
||||
Webhook labels allow to pass labels data to a 3'rd party.
|
||||
Label data will be included in the webhook payload, along with alert group and integration labels.
|
||||
It could be useful such use-cases as delivering Alert Groups with severity to the ServiceNow or
|
||||
forwarding the cluster name to the GitHub issue.
|
||||
Check this [template example][labels_webhook_template] to see how you can include labels in the webhook data.
|
||||
|
||||
Editing Webhook Labels:
|
||||
To edit the labels associated with a webhook, follow these steps:
|
||||
|
||||
1. Navigate to the Webhooks tab.
|
||||
2. Select an integration from the list of enabled integrations.
|
||||
3. Click the three dots next to the webhook name and choose Edit Settings.
|
||||
4. Define a key and value for the label:
|
||||
- Select existing keys and values from the dropdown list, or
|
||||
- Type new keys and values into the fields, accepting with the enter/return key.
|
||||
5. To add more labels, click the Add button. Labels can also be removed using the X button next to the key-value pair.
|
||||
6. Click Save to apply the changes.
|
||||
|
||||
To filter webhooks based on labels, use the following steps:
|
||||
|
||||
1. Navigate to the Webhooks tab.
|
||||
2. Locate the Search or Filter Results… dropdown and select Label.
|
||||
3. Start typing to find suggestions and select the desired key-value pair for filtering. Currently, it's only possible to filter by key-value pairs.
|
||||
|
||||
## Outgoing webhook templates
|
||||
|
||||
The fields that accept a Jinja2 template in outgoing webhooks are able to process data to customize the output.
|
||||
|
|
@ -209,6 +237,9 @@ must match the structure of how the fields are nested in the data.
|
|||
"slack": null,
|
||||
"telegram": null,
|
||||
"web": "https://**********.grafana.net/a/grafana-oncall-app/alert-groups/I6HNZGUFG4K11"
|
||||
},
|
||||
"labels": {
|
||||
"region": "eu-1"
|
||||
}
|
||||
},
|
||||
"alert_group_id": "I6HNZGUFG4K11",
|
||||
|
|
@ -229,7 +260,10 @@ must match the structure of how the fields are nested in the data.
|
|||
"id": "CZ7URAT4V3QF2",
|
||||
"type": "webhook",
|
||||
"name": "Main Integration - Webhook",
|
||||
"team": "Webhooks Demo"
|
||||
"team": "Webhooks Demo",
|
||||
"labels": {
|
||||
"component": "demo"
|
||||
}
|
||||
},
|
||||
"notified_users": [],
|
||||
"users_to_be_notified": [],
|
||||
|
|
@ -243,6 +277,10 @@ must match the structure of how the fields are nested in the data.
|
|||
"region": "eu"
|
||||
}
|
||||
}
|
||||
},
|
||||
"webhook": {
|
||||
"name": "demo_hook",
|
||||
"labels": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -280,6 +318,7 @@ Details about the alert group associated with this event.
|
|||
- `{{ alert_group.acknowledged_at }}` - Timestamp alert group was acknowledged (None if not acknowledged yet)
|
||||
- `{{ alert_group.title }}` - Title of alert group
|
||||
- `{{ alert_group.permalinks }}` - Links to alert group in web and chat ops if available
|
||||
- `{{ alert_group.labels }}` - Labels parsed by OnCall from the first alert in the alert group
|
||||
|
||||
#### `{{ alert_group_id }}`
|
||||
|
||||
|
|
@ -301,6 +340,7 @@ Details about the integration that received this alert
|
|||
- `{{ integration.type }}` - Type of integration (grafana, alertmanager, webhook, etc.)
|
||||
- `{{ integration.name }}` - Name of integration
|
||||
- `{{ integration.team }}` - Team integration belongs to if integration is assigned to a team
|
||||
- `{{ integration.labels }}` - Labels assigned to integration
|
||||
|
||||
#### `notified_users`
|
||||
|
||||
|
|
@ -324,6 +364,14 @@ response of the referenced webhook when it was executed on behalf of the current
|
|||
See [Advanced Usage - Using response data](#using-response-data) for more details. Access as
|
||||
`{{ responses["WHP936BM1GPVHQ"].content.message }}` for example
|
||||
|
||||
#### `webhook`
|
||||
|
||||
Details about the triggered webhook
|
||||
|
||||
- `{{ webhook.id }}` - [UID](#uid) of webhook
|
||||
- `{{ webhook.name }}` - Name of webhook
|
||||
- `{{ webhook.labels }}` - Labels assigned to webhook
|
||||
|
||||
### UID
|
||||
|
||||
Templates often use UIDs to make decisions about what actions to take if you need to find the UID of an object
|
||||
|
|
@ -338,8 +386,19 @@ in the user interface to reference they can be found in the following places:
|
|||
|
||||
UIDs are also visible in the browser URL when a specific object is selected for view or edit.
|
||||
|
||||
- Outgoing Webhook - In the table there is an info icon, UID displayed on hover, click to copy to clipboard
|
||||
- Integration - In integrations beside the name is an info icon, UID displayed on hover, click to copy to clipboard
|
||||
- Routes - With an integration selected beside Send Demo Alert is an infor icon, UID displayed on hover,
|
||||
click to copy to clipboard
|
||||
- Alert group - When viewing an alert group UID is visible in the browser URL
|
||||
- User - When viewing a user's profile UID is visible in the browser URL
|
||||
|
||||
UIDs are also visible in the browser URL when a specific object is selected for view or edit.
|
||||
|
||||
### Template examples
|
||||
|
||||
#### Data in a json body
|
||||
|
||||
The following is an example of an entry in the Data field that would return an alert name and description.
|
||||
|
||||
```json
|
||||
|
|
@ -349,17 +408,33 @@ The following is an example of an entry in the Data field that would return an a
|
|||
}
|
||||
```
|
||||
|
||||
#### Data in a query parameter
|
||||
|
||||
Here is an example using the user's email address as part of a URL:
|
||||
|
||||
```bash
|
||||
```jinja2
|
||||
https://someticketsystem.com/new-ticket?assign-user={{ user.email }}
|
||||
```
|
||||
|
||||
#### JSON webhook payload with the alert-group labels
|
||||
|
||||
This example shows how to construct a custom webhook payload from various webhook data fields and output it as a JSON object
|
||||
|
||||
```jinja2
|
||||
{%- set payload = {} -%}
|
||||
{# add alert group labels #}
|
||||
{%- set payload = dict(payload, **{"labels": alert_group.labels}) -%}
|
||||
{# add some other fields from webhook data just for example #}
|
||||
{%- set payload = dict(payload, **{"event": event.type, "integration": integration.name}) -%}
|
||||
{# encode payload dict to json #}
|
||||
{{ payload | tojson }}
|
||||
```
|
||||
|
||||
#### Note about JSON
|
||||
|
||||
Take this template for example:
|
||||
|
||||
```json
|
||||
```jinja2
|
||||
{
|
||||
"labels": "{{ alert_payload.labels }}"
|
||||
}
|
||||
|
|
@ -375,9 +450,9 @@ It will result in the following (Invalid JSON due to single quotes):
|
|||
|
||||
To fix change the template to:
|
||||
|
||||
```json
|
||||
```tempate
|
||||
{
|
||||
"labels": {{ alert_payload.labels | tojson()}}
|
||||
"labels": {{ alert_payload.labels | tojson }}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -496,3 +571,8 @@ Integrate with third-party services:
|
|||
- [Zendesk]({{< relref "../integrations/zendesk" >}})
|
||||
|
||||
{{< section >}}
|
||||
|
||||
{{% docs/reference %}}
|
||||
[labels_webhook_template]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/outgoing-webhooks/#json-webhook-payload-with-the-alert-group-labels
|
||||
[labels_webhook_template]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/outgoing-webhooks/#json-webhook-payload-with-the-alert-group-labels
|
||||
{{% /docs/reference %}}
|
||||
|
|
|
|||
|
|
@ -93,7 +93,7 @@ const IntegrationLabelsForm = observer((props: IntegrationLabelsFormProps) => {
|
|||
subtitle={
|
||||
<Text size="small" className="u-margin-top-xs">
|
||||
Combination of settings that manage the labeling of alert groups. More information in{' '}
|
||||
<a href={DOCS_ROOT} target="_blank" rel="noreferrer">
|
||||
<a href={`${DOCS_ROOT}/integrations/#alert-group-labels`} target="_blank" rel="noreferrer">
|
||||
<Text type="link">documentation</Text>
|
||||
</a>
|
||||
.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue