Related to https://github.com/grafana/oncall-private/issues/2544

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
This commit is contained in:
Vadim Stepanov 2024-04-16 14:39:23 +01:00 committed by GitHub
parent 689fdba52b
commit a30f4d9527
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 119 additions and 111 deletions

View file

@ -215,6 +215,9 @@ must match the structure of how the fields are nested in the data.
"slack": null,
"telegram": null,
"web": "https://**********.grafana.net/a/grafana-oncall-app/alert-groups/I6HNZGUFG4K11"
},
"labels": {
"urgency": "3"
}
},
"alert_group_id": "I6HNZGUFG4K11",
@ -235,7 +238,10 @@ must match the structure of how the fields are nested in the data.
"id": "CZ7URAT4V3QF2",
"type": "webhook",
"name": "Main Integration - Webhook",
"team": "Webhooks Demo"
"team": "Webhooks Demo",
"labels": {
"urgency": "3"
}
},
"notified_users": [],
"users_to_be_notified": [],
@ -249,6 +255,11 @@ must match the structure of how the fields are nested in the data.
"region": "eu"
}
}
},
"webhook": {
"id": "WH9NSKXWPXSNY3",
"name": "Demo Webhook",
"labels": {}
}
}
```
@ -286,6 +297,7 @@ Details about the alert group associated with this event.
- `{{ alert_group.acknowledged_at }}` - Timestamp alert group was acknowledged (None if not acknowledged yet)
- `{{ alert_group.title }}` - Title of alert group
- `{{ alert_group.permalinks }}` - Links to alert group in web and chat ops if available
- `{{ alert_group.labels }}` - [Alert group labels]
#### `{{ alert_group_id }}`
@ -307,6 +319,7 @@ Details about the integration that received this alert
- `{{ integration.type }}` - Type of integration (grafana, alertmanager, webhook, etc.)
- `{{ integration.name }}` - Name of integration
- `{{ integration.team }}` - Team integration belongs to if integration is assigned to a team
- `{{ integration.labels }}` - [Integration labels]
#### `notified_users`
@ -330,6 +343,14 @@ response of the referenced webhook when it was executed on behalf of the current
See [Advanced Usage - Using response data](#using-response-data) for more details. Access as
`{{ responses["WHP936BM1GPVHQ"].content.message }}` for example
#### `webhook`
Triggered webhook details
- `{{ webhook.id }}` - [UID](#uid) of webhook
- `{{ webhook.name }}` - Name of webhook
- `{{ webhook.labels }}` - Webhook labels
### UID
Templates often use UIDs to make decisions about what actions to take if you need to find the UID of an object
@ -509,7 +530,13 @@ Integrate with third-party services:
- [ServiceNow][]
- [Zendesk][]
{{% docs/reference %}}
- {{% docs/reference %}}
[Alert group labels]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/integrations#alert-group-labels"
[Alert group labels]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations#alert-group-labels"
[Integration labels]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/integrations#labels"
[Integration labels]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations#labels"
[JIRA]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/integrations/jira"
[JIRA]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations/jira"

View file

@ -172,7 +172,7 @@ The Alert Group Labeling feature allows users to:
- Assign labels to alert groups
- Filter alert groups by labels
- Customize the Alert Group table
- Pass labels in [webhooks][webhook-labels]
- Pass labels in [Webhooks]
##### Label Assignment Limits
@ -351,8 +351,8 @@ Users with admin permissions have the ability to add custom columns based on lab
[Routing template]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/jinja2-templating#routing-template"
[Routing template]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/jinja2-templating#routing-template"
[webhook-labels]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/outgoing-webhooks#labels"
[webhook-labels]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/outgoing-webhooks#labels"
[Webhooks]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/outgoing-webhooks"
[Webhooks]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/outgoing-webhooks"
[integration-labels]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/integrations/#labels"
[integration-labels]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations/#labels"

View file

@ -21,130 +21,111 @@ weight: 500
> This integration is not available in OSS version
Grafana OnCall can automatically create, assign and resolve incidents in ServiceNow via [outgoing webhooks][outgoing-webhooks].
This guide provides example webhook configurations for common use cases, as well as information on how to set up a user in ServiceNow to be used by Grafana OnCall.
The bi-directional ServiceNow integration can create and update incidents in ServiceNow based on Grafana OnCall alert
groups, and vice-versa. This integration supports alerts originating from ServiceNow or other integrations such as
Alertmanager, Grafana Alerting, and others.
The integration can automatically:
* Create an incident in ServiceNow when an alert group is created in OnCall.
* Update the incident state in ServiceNow when the alert group status changes in OnCall.
* Create an alert group in OnCall when an incident is created in ServiceNow.
* Update the alert group status in OnCall when the incident state changes in ServiceNow.
## Prerequisites
1. Create a new user in ServiceNow to be used by Grafana OnCall. Obtain the username and password for the user,
these credentials will be used to communicate with ServiceNow REST API.
2. Make sure the user has appropriate permissions to create and update incidents in ServiceNow. By default, the user will need to have the `sn_incident_write` role.
1. Create a new ServiceNow user to be used by Grafana OnCall. On your ServiceNow instance,
navigate to **User Administration** > **Users** and click **New**. Fill in the following details:
* Username: `grafana-oncall`
* First name: `Grafana OnCall`
* Active: ✔
* Web service access only: ✔
## Create incidents in ServiceNow
After creating the user, generate and save a password using the Set Password button for later use.
2. Grant the following roles to the user (use the **Roles** tab):
* `itil` (allows creating and updating incidents)
* `personalize_choices` (allows fetching the list of available incident states)
The steps below describe how to create an outgoing webhook in Grafana OnCall that will allow to automatically create
incidents in ServiceNow from Grafana OnCall alert groups.
## Create integration
Create a new Outgoing Webhook in Grafana OnCall, and configure it as follows:
1. On the **Integrations** tab, click **+ New integration**.
2. Select **ServiceNow** from the list of available integrations.
3. Enter a name and description for the integration.
4. Enter ServiceNow credentials (instance URL, username, and password) and verify the connection.
5. Make sure **Create default outgoing webhooks** is enabled. This will create the necessary webhooks in Grafana OnCall
to send alerts to ServiceNow.
6. Click **Create integration**.
7. Map ServiceNow incident states to OnCall alert group statuses. Example:
* `Firing -> New`
* `Acknowledged -> In Progress`
* `Resolved -> Resolved`
* `Silenced -> Not Selected`
8. Generate a ServiceNow Business Rule script and copy it to your clipboard. This script will allow your ServiceNow
instance to send updates to Grafana OnCall. You won't be able to see the script again after closing the
dialog, but you can regenerate it at any time in integration settings. See the next step for more details on how to
create a Business Rule in ServiceNow using the generated script.
9. On your ServiceNow instance, navigate to **System Definition** > **Business Rules** and click **New**.
Fill in the following details:
* Name: `grafana-oncall`
* Table: `incident`
* Active: ✔
* Advanced: ✔
* When to run > When: `before`
* When to run > Insert: ✔
* When to run > Update: ✔
* Advanced > Script: Paste the generated script
- Trigger type: `Alert Group Created`
Click **Submit** to save the Business Rule.
10. In Grafana OnCall, click **Proceed** to complete the integration setup.
- Integrations: Select integrations that will trigger the webhook
## Test the integration
- HTTP method: `POST`
1. Create a new incident in ServiceNow.
2. Verify that a new alert group is created in Grafana OnCall.
3. Acknowledge the alert group in Grafana OnCall, and verify that the incident state is updated in ServiceNow.
4. Resolve the incident in ServiceNow, and verify that the alert group status is updated in Grafana OnCall.
- Webhook URL:
## Connect other integrations
```text
https://<INSTANCE>.service-now.com/api/now/table/incident
```
You can connect other integrations such as Alertmanager, Grafana Alerting, and others to an existing ServiceNow
integration. When connected, Grafana OnCall will send alerts from the connected integrations to ServiceNow, and update
alert groups on the connected integrations based on incident state changes in ServiceNow. Connected integrations will
use the same ServiceNow credentials and outgoing webhooks as the ServiceNow integration they are connected to.
Replace `<INSTANCE>` with your ServiceNow instance.
To connect other integrations:
- Username: Username of the [ServiceNow user](#prerequisites)
- Password: Password of the [ServiceNow user](#prerequisites)
Use the following JSON template as webhook data:
```json
{
"short_description": "{{alert_group.title}}",
"description": "This incident is created automatically by Grafana OnCall.",
"work_notes": "Grafana OnCall alert group: [code]<a target='_blank' href='{{alert_group.permalinks.web}}'>{{alert_group.id}}</a>[/code]",
"category": "Software"
}
```
## Assign incidents in ServiceNow
The steps below describe how to create an outgoing webhook in Grafana OnCall that will allow to automatically assign incidents in ServiceNow.
The assignment will be performed when an alert group is acknowledged in Grafana OnCall.
- Trigger type: `Acknowledged`
- Integrations: Select integrations that will trigger the webhook
- HTTP method: `PUT`
- Webhook URL:
```text
https://<INSTANCE>.service-now.com/api/now/table/incident/{{responses.<WEBHOOK_ID>.result.sys_id}}
```
Replace `<INSTANCE>` with your ServiceNow instance, and `<WEBHOOK_ID>` with the ID of the [webhook used for creating incidents](#create-incidents-in-servicenow).
- Username: Username of the [ServiceNow user](#prerequisites)
- Password: Password of the [ServiceNow user](#prerequisites)
Use the following JSON template as webhook data:
```json
{
"assigned_to": "{{user.email}}"
}
```
>**Note**: The incident will be assigned to the user that acknowledged the alert group in Grafana OnCall.
The assignment will fail if the user email does not exist in ServiceNow.
## Resolve incidents in ServiceNow
The steps below describe how to create an outgoing webhook in Grafana OnCall that will allow to automatically close
incidents in ServiceNow when an alert group is resolved in Grafana OnCall.
- Trigger type: `Resolved`
- Integrations: Select integrations that will trigger the webhook
- HTTP method: `PUT`
- Webhook URL:
```text
https://<INSTANCE>.service-now.com/api/now/table/incident/{{responses.<WEBHOOK_ID>.result.sys_id}}
```
Replace `<INSTANCE>` with your ServiceNow instance, and `<WEBHOOK_ID>` with the ID of the [webhook used for creating incidents](#create-incidents-in-servicenow).
- Username: Username of the [ServiceNow user](#prerequisites)
- Password: Password of the [ServiceNow user](#prerequisites)
Use the following JSON template as webhook data:
```json
{
"state": 6,
"close_code": "Resolved by caller",
"close_notes": "Resolved by Grafana OnCall."
}
```
>**Note**: Values for fields `state` and `close_code` may be different for your ServiceNow instance, please check and update the values accordingly.
1. Navigate to the **Outgoing** tab of an existing ServiceNow integration.
2. Use the **Send data from other integrations** section to connect other integrations.
3. Enable the **backsync** option if you want alert groups from connected integrations to be updated from ServiceNow.
If disabled, Grafana OnCall will only send alerts to ServiceNow, but not receive updates back.
4. Test the connection by creating a demo alert for the connected integration.
* Verify that an incident is created in ServiceNow.
* Verify that incident state changes in ServiceNow are reflected in Grafana OnCall, and vice-versa.
## Advanced usage
The examples above describe how to create outgoing webhooks in Grafana OnCall that will allow to automatically create, assign and resolve incidents in ServiceNow.
You can customize the integration behaviour by editing the outgoing webhooks on the **Outgoing** tab of the integration.
Consider modifying example templates to fit your use case (e.g. to include more information on alert groups).
Refer to [outgoing webhooks documentation][outgoing-webhooks] for more information on available template variables and webhook configuration.
### Custom incident fields
For more information on ServiceNow REST API, refer to [ServiceNow REST API documentation](https://developer.servicenow.com/dev.do#!/reference/api/sandiego/rest).
You can set custom fields on ServiceNow incidents. To do so, edit the **Alert group created** webhook on
the **Outgoing** tab of the integration.
Example: to set the "urgency" field based on alert group labels, add the following to **Data template**:
```json
{
...,
"urgency": "{{ alert_group.labels.urgency }}"
}
```
Refer to [Outgoing webhook templates] and [Alert group labels] for more info.
{{% docs/reference %}}
[outgoing-webhooks]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/outgoing-webhooks"
[outgoing-webhooks]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/outgoing-webhooks"
[Alert group labels]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/integrations#alert-group-labels"
[Alert group labels]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations#alert-group-labels"
[Outgoing webhook templates]: "/docs/oncall/ -> /docs/oncall/<ONCALL_VERSION>/configure/outgoing-webhooks#outgoing-webhook-templates"
[Outgoing webhook templates]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/outgoing-webhooks#outgoing-webhook-templates"
{{% /docs/reference %}}