centralcloud/oncall-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Refactor docs for the new integrations page (#2290)

Browse source 
# What this PR does

## Which issue(s) this PR fixes

## Checklist

- [ ] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)

---------

Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
This commit is contained in:
Ildar Iskhakov 2023-06-27 09:57:27 +08:00 • committed by GitHub
parent dd713bac55
commit e265bade4f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
22 changed files with 1101 additions and 216 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

134
docs/sources/escalation-chains-and-routes/_index.md
View file

@ -6,79 +6,99 @@ weight: 600
# Escalation Chains and Routes
Escalation chains and routes for Grafana OnCall
Administrators can create escalation policies to automatically send alert group notifications to recipients.
These policies define how, where, and when to send notifications.
Escalation policies dictate how users and groups are notified when an alert notification is created. They can be very
simple, or very complex. You can define as many escalation configurations for an integration as you need, and you can
send notifications for certain alerts to a designated place when certain conditions are met, or not met.
Escalation policies have three main parts:
- User settings, where a user sets up their preferred or required notification method.
- An **escalation chain**, which can have one or more steps that are followed in order when a notification is triggered.
- A **route**, that allows administrators to manage notifications by flagging expressions in an alert payload.
## Escalation chains
An escalation chain can have many steps, or only one step. For example, steps can be configured to notify multiple users
in some order, notify users that are scheduled for on-call shifts, ping groups in Slack, use outgoing webhooks to
integrate with other services, such as JIRA, and do a number of other automated notification tasks.
Often alerts from monitoring systems need to be sent to different escalation chains and messaging channels, based on their severity, or other alert content.
## Routes
An escalation workflow can employ **routes** that administrators can configure to filter alerts by regular expressions
(outdated) or Jinja2 templates
in their payloads. Notifications for these alerts can be sent to individuals, or they can make use of a new
or existing escalation chain.
Routes are used to determine which escalation chain should be used for a specific alert
group. A route's ["Routing Templates"]({{< relref "jinja2-templating#routing-template" >}})
are evaluated for each alert and **the first matching route** is used to determine the
escalation chain and chatops channels.
## Configure escalation chains
> **Example:**
>
>
> * trigger escalation chain called `Database Critical` for alerts with `{{ payload.severity == "critical" and payload.service == "database" }}` in the payload
> * create a different route for alerts with the payload `{{ "synthetic-monitoring-dev-" in payload.namespace }}` and select a escalation chain called `Security`.
You can create and edit escalation chains in two places: within **Integrations**, by clicking on an integration tile,
and in **Escalation Chains**. The following steps are for the **Integrations** workflow, but are generally applicable
in both situations.
### Manage routes
You can use **escalation chains** and **routes** to determine ordered escalation procedures. Escalation chains allow
you to set up a series of alert group notification actions that trigger if certain conditions that you specify are
met or not met.
1. Open Integration page
2. Click **Add route** button to create a new route
3. Click **Edit** button to edit `Routing Template`. The routing template must evaluate to `True` for it to apply
4. Select channels in **Publish to Chatops** section
> **Note:** If **Publish to Chatops** section does not exist, connect Chatops integrations first, see more in [docs]({{< relref notify >}})
5. Select **Escalation Chain** from the list
6. If **Escalation Chain** does not exist, click **Add new escalation chain** button to create a new one, it will open in a new tab.
7. Once created, **Reload list**, and select the new escalation chain
8. Click **Arrow Up** and **Arrow Down** on the right to change the order of routes
9. Click **Three dots** and **Delete Route** to delete the route
1. Click on the integration tile for which you want to define escalation policies.
## Escalation Chains
The **Escalations** section for the notification is in the pane to the right of the list of notifications.
You can click **Change alert template and grouping** to customize the look of the alert. You can also do this by
clicking the **Settings** (gear) icon in the integration tile.
Once an alert group is created and assigned to the route with escalation chain, the
escalation chain will be executed. Until user performs an action, which stops the escalation
chain (e.g. acknowledge, resolve, silence etc), the escalation chain will continue to
execute.
1. Create an escalation chain.
Users can create escalation chains to configure different type of escalation workflows.
For example, you can create a chain that will notify on-call users with high priopity, and
another chain that will only send a message into a Slack channel.
In the escalation pane, click **Escalate to** to choose from previously added escalation chains, or create a new one
by clicking **Make a copy** or **Create a new chain**. This will be the name of the escalation policy you define.
Escalation chains determine Who and When to notify. [How to notify]({{< relref notify >}}) is set by the user, based on their own preferences.
1. Add escalation steps.
### Types of escalation steps
Click **Add escalation step** to choose from a set of actions and specify their triggering conditions. By default, the
first step is to notify a slack channel or user. Specify users or channels or toggle the switch to turn this step off.
* `Wait` - wait for a specified amount of time before proceeding to the next step. If you
need a larger time interval, use multiple wait steps in a row.
* `Notify users` - send a notification to a user or a group of users.
* `Notify users from on-call schedule` - send a notification to a user or a group of users
from an on-call schedule.
* `Resolve incident automatically` - resolve the alert group right now with status
`Resolved automatically`.
* `Notify whole slack channel` - send a notification to a slack channel (not recommended
to use as it will spam the channel).
* `Notify Slack User Group` - send a notification to a slack user group.
* `Trigger outgoing webhook` - trigger an [outgoing webhook]({{< relref outgoing-webhooks
To mark an escalation as **Important**, select the option from the step **Start** dropdown menu. User notification
policies can be separately defined for **Important** and **Default** escalations.
>}}).
<https://en.wikipedia.org/>
## Create a route
* `Notify users one by one (round robin)` - each notification will be sent to a group of
users one by one, in sequential order in [round robin fashion](<https://en.wikipedia.org/>
wiki/Round-robin_item_allocation).
* `Continue escalation if current time is in range` - continue escalation only if current
time is in specified range. It will wait for the specfied time to continue escalation.
Useful when you want to get escalation only during working hours
* `Continue escalation if >X alerts per Y minutes (beta)` - continue escalation only if it
passes some threshold
* `Repeat escalation from beginning (5 times max)` - loop the escalation chain
To add a route, click **Add Route**.
### Notification types
You can set up a single route and specify notification escalation steps, or you can add multiple routes, each with
its own configuration.
Each escalation step that notifies a user, does so by triggering their personal notification steps. These are configured in the Grafana
OnCall users page (by clicking "View my profile").
It will be executed for each user in the escalation step
User can configure two types of personal notification chains:
Each route added to an escalation policy follows an `IF`, `ELSE IF`, or `ELSE` path and depends on the type of alert you
specify using a Jinja template that matches content in the payload body of the first alert in alert group. You can also
specify where to send the notification for each route.
* **Default Notifications**
For example, you can send notifications for alerts with `{{ payload.severity == "critical" and payload.service ==
"database" }}` [(Check Jinja2 reference)]({{< relref "jinja2-templating" >}}) in the payload to an escalation chain
called `Bob_OnCall`. You can create a different route for alerts
with the payload `{{ "synthetic-monitoring-dev-" in payload.namespace }}` and select a escalation chain called
`NotifySecurity`.
* **Important Notifications**
> **NOTE:** When you modify an escalation chain or a route, it will modify that escalation chain across
> all integrations that use it.
In the escalation step, user can select which type of notification to use.
Check more information on [Personal Notification Preferences]({{< relref notify >}}) page.
### Manage Escalation Chains
1. Open **Escalation Chains** page
2. Click **New escalation chain** button to create a new escalation chain
3. Enter a name and assign it to a team
> **Note:** Name must be unique across organization
> **Note:** Alert Groups inherit the team from the Integration, not the Escalation Chain
4. Click **Add escalation step** button to add a new step
5. Click **Delete** to delete the Escalation Chain, and **Edit** to edit the name or the team.
> **Important:** Linked Integrations and Routes are displayed in the right panel. Any change in the Escalation Chain will
affect all linked Integrations and Routes.

115
docs/sources/get-started/_index.md
View file

@ -12,10 +12,9 @@ weight: 300
# Get started with Grafana OnCall
Grafana OnCall is an incident response tool built to help DevOps and SRE teams improve their collaboration and resolve
incidents faster.
Grafana OnCall is an incident response tool built to help DevOps and SRE teams improve their collaboration, and resolve incidents faster.
With a centralized view of all your alerts, automated alert escalation and grouping, and on-call scheduling, Grafana
With a centralized view of all your alerts and alert groups, automated escalations and grouping, and on-call scheduling, Grafana
OnCall helps ensure that alert notifications reach the right people, at the right time using the right notification method.
The following diagram details an example alert workflow with Grafana OnCall:
@ -25,23 +24,34 @@ The following diagram details an example alert workflow with Grafana OnCall:
These procedures introduce you to initial Grafana OnCall configuration steps, including monitoring system integration,
how to set up escalation chains, and how to set up calendar for on-call scheduling.
## Before you begin
## Grafana Cloud OnCall vs Open Source Grafana OnCall
Grafana OnCall is available for Grafana Cloud as well as Grafana open source users. You must have a Grafana Cloud account
or use [Open Source Grafana OnCall]({{< relref "../open-source" >}})
Grafana OnCall is available both in Grafana Cloud and Grafana Open Source.
## Install Open Source Grafana OnCall
OnCall is available in Grafana Cloud automatically:
For Open Source Grafana OnCall installation guidance, refer to
[Open Source Grafana OnCall]({{< relref "../open-source" >}})
1. Create or log in into [Grafana Cloud account](https://grafana.com/auth/sign-up/create-user)
2. Sign in to your Grafana stack
3. Choose **Alerts and IRM** from the left menu
4. Click **OnCall** to access Grafana OnCall
> **Note:** If you are using Grafana OnCall with your Grafana Cloud instance there are no install steps. Access Grafana
> OnCall from your Grafana Cloud account and skip ahead to “Get alerts into Grafana OnCall”
Otherwise you'll need to install [Open Source Grafana OnCall]({{< relref "../open-source" >}}) on your own.
## Get alerts into Grafana OnCall
## How to configure Grafana OnCall
Once you’ve installed Grafana OnCall or accessed it from your Grafana Cloud instance, you can begin integrating with
monitoring systems, configuring escalation chains, and get alerts into Grafana OnCall.
* Users with [Admin role]({{< relref "user-and-team-management" >}}) can configure Alert rules (Integrations, Routes, etc)
to define **when and which users to notify**
* OnCall users with [Editor role]({{< relref "user-and-team-management" >}}) can work with Alerts Groups and set up personal settings,
e.g. **how to notify**.
> **Note:** If your role is **Editor**, you can skip to [**Learn Alert Workflow**]({{< relref "#learn-about-the-alert-workflow" >}}) section
of this doc
## Get alerts into Grafana OnCall and configure rules
Once you’ve installed Grafana OnCall, or accessed it from your Grafana Cloud instance, you can begin integrating with
monitoring systems to get alerts into Grafana OnCall. Additionally, you can configure when, and which, users get notified, by setting templates, routes,
escalation chains, etc.
### Integrate with a monitoring system
@ -51,47 +61,31 @@ send a demo alert.
#### Configure your first integration
1. In Grafana OnCall, navigate to the **Integrations** tab and click **+ New integration to receive alerts**.
1. In Grafana OnCall, navigate to the **Integrations** tab and click **+ New integration**.
2. Select an integration from the provided options, if the integration you’re looking for isn’t listed, then select Webhook.
3. Follow the configuration steps on the integration settings page.
4. Complete any necessary configurations in your monitoring system to send alerts to Grafana OnCall.
3. Click **How to connect** to view the instructions specific to your monitoring system
#### Send a demo alert
1. In the integration tab, click **Send demo alert** then navigate to the **Alert Groups** tab to see your test alert firing.
2. Explore the alert by clicking on the title of the alert.
3. Acknowledge and resolve the test alert.
1. In the integration tab, click **Send demo alert**, review and modify the alert payload as needed, and click **Send**
2. Navigate to the **Alert Groups** tab to see your test alert firing
3. Explore the Alert Group by clicking on the title
4. Acknowledge and resolve the test alert group
For more information on Grafana OnCall integrations and further configuration guidance, refer to
[Grafana OnCall integrations]({{< relref "../integrations" >}})
### Learn Alert Flow
### Review and modify alert templates
All Alerts in OnCall are grouped to Alert Groups ([read more about Grouping ID]({{< relref "../jinja2-templating" >}})). Alert Group could have mutually
exclusive states:
Review and customize templates to interpret monitoring alerts and minimize noise. Group alerts, enable auto-resolution,
customize visualizations and notifications by extracting data from alerts. See more details in the
[Jinja2 templating]({{< relref "../jinja2-templating" >}}) section.
- **Firing:** Once Alert Group is registered, Escalation Policy associated with it is getting started. Escalation policy will work while Alert Group is in this status.
- **Acknowledged:** Ongoing Escalation Chain will be interrupted. Unacknowledge will move Alert Group to the "Firing" state and will re-launch Escalation Chain.
- **Silenced:** Similar to "Acknowledged" but designed to be temporary with a timeout. Once time is out, will re-launch Escalation Chain and move Alert Group
to the "Firing" state.
- **Resolved:** Similar to "Acknowledged".
### Configure scalation Chains
Possible transitions:
Escalation chains are a set of steps that define who to notify, and when.
- Firing -> Acknowledged
- Firing -> Silenced
- Firing -> Resolved
- Silenced -> Firing
- Silenced -> Acknowledged
- Silenced -> Resolved
- Acknowledged -> Silenced
- Acknowledged -> Firing
- Acknowledged -> Resolved
- Resolved -> Firing
Transitions change trigger Escalation Chain launch with a few-seconds delay to avoid unexpected notifications.
### Configure Escalation Chains
See more details in the [Escalation Chains]({{< relref "../escalation-chains-and-routes#escalation-chains" >}}) section.
Escalation Chains are customizable automated alert routing steps that enable you to specify who is notified for a certain
alert. In addition to escalation chains, you can configure Routes to send alerts to different escalation chains depending
@ -111,8 +105,40 @@ To configure Escalation Chains:
Alerts from this integration will now follow the escalation steps configured in your Escalation Chain.
For more information on Escalation Chains and more ways to customize them, refer to
[Configure and manage Escalation Chains]({{< relref "escalation-chains-and-routes" >}})
Routes define which messenger channels and escalation chains to use for notifications. See more details in
the [Routes]({{< relref "../escalation-chains-and-routes#routes" >}}) section.
### Learn about the Alert Workflow
* All Alerts in OnCall are grouped into Alert Groups ([read more about Grouping ID]({{< relref "../jinja2-templating" >}})).
An Alert Group can have the following, mutually exclusive states:
* **Firing:** Once Alert Group is registered, Escalation Policy associated with it is getting started.
Escalation policy will work while Alert Group is in this status.
* **Acknowledged:** Ongoing Escalation Chain will be interrupted. Unacknowledge will move Alert Group to
the "Firing" state and will re-launch Escalation Chain.
* **Silenced:** Similar to "Acknowledged" but designed to be temporary with a timeout. Once time is out, will
re-launch Escalation Chain and move Alert Group
to the "Firing" state.
* **Resolved:** Similar to "Acknowledged".
*
* Possible transitions:
* Firing -> Acknowledged
* Firing -> Silenced
* Firing -> Resolved
* Silenced -> Firing
* Silenced -> Acknowledged
* Silenced -> Resolved
* Acknowledged -> Silenced
* Acknowledged -> Firing
* Acknowledged -> Resolved
* Resolved -> Firing
Transition changes trigger Escalation Chains to launch, with a few-second delay (to avoid unexpected notifications).
## Get notified of an alert
In order for Grafana OnCall to notify you of an alert, you must configure how you want to be notified. Personal notification
@ -121,7 +147,8 @@ policies, chatops integrations, and on-call schedules allow you to automate how
### Configure personal notification policies
Personal notification policies determine how a user is notified for a certain type of alert. Get notified by SMS,
phone call, Slack mentions, or mobile push notification. Administrators can configure how users receive notification for certain types of alerts.
phone call, Slack mentions, or mobile push notification. Administrators can configure how users receive notifications
for certain types of alerts.
For more information on personal notification policies, refer to
[Manage users and teams for Grafana OnCall]({{< relref "user-and-team-management" >}})

96
docs/sources/integrations/_index.md
View file

@ -14,22 +14,24 @@ weight: 500
# Integrations
"Integration" is a main entry point for alerts being consumed by OnCall. Rendering, grouping and routing are configured
within integrations.
"Integration" is a set of Jinja2 templates which is transforming alert payload to the format suitable to OnCall.
You could check pre-configured templates in the list of avaliable integrations (Integrations ->
"New integration to receive alerts"), create your own or adjust existing.
An "Integration" is a main entry point for alerts being consumed by Grafana OnCall.
Integrations receive alerts on a unique API URL, interprets them using a set of templates tailored for the monitoring system, and starts
escalations.
Read more about Jinja2 templating used in OnCall [here]({{< relref "../jinja2-templating" >}}).
Alert flow within integration:
## Learn Alert Flow Within Integration
1. Alert is registered by unique integration url (or [e-mail]({{< relref "inbound-email" >}}) in case of inbound e-mail
integration)
2. If there is a non-resolved "alert group" with the same "grouping id", alert will be added to this "alert group".
3. If there is no non-resolved "alert group" with the same "grouping id", new "alert group" will be issued.
4. New "alert group" will be routed using routing engine and escalation chain will be started (TODO: link).
1. An Alert is received on an integration's **Unique URL** as an HTTP POST request with a JSON payload (or via
[e-mail]({{< relref "inbound-email" >}}), for inbound e-mail integrations)
1. Routing is determined for the incoming alert, by applying the [Routing Template]({{< relref "jinja2-templating#routing-template" >}})
1. Alert Grouping is determined based on [Grouping Id Template]({{< relref "jinja2-templating#behavioral-template" >}})
1. An Alert Group may be acknowledged or resolved with status `_ by source` based on
[Behaviour Templates]({{< relref "jinja2-templating#behavioral-template" >}})
1. The Alert Group is available in Web, and can be published to messengers, based on the Route's **Publish to Chatops** configuration.
It is rendered using [Appearance Templates]({{< relref "jinja2-templating#appearance-template" >}})
1. The Alert Group is escalated to uers based on the Escalation Chains selected for the Route
1. Users can perform actions listed in [Learn Alert Workflow]({{< relref "get-started#learn-alert-workflow" >}}) section
## Configure and manage integrations
@ -40,19 +42,69 @@ describe how to configure and customize your integrations to ensure alerts are t
To configure an integration for Grafana OnCall:
1. In Grafana OnCall, navigate to the **Integrations** tab and click **+ New integration to receive alerts**.
2. Select an integration from the provided options, if the integration you want isn’t listed, then select **Webhook**.
3. Follow the configuration steps on the integration settings page.
4. Complete any necessary configurations in your tool to send alerts to Grafana OnCall.
1. In Grafana OnCall, navigate to the **Integrations** tab and click **+ New integration**.
1. Select an integration type from the [list of available integrations]({{< relref "#list-of-available-integrations" >}}).
If the integration you want isn’t listed, then select **Webhook**.
1. Fill in a title and a description for your integration, assign it to a team, and click **Create Integration**.
1. The Integration page will open. Here you will see details about the Integration.
You can use the HTTP Endpoint url to send events from an external monitoring system.
Click the **How to connect** link for more information.
1. Complete any necessary configurations in your tool to send alerts to Grafana OnCall.
1. Click **Send demo alert** to send a test alert to Grafana OnCall.
### Complete the integration configuration
- Review and customise grouping, autoresolution, autoacknowledge, etc [templates]({{< relref "../jinja2-templating" >}})
if you want to customise alert behaviour for your team
- Review and customise [other templates]({{< relref "../jinja2-templating" >}}) to change how alert groups are displayed
in different parts of Grafana OnCall: UI, messengers, emails, notifications, etc.
- Add routes to your integration to route alerts to different users and teams based on labels or other data
- Connect your escalation chains to routes to notify the right people, at the right time
- Learn [how to start Maintenance Mode]({{< relref "#maintenance-mode" >}}) for an integration
- Send demo alerts to an integration to make sure routes, templates, and escalations, are working as expected. Consider using
`Debug Maintenance mode` to avoid sending real notifications to your team
### Manage integrations
To manage existing integrations, navigate to the **Integrations** tab in Grafana OnCall and select the integration
you want to manage.
#### Manage integration behaviour and rendering
#### Maintenance Mode
Start maintenance mode when performing scheduled maintenance or updates on your infrastructure, which may trigger false alarms.
There are two possible maintenance modes:
- **Debug** - test routing and escalations without real notifications. Alerts will be processed as usual, but no notifications
will be sent to users.
- **Maintenance** - group alerts into one during infrastructure work.
##### Manage maintenance Mode
1. Go to the Integration page and click **Three dots**
1. Select **Start Maintenance Mode**
1. Select **Debug** or **Maintenance** mode
1. Set the **Duration** of Maintenance Mode
1. Click **Start**
1. If you want to stop maintenance mode before it ends, click **Three dots** and select **Stop Maintenance Mode**
#### Heartbeat monitoring
An OnCall heartbeat acts as a healthcheck for alert group monitoring. You can configure you monitoring to regularly send alerts
to the heartbeat endpoint. If OnCall doen't receive one of these alerts, it will create an new alert group and escalate it
1. Go to Integration page and click **Three dots**
1. Select **Heartbeat Settings**
1. Set **Heartbeat interval**
1. Copy **Endpoint** into you monitoring system.
More specific instructions can be found in a specific integration's documentation.
#### Behaviour and rendering templates example
"Integration templates" are Jinja2 templates which are applied to each alert to define it's rendering and behaviour.
Read more in [Templates guide]({{< relref jinja2-templating>}})
For templates editor:
1. Navigate to the **Integrations** tab, select an integration from the list.
@ -80,14 +132,14 @@ template: `{{ payload.region }}}`
Should point to the most specific place in the alert source related to the alert group. Also rendering result will be
available in other templates as a variable `{{ source_link }}`.
Read more about Jinja2 (TODO: link) in a specific section.
#### Edit integration name
#### Edit integration name, description and assigned team
To edit the name of an integration:
1. Navigate to the **Integrations** tab, select an integration from the list of enabled integrations.
2. Click the **pencil icon** next to the integration name.
3. Provide a new name and click **Update**.
1. Click the **three dots** next to the integration name and select **Integration settings**.
1. Provide a new name, description, and team, and click **Save**.
## List of available integrations
{{< section >}}

96
docs/sources/integrations/alertmanager/index.md
View file

@ -15,24 +15,33 @@ weight: 300
# Alertmanager integration for Grafana OnCall
The Alertmanager integration for Grafana OnCall handles alerts sent by client applications such as the Prometheus server.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
Grafana OnCall provides<!--[grouping](#alertmanager-grouping-amp-oncall-grouping)--> grouping abilities when processing
alerts from Alertmanager, including initial deduplicating, grouping, and routing the alerts to Grafana OnCall.
The Alertmanager integration handles alerts from [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/).
This integration is the recommended way to send alerts from Prometheus deployed in your infrastructure, to Grafana OnCall.
## Configure Alertmanager integration for Grafana OnCall
> **Pro tip:** Create one integration per team, and configure alertmanager labels selector to send alerts only related to that team
You must have an Admin role to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Prometheus Alertmanager
1. In the **Integrations** tab, click **+ New integration to receive alerts**.
2. Select **Alertmanager** from the list of available integrations.
3. Follow the instructions in the **How to connect** window to get your unique integration URL and identify next steps.
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Alertmanager Prometheus** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
You will need it when configuring Alertmanager.
<!--![123](../_images/connect-new-monitoring.png)-->
## Configure Alertmanager
## Configuring Alertmanager to Send Alerts to Grafana OnCall
Update the `receivers` section of your Alertmanager configuration to use a unique integration URL:
1. Add a new [Webhook](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config) receiver to `receivers`
section of your Alertmanager configuration
2. Set `url` to the **OnCall Integration URL** from previous section
3. Set `send_resolved` to `true`, so Grafana OnCall can autoresolve alert groups when they are resolved in Alertmanager
4. It is recommended to set `max_alerts` to less than `300` to avoid rate-limiting issues
5. Use this receiver in your route configuration
Here is the example of final configuration:
```yaml
route:
@ -44,28 +53,63 @@ receivers:
webhook_configs:
- url: <integation-url>
send_resolved: true
max_alerts: 300
```
## Configure grouping with Alertmanager and Grafana OnCall
## Complete the Integration Configuration
You can use the alert grouping mechanics of Alertmanager and Grafana OnCall to configure your alert grouping preferences.
Complete configuration by setting routes, templates, maintenances, etc. Read more in
[this section]({{< relref "../../integrations/#complete-the-integration-configuration" >}})
Alertmanager offers three alert grouping options:
## Configuring OnCall Heartbeats (optional)
- `group_by` provides two options, `instance` or `job`.
- `group_wait` sets the length of time to initially wait before sending a notification for a particular group of alerts.
For example, `group_wait` can be set to 45s.
An OnCall heartbeat acts as a monitoring for monitoring systems. If your monitoring is down and stop sending alerts,
Grafana OnCall will notify you about that.
Setting a high value for `group_wait` reduces alert noise and minimizes interruption, but it may introduce delays in
receiving alert notifications. To set an appropriate wait time, consider whether the group of alerts will be the same
as those previously sent.
### Configuring Grafana OnCall Heartbeat
- `group_interval` sets the length of time to wait before sending notifications about new alerts that have been added to
a group of alerts that have been previously alerted on. This setting is usually set to five minutes or more.
1. Go to **Integration Page**, click on three dots on top right, click **Heartbeat settings**
2. Copy **OnCall Heartbeat URL**, you will need it when configuring Alertmanager
3. Set up **Heartbeat Interval**, time period after which Grafana OnCall will start a new alert group if it
doesn't receive a heartbeat request
During high alert volume periods, Alertmanager will send alerts at each `group_interval`, which can mean a lot of
distraction. Grafana OnCall grouping will help manage this in the following ways:
### Configuring Alertmanager to send heartbeats to Grafana OnCall Heartbeat
- Grafana OnCall groups alerts based on the first label of each alert.
- Grafana OnCall marks an alert group as resolved only when there are fewer than 500 grouped
alerts, and every `firing` alert with the same labels has a corresponding `resolved` alert.
You can configure Alertmanager to regularly send alerts to the heartbeat endpoint. Add `vector(1)` as a heartbeat
generator to `prometheus.yaml`. It will always return true and act like always firing alert, which will be sent to
Grafana OnCall once in a given period of time:
```yaml
groups:
- name: meta
rules:
- alert: heartbeat
expr: vector(1)
labels:
severity: none
annotations:
description: This is a heartbeat alert for Grafana OnCall
summary: Heartbeat for Grafana OnCall
```
Add receiver configuration to `prometheus.yaml` with the **OnCall Heartbeat URL**:
```yaml
...
route:
...
routes:
- match:
alertname: heartbeat
receiver: 'grafana-oncall-heartbeat'
group_wait: 0s
group_interval: 1m
repeat_interval: 50s
receivers:
- name: 'grafana-oncall-heartbeat'
webhook_configs:
- url: https://oncall-dev-us-central-0.grafana.net/oncall/integrations/v1/alertmanager/1234567890/heartbeat/
send_resolved: false
```

37
docs/sources/integrations/amazon-sns/index.md Normal file
View file

@ -0,0 +1,37 @@
---
aliases:
- add-amazon-sns/
- /docs/oncall/latest/integrations/available-integrations/configure-amazon-sns/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-amazon-sns/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- amazon-sns
title: Amazon SNS
weight: 500
---
# Amazon SNS integration for Grafana OnCall
The Amazon SNS integration for Grafana OnCall handles ticket events sent from Amazon SNS webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations
in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Amazon SNS
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Amazon SNS** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from
**HTTP Endpoint** section.
## Configuring Amazon SNS to Send Alerts to Grafana OnCall
1. Create a new Topic in <https://console.aws.amazon.com/sns>
2. Open this topic, then create a new subscription
3. Choose the protocol HTTPS
4. Add the **OnCall Integration URL** to the Amazon SNS Endpoint

88
docs/sources/integrations/appdynamics/index.md
View file

@ -18,19 +18,91 @@ weight: 500
The AppDynamics integration for Grafana OnCall handles health rule violation events sent from AppDynamics actions.
The integration provides grouping and auto-resolve logic via customizable alert templates.
## Configure AppDynamics integration for Grafana OnCall
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
You must have an Admin role to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from AppDynamics
1. In the **Integrations** tab, click **+ New integration to receive alerts**.
1. In the **Integrations** tab, click **+ New integration**.
2. Select **AppDynamics** from the list of available integrations.
3. Follow the instructions in the **How to connect** window to get your unique integration URL and review next steps.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
You will need it when configuring AppDynamics.
## Grouping and auto-resolve
## Configuring AppDynamics to Send Alerts to Grafana OnCall
Create a new HTTP Request Template in AppDynamics to send events to Grafana OnCall using the integration URL above.
Refer to
[AppDynamics documentation]
(<https://docs.appdynamics.com/appd/23.x/latest/en/appdynamics-essentials/alert-and-respond/actions/http-request-actions-and-templates>)
for more information on **how to create HTTP Request Templates**:
Use the following values when configuring a new HTTP Request Template:
* Request URL:
* Method: POST
* Raw URL: **OnCall Integration URL** from previous section
* Authentication:
* Type: None
* Payload:
* MIME Type: application/json
* Template:
```json
{
"event": {
"eventType": "${latestEvent.eventType}",
"id": "${latestEvent.id}",
"guid": "${latestEvent.guid}",
"eventTypeKey": "${latestEvent.eventTypeKey}",
"eventTime": "${latestEvent.eventTime}",
"displayName": "${latestEvent.displayName}",
"summaryMessage": "${latestEvent.summaryMessage}",
"eventMessage": "${latestEvent.eventMessage}",
"application": {
"name": "${latestEvent.application.name}"
},
"node": {
"name": "${latestEvent.node.name}"
},
"severity": "${latestEvent.severity}",
"deepLink": "${latestEvent.deepLink}"
}
}
```
* Response Handling Criteria:
* Success Criteria: Status Code 200
* Settings:
* One Request Per Event: Enabled
After setting up a template, create a new action in AppDynamics and select the template you created earlier.
Now you can configure policies to trigger the action when certain events occur in AppDynamics.
When configuring a policy, select the following events to trigger the action:
```plain
Health Rule Violation Started - Warning
Health Rule Violation Started - Critical
Health Rule Violation Continues - Warning
Health Rule Violation Continues - Critical
Health Rule Violation Upgraded - Warning to Critical
Health Rule Violation Downgraded - Critical to Warning
Health Rule Violation Ended - Warning
Health Rule Violation Ended - Critical
Health Rule Violation Canceled - Warning
Health Rule Violation Canceled - Critical
```
After setting up the connection, you can test it by sending a test request from the AppDynamics UI.
## Understanding How Alerts Grouped and Auto-resolved
Grafana OnCall provides grouping and auto-resolve logic for the AppDynamics integration:
- Alerts created from health rule violation events are grouped by application and node name
- Alert groups are auto-resolved when the health rule violation is ended or canceled
* Alerts created from health rule violation events are grouped by application and node name
* Alert groups are auto-resolved when the health rule violation is ended or canceled
To customize this behaviour, consider modifying alert templates in integration settings.
## Complete the Integration Configuration
Complete configuration by setting routes, templates, maintenances, etc. Read more in
[this section]({{< relref "../../integrations/#complete-the-integration-configuration" >}})

38
docs/sources/integrations/datadog/index.md Normal file
View file

@ -0,0 +1,38 @@
---
aliases:
- add-datadog/
- /docs/oncall/latest/integrations/available-integrations/configure-datadog/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-datadog/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- datadog
title: Datadog
weight: 500
---
# Datadog integration for Grafana OnCall
The Datadog integration for Grafana OnCall handles ticket events sent from Datadog webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Datadog
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Datadog** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring Datadog to Send Alerts to Grafana OnCall
1. Navigate to the Integrations page from the sidebar
2. Search for webhook in the search bar
3. Enter a name for the integration, for example: grafana-oncall-alerts
4. Paste the **OnCall Integration URL**, then save
5. Navigate to the Events page from the sidebar to send the test alert
6. Type @webhook-grafana-oncall-alerts test alert
7. Click the post button

76
docs/sources/integrations/elastalert/index.md Normal file
View file

@ -0,0 +1,76 @@
---
aliases:
- add-elastalert/
- /docs/oncall/latest/integrations/available-integrations/configure-elastalert/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-elastalert/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- elastalert
title: ElastAlert
weight: 500
---
# ElastAlert integration for Grafana OnCall
The ElastAlert integration for Grafana OnCall handles ticket events sent from ElastAlert webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from ElastAlert
1. In the **Integrations** tab, click **+ New integration**.
2. Select **ElastAlert** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring ElastAlert to Send Alerts to Grafana OnCall
To send an alert from ElastAlert to a webhook, follow these steps:
> Refer to [ElastAlert http-post docs](https://elastalert.readthedocs.io/en/latest/ruletypes.html#http-post) for more details
1. Open your ElastAlert configuration file (e.g., `config.yaml`).
2. Locate the `alert` section.
3. Add the following configuration for the webhook alert:
```yaml
alert: post
http_post_url: "http://example.com/api"
http_post_static_payload:
title: abc123
```
Replace `"abc123"` with a suitable name for your alert, and `"http://example.com/api"` with **OnCall Integration URL**.
4. Save the configuration file.
After configuring the webhook, ElastAlert will send alerts to the specified endpoint when triggered.
Make sure your webhook endpoint is configured to receive and process the incoming alerts.
## Grouping, auto-acknowledge and auto-resolve
Grafana OnCall provides grouping, auto-acknowledge and auto-resolve logic for the ElastAlert integration:
- Alerts created from ticket events are grouped by ticket ID
- Alert groups are auto-acknowledged when the ticket status is set to "Pending"
- Alert groups are auto-resolved when the ticket status is set to "Solved"
To customize this behaviour, consider modifying alert templates in integration settings.
### Configuring Elastalert to send heartbeats to Grafana OnCall Heartbeat
Add the following rule to ElastAlert
```yaml
index: elastalert_status
type: any
alert: post
http_post_url: {{ heartbeat_url }}
realert:
minutes: 1
alert_text: elastalert is still running
alert_text_type: alert_text_only
```

37
docs/sources/integrations/fabric/index.md Normal file
View file

@ -0,0 +1,37 @@
---
aliases:
- add-fabric/
- /docs/oncall/latest/integrations/available-integrations/configure-fabric/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-fabric/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- fabric
title: Fabric
weight: 500
---
# Fabric integration for Grafana OnCall
The Fabric integration for Grafana OnCall handles ticket events sent from Fabric webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Fabric
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Fabric** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring Fabric to Send Alerts to Grafana OnCall
1. Go to <https://www.fabric.io/settings/apps>
2. Choose your application
3. Navigate to Service Hooks -> WebHook
4. Enter URL: **OnCall Integration URL**
5. Click Verify
6. Choose "SEND IMPACT CHANGE ALERTS" and "ALSO SEND NON-FATAL ALERTS"

3
docs/sources/integrations/grafana-alerting/index.md
View file

@ -17,8 +17,7 @@ weight: 100
Grafana Alerting for Grafana OnCall can be set up using two methods:
- Grafana Alerting: Grafana OnCall is connected to the same Grafana instance being used to manage Grafana OnCall.
- Grafana (Other Grafana): Grafana OnCall is connected to one or more Grafana instances separate from the one being
used to manage Grafana OnCall.
- Grafana (Other Grafana): Grafana OnCall is connected to one or more Grafana instances, separate from the one being used to manage Grafana OnCall.
## Configure Grafana Alerting for Grafana OnCall

4
docs/sources/integrations/inbound-email/index.md
View file

@ -22,14 +22,14 @@ You must have an Admin role to create integrations in Grafana OnCall.
1. In the **Integrations** tab, click **+ New integration to receive alerts**.
2. Select **Inbound Email** from the list of available integrations.
3. Get your dedicated email address in the **How to connect** window.
3. Get your dedicated email address in the **Integration email** section and use it to send your emails.
## Grouping and auto-resolve
Alert groups will be grouped by email subject and auto-resolved if the email message text equals "OK".
This behaviour can be modified via [custom templates]({{< relref "jinja2-templating" >}}).
Alerts from Inbound Email integration have followng payload:
Alerts from Inbound Email integration have the following payload:
```json
{

22
docs/sources/integrations/jira/index.md
View file

@ -18,13 +18,27 @@ weight: 500
The Jira integration for Grafana OnCall handles issue events sent from Jira webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
## Configure Jira integration for Grafana OnCall
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
You must have an Admin role to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Jira
1. In the **Integrations** tab, click **+ New integration to receive alerts**.
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Jira** from the list of available integrations.
3. Follow the instructions in the **How to connect** window to get your unique integration URL and review next steps.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section. You will need it when configuring Jira.
## Configuring Jira to Send Alerts to Grafana OnCall
Create a new webhook connection in Jira to send events to Grafana OnCall using the integration URL above.
Refer to [Jira documentation](https://developer.atlassian.com/server/jira/platform/webhooks/) for more information on how to create and manage webhooks
When creating a webhook in Jira, select the following events to be sent to Grafana OnCall:
1. Issue - created
2. Issue - updated
3. Issue - deleted
After setting up the connection, you can test it by creating a new issue in Jira. You should see a new alert group in Grafana OnCall.
## Grouping, auto-acknowledge and auto-resolve

61
docs/sources/integrations/kapacitor/index.md Normal file
View file

@ -0,0 +1,61 @@
---
aliases:
- add-kapacitor/
- /docs/oncall/latest/integrations/available-integrations/configure-kapacitor/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-kapacitor/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- kapacitor
title: Kapacitor
weight: 500
---
# Kapacitor integration for Grafana OnCall
The Kapacitor integration for Grafana OnCall handles ticket events sent from Kapacitor webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Kapacitor
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Kapacitor** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring Kapacitor to Send Alerts to Grafana OnCall
To send an alert from Kapacitor, you can follow these steps:
1. Create a Kapacitor TICKscript or modify an existing one to define the conditions for triggering the alert.
The TICKscript specifies the data source, data processing, and the alert rule. Here's an example of a simple TICKscript:
```tickscript
stream
|from()
.measurement('measurement_name')
.where(lambda: <condition>)
|alert()
.webhook('<webhook_url>')
```
Replace `'measurement_name'` with the name of the measurement you want to monitor, `<condition>`
with the condition that triggers the alert, and `'<webhook_url>'` with **OnCall Integration URL**
2. Save the TICKscript file with a `.tick` extension, for example, `alert_script.tick`.
3. Start the Kapacitor service using the TICKscript:
```bash
kapacitor define <alert_name> -tick /path/to/alert_script.tick
kapacitor enable <alert_name>
kapacitor reload
```
Replace `<alert_name>` with a suitable name for your alert.
4. Ensure that the Kapacitor service is running and actively monitoring the data.
When the condition defined in the TICKscript is met, Kapacitor will trigger the alert and send
a POST request to the specified webhook URL with the necessary information. Make sure your webhook
endpoint is configured to receive and process the incoming alerts from Kapacitor.

36
docs/sources/integrations/newrelic/index.md Normal file
View file

@ -0,0 +1,36 @@
---
aliases:
- add-newrelic/
- /docs/oncall/latest/integrations/available-integrations/configure-newrelic/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-newrelic/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- newrelic
title: New Relic
weight: 500
---
# New Relic integration for Grafana OnCall
The New Relic integration for Grafana OnCall handles ticket events sent from New Relic webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from New Relic
1. In the **Integrations** tab, click **+ New integration**.
2. Select **New Relic** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring New Relic to Send Alerts to Grafana OnCall
1. Go to "Alerts".
2. Go to "Notification Channels".
3. Create "Webhook" notification channel.
4. Set the following URL: **OnCall Integration URL**
5. Check "Payload type" is JSON.

37
docs/sources/integrations/pingdom/index.md Normal file
View file

@ -0,0 +1,37 @@
---
aliases:
- add-pingdom/
- /docs/oncall/latest/integrations/available-integrations/configure-pingdom/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-pingdom/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- pingdom
title: Pingdom
weight: 500
---
# Pingdom integration for Grafana OnCall
The Pingdom integration for Grafana OnCall handles ticket events sent from Pingdom webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Pingdom
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Pingdom** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring Pingdom to Send Alerts to Grafana OnCall
1. Go to <https://my.pingdom.com/integrations/settings>
2. Click "Add Integration".
3. Type: Webhook. Name: `Grafana OnCall`. URL: **OnCall Integration URL**
4. Go to "Reports" -> "Uptime" -> "Edit Check".
5. Select `Grafana OnCall` integration in the bottom.
6. Click "Modify Check" to save.

113
docs/sources/integrations/prtg/index.md Normal file
View file

@ -0,0 +1,113 @@
---
aliases:
- add-prtg/
- /docs/oncall/latest/integrations/available-integrations/configure-prtg/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-prtg/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- prtg
title: PRTG
weight: 500
---
# PRTG integration for Grafana OnCall
The PRTG integration for Grafana OnCall handles ticket events sent from PRTG webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from PRTG
1. In the **Integrations** tab, click **+ New integration**.
2. Select **PRTG** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring PRTG to Send Alerts to Grafana OnCall
PRTG can use the script to send the alerts to Grafana OnCall. Please use the format below
Body Fields Format:
```plaintext
alert_uid [char][not required] - unique alert ID for grouping;
title [char][not required] - title;
image_url [char][not required] - url for image attached to alert;
state [char][not required] - could be "ok" or "alerting", helpful for auto-resolving;
link_to_upstream_details [char][not required] - link back to your monitoring system;
message [char][not required] - alert details;
```
ps1 script example:
```ps1
# This script sends alerts from PRTG to Grafana OnCall
Param(
[string]$sensorid,
[string]$date,
[string]$device,
[string]$shortname,
[string]$status,
[string]$message,
[string]$datetime,
[string]$linksensor,
[string]$url
)
# PRTG Server
$PRTGServer = "localhost:8080"
$PRTGUsername = "amixr"
$PRTGPasshash = *****
#Directory for logging
$LogDirectory = "C:\temp\prtg-notifications-msteam.log"
#Acknowledgement Message for alerts ack'd via Teams
$ackmessage = "Problem has been acknowledged via Amixr."
# the acknowledgement URL
$ackURL = [string]::Format("{0}/api/acknowledgealarm.htm?id={1}&ackmsg={2}&username={3}&passhash={4}",
$PRTGServer,$sensorID,$ackmessage,$PRTGUsername,$PRTGPasshash);
# Autoresolve an alert in Amixr
if($status -eq "Up")
{ $state = "ok" }
ElseIf($status -match "now: Up")
{ $state = "ok" }
ElseIf($status -match "Up (was:")
{ $state = "ok" }
Else
{ $state = "alerting" }
$image_datetime = [datetime]::parse($datetime)
$sdate = $image_datetime.AddHours(-1).ToString("yyyy-MM-dd-HH-mm-ss")
$edate = $image_datetime.ToString("yyyy-MM-dd-HH-mm-ss")
$image_url = "$PRTGServer/chart.png?type=graph&graphid=-1&avg=0&width=1000&height=400
&username=$PRTGUsername&passhash=$PRTGPasshash&id=$sensorid&sdate=$sdate&edate=$edate"
$Body = @{
"alert_uid"="$sensorid $date";
"title"="$device $shortname $status at $datetime ";
"image_url"=$image_url;
"state"=$state;
"link_to_upstream_details"="$linksensor";
"message"="$message";
"ack_url_get"="$ackURL"
} | ConvertTo-Json
$Body
try
{ Invoke-RestMethod -uri $url -Method Post -body $Body -ContentType 'application/json; charset=utf-8'; exit 0; }
Catch
{
$ErrorMessage = $_.Exception.Message
(Get-Date).ToString() +" - "+ $ErrorMessage | Out-File -FilePath $LogDirectory -Append
exit 2;
}
```

56
docs/sources/integrations/sentry/index.md Normal file
View file

@ -0,0 +1,56 @@
---
aliases:
- add-sentry/
- /docs/oncall/latest/integrations/available-integrations/configure-Sentry/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-sentry/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- sentry
title: Sentry
weight: 500
---
# Sentry integration for Grafana OnCall
The Sentry integration for Grafana OnCall handles ticket events sent from Sentry webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Sentry
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Sentry** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring Sentry to Send Alerts to Grafana OnCall
To send a webhook alert from Sentry, you can follow these steps:
1. Log in to your Sentry account.
2. Navigate to your project's settings.
3. Click on "Alerts" in the sidebar menu.
4. Click on "New Alert Rule" to create a new alert rule.
5. Configure the conditions for the alert rule based on your requirements. For example, you can set conditions based on issue
level, event frequency, or specific tags.
6. In the "Actions" section, select "Webhook" as the action type.
7. Provide the necessary details for the webhook:
- **URL**: **OnCall Integration URL**
- **Method**: POST
- **Payload**: Define the payload structure and content that you want to send to the webhook endpoint. You can use Sentry's
dynamic variables to include relevant information in the payload.
8. Save the alert rule.
Once the alert conditions are met, Sentry will trigger the webhook action and send a request to the Grafana OnCall.

34
docs/sources/integrations/stackdriver/index.md Normal file
View file

@ -0,0 +1,34 @@
---
aliases:
- add-stackdriver/
- /docs/oncall/latest/integrations/available-integrations/configure-stackdriver/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-stackdriver/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- stackdriver
title: Stackdriver
weight: 500
---
# Stackdriver integration for Grafana OnCall
The Stackdriver integration for Grafana OnCall handles ticket events sent from Stackdriver webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Stackdriver
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Stackdriver** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring Stackdriver to Send Alerts to Grafana OnCall
1. Create a notification channel in Stackdriver by navigating to Workspace Settings -> WEBHOOKS -> Add Webhook **OnCall Integration URL**
2. Create and alert in Stackdriver by navigating to Alerting -> Policies -> Add Policy -> Choose Notification Channel using the channel set up in step 1

61
docs/sources/integrations/uptimerobot/index.md Normal file
View file

@ -0,0 +1,61 @@
---
aliases:
- add-uptimerobot/
- /docs/oncall/latest/integrations/available-integrations/configure-uptimerobot/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-uptimerobot/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- uptimerobot
title: UptimeRobot
weight: 500
---
# UptimeRobot integration for Grafana OnCall
The UptimeRobot integration for Grafana OnCall handles ticket events sent from UptimeRobot webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from UptimeRobot
1. In the **Integrations** tab, click **+ New integration**.
2. Select **UptimeRobot** from the list of available integrations.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring UptimeRobot to Send Alerts to Grafana OnCall
1. Open <https://uptimerobot.com> and log in
1. Go to My Settings > Add Alert Contact and set the following fields:
1. Alert Contact Type: Webhook
1. Friendly Name: Grafana OnCall
1. URL to Notify: **OnCall Integration URL**
POST Value (JSON Format):
```yaml
{
"monitorURL": "monitorURL",
"monitorFriendlyName": "monitorFriendlyName",
"alertType": "alertType",
"alertTypeFriendlyName": "alertTypeFriendlyName",
"alertDetails": "alertDetails",
"alertDuration": "alertDuration",
"sslExpiryDate": "sslExpiryDate",
"sslExpiryDaysLeft": "sslExpiryDaysLeft"
}
```
1. Flag Send as JSON
1. Click Save Changes and Close
1. Send Test Alert to Grafana OnCall
1. Click Add New Monitor
1. Monitor Type HTTP(s)
1. Friendly Name Test Amixr
1. Set URL to <http://devnull.amixr.io> or any other non-existent domain
1. Click Checkbox next to Amixr Alert Contact (created in the previous step)
1. Click Create Monitor

37
docs/sources/integrations/zendesk/index.md
View file

@ -18,13 +18,42 @@ weight: 500
The Zendesk integration for Grafana OnCall handles ticket events sent from Zendesk webhooks.
The integration provides grouping, auto-acknowledge and auto-resolve logic via customizable alert templates.
## Configure Zendesk integration for Grafana OnCall
> You must have the [role of Admin]({{< relref "user-and-team-management" >}}) to be able to create integrations in Grafana OnCall.
You must have an Admin role to create integrations in Grafana OnCall.
## Configuring Grafana OnCall to Receive Alerts from Zendesk
1. In the **Integrations** tab, click **+ New integration to receive alerts**.
1. In the **Integrations** tab, click **+ New integration**.
2. Select **Zendesk** from the list of available integrations.
3. Follow the instructions in the **How to connect** window to get your unique integration URL and review next steps.
3. Enter a name and description for the integration, click **Create**
4. A new page will open with the integration details. Copy the **OnCall Integration URL** from **HTTP Endpoint** section.
## Configuring Zendesk to Send Alerts to Grafana OnCall
Create a new "Trigger or automation" webhook connection in Zendesk to send events to Grafana OnCall using the integration URL above.
Refer to [Zendesk documentation]
(<https://support.zendesk.com/hc/en-us/articles/4408839108378-Creating-webhooks-to-interact-with-third-party-systems>
) for more information on how to create and manage webhooks.
After setting up a webhook in Zendesk, create a new trigger with the following condition:
`Meet ANY of the following conditions: "Ticket Is Created", "Ticket status Changed"`
Set `Notify webhook` as the trigger action and select the webhook you created earlier.
In the JSON body field, use the following JSON template:
```json
{
"ticket": {
"id": "{{ticket.id}}",
"url": "{{ticket.url}}",
"status": "{{ticket.status}}",
"title": "{{ticket.title}}",
"description": "{{ticket.description}}"
}
}
```
After setting up the connection, you can test it by creating a new ticket in Zendesk. You should see a new alert group in Grafana OnCall.
## Grouping, auto-acknowledge and auto-resolve

131
docs/sources/jinja2-templating/_index.md
View file

@ -6,10 +6,12 @@ weight: 1000
## Jinja2 templating
Grafana OnCall can integrate with any monitoring systems that can send alerts using webhooks with JSON payloads. By
default, webhooks deliver raw JSON payloads. When Grafana OnCall receives an alert and parses its payload, a default
pre-configured alert template is applied to modify the alert payload to be more human-readable. These alert templates
are customizable for any integration.
Grafana OnCall can integrate with any monitoring system that can send alerts via
webhooks with JSON payloads. By default, webhooks deliver raw JSON payloads. When Grafana
OnCall receives an alert and parses its payload, a default pre-configured alert template
is applied to modify the alert payload to be more human-readable. These alert templates
are customizable for any integration. Templates are also used to notify different
escalation chains based on the content of the alert payload.
<iframe width="560" height="315" src="https://www.youtube.com/embed/S6Is8hhyCos" title="YouTube video player"
frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture;
@ -17,8 +19,9 @@ web-share" allowfullscreen></iframe>
## Alert payload
Alerts received by Grafana OnCall contain metadata as keys and values in a JSON object. The following is an example of
an alert received by Grafana OnCall initiated by Grafana Alerting:
Alerts received by Grafana OnCall contain metadata as keys and values in a JSON object.
The following is an example of an alert which was initiated by Grafana Alerting, and
received by Grafana OnCall:
```json
{
@ -45,67 +48,106 @@ an alert received by Grafana OnCall initiated by Grafana Alerting:
}
```
In Grafana OnCall every alert and alert group has the following fields:
In Grafana OnCall every alert and alert group have the following fields:
- `Title`, `message` and `image url`
- `Grouping Id`
- `Resolve Signal`
- `Title`, `Message` and `Image Url` for each notification method (Web, Slack, Ms Teams, SMS, Phone, Email, etc.)
- `Grouping Id` - unique identifier for each non-resolved alert group
- `Resolved by source`
- `Acknowledged by source`
- `Source link`
The JSON payload is converted. For example:
The JSON payload is converted to OnCall fields. For example:
- `{{ payload.title }}` -> Title
- `{{ payload.message }}` -> Message
- `{{ payload.imageUrl }}` -> Image Url
- `{{ payload.title }}` -> `Title`
- `{{ payload.message }}` -> `Message`
- `{{ payload.imageUrl }}` -> `Image Url`
The result is that each field of the alert in OnCall is now mapped to the JSON payload keys. This also true for the
The result is that each field of the alert in OnCall is now mapped to the JSON payload
keys. This also true for the
alert behavior:
- `{{ payload.ruleId }}` -> Grouping Id
- `{{ 1 if payload.state == 'OK' else 0 }}` -> Resolve Signal
Grafana OnCall provides a pre configured default Jinja template for supported integrations. If your monitoring system is
not in the Grafana OnCall integrations list, you can create a generic `webhook` integration, send an alert, and configure
Grafana OnCall provides pre-configured default Jinja templates for supported
integrations. If your monitoring system is
not in the Grafana OnCall integrations list, you can create a generic `webhook`
integration, send an alert, and configure
your templates.
## Customize alerts with alert templates
## Types of templates
Alert templates allow you to format any alert fields recognized by Grafana OnCall. You can customize default alert
templates for all the different ways you receive your alerts such as web, slack, SMS, and email. For more advanced
Alert templates allow you to format any alert fields recognized by Grafana OnCall. You can
customize default alert
templates for all the different notification methods. For more advanced
customization, use Jinja templates.
As a best practice, add _Playbooks_, _Useful links_, or _Checklists_ to the alert message.
### Routing template
To customize alert templates in Grafana OnCall:
- `Routing Template` - used to route alerts to different Escalation Chains based on alert content (conditional template, output should be `True`)
1. Navigate to the **Integrations** tab, select the integration, then click **Change alert template and grouping**.
> **Note:** For conditional templates, the output should be `True` to be applied, for example `{{ True if payload.state == 'OK' else False }}`
2. In Alert Templates, select a template from the **Edit template for** dropdown.
#### Appearance templates
3. Edit the Appearances template as needed:
How alerts are displayed in the UI, messengers, and notifications
- `Title`, `Message`, `Image url` for Web
- `Title`, `Message`, `Image url` for Slack
- `Title` used for SMS
- `Title` used for Phone
- `Title`, `Message` used for Email
- `Title`, `Message`, `Image url` for Web
- `Title`, `Message`, `Image url` for Slack
- `Title`, `Message`, `Image url` for MS Teams
- `Title`, `Message`, `Image url` for Telegram
- `Title` for SMS
- `Title` for Phone Call
- `Title`, `Message` for Email
4. Edit the alert behavior as needed:
- `Grouping Id` - Alerts with the same `Grouping Id` will be grouped into the same Alert Group
if Alert Group in the state "Firing", "Acked" or "Silenced" exists. If previous Alert Group is in the state "Resolved",
a new Alert Group will be issued.
- `Acknowledge Condition` - The output should be `ok`, `true`, or `1` to auto-acknowledge the alert group.
For example, `{{ 1 if payload.state == 'OK' else 0 }}`.
- `Resolve Condition` - The output should be `ok`, `true` or `1` to auto-resolve the alert group.
For example, `{{ 1 if payload.state == 'OK' else 0 }}`.
- `Source Link` - Used to customize the URL link to provide as the "source" of the alert.
#### Behavioral templates
- `Grouping Id` - applied to every incoming alert payload after the `Routing Template`. It
can be based on time, alert content, or both. If the resulting grouping id matches an
existing non-resolved alert group grouping id, the alert will be grouped accordingly.
Otherwise, a new alert group will be created
- `Autoresolution` - used to auto-resolve alert groups with status `Resolved by source`
(Conditional template, output should be `True`)
- `Auto acknowledge` - used to auto-acknowledge alert groups with status `Acknowledged by
source` (Conditional template, output should be `True`)
- `Source link` - Used to customize the URL link to provide as the "source" of the alert.
> **Note:** For conditional templates, the output should be `True` to be applied, for
example `{{ True if payload.state == 'OK' else False }}`
> **Pro Tip:** As a best practice, add _Playbooks_, _Useful links_, or _Checklists_ to the
alert message.
#### How to edit templates
1. Open the **Integration** page for the integration you want to edit
1`. Click the **Edit** button for the Templates Section. Now you can see previews of all
templates for the Integration
1. Select the template you want to edit and click the **Edit** button to the right to the template
name. The template editor will open. The first column is the example alert payload, second
column is the Template itself, and third column is used to view rendered result.
1. Select one of the **Recent Alert groups** for the integration to see its `latest alert
payload`. If you want to edit this payload, click the **Edit** button right to the Alert Group
Name.
1. Alternatively, you can click **Use custom payload** and write your own payload to see
how it will be rendered
1. Press `Control + Enter` in the editor to see suggestions
1. Click **Cheatsheet** in the second column to get some inspiration.
1. If you edit Messenger templates, click **Save and open Alert Group in ChatOps** to see
how the alert will be rendered in the messenger, right in the messenger (Only works for
an Alert Group that exists in the messenger)
1. Click **Save** to save the template
## Advanced Jinja templates
Grafana OnCall uses [Jinja templating language](http://jinja.pocoo.org/docs/2.10/) to format alert groups for the Web,
Slack, phone calls, SMS messages, and more because the JSON format is not easily readable by humans. As a result, you
can decide what you want to see when an alert group is triggered as well as how it should be presented.
Grafana OnCall uses the [Jinja templating language](http://jinja.pocoo.org/docs/2.10/) to
format alert groups for the Web,
Slack, phone calls, SMS messages, and more. As a result, you
can decide what you want to see when an alert group is triggered, as well as how it should
be presented.
Jinja2 offers simple but multi-faceted functionality by using loops, conditions, functions, and more.
Jinja2 offers simple but multi-faceted functionality by using loops, conditions,
functions, and more.
> **NOTE:** Every alert from a monitoring system comes in the key/value format.
@ -113,7 +155,8 @@ Grafana OnCall has rules about which of the keys match to: `__title`, `message`,
### Loops
Monitoring systems can send an array of values. In this example, you can use Jinja to iterate and format the alert
Monitoring systems can send an array of values. In this example, you can use Jinja to
iterate and format the alert
using a Grafana example:
```.jinja2

5
grafana-plugin/src/containers/IntegrationContainers/IntegrationCommonTemplatesList.config.ts
View file

@ -19,15 +19,14 @@ export const commonTemplatesToRender: TemplateBlock[] = [
{
name: 'grouping_id_template',
label: 'Grouping',
labelTooltip:
'The Grouping Template is applied to every incoming alert payload after the Routing Template. It can be based on time, or alert content, or both. If the resulting grouping key matches an existing non-resolved alert group, the alert will be grouped accordingly. Otherwise, a new alert group will be created',
labelTooltip: 'Alerts with the same Grouping Id will be grouped together. See docs for more information',
height: MONACO_INPUT_HEIGHT_SMALL,
},
{
name: 'resolve_condition_template',
label: 'Autoresolution',
labelTooltip:
'If Autoresolution Template is True, the alert will resolve its group as "resolved by source". If the group is already resolved, the alert will be added to that group',
'If Autoresolution Template is True, the alert will resolve its group as "resolved by source", See docs for more information',
height: MONACO_INPUT_HEIGHT_SMALL,
},
],