# What this PR does The `docs/reference` shortcode supports contextual destinations and version inference. `<ONCALL VERSION>` is inferred to match the version of the documentation set. For example, the inferred version for the page /docs/grafana/oncall/latest/get-started/ is "latest". It can also be overriden using front matter. Given the same page, but with the additional front matter `oncall_version: next`, the variable is substituted with "next" rather than "latest". Contextual destinations are achieved using repeated labels in the shortcode inner text. The format is [<LABEL>]: "<PAGE PATH PREFIX> -> <HUGO REFERENCE>". - _`<LABEL>`_ matches the reference style link label used in the rest of the text. - _`<PAGE PATH PREFIX>`_ is matched against the page during the production build. If the match is successful, the destination that is used is _`<HUGO REFERENCE>`_. The first matching prefix is used, not the longest matching prefix. ## Which issue(s) this PR fixes - Broken links due to ambiguous relref resolution. Any relref parameter that does not start with either `/`, `./`, or `../` can resolve ambiguously and is resulting in broken link behavior on the current site. - Broken links in Grafana Cloud. We mount OnCall documentation in Grafana Cloud. In https://github.com/grafana/website/pull/13872 the location will become /docs/grafana-cloud/alerting-and-irm/oncall. This PR is intended to be merged alongside that PR. --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
10 KiB
| aliases | canonical | keywords | title | weight | ||||
|---|---|---|---|---|---|---|---|---|
|
https://grafana.com/docs/oncall/latest/get-started/ |
|
Get started with Grafana OnCall | 300 |
Get started with Grafana OnCall
Grafana OnCall is an incident response tool built to help DevOps and SRE teams improve their collaboration, and resolve incidents faster.
With a centralized view of all your alerts and alert groups, automated escalations and grouping, and on-call scheduling, Grafana OnCall helps ensure that alert notifications reach the right people, at the right time using the right notification method.
The following diagram details an example alert workflow with Grafana OnCall:
These procedures introduce you to initial Grafana OnCall configuration steps, including monitoring system integration, how to set up escalation chains, and how to set up calendar for on-call scheduling.
Grafana Cloud OnCall vs Open Source Grafana OnCall
Grafana OnCall is available both in Grafana Cloud and Grafana Open Source.
OnCall is available in Grafana Cloud automatically:
- Create or log in into Grafana Cloud account
- Sign in to your Grafana stack
- Choose Alerts and IRM from the left menu
- Click OnCall to access Grafana OnCall
Otherwise you'll need to install [Open Source Grafana OnCall][open-source] on your own.
How to configure Grafana OnCall
- Users with [Admin role][user-and-team-management] can configure Alert rules (Integrations, Routes, etc) to define when and which users to notify
- OnCall users with [Editor role][user-and-team-management] can work with Alerts Groups and set up personal settings, e.g. how to notify.
Note: If your role is Editor, you can skip to Learn Alert Workflow section of this doc
Get alerts into Grafana OnCall and configure rules
Once you’ve installed Grafana OnCall, or accessed it from your Grafana Cloud instance, you can begin integrating with monitoring systems to get alerts into Grafana OnCall. Additionally, you can configure when, and which, users get notified, by setting templates, routes, escalation chains, etc.
Integrate with a monitoring system
Regardless of where your alerts originate, you can send them to Grafana OnCall via available integrations or customizable webhooks. To start receiving alerts in Grafana OnCall, use the following steps to configure your first integration and send a demo alert.
Configure your first integration
- In Grafana OnCall, navigate to the Integrations tab and click + New integration.
- Select an integration from the provided options, if the integration you’re looking for isn’t listed, then select Webhook.
- Click How to connect to view the instructions specific to your monitoring system
Send a demo alert
- In the integration tab, click Send demo alert, review and modify the alert payload as needed, and click Send
- Navigate to the Alert Groups tab to see your test alert firing
- Explore the Alert Group by clicking on the title
- Acknowledge and resolve the test alert group
For more information on Grafana OnCall integrations and further configuration guidance, refer to [Grafana OnCall integrations][integrations]
Review and modify alert templates
Review and customize templates to interpret monitoring alerts and minimize noise. Group alerts, enable auto-resolution, customize visualizations and notifications by extracting data from alerts. See more details in the [Jinja2 templating][jinja2-templating] section.
Configure scalation Chains
Escalation chains are a set of steps that define who to notify, and when.
See more details in the [Escalation Chains][escalation-chains] section.
Escalation Chains are customizable automated alert routing steps that enable you to specify who is notified for a certain alert. In addition to escalation chains, you can configure Routes to send alerts to different escalation chains depending on the alert details.
Once your integration is configured, you can set up an escalation chain to determine how alerts from your integration are handled. Multi-step escalation chains help ensure thorough alert escalation to prevent alerts from being missed.
To configure Escalation Chains:
- Navigate to the Escalation Chains tab and click + New Escalation Chain
- Give your Escalation Chain a useful name and click Create
- Add a series of escalation steps from the available dropdown options.
- To link your Escalation Chain to your integration, navigate back to the Integrations tab, Select your newly created Escalation Chain from the “Escalate to” dropdown.
Alerts from this integration will now follow the escalation steps configured in your Escalation Chain.
For more information on Escalation Chains and more ways to customize them, refer to
[Configure and manage Escalation Chains][escalation-chains-and-routes]
Routes define which messenger channels and escalation chains to use for notifications. See more details in the [Routes][routes] section.
Learn about the Alert Workflow
-
All Alerts in OnCall are grouped into Alert Groups ([read more about Grouping ID][jinja2-templating]). An Alert Group can have the following, mutually exclusive states:
-
Firing: Once Alert Group is registered, Escalation Policy associated with it is getting started. Escalation policy will work while Alert Group is in this status.
-
Acknowledged: Ongoing Escalation Chain will be interrupted. Unacknowledge will move Alert Group to the "Firing" state and will re-launch Escalation Chain.
-
Silenced: Similar to "Acknowledged" but designed to be temporary with a timeout. Once time is out, will re-launch Escalation Chain and move Alert Group to the "Firing" state.
-
Resolved: Similar to "Acknowledged".
-
Possible transitions:
-
Firing -> Acknowledged
-
Firing -> Silenced
-
Firing -> Resolved
-
Silenced -> Firing
-
Silenced -> Acknowledged
-
Silenced -> Resolved
-
Acknowledged -> Silenced
-
Acknowledged -> Firing
-
Acknowledged -> Resolved
-
Resolved -> Firing
Transition changes trigger Escalation Chains to launch, with a few-second delay (to avoid unexpected notifications).
Get notified of an alert
In order for Grafana OnCall to notify you of an alert, you must configure how you want to be notified. Personal notification policies, chatops integrations, and on-call schedules allow you to automate how users are notified of alerts.
Configure personal notification policies
Personal notification policies determine how a user is notified for a certain type of alert. Get notified by SMS, phone call, Slack mentions, or mobile push notification. Administrators can configure how users receive notifications for certain types of alerts. For more information on personal notification policies, refer to [Manage users and teams for Grafana OnCall][user-and-team-management]
To configure users personal notification policies:
- Navigate to the Users tab in Grafana OnCall
- Select a user from the user list and click Edit
- Configure Default Notifications and Important Notification
Configure Slack for Grafana OnCall
Grafana OnCall integrates closely with your Slack workspace to deliver alert notifications to individuals, user groups, and channels. Slack notifications can be triggered by steps in an escalation chain or as a step in users personal notification policies.
To configure Slack for Grafana OnCall:
- In OnCall, click on the ChatOps tab and select Slack in the side menu.
- Click Install Slack integration.
- Read the notice and confirm to proceed to the Slack website.
- Sign in to your organization's Slack workspace.
- Click Allow to allow Grafana OnCall to access Slack.
- Ensure users verify their Slack accounts in their user profile in Grafana OnCall.
For further instruction on connecting to your Slack workspace, refer to [Slack integration for Grafana OnCall][slack]
Grafana OnCall also supports other ChatOps integration like [Microsoft Teams and Telegram][notify].
Add your on-call schedule
Grafana OnCall allows you to manage your on-call schedule in your preferred calendar app such as Google Calendar or Microsoft Outlook.
To integrate your on-call calendar with Grafana OnCall:
- In the Schedules tab of Grafana OnCall, click + Add team schedule for on-call rotation.
- Provide a schedule name.
- Configure the rest of the schedule settings and click Create Schedule
[More information on on-call schedules.][on-call-schedules]
{{% docs/reference %}} [escalation-chains-and-routes]: "/docs/oncall/ -> /docs/oncall//escalation-chains-and-routes" [escalation-chains-and-routes]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes"
[escalation-chains]: "/docs/oncall/ -> /docs/oncall//escalation-chains-and-routes#escalation-chains" [escalation-chains]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes#escalation-chains"
[integrations]: "/docs/oncall/ -> /docs/oncall//integrations" [integrations]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations"
[jinja2-templating]: "/docs/oncall/ -> /docs/oncall//jinja2-templating" [jinja2-templating]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/jinja2-templating"
[notify]: "/docs/oncall/ -> /docs/oncall//notify" [notify]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/notify"
[on-call-schedules]: "/docs/oncall/ -> /docs/oncall//on-call-schedules" [on-call-schedules]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/on-call-schedules"
[open-source]: "/docs/oncall/ -> /docs/oncall//open-source" [open-source]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/open-source"
[routes]: "/docs/oncall/ -> /docs/oncall//escalation-chains-and-routes#routes" [routes]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes#routes"
[slack]: "/docs/oncall/ -> /docs/oncall//notify/slack" [slack]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/notify/slack"
[user-and-team-management]: "/docs/oncall/ -> /docs/oncall//user-and-team-management" [user-and-team-management]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/user-and-team-management" {{% /docs/reference %}}