oncall-engine/docs/sources/get-started/_index.md
Jack Baldry 1735db4c3b
Update all links to use docs/reference style (#2456)
# What this PR does

The `docs/reference` shortcode supports contextual destinations and
version inference.

`<ONCALL VERSION>` is inferred to match the version of the documentation
set. For example, the inferred version for the page
/docs/grafana/oncall/latest/get-started/ is "latest". It can also be
overriden using front matter.
Given the same page, but with the additional front matter
`oncall_version: next`, the variable is substituted with "next" rather
than "latest".

Contextual destinations are achieved using repeated labels in the
shortcode inner text. The format is [<LABEL>]: "<PAGE PATH PREFIX> ->
<HUGO REFERENCE>".

- _`<LABEL>`_ matches the reference style link label used in the rest of
the text.
- _`<PAGE PATH PREFIX>`_ is matched against the page during the
production build. If the match is successful, the destination that is
used is _`<HUGO REFERENCE>`_. The first matching prefix is used, not the
longest matching prefix.

## Which issue(s) this PR fixes

- Broken links due to ambiguous relref resolution. Any relref parameter
that does not start with either `/`, `./`, or `../` can resolve
ambiguously and is resulting in broken link behavior on the current
site.
- Broken links in Grafana Cloud. We mount OnCall documentation in
Grafana Cloud. In https://github.com/grafana/website/pull/13872 the
location will become /docs/grafana-cloud/alerting-and-irm/oncall. This
PR is intended to be merged alongside that PR.

---------

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
2023-07-13 14:38:35 +00:00

10 KiB
Raw Blame History

aliases canonical keywords title weight
/getting-started/
https://grafana.com/docs/oncall/latest/get-started/
Get started
On call
Grafana Cloud
Get started with Grafana OnCall 300

Get started with Grafana OnCall

Grafana OnCall is an incident response tool built to help DevOps and SRE teams improve their collaboration, and resolve incidents faster.

With a centralized view of all your alerts and alert groups, automated escalations and grouping, and on-call scheduling, Grafana OnCall helps ensure that alert notifications reach the right people, at the right time using the right notification method.

The following diagram details an example alert workflow with Grafana OnCall:

These procedures introduce you to initial Grafana OnCall configuration steps, including monitoring system integration, how to set up escalation chains, and how to set up calendar for on-call scheduling.

Grafana Cloud OnCall vs Open Source Grafana OnCall

Grafana OnCall is available both in Grafana Cloud and Grafana Open Source.

OnCall is available in Grafana Cloud automatically:

  1. Create or log in into Grafana Cloud account
  2. Sign in to your Grafana stack
  3. Choose Alerts and IRM from the left menu
  4. Click OnCall to access Grafana OnCall

Otherwise you'll need to install [Open Source Grafana OnCall][open-source] on your own.

How to configure Grafana OnCall

  • Users with [Admin role][user-and-team-management] can configure Alert rules (Integrations, Routes, etc) to define when and which users to notify
  • OnCall users with [Editor role][user-and-team-management] can work with Alerts Groups and set up personal settings, e.g. how to notify.

Note: If your role is Editor, you can skip to Learn Alert Workflow section of this doc

Get alerts into Grafana OnCall and configure rules

Once youve installed Grafana OnCall, or accessed it from your Grafana Cloud instance, you can begin integrating with monitoring systems to get alerts into Grafana OnCall. Additionally, you can configure when, and which, users get notified, by setting templates, routes, escalation chains, etc.

Integrate with a monitoring system

Regardless of where your alerts originate, you can send them to Grafana OnCall via available integrations or customizable webhooks. To start receiving alerts in Grafana OnCall, use the following steps to configure your first integration and send a demo alert.

Configure your first integration

  1. In Grafana OnCall, navigate to the Integrations tab and click + New integration.
  2. Select an integration from the provided options, if the integration youre looking for isnt listed, then select Webhook.
  3. Click How to connect to view the instructions specific to your monitoring system

Send a demo alert

  1. In the integration tab, click Send demo alert, review and modify the alert payload as needed, and click Send
  2. Navigate to the Alert Groups tab to see your test alert firing
  3. Explore the Alert Group by clicking on the title
  4. Acknowledge and resolve the test alert group

For more information on Grafana OnCall integrations and further configuration guidance, refer to [Grafana OnCall integrations][integrations]

Review and modify alert templates

Review and customize templates to interpret monitoring alerts and minimize noise. Group alerts, enable auto-resolution, customize visualizations and notifications by extracting data from alerts. See more details in the [Jinja2 templating][jinja2-templating] section.

Configure scalation Chains

Escalation chains are a set of steps that define who to notify, and when.

See more details in the [Escalation Chains][escalation-chains] section.

Escalation Chains are customizable automated alert routing steps that enable you to specify who is notified for a certain alert. In addition to escalation chains, you can configure Routes to send alerts to different escalation chains depending on the alert details.

Once your integration is configured, you can set up an escalation chain to determine how alerts from your integration are handled. Multi-step escalation chains help ensure thorough alert escalation to prevent alerts from being missed.

To configure Escalation Chains:

  1. Navigate to the Escalation Chains tab and click + New Escalation Chain
  2. Give your Escalation Chain a useful name and click Create
  3. Add a series of escalation steps from the available dropdown options.
  4. To link your Escalation Chain to your integration, navigate back to the Integrations tab, Select your newly created Escalation Chain from the “Escalate to” dropdown.

Alerts from this integration will now follow the escalation steps configured in your Escalation Chain.

For more information on Escalation Chains and more ways to customize them, refer to

[Configure and manage Escalation Chains][escalation-chains-and-routes]

Routes define which messenger channels and escalation chains to use for notifications. See more details in the [Routes][routes] section.

Learn about the Alert Workflow

  • All Alerts in OnCall are grouped into Alert Groups ([read more about Grouping ID][jinja2-templating]). An Alert Group can have the following, mutually exclusive states:

  • Firing: Once Alert Group is registered, Escalation Policy associated with it is getting started. Escalation policy will work while Alert Group is in this status.

  • Acknowledged: Ongoing Escalation Chain will be interrupted. Unacknowledge will move Alert Group to the "Firing" state and will re-launch Escalation Chain.

  • Silenced: Similar to "Acknowledged" but designed to be temporary with a timeout. Once time is out, will re-launch Escalation Chain and move Alert Group to the "Firing" state.

  • Resolved: Similar to "Acknowledged".

  • Possible transitions:

  • Firing -> Acknowledged

  • Firing -> Silenced

  • Firing -> Resolved

  • Silenced -> Firing

  • Silenced -> Acknowledged

  • Silenced -> Resolved

  • Acknowledged -> Silenced

  • Acknowledged -> Firing

  • Acknowledged -> Resolved

  • Resolved -> Firing

Transition changes trigger Escalation Chains to launch, with a few-second delay (to avoid unexpected notifications).

Get notified of an alert

In order for Grafana OnCall to notify you of an alert, you must configure how you want to be notified. Personal notification policies, chatops integrations, and on-call schedules allow you to automate how users are notified of alerts.

Configure personal notification policies

Personal notification policies determine how a user is notified for a certain type of alert. Get notified by SMS, phone call, Slack mentions, or mobile push notification. Administrators can configure how users receive notifications for certain types of alerts. For more information on personal notification policies, refer to [Manage users and teams for Grafana OnCall][user-and-team-management]

To configure users personal notification policies:

  1. Navigate to the Users tab in Grafana OnCall
  2. Select a user from the user list and click Edit
  3. Configure Default Notifications and Important Notification

Configure Slack for Grafana OnCall

Grafana OnCall integrates closely with your Slack workspace to deliver alert notifications to individuals, user groups, and channels. Slack notifications can be triggered by steps in an escalation chain or as a step in users personal notification policies.

To configure Slack for Grafana OnCall:

  1. In OnCall, click on the ChatOps tab and select Slack in the side menu.
  2. Click Install Slack integration.
  3. Read the notice and confirm to proceed to the Slack website.
  4. Sign in to your organization's Slack workspace.
  5. Click Allow to allow Grafana OnCall to access Slack.
  6. Ensure users verify their Slack accounts in their user profile in Grafana OnCall.

For further instruction on connecting to your Slack workspace, refer to [Slack integration for Grafana OnCall][slack]

Grafana OnCall also supports other ChatOps integration like [Microsoft Teams and Telegram][notify].

Add your on-call schedule

Grafana OnCall allows you to manage your on-call schedule in your preferred calendar app such as Google Calendar or Microsoft Outlook.

To integrate your on-call calendar with Grafana OnCall:

  1. In the Schedules tab of Grafana OnCall, click + Add team schedule for on-call rotation.
  2. Provide a schedule name.
  3. Configure the rest of the schedule settings and click Create Schedule

[More information on on-call schedules.][on-call-schedules]

{{% docs/reference %}} [escalation-chains-and-routes]: "/docs/oncall/ -> /docs/oncall//escalation-chains-and-routes" [escalation-chains-and-routes]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes"

[escalation-chains]: "/docs/oncall/ -> /docs/oncall//escalation-chains-and-routes#escalation-chains" [escalation-chains]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes#escalation-chains"

[integrations]: "/docs/oncall/ -> /docs/oncall//integrations" [integrations]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations"

[jinja2-templating]: "/docs/oncall/ -> /docs/oncall//jinja2-templating" [jinja2-templating]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/jinja2-templating"

[notify]: "/docs/oncall/ -> /docs/oncall//notify" [notify]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/notify"

[on-call-schedules]: "/docs/oncall/ -> /docs/oncall//on-call-schedules" [on-call-schedules]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/on-call-schedules"

[open-source]: "/docs/oncall/ -> /docs/oncall//open-source" [open-source]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/open-source"

[routes]: "/docs/oncall/ -> /docs/oncall//escalation-chains-and-routes#routes" [routes]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes#routes"

[slack]: "/docs/oncall/ -> /docs/oncall//notify/slack" [slack]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/notify/slack"

[user-and-team-management]: "/docs/oncall/ -> /docs/oncall//user-and-team-management" [user-and-team-management]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/user-and-team-management" {{% /docs/reference %}}