dcae98b02a
Related to https://github.com/grafana/oncall/issues/96 --------- Co-authored-by: Ravishankar <ravishankar.gnanaprakasam@gmail.com>
386 lines
18 KiB
Markdown
386 lines
18 KiB
Markdown
---
|
|
title: Grafana OnCall open source guide
|
|
menuTitle: Open source
|
|
description: Installation and setup configuration details for open source Grafana OnCall
|
|
weight: 300
|
|
keywords:
|
|
- OnCall
|
|
- Open source
|
|
- OSS
|
|
- Install
|
|
canonical: https://grafana.com/docs/oncall/latest/set-up/open-source/
|
|
aliases:
|
|
- /docs/grafana-cloud/alerting-and-irm/oncall/set-up/open-source/
|
|
- /docs/grafana-cloud/alerting-and-irm/oncall/open-source/
|
|
- ../open-source/ # /docs/oncall/<ONCALL_VERSION>/open-source/
|
|
refs:
|
|
grafana-oncall-mobile-app:
|
|
- pattern: /docs/oncall/
|
|
destination: /docs/oncall/<ONCALL_VERSION>/manage/mobile-app/
|
|
- pattern: /docs/grafana-cloud/
|
|
destination: /docs/grafana-cloud/alerting-and-irm/oncall/manage/mobile-app/
|
|
inbound-email-integration:
|
|
- pattern: /docs/oncall/
|
|
destination: /docs/oncall/<ONCALL_VERSION>/configure/integrations/references/inbound-email/
|
|
- pattern: /docs/grafana-cloud/
|
|
destination: /docs/grafana-cloud/alerting-and-irm/oncall/configure/integrations/references/inbound-email/
|
|
---
|
|
|
|
# Grafana OnCall open source guide
|
|
|
|
Grafana OnCall is a developer-friendly incident response tool that's available to Grafana open source and Grafana Cloud
|
|
users. The OSS version of Grafana OnCall provides the same reliable on-call management solution along with the
|
|
flexibility of a self-managed environment.
|
|
|
|
This guide describes the necessary installation and configuration steps needed to configure OSS Grafana OnCall.
|
|
|
|
## Install Grafana OnCall OSS
|
|
|
|
There are three Grafana OnCall OSS environments available:
|
|
|
|
- **Hobby** playground environment for local usage: [README.md](https://github.com/grafana/oncall#getting-started)
|
|
- **Development** environment for contributors: [development README.md](https://github.com/grafana/oncall/blob/dev/dev/README.md)
|
|
- **Production** environment for reliable cloud installation using Helm: [Production Environment](#production-environment)
|
|
|
|
## Production Environment
|
|
|
|
We suggest using our official helm chart for the reliable production deployment of Grafana OnCall. It will deploy
|
|
Grafana OnCall engine and celery workers, along with RabbitMQ cluster, Redis Cluster, and the database.
|
|
|
|
> **Note:** The Grafana OnCall engine currently supports one instance of the Grafana OnCall plugin at a time.
|
|
|
|
Check the [helm chart](https://github.com/grafana/oncall/tree/dev/helm/oncall) for more details.
|
|
|
|
We'll always be happy to provide assistance with production deployment in [our communities](https://github.com/grafana/oncall#join-community)!
|
|
|
|
## Update Grafana OnCall OSS
|
|
|
|
To update an OSS installation of Grafana OnCall, please see the update docs:
|
|
|
|
- **Hobby** playground environment: [README.md](https://github.com/grafana/oncall#update-version)
|
|
- **Production** Helm environment: [Helm update](https://github.com/grafana/oncall/tree/dev/helm/oncall#update)
|
|
|
|
## Slack Setup
|
|
|
|
The Slack integration for Grafana OnCall leverages Slack API features to provide a customizable and useful integration.
|
|
Refer to the following steps to configure the Slack integration:
|
|
|
|
1. Ensure your Grafana OnCall environment is up and running
|
|
|
|
1. Set `FEATURE_SLACK_INTEGRATION_ENABLED` as True
|
|
|
|
1. Grafana OnCall must be accessible through HTTPS. For development purposes, use [localtunnel](https://github.com/localtunnel/localtunnel).
|
|
For production purposes, consider establishing a proper web server with HTTPS termination.
|
|
For localtunnel, refer to the following configuration:
|
|
|
|
```bash
|
|
# Choose the unique prefix instead of pretty-turkey-83
|
|
# Localtunnel will generate an url, e.g. https://pretty-turkey-83.loca.lt
|
|
# it is referred as <ONCALL_ENGINE_PUBLIC_URL> below
|
|
lt --port 8080 -s pretty-turkey-83 --print-requests
|
|
```
|
|
|
|
1. If using localtunnel, open your external URL and click **Continue** to allow requests to bypass the warning page.
|
|
|
|
1. [Create a Slack Workspace](https://slack.com/create) for development, or use your company workspace.
|
|
|
|
1. Go to <https://api.slack.com/apps> and click **Create an App** .
|
|
|
|
1. Select `From an app manifest` option and select your workspace.
|
|
|
|
1. Replace the text with the following YAML code block . Be sure to replace `<YOUR_BOT_NAME>` and `<ONCALL_ENGINE_PUBLIC_URL>`
|
|
fields with the appropriate information.
|
|
|
|
```yaml
|
|
_metadata:
|
|
major_version: 1
|
|
minor_version: 1
|
|
display_information:
|
|
name: <YOUR_BOT_NAME>
|
|
features:
|
|
app_home:
|
|
home_tab_enabled: false
|
|
messages_tab_enabled: true
|
|
messages_tab_read_only_enabled: false
|
|
bot_user:
|
|
display_name: <YOUR_BOT_NAME>
|
|
always_online: true
|
|
shortcuts:
|
|
- name: Add to resolution note
|
|
type: message
|
|
callback_id: add_resolution_note
|
|
description: Add this message to resolution note
|
|
slash_commands:
|
|
- command: /escalate
|
|
url: <ONCALL_ENGINE_PUBLIC_URL>/slack/interactive_api_endpoint/
|
|
description: Direct page a team or user(s)
|
|
should_escape: false
|
|
oauth_config:
|
|
redirect_urls:
|
|
- <ONCALL_ENGINE_PUBLIC_URL>/api/internal/v1/complete/slack-install-free/
|
|
- <ONCALL_ENGINE_PUBLIC_URL>/api/internal/v1/complete/slack-login/
|
|
scopes:
|
|
user:
|
|
- channels:read
|
|
- identify
|
|
- users.profile:read
|
|
bot:
|
|
- app_mentions:read
|
|
- channels:history
|
|
- channels:read
|
|
- chat:write
|
|
- chat:write.customize
|
|
- chat:write.public
|
|
- commands
|
|
- files:write
|
|
- groups:history
|
|
- groups:read
|
|
- im:history
|
|
- im:read
|
|
- im:write
|
|
- mpim:history
|
|
- mpim:read
|
|
- mpim:write
|
|
- reactions:write
|
|
- team:read
|
|
- usergroups:read
|
|
- usergroups:write
|
|
- users.profile:read
|
|
- users:read
|
|
- users:read.email
|
|
- users:write
|
|
settings:
|
|
event_subscriptions:
|
|
request_url: <ONCALL_ENGINE_PUBLIC_URL>/slack/event_api_endpoint/
|
|
bot_events:
|
|
- app_home_opened
|
|
- app_mention
|
|
- channel_archive
|
|
- channel_created
|
|
- channel_deleted
|
|
- channel_rename
|
|
- channel_unarchive
|
|
- member_joined_channel
|
|
- message.channels
|
|
- message.im
|
|
- subteam_created
|
|
- subteam_members_changed
|
|
- subteam_updated
|
|
- user_profile_changed
|
|
interactivity:
|
|
is_enabled: true
|
|
request_url: <ONCALL_ENGINE_PUBLIC_URL>/slack/interactive_api_endpoint/
|
|
org_deploy_enabled: false
|
|
socket_mode_enabled: false
|
|
```
|
|
|
|
1. Set environment variables by navigating to your Grafana OnCall, then click **Env Variables** and set the following:
|
|
|
|
```text
|
|
SLACK_CLIENT_OAUTH_ID = Basic Information -> App Credentials -> Client ID
|
|
SLACK_CLIENT_OAUTH_SECRET = Basic Information -> App Credentials -> Client Secret
|
|
SLACK_SIGNING_SECRET = Basic Information -> App Credentials -> Signing Secret
|
|
SLACK_INSTALL_RETURN_REDIRECT_HOST = << OnCall external URL >>
|
|
```
|
|
|
|
1. In OnCall, navigate to **ChatOps**, select Slack and click **Install Slack integration**.
|
|
|
|
1. Configure additional Slack settings.
|
|
|
|
## Telegram Setup
|
|
|
|
<iframe width="560" height="315" src="https://www.youtube.com/embed/YMDFZP7Dxns" title="YouTube video player"
|
|
frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture;
|
|
web-share" allowfullscreen></iframe>
|
|
|
|
The Telegram integration for Grafana OnCall is designed for collaborative team work and improved incident response.
|
|
It's available in two options: using Webhooks or Long Polling.
|
|
Refer to the following steps to configure the Telegram integration:
|
|
|
|
1. Ensure your Grafana OnCall environment is up and running.
|
|
2. Set `FEATURE_TELEGRAM_INTEGRATION_ENABLED` as "True"
|
|
3. (Long Polling only) Set `FEATURE_TELEGRAM_LONG_POLLING_ENABLED` as "True"
|
|
4. Create a Telegram bot using [BotFather](https://t.me/BotFather) and save the token provided by BotFather. Please make
|
|
sure to disable **Group Privacy** for the bot (Bot Settings -> Group Privacy -> Turn off).
|
|
5. Paste the token provided by BotFather to the `TELEGRAM_TOKEN` variable on the **Env Variables** page of your
|
|
Grafana OnCall instance.
|
|
6. (Webhook only) Set the `TELEGRAM_WEBHOOK_HOST` variable to the external address of your Grafana OnCall instance. Please note
|
|
that `TELEGRAM_WEBHOOK_HOST` must start with `https://` and be publicly available (meaning that it can be reached by
|
|
Telegram servers). If your host is private or local, consider using a reverse proxy (e.g. [ngrok](https://ngrok.com)).
|
|
7. Now you can connect Telegram accounts on the **Users** page and receive alert groups to Telegram direct messages.
|
|
Alternatively, in case you want to connect Telegram channels to your Grafana OnCall environment, navigate
|
|
to the **ChatOps** tab.
|
|
|
|
## Mattermost Setup
|
|
|
|
The Mattermost integration of the Grafana OnCall is designed for collobarative team work and improved incident response.
|
|
|
|
Refer to the following steps to configure the Mattermost integration:
|
|
|
|
1. Ensure your Grafana OnCall environment is up and running.
|
|
2. Set `FEATURE_MATTERMOST_INTEGRATION_ENABLED` as "True".
|
|
3. Create a Mattermost bot account [Ref](https://developers.mattermost.com/integrate/reference/bot-accounts/#bot-account-creation)
|
|
and save the token generated (NOTE: you may need to give the bot admin permissions
|
|
to make it possible for it to update alert group notifications).
|
|
4. Add the bot to the Mattermost team(s) owning the channels you want to connect to
|
|
(via Mattermost System console -> User Management -> Teams).
|
|
5. Set the token generated in the `MATTERMOST_BOT_TOKEN` variable on the **ENV Variables** page
|
|
of your Grafana OnCall instance.
|
|
6. [Create OAuth 2.0 Application In Mattermost](https://developers.mattermost.com/integrate/apps/authentication/oauth2/#register-an-oauth-20-application).
|
|
The callback url for the OAuth application should be,
|
|
|
|
```text
|
|
https://<ONCALL_ENGINE_PUBLIC_URL>/api/internal/v1/complete/mattermost-login/
|
|
```
|
|
|
|
This will allow users to connect their OnCall accounts with Mattermost. The OAuth credentials will be needed later.
|
|
7. Generate a JWT secret for authenticating the incoming event messages from Mattermost
|
|
and set it as the `MATTERMOST_SIGNING_SECRET` variable on the **ENV Variables** page of your Grafana OnCall instance.
|
|
8. Set the following environment variables too:
|
|
|
|
```text
|
|
MATTERMOST_CLIENT_OAUTH_ID = << Integrations -> OAuth 2.0 Applications -> Client ID >>
|
|
MATTERMOST_CLIENT_OAUTH_SECRET = << Integrations -> OAuth 2.0 Applications -> Client Secret >>
|
|
MATTERMOST_HOST = << Mattermost server URL >>
|
|
MATTERMOST_LOGIN_RETURN_REDIRECT_HOST = << OnCall external URL >>
|
|
```
|
|
|
|
- SMS for user notifications
|
|
- Phone calls for user notifications.
|
|
|
|
To connect to Grafana Cloud OnCall, refer to the **Cloud** page in your OSS Grafana OnCall instance.
|
|
|
|
Check the Settings page for the Grafana Cloud OnCall API URL. If it's not `https://oncall-prod-us-central-0.grafana.net/oncall`,
|
|
you will need to set the `GRAFANA_CLOUD_ONCALL_API_URL` variable in your self-hosted OnCall, so that it can connect properly.
|
|
|
|
## Supported Phone Providers
|
|
|
|
### Exotel
|
|
|
|
Grafana OnCall supports Exotel phone call notifications delivery. To configure phone call notifications using Exotel,
|
|
complete the following steps:
|
|
|
|
1. Set `GRAFANA_CLOUD_NOTIFICATIONS_ENABLED` as **False** to ensure the Grafana OSS <-> Cloud connector is disabled.
|
|
2. Change `PHONE_PROVIDER` value to `exotel`.
|
|
3. `EXOTEL_ACCOUNT_SID` can be found under DEVELOPER SETTINGS->API Settings
|
|
4. `EXOTEL_API_KEY` and `EXOTEL_API_TOKEN` can also be found under DEVELOPER SETTINGS->API Settings
|
|
5. `EXOTEL_APP_ID` is the identifier of the flow (or applet) which can be found under MANAGE->App Bazaar (Installed apps)
|
|
6. `EXOTEL_CALLER_ID` is the Exophone / Exotel virtual number.
|
|
7. `EXOTEL_SMS_SENDER_ID` is the SMS Sender ID to use for sending verification SMS, which can be found under
|
|
SMS SETTINGS->Sender ID.
|
|
8. `EXOTEL_SMS_VERIFICATION_TEMPLATE` is the SMS text template to be used for sending verification SMS, add
|
|
$verification_code as a placeholder.
|
|
9. `EXOTEL_SMS_DLT_ENTITY_ID` is the DLT Entity ID registered with TRAI.
|
|
|
|
### Twilio
|
|
|
|
Grafana OnCall supports Twilio SMS and phone call notifications delivery. If you prefer to configure SMS and phone call
|
|
notifications using Twilio, complete the following steps:
|
|
|
|
1. Set `GRAFANA_CLOUD_NOTIFICATIONS_ENABLED` as **False** to ensure the Grafana OSS <-> Cloud connector is disabled.
|
|
2. From your **OnCall** environment, select **Env Variables** and configure all variables starting with `TWILIO_`.
|
|
|
|
### Zvonok.com
|
|
|
|
Grafana OnCall supports Zvonok.com phone call notifications delivery. To configure phone call notifications using
|
|
Zvonok.com, complete the following steps:
|
|
|
|
1. Set `GRAFANA_CLOUD_NOTIFICATIONS_ENABLED` as **False** to ensure the Grafana OSS <-> Cloud connector is disabled.
|
|
2. Change `PHONE_PROVIDER` value to `zvonok`.
|
|
3. Create a public API key on the Profile->Settings page, and assign its value to `ZVONOK_API_KEY`.
|
|
4. Create campaign and assign its ID value to `ZVONOK_CAMPAIGN_ID`.
|
|
5. If you are planning to use pre-recorded audio instead of a speech synthesizer, you can copy the ID of the audio clip
|
|
to the variable `ZVONOK_AUDIO_ID` (optional step).
|
|
6. To make a call with a specific voice, you can set the `ZVONOK_SPEAKER_ID`.
|
|
By default, the ID used is `Salli` (optional step).
|
|
7. Create phone number verification campaign with type `tellcode` and assign its ID value to `ZVONOK_VERIFICATION_CAMPAIGN_ID`.
|
|
8. To process the call status, it is required to add a postback with the GET/POST method on the side of the zvonok.com
|
|
service with the following format (optional step):
|
|
`${ONCALL_BASE_URL}/zvonok/call_status_events?campaign_id={ct_campaign_id}&call_id={ct_call_id}&status={ct_status}&user_choice={ct_user_choice}`
|
|
|
|
The names of the transmitted parameters can be redefined through environment variables:
|
|
|
|
- `ZVONOK_POSTBACK_CALL_ID` - call id (ct_call_id) query parameter name
|
|
- `ZVONOK_POSTBACK_CAMPAIGN_ID` - company id (ct_campaign_id) query parameter name
|
|
- `ZVONOK_POSTBACK_STATUS` - status (ct_status) query parameter name
|
|
- `ZVONOK_POSTBACK_USER_CHOICE` - user choice (ct_user_choice) query parameter name
|
|
- `ZVONOK_POSTBACK_USER_CHOICE_ACK` - user choice (ct_user_choice) query parameter value for acknowledge alert group
|
|
|
|
## Email Setup
|
|
|
|
Grafana OnCall is capable of sending emails using SMTP as a user notification step. To setup email notifications, populate
|
|
the following env variables with your SMTP server credentials:
|
|
|
|
- `EMAIL_HOST` - SMTP server host
|
|
- `EMAIL_HOST_USER` - SMTP server user
|
|
- `EMAIL_HOST_PASSWORD` - SMTP server password
|
|
- `EMAIL_PORT` (default is `587`) - SMTP server port
|
|
- `EMAIL_USE_TLS` (default is `True`) - To enable/disable TLS
|
|
- `EMAIL_FROM_ADDRESS` (optional) - Email address used to send emails. If not specified, `EMAIL_HOST_USER` will be used.
|
|
|
|
After enabling the email integration, it will be possible to use the `Notify by email` notification step in user settings.
|
|
|
|
## Inbound Email Setup
|
|
|
|
Grafana OnCall is capable of creating alert groups from the [Inbound email integration](ref:inbound-email-integration).
|
|
|
|
To configure Inbound Email integration for Grafana OnCall OSS populate env variables with your Email Service Provider data:
|
|
|
|
- `INBOUND_EMAIL_ESP` - Inbound email ESP name. Available options: `amazon_ses`, `mailgun`, `mailjet`, `mandrill`, `postal`, `postmark`, `sendgrid`, `sparkpost`
|
|
- `INBOUND_EMAIL_DOMAIN` - Inbound email domain
|
|
- `INBOUND_EMAIL_WEBHOOK_SECRET` - Inbound email webhook secret
|
|
|
|
Required secret syntax: `part1ofsecret:part2ofsecret` (The colon `:` is a mandatory delimiter separating both parts of your secret.)
|
|
|
|
You will also need to configure your ESP to forward messages to the following URL: `scheme://<INBOUND_EMAIL_WEBHOOK_SECRET>@<ONCALL_ENGINE_PUBLIC_URL>/integrations/v1/inbound_email_webhook`.
|
|
|
|
## Limits
|
|
|
|
By default, Grafana OnCall limits email and phone notifications (calls, SMS) to 200 per user per day.
|
|
The limit can be changed using env variables:
|
|
|
|
- `PHONE_NOTIFICATIONS_LIMIT` (default is `200`) - phone notifications per user
|
|
- `EMAIL_NOTIFICATIONS_LIMIT` (default is `200`) - emails per user
|
|
|
|
## Mobile application set up
|
|
|
|
Grafana OnCall OSS users can use the mobile app to receive push notifications from OnCall.
|
|
Grafana OnCall OSS relies on Grafana Cloud as on relay for push notifications.
|
|
You must first connect your Grafana OnCall OSS to Grafana Cloud for the mobile app to work.
|
|
|
|
Refer to [Install Grafana OnCall OSS](#install-grafana-oncall-oss) before continuing with the mobile app.
|
|
|
|
For Grafana OnCall OSS, the mobile app QR code includes an authentication token along with a backend URL.
|
|
Your Grafana OnCall OSS instance should be reachable from the same network as your mobile device, preferably from the internet.
|
|
|
|
For more information, refer to [Grafana OnCall mobile app](ref:grafana-oncall-mobile-app).
|
|
|
|
## Alert Group Escalation Auditor
|
|
|
|
Grafana OnCall has a periodic background task, which runs to check that all alert group escalations have finished
|
|
properly. This feature, if configured, can also ping an OnCall Webhook Integration's heartbeat URL, so that you can be
|
|
alerted, in the event that something goes wrong.
|
|
|
|
Logs originating from the celery worker, for the `apps.alerts.tasks.check_escalation_finished.check_escalation_finished_task`
|
|
task, that reference a `AlertGroupEscalationPolicyExecutionAuditException` exception
|
|
indicate that the auditor periodic task is failing check(s) on one or more alert groups. Logs for this task which
|
|
mention `.. passed the audit checks` indicate that there were no issues with with the escalation on the audited
|
|
alert groups.
|
|
|
|
To configure this feature as such:
|
|
|
|
1. Create a Webhook, or Formatted Webhook, Integration type.
|
|
1. Under the "Heartbeat" tab in the Integration modal, copy the unique heartbeat URL that is shown.
|
|
1. Set the heartbeat's expected time interval to 15 minutes (see note below regarding `ALERT_GROUP_ESCALATION_AUDITOR_CELERY_TASK_HEARTBEAT_INTERVAL`)
|
|
1. Configure the integration's escalation chain as necessary
|
|
1. Populate the following env variables:
|
|
|
|
- `ALERT_GROUP_ESCALATION_AUDITOR_CELERY_TASK_HEARTBEAT_URL` - integration's unique heartbeat URL
|
|
- `ALERT_GROUP_ESCALATION_AUDITOR_CELERY_TASK_HEARTBEAT_INTERVAL` - how often the auditor task should run. By default the
|
|
task runs every 13 minutes so we therefore recommend setting the heartbeat's expected time interval to 15 minutes. If you
|
|
would like to modify this, we recommend configuring this env variable to 1 or 2 minutes less than the value set for the
|
|
integration's heartbeat expected time interval.
|
|
|
|
Additionally, if you prefer to disable this feature, you can set the `ESCALATION_AUDITOR_ENABLED` environment variable
|
|
to `False`.
|
|
e`.
|