centralcloud/oncall-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #501 from grafana/dev

Browse source 
Merge dev to main
This commit is contained in:
Vadim Stepanov 2022-09-07 11:41:25 +01:00 • committed by GitHub
commit 04d8daea87
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
17 changed files with 94 additions and 45 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
CHANGELOG.md
View file

@ -1,5 +1,11 @@
# Change Log
## v1.0.35 (2022-09-07)
- Bug fixes
## v1.0.34 (2022-09-06)
- Fix schedule notification spam
## v1.0.33 (2022-09-06)
- Add raw alert view
- Add GitHub star button for OSS installations

14
docs/sources/chat-options/configure-slack.md
View file

@ -16,16 +16,16 @@ weight: 100
# Slack integration for Grafana OnCall
The Slack integration for Grafana OnCall incorporates your Slack workspace directly into your incident response workflow to help your team focus on alert resolution with less friction.
The Slack integration for Grafana OnCall incorporates your Slack workspace directly into your incident response workflow to help your team focus on alert resolution with less friction.
Integrating your Slack workspace with Grafana OnCall allows users and teams to be notified of alerts directly in Slack with automated alert escalation steps and user notification preferences. There are a number of alert actions that users can take directly from Slack, including acknowledge, resolve, add resolution notes, and more.
## Before you begin
To install the Slack integration, you must have Admin permissions in your Grafana instance as well as the Slack workspace that you’d like to integrate with.
To install the Slack integration, you must have Admin permissions in your Grafana instance as well as the Slack workspace that you’d like to integrate with.
For Open Source Grafana OnCall Slack installation guidance, refer to [Open Source Grafana OnCall]({{< relref "../open-source.md" >}}).
For Open Source Grafana OnCall Slack installation guidance, refer to [Open Source Grafana OnCall]({{< relref "../open-source" >}}).
## Install Slack integration for Grafana OnCall
@ -41,23 +41,23 @@ For Open Source Grafana OnCall Slack installation guidance, refer to [Open Sourc
Configure the following additional settings to ensure Grafana OnCall alerts are routed to the intended Slack channels and users:
1. From your **Slack integration** settings, select a default slack channel in the first dropdown menu. This is where alerts will be sent unless otherwise specified in escalation chains.
2. In **Additional Settings**, configure alert reminders for alerts to retrigger after being acknowledged for some amount of time.
2. In **Additional Settings**, configure alert reminders for alerts to retrigger after being acknowledged for some amount of time.
3. Ensure all users verify their slack account in their Grafana OnCall **users info**.
### Configure Escalation Chains with Slack notifications
Once your Slack integration is configured you can configure Escalation Chains to notify via Slack messages for alerts in Grafana OnCall.
Once your Slack integration is configured you can configure Escalation Chains to notify via Slack messages for alerts in Grafana OnCall.
There are two Slack notification options that you can configure into escalation chains, notify whole Slack channel and notify Slack user group:
1. In Grafana OnCall, navigate to the **Escalation Chains** tab then select an existing escalation chain or click **+ New escalation chain**.
2. Click the dropdown for **Add escalation step**.
3. Configure your escalation chain with automated Slack notifications.
3. Configure your escalation chain with automated Slack notifications.
### Configure user notifications with Slack mentions
To be notified of alerts in Grafana OnCall via Slack mentions:
1. Navigate to the **Users** tab in Grafana OnCall, click **Edit** next to a user.
2. In the **User Info** tab, edit or configure notification steps by clicking + Add Notification step
2. In the **User Info** tab, edit or configure notification steps by clicking + Add Notification step
3. select **Notify by** in the first dropdown and select **Slack mentions** in the second dropdown to receive alert notifications via Slack mentions.
### Configure on-call notifications in Slack

0
docs/sources/configure-user-settings.md → docs/sources/configure-user-settings/_index.md
View file

28
docs/sources/getting-started.md → docs/sources/getting-started/_index.md
View file

@ -24,16 +24,16 @@ The following diagram details an example alert workflow with Grafana OnCall:
These procedures introduce you to initial Grafana OnCall configuration steps, including monitoring system integration, how to set up escalation chains, and how to use your calendar service for on-call scheduling.
## Before you begin
## Before you begin
Grafana OnCall is available for Grafana Cloud as well as Grafana open source users. You must have a Grafana Cloud account or [Open Source Grafana OnCall]({{< relref "open-source.md" >}})
Grafana OnCall is available for Grafana Cloud as well as Grafana open source users. You must have a Grafana Cloud account or [Open Source Grafana OnCall]({{< relref "../open-source" >}})
For more information, see [Grafana Pricing](https://grafana.com/pricing/) for details.
## Install Open Source Grafana OnCall
For Open Source Grafana OnCall installation guidance, refer to [Open Source Grafana OnCall]({{< relref "open-source.md" >}})
For Open Source Grafana OnCall installation guidance, refer to [Open Source Grafana OnCall]({{< relref "../open-source" >}})
>**Note:** If you are using Grafana OnCall with your Grafana Cloud instance there are no install steps. Access Grafana OnCall from your Grafana Cloud account and skip ahead to “Get alerts into Grafana OnCall”
@ -53,13 +53,13 @@ Regardless of where your alerts originate, you can send them to Grafana OnCall v
4. Complete any necessary configurations in your monitoring system to send alerts to Grafana OnCall.
#### Send a demo alert
#### Send a demo alert
1. In the integration tab, click **Send demo alert** then navigate to the **Alert Groups** tab to see your test alert firing.
2. Explore the alert by clicking on the title of the alert.
3. Acknowledge and resolve the test alert.
For more information on Grafana OnCall integrations and further configuration guidance, refer to, [Connect to Grafana OnCall]({{< relref "integrations/" >}})
For more information on Grafana OnCall integrations and further configuration guidance, refer to, [Connect to Grafana OnCall]({{< relref "../integrations" >}})
### Configure Escalation Chains
@ -72,18 +72,18 @@ To configure Escalation Chains:
1. Navigate to the **Escalation Chains** tab and click **+ New Escalation Chain**
2. Give your Escalation Chain a useful name and click **Create**
3. Add a series of escalation steps from the available dropdown options.
4. To link your Escalation Chain to your integration, navigate back to the **Integrations tab**, Select your newly created Escalation Chain from the “**Escalate to**” dropdown.
4. To link your Escalation Chain to your integration, navigate back to the **Integrations tab**, Select your newly created Escalation Chain from the “**Escalate to**” dropdown.
Alerts from this integration will now follow the escalation steps configured in your Escalation Chain.
For more information on Escalation Chains and more ways to customize them, refer to [Configure and manage Escalation Chains]({{< relref "escalation-policies/configure-escalation-chains/" >}})
For more information on Escalation Chains and more ways to customize them, refer to [Configure and manage Escalation Chains]({{< relref "../escalation-policies/configure-escalation-chains" >}})
## Get notified of an alert
In order for Grafana OnCall to notify you of an alert, you must configure how you want to be notified. Personal notification policies, chatops integrations, and on-call schedules allow you to automate how users are notified of alerts.
### Configure personal notification policies
Personal notification policies determine how a user is notified for a certain type of alert. Get notified by SMS, phone call, or Slack mentions. Administrators can configure how users receive notification for certain types of alerts. For more information on personal notification policies, refer to [Manage users and teams for Grafana OnCall]({{< relref "configure-user-settings/" >}})
### Configure personal notification policies
Personal notification policies determine how a user is notified for a certain type of alert. Get notified by SMS, phone call, or Slack mentions. Administrators can configure how users receive notification for certain types of alerts. For more information on personal notification policies, refer to [Manage users and teams for Grafana OnCall]({{< relref "../configure-user-settings" >}})
To configure users personal notification policies:
@ -94,7 +94,7 @@ To configure users personal notification policies:
### Configure Slack for Grafana OnCall
Grafana OnCall integrates closely with your Slack workspace to deliver alert notifications to individuals, user groups, and channels. Slack notifications can be triggered by steps in an escalation chain or as a step in users personal notification policies.
Grafana OnCall integrates closely with your Slack workspace to deliver alert notifications to individuals, user groups, and channels. Slack notifications can be triggered by steps in an escalation chain or as a step in users personal notification policies.
To configure Slack for Grafana OnCall:
@ -105,20 +105,20 @@ To configure Slack for Grafana OnCall:
5. Click Allow to allow Grafana OnCall to access Slack.
6. Ensure users verify their Slack accounts in their user profile in Grafana OnCall.
For further instruction on connecting to your Slack workspace, refer to [Connect Slack to Grafana OnCall]({{< relref "chat-options/configure-slack/" >}})
For further instruction on connecting to your Slack workspace, refer to [Connect Slack to Grafana OnCall]({{< relref "../chat-options/configure-slack" >}})
### Add your on-call schedule
Grafana OnCall allows you to manage your on-call schedule in your preferred calendar app such as Google Calendar or Microsoft Outlook.
Grafana OnCall allows you to manage your on-call schedule in your preferred calendar app such as Google Calendar or Microsoft Outlook.
To integrate your on-call calendar with Grafana OnCall:
1. In the **Schedules** tab of Grafana OnCall, click **+ Add team schedule for on-call rotation**.
2. Provide a schedule name.
3. Copy the iCal URL associated with your on-call calendar from your calendar integration settings.
3. Copy the iCal URL associated with your on-call calendar from your calendar integration settings.
4. Configure the rest of the schedule settings and click Create Schedule
For more information on on-call schedules, refer to [Configure and manage on-call schedules]({{< relref "calendar-schedules/" >}})
For more information on on-call schedules, refer to [Configure and manage on-call schedules]({{< relref "../calendar-schedules" >}})

0
docs/sources/open-source.md → docs/sources/open-source/_index.md
View file

6
engine/apps/alerts/escalation_snapshot/escalation_snapshot_mixin.py
View file

@ -7,6 +7,7 @@ from dateutil.parser import parse
from django.apps import apps
from django.utils import timezone
from django.utils.functional import cached_property
from rest_framework.exceptions import ValidationError
from apps.alerts.constants import NEXT_ESCALATION_DELAY
from apps.alerts.escalation_snapshot.snapshot_classes import (
@ -189,7 +190,10 @@ class EscalationSnapshotMixin:
escalation_snapshot_object = None
raw_escalation_snapshot = self.raw_escalation_snapshot
if raw_escalation_snapshot is not None:
escalation_snapshot_object = self._deserialize_escalation_snapshot(raw_escalation_snapshot)
try:
escalation_snapshot_object = self._deserialize_escalation_snapshot(raw_escalation_snapshot)
except ValidationError as e:
logger.error(f"Error trying to deserialize raw escalation snapshot: {e}")
return escalation_snapshot_object
def _deserialize_escalation_snapshot(self, raw_escalation_snapshot) -> EscalationSnapshot:

5
engine/apps/api/serializers/channel_filter.py
View file

@ -96,7 +96,7 @@ class ChannelFilterSerializer(OrderedModelSerializerMixin, EagerLoadingMixin, se
organization = self.context["request"].auth.organization
if not isinstance(notification_backends, dict):
raise serializers.ValidationError(["Invalid messaging backend data"])
current = self.instance.notification_backends or {}
updated = self.instance.notification_backends or {}
for backend_id in notification_backends:
backend = get_messaging_backend_from_id(backend_id)
if backend is None:
@ -106,7 +106,8 @@ class ChannelFilterSerializer(OrderedModelSerializerMixin, EagerLoadingMixin, se
notification_backends[backend_id],
)
# update existing backend data
notification_backends[backend_id] = current.get(backend_id, {}) | updated_data
updated[backend_id] = updated.get(backend_id, {}) | updated_data
notification_backends = updated
return notification_backends

13
engine/apps/api/tests/test_channel_filter.py
View file

@ -437,7 +437,10 @@ def test_channel_filter_update_notification_backends_updates_existing_data(
):
organization, user, token = make_organization_and_user_with_plugin_token()
alert_receive_channel = make_alert_receive_channel(organization)
existing_notification_backends = {"TESTONLY": {"enabled": True, "channel": "ABCDEF"}}
existing_notification_backends = {
"TESTONLY": {"enabled": True, "channel": "ABCDEF"},
"ANOTHERONE": {"enabled": False, "channel": "123456"},
}
channel_filter = make_channel_filter(alert_receive_channel, notification_backends=existing_notification_backends)
client = APIClient()
@ -448,7 +451,13 @@ def test_channel_filter_update_notification_backends_updates_existing_data(
"notification_backends": notification_backends_update,
}
response = client.put(url, data=data_for_update, format="json", **make_user_auth_headers(user, token))
class FakeBackend:
def validate_channel_filter_data(self, organization, data):
return data
with patch("apps.api.serializers.channel_filter.get_messaging_backend_from_id") as mock_get_backend:
mock_get_backend.return_value = FakeBackend()
response = client.put(url, data=data_for_update, format="json", **make_user_auth_headers(user, token))
channel_filter.refresh_from_db()

8
engine/apps/base/models/user_notification_policy_log_record.py
View file

@ -69,7 +69,8 @@ class UserNotificationPolicyLogRecord(models.Model):
ERROR_NOTIFICATION_IN_SLACK_RATELIMIT,
ERROR_NOTIFICATION_MESSAGING_BACKEND_ERROR,
ERROR_NOTIFICATION_NOT_ALLOWED_USER_ROLE,
) = range(26)
ERROR_NOTIFICATION_TELEGRAM_USER_IS_DEACTIVATED,
) = range(27)
# for this errors we want to send message to general log channel
ERRORS_TO_SEND_IN_SLACK_CHANNEL = [
@ -272,6 +273,11 @@ class UserNotificationPolicyLogRecord(models.Model):
self.notification_error_code == UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_NOT_ALLOWED_USER_ROLE
):
result += f"failed to notify {user_verbal}, not allowed role"
elif (
self.notification_error_code
== UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_TELEGRAM_USER_IS_DEACTIVATED
):
result += f"failed to send telegram message to {user_verbal} because user has been deactivated"
else:
# TODO: handle specific backend errors
try:

2
engine/apps/schedules/ical_utils.py
View file

@ -33,7 +33,7 @@ This is a hack to allow us to load models for type checking without circular dep
This module likely needs to refactored to be part of the OnCallSchedule module.
"""
if TYPE_CHECKING:
from apps.schedules.models import OnCallSchedule, OnCallScheduleICal # noqa
from apps.schedules.models import OnCallSchedule
from apps.user_management.models import User

9
engine/apps/slack/scenarios/notification_delivery.py
View file

@ -87,12 +87,3 @@ class NotificationDeliveryStep(scenario_step.ScenarioStep):
print(e)
else:
raise e
def get_color_id(self, color):
if color == "red":
color_id = "#FF0000"
elif color == "yellow":
color_id = "#c6c000"
else:
color_id = color
return color_id

7
engine/apps/telegram/models/connectors/personal.py
View file

@ -111,6 +111,13 @@ class TelegramToUserConnector(models.Model):
notification_policy,
UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_TELEGRAM_TOKEN_ERROR,
)
elif e.message == "Forbidden: user is deactivated":
TelegramToUserConnector.create_telegram_notification_error(
alert_group,
self.user,
notification_policy,
UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_TELEGRAM_USER_IS_DEACTIVATED,
)
else:
raise e
else:

12
engine/apps/telegram/tasks.py
View file

@ -119,7 +119,17 @@ def send_link_to_channel_message_or_fallback_to_full_incident(
@ignore_bot_deleted
def send_log_and_actions_message(self, channel_chat_id, group_chat_id, channel_message_id, reply_to_message_id):
with OkToRetry(task=self, exc=TelegramMessage.DoesNotExist, num_retries=5):
channel_message = TelegramMessage.objects.get(chat_id=channel_chat_id, message_id=channel_message_id)
try:
channel_message = TelegramMessage.objects.get(chat_id=channel_chat_id, message_id=channel_message_id)
except TelegramMessage.DoesNotExist:
if self.request.retries <= 5:
raise
else:
logger.warning(
f"Could not send log and actions message, telegram message does not exist "
f" chat_id={channel_chat_id} message_id={channel_message_id}"
)
return
if channel_message.discussion_group_message_id is None:
channel_message.discussion_group_message_id = reply_to_message_id

12
grafana-plugin/CHANGELOG.md
View file

@ -1,5 +1,17 @@
# Change Log
## v1.0.35 (2022-09-07)
- Bug fixes
## v1.0.34 (2022-09-06)
- Fix schedule notification spam
## v1.0.33 (2022-09-06)
- Add raw alert view
- Add GitHub star button for OSS installations
- Restore alert group search functionality
- Bug fixes
## v1.0.32 (2022-09-01)
- Bug fixes

3
grafana-plugin/src/models/alertgroup/alertgroup.ts
View file

@ -414,8 +414,7 @@ export class AlertGroupStore extends BaseStore {
console.log('undoAction', undoAction);
} catch (e) {
this.updateAlert(alertId, { loading: false });
openErrorNotification(e.response.data?.detail);
openErrorNotification(e.response.data?.detail || e.response.data);
}
}

4
tools/pagerduty-migrator/README.md
View file

@ -20,7 +20,7 @@ Resources that can be migrated using this tool:
1. Make sure you have `docker` installed
2. Build the docker image: `docker build -t pd-oncall-migrator .`
3. Obtain a PagerDuty API token: https://support.pagerduty.com/docs/api-access-keys
3. Obtain a PagerDuty API user token: https://support.pagerduty.com/docs/api-access-keys#generate-a-user-token-rest-api-key
4. Obtain a Grafana OnCall API token and API URL on the "Settings" page of your Grafana OnCall instance
## Migration plan
@ -84,4 +84,4 @@ It's possible to specify a default contact method type for user notification rul
* Connect integrations (press the "How to connect" button on the integration page)
* Make sure users connect their phone numbers, Slack accounts, etc. in their user settings
* At some point you would probably want to recreate schedules using Google Calendar or Terraform to be able to modify migrated on-call schedules in Grafana OnCall

10
tools/pagerduty-migrator/migrator/resources/notification_rules.py
View file

@ -31,12 +31,16 @@ def migrate_notification_rules(user: dict) -> None:
notification_rules, user["oncall_user"]["id"]
)
for rule in user["oncall_user"]["notification_rules"]:
oncall_api_client.delete("personal_notification_rules/{}".format(rule["id"]))
for rule in oncall_rules:
oncall_api_client.create("personal_notification_rules", rule)
if oncall_rules:
# delete old notification rules if any new rules were created
for rule in user["oncall_user"]["notification_rules"]:
oncall_api_client.delete(
"personal_notification_rules/{}".format(rule["id"])
)
def transform_notification_rules(
notification_rules: list[dict], user_id: str