centralcloud/oncall-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Inbound email integration (#837)

Browse source 
This PR add Inbound Email integration.

It designed to support some variety of ESPs, but in prod we will use
Mailgun, so locally I tested it only with mailgun ESP.

**Important:**
To make it work on different clusters I'm planning to provide different
email domains for different regions, like ....@us.oncall.grafana.net,
...@eu.oncall.grafana.net

---------

Co-authored-by: Innokentii Konstantinov <innokenty.konstantinov@grafana.com>
This commit is contained in:
Vadim Stepanov 2023-03-16 05:59:21 +00:00 • committed by GitHub
parent 73c394112c
commit ea60c0d247
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
16 changed files with 267 additions and 53 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
CHANGELOG.md
View file

@ -5,6 +5,10 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## Unreleased
- Inbound email integration ([837](https://github.com/grafana/oncall/pull/837))
## v1.1.38 (2023-03-14)
### Added
@ -27,6 +31,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Direct user paging improvements ([1358](https://github.com/grafana/oncall/issues/1358))
- Added Schedule Score quality within the schedule view ([118](https://github.com/grafana/oncall/issues/118))
## v1.1.36 (2023-03-09)
### Fixed

2
docs/sources/calendar-schedules/_index.md
View file

@ -61,4 +61,4 @@ via Terraform are automatically added to your schedules in Grafana OnCall. Simil
read-only and cannot be edited from the UI.
To learn more, read our [Get started with Grafana OnCall and Terraform](
https://grafana.com/blog/2022/08/29/get-started-with-grafana-oncall-and-terraform/) blog post.
<https://grafana.com/blog/2022/08/29/get-started-with-grafana-oncall-and-terraform/>) blog post.

41
docs/sources/integrations/available-integrations/configure-inbound-email/index.md Normal file
View file

@ -0,0 +1,41 @@
---
aliases:
- add-inbound-email/
- /docs/oncall/latest/integrations/available-integrations/configure-inbound-email/
canonical: https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-inbound-email/
keywords:
- Grafana Cloud
- Alerts
- Notifications
- on-call
- Email
title: Inbound Email integration for Grafana OnCall
weight: 500
---
# Inbound Email integration for Grafana OnCall
Inbound Email integration will consume emails from dedicated email address and make alert groups from them.
## Configure Inbound Email integration for Grafana OnCall
You must have an Admin role to create integrations in Grafana OnCall.
1. In the **Integrations** tab, click **+ New integration to receive alerts**.
2. Select **Inbound Email** from the list of available integrations.
3. Get your dedicated email address in the **How to connect** window.
## Grouping and auto-resolve
Alert groups will be grouped by email subject and auto-resolved if the email message text equals "OK".
This behaviour can be modified via custom templates.
Alerts from Inbound Email integration have followng payload:
```json
{
"subject": "<your_email_subject>",
"message": "<your_email_message>",
"sender": "<your_email_sender_address>"
}
```

9
docs/sources/open-source/_index.md
View file

@ -224,6 +224,15 @@ the following env variables with your SMTP server credentials:
After enabling the email integration, it will be possible to use the `Notify by email` notification step in user settings.
Grafana OnCall is also capable of creating alert groups from
[Inbound Email integration]({{< relref "../integrations/available-integrations/configure-inbound-email" >}}).
To configure Inbound Email integration for Grafana OnCall OSS populate env variables with your Email Service Provider data:
- `INBOUND_EMAIL_ESP` - Inbound email ESP name. Available options: amazon_ses, mailgun, mailjet, mandrill, postal, postmark, sendgrid, sparkpost
- `INBOUND_EMAIL_DOMAIN` - Inbound email domain
- `INBOUND_EMAIL_WEBHOOK_SECRET` - Inbound email webhook secret
## Limits
By default, Grafana OnCall limits email and phone notifications (calls, SMS) to 200 per user per day.

4
engine/apps/alerts/models/alert_receive_channel.py
View file

@ -20,6 +20,7 @@ from apps.alerts.integration_options_mixin import IntegrationOptionsMixin
from apps.alerts.models.maintainable_object import MaintainableObject
from apps.alerts.tasks import disable_maintenance, sync_grafana_alerting_contact_points
from apps.base.messaging import get_messaging_backend_from_id
from apps.base.utils import live_settings
from apps.integrations.metadata import heartbeat
from apps.integrations.tasks import create_alert, create_alertmanager_alerts
from apps.slack.constants import SLACK_RATE_LIMIT_DELAY, SLACK_RATE_LIMIT_TIMEOUT
@ -420,8 +421,7 @@ class AlertReceiveChannel(IntegrationOptionsMixin, MaintainableObject):
@property
def inbound_email(self):
# todo: implement inbound emails
pass
return f"{self.token}@{live_settings.INBOUND_EMAIL_DOMAIN}"
@property
def default_channel_filter(self):

10
engine/apps/base/models/live_setting.py
View file

@ -39,6 +39,9 @@ class LiveSetting(models.Model):
"EMAIL_HOST_PASSWORD",
"EMAIL_USE_TLS",
"EMAIL_FROM_ADDRESS",
"INBOUND_EMAIL_ESP",
"INBOUND_EMAIL_DOMAIN",
"INBOUND_EMAIL_WEBHOOK_SECRET",
"TWILIO_ACCOUNT_SID",
"TWILIO_AUTH_TOKEN",
"TWILIO_API_KEY_SID",
@ -65,6 +68,12 @@ class LiveSetting(models.Model):
"EMAIL_HOST_PASSWORD": "SMTP server password",
"EMAIL_USE_TLS": "SMTP enable/disable TLS",
"EMAIL_FROM_ADDRESS": "Email address used to send emails. If not specified, EMAIL_HOST_USER will be used.",
"INBOUND_EMAIL_DOMAIN": "Inbound email domain",
"INBOUND_EMAIL_ESP": (
"Inbound email ESP name. "
"Available options: amazon_ses, mailgun, mailjet, mandrill, postal, postmark, sendgrid, sparkpost"
),
"INBOUND_EMAIL_WEBHOOK_SECRET": "Inbound email webhook secret",
"SLACK_SIGNING_SECRET": (
"Check <a href='"
"https://grafana.com/docs/grafana-cloud/oncall/open-source/#slack-setup"
@ -141,6 +150,7 @@ class LiveSetting(models.Model):
SECRET_SETTING_NAMES = (
"EMAIL_HOST_PASSWORD",
"INBOUND_EMAIL_WEBHOOK_SECRET",
"TWILIO_ACCOUNT_SID",
"TWILIO_AUTH_TOKEN",
"TWILIO_API_KEY_SID",

142
engine/apps/email/inbound.py Normal file
View file

@ -0,0 +1,142 @@
import logging
from typing import Optional, TypedDict
from anymail.exceptions import AnymailWebhookValidationFailure
from anymail.inbound import AnymailInboundMessage
from anymail.signals import AnymailInboundEvent
from anymail.webhooks import amazon_ses, mailgun, mailjet, mandrill, postal, postmark, sendgrid, sparkpost
from django.http import HttpResponse, HttpResponseNotAllowed
from rest_framework import status
from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView
from apps.base.utils import live_settings
from apps.integrations.mixins import AlertChannelDefiningMixin
from apps.integrations.tasks import create_alert
logger = logging.getLogger(__name__)
# {<ESP name>: (<django-anymail inbound webhook view class>, <webhook secret argument name to pass to the view>), ...}
INBOUND_EMAIL_ESP_OPTIONS = {
"amazon_ses": (amazon_ses.AmazonSESInboundWebhookView, None),
"mailgun": (mailgun.MailgunInboundWebhookView, "webhook_signing_key"),
"mailjet": (mailjet.MailjetInboundWebhookView, "webhook_secret"),
"mandrill": (mandrill.MandrillCombinedWebhookView, "webhook_key"),
"postal": (postal.PostalInboundWebhookView, "webhook_key"),
"postmark": (postmark.PostmarkInboundWebhookView, "webhook_secret"),
"sendgrid": (sendgrid.SendGridInboundWebhookView, "webhook_secret"),
"sparkpost": (sparkpost.SparkPostInboundWebhookView, "webhook_secret"),
}
class EmailAlertPayload(TypedDict):
subject: str
message: str
sender: str
class InboundEmailWebhookView(AlertChannelDefiningMixin, APIView):
def dispatch(self, request):
"""
Wrapper to parse integration_token from inbound email address and pass this token to
AlertChannelDefiningMixin
"""
# http_method_names can't be used due to how AlertChannelDefiningMixin is implemented
# todo: refactor AlertChannelDefiningMixin
if not request.method.lower() in ["head", "post"]:
return HttpResponseNotAllowed(permitted_methods=["head", "post"])
self.check_inbound_email_settings_set()
# Some ESPs verify the webhook with a HEAD request at configuration time
if request.method.lower() == "head":
return HttpResponse(status=status.HTTP_200_OK)
integration_token = self.get_integration_token_from_request(request)
if integration_token is None:
return HttpResponse(status=status.HTTP_400_BAD_REQUEST)
return super().dispatch(request, alert_channel_key=integration_token)
def post(self, request, alert_receive_channel):
for message in self.get_messages_from_esp_request(request):
payload = self.get_alert_payload_from_email_message(message)
create_alert.delay(
title=payload["subject"],
message=payload["message"],
alert_receive_channel_pk=alert_receive_channel.pk,
image_url=None,
link_to_upstream_details=None,
integration_unique_data=request.data,
raw_request_data=payload,
)
return Response("OK", status=status.HTTP_200_OK)
def get_integration_token_from_request(self, request) -> Optional[str]:
messages = self.get_messages_from_esp_request(request)
if not messages:
return None
message = messages[0]
# First try envelope_recipient field.
# According to AnymailInboundMessage it's provided not by all ESPs.
if message.envelope_recipient:
token, domain = message.envelope_recipient.split("@")
if domain == live_settings.INBOUND_EMAIL_DOMAIN:
return token
else:
logger.info(f"get_integration_token_from_request: message.envelope_recipient is not present")
"""
TODO: handle case when envelope_recipient is not provided.
Now we can't just compare to/cc domains one by one with INBOUND_EMAIL_DOMAIN
because this check will not work in case of OrganizationMovedException
"""
# for to in message.to:
# if to.domain == live_settings.INBOUND_EMAIL_DOMAIN:
# return to.address.split("@")[0]
# for cc in message.cc:
# if cc.domain == live_settings.INBOUND_EMAIL_DOMAIN:
# return cc.address.split("@")[0]
return None
def get_messages_from_esp_request(self, request: Request) -> list[AnymailInboundMessage]:
view_class, secret_name = INBOUND_EMAIL_ESP_OPTIONS[live_settings.INBOUND_EMAIL_ESP]
kwargs = {secret_name: live_settings.INBOUND_EMAIL_WEBHOOK_SECRET} if secret_name else {}
view = view_class(**kwargs)
try:
view.run_validators(request)
events = view.parse_events(request)
except AnymailWebhookValidationFailure as e:
logger.info(f"get_messages_from_esp_request: inbound email webhook validation failed: {e}")
return []
return [event.message for event in events if isinstance(event, AnymailInboundEvent)]
def check_inbound_email_settings_set(self):
"""
Guard method to checks if INBOUND_EMAIL settings present.
Returns InternalServerError if not.
"""
# TODO: These settings should be checked before app start.
if not live_settings.INBOUND_EMAIL_ESP:
logger.error(f"InboundEmailWebhookView: INBOUND_EMAIL_ESP env variable must be set.")
return HttpResponse(
status=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
if not live_settings.INBOUND_EMAIL_DOMAIN:
logger.error("InboundEmailWebhookView: INBOUND_EMAIL_DOMAIN env variable must be set.")
return HttpResponse(status=status.HTTP_500_INTERNAL_SERVER_ERROR)
def get_alert_payload_from_email_message(self, email: AnymailInboundMessage) -> EmailAlertPayload:
subject = email.subject or ""
subject = subject.strip()
message = email.text or ""
message = message.strip()
sender = email.from_email.addr_spec
return {"subject": subject, "message": message, "sender": sender}

16
engine/apps/integrations/templates/html/integration_inbound_email.html
View file

@ -1,15 +1,5 @@
<p>This integration will consume emails from dedicated email address and make incidents from them.</p>
<p>It’s useful for:</p>
<ol>
<li>Service desk.</li>
<li>Consuming alerts from other systems using emails as a message bus.</li>
</ol>
<p>Dedicated email address for incidents:</p>
<h4>This is the dedicated email address to create alert groups:</h4>
<pre><code class='code-multiline'>{{ alert_receive_channel.inbound_email }}</code></pre>
<p><i>Fields:</i></p>
<ul>
<li><i>email_subject</i> - alert title;</li>
<li><i>email_body</i> - alert details;</li>
</ul>
<a href="https://grafana.com/docs/oncall/latest/integrations/available-integrations/configure-inbound-email/">Docs</a>

11
engine/apps/integrations/urls.py
View file

@ -1,14 +1,17 @@
from pathlib import Path
from django.conf import settings
from django.urls import path
from apps.email.inbound import InboundEmailWebhookView
from common.api_helpers.optional_slash_router import optional_slash_path
from .views import (
AlertManagerAPIView,
AmazonSNS,
GrafanaAlertingAPIView,
GrafanaAPIView,
HeartBeatAPIView,
InboundWebhookEmailView,
IntegrationHeartBeatAPIView,
UniversalAPIView,
)
@ -28,12 +31,16 @@ urlpatterns = [
path("grafana/<str:alert_channel_key>/", GrafanaAPIView.as_view(), name="grafana"),
path("grafana_alerting/<str:alert_channel_key>/", GrafanaAlertingAPIView.as_view(), name="grafana_alerting"),
path("alertmanager/<str:alert_channel_key>/", AlertManagerAPIView.as_view(), name="alertmanager"),
path("inbound_webhook_email/", InboundWebhookEmailView.as_view(), name="inbound_email"),
path("amazon_sns/<str:alert_channel_key>/", AmazonSNS.as_view(), name="amazon_sns"),
path("heartbeat/<str:alert_channel_key>/", HeartBeatAPIView.as_view(), name="heartbeat"),
path("<str:integration_type>/<str:alert_channel_key>/", UniversalAPIView.as_view(), name="universal"),
]
if settings.FEATURE_INBOUND_EMAIL_ENABLED:
urlpatterns += [
optional_slash_path("inbound_email_webhook", InboundEmailWebhookView.as_view(), name="inbound_email_webhook"),
]
def create_heartbeat_path(integration_url):
return path(

5
engine/apps/integrations/views.py
View file

@ -381,11 +381,6 @@ class HeartBeatAPIView(AlertChannelDefiningMixin, APIView):
return Response("Ok.")
class InboundWebhookEmailView(AlertChannelDefiningMixin, APIView):
# todo: implement inbound emails
pass
class IntegrationHeartBeatAPIView(AlertChannelDefiningMixin, IntegrationHeartBeatRateLimitMixin, APIView):
def get(self, request, alert_receive_channel):
self._process_heartbeat_signal(request, alert_receive_channel)

19
engine/config_integrations/inbound_email.py
View file

@ -1,19 +1,20 @@
from django.conf import settings
# Main
enabled = True
title = "Inboubd Email"
title = "Inbound Email"
slug = "inbound_email"
short_description = None
description = None
is_displayed_on_web = False
is_displayed_on_web = settings.FEATURE_INBOUND_EMAIL_ENABLED
is_featured = False
is_able_to_autoresolve = False
is_able_to_autoresolve = True
is_demo_alert_enabled = False
description = None
# Default templates
slack_title = """\
*<{{ grafana_oncall_link }}|#{{ grafana_oncall_incident_id }} {{ payload.get("title", "Title undefined (check Slack Title Template)") }}>* via {{ integration_name }}
*<{{ grafana_oncall_link }}|#{{ grafana_oncall_incident_id }} {{ payload.get("subject", "Title undefined (check Slack Title Template)") }}>* via {{ integration_name }}
{% if source_link %}
(*<{{ source_link }}|source>*)
{%- endif %}"""
@ -22,7 +23,7 @@ slack_message = "{{ payload.message }}"
slack_image_url = "{{ payload.image_url }}"
web_title = '{{ payload.get("title", "Title undefined (check Web Title Template)") }}'
web_title = '{{ payload.get("subject", "Title undefined (check Web Title Template)") }}'
web_message = slack_message
@ -38,10 +39,10 @@ telegram_message = slack_message
telegram_image_url = slack_image_url
source_link = "{{ payload.link_to_upstream_details }}"
source_link = None
grouping_id = '{{ payload.get("title", "")}}'
grouping_id = '{{ payload.get("subject", "").upper() }}'
resolve_condition = '{{ payload.get("state", "").upper() == "OK" }}'
resolve_condition = '{{ payload.get("message", "").upper() == "OK" }}'
acknowledge_condition = None

4
engine/config_integrations/zabbix.py
View file

@ -32,10 +32,6 @@ sms_title = web_title
phone_call_title = sms_title
email_title = web_title
email_message = web_message
telegram_title = sms_title
telegram_message = slack_message

9
engine/engine/middlewares.py
View file

@ -40,7 +40,14 @@ class RequestTimeLoggingMiddleware(MiddlewareMixin):
if request.path.startswith("/integrations/v1"):
split_path = request.path.split("/")
integration_type = split_path[3]
integration_token = split_path[4]
# integration token is not always present in the URL,
# e.g. for inbound emails integration token is passed in the request payload
if len(split_path) >= 5:
integration_token = split_path[4]
else:
integration_token = None
message += f"integration_type={integration_type} integration_token={integration_token} "
logging.info(message)

1
engine/requirements.txt
View file

@ -50,3 +50,4 @@ opentelemetry-exporter-otlp-proto-grpc==1.15.0
pyroscope-io==0.8.1
django-dbconn-retry==0.1.7
django-ipware==4.0.2
django-anymail==8.6

6
engine/settings/base.py
View file

@ -58,6 +58,7 @@ FEATURE_EMAIL_INTEGRATION_ENABLED = getenv_boolean("FEATURE_EMAIL_INTEGRATION_EN
FEATURE_SLACK_INTEGRATION_ENABLED = getenv_boolean("FEATURE_SLACK_INTEGRATION_ENABLED", default=True)
FEATURE_WEB_SCHEDULES_ENABLED = getenv_boolean("FEATURE_WEB_SCHEDULES_ENABLED", default=False)
FEATURE_MULTIREGION_ENABLED = getenv_boolean("FEATURE_MULTIREGION_ENABLED", default=False)
FEATURE_INBOUND_EMAIL_ENABLED = getenv_boolean("FEATURE_INBOUND_EMAIL_ENABLED", default=False)
GRAFANA_CLOUD_ONCALL_HEARTBEAT_ENABLED = getenv_boolean("GRAFANA_CLOUD_ONCALL_HEARTBEAT_ENABLED", default=True)
GRAFANA_CLOUD_NOTIFICATIONS_ENABLED = getenv_boolean("GRAFANA_CLOUD_NOTIFICATIONS_ENABLED", default=True)
@ -610,6 +611,11 @@ EMAIL_NOTIFICATIONS_LIMIT = getenv_integer("EMAIL_NOTIFICATIONS_LIMIT", 200)
if FEATURE_EMAIL_INTEGRATION_ENABLED:
EXTRA_MESSAGING_BACKENDS += [("apps.email.backend.EmailBackend", 8)]
# Inbound email settings
INBOUND_EMAIL_ESP = os.getenv("INBOUND_EMAIL_ESP")
INBOUND_EMAIL_DOMAIN = os.getenv("INBOUND_EMAIL_DOMAIN")
INBOUND_EMAIL_WEBHOOK_SECRET = os.getenv("INBOUND_EMAIL_WEBHOOK_SECRET")
INSTALLED_ONCALL_INTEGRATIONS = [
"config_integrations.alertmanager",
"config_integrations.grafana",

36
grafana-plugin/src/containers/IntegrationSettings/IntegrationSettings.tsx
View file

@ -136,22 +136,26 @@ const IntegrationSettings = observer((props: IntegrationSettingsProps) => {
{activeTab === IntegrationSettingsTab.HowToConnect && (
<div className="container">
<VerticalGroup>
<h4>This is the unique webhook URL for the integration:</h4>
<div style={{ width: '70%' }}>
<Input
value={alertReceiveChannel.integration_url}
addonAfter={
<CopyToClipboard
text={alertReceiveChannel.integration_url}
onCopy={() => {
openNotification('Unique webhook URL copied');
}}
>
<Button icon="copy" variant="primary" />
</CopyToClipboard>
}
/>
</div>
{alertReceiveChannel.integration_url && (
<div>
<h4>This is the unique webhook URL for the integration:</h4>
<div style={{ width: '70%' }}>
<Input
value={alertReceiveChannel.integration_url}
addonAfter={
<CopyToClipboard
text={alertReceiveChannel.integration_url}
onCopy={() => {
openNotification('Unique webhook URL copied');
}}
>
<Button icon="copy" variant="primary" />
</CopyToClipboard>
}
/>
</div>
</div>
)}
<div dangerouslySetInnerHTML={{ __html: alertReceiveChannel?.instructions }} />
<Button variant="primary" onClick={onHide}>
Open Escalations Settings