centralcloud/oncall-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'dev' of github.com:grafana/oncall into dev

Browse source
This commit is contained in:
Maxim 2022-10-25 14:39:38 +01:00
commit 0197d945e0
303 changed files with 4023 additions and 4127 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
.env.dev.example
View file

@ -14,11 +14,6 @@ TWILIO_VERIFY_SERVICE_SID=
TWILIO_AUTH_TOKEN=
TWILIO_NUMBER=
SENDGRID_SECRET_KEY=
SENDGRID_INBOUND_EMAIL_DOMAIN=
SENDGRID_API_KEY=
SENDGRID_FROM_EMAIL=
DJANGO_SETTINGS_MODULE=settings.dev
SECRET_KEY=jkashdkjashdkjh
BASE_URL=http://localhost:8080

30
.github/workflows/ci.yml vendored
View file

@ -27,6 +27,22 @@ jobs:
run: |
pre-commit run --all-files
test:
runs-on: ubuntu-latest
container: python:3.9
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 14.17.0
- name: Unit Testing Frontend
run: |
pip install $(grep "pre-commit" engine/requirements.txt)
npm install -g yarn
cd grafana-plugin/
yarn --network-timeout 500000
yarn test
test-technical-documentation:
runs-on: ubuntu-latest
steps:
@ -124,3 +140,17 @@ jobs:
cd engine/
pip install -r requirements.txt
pytest --ds=settings.ci-test -x
docker-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Test docker build (no push)
id: docker_build
uses: docker/build-push-action@v2
with:
context: ./engine
file: ./engine/Dockerfile
push: false
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}

18
.pre-commit-config.yaml
View file

@ -23,16 +23,26 @@ repos:
- flake8-tidy-imports
- repo: https://github.com/pre-commit/mirrors-eslint
rev: v7.21.0
rev: v8.25.0
hooks:
- id: eslint
entry: bash -c 'cd grafana-plugin && eslint --fix ${@/grafana-plugin\//}' --
entry: bash -c 'cd grafana-plugin && eslint --max-warnings=0 --fix ${@/grafana-plugin\//}' --
types: [file]
files: ^grafana-plugin/src/.*\.(js|jsx|ts|tsx)$
additional_dependencies:
- eslint@7.21.0
- eslint@^8.25.0
- eslint-plugin-import@^2.25.4
- eslint-plugin-rulesdir@^0.2.1
- "@grafana/eslint-config@^5.0.0"
- repo: https://github.com/pre-commit/mirrors-prettier
rev: "v2.7.1"
hooks:
- id: prettier
types_or: [css, javascript, jsx, ts, tsx, json]
files: ^grafana-plugin/src
additional_dependencies:
- prettier@^2.7.1
- repo: https://github.com/thibaudcolas/pre-commit-stylelint
rev: v13.13.1
@ -43,4 +53,6 @@ repos:
files: ^grafana-plugin/src/.*\.css$
additional_dependencies:
- stylelint@^13.13.1
- stylelint-prettier@^2.0.0
- stylelint-config-standard@^22.0.0
- stylelint-config-prettier@^9.0.3

7
CHANGELOG.md
View file

@ -1,5 +1,12 @@
# Change Log
## v1.0.42 (2022-10-24)
- Fix posting resolution notes to Slack
## v1.0.41 (2022-10-24)
- Add personal email notifications
- Bug fixes
## v1.0.40 (2022-10-05)
- Improved database and celery backends support
- Added script to import PagerDuty users to Grafana

40
DEVELOPER.md
View file

@ -2,6 +2,7 @@
- [Code style](#code-style)
- [Backend setup](#backend-setup)
- [Frontend setup](#frontend-setup)
- [Setup using Makefile](#setup-using-makefile)
- [Slack application setup](#slack-application-setup)
- [Update drone build](#update-drone-build)
- [Troubleshooting](#troubleshooting)
@ -142,6 +143,45 @@ extra_hosts:
```
### Setup using Makefile
- Make sure you have `make` installed
- Backend setup:
- Run stateful services:
`$ make docker-services-start`
(you can change your preferred docker file by defining the `DOCKER_FILE` env variable)
- Setup environment:
`$ make bootstrap`
(you can change your preferred directory for your Python virtualenv by defining the `ENV_DIR` env variable)
- Start the server (this will run bootstrap if needed and apply db migrations):
`$ make run`
- Start the celery workers:
`$ make start-celery`
- Start celery beat:
`$ make start-celery-beat`
- Frontend:
- Build and watch plugin:
`$ make watch-plugin`
- Generate invitation token:
`$ make manage ARGS="issue_invite_for_the_frontend --override"`
- Follow instructions above to setup plugin (see steps 5 and 6)
- Other useful targets:
- `$ make shell` (open Django shell)
- `$ make dbshell` (open DB shell)
- `$ make test` (run tests)
- `$ make lint` (run lint checks)
### Slack application setup
For Slack app configuration check our docs: https://grafana.com/docs/grafana-cloud/oncall/open-source/#slack-setup

6
README.md
View file

@ -20,7 +20,7 @@ Developer-friendly incident response with brilliant Slack integration.
We prepared multiple environments: [production](https://grafana.com/docs/grafana-cloud/oncall/open-source/#production-environment), [developer](DEVELOPER.md) and hobby:
1. Download docker-compose.yaml:
1. Download [`docker-compose.yml`](docker-compose.yml):
```bash
curl -fsSL https://raw.githubusercontent.com/grafana/oncall/dev/docker-compose.yml -o docker-compose.yml
@ -31,9 +31,7 @@ curl -fsSL https://raw.githubusercontent.com/grafana/oncall/dev/docker-compose.y
```bash
echo "DOMAIN=http://localhost:8080
COMPOSE_PROFILES=with_grafana # Remove this line if you want to use existing grafana
SECRET_KEY=my_random_secret_must_be_more_than_32_characters_long
RABBITMQ_PASSWORD=rabbitmq_secret_pw
MYSQL_PASSWORD=mysql_secret_pw" > .env
SECRET_KEY=my_random_secret_must_be_more_than_32_characters_long" > .env
```
3. Launch services:

162
docker-compose-mysql-rabbitmq.yml Normal file
View file

@ -0,0 +1,162 @@
version: "3.8"
x-environment:
&oncall-environment
BASE_URL: $DOMAIN
SECRET_KEY: $SECRET_KEY
RABBITMQ_USERNAME: "rabbitmq"
RABBITMQ_PASSWORD: $RABBITMQ_PASSWORD
RABBITMQ_HOST: "rabbitmq"
RABBITMQ_PORT: "5672"
RABBITMQ_DEFAULT_VHOST: "/"
MYSQL_PASSWORD: $MYSQL_PASSWORD
MYSQL_DB_NAME: oncall_hobby
MYSQL_USER: ${MYSQL_USER:-root}
MYSQL_HOST: ${MYSQL_HOST:-mysql}
MYSQL_PORT: 3306
REDIS_URI: redis://redis:6379/0
DJANGO_SETTINGS_MODULE: settings.hobby
CELERY_WORKER_QUEUE: "default,critical,long,slack,telegram,webhook,retry,celery"
CELERY_WORKER_CONCURRENCY: "1"
CELERY_WORKER_MAX_TASKS_PER_CHILD: "100"
CELERY_WORKER_SHUTDOWN_INTERVAL: "65m"
CELERY_WORKER_BEAT_ENABLED: "True"
services:
engine:
image: grafana/oncall
restart: always
ports:
- "8080:8080"
command: >
sh -c "uwsgi --ini uwsgi.ini"
environment: *oncall-environment
depends_on:
mysql:
condition: service_healthy
oncall_db_migration:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
redis:
condition: service_started
celery:
image: grafana/oncall
restart: always
command: sh -c "./celery_with_exporter.sh"
environment: *oncall-environment
depends_on:
mysql:
condition: service_healthy
oncall_db_migration:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
redis:
condition: service_started
oncall_db_migration:
image: grafana/oncall
command: python manage.py migrate --noinput
environment: *oncall-environment
depends_on:
mysql:
condition: service_healthy
rabbitmq:
condition: service_healthy
mysql:
image: mysql:5.7
platform: linux/x86_64
command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
restart: always
expose:
- 3306
volumes:
- dbdata:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: $MYSQL_PASSWORD
MYSQL_DATABASE: oncall_hobby
deploy:
resources:
limits:
memory: 500m
cpus: '0.5'
healthcheck:
test: "mysql -uroot -p$MYSQL_PASSWORD oncall_hobby -e 'select 1'"
timeout: 20s
retries: 10
redis:
image: redis
restart: always
expose:
- 6379
deploy:
resources:
limits:
memory: 100m
cpus: '0.1'
rabbitmq:
image: "rabbitmq:3.7.15-management"
restart: always
hostname: rabbitmq
volumes:
- rabbitmqdata:/var/lib/rabbitmq
environment:
RABBITMQ_DEFAULT_USER: "rabbitmq"
RABBITMQ_DEFAULT_PASS: $RABBITMQ_PASSWORD
RABBITMQ_DEFAULT_VHOST: "/"
deploy:
resources:
limits:
memory: 1000m
cpus: '0.5'
healthcheck:
test: rabbitmq-diagnostics -q ping
interval: 30s
timeout: 30s
retries: 3
mysql_to_create_grafana_db:
image: mysql:5.7
platform: linux/x86_64
command: bash -c "mysql -h ${MYSQL_HOST:-mysql} -uroot -p${MYSQL_PASSWORD:?err} -e 'CREATE DATABASE IF NOT EXISTS grafana CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;'"
depends_on:
mysql:
condition: service_healthy
profiles:
- with_grafana
grafana:
image: "grafana/grafana:9.0.0-beta3"
restart: always
ports:
- "3000:3000"
environment:
GF_DATABASE_TYPE: mysql
GF_DATABASE_HOST: ${MYSQL_HOST:-mysql}
GF_DATABASE_USER: ${MYSQL_USER:-root}
GF_DATABASE_PASSWORD: ${MYSQL_PASSWORD:?err}
GF_SECURITY_ADMIN_USER: ${GRAFANA_USER:-admin}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-admin}
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: grafana-oncall-app
GF_INSTALL_PLUGINS: grafana-oncall-app
deploy:
resources:
limits:
memory: 500m
cpus: '0.5'
depends_on:
mysql_to_create_grafana_db:
condition: service_completed_successfully
mysql:
condition: service_healthy
profiles:
- with_grafana
volumes:
dbdata:
rabbitmqdata:

110
docker-compose.yml
View file

@ -2,18 +2,10 @@ version: "3.8"
x-environment:
&oncall-environment
DATABASE_TYPE: sqlite3
BROKER_TYPE: redis
BASE_URL: $DOMAIN
SECRET_KEY: $SECRET_KEY
RABBITMQ_USERNAME: "rabbitmq"
RABBITMQ_PASSWORD: $RABBITMQ_PASSWORD
RABBITMQ_HOST: "rabbitmq"
RABBITMQ_PORT: "5672"
RABBITMQ_DEFAULT_VHOST: "/"
MYSQL_PASSWORD: $MYSQL_PASSWORD
MYSQL_DB_NAME: oncall_hobby
MYSQL_USER: ${MYSQL_USER:-root}
MYSQL_HOST: ${MYSQL_HOST:-mysql}
MYSQL_PORT: 3306
REDIS_URI: redis://redis:6379/0
DJANGO_SETTINGS_MODULE: settings.hobby
CELERY_WORKER_QUEUE: "default,critical,long,slack,telegram,webhook,retry,celery"
@ -31,104 +23,54 @@ services:
command: >
sh -c "uwsgi --ini uwsgi.ini"
environment: *oncall-environment
volumes:
- oncall_data:/var/lib/oncall
depends_on:
mysql:
condition: service_healthy
oncall_db_migration:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
redis:
condition: service_started
condition: service_healthy
celery:
image: grafana/oncall
restart: always
command: sh -c "./celery_with_exporter.sh"
environment: *oncall-environment
volumes:
- oncall_data:/var/lib/oncall
depends_on:
mysql:
condition: service_healthy
oncall_db_migration:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
redis:
condition: service_started
condition: service_healthy
oncall_db_migration:
image: grafana/oncall
command: python manage.py migrate --noinput
environment: *oncall-environment
depends_on:
mysql:
condition: service_healthy
rabbitmq:
condition: service_healthy
mysql:
image: mysql:5.7
platform: linux/x86_64
command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
restart: always
expose:
- 3306
volumes:
- dbdata:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: $MYSQL_PASSWORD
MYSQL_DATABASE: oncall_hobby
deploy:
resources:
limits:
memory: 500m
cpus: '0.5'
healthcheck:
test: "mysql -uroot -p$MYSQL_PASSWORD oncall_hobby -e 'select 1'"
timeout: 20s
retries: 10
- oncall_data:/var/lib/oncall
depends_on:
redis:
condition: service_healthy
redis:
image: redis
restart: always
expose:
- 6379
deploy:
resources:
limits:
memory: 100m
cpus: '0.1'
rabbitmq:
image: "rabbitmq:3.7.15-management"
restart: always
hostname: rabbitmq
volumes:
- rabbitmqdata:/var/lib/rabbitmq
environment:
RABBITMQ_DEFAULT_USER: "rabbitmq"
RABBITMQ_DEFAULT_PASS: $RABBITMQ_PASSWORD
RABBITMQ_DEFAULT_VHOST: "/"
- redis_data:/data
deploy:
resources:
limits:
memory: 1000m
memory: 500m
cpus: '0.5'
healthcheck:
test: rabbitmq-diagnostics -q ping
interval: 30s
timeout: 30s
retries: 3
mysql_to_create_grafana_db:
image: mysql:5.7
platform: linux/x86_64
command: bash -c "mysql -h ${MYSQL_HOST:-mysql} -uroot -p${MYSQL_PASSWORD:?err} -e 'CREATE DATABASE IF NOT EXISTS grafana CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;'"
depends_on:
mysql:
condition: service_healthy
profiles:
- with_grafana
test: ["CMD", "redis-cli", "ping"]
timeout: 5s
interval: 5s
retries: 10
grafana:
image: "grafana/grafana:9.0.0-beta3"
@ -136,27 +78,21 @@ services:
ports:
- "3000:3000"
environment:
GF_DATABASE_TYPE: mysql
GF_DATABASE_HOST: ${MYSQL_HOST:-mysql}
GF_DATABASE_USER: ${MYSQL_USER:-root}
GF_DATABASE_PASSWORD: ${MYSQL_PASSWORD:?err}
GF_SECURITY_ADMIN_USER: ${GRAFANA_USER:-admin}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-admin}
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: grafana-oncall-app
GF_INSTALL_PLUGINS: grafana-oncall-app
volumes:
- grafana_data:/var/lib/grafana
deploy:
resources:
limits:
memory: 500m
cpus: '0.5'
depends_on:
mysql_to_create_grafana_db:
condition: service_completed_successfully
mysql:
condition: service_healthy
profiles:
- with_grafana
volumes:
dbdata:
rabbitmqdata:
grafana_data:
oncall_data:
redis_data:

11
docs/sources/open-source/_index.md
View file

@ -187,3 +187,14 @@ Grafana OnCall supports Twilio SMS and phone call notifications delivery. If you
1. Set `GRAFANA_CLOUD_NOTIFICATIONS_ENABLED` as **False** to ensure the Grafana OSS <-> Cloud connector is disabled.
1. From your **OnCall** environment, select **Env Variables** and configure all variables starting with `TWILIO_`.
## Email Setup
Grafana OnCall is capable of sending emails using SMTP as a user notification step. To setup email notifications, populate the following env variables with your SMTP server credentials:
- `EMAIL_HOST` - SMTP server host
- `EMAIL_HOST_USER` - SMTP server user
- `EMAIL_HOST_PASSWORD` - SMTP server password
- `EMAIL_PORT` (default is `587`) - SMTP server port
- `EMAIL_USE_TLS` (default is `True`) - to enable/disable TLS
After enabling the email integration, it will be possible to use the `Notify by email` notification step in user settings.

42
engine/apps/alerts/incident_appearance/renderers/email_renderer.py
View file

@ -1,42 +0,0 @@
from django.template.loader import render_to_string
from apps.alerts.incident_appearance.renderers.base_renderer import AlertBaseRenderer, AlertGroupBaseRenderer
from apps.alerts.incident_appearance.renderers.constants import DEFAULT_BACKUP_TITLE
from apps.alerts.incident_appearance.templaters import AlertEmailTemplater
from common.utils import str_or_backup
class AlertEmailRenderer(AlertBaseRenderer):
@property
def templater_class(self):
return AlertEmailTemplater
class AlertGroupEmailRenderer(AlertGroupBaseRenderer):
@property
def alert_renderer_class(self):
return AlertEmailRenderer
def render(self, limit_notification=False):
subject = "You are invited to check an incident from Grafana OnCall"
templated_alert = self.alert_renderer.templated_alert
title_fallback = (
f"#{self.alert_group.inside_organization_number} "
f"{DEFAULT_BACKUP_TITLE} via {self.alert_group.channel.verbal_name}"
)
content = render_to_string(
"email_notification.html",
{
"url": self.alert_group.slack_permalink or self.alert_group.web_link,
"title": str_or_backup(templated_alert.title, title_fallback),
"message": str_or_backup(templated_alert.message, ""), # not render message it all if smth go wrong
"amixr_team": self.alert_group.channel.organization,
"alert_channel": self.alert_group.channel.short_name,
"limit_notification": limit_notification,
"emails_left": self.alert_group.channel.organization.emails_left,
},
)
return subject, content

6
engine/apps/alerts/incident_appearance/renderers/telegram_renderer.py
View file

@ -49,8 +49,10 @@ class AlertGroupTelegramRenderer(AlertGroupBaseRenderer):
status_verbose = self.alert_group.get_resolve_text()
elif self.alert_group.acknowledged:
status_verbose = self.alert_group.get_acknowledge_text()
text = f"{status_emoji} #{self.alert_group.inside_organization_number}, {title}\n"
# First line in the invisible link with id of organization.
# It is needed to add info about organization to the telegram message for the oncall-gateway.
text = f"<a href='{self.alert_group.channel.organization.web_link_with_id}'>&#8205;</a>"
text += f"{status_emoji} #{self.alert_group.inside_organization_number}, {title}\n"
text += f"{status_verbose}, alerts: {alerts_count_str}\n"
text += f"Source: {self.alert_group.channel.short_name}\n"
text += f"{self.alert_group.web_link}"

1
engine/apps/alerts/incident_appearance/templaters/__init__.py
View file

@ -1,6 +1,5 @@
from .alert_templater import TemplateLoader # noqa: F401
from .classic_markdown_templater import AlertClassicMarkdownTemplater # noqa: F401
from .email_templater import AlertEmailTemplater # noqa: F401
from .phone_call_templater import AlertPhoneCallTemplater # noqa: F401
from .slack_templater import AlertSlackTemplater # noqa: F401
from .sms_templater import AlertSmsTemplater # noqa: F401

18
engine/apps/alerts/incident_appearance/templaters/email_templater.py
View file

@ -1,18 +0,0 @@
from apps.alerts.incident_appearance.templaters.alert_templater import AlertTemplater
class AlertEmailTemplater(AlertTemplater):
RENDER_FOR_EMAIL = "email"
def _render_for(self):
return self.RENDER_FOR_EMAIL
def _postformat(self, templated_alert):
templated_alert.title = self._slack_format_for_email(templated_alert.title)
templated_alert.message = self._slack_format_for_email(templated_alert.message)
return templated_alert
def _slack_format_for_email(self, data):
sf = self.slack_formatter
sf.hyperlink_mention_format = "{title} - {url}"
return sf.format(data)

3
engine/apps/alerts/incident_log_builder/incident_log_builder.py
View file

@ -589,9 +589,6 @@ class IncidentLogBuilder:
result += f"call {user_verbal} by phone"
elif notification_policy.notify_by == UserNotificationPolicy.NotificationChannel.TELEGRAM:
result += f"send telegram message to {user_verbal}"
# TODO: restore email notifications
# elif notification_policy.notify_by == UserNotificationPolicy.NotificationChannel.EMAIL:
# result += f"send email to {user_verbal}"
else:
try:
backend_id = UserNotificationPolicy.NotificationChannel(notification_policy.notify_by).name

2
engine/apps/alerts/integration_options_mixin.py
View file

@ -59,8 +59,6 @@ class IntegrationOptionsMixin:
"web_title",
"web_message",
"web_image_url",
"email_title",
"email_message",
"sms_title",
"phone_call_title",
"telegram_title",

6
engine/apps/alerts/migrations/0005_alertgroup_cached_render_for_web.py
View file

@ -1,6 +1,6 @@
# Generated by Django 3.2.13 on 2022-07-20 09:04
from django.db import migrations, models, OperationalError
from django.db import migrations, models, OperationalError, ProgrammingError
class AddFieldIfNotExists(migrations.AddField):
@ -14,6 +14,9 @@ class AddFieldIfNotExists(migrations.AddField):
super().database_forwards(app_label, schema_editor, from_state, to_state)
except OperationalError:
pass
except ProgrammingError as e: # ignore if the field already exists
if "already exists" in str(e):
pass
def database_backwards(self, app_label, schema_editor, from_state, to_state):
pass
@ -27,6 +30,7 @@ class Migration(migrations.Migration):
it will recreate these fields.
"""
atomic = False
dependencies = [
('alerts', '0004_auto_20220711_1106'),
]

29
engine/apps/alerts/models/alert_receive_channel.py
View file

@ -21,7 +21,6 @@ from apps.alerts.integration_options_mixin import IntegrationOptionsMixin
from apps.alerts.models.maintainable_object import MaintainableObject
from apps.alerts.tasks import disable_maintenance, sync_grafana_alerting_contact_points
from apps.base.messaging import get_messaging_backend_from_id
from apps.base.utils import live_settings
from apps.integrations.metadata import heartbeat
from apps.integrations.tasks import create_alert, create_alertmanager_alerts
from apps.slack.constants import SLACK_RATE_LIMIT_DELAY, SLACK_RATE_LIMIT_TIMEOUT
@ -162,8 +161,10 @@ class AlertReceiveChannel(IntegrationOptionsMixin, MaintainableObject):
web_message_template = models.TextField(null=True, default=None)
web_image_url_template = models.TextField(null=True, default=None)
email_title_template = models.TextField(null=True, default=None)
email_message_template = models.TextField(null=True, default=None)
# email related fields are deprecated in favour of messaging backend based templates
# these templates are stored in the messaging_backends_templates field
email_title_template = models.TextField(null=True, default=None) # deprecated
email_message_template = models.TextField(null=True, default=None) # deprecated
telegram_title_template = models.TextField(null=True, default=None)
telegram_message_template = models.TextField(null=True, default=None)
@ -194,10 +195,6 @@ class AlertReceiveChannel(IntegrationOptionsMixin, MaintainableObject):
"phone_call": {
"title": "phone_call_title_template",
},
"email": {
"title": "email_title_template",
"message": "email_message_template",
},
"telegram": {
"title": "telegram_title_template",
"message": "telegram_message_template",
@ -388,10 +385,19 @@ class AlertReceiveChannel(IntegrationOptionsMixin, MaintainableObject):
organization=kwargs["organization"],
integration=kwargs["integration"],
team=kwargs["team"],
deleted_at=None,
)
except cls.DoesNotExist:
kwargs.update(defaults)
alert_receive_channel = cls.create(**kwargs)
except cls.MultipleObjectsReturned:
# general team may inherit integrations from deleted teams
alert_receive_channel = cls.objects.filter(
organization=kwargs["organization"],
integration=kwargs["integration"],
team=kwargs["team"],
deleted_at=None,
).first()
return alert_receive_channel
@property
@ -438,7 +444,8 @@ class AlertReceiveChannel(IntegrationOptionsMixin, MaintainableObject):
@property
def inbound_email(self):
return f"{self.token}@{live_settings.SENDGRID_INBOUND_EMAIL_DOMAIN}"
# todo: implement inbound emails
pass
@property
def default_channel_filter(self):
@ -461,10 +468,6 @@ class AlertReceiveChannel(IntegrationOptionsMixin, MaintainableObject):
"message": self.web_message_template,
"image_url": self.web_image_url_template,
},
"email": {
"title": self.email_title_template,
"message": self.email_message_template,
},
"sms": {
"title": self.sms_title_template,
},
@ -620,8 +623,6 @@ class AlertReceiveChannel(IntegrationOptionsMixin, MaintainableObject):
"web_title": self.web_title_template or "default",
"web_message": self.web_message_template or "default",
"web_image_url_template": self.web_image_url_template or "default",
"email_title_template": self.email_title_template or "default",
"email_message": self.email_message_template or "default",
"telegram_title": self.telegram_title_template or "default",
"telegram_message": self.telegram_message_template or "default",
"telegram_image_url": self.telegram_image_url_template or "default",

6
engine/apps/alerts/models/channel_filter.py
View file

@ -116,7 +116,11 @@ class ChannelFilter(OrderedModel):
return self.is_default or self.check_filter(json.dumps(raw_request_data)) or self.check_filter(str(title))
def check_filter(self, value):
return re.search(self.filtering_term, value)
try:
return re.search(self.filtering_term, value)
except re.error:
logger.error(f"channel_filter={self.id} failed to parse regex={self.filtering_term}")
return False
@property
def slack_channel_id_or_general_log_id(self):

5
engine/apps/alerts/tasks/notify_user.py
View file

@ -228,7 +228,6 @@ def notify_user_task(
def perform_notification(log_record_pk):
SMSMessage = apps.get_model("twilioapp", "SMSMessage")
PhoneCall = apps.get_model("twilioapp", "PhoneCall")
# EmailMessage = apps.get_model("sendgridapp", "EmailMessage") TODO: restore email notifications
UserNotificationPolicy = apps.get_model("base", "UserNotificationPolicy")
TelegramToUserConnector = apps.get_model("telegram", "TelegramToUserConnector")
UserNotificationPolicyLogRecord = apps.get_model("base", "UserNotificationPolicyLogRecord")
@ -280,10 +279,6 @@ def perform_notification(log_record_pk):
elif notification_channel == UserNotificationPolicy.NotificationChannel.TELEGRAM:
TelegramToUserConnector.notify_user(user, alert_group, notification_policy)
# TODO: restore email notifications
# elif notification_channel == UserNotificationPolicy.NotificationChannel.EMAIL:
# EmailMessage.send_incident_mail(user, alert_group, notification_policy)
elif notification_channel == UserNotificationPolicy.NotificationChannel.SLACK:
# TODO: refactor checking the possibility of sending a notification in slack
# Code below is not consistent.

25
engine/apps/alerts/tests/test_alert_receiver_channel.py
View file

@ -144,3 +144,28 @@ def test_notify_maintenance_with_general_channel(make_organization, make_alert_r
mock_post_message.assert_called_once_with(
organization, organization.general_log_channel_id, "maintenance mode enabled"
)
@pytest.mark.django_db
def test_get_or_create_manual_integration_deleted_team(make_organization, make_team, make_alert_receive_channel):
organization = make_organization(general_log_channel_id="CHANNEL-ID")
# setup general manual integration
general_manual = AlertReceiveChannel.get_or_create_manual_integration(
organization=organization, team=None, integration=AlertReceiveChannel.INTEGRATION_MANUAL, defaults={}
)
# setup another team manual integration
team1 = make_team(organization)
team1_manual = AlertReceiveChannel.get_or_create_manual_integration(
organization=organization, team=team1, integration=AlertReceiveChannel.INTEGRATION_MANUAL, defaults={}
)
# team is deleted
team1.delete()
team1_manual.refresh_from_db()
assert team1_manual.team is None
# it should still be possible to get a manual integration for general team
integration = AlertReceiveChannel.get_or_create_manual_integration(
organization=organization, team=None, integration=AlertReceiveChannel.INTEGRATION_MANUAL, defaults={}
)
assert integration == general_manual

3
engine/apps/alerts/tests/test_default_templates.py
View file

@ -2,7 +2,6 @@ import pytest
from jinja2 import TemplateSyntaxError
from apps.alerts.incident_appearance.templaters import (
AlertEmailTemplater,
AlertPhoneCallTemplater,
AlertSlackTemplater,
AlertSmsTemplater,
@ -45,14 +44,12 @@ def test_default_templates(
slack_templater = AlertSlackTemplater(alert)
web_templater = AlertWebTemplater(alert)
sms_templater = AlertSmsTemplater(alert)
email_templater = AlertEmailTemplater(alert)
telegram_templater = AlertTelegramTemplater(alert)
phone_call_templater = AlertPhoneCallTemplater(alert)
templaters = {
"slack": slack_templater,
"web": web_templater,
"sms": sms_templater,
"email": email_templater,
"telegram": telegram_templater,
"phone_call": phone_call_templater,
}

40
engine/apps/api/serializers/alert_receive_channel.py
View file

@ -254,18 +254,6 @@ class AlertReceiveChannelTemplatesSerializer(EagerLoadingMixin, serializers.Mode
validators=[valid_jinja_template_for_serializer_method_field],
required=False,
)
email_title_template = WritableSerializerMethodField(
allow_null=True,
deserializer_field=serializers.CharField(),
validators=[valid_jinja_template_for_serializer_method_field],
required=False,
)
email_message_template = WritableSerializerMethodField(
allow_null=True,
deserializer_field=serializers.CharField(),
validators=[valid_jinja_template_for_serializer_method_field],
required=False,
)
source_link_template = WritableSerializerMethodField(
allow_null=True,
deserializer_field=serializers.CharField(),
@ -306,8 +294,6 @@ class AlertReceiveChannelTemplatesSerializer(EagerLoadingMixin, serializers.Mode
"web_title_template",
"web_message_template",
"web_image_url_template",
"email_title_template",
"email_message_template",
"telegram_title_template",
"telegram_message_template",
"telegram_image_url_template",
@ -391,17 +377,6 @@ class AlertReceiveChannelTemplatesSerializer(EagerLoadingMixin, serializers.Mode
elif default_template is not None and default_template.strip() == value.strip():
self.instance.web_title_template = None
def get_email_title_template(self, obj):
default_template = AlertReceiveChannel.INTEGRATION_TO_DEFAULT_EMAIL_TITLE_TEMPLATE[obj.integration]
return obj.email_title_template or default_template
def set_email_title_template(self, value):
default_template = AlertReceiveChannel.INTEGRATION_TO_DEFAULT_EMAIL_TITLE_TEMPLATE[self.instance.integration]
if default_template is None or default_template.strip() != value.strip():
self.instance.email_title_template = value.strip()
elif default_template is not None and default_template.strip() == value.strip():
self.instance.email_title_template = None
def get_web_message_template(self, obj):
default_template = AlertReceiveChannel.INTEGRATION_TO_DEFAULT_WEB_MESSAGE_TEMPLATE[obj.integration]
return obj.web_message_template or default_template
@ -424,17 +399,6 @@ class AlertReceiveChannelTemplatesSerializer(EagerLoadingMixin, serializers.Mode
elif default_template is not None and default_template.strip() == value.strip():
self.instance.web_image_url_template = None
def get_email_message_template(self, obj):
default_template = AlertReceiveChannel.INTEGRATION_TO_DEFAULT_EMAIL_MESSAGE_TEMPLATE[obj.integration]
return obj.email_message_template or default_template
def set_email_message_template(self, value):
default_template = AlertReceiveChannel.INTEGRATION_TO_DEFAULT_EMAIL_MESSAGE_TEMPLATE[self.instance.integration]
if default_template is None or default_template.strip() != value.strip():
self.instance.email_message_template = value.strip()
elif default_template is not None and default_template.strip() == value.strip():
self.instance.email_message_template = None
def get_telegram_title_template(self, obj):
default_template = AlertReceiveChannel.INTEGRATION_TO_DEFAULT_TELEGRAM_TITLE_TEMPLATE[obj.integration]
return obj.telegram_title_template or default_template
@ -644,8 +608,8 @@ class AlertReceiveChannelTemplatesSerializer(EagerLoadingMixin, serializers.Mode
def _get_messaging_backend_templates(self, obj):
"""Return additional messaging backend templates if any."""
templates = {}
for backend_id, _ in get_messaging_backends():
for field in ("title", "message", "image_url"):
for backend_id, backend in get_messaging_backends():
for field in backend.template_fields:
value = None
if obj.messaging_backends_templates:
value = obj.messaging_backends_templates.get(backend_id, {}).get(field)

31
engine/apps/api/serializers/custom_button.py
View file

@ -1,4 +1,5 @@
import json
from collections import defaultdict
from django.core.validators import URLValidator, ValidationError
from jinja2 import Template, TemplateError
@ -51,15 +52,33 @@ class CustomButtonSerializer(serializers.ModelSerializer):
return None
try:
json.loads(data)
except ValueError:
raise serializers.ValidationError("Data has incorrect format")
try:
Template(data)
template = Template(data)
except TemplateError:
raise serializers.ValidationError("Data has incorrect template")
try:
rendered = template.render(
{
# Validate that the template can be rendered with a JSON-ish alert payload.
# We don't know what the actual payload will be, so we use a defaultdict
# so that attribute access within a template will never fail
# (provided it's only one level deep - we won't accept templates that attempt
# to do nested attribute access).
# Every attribute access should return a string to ensure that users are
# correctly using `tojson` or wrapping fields in strings.
# If we instead used a `defaultdict(dict)` or `defaultdict(lambda: 1)` we
# would accidentally accept templates such as `{"name": {{ alert_payload.name }}}`
# which would then fail at the true render time due to the
# lack of explicit quotes around the template variable; this would render
# as `{"name": some_alert_name}` which is not valid JSON.
"alert_payload": defaultdict(str),
"alert_group_id": "abcd",
}
)
json.loads(rendered)
except ValueError:
raise serializers.ValidationError("Data has incorrect format")
return data
def validate_forward_whole_payload(self, data):

3
engine/apps/api/serializers/schedule_base.py
View file

@ -76,7 +76,8 @@ class ScheduleBaseSerializer(EagerLoadingMixin, serializers.ModelSerializer):
def get_number_of_escalation_chains(self, obj):
# num_escalation_chains param added in queryset via annotate. Check ScheduleView.get_queryset
# return 0 for just created schedules
return getattr(obj, "num_escalation_chains", 0)
num = getattr(obj, "num_escalation_chains", 0)
return num or 0
def validate(self, attrs):
if "slack_channel_id" in attrs:

4
engine/apps/api/tests/test_alert_group.py
View file

@ -1443,7 +1443,7 @@ def test_alert_group_preview_body_non_existent_template_var(
client = APIClient()
url = reverse("api-internal:alertgroup-preview-template", kwargs={"pk": alert_group.public_primary_key})
data = {"template_name": "email_title_template", "template_body": "foobar: {{ foobar.does_not_exist }}"}
data = {"template_name": "testonly_title_template", "template_body": "foobar: {{ foobar.does_not_exist }}"}
response = client.post(url, data, format="json", **make_user_auth_headers(user, token))
assert response.status_code == status.HTTP_200_OK
@ -1465,7 +1465,7 @@ def test_alert_group_preview_body_invalid_template_syntax(
client = APIClient()
url = reverse("api-internal:alertgroup-preview-template", kwargs={"pk": alert_group.public_primary_key})
data = {"template_name": "email_title_template", "template_body": "{{'' if foo is None else foo}}"}
data = {"template_name": "testonly_title_template", "template_body": "{{'' if foo is None else foo}}"}
response = client.post(url, data, format="json", **make_user_auth_headers(user, token))
assert response.status_code == status.HTTP_400_BAD_REQUEST

3
engine/apps/api/tests/test_alert_receive_channel_template.py
View file

@ -6,6 +6,7 @@ from rest_framework import status
from rest_framework.response import Response
from rest_framework.test import APIClient
from apps.base.messaging import BaseMessagingBackend
from common.constants.role import Role
@ -224,7 +225,7 @@ def test_update_alert_receive_channel_backend_template_update_values(
# patch messaging backends to add OTHER as a valid backend
with patch(
"apps.api.serializers.alert_receive_channel.get_messaging_backends",
return_value=[("TESTONLY", None), ("OTHER", None)],
return_value=[("TESTONLY", BaseMessagingBackend), ("OTHER", BaseMessagingBackend)],
):
response = client.put(
url, format="json", data={"testonly_title_template": "updated-title"}, **make_user_auth_headers(user, token)

101
engine/apps/api/tests/test_custom_button.py
View file

@ -128,6 +128,91 @@ def test_create_valid_data_button(custom_button_internal_api_setup, make_user_au
assert response.json() == expected_response
@pytest.mark.django_db
def test_create_valid_nested_data_button(custom_button_internal_api_setup, make_user_auth_headers):
user, token, custom_button = custom_button_internal_api_setup
client = APIClient()
url = reverse("api-internal:custom_button-list")
data = {
"name": "amixr_button_with_valid_data",
"webhook": TEST_URL,
# Assert that nested field access still works as long as the variable
# is quoted, making it valid JSON.
# This ensures backwards compatibility from when templates were required
# to be JSON.
"data": '{"nested_item": "{{ alert_payload.foo.bar }}"}',
"team": None,
}
response = client.post(url, data, format="json", **make_user_auth_headers(user, token))
# modify initial data by adding id and None for optional fields
custom_button = CustomButton.objects.get(public_primary_key=response.data["id"])
expected_response = data | {
"id": custom_button.public_primary_key,
"user": None,
"password": None,
"authorization_header": None,
"forward_whole_payload": False,
}
assert response.status_code == status.HTTP_201_CREATED
assert response.json() == expected_response
@pytest.mark.django_db
def test_create_valid_data_after_render_button(custom_button_internal_api_setup, make_user_auth_headers):
user, token, custom_button = custom_button_internal_api_setup
client = APIClient()
url = reverse("api-internal:custom_button-list")
data = {
"name": "amixr_button_with_valid_data",
"webhook": TEST_URL,
"data": '{"name": "{{ alert_payload.name }}", "labels": {{ alert_payload.labels | tojson }}}',
"team": None,
}
response = client.post(url, data, format="json", **make_user_auth_headers(user, token))
# modify initial data by adding id and None for optional fields
custom_button = CustomButton.objects.get(public_primary_key=response.data["id"])
expected_response = data | {
"id": custom_button.public_primary_key,
"user": None,
"password": None,
"authorization_header": None,
"forward_whole_payload": False,
}
assert response.status_code == status.HTTP_201_CREATED
assert response.json() == expected_response
@pytest.mark.django_db
def test_create_valid_data_after_render_use_all_data_button(custom_button_internal_api_setup, make_user_auth_headers):
user, token, custom_button = custom_button_internal_api_setup
client = APIClient()
url = reverse("api-internal:custom_button-list")
data = {
"name": "amixr_button_with_valid_data",
"webhook": TEST_URL,
"data": "{{ alert_payload | tojson }}",
"team": None,
}
response = client.post(url, data, format="json", **make_user_auth_headers(user, token))
# modify initial data by adding id and None for optional fields
custom_button = CustomButton.objects.get(public_primary_key=response.data["id"])
expected_response = data | {
"id": custom_button.public_primary_key,
"user": None,
"password": None,
"authorization_header": None,
"forward_whole_payload": False,
}
assert response.status_code == status.HTTP_201_CREATED
assert response.json() == expected_response
@pytest.mark.django_db
def test_create_invalid_url_custom_button(custom_button_internal_api_setup, make_user_auth_headers):
user, token, custom_button = custom_button_internal_api_setup
@ -157,6 +242,22 @@ def test_create_invalid_data_custom_button(custom_button_internal_api_setup, mak
assert response.status_code == status.HTTP_400_BAD_REQUEST
@pytest.mark.django_db
def test_create_invalid_templated_data_custom_button(custom_button_internal_api_setup, make_user_auth_headers):
user, token, custom_button = custom_button_internal_api_setup
client = APIClient()
url = reverse("api-internal:custom_button-list")
data = {
"name": "amixr_button_invalid_data",
"webhook": TEST_URL,
# This would need a `| tojson` or some double quotes around it to pass validation.
"data": "{{ alert_payload.name }}",
}
response = client.post(url, data, format="json", **make_user_auth_headers(user, token))
assert response.status_code == status.HTTP_400_BAD_REQUEST
@pytest.mark.django_db
def test_update_custom_button(custom_button_internal_api_setup, make_user_auth_headers):
user, token, custom_button = custom_button_internal_api_setup

4
engine/apps/api/views/organization.py
View file

@ -59,7 +59,9 @@ class GetTelegramVerificationCode(APIView):
telegram_client = TelegramClient()
bot_username = telegram_client.api_client.username
bot_link = f"https://t.me/{bot_username}"
return Response({"telegram_code": str(new_code.uuid), "bot_link": bot_link}, status=status.HTTP_200_OK)
return Response(
{"telegram_code": str(new_code.uuid_with_org_id), "bot_link": bot_link}, status=status.HTTP_200_OK
)
class GetChannelVerificationCode(APIView):

22
engine/apps/api/views/schedule.py
View file

@ -13,7 +13,7 @@ from rest_framework.permissions import IsAuthenticated
from rest_framework.views import Response
from rest_framework.viewsets import ModelViewSet
from apps.alerts.models import EscalationChain
from apps.alerts.models import EscalationChain, EscalationPolicy
from apps.api.permissions import MODIFY_ACTIONS, READ_ACTIONS, ActionPermission, AnyRole, IsAdmin, IsAdminOrEditor
from apps.api.serializers.schedule_base import ScheduleFastSerializer
from apps.api.serializers.schedule_polymorphic import (
@ -108,22 +108,28 @@ class ScheduleView(
slack_team_identity=organization.slack_team_identity,
slack_id=OuterRef("channel"),
)
escalation_policies = (
EscalationPolicy.objects.values("notify_schedule")
.order_by("notify_schedule")
.annotate(num_escalation_chains=Count("notify_schedule"))
.filter(notify_schedule=OuterRef("id"))
)
queryset = queryset.annotate(
slack_channel_name=Subquery(slack_channels.values("name")[:1]),
slack_channel_pk=Subquery(slack_channels.values("public_primary_key")[:1]),
num_escalation_chains=Count(
"escalation_policies__escalation_chain",
distinct=True,
),
num_escalation_chains=Subquery(escalation_policies.values("num_escalation_chains")[:1]),
)
return queryset
def get_queryset(self):
is_short_request = self.request.query_params.get("short", "false") == "true"
organization = self.request.auth.organization
queryset = OnCallSchedule.objects.filter(
organization=organization,
team=self.request.user.current_team,
queryset = OnCallSchedule.objects.filter(organization=organization, team=self.request.user.current_team).defer(
# avoid requesting large text fields which are not used when listing schedules
"cached_ical_file_primary",
"prev_ical_file_primary",
"cached_ical_file_overrides",
"prev_ical_file_overrides",
)
if not is_short_request:
queryset = self._annotate_queryset(queryset)

4
engine/apps/api/views/user.py
View file

@ -374,7 +374,9 @@ class UserView(
bot_username = telegram_client.api_client.username
bot_link = f"https://t.me/{bot_username}"
return Response({"telegram_code": str(new_code.uuid), "bot_link": bot_link}, status=status.HTTP_200_OK)
return Response(
{"telegram_code": str(new_code.uuid_with_org_id), "bot_link": bot_link}, status=status.HTTP_200_OK
)
@action(detail=True, methods=["post"])
def unlink_slack(self, request, pk):

5
engine/apps/api/views/user_notification_policy.py
View file

@ -160,9 +160,6 @@ class UserNotificationPolicyView(UpdateSerializerMixin, ModelViewSet):
notification_channel
in NotificationChannelAPIOptions.TELEGRAM_INTEGRATION_REQUIRED_NOTIFICATION_CHANNELS
)
email_integration_required = (
notification_channel in NotificationChannelAPIOptions.EMAIL_INTEGRATION_REQUIRED_NOTIFICATION_CHANNELS
)
mobile_app_integration_required = (
notification_channel
in NotificationChannelAPIOptions.MOBILE_APP_INTEGRATION_REQUIRED_NOTIFICATION_CHANNELS
@ -171,8 +168,6 @@ class UserNotificationPolicyView(UpdateSerializerMixin, ModelViewSet):
continue
if telegram_integration_required and not settings.FEATURE_TELEGRAM_INTEGRATION_ENABLED:
continue
if email_integration_required and not settings.FEATURE_EMAIL_INTEGRATION_ENABLED:
continue
if mobile_app_integration_required and not settings.MOBILE_APP_PUSH_NOTIFICATIONS_ENABLED:
continue

2
engine/apps/base/messaging.py
View file

@ -7,7 +7,9 @@ class BaseMessagingBackend:
label = "The Backend"
short_label = "Backend"
available_for_use = False
templater = None
template_fields = ("title", "message", "image_url")
def __init__(self, *args, **kwargs):
self.notification_channel_id = kwargs.get("notification_channel_id")

23
engine/apps/base/models/live_setting.py
View file

@ -33,6 +33,11 @@ class LiveSetting(models.Model):
error = models.TextField(null=True, default=None)
AVAILABLE_NAMES = (
"EMAIL_HOST",
"EMAIL_PORT",
"EMAIL_HOST_USER",
"EMAIL_HOST_PASSWORD",
"EMAIL_USE_TLS",
"TWILIO_ACCOUNT_SID",
"TWILIO_AUTH_TOKEN",
"TWILIO_NUMBER",
@ -51,6 +56,11 @@ class LiveSetting(models.Model):
)
DESCRIPTIONS = {
"EMAIL_HOST": "SMTP server host. This email server will be used to notify users via email.",
"EMAIL_PORT": "SMTP server port",
"EMAIL_HOST_USER": "SMTP server user",
"EMAIL_HOST_PASSWORD": "SMTP server password",
"EMAIL_USE_TLS": "SMTP enable/disable TLS",
"SLACK_SIGNING_SECRET": (
"Check <a href='"
"https://grafana.com/docs/grafana-cloud/oncall/open-source/#slack-setup"
@ -98,16 +108,6 @@ class LiveSetting(models.Model):
"You can create a service in Twilio web interface. "
"twilio.com -> verify -> create new service."
),
"SENDGRID_API_KEY": (
"Sendgrid api key to send emails, "
"<a href='https://sendgrid.com/docs/ui/account-and-settings/api-keys/' target='_blank'>more info</a>."
),
"SENDGRID_FROM_EMAIL": (
"Address to send emails, <a href='https://sendgrid.com/docs/ui/sending-email/senders/' target='_blank'>"
"more info</a>."
),
"SENDGRID_SECRET_KEY": "It is the secret key to secure receiving inbound emails.",
"SENDGRID_INBOUND_EMAIL_DOMAIN": "Domain to receive emails for inbound emails integration.",
"TELEGRAM_TOKEN": (
"Secret token for Telegram bot, you can get one via <a href='https://t.me/BotFather' target='_blank'>BotFather</a>."
),
@ -126,11 +126,10 @@ class LiveSetting(models.Model):
}
SECRET_SETTING_NAMES = (
"EMAIL_HOST_PASSWORD",
"TWILIO_ACCOUNT_SID",
"TWILIO_AUTH_TOKEN",
"TWILIO_VERIFY_SERVICE_SID",
"SENDGRID_API_KEY",
"SENDGRID_SECRET_KEY",
"SLACK_CLIENT_OAUTH_ID",
"SLACK_CLIENT_OAUTH_SECRET",
"SLACK_SIGNING_SECRET",

5
engine/apps/base/models/user_notification_policy.py
View file

@ -35,7 +35,6 @@ BUILT_IN_BACKENDS = (
("SMS", 1),
("PHONE_CALL", 2),
("TELEGRAM", 3),
("EMAIL", 4),
("MOBILE_PUSH_GENERAL", 5),
("MOBILE_PUSH_CRITICAL", 6),
)
@ -214,7 +213,6 @@ class NotificationChannelOptions:
SLACK_INTEGRATION_REQUIRED_NOTIFICATION_CHANNELS = [UserNotificationPolicy.NotificationChannel.SLACK]
TELEGRAM_INTEGRATION_REQUIRED_NOTIFICATION_CHANNELS = [UserNotificationPolicy.NotificationChannel.TELEGRAM]
EMAIL_INTEGRATION_REQUIRED_NOTIFICATION_CHANNELS = [UserNotificationPolicy.NotificationChannel.EMAIL]
MOBILE_APP_INTEGRATION_REQUIRED_NOTIFICATION_CHANNELS = [
UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_GENERAL,
UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_CRITICAL,
@ -227,7 +225,6 @@ class NotificationChannelAPIOptions(NotificationChannelOptions):
UserNotificationPolicy.NotificationChannel.SMS: "SMS \U00002709\U0001F4F2",
UserNotificationPolicy.NotificationChannel.PHONE_CALL: "Phone call \U0000260E",
UserNotificationPolicy.NotificationChannel.TELEGRAM: "Telegram \U0001F916",
UserNotificationPolicy.NotificationChannel.EMAIL: "Email \U0001F4E8",
UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_GENERAL: "Mobile App",
UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_CRITICAL: "Mobile App Critical",
}
@ -243,7 +240,6 @@ class NotificationChannelAPIOptions(NotificationChannelOptions):
UserNotificationPolicy.NotificationChannel.SMS: "SMS",
UserNotificationPolicy.NotificationChannel.PHONE_CALL: "\U0000260E",
UserNotificationPolicy.NotificationChannel.TELEGRAM: "Telegram",
UserNotificationPolicy.NotificationChannel.EMAIL: "Email",
UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_GENERAL: "Mobile App",
UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_CRITICAL: "Mobile App Critical",
}
@ -261,7 +257,6 @@ class NotificationChannelPublicAPIOptions(NotificationChannelAPIOptions):
UserNotificationPolicy.NotificationChannel.SMS: "notify_by_sms",
UserNotificationPolicy.NotificationChannel.PHONE_CALL: "notify_by_phone_call",
UserNotificationPolicy.NotificationChannel.TELEGRAM: "notify_by_telegram",
UserNotificationPolicy.NotificationChannel.EMAIL: "notify_by_email",
UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_GENERAL: "notify_by_mobile_app",
UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_CRITICAL: "notify_by_mobile_app_critical",
}

24
engine/apps/base/models/user_notification_policy_log_record.py
View file

@ -49,14 +49,14 @@ class UserNotificationPolicyLogRecord(models.Model):
ERROR_NOTIFICATION_PHONE_CALLS_LIMIT_EXCEEDED,
ERROR_NOTIFICATION_PHONE_NUMBER_IS_NOT_VERIFIED,
ERROR_NOTIFICATION_NOT_ABLE_TO_SEND_MAIL,
ERROR_NOTIFICATION_MAIL_LIMIT_EXCEEDED,
ERROR_NOTIFICATION_EMAIL_IS_NOT_VERIFIED,
ERROR_NOTIFICATION_MAIL_LIMIT_EXCEEDED, # todo: manage backend specific limits in messaging backend
ERROR_NOTIFICATION_EMAIL_IS_NOT_VERIFIED, # deprecated
ERROR_NOTIFICATION_TELEGRAM_IS_NOT_LINKED_TO_SLACK_ACC,
ERROR_NOTIFICATION_PHONE_CALL_LINE_BUSY,
ERROR_NOTIFICATION_PHONE_CALL_FAILED,
ERROR_NOTIFICATION_PHONE_CALL_NO_ANSWER,
ERROR_NOTIFICATION_SMS_DELIVERY_FAILED,
ERROR_NOTIFICATION_MAIL_DELIVERY_FAILED,
ERROR_NOTIFICATION_MAIL_DELIVERY_FAILED, # deprecated
ERROR_NOTIFICATION_TELEGRAM_BOT_IS_DELETED,
ERROR_NOTIFICATION_POSTING_TO_SLACK_IS_DISABLED,
ERROR_NOTIFICATION_POSTING_TO_TELEGRAM_IS_DISABLED, # deprecated
@ -78,7 +78,6 @@ class UserNotificationPolicyLogRecord(models.Model):
ERROR_NOTIFICATION_PHONE_CALLS_LIMIT_EXCEEDED,
ERROR_NOTIFICATION_MAIL_LIMIT_EXCEEDED,
ERROR_NOTIFICATION_PHONE_NUMBER_IS_NOT_VERIFIED,
ERROR_NOTIFICATION_EMAIL_IS_NOT_VERIFIED,
]
type = models.IntegerField(choices=TYPE_CHOICES)
@ -172,9 +171,6 @@ class UserNotificationPolicyLogRecord(models.Model):
result += f"SMS to {user_verbal} was delivered successfully"
elif notification_channel == UserNotificationPolicy.NotificationChannel.PHONE_CALL:
result += f"phone call to {user_verbal} was successful"
# TODO: restore email notifications
# elif notification_channel == UserNotificationPolicy.NotificationChannel.EMAIL:
# result += f"email to {user_verbal} was delivered successfully"
elif notification_channel is None:
result += f"notification to {user_verbal} was delivered successfully"
elif self.type == UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED:
@ -185,6 +181,7 @@ class UserNotificationPolicyLogRecord(models.Model):
== UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_PHONE_CALLS_LIMIT_EXCEEDED
):
result += f"attempt to call to {user_verbal} has been failed due to a plan limit"
# todo: manage backend specific limits in messaging backend
elif self.notification_error_code == UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_MAIL_LIMIT_EXCEEDED:
result += f"failed to send email to {user_verbal}. Exceeded limit for mails"
elif (
@ -201,10 +198,6 @@ class UserNotificationPolicyLogRecord(models.Model):
result += f"OnCall was not able to send an SMS to {user_verbal}"
elif self.notification_error_code == UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_NOT_ABLE_TO_CALL:
result += f"OnCall was not able to call to {user_verbal}"
elif (
self.notification_error_code == UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_NOT_ABLE_TO_SEND_MAIL
):
result += f"OnCall was not able to send an email to {user_verbal}"
elif (
self.notification_error_code
== UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_POSTING_TO_SLACK_IS_DISABLED
@ -242,10 +235,6 @@ class UserNotificationPolicyLogRecord(models.Model):
result += f"phone call to {user_verbal} ended without being answered"
elif self.notification_error_code == UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_SMS_DELIVERY_FAILED:
result += f"SMS {user_verbal} was not delivered"
elif (
self.notification_error_code == UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_MAIL_DELIVERY_FAILED
):
result += f"email to {user_verbal} was not delivered"
elif self.notification_error_code == UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_IN_SLACK:
result += f"failed to notify {user_verbal} in Slack"
elif (
@ -286,7 +275,7 @@ class UserNotificationPolicyLogRecord(models.Model):
except ValueError:
backend = None
result += (
f"failed to notify {user_verbal} in {backend.label.lower() if backend else 'disabled backend'}"
f"failed to notify {user_verbal} by {backend.label.lower() if backend else 'disabled backend'}"
)
elif self.type == UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_TRIGGERED:
if notification_step == UserNotificationPolicy.Step.NOTIFY:
@ -298,9 +287,6 @@ class UserNotificationPolicyLogRecord(models.Model):
result += f"called {user_verbal} by phone"
elif notification_channel == UserNotificationPolicy.NotificationChannel.TELEGRAM:
result += f"sent telegram message to {user_verbal}"
# TODO: restore email notifications
# elif notification_channel == UserNotificationPolicy.NotificationChannel.EMAIL:
# result += f"sent email to {user_verbal}"
elif notification_channel == UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_GENERAL:
result += f"sent push notifications to {user_verbal}"
elif notification_channel == UserNotificationPolicy.NotificationChannel.MOBILE_PUSH_CRITICAL:

2
engine/apps/base/tests/test_user_notification_policy_log_record.py
View file

@ -31,7 +31,7 @@ def test_extra_messaging_backends_error_log(
)
output = log_record.render_log_line_action()
assert output == f"failed to notify {user_1.username} in {TestOnlyBackend.label.lower()}"
assert output == f"failed to notify {user_1.username} by {TestOnlyBackend.label.lower()}"
@pytest.mark.django_db

23
engine/apps/base/utils.py
View file

@ -5,8 +5,6 @@ from urllib.parse import urlparse
import phonenumbers
from django.apps import apps
from phonenumbers import NumberParseException
from python_http_client import UnauthorizedError
from sendgrid import SendGridAPIClient
from telegram import Bot
from twilio.base.exceptions import TwilioException
from twilio.rest import Client
@ -77,20 +75,6 @@ class LiveSettingValidator:
if not cls._is_phone_number_valid(twilio_number):
return "Please specify a valid phone number in the following format: +XXXXXXXXXXX"
@classmethod
def _check_sendgrid_api_key(cls, sendgrid_api_key):
sendgrid_client = SendGridAPIClient(sendgrid_api_key)
try:
sendgrid_client.client.mail_settings.get()
except Exception as e:
return cls._prettify_sendgrid_error(e)
@classmethod
def _check_sendgrid_from_email(cls, sendgrid_from_email):
if not cls._is_email_valid(sendgrid_from_email):
return "Please specify a valid email"
@classmethod
def _check_slack_install_return_redirect_host(cls, slack_install_return_redirect_host):
scheme = urlparse(slack_install_return_redirect_host).scheme
@ -147,10 +131,3 @@ class LiveSettingValidator:
return f"Twilio error: {exc.args[0]}"
else:
return f"Twilio error: {str(exc)}"
@staticmethod
def _prettify_sendgrid_error(exc):
if isinstance(exc, UnauthorizedError):
return "Sendgrid error: couldn't authorize with given credentials"
else:
return f"Sendgrid error: {str(exc)}"

0
engine/apps/sendgridapp/__init__.py → engine/apps/email/__init__.py
View file

55
engine/apps/email/alert_rendering.py Normal file
View file

@ -0,0 +1,55 @@
from django.template.loader import render_to_string
from emoji.core import emojize
from apps.alerts.incident_appearance.renderers.constants import DEFAULT_BACKUP_TITLE
from apps.alerts.incident_appearance.templaters.alert_templater import AlertTemplater
from common.utils import convert_md_to_html, str_or_backup
class AlertEmailTemplater(AlertTemplater):
RENDER_FOR_EMAIL = "email"
def _render_for(self):
return self.RENDER_FOR_EMAIL
def _postformat(self, templated_alert):
templated_alert.title = self._slack_format_for_email(templated_alert.title)
templated_alert.message = self._slack_format_for_email(templated_alert.message)
return templated_alert
def _slack_format_for_email(self, data):
sf = self.slack_formatter
sf.hyperlink_mention_format = "{title} - {url}"
return sf.format(data)
def build_subject_and_message(alert_group, emails_left):
alert = alert_group.alerts.first()
templated_alert = AlertEmailTemplater(alert).render()
title_fallback = (
f"#{alert_group.inside_organization_number} " f"{DEFAULT_BACKUP_TITLE} via {alert_group.channel.verbal_name}"
)
# default templates are the same as web templates, which are in Markdown format
message = templated_alert.message
if message:
message = convert_md_to_html(templated_alert.message) if templated_alert.message else ""
content = render_to_string(
"email_notification.html",
{
"url": alert_group.slack_permalink or alert_group.web_link,
"title": str_or_backup(templated_alert.title, title_fallback),
"message": str_or_backup(message, ""), # not render message at all if smth goes wrong
"organization": alert_group.channel.organization.org_title,
"integration": emojize(alert_group.channel.short_name, use_aliases=True),
"limit_notification": emails_left <= 20,
"emails_left": emails_left,
},
)
title = str_or_backup(templated_alert.title, title_fallback)
subject = f"[{title}] You are invited to check an alert group"
return subject, content

20
engine/apps/email/backend.py Normal file
View file

@ -0,0 +1,20 @@
from apps.base.messaging import BaseMessagingBackend
from apps.email.tasks import notify_user_async
class EmailBackend(BaseMessagingBackend):
backend_id = "EMAIL"
label = "Email"
short_label = "Email"
available_for_use = True
templater = "apps.email.alert_rendering.AlertEmailTemplater"
template_fields = ("title", "message")
def serialize_user(self, user):
return {"email": user.email}
def notify_user(self, user, alert_group, notification_policy):
notify_user_async.delay(
user_pk=user.pk, alert_group_pk=alert_group.pk, notification_policy_pk=notification_policy.pk
)

31
engine/apps/email/migrations/0001_initial.py Normal file
View file

@ -0,0 +1,31 @@
# Generated by Django 3.2.15 on 2022-10-10 12:06
from django.db import migrations, models
import django.db.models.deletion
import uuid
class Migration(migrations.Migration):
initial = True
dependencies = [
('user_management', '0003_user_hide_phone_number'),
('alerts', '0007_populate_web_title_cache'),
('base', '0003_delete_organizationlogrecord'),
]
operations = [
migrations.CreateModel(
name='EmailMessage',
fields=[
('message_uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
('exceeded_limit', models.BooleanField(default=None, null=True)),
('created_at', models.DateTimeField(auto_now_add=True)),
('notification_policy', models.ForeignKey(default=None, null=True, on_delete=django.db.models.deletion.SET_NULL, to='base.usernotificationpolicy')),
('receiver', models.ForeignKey(default=None, null=True, on_delete=django.db.models.deletion.PROTECT, to='user_management.user')),
('represents_alert', models.ForeignKey(default=None, null=True, on_delete=django.db.models.deletion.SET_NULL, to='alerts.alert')),
('represents_alert_group', models.ForeignKey(default=None, null=True, on_delete=django.db.models.deletion.SET_NULL, to='alerts.alertgroup')),
],
),
]

0
engine/apps/sendgridapp/migrations/__init__.py → engine/apps/email/migrations/__init__.py
View file

20
engine/apps/email/models.py Normal file
View file

@ -0,0 +1,20 @@
import logging
import uuid
from django.db import models
logger = logging.getLogger(__name__)
class EmailMessage(models.Model):
message_uuid = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
exceeded_limit = models.BooleanField(null=True, default=None)
represents_alert = models.ForeignKey("alerts.Alert", on_delete=models.SET_NULL, null=True, default=None)
represents_alert_group = models.ForeignKey("alerts.AlertGroup", on_delete=models.SET_NULL, null=True, default=None)
notification_policy = models.ForeignKey(
"base.UserNotificationPolicy", on_delete=models.SET_NULL, null=True, default=None
)
receiver = models.ForeignKey("user_management.User", on_delete=models.PROTECT, null=True, default=None)
created_at = models.DateTimeField(auto_now_add=True)

99
engine/apps/email/tasks.py Normal file
View file

@ -0,0 +1,99 @@
from socket import gaierror
from celery.utils.log import get_task_logger
from django.conf import settings
from django.core.mail import BadHeaderError, get_connection, send_mail
from django.utils.html import strip_tags
from apps.alerts.models import AlertGroup
from apps.base.utils import live_settings
from apps.email.alert_rendering import build_subject_and_message
from apps.email.models import EmailMessage
from apps.user_management.models import User
from common.custom_celery_tasks import shared_dedicated_queue_retry_task
MAX_RETRIES = 1 if settings.DEBUG else 10
logger = get_task_logger(__name__)
@shared_dedicated_queue_retry_task(autoretry_for=(Exception,), retry_backoff=True, max_retries=MAX_RETRIES)
def notify_user_async(user_pk, alert_group_pk, notification_policy_pk):
# imported here to avoid circular import error
from apps.base.models import UserNotificationPolicy, UserNotificationPolicyLogRecord
try:
user = User.objects.get(pk=user_pk)
except User.DoesNotExist:
logger.warning(f"User {user_pk} does not exist")
return
try:
alert_group = AlertGroup.all_objects.get(pk=alert_group_pk)
except AlertGroup.DoesNotExist:
logger.warning(f"Alert group {alert_group_pk} does not exist")
return
try:
notification_policy = UserNotificationPolicy.objects.get(pk=notification_policy_pk)
except UserNotificationPolicy.DoesNotExist:
logger.warning(f"User notification policy {notification_policy_pk} does not exist")
return
emails_left = user.organization.emails_left(user)
if emails_left <= 0:
UserNotificationPolicyLogRecord.objects.create(
author=user,
type=UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED,
notification_policy=notification_policy,
alert_group=alert_group,
reason="Error while sending email",
notification_step=notification_policy.step,
notification_channel=notification_policy.notify_by,
notification_error_code=UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_MAIL_LIMIT_EXCEEDED,
)
EmailMessage.objects.create(
represents_alert_group=alert_group,
notification_policy=notification_policy,
receiver=user,
exceeded_limit=True,
)
return
subject, html_message = build_subject_and_message(alert_group, emails_left)
message = strip_tags(html_message)
email_from = settings.EMAIL_HOST_USER
recipient_list = [user.email]
connection = get_connection(
host=live_settings.EMAIL_HOST,
port=live_settings.EMAIL_PORT,
username=live_settings.EMAIL_HOST_USER,
password=live_settings.EMAIL_HOST_PASSWORD,
use_tls=live_settings.EMAIL_USE_TLS,
fail_silently=False,
timeout=5,
)
try:
send_mail(subject, message, email_from, recipient_list, html_message=html_message, connection=connection)
EmailMessage.objects.create(
represents_alert_group=alert_group,
notification_policy=notification_policy,
receiver=user,
exceeded_limit=False,
)
except (gaierror, BadHeaderError) as e:
# gaierror is raised when EMAIL_HOST is invalid
# BadHeaderError is raised when there's newlines in the subject
UserNotificationPolicyLogRecord.objects.create(
author=user,
type=UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED,
notification_policy=notification_policy,
alert_group=alert_group,
reason="Error while sending email",
notification_step=notification_policy.step,
notification_channel=notification_policy.notify_by,
)
logger.error(f"Error while sending email: {e}")
return

28
engine/apps/email/templates/email_notification.html Normal file
View file

@ -0,0 +1,28 @@
<!DOCTYPE html>
You are invited to check an alert group in Grafana OnCall!
<br><br>
Organization: {{ organization }}
<br>
Integration: {{ integration }}
<br>
Title: {{ title }}
{% if message %}
<br>
Message:
<br>
{% autoescape off %}
{{ message }}
{% endautoescape %}
{% endif %}
<br>
<strong><a href="{{ url }}">Go to the alert group</a></strong>
<br>
Your Grafana OnCall
{% if limit_notification %}
<br><br>
<span style="color: #333333;"><em>{{ emails_left }} emails left for the organization today. Contact your admin.</em></span>
{% endif %}
<!-- this ensures Gmail doesn't trim the email -->
<span style="display:none;">{% now "H:i.u e"%}</span>
<!---->

0
engine/apps/sendgridapp/tests/__init__.py → engine/apps/email/tests/__init__.py
View file

8
engine/apps/email/tests/factories.py Normal file
View file

@ -0,0 +1,8 @@
import factory
from apps.email.models import EmailMessage
class EmailMessageFactory(factory.DjangoModelFactory):
class Meta:
model = EmailMessage

118
engine/apps/email/tests/test_notify_user.py Normal file
View file

@ -0,0 +1,118 @@
import socket
from unittest.mock import patch
import pytest
from django.core import mail
from django.core.mail.backends.locmem import EmailBackend
from apps.base.models import UserNotificationPolicy, UserNotificationPolicyLogRecord
from apps.email.tasks import notify_user_async
from apps.user_management.subscription_strategy.free_public_beta_subscription_strategy import (
FreePublicBetaSubscriptionStrategy,
)
@pytest.mark.django_db
def test_notify_user(
settings,
make_organization,
make_user_for_organization,
make_token_for_organization,
make_alert_receive_channel,
make_alert_group,
make_alert,
make_user_notification_policy,
):
settings.EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
organization = make_organization()
user = make_user_for_organization(organization)
alert_receive_channel = make_alert_receive_channel(organization)
alert_group = make_alert_group(alert_receive_channel)
make_alert(alert_group=alert_group, raw_request_data=alert_receive_channel.config.example_payload)
notification_policy = make_user_notification_policy(
user,
UserNotificationPolicy.Step.NOTIFY,
notify_by=8,
important=False,
)
notify_user_async(user.pk, alert_group.pk, notification_policy.pk)
assert len(mail.outbox) == 1
@pytest.mark.django_db
def test_notify_user_bad_smtp_host(
settings,
make_organization,
make_user_for_organization,
make_token_for_organization,
make_alert_receive_channel,
make_alert_group,
make_alert,
make_user_notification_policy,
):
settings.EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
organization = make_organization()
user = make_user_for_organization(organization)
alert_receive_channel = make_alert_receive_channel(organization)
alert_group = make_alert_group(alert_receive_channel)
make_alert(alert_group=alert_group, raw_request_data=alert_receive_channel.config.example_payload)
notification_policy = make_user_notification_policy(
user,
UserNotificationPolicy.Step.NOTIFY,
notify_by=8,
important=False,
)
with patch.object(EmailBackend, "send_messages", side_effect=socket.gaierror):
notify_user_async(user.pk, alert_group.pk, notification_policy.pk)
assert len(mail.outbox) == 0
log_record = notification_policy.personal_log_records.last()
assert log_record.type == UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED
@pytest.mark.django_db
def test_notify_user_no_emails_left(
settings,
make_organization,
make_user_for_organization,
make_token_for_organization,
make_alert_receive_channel,
make_alert_group,
make_alert,
make_user_notification_policy,
):
settings.EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
organization = make_organization()
user = make_user_for_organization(organization)
alert_receive_channel = make_alert_receive_channel(organization)
alert_group = make_alert_group(alert_receive_channel)
make_alert(alert_group=alert_group, raw_request_data=alert_receive_channel.config.example_payload)
notification_policy = make_user_notification_policy(
user,
UserNotificationPolicy.Step.NOTIFY,
notify_by=8,
important=False,
)
with patch.object(FreePublicBetaSubscriptionStrategy, "emails_left", return_value=0):
notify_user_async(user.pk, alert_group.pk, notification_policy.pk)
assert len(mail.outbox) == 0
log_record = notification_policy.personal_log_records.last()
assert log_record.type == UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED
assert log_record.notification_error_code == UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_MAIL_LIMIT_EXCEEDED

14
engine/apps/integrations/middlewares.py Normal file
View file

@ -0,0 +1,14 @@
import logging
from django.core.exceptions import PermissionDenied
from django.http import HttpResponse
from django.utils.deprecation import MiddlewareMixin
from rest_framework import status
logger = logging.getLogger(__name__)
class IntegrationExceptionMiddleware(MiddlewareMixin):
def process_exception(self, request, exception):
if request.path.startswith("/integrations/v1") and isinstance(exception, PermissionDenied):
return HttpResponse(exception, status=status.HTTP_403_FORBIDDEN)

70
engine/apps/integrations/views.py
View file

@ -25,8 +25,6 @@ from apps.integrations.mixins import (
is_ratelimit_ignored,
)
from apps.integrations.tasks import create_alert, create_alertmanager_alerts
from apps.sendgridapp.parse import Parse
from apps.sendgridapp.permissions import AllowOnlySendgrid
from common.api_helpers.utils import create_engine_url
logger = logging.getLogger(__name__)
@ -384,72 +382,8 @@ class HeartBeatAPIView(AlertChannelDefiningMixin, APIView):
class InboundWebhookEmailView(AlertChannelDefiningMixin, APIView):
permission_classes = [AllowOnlySendgrid]
def dispatch(self, *args, **kwargs):
parse = Parse(self.request)
self.email_data = parse.key_values()
# When email is forwarded recipient field can be stored both in "to" and in "envelope" fields.
token_from_to = self._parse_token_from_to(self.email_data)
try:
kwargs["alert_channel_key"] = token_from_to
return super().dispatch(*args, **kwargs)
except KeyError as e:
logger.warning(f"InboundWebhookEmailView: {e}")
except PermissionDenied as e:
self._log_permission_denied(token_from_to, e)
kwargs.pop("alert_channel_key")
token_from_envelope = self._parse_token_from_envelope(self.email_data)
try:
kwargs["alert_channel_key"] = token_from_envelope
return super().dispatch(*args, **kwargs)
except KeyError as e:
logger.warning(f"InboundWebhookEmailView: {e}")
except PermissionDenied as e:
self._log_permission_denied(token_from_to, e)
kwargs.pop("alert_channel_key")
raise PermissionDenied("Integration key was not found. Permission denied.")
def _log_permission_denied(self, token, e):
logger.info(
f"InboundWebhookEmailView: Permission denied. token {token}. "
f"To {self.email_data.get('to')}. "
f"Envelope {self.email_data.get('envelope')}."
f"Exception: {e}"
)
def _parse_token_from_envelope(self, email_data):
envelope = email_data["envelope"]
envelope = json.loads(envelope)
token = envelope.get("to")[0].split("@")[0]
return token
def _parse_token_from_to(self, email_data):
return email_data["to"].split("@")[0]
def post(self, request, alert_receive_channel=None):
title = self.email_data["subject"]
message = self.email_data.get("text", "").strip()
payload = {"title": title, "message": message}
if alert_receive_channel:
create_alert.apply_async(
[],
{
"title": title,
"message": message,
"alert_receive_channel_pk": alert_receive_channel.pk,
"image_url": None,
"link_to_upstream_details": payload.get("link_to_upstream_details"),
"integration_unique_data": payload,
"raw_request_data": request.data,
},
)
return Response("OK")
# todo: implement inbound emails
pass
class IntegrationHeartBeatAPIView(AlertChannelDefiningMixin, IntegrationHeartBeatRateLimitMixin, APIView):

31
engine/apps/public_api/serializers/action.py
View file

@ -1,4 +1,5 @@
import json
from collections import defaultdict
from django.core.validators import URLValidator, ValidationError
from jinja2 import Template, TemplateError
@ -55,15 +56,33 @@ class ActionCreateSerializer(serializers.ModelSerializer):
return None
try:
json.loads(data)
except ValueError:
raise serializers.ValidationError("Data has incorrect format")
try:
Template(data)
template = Template(data)
except TemplateError:
raise serializers.ValidationError("Data has incorrect template")
try:
rendered = template.render(
{
# Validate that the template can be rendered with a JSON-ish alert payload.
# We don't know what the actual payload will be, so we use a defaultdict
# so that attribute access within a template will never fail
# (provided it's only one level deep - we won't accept templates that attempt
# to do nested attribute access).
# Every attribute access should return a string to ensure that users are
# correctly using `tojson` or wrapping fields in strings.
# If we instead used a `defaultdict(dict)` or `defaultdict(lambda: 1)` we
# would accidentally accept templates such as `{"name": {{ alert_payload.name }}}`
# which would then fail at the true render time due to the
# lack of explicit quotes around the template variable; this would render
# as `{"name": some_alert_name}` which is not valid JSON.
"alert_payload": defaultdict(str),
"alert_group_id": "abcd",
}
)
json.loads(rendered)
except ValueError:
raise serializers.ValidationError("Data has incorrect format")
return data
def validate_forward_whole_payload(self, data):

53
engine/apps/public_api/serializers/integrations.py
View file

@ -5,9 +5,10 @@ from rest_framework import fields, serializers
from apps.alerts.grafana_alerting_sync_manager.grafana_alerting_sync import GrafanaAlertingSyncManager
from apps.alerts.models import AlertReceiveChannel
from apps.base.messaging import get_messaging_backends
from common.api_helpers.custom_fields import TeamPrimaryKeyRelatedField
from common.api_helpers.exceptions import BadRequest
from common.api_helpers.mixins import EagerLoadingMixin
from common.api_helpers.mixins import NOTIFICATION_CHANNEL_OPTIONS, EagerLoadingMixin
from common.jinja_templater import jinja_template_env
from common.utils import timed_lru_cache
@ -62,6 +63,9 @@ class IntegrationSerializer(EagerLoadingMixin, serializers.ModelSerializer, Main
serializer = DefaultChannelFilterSerializer(default_route, context=self.context)
result["default_route"] = serializer.data
# add additional templates for messaging backends
result["templates"].update(self._get_messaging_backend_templates(instance))
return result
def create(self, validated_data):
@ -102,6 +106,8 @@ class IntegrationSerializer(EagerLoadingMixin, serializers.ModelSerializer, Main
raise BadRequest(detail="Integration with this name already exists")
def _correct_validated_data(self, validated_data):
validated_data = self._correct_validated_data_for_messaging_backends(validated_data)
templates = validated_data.pop("templates", {})
for template_name, templates_for_notification_channel in templates.items():
if type(templates_for_notification_channel) is dict:
@ -134,7 +140,7 @@ class IntegrationSerializer(EagerLoadingMixin, serializers.ModelSerializer, Main
if not isinstance(templates, dict):
raise BadRequest(detail="Invalid template data")
for notification_channel in ["slack", "web", "sms", "phone_call", "email", "telegram"]:
for notification_channel in NOTIFICATION_CHANNEL_OPTIONS:
template_data = templates.get(notification_channel, {})
if template_data is None:
continue
@ -160,6 +166,49 @@ class IntegrationSerializer(EagerLoadingMixin, serializers.ModelSerializer, Main
raise BadRequest(detail=f"Invalid {common_template} template data")
return templates
def _correct_validated_data_for_messaging_backends(self, validated_data):
templates = validated_data.get("templates", {})
messaging_backends_templates = self.instance.messaging_backends_templates if self.instance else None
for backend_id, backend in get_messaging_backends():
backend_templates = {}
if messaging_backends_templates is not None:
backend_templates = messaging_backends_templates.get(backend_id, {})
for field in backend.template_fields:
try:
template = templates[backend_id.lower()][field]
except KeyError:
continue
backend_templates[field] = template
# remove backend-specific template from payload
templates.pop(backend_id.lower(), None)
if backend_templates:
validated_data["messaging_backends_templates"] = messaging_backends_templates or {} | {
backend_id: backend_templates
}
return validated_data
@staticmethod
def _get_messaging_backend_templates(instance):
result = {}
messaging_backends_templates = instance.messaging_backends_templates or {}
for backend_id, backend in get_messaging_backends():
if not backend.template_fields:
continue
result[backend_id.lower()] = {
field: messaging_backends_templates.get(backend_id, {}).get(field) for field in backend.template_fields
}
return result
def get_heartbeat(self, obj):
try:
heartbeat = obj.integration_heartbeat

137
engine/apps/public_api/tests/test_custom_actions.py
View file

@ -167,6 +167,120 @@ def test_create_custom_action(make_organization_and_user_with_token):
assert response.data == expected_result
@pytest.mark.django_db
def test_create_custom_action_nested_data(make_organization_and_user_with_token):
organization, user, token = make_organization_and_user_with_token()
client = APIClient()
url = reverse("api-public:actions-list")
data = {
"name": "Test outgoing webhook with nested data",
"url": "https://example.com",
# Assert that nested field access still works as long as the variable
# is quoted, making it valid JSON.
# This ensures backwards compatibility from when templates were required
# to be JSON.
"data": '{"nested_item": "{{ alert_payload.foo.bar }}"}',
}
response = client.post(url, data=data, format="json", HTTP_AUTHORIZATION=f"{token}")
custom_action = CustomButton.objects.get(public_primary_key=response.data["id"])
expected_result = {
"id": custom_action.public_primary_key,
"name": custom_action.name,
"team_id": None,
"url": custom_action.webhook,
"data": custom_action.data,
"user": custom_action.user,
"password": custom_action.password,
"authorization_header": custom_action.authorization_header,
"forward_whole_payload": custom_action.forward_whole_payload,
}
assert response.status_code == status.HTTP_201_CREATED
assert response.json() == expected_result
@pytest.mark.django_db
def test_create_custom_action_valid_after_render(make_organization_and_user_with_token):
organization, user, token = make_organization_and_user_with_token()
client = APIClient()
url = reverse("api-public:actions-list")
data = {
"name": "Test outgoing webhook with nested data",
"url": "https://example.com",
# Assert that nested field access still works as long as the variable
# is quoted, making it valid JSON.
# This ensures backwards compatibility from when templates were required
# to be JSON.
"data": '{"name": "{{ alert_payload.name }}", "labels": {{ alert_payload.labels | tojson }}}',
}
response = client.post(url, data=data, format="json", HTTP_AUTHORIZATION=f"{token}")
custom_action = CustomButton.objects.get(public_primary_key=response.data["id"])
expected_result = {
"id": custom_action.public_primary_key,
"name": custom_action.name,
"team_id": None,
"url": custom_action.webhook,
"data": custom_action.data,
"user": custom_action.user,
"password": custom_action.password,
"authorization_header": custom_action.authorization_header,
"forward_whole_payload": custom_action.forward_whole_payload,
}
assert response.status_code == status.HTTP_201_CREATED
assert response.json() == expected_result
@pytest.mark.django_db
def test_create_custom_action_valid_after_render_use_all_data(make_organization_and_user_with_token):
organization, user, token = make_organization_and_user_with_token()
client = APIClient()
url = reverse("api-public:actions-list")
data = {
"name": "Test outgoing webhook with nested data",
"url": "https://example.com",
# Assert that nested field access still works as long as the variable
# is quoted, making it valid JSON.
# This ensures backwards compatibility from when templates were required
# to be JSON.
"data": "{{ alert_payload | tojson }}",
}
response = client.post(url, data=data, format="json", HTTP_AUTHORIZATION=f"{token}")
custom_action = CustomButton.objects.get(public_primary_key=response.data["id"])
expected_result = {
"id": custom_action.public_primary_key,
"name": custom_action.name,
"team_id": None,
"url": custom_action.webhook,
"data": custom_action.data,
"user": custom_action.user,
"password": custom_action.password,
"authorization_header": custom_action.authorization_header,
"forward_whole_payload": custom_action.forward_whole_payload,
}
assert response.status_code == status.HTTP_201_CREATED
assert response.json() == expected_result
@pytest.mark.django_db
def test_create_custom_action_invalid_data(
make_organization_and_user_with_token,
@ -205,6 +319,29 @@ def test_create_custom_action_invalid_data(
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert response.data["name"][0] == "This field is required."
data = {
"name": "Test outgoing webhook",
"url": "https://example.com",
"data": "invalid_json",
}
response = client.post(url, data=data, format="json", HTTP_AUTHORIZATION=f"{token}")
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert response.data["data"][0] == "Data has incorrect format"
data = {
"name": "Test outgoing webhook",
"url": "https://example.com",
# This would need a `| tojson` or some double quotes around it to pass validation.
"data": "{{ alert_payload.name }}",
}
response = client.post(url, data=data, format="json", HTTP_AUTHORIZATION=f"{token}")
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert response.data["data"][0] == "Data has incorrect format"
@pytest.mark.django_db
def test_update_custom_action(

89
engine/apps/public_api/tests/test_integrations.py
View file

@ -54,11 +54,12 @@ def test_get_list_integrations(
"phone_call": {
"title": None,
},
"email": {
"telegram": {
"title": None,
"message": None,
"image_url": None,
},
"telegram": {
TEST_MESSAGING_BACKEND_FIELD: {
"title": None,
"message": None,
"image_url": None,
@ -117,7 +118,6 @@ def test_create_integrations_with_none_templates(
"web": None,
"sms": None,
"phone_call": None,
"email": None,
"telegram": None,
},
}
@ -184,15 +184,76 @@ def test_update_integration_template(
"phone_call": {
"title": None,
},
"email": {
"telegram": {
"title": None,
"message": None,
"image_url": None,
},
TEST_MESSAGING_BACKEND_FIELD: {
"title": None,
"message": None,
"image_url": None,
},
},
"maintenance_mode": None,
"maintenance_started_at": None,
"maintenance_end_at": None,
}
url = reverse("api-public:integrations-detail", args=[integration.public_primary_key])
response = client.put(url, data=data_for_update, format="json", HTTP_AUTHORIZATION=f"{token}")
assert response.status_code == status.HTTP_200_OK
assert response.data == expected_response
@pytest.mark.django_db
def test_update_integration_template_messaging_backend(
make_organization_and_user_with_token, make_alert_receive_channel, make_channel_filter, make_integration_heartbeat
):
organization, user, token = make_organization_and_user_with_token()
integration = make_alert_receive_channel(organization, verbal_name="grafana")
default_channel_filter = make_channel_filter(integration, is_default=True)
make_integration_heartbeat(integration)
client = APIClient()
data_for_update = {"templates": {"grouping_key": "ip_addr", TEST_MESSAGING_BACKEND_FIELD: {"title": "Incident"}}}
expected_response = {
"id": integration.public_primary_key,
"team_id": None,
"name": "grafana",
"link": integration.integration_url,
"type": "grafana",
"default_route": {
"escalation_chain_id": None,
"id": default_channel_filter.public_primary_key,
"slack": {"channel_id": None, "enabled": True},
"telegram": {"id": None, "enabled": False},
TEST_MESSAGING_BACKEND_FIELD: {"id": None, "enabled": False},
},
"heartbeat": {
"link": f"{integration.integration_url}heartbeat/",
},
"templates": {
"grouping_key": "ip_addr",
"resolve_signal": None,
"acknowledge_signal": None,
"slack": {"title": None, "message": None, "image_url": None},
"web": {"title": None, "message": None, "image_url": None},
"sms": {
"title": None,
},
"phone_call": {
"title": None,
},
"telegram": {
"title": None,
"message": None,
"image_url": None,
},
TEST_MESSAGING_BACKEND_FIELD: {
"title": "Incident",
"message": None,
"image_url": None,
},
},
"maintenance_mode": None,
"maintenance_started_at": None,
@ -259,11 +320,12 @@ def test_update_resolve_signal_template(
"phone_call": {
"title": None,
},
"email": {
"telegram": {
"title": None,
"message": None,
"image_url": None,
},
"telegram": {
TEST_MESSAGING_BACKEND_FIELD: {
"title": None,
"message": None,
"image_url": None,
@ -366,11 +428,12 @@ def test_update_sms_template_with_empty_dict(
"phone_call": {
"title": None,
},
"email": {
"telegram": {
"title": None,
"message": None,
"image_url": None,
},
"telegram": {
TEST_MESSAGING_BACKEND_FIELD: {
"title": None,
"message": None,
"image_url": None,
@ -425,11 +488,12 @@ def test_update_integration_name(
"phone_call": {
"title": None,
},
"email": {
"telegram": {
"title": None,
"message": None,
"image_url": None,
},
"telegram": {
TEST_MESSAGING_BACKEND_FIELD: {
"title": None,
"message": None,
"image_url": None,
@ -487,11 +551,12 @@ def test_set_default_template(
"phone_call": {
"title": None,
},
"email": {
"telegram": {
"title": None,
"message": None,
"image_url": None,
},
"telegram": {
TEST_MESSAGING_BACKEND_FIELD: {
"title": None,
"message": None,
"image_url": None,

11
engine/apps/schedules/ical_utils.py
View file

@ -42,13 +42,13 @@ def users_in_ical(usernames_from_ical, organization, include_viewers=False):
Parse ical file and return list of users found
"""
# Only grafana username will be used, consider adding grafana email and id
users_found_in_ical = organization.users
if not include_viewers:
users_found_in_ical = users_found_in_ical.filter(role__in=(Role.ADMIN, Role.EDITOR))
user_emails = [v.lower() for v in usernames_from_ical]
users_found_in_ical = users_found_in_ical.filter(
(Q(username__in=usernames_from_ical) | Q(email__in=usernames_from_ical))
(Q(username__in=usernames_from_ical) | Q(email__lower__in=user_emails))
).distinct()
# Here is the example how we extracted users previously, using slack fields too
@ -394,8 +394,8 @@ def get_missing_users_from_ical_event(event, organization):
all_usernames, _ = get_usernames_from_ical_event(event)
users = list(get_users_from_ical_event(event, organization))
found_usernames = [u.username for u in users]
found_emails = [u.email for u in users]
return [u for u in all_usernames if u != "" and u not in found_usernames and u not in found_emails]
found_emails = [u.email.lower() for u in users]
return [u for u in all_usernames if u != "" and u not in found_usernames and u.lower() not in found_emails]
def get_users_from_ical_event(event, organization):
@ -536,7 +536,8 @@ def get_user_events_from_calendars(ical_obj: Calendar, calendars: tuple, user: U
for component in calendar.walk():
if component.name == "VEVENT":
event_user = get_usernames_from_ical_event(component)
if event_user[0][0] in [user.username, user.email]:
event_user_value = event_user[0][0]
if event_user_value == user.username or event_user_value.lower() == user.email.lower():
ical_obj.add_component(component)

10
engine/apps/schedules/tests/test_ical_utils.py
View file

@ -15,6 +15,16 @@ from apps.schedules.models import CustomOnCallShift, OnCallScheduleCalendar
from common.constants.role import Role
@pytest.mark.django_db
def test_users_in_ical_email_case_insensitive(make_organization_and_user, make_user_for_organization):
organization, user = make_organization_and_user()
user = make_user_for_organization(organization, username="foo", email="TestingUser@test.com")
usernames = ["testinguser@test.com"]
result = users_in_ical(usernames, organization)
assert set(result) == {user}
@pytest.mark.django_db
@pytest.mark.parametrize(
"include_viewers",

49
engine/apps/sendgridapp/constants.py
View file

@ -1,49 +0,0 @@
class SendgridEmailMessageStatuses(object):
"""
https://sendgrid.com/docs/for-developers/tracking-events/event/#delivery-events
"""
# Delivery events
ACCEPTED = 10
PROCESSED = 20
DEFERRED = 30
DELIVERED = 40
DROPPED = 50
BOUNCE = 60 # "event": "bounce", "type: "bounce"
BLOCKED = 70 # "event": "bounce", "type: "blocked"
# Engagement events
OPEN = 80
CLICK = 90
UNSUBSCRIBE = 100
SPAMREPORT = 110
# Group Unsubscribe - ?
# Group Resubscribe - ?
CHOICES = (
(ACCEPTED, "accepted"),
(PROCESSED, "processed"),
(DEFERRED, "deferred"),
(DELIVERED, "delivered"),
(DROPPED, "dropped"),
(BOUNCE, "bounce"),
(BLOCKED, "blocked"),
(OPEN, "open"),
(CLICK, "click"),
(UNSUBSCRIBE, "unsubscribe"),
(SPAMREPORT, "spamreport"),
)
DETERMINANT = {
"accepted": ACCEPTED,
"processed": PROCESSED,
"deferred": DEFERRED,
"delivered": DELIVERED,
"dropped": DROPPED,
"bounce": BOUNCE,
"blocked": BLOCKED,
"open": OPEN,
"click": CLICK,
"unsubscribe": UNSUBSCRIBE,
"spamreport": SPAMREPORT,
}

185
engine/apps/sendgridapp/models.py
View file

@ -1,185 +0,0 @@
import logging
import uuid
from django.apps import apps
from django.db import models
from python_http_client.exceptions import BadRequestsError, ForbiddenError, UnauthorizedError
from sendgrid import SendGridAPIClient
from sendgrid.helpers.mail import CustomArg, Mail
from apps.alerts.incident_appearance.renderers.email_renderer import AlertGroupEmailRenderer
from apps.alerts.signals import user_notification_action_triggered_signal
from apps.base.utils import live_settings
from apps.sendgridapp.constants import SendgridEmailMessageStatuses
logger = logging.getLogger(__name__)
class EmailMessageManager(models.Manager):
def update_status(self, message_uuid, message_status):
"""The function checks existence of EmailMessage
instance according to message_uuid and updates status on
message_status
Args:
message_uuid (str): uuid of Email message
message_status (str): new status
Returns:
"""
UserNotificationPolicyLogRecord = apps.get_model("base", "UserNotificationPolicyLogRecord")
if message_uuid and message_status:
email_message_qs = self.filter(message_uuid=message_uuid)
status = SendgridEmailMessageStatuses.DETERMINANT.get(message_status)
if email_message_qs.exists() and status:
email_message_qs.update(status=status)
email_message = email_message_qs.first()
log_record = None
if status == SendgridEmailMessageStatuses.DELIVERED:
log_record = UserNotificationPolicyLogRecord(
author=email_message.receiver,
type=UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_SUCCESS,
notification_policy=email_message.notification_policy,
alert_group=email_message.represents_alert_group,
notification_step=email_message.notification_policy.step
if email_message.notification_policy
else None,
notification_channel=email_message.notification_policy.notify_by
if email_message.notification_policy
else None,
)
elif status in [
SendgridEmailMessageStatuses.BOUNCE,
SendgridEmailMessageStatuses.BLOCKED,
SendgridEmailMessageStatuses.DROPPED,
]:
log_record = UserNotificationPolicyLogRecord(
author=email_message.receiver,
type=UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED,
notification_policy=email_message.notification_policy,
alert_group=email_message.represents_alert_group,
notification_error_code=email_message.get_error_code_by_sendgrid_status(status),
notification_step=email_message.notification_policy.step
if email_message.notification_policy
else None,
notification_channel=email_message.notification_policy.notify_by
if email_message.notification_policy
else None,
)
if log_record is not None:
log_record.save()
user_notification_action_triggered_signal.send(
sender=EmailMessage.objects.update_status, log_record=log_record
)
class EmailMessage(models.Model):
objects = EmailMessageManager()
message_uuid = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
exceeded_limit = models.BooleanField(null=True, default=None)
represents_alert = models.ForeignKey("alerts.Alert", on_delete=models.SET_NULL, null=True, default=None)
represents_alert_group = models.ForeignKey("alerts.AlertGroup", on_delete=models.SET_NULL, null=True, default=None)
notification_policy = models.ForeignKey(
"base.UserNotificationPolicy", on_delete=models.SET_NULL, null=True, default=None
)
receiver = models.ForeignKey("user_management.User", on_delete=models.PROTECT, null=True, default=None)
status = models.PositiveSmallIntegerField(blank=True, null=True, choices=SendgridEmailMessageStatuses.CHOICES)
created_at = models.DateTimeField(auto_now_add=True)
@staticmethod
def send_incident_mail(user, alert_group, notification_policy):
UserNotificationPolicyLogRecord = apps.get_model("base", "UserNotificationPolicyLogRecord")
log_record = None
alert = alert_group.alerts.first()
email_message = EmailMessage(
represents_alert_group=alert_group,
represents_alert=alert,
receiver=user,
notification_policy=notification_policy,
)
emails_left = alert_group.channel.organization.emails_left(user)
if emails_left > 0:
email_message.exceeded_limit = False
limit_notification = False
if emails_left < 5:
limit_notification = True
subject, html_content = AlertGroupEmailRenderer(alert_group).render(limit_notification)
message = Mail(
from_email=live_settings.SENDGRID_FROM_EMAIL,
to_emails=user.email,
subject=subject,
html_content=html_content,
)
custom_arg = CustomArg("message_uuid", str(email_message.message_uuid))
message.add_custom_arg(custom_arg)
sendgrid_client = SendGridAPIClient(live_settings.SENDGRID_API_KEY)
try:
response = sendgrid_client.send(message)
sending_status = True
except (BadRequestsError, UnauthorizedError, ForbiddenError) as e:
logger.error(f"Error email sending: {e}")
sending_status = False
else:
if response.status_code == 202:
email_message.status = SendgridEmailMessageStatuses.ACCEPTED
email_message.save()
else:
logger.error(f"Error email sending: status code: {response.status_code}")
sending_status = False
if not sending_status:
log_record = UserNotificationPolicyLogRecord(
author=user,
type=UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED,
notification_policy=notification_policy,
alert_group=alert_group,
notification_error_code=UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_NOT_ABLE_TO_SEND_MAIL,
notification_step=notification_policy.step if notification_policy else None,
notification_channel=notification_policy.notify_by if notification_policy else None,
)
else:
log_record = UserNotificationPolicyLogRecord(
author=user,
type=UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED,
notification_policy=notification_policy,
alert_group=alert_group,
notification_error_code=UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_MAIL_LIMIT_EXCEEDED,
notification_step=notification_policy.step if notification_policy else None,
notification_channel=notification_policy.notify_by if notification_policy else None,
)
email_message.exceeded_limit = True
email_message.save()
if log_record is not None:
log_record.save()
user_notification_action_triggered_signal.send(
sender=EmailMessage.send_incident_mail, log_record=log_record
)
@staticmethod
def get_error_code_by_sendgrid_status(status):
UserNotificationPolicyLogRecord = apps.get_model("base", "UserNotificationPolicyLogRecord")
SENDGRID_ERRORS_TO_ERROR_CODES_MAP = {
SendgridEmailMessageStatuses.BOUNCE: UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_MAIL_DELIVERY_FAILED,
SendgridEmailMessageStatuses.BLOCKED: UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_MAIL_DELIVERY_FAILED,
SendgridEmailMessageStatuses.DROPPED: UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_MAIL_DELIVERY_FAILED,
}
return SENDGRID_ERRORS_TO_ERROR_CODES_MAP.get(status, None)

119
engine/apps/sendgridapp/parse.py
View file

@ -1,119 +0,0 @@
import base64
import email
import mimetypes
from six import iteritems
from werkzeug.utils import secure_filename
class Parse(object):
"""Parse data received from the SendGrid Inbound Parse webhook.
It's based on https://github.com/sendgrid/sendgrid-python/blob/master/sendgrid/helpers/inbound/parse.py
"""
def __init__(self, request):
self._keys = [
"attachments",
"headers",
"text",
"envelope",
"to",
"html",
"sender_ip",
"attachment-info",
"subject",
"dkim",
"SPF",
"charsets",
"content-ids",
"spam_report",
"spam_score",
"email",
]
self._request = request
self._payload = request.POST.dict()
self._raw_payload = request.POST
def key_values(self):
"""
Return a dictionary of key/values in the payload received from
the webhook
"""
key_values = {}
for key in self.keys:
if key in self.payload:
key_values[key] = self.payload[key]
return key_values
def get_raw_email(self):
"""
This only applies to raw payloads:
https://sendgrid.com/docs/Classroom/Basics/Inbound_Parse_Webhook/setting_up_the_inbound_parse_webhook.html#-Raw-Parameters
"""
if "email" in self.payload:
raw_email = email.message_from_string(self.payload["email"])
return raw_email
else:
return None
def attachments(self):
"""Returns an object with:
type = file content type
file_name = the name of the file
contents = base64 encoded file contents"""
attachments = None
if "attachment-info" in self.payload:
attachments = self._get_attachments(self.request)
# Check if we have a raw message
raw_email = self.get_raw_email()
if raw_email is not None:
attachments = self._get_attachments_raw(raw_email)
return attachments
def _get_attachments(self, request):
attachments = []
for _, filestorage in iteritems(request.files):
attachment = {}
if filestorage.filename not in (None, "fdopen", "<fdopen>"):
filename = secure_filename(filestorage.filename)
attachment["type"] = filestorage.content_type
attachment["file_name"] = filename
attachment["contents"] = base64.b64encode(filestorage.read())
attachments.append(attachment)
return attachments
def _get_attachments_raw(self, raw_email):
attachments = []
counter = 1
for part in raw_email.walk():
attachment = {}
if part.get_content_maintype() == "multipart":
continue
filename = part.get_filename()
if not filename:
ext = mimetypes.guess_extension(part.get_content_type())
if not ext:
ext = ".bin"
filename = "part-%03d%s" % (counter, ext)
counter += 1
attachment["type"] = part.get_content_type()
attachment["file_name"] = filename
attachment["contents"] = part.get_payload(decode=False)
attachments.append(attachment)
return attachments
@property
def keys(self):
return self._keys
@property
def request(self):
return self._request
@property
def payload(self):
return self._payload
@property
def raw_payload(self):
return self._raw_payload

14
engine/apps/sendgridapp/permissions.py
View file

@ -1,14 +0,0 @@
from rest_framework.permissions import BasePermission
from apps.base.utils import live_settings
class AllowOnlySendgrid(BasePermission):
def has_permission(self, request, view):
# https://stackoverflow.com/questions/20865673/sendgrid-incoming-mail-webhook-how-do-i-secure-my-endpoint
sendgrid_key = request.query_params.get("key")
if sendgrid_key is None:
return False
return live_settings.SENDGRID_SECRET_KEY == sendgrid_key

26
engine/apps/sendgridapp/templates/email_notification.html
View file

@ -1,26 +0,0 @@
<!DOCTYPE html>
<!-- this ensures Gmail doesn't trim the email. Add this line at the beginning and end of email content. -->
<span style="display:none;">{% now "H:i.u e"%}</span>
<!---->
You are invited to check Incident
<br><br>
<strong><a href="{{ url }}">{{ title }}</a></strong>
{% if message %}
{{ message|linebreaks }}
{% endif %}
{#<br>#}
{#<img src="{{ image_url }}" alt="image" />#}
Amixr team: {{ amixr_team }}
<br>
Alert channel: {{ alert_channel }}
<br><br>
<strong><a href="{{ url }}">Check Incident</a></strong>
<br><br>
Your Amixr.IO
{% if limit_notification %}
<br><br>
<span style="color: #333333;"><em>{{ emails_left }} mail(s) left for this week. Contact your admin.</em></span>
{% endif %}
<!-- this ensures Gmail doesn't trim the email. Add this line at the beginning and end of email content. -->
<span style="display:none;">{% now "H:i.u e"%}</span>
<!---->

15
engine/apps/sendgridapp/templates/email_verification.html
View file

@ -1,15 +0,0 @@
<!-- this ensures Gmail doesn't trim the email. Add this line at the beginning and end of email content. -->
<span style="display:none;">{% now "H:i.u e"%}</span>
<!---->
<strong>Welcome to OnCall!</strong>
<br><br>
To verify your email address, please click the button below. If you did not sign up for OnCall, please ignore this email.
<br>
<a href='{{ url }}' target='_blank'>Confirm email</a>
<br><br>
Thanks,
<br>
OnCall Team
<!-- this ensures Gmail doesn't trim the email. Add this line at the beginning and end of email content. -->
<span style="display:none;">{% now "H:i.u e"%}</span>
<!---->

8
engine/apps/sendgridapp/tests/factories.py
View file

@ -1,8 +0,0 @@
# import factory
#
# from apps.sendgridapp.models import EmailMessage
#
#
# class EmailMessageFactory(factory.DjangoModelFactory):
# class Meta:
# model = EmailMessage

135
engine/apps/sendgridapp/tests/test_emails.py
View file

@ -1,135 +0,0 @@
# from unittest.mock import patch
#
# import pytest
# from django.urls import reverse
# from django.utils import timezone
# from rest_framework.test import APIClient
#
# from apps.sendgridapp.constants import SendgridEmailMessageStatuses
# from apps.sendgridapp.verification_token import email_verification_token_generator
#
#
# @pytest.mark.skip(reason="email disabled")
# @pytest.mark.django_db
# def test_email_verification(
# make_team,
# make_user_for_team,
# make_email_message,
# make_alert_receive_channel,
# make_alert_group,
# ):
# amixr_team = make_team()
# admin = make_user_for_team(amixr_team, role=ROLE_ADMIN)
# alert_receive_channel = make_alert_receive_channel(amixr_team)
# alert_group = make_alert_group(alert_receive_channel)
# make_email_message(
# receiver=admin, status=SendgridEmailMessageStatuses.ACCEPTED, represents_alert_group=alert_group
# ),
# client = APIClient()
# correct_token = email_verification_token_generator.make_token(admin)
# url = reverse("sendgridapp:verify_email", kwargs={"token": correct_token, "uid": admin.pk, "slackteam": None})
# response = client.get(url, content_type="application/json")
# assert response.status_code == 200
# admin.refresh_from_db()
# assert admin.email_verified is True
#
#
# @pytest.mark.skip(reason="email disabled")
# @pytest.mark.django_db
# def test_email_verification_incorrect_token(
# make_team,
# make_user_for_team,
# make_email_message,
# make_alert_receive_channel,
# make_alert_group,
# ):
# amixr_team = make_team()
# admin = make_user_for_team(amixr_team, role=ROLE_ADMIN)
# alert_receive_channel = make_alert_receive_channel(amixr_team)
# alert_group = make_alert_group(alert_receive_channel)
# make_email_message(
# receiver=admin, status=SendgridEmailMessageStatuses.ACCEPTED, represents_alert_group=alert_group
# ),
#
# client = APIClient()
# url = reverse("sendgridapp:verify_email", kwargs={"token": "incorrect_token", "uid": admin.pk, "slackteam": None})
#
# response = client.get(path=url, content_type="application/json")
# assert response.status_code == 403
# admin.refresh_from_db()
# assert admin.email_verified is False
#
#
# @pytest.mark.skip(reason="email disabled")
# @pytest.mark.django_db
# def test_email_verification_incorrect_uid(
# make_team,
# make_user_for_team,
# make_email_message,
# make_alert_receive_channel,
# make_alert_group,
# ):
# amixr_team = make_team()
# admin = make_user_for_team(amixr_team, role=ROLE_ADMIN)
# alert_receive_channel = make_alert_receive_channel(amixr_team)
# alert_group = make_alert_group(alert_receive_channel)
# make_email_message(
# receiver=admin, status=SendgridEmailMessageStatuses.ACCEPTED, represents_alert_group=alert_group
# ),
# client = APIClient()
#
# correct_token = email_verification_token_generator.make_token(admin)
# url = reverse(
# "sendgridapp:verify_email", kwargs={"token": correct_token, "uid": 100, "slackteam": None} # incorrect user uid
# )
# response = client.get(path=url, content_type="application/json")
# assert response.status_code == 403
# admin.refresh_from_db()
# assert admin.email_verified is False
#
#
# @pytest.mark.skip(reason="email disabled")
# @patch("apps.integrations.helpers.inbound_emails.AllowOnlySendgrid.has_permission", return_value=True)
# @patch(
# "apps.slack.helpers.slack_client.SlackClientWithErrorHandling.api_call",
# return_value={"ok": True, "ts": timezone.now().timestamp()},
# )
# @pytest.mark.django_db
# @pytest.mark.parametrize("status", ["delivered", "bounce", "dropped"])
# def test_update_email_status(
# mocked_slack_api_call,
# mocked_sendgrid_permission,
# make_team,
# make_user_for_team,
# make_email_message,
# make_alert_receive_channel,
# make_alert_group,
# status,
# ):
# """The test for Email message status update via api"""
# amixr_team = make_team()
# admin = make_user_for_team(amixr_team, role=ROLE_ADMIN)
# alert_receive_channel = make_alert_receive_channel(amixr_team)
# alert_group = make_alert_group(alert_receive_channel)
# email_message = make_email_message(
# receiver=admin, status=SendgridEmailMessageStatuses.ACCEPTED, represents_alert_group=alert_group
# )
# client = APIClient()
# url = reverse("sendgridapp:email_status_event")
#
# data = [
# {
# "message_uuid": str(email_message.message_uuid),
# "event": status,
# }
# ]
# response = client.post(
# url,
# data,
# format="json",
# )
#
# assert response.status_code == 204
# assert response.data == ""
# email_message.refresh_from_db()
# assert email_message.status == SendgridEmailMessageStatuses.DETERMINANT[status]

9
engine/apps/sendgridapp/urls.py
View file

@ -1,9 +0,0 @@
from django.urls import path
from apps.sendgridapp.views import EmailStatusCallback
app_name = "sendgridapp"
urlpatterns = [
path(r"email_status_event/", EmailStatusCallback.as_view(), name="email_status_event"),
]

20
engine/apps/sendgridapp/verification_token.py
View file

@ -1,20 +0,0 @@
"""Based on example https://simpleisbetterthancomplex.com/tutorial/2016/08/24/how-to-create-one-time-link.html"""
from django.conf import settings
from django.contrib.auth.tokens import PasswordResetTokenGenerator
class EmailVerificationTokenGenerator(PasswordResetTokenGenerator):
# There are the default setting of PASSWORD_RESET_TIMEOUT_DAYS = 3 (days)
key_salt = "EmailVerificationTokenGenerator" + settings.TOKEN_SALT
secret = settings.TOKEN_SECRET
def _make_hash_value(self, user, timestamp):
team_datetime_timestamp = (
"" if user.teams.first() is None else user.teams.first().datetime.replace(microsecond=0, tzinfo=None)
)
return str(user.pk) + str(timestamp) + str(team_datetime_timestamp) + str(user.email_verified)
email_verification_token_generator = EmailVerificationTokenGenerator()

29
engine/apps/sendgridapp/views.py
View file

@ -1,29 +0,0 @@
import logging
from django.apps import apps
from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
from apps.sendgridapp.permissions import AllowOnlySendgrid
logger = logging.getLogger(__name__)
# Receive Email Status Update from Sendgrid
class EmailStatusCallback(APIView):
# https://sendgrid.com/docs/for-developers/tracking-events/event/#delivery-events
permission_classes = [AllowOnlySendgrid]
def post(self, request):
for data in request.data:
message_uuid = data.get("message_uuid")
message_status = data.get("event")
if message_status is not None and "type" in message_status:
message_status = message_status["type"]
logger.info(f"UUID: {message_uuid}, Status: {message_status}")
EmailMessage = apps.get_model("sendgridapp", "EmailMessage")
EmailMessage.objects.update_status(message_uuid=message_uuid, message_status=message_status)
return Response(data="", status=status.HTTP_204_NO_CONTENT)

4
engine/apps/slack/scenarios/alertgroup_appearance.py
View file

@ -90,7 +90,7 @@ class OpenAlertAppearanceDialogStep(
blocks.append(block)
blocks.append({"type": "divider"})
for notification_channel in ["slack", "web", "sms", "phone_call", "email", "telegram"]:
for notification_channel in ["slack", "web", "sms", "phone_call", "telegram"]:
blocks.append(
{
"type": "header",
@ -236,7 +236,7 @@ class UpdateAppearanceStep(scenario_step.ScenarioStep):
prev_state = alert_receive_channel.insight_logs_serialized
for templatizable_attr in ["title", "message", "image_url"]:
for notification_channel in ["slack", "web", "sms", "phone_call", "email", "telegram"]:
for notification_channel in ["slack", "web", "sms", "phone_call", "telegram"]:
attr_name = f"{notification_channel}_{templatizable_attr}_template"
try:
old_value = getattr(alert_receive_channel, attr_name)

2
engine/apps/slack/scenarios/resolution_note.py
View file

@ -358,7 +358,7 @@ class UpdateResolutionNoteStep(scenario_step.ScenarioStep):
author_verbal = resolution_note.author_verbal(mention=True)
resolution_note_text_block = {
"type": "section",
"text": {"type": "plain_text", "text": resolution_note.text, "emoji": True},
"text": {"type": "mrkdwn", "text": resolution_note.text},
}
blocks.append(resolution_note_text_block)
context_block = {

25
engine/apps/telegram/models/verification/channel.py
View file

@ -23,26 +23,39 @@ class TelegramChannelVerificationCode(models.Model):
def is_active(self) -> bool:
return self.datetime + timezone.timedelta(days=1) < timezone.now()
@property
def uuid_with_org_id(self) -> str:
return f"{self.organization.public_primary_key}_{self.uuid}"
@classmethod
def uuid_without_org_id(cls, verification_code: str) -> str:
try:
return verification_code.split("_")[1]
except IndexError:
raise ValidationError("Invalid verification code format")
@classmethod
def verify_channel_and_discussion_group(
cls,
uuid_code: str,
verification_code: str,
channel_chat_id: int,
channel_name: str,
discussion_group_chat_id: int,
discussion_group_name: str,
) -> Tuple[Optional[TelegramToOrganizationConnector], bool]:
try:
verification_code = cls.objects.get(uuid=uuid_code)
uuid_code = cls.uuid_without_org_id(verification_code)
code_instance = cls.objects.get(uuid=uuid_code)
# see if a organization has other channels connected
# if it is the first channel, make it default for the organization
connector_exists = verification_code.organization.telegram_channel.exists()
connector_exists = code_instance.organization.telegram_channel.exists()
connector, created = TelegramToOrganizationConnector.objects.get_or_create(
channel_chat_id=channel_chat_id,
defaults={
"organization": verification_code.organization,
"organization": code_instance.organization,
"channel_name": channel_name,
"discussion_group_chat_id": discussion_group_chat_id,
"discussion_group_name": discussion_group_name,
@ -51,14 +64,14 @@ class TelegramChannelVerificationCode(models.Model):
)
write_chatops_insight_log(
author=verification_code.author,
author=code_instance.author,
event_name=ChatOpsEvent.CHANNEL_CONNECTED,
chatops_type=ChatOpsType.TELEGRAM,
channel_name=channel_name,
)
if not connector_exists:
write_chatops_insight_log(
author=verification_code.author,
author=code_instance.author,
event_name=ChatOpsEvent.DEFAULT_CHANNEL_CHANGED,
chatops_type=ChatOpsType.TELEGRAM,
prev_channel=None,

19
engine/apps/telegram/models/verification/personal.py
View file

@ -21,13 +21,26 @@ class TelegramVerificationCode(models.Model):
def is_active(self) -> bool:
return self.datetime + timezone.timedelta(days=1) < timezone.now()
@property
def uuid_with_org_id(self) -> str:
return f"{self.user.organization.public_primary_key}_{self.uuid}"
@classmethod
def uuid_without_org_id(cls, verification_code: str) -> str:
try:
return verification_code.split("_")[1]
except IndexError:
raise ValidationError("Invalid verification code format")
@classmethod
def verify_user(
cls, uuid_code: str, telegram_chat_id: int, telegram_nick_name: str
cls, verification_code: str, telegram_chat_id: int, telegram_nick_name: str
) -> Tuple[Optional[TelegramToUserConnector], bool]:
try:
verification_code = cls.objects.get(uuid=uuid_code)
user = verification_code.user
uuid_code = cls.uuid_without_org_id(verification_code)
code_instance = cls.objects.get(uuid=uuid_code)
user = code_instance.user
connector, created = TelegramToUserConnector.objects.get_or_create(
user=user, defaults={"telegram_nick_name": telegram_nick_name, "telegram_chat_id": telegram_chat_id}

5
engine/apps/telegram/renderers/keyboard.py
View file

@ -83,7 +83,10 @@ class TelegramKeyboardRenderer:
callback_data_args = [self.alert_group.pk, action.value]
if action_data is not None:
callback_data_args.append(action_data)
# Add org id with 'x-oncall-org-id' prefix to callback data.
# It's a workaroung to pass org_id to the oncall-gateway while proxying requests.
# TODO: switch to json str instead of ':' separated string.
callback_data_args.append(f"x-oncall-org-id{self.alert_group.channel.organization.public_primary_key}")
button = InlineKeyboardButton(text=text, callback_data=CallbackQueryFactory.encode_data(*callback_data_args))
return button

71
engine/apps/telegram/tests/test_keyboard_renderer.py
View file

@ -46,12 +46,31 @@ def test_actions_keyboard_alerting(make_organization, make_alert_receive_channel
keyboard = renderer.render_actions_keyboard()
expected_keyboard = [
[InlineKeyboardButton(text="Acknowledge", callback_data=f"{alert_group.pk}:acknowledge")],
[InlineKeyboardButton(text="Resolve", callback_data=f"{alert_group.pk}:resolve")],
[
InlineKeyboardButton(text="🔕 forever", callback_data=f"{alert_group.pk}:silence"),
InlineKeyboardButton(text="... for 1h", callback_data=f"{alert_group.pk}:silence:3600"),
InlineKeyboardButton(text="... for 4h", callback_data=f"{alert_group.pk}:silence:14400"),
InlineKeyboardButton(
text="Acknowledge",
callback_data=f"{alert_group.pk}:acknowledge:x-oncall-org-id{organization.public_primary_key}",
)
],
[
InlineKeyboardButton(
text="Resolve",
callback_data=f"{alert_group.pk}:resolve:x-oncall-org-id{organization.public_primary_key}",
)
],
[
InlineKeyboardButton(
text="🔕 forever",
callback_data=f"{alert_group.pk}:silence:x-oncall-org-id{organization.public_primary_key}",
),
InlineKeyboardButton(
text="... for 1h",
callback_data=f"{alert_group.pk}:silence:3600:x-oncall-org-id{organization.public_primary_key}",
),
InlineKeyboardButton(
text="... for 4h",
callback_data=f"{alert_group.pk}:silence:14400:x-oncall-org-id{organization.public_primary_key}",
),
],
]
@ -75,8 +94,18 @@ def test_actions_keyboard_acknowledged(
keyboard = renderer.render_actions_keyboard()
expected_keyboard = [
[InlineKeyboardButton(text="Unacknowledge", callback_data=f"{alert_group.pk}:unacknowledge")],
[InlineKeyboardButton(text="Resolve", callback_data=f"{alert_group.pk}:resolve")],
[
InlineKeyboardButton(
text="Unacknowledge",
callback_data=f"{alert_group.pk}:unacknowledge:x-oncall-org-id{organization.public_primary_key}",
)
],
[
InlineKeyboardButton(
text="Resolve",
callback_data=f"{alert_group.pk}:resolve:x-oncall-org-id{organization.public_primary_key}",
)
],
]
assert are_keyboards_equal(keyboard.inline_keyboard, expected_keyboard) is True
@ -99,7 +128,12 @@ def test_actions_keyboard_resolved(
keyboard = renderer.render_actions_keyboard()
expected_keyboard = [
[InlineKeyboardButton(text="Unresolve", callback_data=f"{alert_group.pk}:unresolve")],
[
InlineKeyboardButton(
text="Unresolve",
callback_data=f"{alert_group.pk}:unresolve:x-oncall-org-id{organization.public_primary_key}",
)
],
]
assert are_keyboards_equal(keyboard.inline_keyboard, expected_keyboard) is True
@ -122,9 +156,24 @@ def test_actions_keyboard_silenced(
keyboard = renderer.render_actions_keyboard()
expected_keyboard = [
[InlineKeyboardButton(text="Acknowledge", callback_data=f"{alert_group.pk}:acknowledge")],
[InlineKeyboardButton(text="Resolve", callback_data=f"{alert_group.pk}:resolve")],
[InlineKeyboardButton(text="Unsilence", callback_data=f"{alert_group.pk}:unsilence")],
[
InlineKeyboardButton(
text="Acknowledge",
callback_data=f"{alert_group.pk}:acknowledge:x-oncall-org-id{organization.public_primary_key}",
)
],
[
InlineKeyboardButton(
text="Resolve",
callback_data=f"{alert_group.pk}:resolve:x-oncall-org-id{organization.public_primary_key}",
)
],
[
InlineKeyboardButton(
text="Unsilence",
callback_data=f"{alert_group.pk}:unsilence:x-oncall-org-id{organization.public_primary_key}",
)
],
]
assert are_keyboards_equal(keyboard.inline_keyboard, expected_keyboard) is True

5
engine/apps/telegram/tests/test_message_renderer.py
View file

@ -71,9 +71,8 @@ def test_alert_group_message(make_organization, make_alert_receive_channel, make
renderer = TelegramMessageRenderer(alert_group=alert_group)
text = renderer.render_alert_group_message()
assert text == (
f"🔴 #{alert_group.inside_organization_number}, {alert_receive_channel.config.tests['telegram']['title']}\n"
f"<a href='{organization.web_link_with_id}'>&#8205;</a>🔴 #{alert_group.inside_organization_number}, {alert_receive_channel.config.tests['telegram']['title']}\n"
"Alerting, alerts: 1\n"
"Source: Test integration - Grafana\n"
f"{alert_group.web_link}\n\n"
@ -157,7 +156,7 @@ def test_personal_message(
text = renderer.render_personal_message()
assert text == (
f"🟠 #{alert_group.inside_organization_number}, {alert_receive_channel.config.tests['telegram']['title']}\n"
f"<a href='{organization.web_link_with_id}'>&#8205;</a>🟠 #{alert_group.inside_organization_number}, {alert_receive_channel.config.tests['telegram']['title']}\n"
f"Acknowledged by {user_name}, alerts: 1\n"
"Source: Test integration - Grafana\n"
f"{alert_group.web_link}\n\n"

4
engine/apps/telegram/tests/test_models.py
View file

@ -17,7 +17,7 @@ def test_user_verification_handler_process_update_another_account_already_linked
user_2 = make_user_for_organization(organization)
code = make_telegram_verification_code(user_2)
connector, created = TelegramVerificationCode.verify_user(code.uuid, chat_id, "nickname")
connector, created = TelegramVerificationCode.verify_user(code.uuid_with_org_id, chat_id, "nickname")
assert created
assert connector.telegram_chat_id == chat_id
@ -38,7 +38,7 @@ def test_user_verification_handler_process_update_user_already_linked(
other_chat_id = 321
code = make_telegram_verification_code(user_1)
connector, created = TelegramVerificationCode.verify_user(code.uuid, other_chat_id, "nickname")
connector, created = TelegramVerificationCode.verify_user(code.uuid_with_org_id, other_chat_id, "nickname")
assert created is False
assert connector.user == user_1

2
engine/apps/telegram/updates/update_handlers/button_press.py
View file

@ -70,7 +70,7 @@ class ButtonPressHandler(UpdateHandler):
action_name = args[1]
action = Action(action_name)
action_data = args[2] if len(args) >= 3 else None
action_data = args[2] if len(args) >= 3 and not args[2].startswith("x-oncall-org-id") else None
return ActionContext(alert_group=alert_group, action=action, action_data=action_data)

14
engine/apps/telegram/updates/update_handlers/start_message.py
View file

@ -1,15 +1,10 @@
from apps.telegram.client import TelegramClient
from apps.telegram.models import TelegramToUserConnector
from apps.telegram.updates.update_handlers.update_handler import UpdateHandler
START_TEXT = """Hi!
This is Grafana OnCall notification bot. You can connect your Grafana OnCall account to Telegram on user settings page.
"""
START_TEXT_FOR_CONNECTED_USER = """Hi!
This is Grafana OnCall notification bot. Your Telegram account is connected to user <b>{username}</b>
"""
class StartMessageHandler(UpdateHandler):
def matches(self) -> bool:
@ -24,12 +19,5 @@ class StartMessageHandler(UpdateHandler):
return is_from_private_chat and is_start_message
def process_update(self) -> None:
connector = TelegramToUserConnector.objects.filter(telegram_chat_id=self.update.effective_user.id).first()
telegram_client = TelegramClient()
if connector is not None:
user = connector.user
text = START_TEXT_FOR_CONNECTED_USER.format(username=user.username)
telegram_client.send_raw_message(chat_id=self.update.effective_user.id, text=text)
else:
telegram_client.send_raw_message(chat_id=self.update.effective_user.id, text=START_TEXT)
telegram_client.send_raw_message(chat_id=self.update.effective_user.id, text=START_TEXT)

2
engine/apps/telegram/updates/update_handlers/verification/channel.py
View file

@ -73,7 +73,7 @@ class ChannelVerificationCodeHandler(UpdateHandler):
return
connector, created = TelegramChannelVerificationCode.verify_channel_and_discussion_group(
uuid_code=verification_code,
verification_code=verification_code,
channel_chat_id=channel_chat_id,
channel_name=channel_name,
discussion_group_chat_id=discussion_group_chat_id,

2
engine/apps/telegram/updates/update_handlers/verification/personal.py
View file

@ -33,7 +33,7 @@ class PersonalVerificationCodeHandler(UpdateHandler):
verification_code = text if is_verification_message(text) else text.split()[1]
connector, created = TelegramVerificationCode.verify_user(
uuid_code=verification_code, telegram_chat_id=user.id, telegram_nick_name=nickname
verification_code=verification_code, telegram_chat_id=user.id, telegram_nick_name=nickname
)
if created:

4
engine/apps/telegram/utils.py
View file

@ -1,11 +1,11 @@
import re
from typing import List, Union
UUID4_REGEX = "^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$"
TELEGRAM_VERIFICATION_CODE_REGEX = "^[A-Z0-9]*_[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$"
def is_verification_message(text: str) -> bool:
return bool(re.match(UUID4_REGEX, text))
return bool(re.match(TELEGRAM_VERIFICATION_CODE_REGEX, text))
class CallbackQueryFactory:

16
engine/apps/user_management/apps.py Normal file
View file

@ -0,0 +1,16 @@
from django.apps import AppConfig
from django.db import models
# enable a __lower field lookup for email fields
# https://docs.djangoproject.com/en/4.1/howto/custom-lookups/#a-bilateral-transformer-example
class LowerCase(models.Transform):
lookup_name = "lower"
function = "LOWER"
class UserManagementConfig(AppConfig):
name = "apps.user_management"
def ready(self):
models.EmailField.register_lookup(LowerCase)

6
engine/apps/user_management/models/organization.py
View file

@ -218,6 +218,7 @@ class Organization(MaintainableObject):
def sms_left(self, user):
return self.subscription_strategy.sms_left(user)
# todo: manage backend specific limits in messaging backend
def emails_left(self, user):
return self.subscription_strategy.emails_left(user)
@ -244,6 +245,11 @@ class Organization(MaintainableObject):
def web_link(self):
return urljoin(self.grafana_url, "a/grafana-oncall-app/")
@property
def web_link_with_id(self):
# It's a workaround to pass org id to the oncall gateway while proxying telegram requests
return urljoin(self.grafana_url, f"a/grafana-oncall-app/?x-oncall-org-id={self.public_primary_key}")
def __str__(self):
return f"{self.pk}: {self.org_title}"

16
engine/apps/user_management/subscription_strategy/free_public_beta_subscription_strategy.py
View file

@ -1,6 +1,7 @@
from datetime import datetime
from django.apps import apps
from django.utils import timezone
from apps.email.models import EmailMessage
from .base_subsription_strategy import BaseSubscriptionStrategy
@ -21,17 +22,16 @@ class FreePublicBetaSubscriptionStrategy(BaseSubscriptionStrategy):
def sms_left(self, user):
return self._calculate_phone_notifications_left(user)
# todo: manage backend specific limits in messaging backend
def emails_left(self, user):
# Email notifications are disabled now.
EmailMessage = apps.get_model("sendgridapp", "EmailMessage")
now = datetime.now()
now = timezone.now()
day_start = now.replace(hour=0, minute=0, second=0, microsecond=0)
emails_this_week = EmailMessage.objects.filter(
emails_today = EmailMessage.objects.filter(
created_at__gte=day_start,
represents_alert_group__channel__organization=self.organization,
receiver=user,
).count()
return self._emails_limit - emails_this_week
return self._emails_limit - emails_today
def notifications_limit_web_report(self, user):
limits_to_show = []
@ -59,7 +59,7 @@ class FreePublicBetaSubscriptionStrategy(BaseSubscriptionStrategy):
"""
PhoneCall = apps.get_model("twilioapp", "PhoneCall")
SMSMessage = apps.get_model("twilioapp", "SMSMessage")
now = datetime.now()
now = timezone.now()
day_start = now.replace(hour=0, minute=0, second=0, microsecond=0)
calls_today = PhoneCall.objects.filter(
created_at__gte=day_start,

15
engine/apps/user_management/tests/test_free_public_beta_subcription_strategy.py
View file

@ -1,8 +1,5 @@
import sys
import pytest
from apps.sendgridapp.constants import SendgridEmailMessageStatuses
from apps.twilioapp.constants import TwilioCallStatuses, TwilioMessageStatuses
from common.constants.role import Role
@ -65,7 +62,6 @@ def test_phone_calls_and_sms_counts_together(
assert organization.sms_left(user) == organization.subscription_strategy._phone_notifications_limit
@pytest.mark.skip(reason="email disabled")
@pytest.mark.django_db
def test_emails_left(
make_organization,
@ -75,10 +71,11 @@ def test_emails_left(
make_alert_group,
):
organization = make_organization()
admin = make_user_for_organization(organization, role=Role.ADMIN)
user = make_user_for_organization(organization)
alert_receive_channel = make_alert_receive_channel(organization)
alert_group = make_alert_group(alert_receive_channel)
make_email_message(
receiver=admin, status=SendgridEmailMessageStatuses.DELIVERED, represents_alert_group=alert_group
),
assert organization.emails_left(admin) == sys.maxsize
make_email_message(receiver=user, represents_alert_group=alert_group)
assert organization.emails_left(user) == organization.subscription_strategy._emails_limit - 1

14
engine/apps/user_management/tests/test_user.py
View file

@ -2,6 +2,7 @@
import pytest
from apps.user_management.models import User
from common.constants.role import Role
@ -22,3 +23,16 @@ def test_self_or_admin(
assert admin.self_or_admin(editor, organization) is False
assert admin.self_or_admin(second_admin, organization) is True
assert admin.self_or_admin(admin_from_another_organization, organization) is False
@pytest.mark.django_db
def test_lower_email_filter(
make_organization,
make_user_for_organization,
):
organization = make_organization()
user = make_user_for_organization(organization, email="TestingUser@test.com")
make_user_for_organization(organization, email="testing_user@test.com")
assert User.objects.get(email__lower="testinguser@test.com") == user
assert User.objects.filter(email__lower__in=["testinguser@test.com"]).get() == user

5
engine/common/api_helpers/mixins.py
View file

@ -11,7 +11,6 @@ from rest_framework.exceptions import NotFound, Throttled
from rest_framework.response import Response
from apps.alerts.incident_appearance.templaters import (
AlertEmailTemplater,
AlertPhoneCallTemplater,
AlertSlackTemplater,
AlertSmsTemplater,
@ -245,9 +244,8 @@ SLACK = "slack"
WEB = "web"
PHONE_CALL = "phone_call"
SMS = "sms"
EMAIL = "email"
TELEGRAM = "telegram"
NOTIFICATION_CHANNEL_OPTIONS = [SLACK, WEB, PHONE_CALL, SMS, EMAIL, TELEGRAM]
NOTIFICATION_CHANNEL_OPTIONS = [SLACK, WEB, PHONE_CALL, SMS, TELEGRAM]
TITLE = "title"
MESSAGE = "message"
IMAGE_URL = "image_url"
@ -261,7 +259,6 @@ NOTIFICATION_CHANNEL_TO_TEMPLATER_MAP = {
WEB: AlertWebTemplater,
PHONE_CALL: AlertPhoneCallTemplater,
SMS: AlertSmsTemplater,
EMAIL: AlertEmailTemplater,
TELEGRAM: AlertTelegramTemplater,
}

33
engine/config_integrations/alertmanager.py
View file

@ -73,22 +73,6 @@ web_image_url = slack_image_url
sms_title = '{{ payload.get("labels", {}).get("alertname", "Title undefined") }}'
phone_call_title = sms_title
email_title = web_title
email_message = """\
{{- payload.messsage }}
{%- if "status" in payload -%}
**Status**: {{ payload.status }}
{% endif -%}
**Labels:** {% for k, v in payload["labels"].items() %}
{{ k }}: {{ v }}{% endfor %}
**Annotations:**
{%- for k, v in payload.get("annotations", {}).items() %}
{#- render annotation as markdown url if it starts with http #}
{{ k }}: {{v}}
{% endfor %}
""" # noqa: W291
telegram_title = sms_title
telegram_message = """\
@ -188,23 +172,6 @@ tests = {
"phone_call": {
"title": "KubeJobCompletion",
},
"email": {
"title": "KubeJobCompletion",
"message": (
"**Status**: firing\n"
"**Labels:** \n"
"job: kube-state-metrics\n"
"instance: 10.143.139.7:8443\n"
"job_name: email-tracking-perform-initialization-1.0.50\n"
"severity: warning\n"
"alertname: KubeJobCompletion\n"
"namespace: default\n"
"prometheus: monitoring/k8s\n"
"**Annotations:**\n"
"message: Job default/email-tracking-perform-initialization-1.0.50 is taking more than one hour to complete.\n\n"
"runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion\n"
),
},
"telegram": {
"title": "KubeJobCompletion",
"message": (

4
engine/config_integrations/elastalert.py
View file

@ -36,10 +36,6 @@ sms_title = web_title
phone_call_title = sms_title
email_title = web_title
email_message = "{{ payload|tojson_pretty }}"
telegram_title = sms_title
telegram_message = "<code>{{ payload|tojson_pretty }}</code>"

4
engine/config_integrations/formatted_webhook.py
View file

@ -32,10 +32,6 @@ sms_title = web_title
phone_call_title = sms_title
email_title = web_title
email_message = slack_message
telegram_title = sms_title
telegram_message = slack_message

40
engine/config_integrations/grafana.py
View file

@ -81,29 +81,6 @@ sms_title = """\
phone_call_title = sms_title
email_title = web_title
email_message = """\
{{- payload.message }}
{%- for value in payload.get("evalMatches", []) %}
**{{ value.metric }}**: {{ value.value }}
{% endfor -%}
{%- if "status" in payload -%}
**Status**: {{ payload.status }}
{% endif -%}
{%- if "labels" in payload -%}
**Labels:** {% for k, v in payload["labels"].items() %}
{{ k }}: {{ v }}{% endfor %}
{% endif -%}
{%- if "annotations" in payload -%}
**Annotations:**
{%- for k, v in payload.get("annotations", {}).items() %}
{#- render annotation as markdown url if it starts with http #}
{{ k }}: {{v}}
{% endfor %}
{%- endif -%}
"""
telegram_title = sms_title
telegram_message = """\
@ -215,23 +192,6 @@ tests = {
"phone_call": {
"title": "KubeJobCompletion",
},
"email": {
"title": "KubeJobCompletion",
"message": (
"**Status**: firing\n"
"**Labels:** \n"
"job: kube-state-metrics\n"
"instance: 10.143.139.7:8443\n"
"job_name: email-tracking-perform-initialization-1.0.50\n"
"severity: warning\n"
"alertname: KubeJobCompletion\n"
"namespace: default\n"
"prometheus: monitoring/k8s\n"
"**Annotations:**\n"
"message: Job default/email-tracking-perform-initialization-1.0.50 is taking more than one hour to complete.\n\n"
"runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion\n"
),
},
"telegram": {
"title": "KubeJobCompletion",
"message": (

33
engine/config_integrations/grafana_alerting.py
View file

@ -77,22 +77,6 @@ web_image_url = slack_image_url
sms_title = '{{ payload.get("labels", {}).get("alertname", "Title undefined") }}'
phone_call_title = sms_title
email_title = web_title
email_message = """\
{{- payload.messsage }}
{%- if "status" in payload -%}
**Status**: {{ payload.status }}
{% endif -%}
**Labels:** {% for k, v in payload["labels"].items() %}
{{ k }}: {{ v }}{% endfor %}
**Annotations:**
{%- for k, v in payload.get("annotations", {}).items() %}
{#- render annotation as markdown url if it starts with http #}
{{ k }}: {{v}}
{% endfor %}
""" # noqa:W291
telegram_title = sms_title
telegram_message = """\
@ -191,23 +175,6 @@ tests = {
"phone_call": {
"title": "KubeJobCompletion",
},
"email": {
"title": "KubeJobCompletion",
"message": (
"**Status**: firing\n"
"**Labels:** \n"
"job: kube-state-metrics\n"
"instance: 10.143.139.7:8443\n"
"job_name: email-tracking-perform-initialization-1.0.50\n"
"severity: warning\n"
"alertname: KubeJobCompletion\n"
"namespace: default\n"
"prometheus: monitoring/k8s\n"
"**Annotations:**\n"
"message: Job default/email-tracking-perform-initialization-1.0.50 is taking more than one hour to complete.\n\n"
"runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion\n"
),
},
"telegram": {
"title": "KubeJobCompletion",
"message": (

4
engine/config_integrations/inbound_email.py
View file

@ -32,10 +32,6 @@ sms_title = web_title
phone_call_title = web_title
email_title = web_title
email_message = slack_message
telegram_title = sms_title
telegram_message = slack_message

4
engine/config_integrations/kapacitor.py
View file

@ -38,10 +38,6 @@ sms_title = web_title
phone_call_title = web_title
email_title = web_title
email_message = slack_message
telegram_title = sms_title
telegram_message = "<code>{{ payload|tojson_pretty }}</code>"

4
engine/config_integrations/maintenance.py
View file

@ -32,10 +32,6 @@ sms_title = web_title
phone_call_title = sms_title
email_title = web_title
email_message = slack_message
telegram_title = sms_title
telegram_message = slack_message

4
engine/config_integrations/manual.py
View file

@ -41,10 +41,6 @@ sms_title = web_title
phone_call_title = sms_title
email_title = web_title
email_message = slack_message
telegram_title = sms_title
telegram_message = slack_message

4
engine/config_integrations/webhook.py
View file

@ -36,10 +36,6 @@ sms_title = web_title
phone_call_title = sms_title
email_title = web_title
email_message = "{{ payload|tojson_pretty }}"
telegram_title = sms_title
telegram_message = "<code>{{ payload|tojson_pretty }}</code>"

16
engine/conftest.py
View file

@ -44,6 +44,7 @@ from apps.base.tests.factories import (
UserNotificationPolicyFactory,
UserNotificationPolicyLogRecordFactory,
)
from apps.email.tests.factories import EmailMessageFactory
from apps.heartbeat.tests.factories import IntegrationHeartBeatFactory
from apps.schedules.tests.factories import (
CustomOnCallShiftFactory,
@ -105,7 +106,7 @@ register(ResolutionNoteSlackMessageFactory)
register(PhoneCallFactory)
register(SMSFactory)
# register(EmailMessageFactory)
register(EmailMessageFactory)
register(IntegrationHeartBeatFactory)
@ -627,13 +628,12 @@ def make_sms():
return _make_sms
# TODO: restore email notifications
# @pytest.fixture()
# def make_email_message():
# def _make_email_message(receiver, status, **kwargs):
# return EmailMessageFactory(receiver=receiver, status=status, **kwargs)
#
# return _make_email_message
@pytest.fixture()
def make_email_message():
def _make_email_message(receiver, **kwargs):
return EmailMessageFactory(receiver=receiver, **kwargs)
return _make_email_message
@pytest.fixture()

1
engine/engine/urls.py
View file

@ -36,7 +36,6 @@ urlpatterns = [
path("api/internal/v1/", include("apps.social_auth.urls", namespace="social_auth")),
path("integrations/v1/", include("apps.integrations.urls", namespace="integrations")),
path("twilioapp/", include("apps.twilioapp.urls")),
# path('sendgridapp/', include('apps.sendgridapp.urls')), TODO: restore email notifications
path("api/v1/", include("apps.public_api.urls", namespace="api-public")),
path("api/internal/v1/", include("apps.migration_tool.urls", namespace="migration-tool")),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

1
engine/requirements.txt
View file

@ -23,7 +23,6 @@ recurring-ical-events==0.1.16b0
slack-export-viewer==1.0.0
beautifulsoup4==4.8.1
social-auth-app-django==3.1.0
sendgrid==6.1.2
cryptography==3.3.2
pytest==5.4.3
pytest-django==3.9.0

23
engine/settings/base.py
View file

@ -64,14 +64,6 @@ TWILIO_VERIFY_SERVICE_SID = os.environ.get("TWILIO_VERIFY_SERVICE_SID")
TELEGRAM_WEBHOOK_HOST = os.environ.get("TELEGRAM_WEBHOOK_HOST", BASE_URL)
TELEGRAM_TOKEN = os.environ.get("TELEGRAM_TOKEN")
# For Sending email
SENDGRID_API_KEY = os.environ.get("SENDGRID_API_KEY")
SENDGRID_FROM_EMAIL = os.environ.get("SENDGRID_FROM_EMAIL")
# For Inbound email
SENDGRID_SECRET_KEY = os.environ.get("SENDGRID_SECRET_KEY")
SENDGRID_INBOUND_EMAIL_DOMAIN = os.environ.get("SENDGRID_INBOUND_EMAIL_DOMAIN")
# For Grafana Cloud integration
GRAFANA_CLOUD_ONCALL_API_URL = os.environ.get(
"GRAFANA_CLOUD_ONCALL_API_URL", "https://oncall-prod-us-central-0.grafana.net/oncall"
@ -198,13 +190,13 @@ INSTALLED_APPS = [
"apps.integrations",
"apps.schedules",
"apps.heartbeat",
"apps.email",
"apps.slack",
"apps.telegram",
"apps.twilioapp",
"apps.api",
"apps.api_for_grafana_incident",
"apps.base",
# "apps.sendgridapp", TODO: restore email notifications
"apps.auth_token",
"apps.public_api",
"apps.grafana_plugin",
@ -244,6 +236,7 @@ MIDDLEWARE = [
"django.middleware.clickjacking.XFrameOptionsMiddleware",
"social_django.middleware.SocialAuthExceptionMiddleware",
"apps.social_auth.middlewares.SocialAuthAuthCanceledExceptionMiddleware",
"apps.integrations.middlewares.IntegrationExceptionMiddleware",
]
LOG_REQUEST_ID_HEADER = "HTTP_X_CLOUD_TRACE_CONTEXT"
@ -569,6 +562,18 @@ SLOW_THRESHOLD_SECONDS = 2.0
EXTRA_MESSAGING_BACKENDS = []
# Email messaging backend
EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = os.getenv("EMAIL_HOST")
EMAIL_HOST_USER = os.getenv("EMAIL_HOST_USER")
EMAIL_HOST_PASSWORD = os.getenv("EMAIL_HOST_PASSWORD")
EMAIL_PORT = getenv_integer("EMAIL_PORT", 587)
EMAIL_USE_TLS = getenv_boolean("EMAIL_USE_TLS", True)
DEFAULT_FROM_EMAIL = os.getenv("DEFAULT_FROM_EMAIL")
if FEATURE_EMAIL_INTEGRATION_ENABLED:
EXTRA_MESSAGING_BACKENDS = [("apps.email.backend.EmailBackend", 8)]
INSTALLED_ONCALL_INTEGRATIONS = [
"config_integrations.alertmanager",
"config_integrations.grafana",

Some files were not shown because too many files have changed in this diff Show more