Grafana OnCall engine fork — self-hosted on-call scheduler and alert router
014a9c2ec2
# What this PR does https://www.loom.com/share/18cc445117de4895a10892d56c7d3699 In preparation to upgrade our cloud databases, this PR makes some minor changes which, after testing locally, allowed the `POST /<integration_type>/<alert_channel_key>` endpoints to successfully receive incoming alerts and queue the celery tasks. I've tested all of the defined `POST /integrations/v1/<integration_type>/<alert_channel_key>` endpoints by sending `POST` requests to an integrations' URL while the MySQL database was down, bringing the database back up, and ensuring the alerts were created. ## Some other findings - the integration heartbeat endpoints will not work as we interact w/ the database to persist the incoming heartbeat instance - if the integration was created in the last 180 seconds, incoming alerts will fail due to the way we cache the integration IDs ([code](https://github.com/grafana/oncall/blob/dev/engine/apps/integrations/mixins/alert_channel_defining_mixin.py#L47-L50)) - The `create_alert` celery task is set to `max_retries=None` and `retry_backoff=True`. This means that the queued tasks will continue retrying forever w/ an exponential backoff, until the alerts can be created in the database (ie. when the database is back online). ## Checklist - [ ] Unit, integration, and e2e (if applicable) tests updated (N/A) - [ ] Documentation added (or `pr:no public docs` PR label added if not required) (N/A) - [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not required) (N/A) |
||
|---|---|---|
| .github | ||
| dev | ||
| docs | ||
| engine | ||
| examples/terraform | ||
| grafana-plugin | ||
| helm | ||
| tools | ||
| .drone.yml | ||
| .gitignore | ||
| .markdownlint.json | ||
| .markdownlintignore | ||
| .nvmrc | ||
| .pre-commit-config.yaml | ||
| CHANGELOG.md | ||
| CODE_OF_CONDUCT.md | ||
| docker-compose-developer.yml | ||
| docker-compose-mysql-rabbitmq.yml | ||
| docker-compose.yml | ||
| GOVERNANCE.md | ||
| LICENSE | ||
| LICENSING.md | ||
| MAINTAINERS.md | ||
| Makefile | ||
| README.md | ||
| screenshot.png | ||
| screenshot_mobile.png | ||
| SECURITY.md | ||
Grafana OnCall
Developer-friendly incident response with brilliant Slack integration.
 |
- Collect and analyze alerts from multiple monitoring systems
- On-call rotations based on schedules
- Automatic escalations
- Phone calls, SMS, Slack, Telegram notifications
Getting Started
We prepared multiple environments:
- production
- developer
- hobby (described in the following steps)
-
Download
docker-compose.yml:curl -fsSL https://raw.githubusercontent.com/grafana/oncall/dev/docker-compose.yml -o docker-compose.yml -
Set variables:
echo "DOMAIN=http://localhost:8080 COMPOSE_PROFILES=with_grafana # Remove this line if you want to use existing grafana SECRET_KEY=my_random_secret_must_be_more_than_32_characters_long" > .env -
Launch services:
docker-compose pull && docker-compose up -d -
Go to OnCall Plugin Configuration, using log in credentials as defined above:
admin/admin(or find OnCall plugin in configuration->plugins) and connect OnCall plugin with OnCall backend:OnCall backend URL: http://engine:8080 -
Enjoy! Check our OSS docs if you want to set up Slack, Telegram, Twilio or SMS/calls through Grafana Cloud.
Update version
To update your Grafana OnCall hobby environment:
# Update Docker image
docker-compose pull engine
# Re-deploy
docker-compose up -d
After updating the engine, you'll also need to click the "Update" button on the plugin version page. See Grafana docs for more info on updating Grafana plugins.
Join community
Stargazers over time
Further Reading
- Migration from PagerDuty - Migrator
- Documentation - Grafana OnCall
- Overview Webinar - YouTube
- How To Add Integration - How to Add Integration
- Blog Post - Announcing Grafana OnCall, the easiest way to do on-call management
- Presentation - Deep dive into the Grafana, Prometheus, and Alertmanager stack for alerting and on-call management