Grafana OnCall engine fork — self-hosted on-call scheduler and alert router
eefe7be56a
Occasionally, the Playwright global setup step (which authenticates w/ the Grafana API + configures the plugin) would fail, leading to the CI job to instantly fail (playwright doesn't retry global setup if it fails). My current hypothesis as to why this is happening is because the `oncall-engine` and `oncall-celery` pods aren't _actually_ ready in these cases based on the way the `jupyterhub/action-k8s-await-workloads` action await k8s workloads: <img width="1076" alt="Screenshot 2023-05-23 at 18 24 36" src="https://github.com/grafana/oncall/assets/9406895/68d8d2d9-4274-4749-8788-e0a9a3dbad83"> By using the `kubectl rollout status deployment/<deployment-name> --timeout=300s` instead, we can be sure that these pods are _actually_ ready to receive traffic before we start the tests. ```bash ❯ kubectl rollout status --help Show the status of the rollout. By default 'rollout status' will watch the status of the latest rollout until it's done. If you don't want to wait for the rollout to finish then you can use --watch=false. Note that if a new rollout starts in-between, then 'rollout status' will continue watching the latest revision. If you want to pin to a specific revision and abort if it is rolled over by another revision, use --revision=N where N is the revision you need to watch for. ``` Lastly, even despite this, sometimes the `POST /api/internal/v1/plugin/sync` endpoint will return HTTP 500 ([example logs](https://github.com/grafana/oncall/actions/runs/5062712137/jobs/9088529416#step:19:2536) from failed CI job). In this case, let's setup the Playwright global setup to retry 3 times. |
||
|---|---|---|
| .github | ||
| dev | ||
| docs | ||
| engine | ||
| examples/terraform | ||
| grafana-plugin | ||
| helm | ||
| tools | ||
| .drone.yml | ||
| .gitignore | ||
| .markdownlint.json | ||
| .markdownlintignore | ||
| .pre-commit-config.yaml | ||
| CHANGELOG.md | ||
| CODE_OF_CONDUCT.md | ||
| docker-compose-developer.yml | ||
| docker-compose-mysql-rabbitmq.yml | ||
| docker-compose.yml | ||
| GOVERNANCE.md | ||
| LICENSE | ||
| LICENSING.md | ||
| MAINTAINERS.md | ||
| Makefile | ||
| README.md | ||
| screenshot.png | ||
| screenshot_mobile.png | ||
Grafana OnCall
Developer-friendly incident response with brilliant Slack integration.
 |
- Collect and analyze alerts from multiple monitoring systems
- On-call rotations based on schedules
- Automatic escalations
- Phone calls, SMS, Slack, Telegram notifications
Getting Started
We prepared multiple environments:
- production
- developer
- hobby (described in the following steps)
-
Download
docker-compose.yml:curl -fsSL https://raw.githubusercontent.com/grafana/oncall/dev/docker-compose.yml -o docker-compose.yml -
Set variables:
echo "DOMAIN=http://localhost:8080 COMPOSE_PROFILES=with_grafana # Remove this line if you want to use existing grafana SECRET_KEY=my_random_secret_must_be_more_than_32_characters_long" > .env -
Launch services:
docker-compose pull && docker-compose up -d -
Go to OnCall Plugin Configuration, using log in credentials as defined above:
admin/admin(or find OnCall plugin in configuration->plugins) and connect OnCall plugin with OnCall backend:OnCall backend URL: http://engine:8080 -
Enjoy! Check our OSS docs if you want to set up Slack, Telegram, Twilio or SMS/calls through Grafana Cloud.
Update version
To update your Grafana OnCall hobby environment:
# Update Docker image
docker-compose pull engine
# Re-deploy
docker-compose up -d
After updating the engine, you'll also need to click the "Update" button on the plugin version page. See Grafana docs for more info on updating Grafana plugins.
Join community
Stargazers over time
Further Reading
- Migration from PagerDuty - Migrator
- Documentation - Grafana OnCall
- Overview Webinar - YouTube
- How To Add Integration - How to Add Integration
- Blog Post - Announcing Grafana OnCall, the easiest way to do on-call management
- Presentation - Deep dive into the Grafana, Prometheus, and Alertmanager stack for alerting and on-call management