Grafana OnCall engine fork — self-hosted on-call scheduler and alert router
ab700fd4a1
We are noticing some [issues](https://ops.grafana-ops.net/goto/sQFLv4XSg?orgId=1) when updating routes via Terraform because when receiving multiple concurrent requests updating position, the order is updated in a transaction but it is then tried to save again in the `.save` call, which could lead to `IntegrityError`s, because the order may have been updated in a concurrent request meanwhile. Test run with the reproduced error (before the serializer update): ``` _____________________________ test_ordered_model_swap_all_to_zero_via_serializer _____________________________ @pytest.mark.django_db(transaction=True) def test_ordered_model_swap_all_to_zero_via_serializer(): THREADS = 300 exceptions = [] TestOrderedModel.objects.all().delete() # clear table instances = [TestOrderedModel.objects.create(test_field="test") for _ in range(THREADS)] # generate random non-unique orders random.seed(42) positions = [random.randint(0, THREADS - 1) for _ in range(THREADS)] def update_order_to_zero(idx): try: instance = instances[idx] serializer = TestOrderedModelSerializer( instance, data={"order": 0, "extra_field": idx}, partial=True ) serializer.is_valid(raise_exception=True) serializer.save() instance.swap(positions[idx]) except Exception as e: exceptions.append(e) threads = [threading.Thread(target=update_order_to_zero, args=(0,)) for _ in range(THREADS)] for thread in threads: thread.start() for thread in threads: thread.join() # can only check that orders are still sequential and that there are no exceptions # can't check the exact order because it changes depending on the order of execution > assert not exceptions E assert not [IntegrityError(1062, "Duplicate entry 'test-0' for key 'base_testorderedmodel.unique_test_field_order'"), IntegrityEr...rder'"), IntegrityError(1062, "Duplicate entry 'test-0' for key 'base_testorderedmodel.unique_test_field_order'"), ...] THREADS = 300 exceptions = [IntegrityError(1062, "Duplicate entry 'test-0' for key 'base_testorderedmodel.unique_test_field_order'"), IntegrityEr...rder'"), IntegrityError(1062, "Duplicate entry 'test-0' for key 'base_testorderedmodel.unique_test_field_order'"), ...] instances = [<TestOrderedModel: TestOrderedModel object (841)>, <TestOrderedModel: TestOrderedModel object (842)>, <TestOrderedMod...ject (844)>, <TestOrderedModel: TestOrderedModel object (845)>, <TestOrderedModel: TestOrderedModel object (846)>, ...] positions = [57, 12, 140, 125, 114, 71, ...] thread = <Thread(Thread-302 (update_order_to_zero), stopped 139636013323840)> threads = [<Thread(Thread-3 (update_order_to_zero), stopped 139646722418240)>, <Thread(Thread-4 (update_order_to_zero), stopped ...ate_order_to_zero), stopped 139646608590400)>, <Thread(Thread-8 (update_order_to_zero), stopped 139646600197696)>, ...] update_order_to_zero = <function test_ordered_model_swap_all_to_zero_via_serializer.<locals>.update_order_to_zero at 0x7f02086274c0> common/tests/test_ordered_model.py:481: AssertionError ``` |
||
|---|---|---|
| .github | ||
| .tilt | ||
| dev | ||
| docs | ||
| engine | ||
| grafana-plugin | ||
| helm | ||
| terraform | ||
| tools | ||
| .dockerignore | ||
| .gitattributes | ||
| .gitignore | ||
| .markdownlint.json | ||
| .markdownlintignore | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| .prettierrc.js | ||
| .yamllint.yml | ||
| CHANGELOG.md | ||
| docker-compose-developer.yml | ||
| docker-compose-mysql-rabbitmq.yml | ||
| docker-compose.yml | ||
| LICENSE | ||
| Makefile | ||
| README.md | ||
| Tiltfile | ||
Grafana OnCall
Developer-friendly incident response with brilliant Slack integration.
 |
- Collect and analyze alerts from multiple monitoring systems
- On-call rotations based on schedules
- Automatic escalations
- Phone calls, SMS, Slack, Telegram notifications
Getting Started
We prepared multiple environments:
- production
- developer
- hobby (described in the following steps)
-
Download
docker-compose.yml:curl -fsSL https://raw.githubusercontent.com/grafana/oncall/dev/docker-compose.yml -o docker-compose.yml -
Set variables:
echo "DOMAIN=http://localhost:8080 # Remove 'with_grafana' below if you want to use existing grafana # Add 'with_prometheus' below to optionally enable a local prometheus for oncall metrics # e.g. COMPOSE_PROFILES=with_grafana,with_prometheus COMPOSE_PROFILES=with_grafana # to setup an auth token for prometheus exporter metrics: # PROMETHEUS_EXPORTER_SECRET=my_random_prometheus_secret # also, make sure to enable the /metrics endpoint: # FEATURE_PROMETHEUS_EXPORTER_ENABLED=True SECRET_KEY=my_random_secret_must_be_more_than_32_characters_long" > .env -
(Optional) If you want to enable/setup the prometheus metrics exporter (besides the changes above), create a
prometheus.ymlfile (replacingmy_random_prometheus_secretaccordingly), next to yourdocker-compose.yml:echo "global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: prometheus metrics_path: /metrics/ authorization: credentials: my_random_prometheus_secret static_configs: - targets: [\"host.docker.internal:8080\"]" > prometheus.ymlNOTE: you will need to setup a Prometheus datasource using
http://prometheus:9090as the URL in the Grafana UI. -
Launch services:
docker-compose pull && docker-compose up -d -
Go to OnCall Plugin Configuration, using log in credentials as defined above:
admin/admin(or find OnCall plugin in configuration->plugins) and connect OnCall plugin with OnCall backend:OnCall backend URL: http://engine:8080 -
Enjoy! Check our OSS docs if you want to set up Slack, Telegram, Twilio or SMS/calls through Grafana Cloud.
Update version
To update your Grafana OnCall hobby environment:
# Update Docker image
docker-compose pull engine
# Re-deploy
docker-compose up -d
After updating the engine, you'll also need to click the "Update" button on the plugin version page. See Grafana docs for more info on updating Grafana plugins.
Join community
Have a question, comment or feedback? Don't be afraid to open an issue!
Stargazers over time
Further Reading
- Automated migration from other on-call tools - Migrator
- Documentation - Grafana OnCall
- Overview Webinar - YouTube
- How To Add Integration - How to Add Integration
- Blog Post - Announcing Grafana OnCall, the easiest way to do on-call management
- Presentation - Deep dive into the Grafana, Prometheus, and Alertmanager stack for alerting and on-call management