Grafana OnCall engine fork — self-hosted on-call scheduler and alert router
Find a file
Michael Derynck 8db285e4e8
Fix escalation chain webhooks executing when disabled (#4319)
# What this PR does
Fixes issue where custom webhooks that are part of an escalation chain
were still being executed even though they were disabled. Now the
attempt will be logged in the escalation log and noted that the webhook
was disabled.

## Which issue(s) this PR closes

<!--
*Note*: if you have more than one GitHub issue that this PR closes, be
sure to preface
each issue link with a [closing
keyword](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue).
This ensures that the issue(s) are auto-closed once the PR has been
merged.
-->

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
2024-05-07 19:58:49 +00:00
.github Use ops-devenv only in expensive tests (#4285) 2024-04-26 17:41:08 -04:00
dev improve frontend dev exp (#4296) 2024-05-06 05:02:32 +00:00
docs Add acknowledged, resolved user information on webhook payload (#4176) 2024-04-26 21:50:08 +00:00
engine Fix escalation chain webhooks executing when disabled (#4319) 2024-05-07 19:58:49 +00:00
grafana-plugin improve frontend dev exp (#4296) 2024-05-06 05:02:32 +00:00
helm Add retry queue to helm chart (#4189) 2024-04-10 15:51:54 +00:00
terraform Remove unnecessary team checks (#2606) 2023-07-21 15:55:57 +01:00
tools Bump idna from 3.6 to 3.7 in /tools/pagerduty-migrator (#4291) 2024-04-26 17:40:39 -04:00
.dockerignore WIP: Direct paging improvements (#3064) 2023-09-28 03:57:49 +00:00
.drone.yml Switch to uv Python package installer/resolver (#4005) 2024-04-26 14:30:38 +00:00
.gitignore Brojd/improve e2e tests dx (#3516) 2023-12-15 08:58:25 +00:00
.markdownlint.json don't enforce line-length rule for markdownlint for code-blocks or tables (#2145) 2023-06-09 06:57:19 +00:00
.markdownlintignore Add tracing support 2022-12-19 17:15:06 +08:00
.pre-commit-config.yaml SNOW minor wording & docs URL updates (#4229) 2024-04-16 12:16:35 +00:00
.prettierignore Brojd/update insights docs (#3692) 2024-01-22 11:26:07 +00:00
.prettierrc.js Brojd/improve e2e tests dx (#3516) 2023-12-15 08:58:25 +00:00
.yamllint.yml configure yamllint pre-commit step (#2728) 2023-08-03 02:35:08 -04:00
CHANGELOG.md Update CHANGELOG.md (#4041) 2024-03-12 07:47:08 +00:00
CODE_OF_CONDUCT.md add precommit rules for markdown/json files (#915) 2022-12-01 14:26:54 +01:00
docker-compose-developer.yml Fix daily e2e tests GitHub Actions cron job (#3923) 2024-02-27 09:21:53 -05:00
docker-compose-mysql-rabbitmq.yml fix make start command when using mysql/postgres as db (#2744) 2023-08-03 11:50:40 -04:00
docker-compose.yml Update docker-compose.yml (#3266) 2023-11-03 17:09:24 +08:00
LICENSE World, meet OnCall! 2022-06-03 08:09:47 -06:00
LICENSING.md add precommit rules for markdown/json files (#915) 2022-12-01 14:26:54 +01:00
Makefile Switch to uv Python package installer/resolver (#4005) 2024-04-26 14:30:38 +00:00
README.md Update readme (#4012) 2024-03-05 16:12:54 +00:00
screenshot.png Merge dev to main (#54) 2022-06-13 16:39:58 -06:00
screenshot_mobile.png Readme updates 2023-04-11 15:43:52 +03:00
Tiltfile improve frontend dev exp (#4296) 2024-05-06 05:02:32 +00:00

Grafana OnCall

Latest Release License Docker Pulls Slack Discussion Build Status

Developer-friendly incident response with brilliant Slack integration.

  • Collect and analyze alerts from multiple monitoring systems
  • On-call rotations based on schedules
  • Automatic escalations
  • Phone calls, SMS, Slack, Telegram notifications

Getting Started

We prepared multiple environments:

  1. Download docker-compose.yml:

    curl -fsSL https://raw.githubusercontent.com/grafana/oncall/dev/docker-compose.yml -o docker-compose.yml
    
  2. Set variables:

    echo "DOMAIN=http://localhost:8080
    # Remove 'with_grafana' below if you want to use existing grafana
    # Add 'with_prometheus' below to optionally enable a local prometheus for oncall metrics
    # e.g. COMPOSE_PROFILES=with_grafana,with_prometheus
    COMPOSE_PROFILES=with_grafana
    # to setup an auth token for prometheus exporter metrics:
    # PROMETHEUS_EXPORTER_SECRET=my_random_prometheus_secret
    # also, make sure to enable the /metrics endpoint:
    # FEATURE_PROMETHEUS_EXPORTER_ENABLED=True
    SECRET_KEY=my_random_secret_must_be_more_than_32_characters_long" > .env
    
  3. (Optional) If you want to enable/setup the prometheus metrics exporter (besides the changes above), create a prometheus.yml file (replacing my_random_prometheus_secret accordingly), next to your docker-compose.yml:

    echo "global:
      scrape_interval:     15s
      evaluation_interval: 15s
    
    scrape_configs:
      - job_name: prometheus
        metrics_path: /metrics/
        authorization:
          credentials: my_random_prometheus_secret
        static_configs:
          - targets: [\"host.docker.internal:8080\"]" > prometheus.yml
    

    NOTE: you will need to setup a Prometheus datasource using http://prometheus:9090 as the URL in the Grafana UI.

  4. Launch services:

    docker-compose pull && docker-compose up -d
    
  5. Go to OnCall Plugin Configuration, using log in credentials as defined above: admin/admin (or find OnCall plugin in configuration->plugins) and connect OnCall plugin with OnCall backend:

    OnCall backend URL: http://engine:8080
    
  6. Enjoy! Check our OSS docs if you want to set up Slack, Telegram, Twilio or SMS/calls through Grafana Cloud.

Update version

To update your Grafana OnCall hobby environment:

# Update Docker image
docker-compose pull engine

# Re-deploy
docker-compose up -d

After updating the engine, you'll also need to click the "Update" button on the plugin version page. See Grafana docs for more info on updating Grafana plugins.

Join community

Have a question, comment or feedback? Don't be afraid to open an issue!

Stargazers over time

Stargazers over time

Further Reading