Grafana OnCall engine fork — self-hosted on-call scheduler and alert router
Find a file
Joey Orlando 3783aeab64
fix a few flaky e2e tests + allow running project locally via k8s/helm (#2751)
# What this PR does

- updates the GitHub Actions workflow to move the e2e tests into a
"[reusable
workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows#creating-a-reusable-workflow)"
which are run in two scenarios:
- all tests _except_ those annotated as `@expensive` are run against
`grafana/grafana:latest` on all feature branches
- all tests _including_ `@expensive` tests are run on weekdays @ 07h00
UTC, against a matrix of 6 grafana versions. Results of these builds
will be posted to `#irm-amixr-flux` Slack channel.
- local development will now be:
  ```bash
  make build-dev-images init-k8s start-k8s
  ```
- `build-dev-images` - builds the engine and UI docker images (only need
to run first time)
- `init-k8s` - creates a `kind` cluster and loads the two Docker images
onto the cluster nodes (only need to run first time)
- `start-k8s` - switches `kubectl` context to the created `kind`
cluster, and uses `helm` to deploy everything as defined in
`./dev/helm-local.yml` and `./dev/helm-local.dev.yml` (that latter file
is `.gitignored` and specific to how _you_ want your setup to look like.
Hot reloading works as before. This is the _start_ of #2381. (I've
marked these `make` commands as beta, because they've not yet been
thoroughly tested for local development).
- modifies the `helm` chart to add the concept of `oncall.devMode`,
`ui`, and ability to run oncall w/ sqlite
- `oncall.devMode` will essentially just add `volumes` and
`volumeMounts` to the various engine/migrate containers +
- `ui.enabled` + `ui.env` - create a ui container (which is needed for
hot reloading locally)
- `sqlite` - this was useful for the e2e test environments where Github
runner resources are scarce. Running `mariadb` eats up precious
resources, instead lets just use sqlite here
- fixes an issue that caused sporadic HTTP 502s from the grafana
plugin-proxy, which led to flaky tests. See [this
comment](https://github.com/grafana/oncall/pull/2751/files#diff-09040e8df192699b9c5742110ebbe8d9d5c3938cb156cc1cb99fa1c3fdee4fefR72-R77)
for more context + a link to a relevant Slack conversation. **tldr;**
there is a bug with the Grafana plugin proxy in Grafana >= v10.0.3.
Let's stop using the `latest`/`main` docker tags in our test and pin to
`10.0.2` for now
- ~~re-enables the e2e test which validates a phone number via SMS, and
asserts that we can receive an alert escalation via SMS (new Mailslurp
API Key has been added as a repo secret)~~ update: this is still blocked
by procurement, will be done in a future PR

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
2023-08-22 19:03:29 +02:00
.github fix a few flaky e2e tests + allow running project locally via k8s/helm (#2751) 2023-08-22 19:03:29 +02:00
dev fix a few flaky e2e tests + allow running project locally via k8s/helm (#2751) 2023-08-22 19:03:29 +02:00
docs add shift swap request demo video to public docs (#2858) 2023-08-22 15:26:08 +02:00
engine fix a few flaky e2e tests + allow running project locally via k8s/helm (#2751) 2023-08-22 19:03:29 +02:00
grafana-plugin fix a few flaky e2e tests + allow running project locally via k8s/helm (#2751) 2023-08-22 19:03:29 +02:00
helm fix a few flaky e2e tests + allow running project locally via k8s/helm (#2751) 2023-08-22 19:03:29 +02:00
terraform Remove unnecessary team checks (#2606) 2023-07-21 15:55:57 +01:00
tools Bump Python version in CI to 3.11.4 (#2707) 2023-08-01 10:11:05 +01:00
.drone.yml configure yamllint pre-commit step (#2728) 2023-08-03 02:35:08 -04:00
.gitignore ignore .http file extensions (#1762) 2023-04-17 10:52:03 +02:00
.markdownlint.json don't enforce line-length rule for markdownlint for code-blocks or tables (#2145) 2023-06-09 06:57:19 +00:00
.markdownlintignore Add tracing support 2022-12-19 17:15:06 +08:00
.pre-commit-config.yaml configure yamllint pre-commit step (#2728) 2023-08-03 02:35:08 -04:00
.yamllint.yml configure yamllint pre-commit step (#2728) 2023-08-03 02:35:08 -04:00
CHANGELOG.md Update CHANGELOG.md 2023-08-22 17:17:10 +02:00
CODE_OF_CONDUCT.md add precommit rules for markdown/json files (#915) 2022-12-01 14:26:54 +01:00
docker-compose-developer.yml increase mobile app verification token TTL from 1 minute to 5 minutes (#2859) 2023-08-22 17:09:59 +02:00
docker-compose-mysql-rabbitmq.yml fix make start command when using mysql/postgres as db (#2744) 2023-08-03 11:50:40 -04:00
docker-compose.yml fix make start command when using mysql/postgres as db (#2744) 2023-08-03 11:50:40 -04:00
GOVERNANCE.md add precommit rules for markdown/json files (#915) 2022-12-01 14:26:54 +01:00
LICENSE World, meet OnCall! 2022-06-03 08:09:47 -06:00
LICENSING.md add precommit rules for markdown/json files (#915) 2022-12-01 14:26:54 +01:00
MAINTAINERS.md add precommit rules for markdown/json files (#915) 2022-12-01 14:26:54 +01:00
Makefile fix a few flaky e2e tests + allow running project locally via k8s/helm (#2751) 2023-08-22 19:03:29 +02:00
README.md [README]fix prometheus yml indentation error (#2327) 2023-06-26 13:44:33 +00:00
screenshot.png Merge dev to main (#54) 2022-06-13 16:39:58 -06:00
screenshot_mobile.png Readme updates 2023-04-11 15:43:52 +03:00

Grafana OnCall

Latest Release License Docker Pulls Slack Discussion Build Status

Developer-friendly incident response with brilliant Slack integration.

  • Collect and analyze alerts from multiple monitoring systems
  • On-call rotations based on schedules
  • Automatic escalations
  • Phone calls, SMS, Slack, Telegram notifications

Getting Started

We prepared multiple environments:

  1. Download docker-compose.yml:

    curl -fsSL https://raw.githubusercontent.com/grafana/oncall/dev/docker-compose.yml -o docker-compose.yml
    
  2. Set variables:

    echo "DOMAIN=http://localhost:8080
    # Remove 'with_grafana' below if you want to use existing grafana
    # Add 'with_prometheus' below to optionally enable a local prometheus for oncall metrics
    # e.g. COMPOSE_PROFILES=with_grafana,with_prometheus
    COMPOSE_PROFILES=with_grafana
    # to setup an auth token for prometheus exporter metrics:
    # PROMETHEUS_EXPORTER_SECRET=my_random_prometheus_secret
    # also, make sure to enable the /metrics endpoint:
    # FEATURE_PROMETHEUS_EXPORTER_ENABLED=True
    SECRET_KEY=my_random_secret_must_be_more_than_32_characters_long" > .env
    
  3. (Optional) If you want to enable/setup the prometheus metrics exporter (besides the changes above), create a prometheus.yml file (replacing my_random_prometheus_secret accordingly), next to your docker-compose.yml:

    echo "global:
      scrape_interval:     15s
      evaluation_interval: 15s
    
    scrape_configs:
      - job_name: prometheus
        metrics_path: /metrics/
        authorization:
          credentials: my_random_prometheus_secret
        static_configs:
          - targets: [\"host.docker.internal:8080\"]" > prometheus.yml
    

    NOTE: you will need to setup a Prometheus datasource using http://prometheus:9090 as the URL in the Grafana UI.

  4. Launch services:

    docker-compose pull && docker-compose up -d
    
  5. Go to OnCall Plugin Configuration, using log in credentials as defined above: admin/admin (or find OnCall plugin in configuration->plugins) and connect OnCall plugin with OnCall backend:

    OnCall backend URL: http://engine:8080
    
  6. Enjoy! Check our OSS docs if you want to set up Slack, Telegram, Twilio or SMS/calls through Grafana Cloud.

Update version

To update your Grafana OnCall hobby environment:

# Update Docker image
docker-compose pull engine

# Re-deploy
docker-compose up -d

After updating the engine, you'll also need to click the "Update" button on the plugin version page. See Grafana docs for more info on updating Grafana plugins.

Join community

Stargazers over time

Stargazers over time

Further Reading