# Developer quickstart - [Running the project](#running-the-project) - [`COMPOSE_PROFILES`](#compose_profiles) - [`GRAFANA_IMAGE`](#grafana_image) - [Configuring Grafana](#configuring-grafana) - [Enabling RBAC for OnCall for local development](#enabling-rbac-for-oncall-for-local-development) - [Django Silk Profiling](#django-silk-profiling) - [Running backend services outside Docker](#running-backend-services-outside-docker) - [UI Integration Tests](#ui-integration-tests) - [Useful `make` commands](#useful-make-commands) - [Setting environment variables](#setting-environment-variables) - [Slack application setup](#slack-application-setup) - [Update drone build](#update-drone-build) - [Troubleshooting](#troubleshooting) - [ld: library not found for -lssl](#ld-library-not-found-for--lssl) - [Could not build wheels for cryptography which use PEP 517 and cannot be installed directly](#could-not-build-wheels-for-cryptography-which-use-pep-517-and-cannot-be-installed-directly) - [django.db.utils.OperationalError: (1366, "Incorrect string value")](#djangodbutilsoperationalerror-1366-incorrect-string-value) - [/bin/sh: line 0: cd: grafana-plugin: No such file or directory](#binsh-line-0-cd-grafana-plugin-no-such-file-or-directory) - [Encountered error while trying to install package - grpcio](#encountered-error-while-trying-to-install-package---grpcio) - [distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1](#distutilserrorscompileerror-command-usrbinclang-failed-with-exit-code-1) - [symbol not found in flat namespace '\_EVP_DigestSignUpdate'](#symbol-not-found-in-flat-namespace-_evp_digestsignupdate) - [IDE Specific Instructions](#ide-specific-instructions) - [PyCharm](#pycharm) - [How to write database migrations](#how-to-write-database-migrations) Related: [How to develop integrations](/engine/config_integrations/README.md) ## Running the project By default everything runs inside Docker. These options can be modified via the [`COMPOSE_PROFILES`](#compose_profiles) environment variable. 1. Firstly, ensure that you have `docker` [installed](https://docs.docker.com/get-docker/) and running on your machine. **NOTE**: the `docker-compose-developer.yml` file uses some syntax/features that are only supported by Docker Compose v2. For instructions on how to enable this (if you haven't already done so), see [here](https://www.docker.com/blog/announcing-compose-v2-general-availability/). Ensure you have Docker Compose version 2.10 or above installed - update instructions are [here](https://docs.docker.com/compose/install/linux/). 2. Run `make init start`. By default this will run everything in Docker, using SQLite as the database and Redis as the message broker/cache. See [`COMPOSE_PROFILES`](#compose_profiles) below for more details on how to swap out/disable which components are run in Docker. 3. Open Grafana in a browser [here](http://localhost:3000/plugins/grafana-oncall-app) (login: `oncall`, password: `oncall`). 4. You should now see the OnCall plugin configuration page. You may safely ignore the warning about the invalid plugin signature. Set "OnCall backend URL" as "http://host.docker.internal:8080". When opening the main plugin page, you may also ignore warnings about version mismatch and lack of communication channels. 5. Enjoy! Check our [OSS docs](https://grafana.com/docs/oncall/latest/open-source/) if you want to set up Slack, Telegram, Twilio or SMS/calls through Grafana Cloud. 6. (Optional) Install `pre-commit` hooks by running `make install-precommit-hook` **Note**: on subsequent startups you can simply run `make start`, this is a bit faster because it skips the frontend build step. ### `COMPOSE_PROFILES` This configuration option represents a comma-separated list of [`docker-compose` profiles](https://docs.docker.com/compose/profiles/). It allows you to swap-out, or disable, certain components in Docker. This option can be configured in two ways: 1. Setting a `COMPOSE_PROFILES` environment variable in `dev/.env.dev`. This allows you to avoid having to set `COMPOSE_PROFILES` for each `make` command you execute afterwards. 2. Passing in a `COMPOSE_PROFILES` argument when running `make` commands. For example: ```bash make start COMPOSE_PROFILES=postgres,engine,grafana,rabbitmq ``` The possible profiles values are: - `grafana` - `prometheus` - `engine` - `oncall_ui` - `redis` - `rabbitmq` - `postgres` - `mysql` The default is `engine,oncall_ui,redis,grafana`. This runs: - all OnCall components (using SQLite as the database) - Redis as the Celery message broker/cache - a Grafana container ### `GRAFANA_IMAGE` If you would like to change the image or version of Grafana being run, simply pass in a `GRAFANA_IMAGE` environment variable to `make start` (or alternatively set it in your root `.env` file). The value of this environment variable should be a valid `grafana` image/tag combination (ex. `grafana:main` or `grafana-enterprise:latest`). ### Configuring Grafana This section is applicable for when you are running a Grafana container inside of `docker-compose` and you would like to modify your Grafana instance's provisioning configuration. The following commands assume you run them from the root of the project: ```bash touch ./dev/grafana.dev.ini # make desired changes to ./dev/grafana.dev.ini then run touch .env && ./dev/add_env_var.sh GRAFANA_DEV_PROVISIONING ./dev/grafana.dev.ini .env ``` For example, if you would like to enable the `topnav` feature toggle, you can modify your `./dev/grafana.dev.ini` as such: ```ini [feature_toggles] enable = top_nav ``` The next time you start the project via `docker-compose`, the `grafana` container will have `./dev/grafana.dev.ini` volume mounted inside the container. ### Enabling RBAC for OnCall for local development To run the project locally w/ RBAC for OnCall enabled, you will first need to run a `grafana-enterprise` container, instead of a `grafana` container. See the instructions [here](#grafana_image) on how to do so. Next, you will need to follow the steps [here](https://grafana.com/docs/grafana/latest/administration/enterprise-licensing/) on setting up/downloading a Grafana Enterprise license. Lastly, you will need to modify the instance's configuration. Follow the instructions [here](#configuring-grafana) on how to do so. You can modify your configuration file (`./dev/grafana.dev.ini`) as such: ```ini [rbac] enabled = true [feature_toggles] enable = accessControlOnCall [server] root_url = https://.grafana.net/ [enterprise] license_text = ``` (_Note_: you may need to restart your `grafana` container after modifying its configuration) ### Enabling OnCall prometheus exporter for local development Add `prometheus` to your `COMPOSE_PROFILES` and set `FEATURE_PROMETHEUS_EXPORTER_ENABLED=True` in your `dev/.env.dev` file. You may need to restart your `grafana` container to make sure the new datasource is added (or add it manually using the UI; Prometheus will be running in `host.docker.internal:9090` by default, using default settings). ### Django Silk Profiling In order to setup [`django-silk`](https://github.com/jazzband/django-silk) for local profiling, perform the following steps: 1. `make backend-debug-enable` 2. `make engine-manage CMD="createsuperuser"` - follow CLI prompts to create a Django superuser 3. Visit and login using the credentials you created in step #2 You should now be able to visit and see the Django Silk UI. See the `django-silk` documentation [here](https://github.com/jazzband/django-silk) for more information. ### Running backend services outside Docker By default everything runs inside Docker. If you would like to run the backend services outside of Docker (for integrating w/ PyCharm for example), follow these instructions: 1. Create a Python 3.11 virtual environment using a method of your choosing (ex. [venv](https://docs.python.org/3.11/library/venv.html) or [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv)). Make sure the virtualenv is "activated". 2. `postgres` is a dependency on some of our Python dependencies (notably `psycopg2` ([docs](https://www.psycopg.org/docs/install.html#prerequisites))). Please visit [here](https://www.postgresql.org/download/) for installation instructions. 3. `make backend-bootstrap` - installs all backend dependencies 4. Modify your `.env.dev` by copying the contents of one of `.env.mysql.dev`, `.env.postgres.dev`, or `.env.sqlite.dev` into `.env.dev` (you should exclude the `GF_` prefixed environment variables). > In most cases where you are running stateful services via `docker-compose`, and backend services outside of > docker, you will simply need to change the database host to `localhost` (or in the case of `sqlite` update > the file-path to your `sqlite` database file). You will need to change the broker host to `localhost` as well. 5. `make backend-migrate` - runs necessary database migrations 6. Open two separate shells and then run the following: - `make run-backend-server` - runs the HTTP server - `make run-backend-celery` - runs Celery workers ## UI Integration Tests We've developed a suite of "end-to-end" integration tests using [Playwright](https://playwright.dev/). These tests are run on pull request CI builds. New features should ideally include a new/modified integration test. To run these tests locally simply do the following: ```bash npx playwright install # install playwright dependencies cp ./grafana-plugin/.env.example ./grafana-plugin/.env # you may need to tweak the values in ./grafana-plugin/.env according to your local setup cd grafana-plugin yarn test:integration ``` ## Useful `make` commands > 🚶‍This part was moved to `make help` command. Run it to see all the available commands and their descriptions ## Setting environment variables If you need to override any additional environment variables, you should set these in a root `.env.dev` file. This file is automatically picked up by the OnCall engine Docker containers. This file is ignored from source control and also overrides any defaults that are set in other `.env*` files ## Slack application setup For Slack app configuration check our docs: ## Update drone build The `.drone.yml` build file must be signed when changes are made to it. Follow these steps: If you have not installed drone CLI follow [these instructions](https://docs.drone.io/cli/install/) To sign the `.drone.yml` file: ```bash export DRONE_SERVER=https://drone.grafana.net # Get your drone token from https://drone.grafana.net/account export DRONE_TOKEN= drone sign --save grafana/oncall .drone.yml ``` ## Troubleshooting ### ld: library not found for -lssl **Problem:** ```bash make backend-bootstrap ... ld: library not found for -lssl clang: error: linker command failed with exit code 1 (use -v to see invocation) error: command 'gcc' failed with exit status 1 ... ``` **Solution:** ```bash export LDFLAGS=-L/usr/local/opt/openssl/lib make backend-bootstrap ``` ### Could not build wheels for cryptography which use PEP 517 and cannot be installed directly Happens on Apple Silicon **Problem:** ```bash build/temp.macosx-12-arm64-3.9/_openssl.c:575:10: fatal error: 'openssl/opensslv.h' file not found #include ^~~~~~~~~~~~~~~~~~~~ 1 error generated. error: command '/usr/bin/clang' failed with exit code 1 ---------------------------------------- ERROR: Failed building wheel for cryptography ``` **Solution:** ```bash LDFLAGS="-L$(brew --prefix openssl@1.1)/lib" CFLAGS="-I$(brew --prefix openssl@1.1)/include" pip install `cat engine/requirements.txt | grep cryptography` ``` ### django.db.utils.OperationalError: (1366, "Incorrect string value") **Problem:** ```bash django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A\\xF0\\x9F...' for column 'cached_name' at row 1") ``` **Solution:** Recreate the database with the correct encoding. ### /bin/sh: line 0: cd: grafana-plugin: No such file or directory **Problem:** When running `make init`: ```bash /bin/sh: line 0: cd: grafana-plugin: No such file or directory make: *** [init] Error 1 ``` This arises when the environment variable `[CDPATH](https://www.theunixschool.com/2012/04/what-is-cdpath.html)` is set _and_ when the current path (`.`) is not explicitly part of `CDPATH`. **Solution:** Either make `.` part of `CDPATH` in your .rc file setup, or temporarily override the variable when running `make` commands: ```bash $ CDPATH="." make init # Setting CDPATH to empty seems to also work - only tested on zsh, YMMV $ CDPATH="" make init ``` **Problem:** When running `make init start`: ```bash Error response from daemon: open /var/lib/docker/overlay2/ac57b871108ee1b98ff4455e36d2175eae90cbc7d4c9a54608c0b45cfb7c6da5/committed: is a directory make: *** [start] Error 1 ``` **Solution:** clear everything in docker by resetting or: ```bash make cleanup ``` ### Encountered error while trying to install package - grpcio **Problem:** We are currently using a library, `fcm-django`, which has a dependency on `grpcio`. Google does not provide `grpcio` wheels built for Apple Silicon Macs. The best solution so far has been to use a `conda` virtualenv. There's apparently a lot of community work put into making packages play well with M1/arm64 architecture. ```bash pip install -r requirements.txt ... note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> grpcio ... ``` **Solution:** Use a `conda` virtualenv, and then run the following when installing the engine dependencies/ [See here for more details](https://stackoverflow.com/a/74307636/3902555) ```bash GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt ``` ### distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1 See solution for "Encountered error while trying to install package - grpcio" [here](#encountered-error-while-trying-to-install-package---grpcio) ### symbol not found in flat namespace '\_EVP_DigestSignUpdate' **Problem:** This problem seems to occur when running the Celery process, outside of `docker-compose` (via `make run-backend-celery`), and using a `conda` virtual environment. ```bash conda create --name oncall-dev python=3.9.13 conda activate oncall-dev make backend-bootstrap make run-backend-celery File "~/oncall/engine/engine/__init__.py", line 5, in from .celery import app as celery_app File "~/oncall/engine/engine/celery.py", line 11, in from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter File "/opt/homebrew/Caskroom/miniconda/base/envs/oncall-dev/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/grpc/trace_exporter/__init__.py", line 20, in from grpc import ChannelCredentials, Compression File "/opt/homebrew/Caskroom/miniconda/base/envs/oncall-dev/lib/python3.9/site-packages/grpc/__init__.py", line 22, in from grpc import _compression File "/opt/homebrew/Caskroom/miniconda/base/envs/oncall-dev/lib/python3.9/site-packages/grpc/_compression.py", line 20, in from grpc._cython import cygrpc ImportError: dlopen(/opt/homebrew/Caskroom/miniconda/base/envs/oncall-dev/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_EVP_DigestSignUpdate' ``` **Solution:** [This solution](https://github.com/grpc/grpc/issues/15510#issuecomment-392012594) posted in a GitHub issue thread for the `grpc/grpc` repository, fixes the issue: ```bash conda install grpcio make run-backend-celery ``` ## IDE Specific Instructions ### PyCharm 1. Follow the instructions listed in ["Running backend services outside Docker"](#running-backend-services-outside-docker). 2. Open the project in PyCharm 3. Settings → Project OnCall - In Python Interpreter click the gear and create a new Virtualenv from existing environment selecting the venv created in Step 1. - In Project Structure make sure the project root is the content root and add /engine to Sources 4. Under Settings → Languages & Frameworks → Django - Enable Django support - Set Django project root to /engine - Set Settings to settings/dev.py 5. Create a new Django Server run configuration to Run/Debug the engine - Use a plugin such as EnvFile to load the .env.dev file - Change port from 8000 to 8080 ## How to write database migrations We use [django-migration-linter](https://github.com/3YOURMIND/django-migration-linter) to keep database migrations backwards compatible - we can automatically run migrations and they are zero-downtime, e.g. old code can work with the migrated database - we can run and rollback migrations without worrying about data safety - OnCall is deployed to the multiple environments core team is not able to control See [django-migration-linter checklist](https://github.com/3YOURMIND/django-migration-linter/blob/main/docs/incompatibilities.md) for the common mistakes and best practices