centralcloud/oncall-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add database migrations linter (#1020)

Browse source 
# What this PR does

This PR adds
[django-migration-linter](https://github.com/3YOURMIND/django-migration-linter)
to keep database migrations
 backwards compatible

- we can automatically run migrations and they are zero-downtime, e.g.
old code can work with the migrated database
 - we can run and rollback migrations without worrying about data safety
- OnCall is deployed to the multiple environments core team is not able
to control

See [django-migration-linter
checklist](https://github.com/3YOURMIND/django-migration-linter/blob/main/docs/incompatibilities.md)
for the common mistakes and best practices


## Which issue(s) this PR fixes

## Checklist

- [ ] Tests updated
- [ ] Documentation added
- [ ] `CHANGELOG.md` updated

---------

Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
This commit is contained in:
Ildar Iskhakov 2023-02-06 16:01:37 +08:00 • committed by GitHub
parent 59ac97438a
commit 1b7ada4315
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
44 changed files with 128 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

29
.github/workflows/ci.yml vendored
View file

@ -23,6 +23,9 @@ jobs:
cd grafana-plugin/
yarn --network-timeout 500000
yarn build
# pre-commit uses git, which is not working in the action without this workaround
# see https://github.com/actions/runner-images/issues/6775
- run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
- name: Lint All
run: |
pre-commit run --all-files
@ -55,6 +58,32 @@ jobs:
run: |
docker run -v ${PWD}/docs/sources:/hugo/content/docs/oncall/latest -e HUGO_REFLINKSERRORLEVEL=ERROR --rm grafana/docs-base:latest /bin/bash -c 'make hugo'
lint-migrations-backend-mysql-rabbitmq:
name: "Lint migrations"
runs-on: ubuntu-latest
container: python:3.9
env:
DJANGO_SETTINGS_MODULE: settings.ci-test
SLACK_CLIENT_OAUTH_ID: 1
services:
rabbit_test:
image: rabbitmq:3.7.19
env:
RABBITMQ_DEFAULT_USER: rabbitmq
RABBITMQ_DEFAULT_PASS: rabbitmq
mysql_test:
image: mysql:5.7.25
env:
MYSQL_DATABASE: oncall_local_dev
MYSQL_ROOT_PASSWORD: local_dev_pwd
steps:
- uses: actions/checkout@v3
- name: Lint migrations
run: |
cd engine/
pip install -r requirements.txt
python manage.py lintmigrations
unit-test-backend-mysql-rabbitmq:
name: "Backend Tests: MySQL + RabbitMQ (RBAC enabled: ${{ matrix.rbac_enabled }})"
runs-on: ubuntu-latest

13
dev/README.md
View file

@ -20,6 +20,7 @@
- [symbol not found in flat namespace '\_EVP_DigestSignUpdate'](#symbol-not-found-in-flat-namespace-_evp_digestsignupdate)
- [IDE Specific Instructions](#ide-specific-instructions)
- [PyCharm](#pycharm)
- [How to write database migrations](#how-to-write-database-migrations)
Related: [How to develop integrations](/engine/config_integrations/README.md)
@ -397,3 +398,15 @@ make run-backend-celery
5. Create a new Django Server run configuration to Run/Debug the engine
- Use a plugin such as EnvFile to load the .env.dev file
- Change port from 8000 to 8080
## How to write database migrations
We use [django-migration-linter](https://github.com/3YOURMIND/django-migration-linter) to keep database migrations
backwards compatible
- we can automatically run migrations and they are zero-downtime, e.g. old code can work with the migrated database
- we can run and rollback migrations without worrying about data safety
- OnCall is deployed to the multiple environments core team is not able to control
See [django-migration-linter checklist](https://github.com/3YOURMIND/django-migration-linter/blob/main/docs/incompatibilities.md)
for the common mistakes and best practices

2
engine/apps/alerts/migrations/0001_squashed_initial.py
View file

@ -15,6 +15,7 @@ import django.core.validators
from django.db import migrations, models
import django.db.models.deletion
import django.db.models.manager
import django_migration_linter as linter
from apps.alerts.integration_options_mixin import IntegrationOptionsMixin
@ -27,6 +28,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='Alert',
fields=[

2
engine/apps/alerts/migrations/0002_squashed_initial.py
View file

@ -3,6 +3,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django.db.models.manager
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -18,6 +19,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='userhasnotification',
name='user',

2
engine/apps/alerts/migrations/0003_grafanaalertingcontactpoint_datasource_uid.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.13 on 2022-06-14 15:18
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='grafanaalertingcontactpoint',
name='datasource_uid',

2
engine/apps/alerts/migrations/0004_auto_20220711_1106.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.13 on 2022-07-11 11:06
from django.db import migrations
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -17,6 +18,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
# migrations.RemoveField(
# model_name='alertgroup',
# name='active_cache_for_web_calculation_id',

2
engine/apps/alerts/migrations/0005_alertgroup_cached_render_for_web.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.13 on 2022-07-20 09:04
from django.db import migrations, models, OperationalError, ProgrammingError
import django_migration_linter as linter
class AddFieldIfNotExists(migrations.AddField):
@ -36,6 +37,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
AddFieldIfNotExists(
model_name='alertgroup',
name='cached_render_for_web',

2
engine/apps/alerts/migrations/0006_alertgroup_alerts_aler_channel_ee84a7_idx.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.13 on 2022-07-27 10:51
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddIndex(
model_name='alertgroup',
index=models.Index(fields=['channel_id', 'resolved', 'acknowledged', 'silenced', 'root_alert_group_id', 'is_archived'], name='alerts_aler_channel_ee84a7_idx'),

2
engine/apps/alerts/migrations/0007_populate_web_title_cache.py
View file

@ -4,6 +4,7 @@ from django.db import migrations
from apps.alerts.models import AlertReceiveChannel
from apps.alerts.tasks import update_web_title_cache_for_alert_receive_channel
import django_migration_linter as linter
def populate_web_title_cache(apps, _):
@ -19,6 +20,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.RenameField(
model_name='alertgroup',
old_name='verbose_name',

2
engine/apps/alerts/migrations/0008_alter_alertgrouplogrecord_type.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.16 on 2023-01-19 18:06
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AlterField(
model_name='alertgrouplogrecord',
name='type',

2
engine/apps/auth_token/migrations/0001_squashed_initial.py
View file

@ -3,6 +3,7 @@ from django.utils import timezone
import apps.auth_token.models.slack_auth_token
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -13,6 +14,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='ApiAuthToken',
fields=[

2
engine/apps/auth_token/migrations/0002_squashed_initial.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -15,6 +16,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='userscheduleexportauthtoken',
name='organization',

2
engine/apps/auth_token/migrations/0003_auto_20221121_1610.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.16 on 2022-11-21 16:10
from django.db import migrations
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.RemoveField(
model_name='mobileappverificationtoken',
name='organization',

2
engine/apps/base/migrations/0001_squashed_initial.py
View file

@ -6,6 +6,7 @@ import datetime
import django.core.validators
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -17,6 +18,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='DynamicSetting',
fields=[

2
engine/apps/base/migrations/0002_squashed_initial.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -14,6 +15,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='usernotificationpolicylogrecord',
name='author',

2
engine/apps/base/migrations/0003_delete_organizationlogrecord.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.5 on 2022-08-23 12:03
from django.db import migrations
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.DeleteModel(
name='OrganizationLogRecord',
),

2
engine/apps/email/migrations/0001_initial.py
View file

@ -3,6 +3,7 @@
from django.db import migrations, models
import django.db.models.deletion
import uuid
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -16,6 +17,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='EmailMessage',
fields=[

2
engine/apps/heartbeat/migrations/0001_squashed_initial.py
View file

@ -4,6 +4,7 @@ import apps.heartbeat.models
import django.core.validators
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -15,6 +16,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='IntegrationHeartBeat',
fields=[

2
engine/apps/mobile_app/migrations/0001_initial.py
View file

@ -3,6 +3,7 @@
import apps.mobile_app.models
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -15,6 +16,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='MobileAppVerificationToken',
fields=[

2
engine/apps/oss_installation/migrations/0001_squashed_initial.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import uuid
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -12,6 +13,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='CloudHeartbeat',
fields=[

2
engine/apps/schedules/migrations/0001_squashed_initial.py
View file

@ -6,6 +6,7 @@ import django.core.validators
from django.db import migrations, models
import django.db.models.deletion
import uuid
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -16,6 +17,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='CustomOnCallShift',
fields=[

2
engine/apps/schedules/migrations/0002_squashed_initial.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -16,6 +17,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='oncallschedule',
name='organization',

2
engine/apps/schedules/migrations/0003_alter_customoncallshift_frequency.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.5 on 2022-06-28 13:06
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AlterField(
model_name='customoncallshift',
name='frequency',

2
engine/apps/schedules/migrations/0004_customoncallshift_until.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.5 on 2022-06-28 13:07
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='customoncallshift',
name='until',

2
engine/apps/schedules/migrations/0005_auto_20220704_1947.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -11,6 +12,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='OnCallScheduleWeb',
fields=[

2
engine/apps/schedules/migrations/0006_customoncallshift_rotation_start.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
from django.db.models import F
import django_migration_linter as linter
def fill_rotation_start_field(apps, schema_editor):
@ -16,6 +17,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='customoncallshift',
name='rotation_start',

2
engine/apps/schedules/migrations/0007_customoncallshift_updated_shift.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -11,6 +12,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='customoncallshift',
name='updated_shift',

2
engine/apps/schedules/migrations/0008_auto_20221201_0809.py
View file

@ -5,6 +5,7 @@ from django.db import migrations
from django.db.models import Q
from common.timezones import is_valid_timezone
import django_migration_linter as linter
def fix_bad_timezone_values(model):
@ -34,6 +35,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.RunPython(fix_bad_timezone_values('CustomOnCallShift')),
migrations.RunPython(fix_bad_timezone_values('OnCallScheduleCalendar')),
migrations.RunPython(fix_bad_timezone_values('OnCallScheduleWeb')),

2
engine/apps/slack/migrations/0001_squashed_initial.py
View file

@ -6,6 +6,7 @@ import django.core.validators
from django.db import migrations, models
import django.db.models.deletion
import uuid
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -17,6 +18,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='SlackActionRecord',
fields=[

2
engine/apps/slack/migrations/0002_squashed_initial.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -14,6 +15,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='slackmessage',
name='organization',

2
engine/apps/telegram/migrations/0001_squashed_initial.py
View file

@ -5,6 +5,7 @@ import django.core.validators
from django.db import migrations, models
import django.db.models.deletion
import uuid
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -17,6 +18,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='TelegramVerificationCode',
fields=[

2
engine/apps/telegram/migrations/0002_alter_telegrammessage_message_type.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.16 on 2022-12-29 14:42
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AlterField(
model_name='telegrammessage',
name='message_type',

2
engine/apps/twilioapp/migrations/0001_squashed_initial.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -15,6 +16,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='TwilioLogRecord',
fields=[

2
engine/apps/twilioapp/migrations/0002_auto_20220604_1008.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.5 on 2022-06-04 10:08
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='phonecall',
name='grafana_cloud_notification',

2
engine/apps/user_management/migrations/0001_squashed_initial.py
View file

@ -8,6 +8,7 @@ import django.core.validators
from django.db import migrations, models
import django.db.models.deletion
import mirage.fields
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -20,6 +21,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='Organization',
fields=[

2
engine/apps/user_management/migrations/0002_auto_20220705_1214.py
View file

@ -2,6 +2,7 @@
import apps.user_management.models.user
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -11,6 +12,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='user',
name='_timezone',

2
engine/apps/user_management/migrations/0003_user_hide_phone_number.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.5 on 2022-08-25 08:51
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='user',
name='hide_phone_number',

2
engine/apps/user_management/migrations/0004_auto_20221025_0316.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import django.db.models.deletion
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -11,6 +12,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.CreateModel(
name='Region',
fields=[

2
engine/apps/user_management/migrations/0005_rbac_permissions.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.15 on 2022-10-25 11:18
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='organization',
name='is_rbac_permissions_enabled',

2
engine/apps/user_management/migrations/0006_organization_uuid.py
View file

@ -2,6 +2,7 @@
from django.db import migrations, models
import uuid
import django_migration_linter as linter
def fill_org_uuid(apps, schema_editor):
@ -20,6 +21,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='organization',
name='uuid',

2
engine/apps/user_management/migrations/0007_organization_deleted_at.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.16 on 2023-01-04 05:06
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='organization',
name='deleted_at',

2
engine/apps/user_management/migrations/0008_organization_is_grafana_incident_enabled.py
View file

@ -1,6 +1,7 @@
# Generated by Django 3.2.16 on 2023-01-16 09:41
from django.db import migrations, models
import django_migration_linter as linter
class Migration(migrations.Migration):
@ -10,6 +11,7 @@ class Migration(migrations.Migration):
]
operations = [
linter.IgnoreMigration(),
migrations.AddField(
model_name='organization',
name='is_grafana_incident_enabled',

1
engine/requirements.txt
View file

@ -41,6 +41,7 @@ psycopg2==2.9.3
emoji==1.7.0
regex==2021.11.2
psutil==5.9.4
django-migration-linter==4.1.0
opentelemetry-instrumentation-celery==0.36b0
opentelemetry-instrumentation-pymysql==0.36b0
opentelemetry-instrumentation-wsgi==0.36b0

5
engine/settings/base.py
View file

@ -219,6 +219,7 @@ INSTALLED_APPS = [
"debug_toolbar",
"social_django",
"polymorphic",
"django_migration_linter",
"fcm_django",
"django_dbconn_retry",
]
@ -654,6 +655,10 @@ if OSS_INSTALLATION:
"args": (),
} # noqa
MIGRATION_LINTER_OPTIONS = {"exclude_apps": ["social_django", "silk", "fcm_django"]}
# Run migrations linter on each `python manage.py makemigrations`
MIGRATION_LINTER_OVERRIDE_MAKEMIGRATIONS = True
PYROSCOPE_PROFILER_ENABLED = getenv_boolean("PYROSCOPE_PROFILER_ENABLED", default=False)
if PYROSCOPE_PROFILER_ENABLED:
import pyroscope