014a9c2ec2
# What this PR does https://www.loom.com/share/18cc445117de4895a10892d56c7d3699 In preparation to upgrade our cloud databases, this PR makes some minor changes which, after testing locally, allowed the `POST /<integration_type>/<alert_channel_key>` endpoints to successfully receive incoming alerts and queue the celery tasks. I've tested all of the defined `POST /integrations/v1/<integration_type>/<alert_channel_key>` endpoints by sending `POST` requests to an integrations' URL while the MySQL database was down, bringing the database back up, and ensuring the alerts were created. ## Some other findings - the integration heartbeat endpoints will not work as we interact w/ the database to persist the incoming heartbeat instance - if the integration was created in the last 180 seconds, incoming alerts will fail due to the way we cache the integration IDs ([code](https://github.com/grafana/oncall/blob/dev/engine/apps/integrations/mixins/alert_channel_defining_mixin.py#L47-L50)) - The `create_alert` celery task is set to `max_retries=None` and `retry_backoff=True`. This means that the queued tasks will continue retrying forever w/ an exponential backoff, until the alerts can be created in the database (ie. when the database is back online). ## Checklist - [ ] Unit, integration, and e2e (if applicable) tests updated (N/A) - [ ] Documentation added (or `pr:no public docs` PR label added if not required) (N/A) - [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not required) (N/A)
41 lines
1.8 KiB
Python
41 lines
1.8 KiB
Python
import logging
|
|
import sys
|
|
|
|
from django.apps import AppConfig, apps
|
|
from django.conf import settings
|
|
from django.db import OperationalError
|
|
|
|
logger = logging.getLogger(__name__)
|
|
logger.setLevel(logging.DEBUG)
|
|
|
|
STARTUP_COMMANDS = ["runserver", "uwsgi"]
|
|
|
|
|
|
class GrafanaPluginConfig(AppConfig):
|
|
name = "apps.grafana_plugin"
|
|
|
|
def ready(self):
|
|
"""
|
|
For OSS installations, validate that GRAFANA_API_URL environment variable is specified, otherwise
|
|
abort app startup.
|
|
|
|
We only care to run this for OSS INSTALLATIONS. The STARTUP_COMMANDS check is to avoid running this check
|
|
for the django migrate command. For a fresh installation this would crash because user_management table would
|
|
[not exist](https://stackoverflow.com/a/63326719).
|
|
"""
|
|
# TODO: this logic should probably be moved out to a common utility
|
|
is_not_migration_script = any(startup_command in sys.argv for startup_command in STARTUP_COMMANDS)
|
|
if is_not_migration_script and settings.IS_OPEN_SOURCE:
|
|
try:
|
|
Organization = apps.get_model("user_management", "Organization")
|
|
has_existing_org = Organization.objects.first() is not None
|
|
|
|
# only enforce the following for new setups - if no organization exists in the database
|
|
# and the GRAFANA_API_URL env var is not specified, exit the application
|
|
if has_existing_org is False and settings.SELF_HOSTED_SETTINGS["GRAFANA_API_URL"] is None:
|
|
logger.error(
|
|
f"For OSS installations, GRAFANA_API_URL is a required environment variable. Please set it and restart the application."
|
|
)
|
|
sys.exit()
|
|
except OperationalError:
|
|
pass
|