oncall-engine/engine/apps/grafana_plugin
Joey Orlando 014a9c2ec2
allow the POST incoming alert endpoints to queue create_alert tasks independent of the database status (#1896)
# What this PR does

https://www.loom.com/share/18cc445117de4895a10892d56c7d3699

In preparation to upgrade our cloud databases, this PR makes some minor
changes which, after testing locally, allowed the `POST
/<integration_type>/<alert_channel_key>` endpoints to successfully
receive incoming alerts and queue the celery tasks.

I've tested all of the defined `POST
/integrations/v1/<integration_type>/<alert_channel_key>` endpoints by
sending `POST` requests to an integrations' URL while the MySQL database
was down, bringing the database back up, and ensuring the alerts were
created.

## Some other findings
- the integration heartbeat endpoints will not work as we interact w/
the database to persist the incoming heartbeat instance
- if the integration was created in the last 180 seconds, incoming
alerts will fail due to the way we cache the integration IDs
([code](https://github.com/grafana/oncall/blob/dev/engine/apps/integrations/mixins/alert_channel_defining_mixin.py#L47-L50))
- The `create_alert` celery task is set to `max_retries=None` and
`retry_backoff=True`. This means that the queued tasks will continue
retrying forever w/ an exponential backoff, until the alerts can be
created in the database (ie. when the database is back online).

## Checklist

- [ ] Unit, integration, and e2e (if applicable) tests updated (N/A)
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required) (N/A)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required) (N/A)
2023-05-10 12:36:23 +00:00
..
helpers properly parse grafana cloud feature toggles (#1880) 2023-05-04 16:38:26 +00:00
tasks Do not retry start_sync_organization (#1856) 2023-05-02 02:33:26 +00:00
tests properly parse grafana cloud feature toggles (#1880) 2023-05-04 16:38:26 +00:00
views Store org cluster_slug (#1480) 2023-03-09 04:10:19 +00:00
__init__.py World, meet OnCall! 2022-06-03 08:09:47 -06:00
apps.py allow the POST incoming alert endpoints to queue create_alert tasks independent of the database status (#1896) 2023-05-10 12:36:23 +00:00
permissions.py Get rid of installation token + add a bunch of tests (#624) 2022-11-21 16:26:00 +01:00
urls.py Get rid of installation token + add a bunch of tests (#624) 2022-11-21 16:26:00 +01:00