oncall-engine/engine/apps/integrations
Joey Orlando 014a9c2ec2
allow the POST incoming alert endpoints to queue create_alert tasks independent of the database status (#1896)
# What this PR does

https://www.loom.com/share/18cc445117de4895a10892d56c7d3699

In preparation to upgrade our cloud databases, this PR makes some minor
changes which, after testing locally, allowed the `POST
/<integration_type>/<alert_channel_key>` endpoints to successfully
receive incoming alerts and queue the celery tasks.

I've tested all of the defined `POST
/integrations/v1/<integration_type>/<alert_channel_key>` endpoints by
sending `POST` requests to an integrations' URL while the MySQL database
was down, bringing the database back up, and ensuring the alerts were
created.

## Some other findings
- the integration heartbeat endpoints will not work as we interact w/
the database to persist the incoming heartbeat instance
- if the integration was created in the last 180 seconds, incoming
alerts will fail due to the way we cache the integration IDs
([code](https://github.com/grafana/oncall/blob/dev/engine/apps/integrations/mixins/alert_channel_defining_mixin.py#L47-L50))
- The `create_alert` celery task is set to `max_retries=None` and
`retry_backoff=True`. This means that the queued tasks will continue
retrying forever w/ an exponential backoff, until the alerts can be
created in the database (ie. when the database is back online).

## Checklist

- [ ] Unit, integration, and e2e (if applicable) tests updated (N/A)
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required) (N/A)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required) (N/A)
2023-05-10 12:36:23 +00:00
..
metadata Conform URLs (#281) 2022-07-25 09:12:50 -06:00
mixins allow the POST incoming alert endpoints to queue create_alert tasks independent of the database status (#1896) 2023-05-10 12:36:23 +00:00
templates Fix and tidy alertmanager heartbeat template (#1865) 2023-05-05 00:25:05 +00:00
tests Allow multiple database and celery broker types (#582) 2022-10-04 09:25:53 +01:00
__init__.py World, meet OnCall! 2022-06-03 08:09:47 -06:00
middlewares.py Add middleware to catch exception for missing integration, reduce spamminess of logs 2022-10-13 17:18:22 -06:00
tasks.py Limit number of alertmanager alerts in alert group to autoresolve (#1779) 2023-04-24 05:38:21 +00:00
urls.py Inbound email integration (#837) 2023-03-16 13:59:21 +08:00
views.py Inbound email integration (#837) 2023-03-16 13:59:21 +08:00