Handle SlackAPIRatelimitError in perform_notification (#4486)

# What this PR does

Fixes retrying perform_notification task when Slack API returns 429
(Ratelimited)
See
[thread](https://raintank-corp.slack.com/archives/C025VMT6SPK/p1717725432075029)
for more details

## Which issue(s) this PR closes

Closes [issue link here]

<!--
*Note*: if you have more than one GitHub issue that this PR closes, be
sure to preface
each issue link with a [closing
keyword](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue).
This ensures that the issue(s) are auto-closed once the PR has been
merged.
-->

## Checklist

- [ ] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.

engine/apps/slack/models/slack_message.py

@@ -11,6 +11,7 @@ from apps.slack.errors import (
     SlackAPIError,
     SlackAPIFetchMembersFailedError,
     SlackAPIMethodNotSupportedForChannelTypeError,
+    SlackAPIRatelimitError,
     SlackAPITokenError,
 )
 
@@ -146,6 +147,18 @@ class SlackMessage(models.Model):
                 thread_ts=slack_message.slack_id,
                 unfurl_links=True,
             )
+        except SlackAPIRatelimitError:
+            UserNotificationPolicyLogRecord(
+                author=user,
+                type=UserNotificationPolicyLogRecord.TYPE_PERSONAL_NOTIFICATION_FAILED,
+                notification_policy=notification_policy,
+                alert_group=alert_group,
+                reason="Slack API rate limit error",
+                notification_step=notification_policy.step,
+                notification_channel=notification_policy.notify_by,
+                notification_error_code=UserNotificationPolicyLogRecord.ERROR_NOTIFICATION_IN_SLACK_RATELIMIT,
+            ).save()
+            return
         except SlackAPITokenError:
             UserNotificationPolicyLogRecord(
                 author=user,