# What this PR does
- removes unused "custom button" backend code now that we've migrated to
outgoing webhooks
- adds new e2e test for webhooks asserting that an `ngrok`/`express`
webhook handler receives the call as expected + payload is as expected
(related to https://github.com/grafana/oncall/issues/2691) - skipped for
now, the test passes locally but fails on GitHub Actions CI, seems to be
networking related
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
---------
Co-authored-by: Michael Derynck <michael.derynck@grafana.com>
# What this PR does
Adds internal API endpoints for
`alert_receive_channels/<id>/webhooks/<webhook_id>`.
## Which issue(s) this PR fixes
Related to https://github.com/grafana/oncall-private/issues/2541
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
This PR adds support for routing alerts based on labels.
https://www.loom.com/share/4401de6e3c4945d5b8961fe43ee373c9
Additionally:
- improve the typing around the `get_object` method that is inherited by
[`PublicPrimaryKeyMixin.get_object`](https://github.com/grafana/oncall/blob/dev/engine/common/api_helpers/mixins.py#L153)
in most of our models. `PublicPrimaryKeyMixin` is generic, so it can be
more strongly typed when it is being subclassed, which results in better
typing of the `get_object` method in child classes
- I decided to do this because I started looking into this task via the
[`AlertReceiveChannelView.send_demo_alert`
method/endpoint](https://github.com/grafana/oncall/blob/dev/engine/apps/api/views/alert_receive_channel.py#L242).
Within that method, `instance` is not typed because the inherited
`get_object` method is not typed.. I digress 😄
- improve typing around `Alert.create` and
`apps.integrations.tasks.create_alert` functions
- make `Alert.render_group_data` more DRY by extracting some logic out
into `Alert._apply_jinja_template_to_alert_payload_and_labels`
- deduplicate the logic of `value.strip().lower() in ["1", "true",
"ok"]` into a shared function,
`common.jinja_templater.apply_jinja_template.templated_value_is_truthy`
Closes https://github.com/grafana/oncall-private/issues/2490
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
- [x] Documentation added (or `pr:no public docs` PR label added if not
required) (will be done in #3762)
# What this PR does
This is a follow up to https://github.com/grafana/oncall/pull/3677.
It appears that when a task uses the [`autoretry_for`
kwarg](https://docs.celeryq.dev/en/stable/userguide/tasks.html#automatic-retry-for-known-exceptions)
in the task decorator, it doesn't log the exception in `on_failure` as
would be expected. Now when retrying, we log out a message + any
exception/stack trace information.
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
Closes https://github.com/grafana/oncall-private/issues/2449
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
Add an additional jinja2 template helper filter to convert a timezone
aware datetime to a different timezone.
## Which issue(s) this PR fixes
Alert payloads that originate from different time zones may include
timestamps having a local time offset. This filter enables
standardization of timestamp timezones.
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
---------
Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
# Which issue(s) this PR fixes
Related to https://github.com/grafana/oncall-private/issues/2363
Addresses this issue that arises when using
`cache.get_many`/`cache.set_many` operations with a Redis Cluster:
```python3
File "/usr/local/lib/python3.11/site-packages/redis/cluster.py", line 1006, in determine_slot
raise RedisClusterException(
redis.exceptions.RedisClusterException: MGET - all keys must map to the same key slot
```
From the Redis Cluster
[docs](https://redis.io/docs/reference/cluster-spec/#hash-tags), this
can be addressed with this 👇 . Basically this will ensure that keys in
multi-key operations will resolve to the same hash slot (read: node):
> Hash tags
> There is an exception for the computation of the hash slot that is
used in order to implement hash tags. Hash tags are a way to ensure that
multiple keys are allocated in the same hash slot. This is used in order
to implement multi-key operations in Redis Cluster.
>
> To implement hash tags, the hash slot for a key is computed in a
slightly different way in certain conditions. If the key contains a
"{...}" pattern only the substring between { and } is hashed in order to
obtain the hash slot. However since it is possible that there are
multiple occurrences of { or } the algorithm is well specified by the
following rules:
>
> IF the key contains a { character.
> AND IF there is a } character to the right of {.
> AND IF there are one or more characters between the first occurrence
of { and the first occurrence of }.
> Then instead of hashing the key, only what is between the first
occurrence of { and the following first occurrence of } is hashed.
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
Add an additional jinja2 template helper filter to decode base64-encoded
strings.
An example of an incoming integration payload that would benefit from
this filter is GCP's pubsub message:
https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
## Which issue(s) this PR fixes
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
---------
Co-authored-by: Jorge Vargas <jorge.vargas@homeprotech.com>
Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
Co-authored-by: Joey Orlando <joseph.t.orlando@gmail.com>
# What this PR does
- add `common.oncall_gateway.tasks.create_slack_connector_async_v2`
celery task to `CELERY_TASK_ROUTES`
- remove two deprecated tasks in `common.oncall_gateway.tasks`
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
These celery tasks have not been used for more than one week (since
[v1.3.25](https://github.com/grafana/oncall/releases/tag/v1.3.25) which
released an improvement for Grafana Alerting integration)
## Checklist
- [ ] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
This task was renamed in a v1.3.30. It is no longer referenced and there
are no pending tasks in production queues:
## Checklist
- [ ] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
- Add a test to ensure celery tasks are assigned to a queue
- Move CELERY_TASK_ROUTES out of settings into its own file for easier
reuse and reference.
## Which issue(s) this PR fixes
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
This PR should allow us to start working on _most_ of the remaining
tasks for this feature set.
- Adds a basic `ShiftSwapRequest` model + CRUD endpoints.
- Adds a `POST /api/internal/v1/shift_swaps/<id>/take` endpoint which
allows a benefactor to take a request (only when certain conditions
about the ssr are met)
Closes#2587
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required) will be done in #2589
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required) (will update once we ship the finalized feature set)
# What this PR does
Fixes an issue when alert group list page fails to load if any alert
group on the page has `- - ` substring in its message.
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
- add user locale field to mobile app user settings table + add a test
that sends `PATCH` requests to this endpoint
- change "you're going on call" push notification text to include
localized shift time. The general format is now:
```python
f"You're going on call in {time_until_going_oncall} for schedule
{schedule.name}, {formatted_shift}"
```
- `time_until_going_oncall` is a "human-readable" format of the time
until the start)
- `schedule.name` is self-explanatory
- `formatted_shift` this depends on the shift. If the shift starts and
ends on the same day, the format will be "HH:mm - HH:mm". Otherwise, if
the shift starts and ends on different days, the format will be
"YYYY-MM-DD HH:mm - YYYY-MM-DD HH:mm". **Note** that all datetime
related formatting will use the new `locale` field that we are now
storing in the mobile app user settings table. If no locale is yet
present we will fallback to "en"
## Which issue(s) this PR fixes
closes https://github.com/grafana/oncall/issues/2024https://github.com/grafana/oncall-mobile-app/issues/187
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
Makes sure that all viewset actions with `detail=True` use
`self.get_object()` to retrieve an instance that's being acted upon.
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
- update `make test` to always use `settings.ci-test`. Right now it will
use whatever the value of `DJANGO_SETTINGS_MODULE` is in
`./dev/.env.dev`, which causes ~45 tests to fail
- Fix several Python warnings that we see when running the tests
```bash
RemovedInDjango40Warning: The providing_args argument is deprecated. As it is purely documentational, it has no replacement. If you rely on this argument as documentation, you can move the text to a code comment or docstring.
alert_create_signal = django.dispatch.Signal(
```
```bash
PytestCollectionWarning: cannot collect test class 'TestOnlyBackend' because it has a __init__ constructor (from: apps/api/tests/test_alert_receive_channel_template.py)
class TestOnlyBackend(BaseMessagingBackend):
```
```bash
DeprecationWarning: The parameter 'use_aliases' in emoji.emojize() is deprecated and will be removed in version 2.0.0. Use language='alias' instead.
To hide this warning, pin/downgrade the package to 'emoji~=1.6.3'
return emoji.emojize(self.verbal_name, use_aliases=True)
```
```bash
DateTimeField CustomOnCallShift.start received a naive datetime (2023-06-01 12:53:12) while time zone support is active.
warnings.warn("DateTimeField %s received a naive datetime (%s)"
```
```bash
apps/twilioapp/tests/test_phone_calls.py::test_resolve_by_phone
/etc/app/apps/twilioapp/tests/test_phone_calls.py:173: DeprecationWarning: The 'text' argument to find()-type methods is deprecated. Use 'string' instead.
content = BeautifulSoup(content, features="html.parser").findAll(text=True)
```
```bash
apps/twilioapp/tests/test_phone_calls.py::test_resolve_by_phone
apps/twilioapp/tests/test_phone_calls.py::test_wrong_pressed_digit
/usr/local/lib/python3.11/site-packages/bs4/builder/__init__.py:545: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
```
```bash
apps/twilioapp/tests/test_phone_calls.py::test_forbidden_requests
/usr/local/lib/python3.11/site-packages/social_django/urls.py:15: RemovedInDjango40Warning: django.conf.urls.url() is deprecated in favor of django.urls.re_path().
url(r'^login/(?P<backend>[^/]+){0}$'.format(extra), views.auth,
```
```bash
apps/twilioapp/tests/test_phone_calls.py: 66 warnings
/usr/local/lib/python3.11/site-packages/debug_toolbar/utils.py:255: DeprecationWarning: currentThread() is deprecated, use current_thread() instead
thread = threading.currentThread()
```
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
This PR adds new API endpoints for migrating routes from regex format to
jinja2 format. The changes include the following:
* `filtering_term_as_jinja2` field to GET `channels_filters` endpoint
* A POST endpoint
`channel_filters/ABCDEF123/convert_from_regex_to_jinja2/` for migrating
routes to jinja2 format.
These new endpoints will allow users to preview and migrate their
existing regex routes to the more flexible and maintainable jinja2
format.
Check the screenshot where this endpoints will be used
<img width="407" alt="Screenshot 2023-04-14 at 09 50 23"
src="https://user-images.githubusercontent.com/2262529/231920771-20792c7e-d6ef-4681-80e1-c82bb4aa4b8e.png">
## Which issue(s) this PR fixes
## Checklist
- [ ] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
This PR, overrides Django URLValidator with a CustomURLValidator. It
just removes tld_re part from the regex, and the other behaviour remains
the same.
The CustomURLValidator is defined in common.api_helpers.utils.py file
and is utilized in custom_button.py.
Please inform me if it needs to be defined somewhere else or be
implemented with some other methods.
## Which issue(s) this PR fixes
Currently, URLValidator raises exception for URLs that don't have TLD.
This leads to not being able to use containers URL for outgoing webhooks
as they usually don't have TLD.
## Checklist
- [x] Tests updated
- [ ] Documentation added
- [x] `CHANGELOG.md` updated
---------
Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
Co-authored-by: Innokentii Konstantinov <innokenty.konstantinov@grafana.com>
Co-authored-by: Joey Orlando <joseph.t.orlando@gmail.com>
# What this PR does
Adds a new `regex_match` filter to Jinja environment.
## Which issue(s) this PR fixes
This should be useful on its own, and also helpful for
https://github.com/grafana/oncall/pull/1555.
## Checklist
- [x] Tests updated
- [x] Documentation added
- [x] `CHANGELOG.md` updated
# What this PR does
- Implement recapthca v3 check. DRF_RECAPTCHA didn't support hostname
validation and it's too complicated to add it.
- Add validation of verification code on oncall side to not to call
twilio with obviously invalid codes
## Checklist
- [x] Tests updated
- [ ] Documentation added
- [ ] `CHANGELOG.md` updated
# What this PR does
This is a minor refactor before implementing
https://github.com/grafana/oncall-private/issues/1558.
Additionally, it cleans up a few spots where we do this:
```
# Re-take in case we are in the readonly db context.
```
We currently don't read anything from a read-only database, so this
should be not necessary.
## Checklist
- [x] Tests updated
- [ ] Documentation added (N/A)
- [ ] `CHANGELOG.md` updated (N/A)
* Centralize timezone validation into one spot + add serializer validation
for schedules and oncall shifts (both public and internal API)
* add engine-manage make command
* Improve feedback so template errors are given to user
* Add security error logging
* Add limits for templates, payloads, results
* Show popup error notification for webhook errors and template errors that don't have a result
* Update tests
* Split exceptions into warnings/errors to give more control when previewing, rendering, saving templates
* Limit title lengths
* Make TypeError a warning
* Adjust title length limit
* Remove length limiting on urlize since it is being done on template render
* Fix tests
* Add KeyError and ValueError to warnings
* No longer enforcing json result when saving webhook in case it is dependent on payload
* Add tests for expected exceptions coming from apply_jinja_template
* Update changelog
* Send raw post if template result is not JSON
