# Which issue(s) this PR fixes
I was seeing some strange logs occasionally pop up from celery:
```bash
[1;33m2023-07-13 10:05:14,772 source=engine:celery worker=MainProcess task_id=??? task_name=??? name=celery.redirected level=WARNING AttributeError: '_Code' object has no attribute 'co_positions'[0m
```
Did a bit of digging on this and
[realized](https://github.com/DataDog/dd-trace-py/issues/4149) that
`celery==5.2.7` does not support Python 3.11 (the version we use):
We should immediately upgrade to `celery==5.3.1`. It mentions in this in
the changelog for that version:
## Checklist
- [ ] Unit, integration, and e2e (if applicable) tests updated (N/A)
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required) (N/A)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required) (N/A)
# What this PR does
See more details comments alongside the code.
Regarding frontend changes, the main changes in this PR are to remove
unused fields on the `Team` interface + unused methods on the `Team`
model.
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required) (N/A)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required) (N/A)
Bumps [django](https://github.com/django/django) from 3.2.19 to 3.2.20.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="19bc11f636"><code>19bc11f</code></a>
[3.2.x] Bumped version for 3.2.20 release.</li>
<li><a
href="454f2fb934"><code>454f2fb</code></a>
[3.2.x] Fixed CVE-2023-36053 -- Prevented potential ReDoS in
EmailValidator a...</li>
<li><a
href="07cc014cb3"><code>07cc014</code></a>
[3.2.x] Added stub release notes for 3.2.20.</li>
<li><a
href="e1bbbbe6ac"><code>e1bbbbe</code></a>
[3.2.x] Fixed MultipleFileFieldTest.test_file_multiple_validation() test
if P...</li>
<li><a
href="47ef12e69c"><code>47ef12e</code></a>
[3.2.x] Added CVE-2023-31047 to security archive.</li>
<li><a
href="15f90ebff3"><code>15f90eb</code></a>
[3.2.x] Post-release version bump.</li>
<li>See full diff in <a
href="https://github.com/django/django/compare/3.2.19...3.2.20">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/grafana/oncall/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# What this PR does
- add user locale field to mobile app user settings table + add a test
that sends `PATCH` requests to this endpoint
- change "you're going on call" push notification text to include
localized shift time. The general format is now:
```python
f"You're going on call in {time_until_going_oncall} for schedule
{schedule.name}, {formatted_shift}"
```
- `time_until_going_oncall` is a "human-readable" format of the time
until the start)
- `schedule.name` is self-explanatory
- `formatted_shift` this depends on the shift. If the shift starts and
ends on the same day, the format will be "HH:mm - HH:mm". Otherwise, if
the shift starts and ends on different days, the format will be
"YYYY-MM-DD HH:mm - YYYY-MM-DD HH:mm". **Note** that all datetime
related formatting will use the new `locale` field that we are now
storing in the mobile app user settings table. If no locale is yet
present we will fallback to "en"
## Which issue(s) this PR fixes
closes https://github.com/grafana/oncall/issues/2024https://github.com/grafana/oncall-mobile-app/issues/187
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
- Adds [`mypy` static type checking](https://mypy-lang.org/) to our CI
pipeline. Currently there is still a **ton** of errors being returned by
the tool, as we'll need to fix pre-existing errors. I think we can
slowly chip away at these errors in small PRs, doing them all in one
large PR is likely very risky.
- Also, this PR starts chipping away at one of the main type errors that
we have which is accessing the `datetime` class (from the `datetime`
library) or `timedelta` function on the `django.utils.timezone` module.
Basically we should be instead accessing these two objects from the
native `datetime` module. This makes sense because the [`__all__`
attribute](https://github.com/django/django/blob/main/django/utils/timezone.py#L14-L30)
in `django.utils.timezone` does not re-export `datetime` or `timedelta`.
- splits `engine` dependencies out into `requirements.txt` and
`requirements-dev.txt`
## Checklist
- [ ] Unit, integration, and e2e (if applicable) tests updated (N/A)
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required) (N/A)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required) (N/A)
# What this PR does
- update `make test` to always use `settings.ci-test`. Right now it will
use whatever the value of `DJANGO_SETTINGS_MODULE` is in
`./dev/.env.dev`, which causes ~45 tests to fail
- Fix several Python warnings that we see when running the tests
```bash
RemovedInDjango40Warning: The providing_args argument is deprecated. As it is purely documentational, it has no replacement. If you rely on this argument as documentation, you can move the text to a code comment or docstring.
alert_create_signal = django.dispatch.Signal(
```
```bash
PytestCollectionWarning: cannot collect test class 'TestOnlyBackend' because it has a __init__ constructor (from: apps/api/tests/test_alert_receive_channel_template.py)
class TestOnlyBackend(BaseMessagingBackend):
```
```bash
DeprecationWarning: The parameter 'use_aliases' in emoji.emojize() is deprecated and will be removed in version 2.0.0. Use language='alias' instead.
To hide this warning, pin/downgrade the package to 'emoji~=1.6.3'
return emoji.emojize(self.verbal_name, use_aliases=True)
```
```bash
DateTimeField CustomOnCallShift.start received a naive datetime (2023-06-01 12:53:12) while time zone support is active.
warnings.warn("DateTimeField %s received a naive datetime (%s)"
```
```bash
apps/twilioapp/tests/test_phone_calls.py::test_resolve_by_phone
/etc/app/apps/twilioapp/tests/test_phone_calls.py:173: DeprecationWarning: The 'text' argument to find()-type methods is deprecated. Use 'string' instead.
content = BeautifulSoup(content, features="html.parser").findAll(text=True)
```
```bash
apps/twilioapp/tests/test_phone_calls.py::test_resolve_by_phone
apps/twilioapp/tests/test_phone_calls.py::test_wrong_pressed_digit
/usr/local/lib/python3.11/site-packages/bs4/builder/__init__.py:545: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
```
```bash
apps/twilioapp/tests/test_phone_calls.py::test_forbidden_requests
/usr/local/lib/python3.11/site-packages/social_django/urls.py:15: RemovedInDjango40Warning: django.conf.urls.url() is deprecated in favor of django.urls.re_path().
url(r'^login/(?P<backend>[^/]+){0}$'.format(extra), views.auth,
```
```bash
apps/twilioapp/tests/test_phone_calls.py: 66 warnings
/usr/local/lib/python3.11/site-packages/debug_toolbar/utils.py:255: DeprecationWarning: currentThread() is deprecated, use current_thread() instead
thread = threading.currentThread()
```
## Checklist
- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
Bumps [requests](https://github.com/psf/requests) from 2.29.0 to 2.31.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/psf/requests/releases">requests's
releases</a>.</em></p>
<blockquote>
<h2>v2.31.0</h2>
<h2>2.31.0 (2023-05-22)</h2>
<p><strong>Security</strong></p>
<ul>
<li>
<p>Versions of Requests between v2.3.0 and v2.30.0 are vulnerable to
potential
forwarding of <code>Proxy-Authorization</code> headers to destination
servers when
following HTTPS redirects.</p>
<p>When proxies are defined with user info (<a
href="https://user:pass@proxy:8080">https://user:pass@proxy:8080</a>),
Requests
will construct a <code>Proxy-Authorization</code> header that is
attached to the request to
authenticate with the proxy.</p>
<p>In cases where Requests receives a redirect response, it previously
reattached
the <code>Proxy-Authorization</code> header incorrectly, resulting in
the value being
sent through the tunneled connection to the destination server. Users
who rely on
defining their proxy credentials in the URL are <em>strongly</em>
encouraged to upgrade
to Requests 2.31.0+ to prevent unintentional leakage and rotate their
proxy
credentials once the change has been fully deployed.</p>
<p>Users who do not use a proxy or do not supply their proxy credentials
through
the user information portion of their proxy URL are not subject to this
vulnerability.</p>
<p>Full details can be read in our <a
href="https://github.com/psf/requests/security/advisories/GHSA-j8r2-6x86-q33q">Github
Security Advisory</a>
and <a
href="https://nvd.nist.gov/vuln/detail/CVE-2023-32681">CVE-2023-32681</a>.</p>
</li>
</ul>
<h2>v2.30.0</h2>
<h2>2.30.0 (2023-05-03)</h2>
<p><strong>Dependencies</strong></p>
<ul>
<li>
<p>⚠️ Added support for urllib3 2.0. ⚠️</p>
<p>This may contain minor breaking changes so we advise careful testing
and
reviewing <a
href="https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html">https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html</a>
prior to upgrading.</p>
<p>Users who wish to stay on urllib3 1.x can pin to
<code>urllib3<2</code>.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/psf/requests/blob/main/HISTORY.md">requests's
changelog</a>.</em></p>
<blockquote>
<h2>2.31.0 (2023-05-22)</h2>
<p><strong>Security</strong></p>
<ul>
<li>
<p>Versions of Requests between v2.3.0 and v2.30.0 are vulnerable to
potential
forwarding of <code>Proxy-Authorization</code> headers to destination
servers when
following HTTPS redirects.</p>
<p>When proxies are defined with user info (<a
href="https://user:pass@proxy:8080">https://user:pass@proxy:8080</a>),
Requests
will construct a <code>Proxy-Authorization</code> header that is
attached to the request to
authenticate with the proxy.</p>
<p>In cases where Requests receives a redirect response, it previously
reattached
the <code>Proxy-Authorization</code> header incorrectly, resulting in
the value being
sent through the tunneled connection to the destination server. Users
who rely on
defining their proxy credentials in the URL are <em>strongly</em>
encouraged to upgrade
to Requests 2.31.0+ to prevent unintentional leakage and rotate their
proxy
credentials once the change has been fully deployed.</p>
<p>Users who do not use a proxy or do not supply their proxy credentials
through
the user information portion of their proxy URL are not subject to this
vulnerability.</p>
<p>Full details can be read in our <a
href="https://github.com/psf/requests/security/advisories/GHSA-j8r2-6x86-q33q">Github
Security Advisory</a>
and <a
href="https://nvd.nist.gov/vuln/detail/CVE-2023-32681">CVE-2023-32681</a>.</p>
</li>
</ul>
<h2>2.30.0 (2023-05-03)</h2>
<p><strong>Dependencies</strong></p>
<ul>
<li>
<p>⚠️ Added support for urllib3 2.0. ⚠️</p>
<p>This may contain minor breaking changes so we advise careful testing
and
reviewing <a
href="https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html">https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html</a>
prior to upgrading.</p>
<p>Users who wish to stay on urllib3 1.x can pin to
<code>urllib3<2</code>.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="147c8511dd"><code>147c851</code></a>
v2.31.0</li>
<li><a
href="74ea7cf7a6"><code>74ea7cf</code></a>
Merge pull request from GHSA-j8r2-6x86-q33q</li>
<li><a
href="3022253346"><code>3022253</code></a>
test on pypy 3.8 and pypy 3.9 on windows and macos (<a
href="https://redirect.github.com/psf/requests/issues/6424">#6424</a>)</li>
<li><a
href="b639e66c81"><code>b639e66</code></a>
test on py3.12 (<a
href="https://redirect.github.com/psf/requests/issues/6448">#6448</a>)</li>
<li><a
href="d3d504436e"><code>d3d5044</code></a>
Fixed a small typo (<a
href="https://redirect.github.com/psf/requests/issues/6452">#6452</a>)</li>
<li><a
href="2ad18e0e10"><code>2ad18e0</code></a>
v2.30.0</li>
<li><a
href="f2629e9e3c"><code>f2629e9</code></a>
Remove strict parameter (<a
href="https://redirect.github.com/psf/requests/issues/6434">#6434</a>)</li>
<li>See full diff in <a
href="https://github.com/psf/requests/compare/v2.29.0...v2.31.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/grafana/oncall/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
Bumps
[pymdown-extensions](https://github.com/facelessuser/pymdown-extensions)
from 9.11 to 10.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/facelessuser/pymdown-extensions/releases">pymdown-extensions's
releases</a>.</em></p>
<blockquote>
<h2>10.0</h2>
<ul>
<li><strong>Break</strong>: Snippets: snippets will restrict snippets to
ensure they are under the <code>base_path</code> preventing snippets
relative to the <code>base_path</code> but not explicitly under it.
<code>restrict_base_path</code> can be set to <code>False</code> for
legacy
behavior.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="5e750734b7"><code>5e75073</code></a>
Update JS doc deps</li>
<li><a
href="b7bb4878d6"><code>b7bb487</code></a>
Merge pull request from GHSA-jh85-wwv9-24hv</li>
<li>See full diff in <a
href="https://github.com/facelessuser/pymdown-extensions/compare/9.11...10.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/grafana/oncall/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
Bumps [django](https://github.com/django/django) from 3.2.18 to 3.2.19.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="fc42edd2e6"><code>fc42edd</code></a>
[3.2.x] Bumped version for 3.2.19 release.</li>
<li><a
href="eed53d0011"><code>eed53d0</code></a>
[3.2.x] Fixed CVE-2023-31047, Fixed <a
href="https://redirect.github.com/django/django/issues/31710">#31710</a>
-- Prevented potential bypass of v...</li>
<li><a
href="007e46d815"><code>007e46d</code></a>
[3.2.x] Added missing backticks in docs/releases/1.7.txt.</li>
<li><a
href="a37e4d5d6e"><code>a37e4d5</code></a>
[3.2.x] Added stub release notes for 3.2.19.</li>
<li><a
href="963f24cff2"><code>963f24c</code></a>
[3.2.x] Added CVE-2023-24580 to security archive.</li>
<li><a
href="e34a2283f2"><code>e34a228</code></a>
[3.2.x] Post-release version bump.</li>
<li>See full diff in <a
href="https://github.com/django/django/compare/3.2.18...3.2.19">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/grafana/oncall/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# What this PR does
Upgrades the backend to Python 3.11.3 (latest stable release) + update
linting step on Drone builds to run **all** the linting steps, not just
the Python ones.
## Checklist
- [ ] Unit, integration, and e2e (if applicable) tests updated (N/A)
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required) (N/A)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
## Which issue(s) this PR fixes
## Checklist
- [ ] Unit, integration, and e2e (if applicable) tests updated
- [ ] Documentation added (or `pr:no public docs` PR label added if not
required)
- [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
# What this PR does
This PR:
- modifies the `check_escalation_finished_task` celery task to:
- do stricter escalation validation based on the alert group's
escalation snapshot (see the `audit_alert_group_escalation` method in
`engine/apps/alerts/tasks/check_escalation_finished.py` for the
validation logic)
- use a read-only database for querying alert-groups if one is
configured, otherwise use the "default" one
- ping a configurable heartbeat (new env var
`ALERT_GROUP_ESCALATION_AUDITOR_CELERY_TASK_HEARTBEAT_URL` added)
- increase the task frequency from every 10 to every 13 minutes (this
can be configured via an env variable)
- adds public documentation on how to configure this auditor task
- modifies the local celery startup command to properly take into
consideration all celery related env vars (similar to the ones we use in
`engine/celery_with_exporter.sh`; this made it easier to enable `celery
beat` locally for testing)
- removes the following code:
- removes references to `AlertGroup.estimate_escalation_finish_time` and
marks the model field as deprecated using the [`django-deprecate-fields`
library](https://pypi.org/project/django-deprecate-fields/). This field
was only used for the previous version of this validation task
- `EscalationSnapshotMixin.calculate_eta_for_finish_escalation` was only
used to calculate the value for
`AlertGroup.estimate_escalation_finish_time`
- `calculate_escalation_finish_time` celery task
## Which issue(s) this PR fixes
https://github.com/grafana/oncall-private/issues/1558
## Checklist
- [x] Tests updated
- [x] Documentation added
- [x] `CHANGELOG.md` updated
This PR add Inbound Email integration.
It designed to support some variety of ESPs, but in prod we will use
Mailgun, so locally I tested it only with mailgun ESP.
**Important:**
To make it work on different clusters I'm planning to provide different
email domains for different regions, like ....@us.oncall.grafana.net,
...@eu.oncall.grafana.net
---------
Co-authored-by: Innokentii Konstantinov <innokenty.konstantinov@grafana.com>
# What this PR does
- Implement recapthca v3 check. DRF_RECAPTCHA didn't support hostname
validation and it's too complicated to add it.
- Add validation of verification code on oncall side to not to call
twilio with obviously invalid codes
## Checklist
- [x] Tests updated
- [ ] Documentation added
- [ ] `CHANGELOG.md` updated
# What this PR does
Adds reCAPTCHA validation to the get mobile verification code endpoint
## Which issue(s) this PR fixes
## Checklist
- [x] Tests updated
- [ ] Documentation added (N/A)
- [x] `CHANGELOG.md` updated
---------
Co-authored-by: Maxim <maxim.mordasov@grafana.com>
# What this PR does
This PR adds
[django-migration-linter](https://github.com/3YOURMIND/django-migration-linter)
to keep database migrations
backwards compatible
- we can automatically run migrations and they are zero-downtime, e.g.
old code can work with the migrated database
- we can run and rollback migrations without worrying about data safety
- OnCall is deployed to the multiple environments core team is not able
to control
See [django-migration-linter
checklist](https://github.com/3YOURMIND/django-migration-linter/blob/main/docs/incompatibilities.md)
for the common mistakes and best practices
## Which issue(s) this PR fixes
## Checklist
- [ ] Tests updated
- [ ] Documentation added
- [ ] `CHANGELOG.md` updated
---------
Co-authored-by: Joey Orlando <joey.orlando@grafana.com>
Bumps [django](https://github.com/django/django) from 3.2.16 to 3.2.17.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="aed1bb56d1"><code>aed1bb5</code></a>
[3.2.x] Bumped version for 3.2.17 release.</li>
<li><a
href="c7e0151fdf"><code>c7e0151</code></a>
[3.2.x] Fixed CVE-2023-23969 -- Prevented DoS with pathological values
for Ac...</li>
<li><a
href="9da46345d8"><code>9da4634</code></a>
[3.2.x] Fixed inspectdb.tests.InspectDBTestCase.test_custom_fields() on
SQLit...</li>
<li><a
href="4c2b26174f"><code>4c2b261</code></a>
[3.2.x] Removed 'tests' path prefix in a couple tests.</li>
<li><a
href="d21543182d"><code>d215431</code></a>
[3.2.x] Adjusted release notes for 3.2.17.</li>
<li><a
href="4e31d3ea55"><code>4e31d3e</code></a>
[3.2.x] Added stub release notes for 3.2.17.</li>
<li><a
href="238e8898ac"><code>238e889</code></a>
[3.2.x] Corrected passenv value for tox 4.0.6+.</li>
<li><a
href="b381ab4906"><code>b381ab4</code></a>
[3.2.x] Disabled auto-created table of contents entries on Sphinx
5.2+.</li>
<li><a
href="f6f0699d01"><code>f6f0699</code></a>
[3.2.x] Removed obsolete doc reference to
asyncio.iscoroutinefunction.</li>
<li><a
href="accdd0576d"><code>accdd05</code></a>
[3.2.x] Added CVE-2022-36359 to security archive.</li>
<li>Additional commits viewable in <a
href="https://github.com/django/django/compare/3.2.16...3.2.17">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/grafana/oncall/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# What this PR does
## Main stuff
- add Python script to populate local Grafana/OnCall setup w/ large
amounts of fake data. Right now the data types that can be generated
are:
- teams and Admin users via the Grafana API (must be synced manually by
going into the UI before going onto the next step)
- Calendar Schedules which have three 8h oncall-shifts, via the OnCall
public API
- fixes `django-debug-toolbar` when being run in `docker-compose`
locally
## Other stuff
- documents how to easily modify the Grafana `docker-compose` container
provisioning configuration
- document solutions for two backend setup related issues encountered
when running the engine/celery workers locally, outside of
`docker-compose`, on an Apple silicon Mac
- fixes small bug in `grafana_plugin.helpers.client.APIClient.call_api`
where it would call `response.json()` for all requests, regardless of
whether or not the response actually contained data or not
- in `engine/settings/dev.py`, properly setup `django-silk` and document
the steps to use it locally
- make it possible to log out debug SQL queries by specifying
`DEV_DEBUG_VIEW_SQL_QUERIES` env var, rather than having to uncomment
out a section of `settings/dev.py`
## Which issue(s) this PR fixes
- Some local setup issues when trying to use `django-silk` and
`django-debug-toolbar`
- Makes it much easier to populate your local setup with a lot of fake
data
- Makes it possible to easily modify your local grafana's provisioning
configuration
## Checklist
- [ ] Tests updated (N/A)
- [ ] Documentation added (N/A)
- [ ] `CHANGELOG.md` updated (N/A)
Fixes issue when running OnCall locally, on an M1 Mac, and using
PostgreSQL as the database. (ie. `COMPOSE_PROFILES=postgres...`).
Currently getting:
```bash
django.db.utils.OperationalError: SCRAM authentication requires libpq version 10 or above
```
I also tried simply adding `libpq-dev` to the `Dockerfile` but this
change alone does not solve the issue. See
[here](https://github.com/MobSF/Mobile-Security-Framework-MobSF/issues/1898)
for a similar reported issue on GitHub.
**Root Cause**
This issue is caused because `psycopg2-binary` 2.9.3 [doesn't
provide](https://pypi.org/project/psycopg2-binary/2.9.3/#files) binary
wheels for MacOS arm64; binary wheels for MacOS are only provided for
Intel x86 64 bits
([reference](https://stackoverflow.com/a/71653850/3902555)).
## Which issue(s) this PR fixes
This change fixes the following issue:
```bash
Traceback (most recent call last):
File "/etc/app/manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 75, in handle
self.check(databases=[database])
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 419, in check
all_issues = checks.run_checks(
File "/usr/local/lib/python3.9/site-packages/django/core/checks/registry.py", line 76, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 23, in check_resolver
return check_method()
File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 416, in check
for pattern in self.url_patterns:
File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 602, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 595, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/etc/app/extensions/engine_enterprise/engine_enterprise/engine_enterprise_urls.py", line 2, in <module>
from engine.urls import urlpatterns
File "/etc/app/engine/urls.py", line 36, in <module>
path("integrations/v1/", include("apps.integrations.urls", namespace="integrations")),
File "/usr/local/lib/python3.9/site-packages/django/urls/conf.py", line 34, in include
urlconf_module = import_module(urlconf_module)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/etc/app/apps/integrations/urls.py", line 5, in <module>
from .views import (
File "/etc/app/apps/integrations/views.py", line 14, in <module>
from django_sns_view.views import SNSEndpoint
File "/usr/local/lib/python3.9/site-packages/django_sns_view/views.py", line 17, in <module>
from django_sns_view.utils import confirm_subscription, verify_notification
File "/usr/local/lib/python3.9/site-packages/django_sns_view/utils.py", line 6, in <module>
import pem
File "/usr/local/lib/python3.9/site-packages/pem/__init__.py", line 26, in <module>
from . import twisted
File "/usr/local/lib/python3.9/site-packages/pem/twisted.py", line 9, in <module>
from OpenSSL.SSL import FILETYPE_PEM
File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import crypto, SSL
File "/usr/local/lib/python3.9/site-packages/OpenSSL/crypto.py", line 3279, in <module>
_lib.OpenSSL_add_all_algorithms()
AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'
```
- swaps out `django-push-notifications` for
[`fcm-django`](https://github.com/grafana/fcm-django). Again.. this is a
fork of the parent repo for exactly the same reason.. the migrations
point to `auth_user` without letting us use our own user model, this has
been patched in the `grafana` fork. The reason why we are using
`fcm-django` vs `django-push-notifications` is that the latter does not
support the new FCM API, only the "legacy" API. The legacy FCM API does
not support certain push notification settings that we would like to
use.
- modifies the iOS/Android specific push notification settings
- adds a `flower` pod in the `docker-compose-developer.yml`, useful for
debugging tasks locally
- sets the mobile app verification token TTL to 5 minutes when
developing locally. The default of 1 minute makes working with device
emulators really tricky..
This PR also swaps out the base image in `engine/Dockerfile` from
`python:3.9-alpine3.16` to `python:3.9-slim-buster`.
As to why.. in short, with the introduction of the `fcm-django` library
there is now a peer-dependency on
[`grpcio`](https://github.com/grpc/grpc) (which is used by
`firebase_admin`.. which I am using in this PR to interact directly with
Firebase Cloud Messaging (FCM)). `grpcio` does not publish wheels (read:
compiled binaries) for the Alpine distro. It does publish wheels for
Debian and hence `pip install -r requirements.txt` does not need to
build this library from the source distribution.
This is a [known
"issue"](https://github.com/grpc/grpc/issues/22815#issuecomment-1107874367)
and the recommended solution in the community is to.. not use alpine.
These were the numbers, when building the image locally, in terms of
image size and build time:
| | Local image size (uncompressed | Build time (may differ based on
your network speed) |
| ------------------------- | -------------------------------------- |
---------- |
| `python:3.9-alpine3.16` | 785MB | 320s |
| `python:3.9-slim-buster` | 1.05GB | 90s |
Co-authored-by: Salvatore Giordano <salvatoregiordanoo@gmail.com>
- removes APNS support
- changes the `django-push-notification` library from the `iskhakov`
fork to the [`grafana`
fork](https://github.com/grafana/django-push-notifications). This new
fork basically just patches an issue which affected the database
migrations of this django app (previously the library would not respect
the `USER_MODEL` setting when creating its tables and would instead
reference the `auth_user` table.. which we don't want)
- add `--no-cache` flag to the `make build` command
**NOTE**
A migration should be applied as follows:
```bash
# remove the four push_notifications tables, which have improper foreign key references
python manage.py migrate push_notifications zero
# recreate the tables with the proper foreign key references
python manage.py migrate
```
* Modify `docker-compose-developer` configuration files, and `Makefile`
to support running everything in containers for local development
- Make use of the COMPOSE_PROFILES env var that is supported by
docker-compose to allow swapping-out/turning off certain docker-compose
services.
- add makefile cleanup command. Will remove all docker resources related
to running the project locally
- The "restart grafana container" issue, where users would need
to restart their grafana container when setting up the project for the
first time, is now fixed (make command now runs yarn build:dev before docker-compose startup;
this ensures grafana-plugin/dist is available for grafana container before it starts up)
- The DEVELOPER.md has been updated as well to reflect these new changes. It
has been moved to ./dev/README.md (and references to the old file have
been updated).
- The redis image that is referenced in the docker-compose files
has been pinned to v7.0.5 (latest version as of this commit) to avoid
any surprises w/ future releases.
- remove root .dockerignore in favour of individual .dockerignore files
in ./engine and ./grafana-plugin
* remove email verification related code
* remove email verification related code
* remove sendgrid callback
* remove sendgrid related code
* remove sendgrid related code
* rename sendgrid app to email
* remove email from built-in channels
* remove email from built-in channels
* remove email from built-in channels
* add email backend: https://github.com/grafana/oncall/pull/50
* add email templater
* add email templater
* convert md to html
* add email settings to live settings
* use task to send email, handle some exceptions to create logs
* remove ERROR_NOTIFICATION_MAIL_DELIVERY_FAILED usage
* add email limit logic
* fix tests
* add docs
* remove old email templates
* remove old email templates
* add template_fields to messaging backend
* add messaging backends templates to public api
* add comment for deprecated fields
* fix test
* fix tests
* disable email by default
* don't retry on SMTPException and TimeoutError
* add tests
* bring email back to public api docs
* return ERROR_NOTIFICATION_MAIL_LIMIT_EXCEEDED
* make template_fields tuple
* build_subject_and_title -> build_subject_and_message
* add one more comment about template deprecation
* use 8 as backend id
* add comment about gaierror and BadHeaderError
* add comment on importing in notify_user_async
* edit oss docs
* add libs for celery + redis
* move redis & cache config to settings/base.py
* move rmq & celery config to settings/base.py
* BROKER -> BROKER_TYPE
* allow multiple database types
* flake8
* add sqlite db creation to dockerfile
* fix ci
* fix ci
* debug
* remove some defaults
* remove prints
* use local memory as cache on ci
* debug
* add DATABASE_DEFAULTS
* add ci test for sqlite + redis
* add ci test for sqlite + redis
* add ci test for sqlite + redis
* debug
* add redis healthcheck
* fix sqlite
* fix dev settings
* refactor dev settings
* tweak ci settings
* clear cache properly between tests
* move db and broker types to constants
* add librabbitmq deps
* use amqp instead of librabbitmq
