Fixes issue when running OnCall locally, on an M1 Mac, and using
PostgreSQL as the database. (ie. `COMPOSE_PROFILES=postgres...`).
Currently getting:
```bash
django.db.utils.OperationalError: SCRAM authentication requires libpq version 10 or above
```
I also tried simply adding `libpq-dev` to the `Dockerfile` but this
change alone does not solve the issue. See
[here](https://github.com/MobSF/Mobile-Security-Framework-MobSF/issues/1898)
for a similar reported issue on GitHub.
**Root Cause**
This issue is caused because `psycopg2-binary` 2.9.3 [doesn't
provide](https://pypi.org/project/psycopg2-binary/2.9.3/#files) binary
wheels for MacOS arm64; binary wheels for MacOS are only provided for
Intel x86 64 bits
([reference](https://stackoverflow.com/a/71653850/3902555)).
## Which issue(s) this PR fixes
This change fixes the following issue:
```bash
Traceback (most recent call last):
File "/etc/app/manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 75, in handle
self.check(databases=[database])
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 419, in check
all_issues = checks.run_checks(
File "/usr/local/lib/python3.9/site-packages/django/core/checks/registry.py", line 76, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/usr/local/lib/python3.9/site-packages/django/core/checks/urls.py", line 23, in check_resolver
return check_method()
File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 416, in check
for pattern in self.url_patterns:
File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 602, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/usr/local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python3.9/site-packages/django/urls/resolvers.py", line 595, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/etc/app/extensions/engine_enterprise/engine_enterprise/engine_enterprise_urls.py", line 2, in <module>
from engine.urls import urlpatterns
File "/etc/app/engine/urls.py", line 36, in <module>
path("integrations/v1/", include("apps.integrations.urls", namespace="integrations")),
File "/usr/local/lib/python3.9/site-packages/django/urls/conf.py", line 34, in include
urlconf_module = import_module(urlconf_module)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/etc/app/apps/integrations/urls.py", line 5, in <module>
from .views import (
File "/etc/app/apps/integrations/views.py", line 14, in <module>
from django_sns_view.views import SNSEndpoint
File "/usr/local/lib/python3.9/site-packages/django_sns_view/views.py", line 17, in <module>
from django_sns_view.utils import confirm_subscription, verify_notification
File "/usr/local/lib/python3.9/site-packages/django_sns_view/utils.py", line 6, in <module>
import pem
File "/usr/local/lib/python3.9/site-packages/pem/__init__.py", line 26, in <module>
from . import twisted
File "/usr/local/lib/python3.9/site-packages/pem/twisted.py", line 9, in <module>
from OpenSSL.SSL import FILETYPE_PEM
File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import crypto, SSL
File "/usr/local/lib/python3.9/site-packages/OpenSSL/crypto.py", line 3279, in <module>
_lib.OpenSSL_add_all_algorithms()
AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'
```
Modifies the Firebase app initialization to explicitly specify the GCP
project ID where the Firebase app is. Previously it would use the
project associated with the service account being used.
Currently unable to exec into a k8s pod. I believe this is because
`bash` is missing from the Docker image after switching from alpine to
debian buster. Debugging this by adding in `bash`.
- swaps out `django-push-notifications` for
[`fcm-django`](https://github.com/grafana/fcm-django). Again.. this is a
fork of the parent repo for exactly the same reason.. the migrations
point to `auth_user` without letting us use our own user model, this has
been patched in the `grafana` fork. The reason why we are using
`fcm-django` vs `django-push-notifications` is that the latter does not
support the new FCM API, only the "legacy" API. The legacy FCM API does
not support certain push notification settings that we would like to
use.
- modifies the iOS/Android specific push notification settings
- adds a `flower` pod in the `docker-compose-developer.yml`, useful for
debugging tasks locally
- sets the mobile app verification token TTL to 5 minutes when
developing locally. The default of 1 minute makes working with device
emulators really tricky..
This PR also swaps out the base image in `engine/Dockerfile` from
`python:3.9-alpine3.16` to `python:3.9-slim-buster`.
As to why.. in short, with the introduction of the `fcm-django` library
there is now a peer-dependency on
[`grpcio`](https://github.com/grpc/grpc) (which is used by
`firebase_admin`.. which I am using in this PR to interact directly with
Firebase Cloud Messaging (FCM)). `grpcio` does not publish wheels (read:
compiled binaries) for the Alpine distro. It does publish wheels for
Debian and hence `pip install -r requirements.txt` does not need to
build this library from the source distribution.
This is a [known
"issue"](https://github.com/grpc/grpc/issues/22815#issuecomment-1107874367)
and the recommended solution in the community is to.. not use alpine.
These were the numbers, when building the image locally, in terms of
image size and build time:
| | Local image size (uncompressed | Build time (may differ based on
your network speed) |
| ------------------------- | -------------------------------------- |
---------- |
| `python:3.9-alpine3.16` | 785MB | 320s |
| `python:3.9-slim-buster` | 1.05GB | 90s |
Co-authored-by: Salvatore Giordano <salvatoregiordanoo@gmail.com>
Adds a make command, `enable-mobile-app-feature-flags`, which sets the
backend feature flag in `./dev/.env.dev`, and updates a record in the
`base_dynamicsetting` database table, which are needed to enable the
mobile app backend features.
- removes APNS support
- changes the `django-push-notification` library from the `iskhakov`
fork to the [`grafana`
fork](https://github.com/grafana/django-push-notifications). This new
fork basically just patches an issue which affected the database
migrations of this django app (previously the library would not respect
the `USER_MODEL` setting when creating its tables and would instead
reference the `auth_user` table.. which we don't want)
- add `--no-cache` flag to the `make build` command
**NOTE**
A migration should be applied as follows:
```bash
# remove the four push_notifications tables, which have improper foreign key references
python manage.py migrate push_notifications zero
# recreate the tables with the proper foreign key references
python manage.py migrate
```
need to check that the user associated
with the request actually has an organization_id
attribute (there are certain cases, such as
if the request.user is a "django user"
where this attribute would not be present)
* Add ability to configure FCM_API_KEY and FCM_POST_URL
* Delete APNSDevice and GCMDevice instances when unlinking the mobile app backend
* Add a simple FCM relay endpoint
* GCM -> FCM
* comment
* Centralize timezone validation into one spot + add serializer validation
for schedules and oncall shifts (both public and internal API)
* add engine-manage make command
* Modify plugin.json to support RBAC role registration
* defines 26 new custom roles in plugin.json. The main roles are:
- Admin: read/write access to everything in OnCall
- Reader: read access to everything in OnCall
- OnCaller : read access to everything in OnCall + edit access to Alert Groups and Schedules
- <object-type> Editor: read/write access to everything related to <object-type>
- <object-type> Reader: read access for <object-type>
- User Settings Admin: read/write access to all user's settings, not just own settings. This is in comparison to User Settings Editor which can only read/write own settings
* update changelog and documentation (#686)
* implement RBAC for OnCall backend
This commit refactors backend authorization. It trys to use RBAC authorization if the org's grafana instance supports it, otherwise it falls back to basic role authorization.
* update RBAC backend tests
* add tests for RBAC changes
- run backend tests as matrix where RBAC is enabled/disabled. When RBAC is enabled, the permissions granted are read from the role grants in the frontend's plugin.json file (instead of relying what we specify in RBACPermission.Permissions)
- remove --reuse-db --nomigrations flags from engine/tox.ini
- minor autoformatting changes to docker-compose-developer.yml
* remove --ds=settings.ci-test from pytest CI command
DJANGO_SETTINGS_MODULE is already specified as an env var so this is just unecessary duplication
* update gitignore
* update github action job name for "test"
* RBAC frontend changes
* refactors the use of basic roles (ex. Viewer, Editor, Admin) use RBAC permissions (when supported), or falling back to basic roles when RBAC is not supported.
- updates the UserAction enum in grafana-plugin/src/state/userAction.ts. Previously this was hardcoded to a list of strings that were being returned by the OnCall API. Now the values here correspond to the permissions in plugin.json (plus a fallback role)
* changes per Gabriel's comments:
- get rid of group attribute in rbac roles
- remove displayName role attribute
- remove hidden role attribute
- add back role to includes section
* don't try to update user timezone if they don't have permission
* Improve feedback so template errors are given to user
* Add security error logging
* Add limits for templates, payloads, results
* Show popup error notification for webhook errors and template errors that don't have a result
* Update tests
* Split exceptions into warnings/errors to give more control when previewing, rendering, saving templates
* Limit title lengths
* Make TypeError a warning
* Adjust title length limit
* Remove length limiting on urlize since it is being done on template render
* Fix tests
* Add KeyError and ValueError to warnings
* No longer enforcing json result when saving webhook in case it is dependent on payload
* Add tests for expected exceptions coming from apply_jinja_template
* Update changelog
* Send raw post if template result is not JSON
* add permalinks list to internal API alertgroup view
* add user's name and full avatar URL to the user view
* make avatar_full_url a property
* fix tests
* fix user connection criteria
