Commit graph

11 commits

Author SHA1 Message Date
Matias Bordese
132bdf235b
feat: update service account auth not to require rbac enabled org (#5360)
Related to https://github.com/grafana/oncall-private/issues/2826

RBAC enabled or not (OSS or cloud), it is possible to get service
account permissions, enabling perm check (for service account tokens) in
public API.

Also allow empty value for users in sync (instead of returning a 400
response).
2024-12-12 22:11:59 +00:00
Joey Orlando
fa071bcd6e
chore: add pytest-socket library + disable network calls in tests (#5315)
# What this PR does

Inspired by [this
discussion](https://github.com/grafana/oncall/pull/5307#discussion_r1862449480).
_tldr;_ ensures that if any of our tests try making an external network
call, they will fail.

Setup an example test:

```python
def test_external_network_call():
    import requests

    response = requests.get('https://www.example.com')
    assert response.status_code == 200
```

and it worked (failed; [example CI test
run](https://github.com/grafana/oncall/actions/runs/12106416991/job/33752144727?pr=5315#step:6:389))
as expected:

```bash
__________________________ test_external_network_call __________________________
    def test_external_network_call():
        import requests
    
>       response = requests.get('https://www.example.com')
requests   = <module 'requests' from '/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requests/__init__.py'>
apps/test_joey.py:4: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requests/api.py:73: in get
    return request("get", url, params=params, **kwargs)
        kwargs     = {}
        params     = None
        url        = 'https://www.example.com'
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requests/api.py:59: in request
    return session.request(method=method, url=url, **kwargs)
        kwargs     = {'params': None}
        method     = 'get'
        session    = <requests.sessions.Session object at 0x7f10ebaada90>
        url        = 'https://www.example.com'
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
        allow_redirects = True
        auth       = None
        cert       = None
        cookies    = None
        data       = None
        files      = None
        headers    = None
        hooks      = None
        json       = None
        method     = 'get'
        params     = None
        prep       = <PreparedRequest [GET]>
        proxies    = {}
        req        = <Request [GET]>
        self       = <requests.sessions.Session object at 0x7f10ebaada90>
        send_kwargs = {'allow_redirects': True, 'cert': None, 'proxies': OrderedDict(), 'stream': False, ...}
        settings   = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'verify': True}
        stream     = None
        timeout    = None
        url        = 'https://www.example.com'
        verify     = None
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
        adapter    = <requests.adapters.HTTPAdapter object at 0x7f10ebaada30>
        allow_redirects = True
        hooks      = {'response': []}
        kwargs     = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': None, ...}
        request    = <PreparedRequest [GET]>
        self       = <requests.sessions.Session object at 0x7f10ebaada90>
        start      = 1733064371.649901
        stream     = False
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/requests/adapters.py:667: in send
    resp = conn.urlopen(
        cert       = None
        chunked    = False
        conn       = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7f10ebaadd30>
        proxies    = OrderedDict()
        request    = <PreparedRequest [GET]>
        self       = <requests.adapters.HTTPAdapter object at 0x7f10ebaada30>
        stream     = False
        timeout    = Timeout(connect=None, read=None, total=None)
        url        = '/'
        verify     = True
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/urllib3/connectionpool.py:715: in urlopen
    httplib_response = self._make_request(
        assert_same_host = False
        body       = None
        body_pos   = None
        chunked    = False
        clean_exit = False
        conn       = None
        destination_scheme = None
        err        = None
        headers    = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
        http_tunnel_required = False
        is_new_proxy_conn = False
        method     = 'GET'
        parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/', query=None, fragment=None)
        pool_timeout = None
        redirect   = False
        release_conn = False
        release_this_conn = True
        response_kw = {'decode_content': False, 'preload_content': False}
        retries    = Retry(total=0, connect=None, read=False, redirect=None, status=None)
        self       = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7f10ebaadd30>
        timeout    = Timeout(connect=None, read=None, total=None)
        timeout_obj = Timeout(connect=None, read=None, total=None)
        url        = '/'
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/urllib3/connectionpool.py:404: in _make_request
    self._validate_conn(conn)
        chunked    = False
        conn       = <urllib3.connection.HTTPSConnection object at 0x7f10ebaadd60>
        httplib_request_kw = {'body': None, 'headers': {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}}
        method     = 'GET'
        self       = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7f10ebaadd30>
        timeout    = Timeout(connect=None, read=None, total=None)
        timeout_obj = Timeout(connect=None, read=None, total=None)
        url        = '/'
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/urllib3/connectionpool.py:1060: in _validate_conn
    conn.connect()
        __class__  = <class 'urllib3.connectionpool.HTTPSConnectionPool'>
        conn       = <urllib3.connection.HTTPSConnection object at 0x7f10ebaadd60>
        self       = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7f10ebaadd30>
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/urllib3/connection.py:363: in connect
    self.sock = conn = self._new_conn()
        self       = <urllib3.connection.HTTPSConnection object at 0x7f10ebaadd60>
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/urllib3/connection.py:174: in _new_conn
    conn = connection.create_connection(
        extra_kw   = {'socket_options': [(6, 1, 1)]}
        self       = <urllib3.connection.HTTPSConnection object at 0x7f10ebaadd60>
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/urllib3/util/connection.py:85: in create_connection
    sock.connect(sa)
        address    = ('www.example.com', 443)
        af         = <AddressFamily.AF_INET: 2>
        canonname  = ''
        err        = None
        family     = <AddressFamily.AF_UNSPEC: 0>
        host       = 'www.example.com'
        port       = 443
        proto      = 6
        res        = (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('93.184.215.14', 443))
        sa         = ('93.184.215.14', 443)
        sock       = <socket.socket fd=12, family=2, type=1, proto=6, laddr=('0.0.0.0', 0)>
        socket_options = [(6, 1, 1)]
        socktype   = <SocketKind.SOCK_STREAM: 1>
        source_address = None
        timeout    = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
inst = <socket.socket fd=12, family=2, type=1, proto=6, laddr=('0.0.0.0', 0)>
args = (('93.184.215.14', 443),), host = '93.184.215.14'
    def guarded_connect(inst, *args):
        host = host_from_connect_args(args)
        if host in allowed_ip_hosts_and_hostnames or (
            _is_unix_socket(inst.family) and allow_unix_socket
        ):
            return _true_connect(inst, *args)
    
>       raise SocketConnectBlockedError(allowed_list, host)
E       pytest_socket.SocketConnectBlockedError: A test tried to use socket.socket.connect() with host "93.184.215.14" (allowed: "calendar.google.com (142.251.167.100,142.251.167.101,142.251.167.102,142.251.167.113,142.251.167.138,142.251.167.139,2607:f8b0:4004:c09::65,2607:f8b0:4004:c09::66,2607:f8b0:4004:c09::71,2607:f8b0:4004:c09::8b),localhost (127.0.0.1,::1),oncall-dev-mariadb ()").
allow_unix_socket = False
allowed_ip_hosts_and_hostnames = {'127.0.0.1', '142.251.167.100', '142.251.167.101', '142.251.167.102', '142.251.167.113', '142.251.167.138', ...}
allowed_list = ['calendar.google.com (142.251.167.100,142.251.167.101,142.251.167.102,142.251.167.113,142.251.167.138,142.251.167.139...8b0:4004:c09::66,2607:f8b0:4004:c09::71,2607:f8b0:4004:c09::8b)', 'localhost (127.0.0.1,::1)', 'oncall-dev-mariadb ()']
args       = (('93.184.215.14', 443),)
host       = '93.184.215.14'
inst       = <socket.socket fd=12, family=2, type=1, proto=6, laddr=('0.0.0.0', 0)>
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/pytest_socket.py:252: SocketConnectBlockedError
```

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
2024-12-02 10:53:18 -05:00
Joey Orlando
673d2e9595
feat: persist is_grafana_irm_enabled from backend plugin sync data (#5171)
# What this PR does

Will start persisting the `organization.is_grafana_irm_enabled` flag
from the backend plugin's sync data that is sent to the oncall backend.
The implications of this are that when `is_grafana_irm_enabled` is set
to True, we will:
- start using `grafana-irm-app` prefixed RBAC permissions (RBAC
permissions for `grafana-irm-app`, as well as `grafana-oncall-app`, are
already being synced to the OnCall backend since
https://github.com/grafana/irm/pull/200 was merged/deployed)
- start building UI URLs w/ `grafana-irm-app` instead of
`grafana-oncall-app`

## Which issue(s) this PR closes

Closes https://github.com/grafana/irm/issues/242

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
2024-10-11 14:57:59 -04:00
Michael Derynck
d1cb862125
Make sync settings configurable (#5002)
# What this PR does
Add settings for how sync jobs get split up to control throughput of
requests.
- `SYNC_V2_MAX_TASKS ` controls how many tasks can run concurrently
- `SYNC_V2_PERIOD_SECONDS` controls the time offset before starting
another set of tasks each time `SYNC_V2_MAX_TASKS` is reached
- `SYNC_V2_BATCH_SIZE` controls how many organizations will be sync'd
per task

## Which issue(s) this PR closes

Related to [issue link here]

<!--
*Note*: If you want the issue to be auto-closed once the PR is merged,
change "Related to" to "Closes" in the line above.
If you have more than one GitHub issue that this PR closes, be sure to
preface
each issue link with a [closing
keyword](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue).
This ensures that the issue(s) are auto-closed once the PR has been
merged.
-->

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
2024-09-10 14:17:46 +00:00
Michael Derynck
cf69289888
Relax sync team serializer validation to match underlying model (#4976)
# What this PR does
- Teams do not require valid email addresses to be sync'd
- AvatarURL can be blank

## Which issue(s) this PR closes

Related to [issue link here]

<!--
*Note*: If you want the issue to be auto-closed once the PR is merged,
change "Related to" to "Closes" in the line above.
If you have more than one GitHub issue that this PR closes, be sure to
preface
each issue link with a [closing
keyword](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue).
This ensures that the issue(s) are auto-closed once the PR has been
merged.
-->

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
2024-09-03 18:47:54 +00:00
Michael Derynck
962cc34432
Compress sync data (#4951)
# What this PR does
- Compresses sync data being sent to engine
- Minor fix to log messages when JSON parse errors occur

## Which issue(s) this PR closes

Related to [issue link here]

<!--
*Note*: If you want the issue to be auto-closed once the PR is merged,
change "Related to" to "Closes" in the line above.
If you have more than one GitHub issue that this PR closes, be sure to
preface
each issue link with a [closing
keyword](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue).
This ensures that the issue(s) are auto-closed once the PR has been
merged.
-->

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
2024-08-29 18:36:35 +00:00
Michael Derynck
586205368e
Sync process and backend plugin tweaks (#4939)
- Split up sync requests into batches to run concurrently
- Add more logging for when API calls to Grafana fail to parse
- Call sync from backend plugin when status is called
- Lock sync from backend plugin to only run every 5 mins
- Add timer display for API call to sync to return remaining time before
sync can execute
- Remove locks from celery task since it's work is low cost and we lock
in the backend plugin anyways.
2024-08-28 16:34:30 +00:00
Michael Derynck
a25d44da1a
Move validate_grafana_token_format to common location, use in sync_v2 (#4919)
# What this PR does
Moves validate_grafana_token_format to GrafanaAPIClient, use it in
sync_v2 to improve logging and skip requests that would not work.

## Which issue(s) this PR closes

Related to [issue link here]

<!--
*Note*: If you want the issue to be auto-closed once the PR is merged,
change "Related to" to "Closes" in the line above.
If you have more than one GitHub issue that this PR closes, be sure to
preface
each issue link with a [closing
keyword](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue).
This ensures that the issue(s) are auto-closed once the PR has been
merged.
-->

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
2024-08-23 19:52:53 +00:00
Michael Derynck
042fb49aaf
Add logging for invalid api_tokens during sync (#4905)
# What this PR does
Add logging for when we skip an organization for sync if it is missing
its api token.

## Which issue(s) this PR closes

Related to [issue link here]

<!--
*Note*: If you want the issue to be auto-closed once the PR is merged,
change "Related to" to "Closes" in the line above.
If you have more than one GitHub issue that this PR closes, be sure to
preface
each issue link with a [closing
keyword](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests#linking-a-pull-request-to-an-issue).
This ensures that the issue(s) are auto-closed once the PR has been
merged.
-->

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.
2024-08-22 20:40:18 +00:00
Dominik Broj
06d19bf6e9
New OnCall plugin initialization process (#4657)
# What this PR does

New OnCall plugin initialization process

## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] Added the relevant release notes label (see labels prefixed w/
`release:`). These labels dictate how your PR will
    show up in the autogenerated release notes.

---------

Co-authored-by: Michael Derynck <michael.derynck@grafana.com>
Co-authored-by: Matias Bordese <mbordese@gmail.com>
2024-08-16 16:43:52 +00:00
Matias Bordese
35f23cdcc6
Rework organization sync and grafana plugin engine backend (#4756)
Related to
https://github.com/grafana/oncall-private/issues/2806#issuecomment-2246286918.

Prepare engine for the backend plugin enablement/migration:

 - Refactor sync code
- Improve plugin user authentication to set up user on-the-fly (when
missing)
- Implement v2 endpoints for install, sync and status (to be used via
the backend plugin)

(most of the changes come from
https://github.com/grafana/oncall/pull/4657; backport all engine changes
that keep backwards compatibility)
2024-07-31 16:12:56 +00:00