oncall-engine/engine/common/tests/test_cache.py

52 lines
2.3 KiB
Python
Raw Normal View History

patch redis cluster multi-key operations (#3496) # Which issue(s) this PR fixes Related to https://github.com/grafana/oncall-private/issues/2363 Addresses this issue that arises when using `cache.get_many`/`cache.set_many` operations with a Redis Cluster: ```python3 File "/usr/local/lib/python3.11/site-packages/redis/cluster.py", line 1006, in determine_slot raise RedisClusterException( redis.exceptions.RedisClusterException: MGET - all keys must map to the same key slot ``` From the Redis Cluster [docs](https://redis.io/docs/reference/cluster-spec/#hash-tags), this can be addressed with this 👇 . Basically this will ensure that keys in multi-key operations will resolve to the same hash slot (read: node): > Hash tags > There is an exception for the computation of the hash slot that is used in order to implement hash tags. Hash tags are a way to ensure that multiple keys are allocated in the same hash slot. This is used in order to implement multi-key operations in Redis Cluster. > > To implement hash tags, the hash slot for a key is computed in a slightly different way in certain conditions. If the key contains a "{...}" pattern only the substring between { and } is hashed in order to obtain the hash slot. However since it is possible that there are multiple occurrences of { or } the algorithm is well specified by the following rules: > > IF the key contains a { character. > AND IF there is a } character to the right of {. > AND IF there are one or more characters between the first occurrence of { and the first occurrence of }. > Then instead of hashing the key, only what is between the first occurrence of { and the following first occurrence of } is hashed. ## Checklist - [x] Unit, integration, and e2e (if applicable) tests updated - [ ] Documentation added (or `pr:no public docs` PR label added if not required) - [ ] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not required)
2023-12-04 13:08:57 -05:00
from django.test import override_settings
from common.cache import ensure_cache_key_allocates_to_the_same_hash_slot
PATTERN = "schedule_oncall_users"
NON_EXISTENT_PATTERN = "nmzxcnvmzxcv"
NUM_CACHE_KEYS = 5
SINGLE_CACHE_KEY = f"{PATTERN}_0"
CACHE_KEYS = [f"{PATTERN}_{pk}" for pk in range(NUM_CACHE_KEYS)]
SET_MANY_CACHE_KEYS_DICT = {k: "foo" for k in CACHE_KEYS}
def test_ensure_cache_key_allocates_to_the_same_hash_slot() -> None:
def _convert_key(key: str) -> str:
return key.replace(PATTERN, f"{{{PATTERN}}}")
# when USE_REDIS_CLUSTER is False the method should just return the cache keys
with override_settings(USE_REDIS_CLUSTER=False):
assert ensure_cache_key_allocates_to_the_same_hash_slot(SINGLE_CACHE_KEY, PATTERN) == SINGLE_CACHE_KEY
assert ensure_cache_key_allocates_to_the_same_hash_slot(CACHE_KEYS, PATTERN) == CACHE_KEYS
assert (
ensure_cache_key_allocates_to_the_same_hash_slot(SET_MANY_CACHE_KEYS_DICT, PATTERN)
== SET_MANY_CACHE_KEYS_DICT
)
# when USE_REDIS_CLUSTER is True the method should wrap the specified pattern within the cache keys in curly brackets
with override_settings(USE_REDIS_CLUSTER=True):
# works with a single str cache key
assert ensure_cache_key_allocates_to_the_same_hash_slot(SINGLE_CACHE_KEY, PATTERN) == _convert_key(
SINGLE_CACHE_KEY
)
# works with a list (useful for cache.get_many operations)
assert ensure_cache_key_allocates_to_the_same_hash_slot(CACHE_KEYS, PATTERN) == [
_convert_key(k) for k in CACHE_KEYS
]
# works with a dict (useful for cache.set_many operations)
assert ensure_cache_key_allocates_to_the_same_hash_slot(SET_MANY_CACHE_KEYS_DICT, PATTERN) == {
_convert_key(k): v for k, v in SET_MANY_CACHE_KEYS_DICT.items()
}
# if the pattern doesn't exist, we don't wrap it in brackets
assert (
ensure_cache_key_allocates_to_the_same_hash_slot(SINGLE_CACHE_KEY, NON_EXISTENT_PATTERN) == SINGLE_CACHE_KEY
)
assert ensure_cache_key_allocates_to_the_same_hash_slot(CACHE_KEYS, NON_EXISTENT_PATTERN) == CACHE_KEYS
assert (
ensure_cache_key_allocates_to_the_same_hash_slot(SET_MANY_CACHE_KEYS_DICT, NON_EXISTENT_PATTERN)
== SET_MANY_CACHE_KEYS_DICT
)