Write our own distributed locking (async etcd3 with concurrency API or aioredis v2 redlock)

[achimnol]

Currently, Python's etcd3 client libraries have limtied support for latest updates such as concurrency API.

The official grpcio began to support asyncio natively since 1.30.0 release (June 2020) and the import namespace is fixed to grpc.aio in 1.33.2, and it's now 1.44.0.

However, etcd3, aioetcd3, etcd3-aio, aetcd3 libraries have not followed these updates and some of them are relying on a 3rd-party aiogrpc library, which seems to be abandoned in favor of the official grpc.aio. Moreover, all those libraries are not being actively maintained.

Here is an issue about adding concurrency API to aetcd, but it seems not to be active:

• Support concurrency API  martyanov/aetcd#2

Our common.etcd module wraps etcd3 with our own async wrappers, but has many issues such as being unable to cleanly shutdown watcher threads. There have been several attempts to replace the underlying implementation, but none was succssful due to watcher API supports and poor cancellation supports.

Now we desperately need to use a proper global lock implementation where locks can be arbitrarily defined at runtime.
Let's write a new etcd v3 client library using the official grpc.aio including the concurrency API support.
We could also add our own high-level API to periodically generate a specific event via watcher API.

Potential alternatives:

• Write our own redlock algorithm as a common module using aioredis-py (v2). The official Redis docs is outdated: redlock-py is absorbed into redis-py and aioredlock has not caught up aioredis-py v2.
  • How about support aioredis 2.0? joanvila/aioredlock#97
  • [2.0] Should Lock default to using contextvars? aio-libs-abandoned/aioredis-py#1040
    • I have tried to port it but it was not successful.
  • Note that if we go this way, the locking connections should be separated from the connection pool.

Rejected alternatives:

• PostgreSQL advisory lock. We are experiencing excessive locking issues with this already, and it has a severe limitation that lock IDs should be represented as predefined integers. We need to be able to define locks with arbitrary names at runtime.