Capture and relay key id for decryption requests

[jot2re]

So far we have been using a single key per ASC on the KMS side and hence kept it as part of the configuration.
However, to support multiple key's per ASC along with appropriate key rotation, we need to support the idea of multiple keys being constructed per ASC.
This involves capturing the key id from the event emitted after key generation on the KMS blockchain and then using it again when requesting the decryption.

This is not high priority right now, but ideally it would be good to work on it during Q4 2024

[RomanBredehoft]

currently, fetching the keyurl endpoint gives the following kind of output. Ex, for pub key (but same for verifier keys):

...
      "fhe_key_info":
        [
          {
            "fhe_public_key":
              {
                "data_id": "beb70cb9fdbabf785242de498d6ec0ed282921d7",
                ...

if we have multiple keys :

• keys are not indexed by key id, so we need to loop over all keys and check key id
• or, output format will need to change (breaking change, but makes more sense imo)

do we agree ? @jot2re

full output 👇

{
  "response":
    {
      "crs":
        {
          "256":
            {
              "data_id": "53d7ffe72d3afa2b80adbba610e1dd465f21955d",
              "param_choice": 1,
              "signatures":
                [
                  "40000000000000009aee1624fda0d5a4e0e292d9731e5d7777711e1b19b140d2820620efe594520f01203776dcbdf32047df622514dff4688be065a0fc1b11d46f68d8e13ead8b4b",
                  "400000000000000080e9c4d80f6e7d3554508bf41f5f96f34593bb05e1bae859566723e1a70cb9cc077adef4d09f4e3c1313deda9a9b8eec8b738f1f697257e68f29bb4a2aec3406",
                  "4000000000000000d567bcb16bb9c20eb88837cebd108df0e7de67c3bea662a07245ba40f123dcd627943ef8c6944c5bb1bdbc1ac797021695ee675dd9c933c1b3bcf2f4dab8e5f0",
                  "40000000000000001a3d8c1b0b38618b9fa62b637a3e7964211b77601e7e8a7168dff3d40d681c544d0aefc282643a619ec3464200fda96b3c0d47ac346168f2de249b905a2a616c",
                ],
              "urls":
                [
                  "s3://kms-dev-v1/threshold/PUB-p3/CRS/53d7ffe72d3afa2b80adbba610e1dd465f21955d",
                  "s3://kms-dev-v1/threshold/PUB-p2/CRS/53d7ffe72d3afa2b80adbba610e1dd465f21955d",
                  "s3://kms-dev-v1/threshold/PUB-p1/CRS/53d7ffe72d3afa2b80adbba610e1dd465f21955d",
                  "s3://kms-dev-v1/threshold/PUB-p4/CRS/53d7ffe72d3afa2b80adbba610e1dd465f21955d",
                ],
            },
        },
      "fhe_key_info":
        [
          {
            "fhe_public_key":
              {
                "data_id": "beb70cb9fdbabf785242de498d6ec0ed282921d7",
                "param_choice": 1,
                "signatures":
                  [
                    "4000000000000000f925d51f2f8e0ced9b1f2c791146799e3310c643db592e9b423ec45f2e6ceeab4acc1a0e850f0fe06f0ae4aac0d72bdf2044339237bb63ae0a2ca065bcc650af",
                    "4000000000000000729d970ac71874a4d4d955db9265a7adfb8cb66f647fc883705f1aaeaa65ab8e65bc391aace9047cff7bf4a75a1afe9f78804f95901bd43c1ea7d8b2591bfc77",
                    "4000000000000000056f580dc2b50072948aad78f5ffa330b43c10b284ed5f18d2606fdc6cd0be0b2704b8070a9032f4a2cf2983562797f04ee1706215d6d718daa06cc1aebea41c",
                    "40000000000000004c008d559d38a6b9718612f644b11db17565af73ecd77e0732fbc4587b46422d1e79b9ec61792b908c51141f2a8b5e735cf04fd238c4294efb9d909585bd007d",
                  ],
                "urls":
                  [
                    "s3://kms-dev-v1/threshold/PUB-p3/PublicKey/beb70cb9fdbabf785242de498d6ec0ed282921d7",
                    "s3://kms-dev-v1/threshold/PUB-p2/PublicKey/beb70cb9fdbabf785242de498d6ec0ed282921d7",
                    "s3://kms-dev-v1/threshold/PUB-p1/PublicKey/beb70cb9fdbabf785242de498d6ec0ed282921d7",
                    "s3://kms-dev-v1/threshold/PUB-p4/PublicKey/beb70cb9fdbabf785242de498d6ec0ed282921d7",
                  ],
              },
            "fhe_server_key":
              {
                "data_id": "beb70cb9fdbabf785242de498d6ec0ed282921d7",
                "param_choice": 1,
                "signatures":
                  [
                    "4000000000000000d78f1868596a90f8e0d6ec253db8de4b37cf8e84975af9076f748a1b752c2f2472cb7af15025f9955c5abe3a40cd1970f59c4c5f2ac5ed4bd8cbeef21108a8fc",
                    "400000000000000083544fc4513190f899f64203f65c261dc9d85ba968f5e3331516006476f3200e41f4ffda1fec767697965d5bade6ce59692effa1f04f83bfccc284997d98ec27",
                    "4000000000000000d2249526d7ab915697db41f3a128fe6296fd8ed69d1eba6615f095c964e5c39c5e702a69a2ef0142424488897fa15819aafc102eb5c9da60accc34a7a697e568",
                    "40000000000000006cd1f8ffce766ce41cb5ef7a77853061d273696912417476c455ffb1b52a857016f5ee04f05926d0d13edcae0ebe8846f3cfbe027697430c5328f8bd185d07f6",
                  ],
                "urls":
                  [
                    "s3://kms-dev-v1/threshold/PUB-p3/ServerKey/beb70cb9fdbabf785242de498d6ec0ed282921d7",
                    "s3://kms-dev-v1/threshold/PUB-p2/ServerKey/beb70cb9fdbabf785242de498d6ec0ed282921d7",
                    "s3://kms-dev-v1/threshold/PUB-p1/ServerKey/beb70cb9fdbabf785242de498d6ec0ed282921d7",
                    "s3://kms-dev-v1/threshold/PUB-p4/ServerKey/beb70cb9fdbabf785242de498d6ec0ed282921d7",
                  ],
              },
          },
        ],
      "verf_public_key":
        [
          {
            "key_id": "e164d9de0bec6656928726433cc56bef6ee8417a",
            "server_id": 3,
            "verf_public_key_address": "s3://kms-dev-v1/threshold/PUB-p3/VerfAddress/e164d9de0bec6656928726433cc56bef6ee8417a",
            "verf_public_key_url": "s3://kms-dev-v1/threshold/PUB-p3/VerfKey/e164d9de0bec6656928726433cc56bef6ee8417a",
          },
          {
            "key_id": "e164d9de0bec6656928726433cc56bef6ee8417a",
            "server_id": 2,
            "verf_public_key_address": "s3://kms-dev-v1/threshold/PUB-p2/VerfAddress/e164d9de0bec6656928726433cc56bef6ee8417a",
            "verf_public_key_url": "s3://kms-dev-v1/threshold/PUB-p2/VerfKey/e164d9de0bec6656928726433cc56bef6ee8417a",
          },
          {
            "key_id": "e164d9de0bec6656928726433cc56bef6ee8417a",
            "server_id": 1,
            "verf_public_key_address": "s3://kms-dev-v1/threshold/PUB-p1/VerfAddress/e164d9de0bec6656928726433cc56bef6ee8417a",
            "verf_public_key_url": "s3://kms-dev-v1/threshold/PUB-p1/VerfKey/e164d9de0bec6656928726433cc56bef6ee8417a",
          },
          {
            "key_id": "e164d9de0bec6656928726433cc56bef6ee8417a",
            "server_id": 4,
            "verf_public_key_address": "s3://kms-dev-v1/threshold/PUB-p4/VerfAddress/e164d9de0bec6656928726433cc56bef6ee8417a",
            "verf_public_key_url": "s3://kms-dev-v1/threshold/PUB-p4/VerfKey/e164d9de0bec6656928726433cc56bef6ee8417a",
          },
        ],
    },
  "status": "success",
}

[jot2re]

@RomanBredehoft just loop for now. We can make an issue about changing the format but for the next long time there will not be more than a few keys. Even if everything takes off it will not be more than 100 keys so I think it might be a bit of a premature optimization since it is an endpoint that requires changes in many different repo

[jot2re]

But in general I think we need to be really careful whenever we make endpoints. I think we should always review them in a group to make sure formats are optimal for everyone

[RomanBredehoft]

yeah ok I see thanks, makes sense

[RomanBredehoft]

looking at the code, I believe that the get_key_id we currently provide in the gateway is not reliable :

• to fetch this key id, we (down the line) call get_all_values_from_operation from the ASC (here)
• but this asc func loops over all (keygen) transactions in a lexicographical order (here)
• so I don't see how the get_key_id could return the latest key id when there are multiple keys

but it'll most likely be fixed by https://github.com/zama-ai/kms-core/issues/1506 if we decide to instead use the key id from the gateway's toml

[jot2re]

@RomanBredehoft the get_key_id method as it is now was never meant as a permanent solution. This is also why it warns if more than 1 key is present.
But I agree the term "latest" is probably not right in this situation as we are not sure about the ordering (at least the ordering is not clear to me) so latest here just means in the list of events returned.
In relation to issue #1506 I added a comment.

It would likely also be possible to simply update the functionality of get_key_id to ensure it is actually the latest key that gets returned. But this might not be the best long term solution as we might want multiple keys in the system.

[mortendahl]

Is it not possible to propagate the request id to the caller (eg CLI tool)? Whoever does this, would then configure the fhEVM chain with it, moving this question of bookkeeping to the fhEVM chain.

[RomanBredehoft]

KeyGenResponseValues has the request_id in it already yes, so I believe it's already accessible from fhevm's side within the events normally (or at least it should be easy to propagate)