kli piped ingestion/excretion without mailbox/polling

[jasoncolburne]

Sometimes (when developing another implementation for instance - when validating correctness) it would be useful to be able to put data directly in a db. Likewise, it would be useful to pull this data from a db.

Consider this scriptable example (assumes -i has been implemented for vc export):

kli vc export -n first_name -s [SAID] -i > input
cat input | kli ingest -n second_name
kli vc export -n second_name -s [SAID] -i > output
diff -u input output

which could be extended to perform integration testing for other implementations:

custom_implementation dump [SAID] > input
cat input | kli ingest -n second_name
kli vc export -n second_name -s [SAID] -i > output
diff -u input output

The idea is that input would look something like an unpretty version of this data (with signatures), at a minimum:

[
    {
        "v": "KERI10JSON0001b7_",
        "t": "icp",
        "d": "EIaJ5gpHSL9nl1XIWDkfMth1uxbD-AfLkqdiZL6S7HkZ",
        "i": "EIaJ5gpHSL9nl1XIWDkfMth1uxbD-AfLkqdiZL6S7HkZ",
        "s": "0",
        "kt": "1",
        "k": [
            "DJv-oxWcj6crSUOvXdIppBhMuIBqoo6rr43xk6A8DWMp"
        ],
        "nt": "1",
        "n": [
            "EGEe3wLra7vhK27WvvnQhookAs2Dc1PPzSGpRtBC4f4N"
        ],
        "bt": "3",
        "b": [
            "BBilc4-L3tFUnfM_wJr4S4OJanAv_VmF_dJNN6vkf2Ha",
            "BLskRTInXnMxWaGqcpSyMgo0nYbalW99cGZESrz3zapM",
            "BIKKuvBwpmDVA4Ds-EpL5bt9OqPzWPja2LigFYZN2YfX"
        ],
        "c": [],
        "a": []
    },
    {
        "v": "KERI10JSON000113_",
        "t": "vcp",
        "d": "EAuvxnObLjS8E2Yn9-7twA8_ZqEsMC3vispqy26NVdaW",
        "i": "EAuvxnObLjS8E2Yn9-7twA8_ZqEsMC3vispqy26NVdaW",
        "ii": "EIaJ5gpHSL9nl1XIWDkfMth1uxbD-AfLkqdiZL6S7HkZ",
        "s": "0",
        "c": [
            "NB"
        ],
        "bt": "0",
        "b": [],
        "n": "AGHblRF4Vdcg_Q5Y4j7LTdFPR04qH4ARR2XrK-Xb6HRz"
    },
    {
        "v": "KERI10JSON00013a_",
        "t": "ixn",
        "d": "ED7MXKROzbMwLMnDojpcOnCLWYBMiDo0LY5DbCzy5TB-",
        "i": "EIaJ5gpHSL9nl1XIWDkfMth1uxbD-AfLkqdiZL6S7HkZ",
        "s": "1",
        "p": "EIaJ5gpHSL9nl1XIWDkfMth1uxbD-AfLkqdiZL6S7HkZ",
        "a": [
            {
                "i": "EAuvxnObLjS8E2Yn9-7twA8_ZqEsMC3vispqy26NVdaW",
                "s": "0",
                "d": "EAuvxnObLjS8E2Yn9-7twA8_ZqEsMC3vispqy26NVdaW"
            }
        ]
    },
    {
        "v": "ACDC10JSON0001d9_",
        "d": "EKriPO1JGk369l5zJGzfFn-yS0dazYu0ktCg8h5pxKZ3",
        "i": "EIaJ5gpHSL9nl1XIWDkfMth1uxbD-AfLkqdiZL6S7HkZ",
        "ri": "EAuvxnObLjS8E2Yn9-7twA8_ZqEsMC3vispqy26NVdaW",
        "s": "EIxAox3KEhiQ_yCwXWeriQ3ruPWbgK94NDDkHAZCuP9l",
        "a": {
            "d": "EBLyBuP_LUGSP3cDUKxQiCe86Sh5J4R6XXqz2hjVEc5H",
            "i": "EJS0-vv_OPAQCdJLmkd5dT0EW-mOfhn_Cje4yzRjTv8q",
            "dt": "2023-04-23T09:10:37.698209+00:00",
            "destination": "Osireion",
            "treasureSplit": "50/50",
            "partyThreshold": 2,
            "journeyEndorser": "Ramiel"
        }
    },
    {
        "v": "KERI10JSON0000ed_",
        "t": "iss",
        "d": "EHD81HVJGDhFWt82QCS_7II44Lqf9iO43ayt4aZ7wqWr",
        "i": "EKriPO1JGk369l5zJGzfFn-yS0dazYu0ktCg8h5pxKZ3",
        "s": "0",
        "ri": "EAuvxnObLjS8E2Yn9-7twA8_ZqEsMC3vispqy26NVdaW",
        "dt": "2023-04-23T09:10:37.698209+00:00"
    },
    {
        "v": "KERI10JSON00013a_",
        "t": "ixn",
        "d": "EKmNtSAju7d3KhdxzIdyZycZNgAWDKfW5YDRUy9eDHis",
        "i": "EIaJ5gpHSL9nl1XIWDkfMth1uxbD-AfLkqdiZL6S7HkZ",
        "s": "2",
        "p": "ED7MXKROzbMwLMnDojpcOnCLWYBMiDo0LY5DbCzy5TB-",
        "a": [
            {
                "i": "EKriPO1JGk369l5zJGzfFn-yS0dazYu0ktCg8h5pxKZ3",
                "s": "0",
                "d": "EHD81HVJGDhFWt82QCS_7II44Lqf9iO43ayt4aZ7wqWr"
            }
        ]
    }
]

This provides everything needed to verify an ACDC. The idea would be that if chained, many ACDCs, KELs and TELs would be returned, sequenced such that dependency SAIDs resolve before their use (suitable for unproxied ingestion).

[2byrds]

+1 for a good idea and another +1 for using the word 'excretion' :)

[jasoncolburne]

Just realized this may already be possible with kli vc export --full -n name -a alias -s [SAID], only the ordering may not be what I'm after. I have to break one down.

[SmithSamuelM]

The direct mode demo code directory examples do something like this. i.e. send the dependencies first so new events immediately resolve. This works well for single sig identifiers. This is more problematic with multi-sig and delegated identifiers.

[jasoncolburne]

Thanks Sam!

[jasoncolburne]

Here's some code for ingestion if anyone is doing integration testing:

from keri.app import habbing
from keri.vdr import (eventing, credentialing, verifying)

from keri import help

import logging

with habbing.openHab(name="test", temp=False) as (hby, hab):
    help.ogler.resetLevel(level=logging.DEBUG)
    
    rgy = credentialing.Regery(hby=hby, name='test', base='')
    vry = verifying.Verifier(hby=hby, reger=rgy.reger)
    vry.resolver.add('.. schema said goes here ..', b'.. schema goes here ..')

    raw = (
"""
    .. messages go here ..
"""
    )

    hby.psr.vry = vry
    hby.psr.tvy = vry.tvy
    hby.psr.parse(bytearray(raw.replace('\n', '').encode("utf-8")))

[jasoncolburne]

With temp set to false, if you run this on the same set of messages, you'll need to rm -rf ~/.keri to avoid duplicity. The reason for setting temp to False is to be able to check using kli after ingestion occurs.

One thing that could be improved is access to the status of credentials not associated with the name in kli. For instance, I can ingest ACDCs unrelated to the AID in question during integration testing, and verification/validation occurs but it's hard to check after the fact that the ACDC is indeed valid, as the command line options only permit access to 'received' and 'issued' credentials, not 'seen' credentials.