Off-chain communication via libp2p

[kwannoel]

In GitLab by @fahree on Jan 23, 2021, 06:20

PoC

- [ ] Do a first pass that communicates via messages on CED and/or the fastest test blockchain we support.

^Fare: It was a good idea when "getting something out of the door ASAP" was the goal, but overall it slows us down.

Barebones

See: #392, for areas to refactor.

Pending review: #398.
This PR supports running the buy_sig interaction locally, over the libp2p channel. It is specific to 2 participants, and it requires users to explicitly declare who starts the interaction.
In the long run, we need to support more than 2 participants, and remove the constraint that users have to explicitly declare who starts the interaction. See the #Fully Featured section below for details for other areas which need to be implemented as well.

• Backend support and CLI to ensure the go-libp2p-daemon and/or JS library is started with the correct parameters (e.g. keypairs).
• Backend support and CLI to send and receive handshakes and agreements.

Fully featured

• Backend support and CLI to send and receive messages between two ETH addresses
• Backend support and CLI to queue, dequeue for active addresses with append-only DB schema
• Backend support and CLI to filter before queuing or select before dequeuing messages for active addresses
• Have a "negotiation" and/or "shopping" UI for negotiating contract terms
  • E.g. someone made you an offer - accept / ignore. Later on we can have reject, spam list.
• Handle exceptions, e.g. cancellations.
• Actually use libp2p for off-chain communication.
• Some way to broadcast availability (future).
• Tag agreement and handshake with uids, such that receiver can identify which agreement and handshake(s) are paired.
• Lock channel buffer (currently not an issue, because only a single daemon writes synchronously).
  Later when daemon starts spawning write processes, we will need this.
  Or, if there are multi daemon's running against a single queue.
• Persist a libp2p stream between participants?
• Enable looking up of contacts table to retrieve participant multiaddr.
  • Provide an entrypoint for users to configure hostaddr.
  • Provide an entrypoint for users to configure hostaddr of peer.
• Add identity multiaddr to glow
  Q: Why not just have a multi_addr field?
  A: During startup we can't supply the multi_addr field to the daemon. It derives the PeerID segment of the multi_addr from the identity's private key.
  We can supply the ip_addr however, to specify where we want to listen from.
  Q: Why not just have an ip_addr field?
  A: Other participants cannot use just ip_addr, they need peerID + ip_addr to communicate through libp2p.
• Refactor identity prompt to occur earlier, since libp2p relies on the identity to peerId mapping to communicate.
  Currently this is done in two places (earlier for libp2p, later for stdio off-chain-channel),
  to support backwards compatibility with the existing integration tests which use stdio.
• Pull in the seckey protobuf format from https://github.com/libp2p/go-libp2p-core/blob/master/crypto/pb/crypto.proto
• Gracefully close connections.
• Ensure ip + port is unused.
• Debug why the buy-sig-libp2p integration test is working locally, but failing in CI.

Overview of interaction via libp2p

Runtime interaction via libp2p (edit here).
Whoever starts the interaction (in this scenario Alice), should already know the multiaddr of the other participant. This information should be in the contacts store.

Internals with libp2p daemon and buffer

Probably not what we want, libp2p does support buffering already. We should just reuse that.
EDIT:
gerbil-libp2p does have buffered streams, but you do not have access to the underlying buffer, only each stream.
Hence you would still need to have a buffer, which the listening thread pushes incoming messages to.
For example if you receive an agreement and handshake,
where the handshake is only used much later in the runtime,
you would store that handshake in a buffer first, and fetch it later when you need it.

So this approach is still needed:

edit here:

[kwannoel]

In GitLab by @fahree on Jan 23, 2021, 06:20

assigned to @Akaa

[kwannoel]

In GitLab by @fahree on Jan 23, 2021, 06:23

Instead of copy/pasting the handshake from one terminal to the other using io-context:terminal, t/buy-sig-manualtest.ss should exchange the handshake via gerbil-libp2p

A followup issue will be negotiating the agreement off-chain, too.

[kwannoel]

In GitLab by @fahree on Feb 15, 2021, 23:57

• Backend support and CLI to ensure the go-libp2p-daemon is started with the correct parameters
• Backend support and CLI to send and receive messages between two ETH addresses
• Backend support and CLI to queue, dequeue for active addresses with append-only DB schema
• Backend support and CLI to filter before queuing or select before dequeuing messages for active addresses

[kwannoel]

In GitLab by @fahree on Feb 17, 2021, 01:58

assigned to @alex413

[kwannoel]

In GitLab by @fahree on Feb 17, 2021, 02:06

unassigned @alex413

[kwannoel]

In GitLab by @ApolloUnicorn on Mar 6, 2021, 01:49

unassigned @Akaa

[kwannoel]

In GitLab by @ApolloUnicorn on Mar 6, 2021, 01:49

assigned to @alexfmpe

[kwannoel]

In GitLab by @ApolloUnicorn on Mar 17, 2021, 19:36

unassigned @alexfmpe

[kwannoel]

In GitLab by @ApolloUnicorn on Mar 18, 2021, 00:01

assigned to @fahree and @Akaa

[kwannoel]

In GitLab by @fahree on Apr 8, 2021, 16:45

changed the description

[kwannoel]

In GitLab by @fahree on Apr 8, 2021, 16:46

changed the description

[kwannoel]

In GitLab by @Akaa on Jun 16, 2021, 07:41

https://gitlab.com/Akaa/glow-example

[kwannoel]

In GitLab by @plotnick on Sep 9, 2021, 02:19

assigned to @plotnick and unassigned @fahree and @Akaa

[kwannoel]

In GitLab by @kwanzknoel on Sep 16, 2021, 02:43

Note:
gerbil-libp2p is built on-top of go-libp2p-daemon which is unmaintained at the moment.

That being said, it should be straightforward to takeover maintenance, as it is just "glue code" around go-libp2p, and go is an easy language to pickup.

We could also consider js-libp2p-daemon, which is actively maintained.

[kwannoel]

In GitLab by @kwanzknoel on Sep 16, 2021, 02:47

Future considerations:

• Handling of exceptions e.g. cancellations
• Handling of negotiations.

[kwannoel]

In GitLab by @kwanzknoel on Sep 18, 2021, 03:37

assigned to @kwanzknoel