Clarion OS Communication Architecture

[bytemaster]

The purpose of this discussion is to explore the evolving design for Clarion OS message passing and work with the community to identify the best approach given the goals. Before getting started on Clarion, lets address the "goto" approach: DHTs.

Why not use a DHT (Distributed Hash Table) such as IPFS?

This is a question that comes up frequently in various forms. Based upon the other discussion threads people frequently reference projects such as Holochain or Berty and others. The first thing that I look for when reviewing these existing projects for ideas is the use of a DHT. At first glance a DHT seems like the "perfect" data store for a peer to peer network; however, I have some serious concerns over how they perform in practice.

1. content must be continually rebroadcast
2. it is difficult to prevent abuse
3. user nodes end up storing / distributing data from untrusted sources
4. there is asymmetry in resource usage among users

Using IPNS to update dynamic content is incredibly slow but to publish and then to fetch. This means that IPFS can be used as a long-term cache of content but is not really suitable for interactive sites.

The conclusion that can be drawn from this is that DHTs and IPFS based solutions are barking up the wrong tree in terms of delivering a scalable, high-performance user experience. In a certain respect they are "logically centralized" in that everyone is part of a single namespace / routing system and everyone is expected to talk to everyone. This generates a large number of outgoing connections and can overload NATs which have a limited number of ports.

Scuttlebutt / Patchwork

Another existing system is known as Patchwork aims to allow you to connect to friends without relying about central servers. It is an electron application and my initial experience was slow and buggy.

Clarion OS

Clarion is being designed with a mobile-first approach for the simple reason that the vast majority of social media interactions occur on a mobile device. It is also being designed as a Progressive Web Application (PWA) so that App stores cannot censor it. These two considerations have a major impact on design. All things must grow organically and start from a single node and expand as new users are invited. What follows below is a user story of how the network grows from a single user to fully distributed.

Everything starts with a single "node", which we will call Eden, owned by a single "identity" who we will call Adam. Adam is a user and maintainer of Eden. Adam is lonely, so he creates an invite and gives it to a beautiful lady he met online, her name is Eve.

Eve visits eden.com (hosted by Adam) using the invite link and is asked to create an "identity" by generating a private key and providing some profile information. Eve now becomes a user on eden.com which she accesses with her browser. She is also automatically following the identity of Adam and Adam is automatically following the identity of Eve. Any message Adam or Eve post will be uploaded to eden.com because they are both users of eden.com.

Eve decides she likes Clarion (as hosted on eden.com) and installs it as a PWA on her phone. Everything Adam and Eve post is replicated to her phone which maintains a local database used to render the interface.

After a while Eve makes friends with Cain and Abel who both send her an invite. Loading the invite into her PWA she connects to cain.com and Abel.com (by DNS or by IP). The PWA will then upload her profile and post history to Cain.com and Abel.com and she they will be following each other automatically.

At this point in time Eve gets to see all of the cat and dog pictures posted by Cain, Abel, and Adam. If Eve likes something Adam posts then Even will forward Adam's message on to Cain and Abel along with the message that she "liked" it. At his point Cain may choose to follow Adam. This will result in a message that Cain sends to Eve. From this point on every time Eve gets a message from Adam, she will automatically forward it on to Cain (assuming Cain hasn't already received Adam's message from another source).

Everything Eve posts is uploaded to every node on which she is a user.
Everything Adam posts is pushed to Eve because Eve is following Adam
Everything that Eve receives from Adam is forwarded to Cain because Eve knows Cain is following Adam
Everyone knows that Eve is a user on every node (so far).

Cain knows about the Identity (profile) of Adam (because Eve share his cat picture and he is following Adam), but Adam is not a user on Cain's node. Only users can establish a connection to nodes.

Even if Eden.com goes away (the server that originally provided the PWA), Eve can still post and interact with Cain and Abel.

While the Clarion PWA can only be installed from a domain with a valid HTTPS certificate, it can communicate with any node it can connect to over websocket HTTPS connections.

At this point Eve decides she likes Clarion and wants to install the desktop version on her home computer. This is an electron application. Upon opening the electron application the first time the app will ask her if she wants to import a profile or create a new one. Since Eve already has a profile on her PWA from Eden.com, she asks to import her profile. She will be given an "invite" link which she will load into Eden.com on her phone. Her PWA application will then connect to her Electron app via WebSocket (connection info provided in invite link) and her electron application will sync her account to her desktop computer. Eve will then have to enter her password in the electron application in order to utilize the electron interface.

At this point Eve's phone is connected to eden.com, cain.com, and able.com, and her own personal desktop electron application. Her electron app will then connect to eden.com, cain.com, and able.com because Eve is a user on those nodes. The next time Adam posts a cat picture it will automatically be sent to her desktop and forwarded on to Cain even if she doesn't have her phone open.

Suppose Noah is friends with Cain and has given Cain a user account on Noah.com. When Cain.com receives a message it will automatically forward it to the mirror provided by Noah. If Cain likes Adam's cat picture as well then Noah could see it and decide to start following Adam.

The next time Adam posts a cat picture, it will flow from Eden.com -> EveDesktop -> Cain.com -> Noah.com.
If Eve happened to be on her phone, then it might flow from Eden.com -> EvePWA -> Cain.com -> Noah.com

To the extent possible Cain.com will de-dupe messages so Noah.com only gets it once and hopefully EvePWA doesn't need to send it to Cain.com if Cain.com already received the message via EveDesktop.

Any time two nodes connect they synchronize any unseen messages. This means that if Cain.com goes down for a while, Noah might not get Adam's cat picture until Cain.com comes backup and synchronizes with one of Eve's computers.

QUIC Connections

Full nodes hosted in the cloud or as part of the Electron application will support QUIC and NAT traversal. This means eventually most people with a desktop computer can connect directly to each other.

WebRTC

Peers can also use WebRTC to connect to each other through a coordinating server.

The next post will explore private messaging.

[noypi]

You lost me when you said you post message to eden. And you failed when you wanted to sync to cain. Didnt you know cain can be the evil cia,.

[bytemaster]

What is your concern? Eve knows exactly who everyone is. And there is no danger in sharing info as nothing is intended to be secret.

[noypi]

Its logically centralized, same with fb, why not send only to adam

[vladtr]

I ran fidonet node before and this is way more efficient approach. It should work without affecting mobile data limits too much. Most challenging problem appears to be communication channel itself - carriers (ATT, Verizon, etc) that can block unwanted IP's/traffic.

[dschlossman]

Similar architecture to this?

OrbitDB is a peer-to-peer database meaning that each peer has its own instance of a specific database. A database is replicated between the peers automatically resulting in an up-to-date view of the database upon updates from any peer. That is to say, the database gets pulled to the clients.

This means that each application contains the full database that they're using. This in turn changes the data modeling as compared to client-server model where there's usually one big database for all entries: in OrbitDB, the data should be stored, "partitioned" or "sharded" based on the access rights for that data. For example, in a twitter-like application, tweets would not be saved in a global "tweets" database to which millions of users write concurrently, but rather, each user would have their own database for their tweets. To follow a user, a peer would subscribe to a user's feed, ie. replicate their feed database.

https://github.com/orbitdb/orbit-db/blob/master/GUIDE.md

[noypi]

This better, the said architecture above sends message to server. I think orbitdb will only need eden for signalling, not storing. Eden is the example node above.

[bytemaster]

Thanks for sharing OrbitDB, I took a look at the documentation and concluded the following:

1. Utilizes IPFS which means DHT which means slow lookup for peers
2. Utilizes JSON in the DB which is also slow
3. Loading an individual "tweet" database seems to be an "all or nothing" requiring a full sync before read
4. I assume it is relying upon a localhost IPFS server which would be required to allow "pinning" and publishing

Compared to Clarion design, which will sync messages from most recent to oldest (because most recent activity is more relevant) and which doesn't require that my node sync all of your tweets for the UI to update.

All of that said, I think there is probably something here that can be used for long-term persistence and for people that are able to run a full clarion node and full IPFS node.

[vaultec81]

Thanks for sharing OrbitDB, I took a look at the documentation and concluded the following:

1. Utilizes IPFS which means DHT which means slow lookup for peers
2. Utilizes JSON in the DB which is also slow
3. Loading an individual "tweet" database seems to be an "all or nothing" requiring a full sync before read
4. I assume it is relying upon a localhost IPFS server which would be required to allow "pinning" and publishing

Compared to Clarion design, which will sync messages from most recent to oldest (because most recent activity is more relevant) and which doesn't require that my node sync all of your tweets for the UI to update.

All of that said, I think there is probably something here that can be used for long-term persistence and for people that are able to run a full clarion node and full IPFS node.

1a. It does use DHT for initial rendezvous, but after creating connections with nodes interested in a database the DHT doesn't need to be used. I believe bitswap has functionality to ask directly connected peers prior to asking the DHT.
2a. I wouldn't consider JSON to be slow, inefficient would be better. Protobuf/Cbor encoding would be much better.
3a. This is true for the CRDT log. However, whether the database entries are actually stored on the CRDT directly or through the use of a linked hash makes the difference between loading up all entries on the database or only the ones necessary. This logic would be pushed onto the database abstraction instead of orbitdb itself. The CRDT log is important for history tracking, ensuring there is no conflicts when multiple peers are modifying a dataset and most importantly ensuring the database is the same across all participating nodes. Maybe a hybrid between CRDT and a lighterweight syncing system could be used? I think it's best to have a solid core CRDT/verifiable log (this is also important for add/removing write access to the dataset) along with something more agile but not as secure.
4a. You can run a javascript version of IPFS in browser which removes the need for a localhost go-ipfs instance.

[noypi]

4. Makes it logically decentralized.

[miron]

Naturally storage requirements will grow with time, service worker in particular on iOS could be a bottleneck. How would something like IPFS pinning work on Clarion OS? Different users would like to keep different data, so unpinned data would have to be kept anyway, even more when original authors disappear.

[bytemaster]

Data would be stored in the browser's / PWA indexDb, so this would be a storage limit. A CPU limit would come into play only when a single person follows a lot of people doing a lot of activity.

[fonty1]

Hey @bytemaster, I think there is an opportunity for layer 0 protocols to work together on this problem. We (Streamr) are in our third year building towards a similar technical specification to Clarion. Our contribution is a layer 0 decentralised P2P network designed on a pub-sub architecture. No consensus layer, and blockchain agnostic, but we require a blockchain to incentivize the nodes to contribute their bandwidth/storage.

We've designed the system to be optimized for performance and scalability. If you reach out to me over email - [email protected] I will setup a call with our Network team to discuss the sort the sort of engineering questions you might have at the moment.

[bytemaster]

Fascinating. How does your system work from a browser?

[hpihkala]

Fascinating. How does your system work from a browser?

@bytemaster there's a JS implementation of the Streamr node, so it can (soon) run in the browser natively. The network uses WebRTC datachannels to relay messages to other nodes in a per-topic P2P topology, as WebRTC datachannels work both in browsers as well as node.js. QUIC is an upcoming alternative transport, but it may be a bit too immature at the moment.

Previously we actually used websockets to communicate data between nodes, making the nodes backend-only, but right now we are in the process of implementing WebRTC datachannels for the very reason of being able to form true P2P connections from, with, and between browsers. So that's why the "soon" above.

There's also an alternative setup which is sometimes desirable for browsers or other types of light/thin clients (low-power devices etc.): publishers/subscribers can also interface with an external (as in, running somewhere else than the application process) node over an off-the-shelf protocol such as MQTT, HTTP, or websocket. In this integration pattern, the node translates traffic to/from that off-the-shelf protocol and the Streamr protocol. Here, the cryptographic keys are at the node (so you interface your app with your own node / a trusted node), and the application remains very thin as all the magic and complexity of the P2P network and cryptography happens at the node. This setup obviously works also with browsers right now.

What you need/propose with Clarion sounds quite similar to the Streamr protocol and its goals, so very happy to connect and swap notes!

[kryptokurrency]

What are your thoughts on RadixDLT?

[bytemaster]

Looks like a blockchain, which is not what Clarion is doing.

[vaultec81]

A few questions:

1. How does this stack up against something like ActivityPubsub (as seen in use by peertube/mastodon/matrix)? What's the differences, advantages and disadvantages of both systems?
2. Reliance on a set of federated servers. How well does it handle against multiple servers going down or being unavailable? Is it possible to operate a node completely independent of a federated node?
3. Reliance on DNS. Since this system appears to be a federated, what would stop a country or ISP from blocking the DNS servers to multiple federated nodes. This may seem like a non issue but still is important in limited edge cases.

[bytemaster]

To the largest extent possible Clarion is being designed to not rely upon DNS. The one place where DNS is required is for Progressive Web Application. If everyone used the full desktop client or an App was allowed to be in the various app stores then no DNS would be required.

[jbuck84]

@bytemaster, which issues with Nostr are you most concerned about that Clarion can improve upon? I don't think you have explicitly detailed this, but apologies if I missed it - github conversation isn't the easiest to follow. Thanks.

[vaultec81]

@bytemaster What you describing is awfully similar to the solid project... While it may have it's fair share of issues, it seems conceptually similar to what is being discussed.

[bytemaster]

@vaultec81 thanks for the suggestion, unfortunately solid is "federated". Solid Pod uses a unique ID, authenticated by a decentralized extension of OpenID Connect. This means you don't own your ID unless you run your own OpenID server (something most people aren't able to do). Furthermore, it doesn't propagate/broadcast messages.

[bytemaster]

@jbuck84 looking over Nostr I have the following critiques:

1. I think performance of JSON will limit scale after a while
2. It is almost too data unaware
3. the division between "client" and "server" is too broad

In effect, Nostr is both immature and overly simple protocol. That said, it does follow the principles of Clarion very well. If it were a more widely accepted standard I could be tempted to build upon it.

[kimkha]

Serious question: How can Clarion avoid the irrelevant messages?

Take Facebook as an example, if Eve follows Adam and Adam posts a message that Eve doesn't want, she can mark as "Hide this post". In future, the similar messages from Adam may be ignored from Eve's device, but not the important updates from Adam. Of course Eve can read all messages from Adam if she likes.

I think Facebook pre-process the message before sending it to New Feed of anyone. Is it possible for the filter to be implemented by ClarionOS?

[jbuck84]

@kimkha in regards to your question “Is it possible for the filter to be implemented by ClarionOS?” the answer is ‘yes.’ Users will have many options for filtering, if they host code that supports it. They may have to build it themselves if the conglomerate isn’t incentived to offer it as a standard.

[kimkha]

alright. That means it's hard for the ordinary person to use it, since it's not a plug and play thing. I think at some point someone must take this seriously and implement. Because if this ClarionOS is aiming to take advance over the big social networks such as Facebook, Twitter,..., there will be a time that a single user follows more than a thousand and got flooded by spam and toxic messages from everywhere. Unless that user has a huge time with nothing to do, they may remove the app immediately.

My point is: having this by default is essential to grow the product in real life.

[miron]

You actively follow more than thousand users on Social media? Then you need a break from SM to not suffer from digital dementia and the filter is called unfollow.
According to Dunbar, our neocortices are too small to maintain more than ~150 cohesive relationships, and that includes offline friends:

https://en.m.wikipedia.org/wiki/Dunbar%27s_number

[kimkha]

Come on, you didn't know about that? I saw many people follow thousand users including news agencies, their idols, celebs, fan pages,... I know people have a limit of their cohesive relationships, but on social media people tend to follow more than that limit. And that's just normal.

I just want to ask if ClarionOS is actually worth for that use case or not. And looks like nobody aware of it, so I'm fine, I don't see any value of Clarion like you do, and I won't join with you guys when the network grow fast in future.

In my opinion, the social network can't let people control what to see more and what to see less is the social network that forces people to use as their way, not as the user's way. Users will find they don't have control at all, and they will abandon, not leave just abandon, the social network.

[kimkha]

Also, I think Clarion will be good to be a messenger or slack competitor if the founders don't take the idea of filtering. And that is just too good to be a success.

[kcoder2016]

I think, the rom space of Phone would used up quickly! How to resolved it !