WAMP IDL support

[oberstet]

Ok, this is an advanced, more complex topic. Let me see if I can find a way to explain it properly. Bare with me, you'll need some patience;) Let me assure you: it's worth it. You will see the light;) Hopefully. Maybe also wrong to explain all that here on the issue tracker for connectanum-dart. It'll be long, multiple comments. Anyways, here we go.

Intro and the Why?

WAMP was designed along a dynamic typing discipline from the beginning (eg even in WAMPv1). This was a deliberate and conscious decision, and it has served us (the WAMP community) well .. I'd say.

What that means is: WAMP application payload in RPC call arguments/results and PubSub events can transport any args (positional payload) and kwargs (keyword payload).

Full run-time dynamic typing has advantages (very flexible, no-frills get started right away, etc), but - as with anything - there is a price to pay.

The price to pay comes in different shapes.

Robustness

Robust app code will need quite a lot of run-time checks. eg say you have a registered WAMP procedure proc add2(a, b) that returns the (numeric?) sum of a and b (two positional args). what if the caller calls your proc with a=="foo" and b==666?

JS, being a duck, will happily return foo666. most of the time this won't be matching what neither the caller nor the callee had in mind.

so you would continue adding "run-time checks" to the implementation of add2. as in: "if type(a) != int then raise .."

This can get boring quite quick, and of course is error prone. did you really check everything? what if the caller send three positional args? etc etc

Code scalability

With one developer doing both sides of a WAMP API, eg the app code for the caller and callee, or the publisher and the subscriber, everything is in the head of 1 person, and fixing stuff and making sure it matches up is straight forward and natural. You can do it.

Now, what if you have 10 developers? In one organization this can get tricky to manage already. If you have developers of different parties involved, it quickly turns into a big problem and time sucker. Was it a,b being ints or also floats? Ah, you changed it, so now it's a,b,c?

How do you even communicate what one side (eg callee) did intend anyways? Is it communicated in a way that even gives the other side (eg caller) a chance to do the right thing?

Documentation

Ok, so now your problem becomes: add2 (the callee side) must have proper docs. Sure, dream on! ;) The problem is manyfold: can you even precisely write natural language docs? even if so on day 1, now add2 changes. Surely, updating the docs is forgotten. Now the situation is even worse: there is docs, but they are wrong. So a caller that wants to do its best can't even. And so on. Wrong docs is worse than no docs.

How to ensure that a proc that claims in the docs it takes exactly 2 positional args of type int does indeed at the technical level?

Statically typed host languages

Dealing with dynamic types and data in statically typed host languages always incurs some form of impedance mismatch. Kinda expected.

Also, when working in such a language, you naturally want native language static types for your app level objects.

Eg say you have

public class Person {
    public String firstname;
    public String lastname;
    public String department;

    public Person() {
        this.firstname = "unknown";
        this.lastname = "unknown";
        this.department = "unknown";
    }

    public Person(String firstname, String lastname, String department) {
        this.firstname = firstname;
        this.lastname = lastname;
        this.department = department;
    }
}

in your app.

Now, wouldn't it be great if you could do (pseudocode):

Person someone = await session.call("com.example.get_last_customer");
print(someone.lastname)

Performance

WAMP supports multiple, pluggable serialization formats (JSON, MsgPack, CBOR, ...) and routers transparently convert between those (roundtripping). Very useful, all cool. We have that today. However ..

• In Crossbar.io, CPU cycles spent in dealing with this are top 1 and dominate (besides WebSocket low-level processing)
• WAMP clients also have to serialize/deserialize the whole payload even if they only need access to a tiny part of the payload (eg thing of a single attribute in a deeply nested dict)
• further, this is not only about CPU cycles, but also (and even more important) about memory and memory pressure. Dynamic typing is .. dynamic also allocation wise. By nature.

---

There are more problems .. I stop here;)

[konsultaner]

I do absolutly understand these issues as I have experienced them too. Do you remember asking me if you are interested in supporting my new protocol idea? My idea does not yet overcome all of these thoughts but maybe we should talk about it. I bet you would like some of my ideas 🤔 maybe not.. but thinking is never wrong 😇😅

[oberstet]

sure, talking is always good! also general protocol ideas.

having said that:

• we have a Flatbuffer schema based IDL for WAMP that can be used with WAMP as is
• that is implemented in ABPy, ABJS and ABJ
• we also have WAMP itself in Flatbuffer define (though that is only experimental in ABPy)
• my main problem is my work load / lack of time;)

just wanted to "manage expectations" ...

---

now, rgd this issue: above is just comment 1 .. the actual solution, describing what we have, I want to do that in follow up comments .. it's just that I have to make progress on other things first. one of which is unblock collegues and provide them with WAMP + Dart + WAMP-cryptosign. Once that other PR is merged and released, that is done (off my table).

in the meantime, let me just link #24 (comment)

[oberstet]

if you want, we could have a 1:1 call coming days .. just let me know, ping me here, or via Signal or Keybase https://keybase.io/oberstet .. we can have a call via Signal or our Jitsi meet ..

[konsultaner]

if you want, we could have a 1:1 call coming days .. just let me know, ping me here, or via Signal or Keybase https://keybase.io/oberstet .. we can have a call via Signal or our Jitsi meet ..

Perfect, lets do that. What time and day is best for you?

[oberstet]

thursday - saturday, between say 12:00-24:00 =) pretty much as you like within that ..

[konsultaner]

right :D

[konsultaner]

@oberstet Coming back to this issue. We've had a lot of talks about this topic and others kind of belonging this.

1. Serialization is and has always been a big issue in general -> IDL + FlatBuffers would do a great job, issues are
   1. performance
   2. multiple serialization types
   3. header + body share the serialization type (opaque payload, E2E, binary data [+json], streams, zero copy, ...)
   4. object serialization issues, dart has no reflections, reflections are slow, code generation sucks, ...
   5. ...
2. Would be great to have descriptive language to also define the message flow like mermaid just not for imaging but for messaging.

Having and IDL, a MSC-language and a Router-Client-Interpreter would enhance a lot of things. This would also solve translation issues between WAMP and other protocols.

• For the IDL I think Flatbuffers is great
• For the MSC-lang I think some stream dsl would probably do a good job
  • https://www.react.uni-saarland.de/publications/ffst16.pdf
  • https://mermaid-js.github.io/mermaid

Do you have any idea where to put this discussion? maybe you have a better place then in this dart client issue?

[oberstet]

so yeah, in fact I've been working on this recently, here are a couple of first examples and screenshots (autobahn has a CLI tool now to dump schemas)

https://github.com/crossbario/crossbar-examples/tree/master/payload-validation

cd crossbar-examples/payload-validation/catalogs/example
make build
xbrnetwork describe-schema --schema=schema/example1.bfbs

cd ~/scm/crossbario/autobahn-python/autobahn/xbr/test/catalog/
xbrnetwork describe-schema --schema=schema/wamp-meta.bfbs

---

this will allow us to:

• API sharing ("API catalogs")
• app payload validation
• docs generation
• client bindings code generation
• run-time type reflection

rgd where to discuss: I'd say the WAMP repo, as issues .. IMO

btw, Mermaid looks quite nice .. visually! need to check it out. in general, an IDL that also adds MSC language: I think there are 2 levels here:

1. the messaging patterns given by WAMP itself in terms of raw WAMP messages sent and received
2. the app level messaging patterns in terms of WAMP calls and events

both are interesting. even though I am a bit hesitant rgd practical value. eg 1) is "merely" helping WAMP router implementors. moreover: any MSC that is correct and complete wrt the WAMP spec could (in principle) be compiled right down to a router implementation. the MSC is just yet another language. This is interesting definitely .. however, probably mostly academic.

now 2) for user apps: it'll depend .. some app contexts like safety-critical and embedded would profit I guess. at least, it's a design approach there .. compile your whole app from state machines described in MSC ...

[oberstet]

forgot another link: https://github.com/crossbario/autobahn-python/tree/master/autobahn/xbr/test/catalog/src

pls have a look at the schemas there, in particular:

• wamp-auth: (dynamic) WAMP authentication and authorization API
• wamp-meta: WAMP meta API
• wamp-control: node control API .. crossbar.io specific .. currently

Now, here is a head spin:

Imagine we'd add reflection procedures to the WAMP meta API .. like "describe_interface()" or "describe_type", and then call:

@wamp.register("wamp.catalog.describe_interface")
def describe_interface(ifc)
   return ..

describe_interface("wamp.catalog.describe_interface")

this will describe the types of the reflection procedure to describe .. at run-time =)

it's a feature of a proper type system: be able to express the type of itself. fully reflected.

[oberstet]

btw, the fbs idl schemas work for payload validation regardless of the serializer used by clients! eg this is a json based client .. note the "validate" etc log lines

[oberstet]

fwiw, I've started to write spec text here wamp-proto/wamp-proto#405 .. but it's WIP .. once I've added everything, I'll try to gather more contributors/implementors feedback ...

[oberstet]

@konsultaner rgd an MSC on top of IDL defined types for app payloads in calls and events: what I'd be looking out for in an MSC language or similar is:

being able to express a message transmission (a link between state machine state nodes) that allows to specify:

• a message transmission latency
• actual a latency distribution ..
• .. with a non-zero probability for complete message loss (equivalent to infinite latency)

this would allow us to model the behavior of WAMP routers and clients wrt to timeouts and retries

[konsultaner]

@oberstet I'm 100% with you! I'll keep my eyes open. But as you said this may be an academical task...