W3C Compliance

[benfrancis]

Since WebThings Gateway 0.12 was released, the W3C WoT Thing Description specification has become a W3C Recommendation.

There are a number of remaining differences between Mozilla's Web Thing API specification (now maintained here) implemented by WebThings Gateway and the W3C specification, which are documented on our wiki.

We would like WebThings to be compliant with the W3C specification, but there are some challenges in achieving that.

The gateway is both a "producer" and a "consumer" of Thing Descriptions. It is a producer in that it exposes Thing Descriptions for all the devices it manages via its Web Thing API, and it is a consumer in that it can consume thing descriptions from native web things via the web thing adapter add-on.

Making the Thing Descriptions exposed by the gateway should be relatively straightforward as it's mainly a case of syntax changes and optional features need not be supported. There would also be some changes needed to payloads in the REST API to be compliant with the default representation in Thing Descriptions. This would be a breaking change, but currently there are few consumers of this API other than the gateway's own web interface, which could be changed in lockstep.

Adding support for the gateway to consume any W3C compliant Thing Description would be much a more challenging/impossible task because of the lack of constraints the specification imposes on different protocols and serialisation formats which web things can use, and the complex system of declarative protocol bindings inside forms. There are also six libraries in six different programming languages and ten third party libraries in the WebThings Framework which would need to be updated in lockstep with this change in order to maintain compatibility.

Supporting the consuming of W3C Thing Descriptions may become more feasible after the definition of a Core Profile for web things, if that profile defines a single required protocol and serialisation format which conforming web things must support and is more prescriptive about how they are used. The related Web Thing Protocol specification would also help enormously, by defining a concrete protocol binding for HTTP (if not already defined by the Core Profile) and WebSockets, which could guarantee ad-hoc interoperability for any conforming web things.

I therefore suggest that our initial focus should be on making the Thing Descriptions and REST API exposed by the gateway compliant with the W3C Thing Description specification. We can then continue to assess the feasibility of support for consuming W3C Thing Descriptions as the standardisation work progresses.

This work will be tracked by the w3c-compliance label.

As this will be a breaking change, I propose we target this work for a 2.0 release which will also have a number of other major differences from 1.x versions.

[benfrancis]

Note: We should also review the draft WoT Discovery specification to see whether we can align our use of mDNS/DNS-SD broadcasts and whether the proposed Directory Service API could replace or augment the Things endpoint of the Web Thing API.

[fatadel]

Hi, @benfrancis!

I was told by @sebastiankb to help you in making the project compatible with the W3C Thing Description and to start with the python implementation of webthing.
If you have any suggestions or questions please let me know. I will investigate the project in the upcoming days and provide some PRs.

Best regards,
Adel

[benfrancis]

Hi @fatadel, thanks very much for the offer of help!

We had planned to work on the WebThings Framework (libraries like webthing-python) last after first implementing the gateway side, but I guess there's no harm in starting that work now if you're willing to help out! There won't be a compliant WebThings consumer for you to test with yet, but maybe you can use a third party implementation for that?

@tim-hellhake is now the module owner for webthing-python so it's really up to him how to go about this. For the gateway we discussed creating a 1.x branch to continue development on 1.x releases while landing the W3C compliance changes on master with the intention of targeting a 2.0 release. For webthing-python it might be a good idea to start this work on a branch or something? I wouldn't let that decision stop you from forking the repo and getting started though, since we can always test your fork.

[fatadel]

@benfrancis

I have forked the repo and will do my work there. As soon as I have some results, I will provide a PR, and then you or @tim-hellhake can decide how to proceed further with it. So, apparently, I will be approaching you from the opposite direction of the W3C-compliance track 😊
Regarding the consumer to test it out, I think I can create a consumer Thing using node-wot as I'm pretty aware of that library and it's W3C-compliant.

[benfrancis]

Great, thanks @fatadel for starting work on this! You will want to keep track of the discussions under the w3c-compliance tag because there are some tricky design decisions to be made about what features to standardise, modify or drop from WebThings which we will need to keep consistent between WebThings Gateway and the WebThings Framework.

@tim-hellhake Please let us know your thoughts on this, particularly a branching strategy for webthing-python.

[tim-hellhake]

@tim-hellhake is now the module owner for webthing-python so it's really up to him how to go about this. For the gateway we discussed creating a 1.x branch to continue development on 1.x releases while landing the W3C compliance changes on master with the intention of targeting a 2.0 release.

I think we be should do the same for the webthing libraries

Regarding the consumer to test it out, I think I can create a consumer Thing using node-wot as I'm pretty aware of that library and it's W3C-compliant.

Perfect!

[tim-hellhake]

Sorry, I'm a bit lost.
There a so many open issues/PRs 😅
Now that the wot-adapter skeleton is ready, what are our next steps?
Is any of the PRs ready to be reviewed?

[benfrancis]

OK, a snapshot of where I think we are. Here is a W3C compliant version of Example 2 from the Web Thing API specification:

{
  "@context": [
     "https://www.w3.org/2019/wot/td/v1",
     "https://webthings.io/schemas/"
  ],
  "@type": ["Light", "OnOffSwitch"],
  "id": "https://mywebthingserver.com/things/lamp",
  "title": "My Lamp",
  "description": "A web connected lamp",
  "securityDefinitions": {
    "oauth2": {
      "scheme": "oauth2",
      "flow": "code",
      "authorization": "https://mywebthingserver.com/oauth/authorize",
      "token": "https://mywebthingserver.com/oauth/token",
      "scopes": [
        "/things/lamp:readwrite",
        "/things/lamp.",
        "/things:readwrite",
        "/things"
      ]
    }
  },
  "security": "oauth2",
  "properties": {
    "on": {
      "@type": "OnOffProperty",
      "type": "boolean",
      "title": "On/Off",
      "description": "Whether the lamp is turned on",
      "forms": [{"href": "/things/lamp/properties/on"}]
    },
    "brightness" : {
      "@type": "BrightnessProperty",
      "type": "integer",
      "title": "Brightness",
      "description": "The level of light from 0-100",
      "minimum" : 0,
      "maximum" : 100,
      "forms": [{"href": "/things/lamp/properties/brightness"}]
    }
  },
  "actions": {
    "fade": {
      "@type": "FadeAction",
      "title": "Fade",
      "description": "Fade the lamp to a given level",
      "input": {
        "type": "object",
        "properties": {
          "level": {
            "type": "integer",
            "minimum": 0,
            "maximum": 100
          },
          "duration": {
            "type": "integer",
            "minimum": 0,
            "unit": "milliseconds"
          }
        }
      },
      "forms": [{"href": "/things/lamp/actions/fade"}]
    }
  },
  "events": {
    "overheated": {
      "title": "Overheated",
      "@type": "OverheatedEvent",
      "data": {
        "type": "number",
        "unit": "degree celsius"
      },
      "description": "The lamp has exceeded its safe operating temperature",
      "forms": [{
        "href": "/things/lamp/events/overheated",
        "subprotocol": "sse"
      }]
    }
  },
  "forms": [
    {
      "op": "readallproperties",
      "href": "/things/lamp/properties"
    },
    {
      "href": "wss://mywebthingserver.com/things/lamp",
      "subprotocol": "webthing"
    }
  ],
  "links": [
    {
      "rel": "alternate",
      "type": "text/html",
      "href": "/things/lamp"
    }
  ]
}

Note: The one part of this TD which doesn't currently validate in the TD playground is that https://webthings.io/schemas/ is supposed to resolve to a valid JSON-LD resource. That's a minor issue but currently a bit tricky because that URL currently resolves to an HTML page hosted on GitHub Pages which doesn't support content negotation. One option might be to change the URL to https://webthings.io/schemas.jsonld

Summary of proposed changes:

• Includes both "https://www.w3.org/2019/wot/td/v1" and "https://webthings.io/schemas/" contexts, see Change @context of Thing Descriptions #2809
• Links in interaction affordances changed to forms, see Change links to forms in API #2806
• Top level link for the Properties endpoint changed to a form with the readallproperties op, see Change links to forms in API #2806
• Top level link to the WebSocket endpoint changed to a form with a subprotocol of webthing (There may be an open issue here with what consumers will assume the default ops are for a top level form. In practice the WebSocket endpoint may support all op types). See Change links to forms in API #2806.
• Changed mediaType to type in links, see Change mediaType to type for links in API #2808
• Added a data wrapper around the Event data schema, see Add data wrapper for event data schema #2810
• (The top level alternate link for the HTML UI remains as a link)

One thing not shown is:

• Adding prefixes to non-standard members, see Remove or add prefix to proprietary members of Thing Descriptions #2832

In order to make the actual API match this description we would need to:

• Remove object wrappers from API payloads, see Remove object wrappers from API payloads #2807
• Add support for Server Sent Events, see Add support for Server Sent Events #2830

There still are still a few missing features in the W3C Thing Description specification that we need:

1. How to describe the top level Events resource, see Add subscribeallevents op to event affordances in Thing Descriptions #2831
2. How to describe action queues. This is being discussed in How do you cancel or query the state of an action request? w3c/wot-thing-description#302. My thinking for now is that we can just provide a form which links to an Action resource and doesn't try to describe the queue. A W3C compliant consumer would assume that only the invokeaction op is supported. The action queue can possibly remain in place in the gateway's API, just not described by the TD.
3. How to describe the top level Actions resource. There's currently no proposal for how to do this. We could propose an invokeanyaction op, but to be honest we could probably live without this. The Actions resource is only really provided as a convenience and consumers could use individual Action endpoints instead.

@relu91: What do you think of all the above? Would you say your pull request for changing links to forms in #2811 is consistent with this?

[benfrancis]

@tim-hellhake wrote:

Is any of the PRs ready to be reviewed?

@relu91 Other than the CI issues, is #2811 ready to review or do you think there are still parts missing? It's currently still marked as Draft. (That seems like the next step).

[relu91]

I think we are moving in the right direction, #2811 is still a draft cause it does not yet cover:

• Top level link for the Properties endpoint changed to a form with the readallproperties
• Top level link to the WebSocket endpoint changed to a form with a subprotocol of webthing

we can still merge it as it is an open a new PR for these two changes. I think it would be easier to review 🤔 I would like to resolve CI issues before merging though. So that we are sure that these changes do not break any test.

[relu91]

Now that the wot-adapter skeleton is ready, what are our next steps?

We worked a little bit on wot-adapter using my fork as the main repo. I am polishing a little bit the changing, expect a couple of PRs soon. 👍🏻

[benfrancis]

Update on open questions...

1. How to describe the top level Events resource, see Add subscribeallevents op to event affordances in Thing Descriptions #2831

This change has now landed. I've started to implement support for Server-Sent Events in #2830.

2. How to describe action queues. This is being discussed in How do you cancel or query the state of an action request? w3c/wot-thing-description#302. My thinking for now is that we can just provide a form which links to an Action resource and doesn't try to describe the queue. A W3C compliant consumer would assume that only the invokeaction op is supported. The action queue can possibly remain in place in the gateway's API, just not described by the TD.

I've submitted a pull request to add queryaction and cancelaction operations to the Thing Description specification. However, I think this discussion could continue for some time and I don't think we should block moving forwards on that landing. I therefore propose a multi-stage implementation for action queues:

Stage 1
Provide a single Form for each action in Thing Descriptions, which can be used to invoke an action. Modify the underlying invoke action API so that it matches the TD (remove the wrapper from request payloads so it just provides the action input). Leave the action queue API in place, but don't try to describe it in the TD. The TDs will be W3C compliant and existing WoT Consumers will be able to invoke actions. The cancel & query functions of the API will still be in place in case we need them (but they're not actually currently used by the built-in UI, only provided as an API).

Stage 2
Modify the action queue API to match the protocol binding in the Core Profile and add the profile member to TDs to denote that they conform with the profile. The current proposal doesn't actually depend on the queryaction and cancelaction operations, it only requires a single Form in an action to which it then applies a set of assumptions.

Stage 3
Implement the Core Profile in wot-adapter so that the gateway can Consume Web Things which use the Core Profile, and use their cancel and query action features if provided.

Stage 4
Once the debate in w3c/wot-thing-description#302 reaches a conclusion, consider adding additional metadata to the gateway's Thing Descriptions so that action queues can theoretically be supported even by WoT Consumers which don't implement the Core Profile. We may decide not to do this, especially if it looks like WoT Consumer implementations can't actually consume such complex Thing Descriptions in practice.

3. How to describe the top level Actions resource. There's currently no proposal for how to do this. We could propose an invokeanyaction op, but to be honest we could probably live without this. The Actions resource is only really provided as a convenience and consumers could use individual Action endpoints instead.

I've opened an issue to discuss potential new operations for the Thing Description specification to support top level Forms for actions like we now have for properties and events. I think the discussion about what (if any) new operations to add could also take some time. In the short term we may just need to leave these endpoints in place in the gateway's API, but not try to describe them in Thing Descriptions.

[benfrancis]

I think the goals of this issue have largely been met, and it has now been replaced by the following more granular issues to track ongoing W3C WoT compliance on the producer side:

• Epic: W3C WoT Thing Description 1.1 Compliance #3136
• Epic: W3C WoT Profile 1.0 Compliance #3137
• Epic: W3C WoT Discovery 1.0 Compliance #3138

And the separate wot-adapter repo for W3C WoT compliance on the consumer side.

Closing.