MediaPlayer Capability UI

[benfrancis]

As a user I want to control and view the status of a media player so that I can control the playback of audio and video in my home.

[benfrancis]

See also: WebThingsIO/schemas#34

[benfrancis]

This is the most complex schema we have added to the WebThings platform and the number of properties and actions really stretches the current hub and spoke design system for the thing detail UI. In #1629 I experimented with a more custom design for a UI capability which takes a more custom approach (which we ultimately didn't use), but I think this may be case where that is even more necessary.

Below are two rough examples of a UI design which demonstrate the problem and a possible solution. Each design represents nearly all the interaction affordances of the schema proposed in WebThingsIO/schemas#34 (the first just misses out ProgressProperty and the second misses out AlbumProperty).

One issue with this less modular design is that it somehow has to support lots of variations on which properties and actions are supported. Should we try to accommodate all possible combinations in the design, or always show all UI elements and disable the ones that aren't supported. I'd like to try the former, but it may result in a messy looking UI for some use cases.

It would be good to have a responsive UI design which accommodate smaller screens, which the first design obviously wouldn't support well.

But the biggest issue is what happens if a device adds other properties and actions which aren't part of the MediaPlayer schema? It's very difficult to accommodate those in the second design. I think those would either have to be ignored, or we need to add a new "other properties" option to the overflow menu in the detail view to discover them, which adds to complexity.

Thoughts?

[rzr]

What about playlists or browser ?

[benfrancis]

What about playlists or browser ?

That's really a question for the schema. While it's a fine line for several of the properties in the current draft of the schema, I'd say playlists and a media browser definitely tip over into things which are not properties of a physical device.

[madb1lly]

Hi @benfrancis,

Is there any way that a device can provide its own interface? In industrial HMI design these per-device interfaces are called "faceplates", after how the physical interface on instruments is called. The HMI is built in a modular way from a library of symbols each of which represent a real-world thing with various properties, much like WebThings Gateway. The properties are standardised in most cases, or a device may bring its own properties if necessary. The faceplates are standard per-device, and these are all taken from a library so there is some consistency of HMI look and behaviour. Complex devices can provide their own faceplate.
See https://www.qwant.com/?q=faceplate%20hmi&t=images&client=opensearch for some examples.

Whilst I like the hub-spoke design it clearly doesn't work at all for the media player. I like the custom "faceplate" you've made for the media player. I think if there is a device that uses more than on capability or adds extra properties then that device should have a way to provide its own faceplate.

Generally however I think the hub-spoke design works very well and is very responsive to smaller displays. Please don't throw the baby out with the bathwater 😉

Cheers 🙂

[benfrancis]

Is there any way that a device can provide its own interface?

Yes, absolutely. A web thing can link to its own web UI using a link relation in the links section of its Thing Description:

  {
    "rel": "alternate",
    "mediaType": "text/html",
    "href": "/things/lamp"
  }

This is then rendered as a link icon which can be clicked to access the custom web UI.

This has been supported since WebThings Gateway 0.5 but unfortunately few web things currently do it and it's non-trivial for add-ons to implement. At one point I envisioned this would be the way that most devices would work and the default automatically generated UI would be a fallback for devices which don't provide their own UI. So far that hasn't turned out to be the case.

Media players would be a good use case for this feature so that they can provide their own optimised UI for their particular feature set.

In industrial HMI design these per-device interfaces are called "faceplates", after how the physical interface on instruments is called. The HMI is built in a modular way from a library of symbols each of which represent a real-world thing with various properties, much like WebThings Gateway. The properties are standardised in most cases, or a device may bring its own properties if necessary. The faceplates are standard per-device, and these are all taken from a library so there is some consistency of HMI look and behaviour. Complex devices can provide their own faceplate.

This does indeed sound very much like capability schemas, with the option to provide a custom web UI.

[madb1lly]

Hi @benfrancis,

So if things can already provide a link to a custom UI then isn't that the solution? Or did you want to build in to the Gateway a default UI for MediaPlayer?

Either are fine. But if your main concern is to try and create a UI which addresses all possible uses of the MediaPlayer capability then I think that may be trying to do too much. If a thing fulfils the standard MediaPlayer capability and does nothing more then your default will suffice; if the thing isn't a perfect match then a custom UI should be provided.

And maybe I've still not full understood problem!

Cheers 🙂

[benfrancis]

@madb1lly wrote:

So if things can already provide a link to a custom UI then isn't that the solution? Or did you want to build in to the Gateway a default UI for MediaPlayer?

Both. I'd like a useable default UI which covers the core use cases, and the ability for web things to provide their own custom UI which is more tailored to a particular device.

if your main concern is to try and create a UI which addresses all possible uses of the MediaPlayer capability then I think that may be trying to do too much. If a thing fulfils the standard MediaPlayer capability and does nothing more then your default will suffice; if the thing isn't a perfect match then a custom UI should be provided.

Exactly. This issue is about what the default should be.

[bewee]

Couldn't we just provide the media player UI as some sort of link relation template and leave the thing view in its bubble design as it is?
I mean, of course, every add-on could implement an own linked custom player view. But that would a lot of (mostly duplicate) effort, and I also think that usability would suffer from many differently styled pages. I really like how the UIs of all things fit together so well.

[benfrancis]

@bewee wrote:

Couldn't we just provide the media player UI as some sort of link relation template

What do you mean by link relation template?

and leave the thing view in its bubble design as it is?

This is what we ended up doing with the Thermostat capability and is an option for MediaPlayer too, but the large number of properties and actions is really stretching the existing design system and doesn't work well on small screens.

[bewee]

but the large number of properties and actions is really stretching the existing design system and doesn't work well on small screens

Totally agree with that

What do you mean by link relation template?

As an add-on developer, I would love to only have to add the @type capability 'MediaPlayer' to my device, and have the gateway generate a link to the prefabricated media player view for my device.

This separate page would look like the the bottom-right one of your 2x2 mockup from above, whereas the thing view would remain as in the top-right (which is almost as it is now). This would allow for all sorts of additions to the predefined schema, but still give a more convenient way of controlling the device in most cases.
(Hope this made my thoughts a little more clear)

[benfrancis]

Ah I see. I would like to avoid having two default user interfaces for a media player capability as I think that adds a lot of unnecessary complexity to the UI. In the longer term I'd be interested in exploring a different design approach for the gateway UI which does a better job of balancing more task-focused user interfaces with the extensibility of the current hub and spoke design pattern to ideally have the best of both worlds.

In the meantime if we take the approach in the second mockup which breaks with the current design pattern, I think we either need to also implement some kind of "other properties" UI to access other custom properties, or accept that if the user chooses media player as a device's primary capability when adding it to the gateway then they can't access any other custom properties of the device.

[bewee]

Alright!
I would love to add the MediaPlayer capability to the frontier-silicon adapter. However, this one requires an OnOffProperty, which is (and I guess shouldn't) be part of the generic schema, as well as a mode property and some mode-specific properties. But without the on/off property, one cannot turn on the device, so it would be practically useless; Similar applies for mode and its specific properties. I guess the Sonos? source property poses about the same problem. Therefore, I would prefer a solution where custom properties can be easily accessed as well.

In general, I think some sort of "extended" thing overview would be practical. This would prevent bloating the thing view with (seldom used) properties and actions, while still keeping their advanced functionality which can also still be used in rules for instance. I can imagine this being useful for way more things than just media players (e.g. putting hue/saturation/brightness of things that also have a colour property there sounds reasonable to me).
Maybe a little offtopic (Please tell me if this doesn't belong here, it just came to my mind): Since the removal of the visibility flag, frontier-silicon devices have to remove and re-add properties depending on their current mode (see image above). This helps making the thing view less bloated, but also breaks rules using (at this moment non-existent) properties. Moving these sort of properties to such an "extended" section may be a better solution here, too?