usb: device_next: new USB Video Class (UVC) implementation

[josuah]

Dependencies:

• drivers: video: introduce "GET" sub-operations #78603 replaced by something else to come
• drivers: video: allow allocation with a header preceding the buffer #79168 not required anymore
• video: emul: virtual driver for an imager and RX engine #79482

This is a preview of an USB Video Class implementation for Zephyr that was only tested on custom devices insofar.

Proper Zephyr samples will be provided on upcoming commits. The API is simply to submit frames to the UVC device like to any Zephyr video device.

There is an unsolved challenge around the Video API: there is no set_format because the Zephyr application cannot decide what the host uses, only get_format for what the host does support. But there is a missing video API to allow the driver to warn the application about a forced format change requested by the host. I thought of maybe reusing set_signal() to also warn about format changes and not just buffer events.

	.get_format = uvc_get_format,
	.get_caps = uvc_get_caps,

I will now work on building examples for existing Zephyr devices, as this was built for a custom USB3 board.
Here is the devicetree configuration used insofar:

#include <zephyr/dt-bindings/usb/video.h>

&zephyr_udc0 {
	uvc: uvc {
		compatible = "zephyr,uvc-device";

		port {
			uvc_ep_in: endpoint {
				remote-endpoint-label = "mipi0_ep_out";
			};
		};
	};
};

The sizes and FPS are to be selected by the developer. The USB descriptors get configured as described above, and the host will request a particular format. Once that is done, the USB Video Class driver can let the application know which of these was selected through the video.h get_format() API.

This is still a draft PR, but I am grateful for comments and suggestions. I am willing to do the work of refactoring this as much as needed.

[EDIT: see this comment for latest description]

[josuah]

I am grateful for the work on the USB and Video stacks of Zephyr, as well as the entire Zephyr tree, on the shoulder of which this is built.

[josuah]

Force push:

• fixes to make it compile with west build -b frdm_mcxn947/mcxn947/cpu0 (a board I happened to have, open to suggestions)
• added another commit (with I hope proper attributions!) to introduce fragments, which UVC and this implementation support.

[josuah]

Force-push:

• Add a sample to illustrate the usage. It is not yet tested but compiles.

[XenuIsWatching]

Where are these values coming from?

[josuah]

These are chosen arbitrarily, and I need to think about what should I do here: pick reasonable defaults? Deduce from other values? Let the user figure out by exposing a devicetree option?

[josuah]

In future work, it will be possible to ask the sensor for VIDEO_CID_PIXEL_RATE in combination with video_bits_per_pixel() to generate this field.

[XenuIsWatching]

I believe UVC can be BULK or ISO.... but I only see BULK supported here. Do you plan to have a way to select which type of EP?

[josuah]

I forgot to mention that only BULK is supported through this early version. I will need to spend more time investigating the device_next APIs for how ISO is implemented.

I cannot guarantee ETAs as I would do this feature on free time, but will add it to the roadmap: this will be needed at some point.

[josuah]

Very grateful for your reviews @XenuIsWatching I will force-push with the changes as soon as I get a chance to do so.

[josuah]

Force-push:

• control a selected source = <&dev>; device from the UVC stack directly
• wait that a format got selected before queuing video buffers to the USB
• use "discrete" frame intervals (FPS) values, which seemed to help getting it to work under Windows
• USB3CV compliant descriptors

[josuah]

force push: rebased on main

[josuah]

force push: changed the way descriptors are declared and introduce video controls at the descriptor level (no support for controls commands yet)

[josuah]

force push: implemented the UVC controls for the supported Video class controls.

I did not test this yet with the samples, but on the internal fork, we could get an IMX219 sensor with exposure and gain control from the host, format selection at startup (but not runtime, see [1]).

I believe that the last step for me is to test the sample on several boards that support device_next and fix the CI.

[josuah]

Known limitations:

[1]: there is currently no <video.h> API to let the application know that a format change occurred. For now, the sample waits that the host makes an initial choice, and does not reconsider its format selection.
A trade-off between supported feature and sample complexity.

[2]: there is currently no <video.h> API to let the video device communicate their min/max/step, so 0 is always picked as min, INT32/16/8_MAX as max, and 1 as step. Maybe the solution is to define an unit for each of the controls and have every sensor map this to their local definition. For instance, 1 µs for exposure, 1/1000 for gain, etc, and have the max value implicit. DONE

[3]: The "default" value is an arbitrary value of 1 instead of querying the controls of each sensor. DONE

[4]: There is no device const struct uvc_conf and only struct uvc_data, which wastes precious memory. DONE.

[5]: There are missing implementations for selector unit, extensions unit, encoding unit. TODO.

[6]: UVC introduces dependencies between some controls, i.e. auto-expopsure needs to be off for exposure to be accepted. TODO.

[7]: UVC has "error" control type reporting the last error. DONE

[8]: Only a single endpoint and output/streaming interface is supported per UVC interface. Composite devices (multiple UVC classes per device) are used instead for supporting multiple video streams per device. DONE

[9]: No documentation outside the devicetree bindings. DONE (no documentation needed anymore as no configuration required).

[10]: Supports custom header size, but not passing custom header data yet.

[11]: Still image capture (capturing one frame at full resolution) not supported.

[12]: USB3CV compliance tests not all passing since last rebase. TODO.

[13]: Announcing different resolutions/FPS for different connection speed not supported.

[14]: Asynchronous controls (the host setting a control, and a notification interrupt alert of completion) not supported.

Supported features:

[A]: Class API and enumeration

[B]: Custom control chains descriptors built from the devicetree layout Not supported by Linux

[C]: Selection of which controls to expose to the host via devicetree toggles Now controlled by the drivers.

[D]: Per-control entity tuning Sending all the controls requests to the first driver of the pipeline.

[E]: Handling of control commands end-to-end from host to Zephyr video device

[F]: Zephyr native Video API that allows to enqueue/dequeue frames like any other Zephyr video device.

[G]: Supports fragmented frames as discussed in #66994 and #72827

[H]: Can configure pixel format and resolution on the devicetree Done automatically by asking the video drivers.

[I]: Supports querying the min/max/current values of every controls end-to-end from host to Zephyr video driver.

[ghost]

@josuah Thanks for considering my comments. They are all strictly optional of course (as you have treated them) because I know virtually nothing about UVC. Happy that there were a few useful things among my misunderstandings. ;-)

The alternative would be to have the application build the descriptors with macros:

I understand your dilemma. These are application configuration properties which we do not (yet) have a good place for (although we're actively working on it). From a dogmatic viewpoint some of these settings are obviously not hardware properties (as they change with application requirements not with hardware). But as we don't have a good alternative for such settings right now, you're not the first to encounter such a dilemma.

I'd be inclined to have those properties (lowercased and as closely modeled around the standard UVC data structures) in DT for the moment being as we'll have to migrate several DT properties to software settings anyway once the config infrastructure is in place. This will be easier if we migrate from DT rather than from macros.

@decsny In the end it is of course entirely your call. Just thinking out loudly. I'll be presenting my software settings poc in the upcoming arch WG probably. This would be one of the first things I'd write a configuration handler for.

[josuah]

Given #77638 it might be interesting to provide a temporary solution that upgrades gracefully to a dedicated configuration system.

	Linux	Current Zephyr PR	Ideal
Configuration time	❌ run-time	⭕ build-time	build-time
Descriptor configuration	❌ manual	❌ simplified	nothing (guessed from drivers)
Configuration storage	❌ RAM	⭕ flash	flash
Use of devicetree	⭕ hardware-only	❌ mixed	hardware-only
USB knowledge required	❌ medium	❌ medium-low	none
Video pipeline supported	❌ limited	⭕ almost any (only one output terminal, WIP)	any
Video interface	⭕ native /dev/video	⭕ native Video API	native

Linux also uses an extra userspace library to abstract the USB protocol which fills ⭕ this grid almost completely.

Here is the Rosetta Stone between Linux, UVC and this PR:

• Left: Linux, which mirrors the elements from the standard as-is.
• Right: current PR with blanks when something is handled with macrobatics or USBD stack.

See the configuration

uvc.0
|- streaming/
|  |- class
|  |  |- h
|  |  |- hs
|  |  |  '- h
|  |  |- fs
|  |  |  '- h
|  |  |- ss
|  |  |  '- h
|  |  '- ...
|  |- color_matching
|  |  '- ...
|  |- header
|  |  '- h
|  |- mjpeg                             format_mjpeg {...};
|  |
|  '- uncompressed                      format_uncompressed {
|     |- u
|     |- bAspectRatioX                      /* not used */
|     |- bAspectRatioY                      /* not used */
|     |- bBitsPerPixel                      bits-per-pixel = <2>;
|     |- bDefaultFrameIndex
|     |- bmInterfaceFlags                   /* not used */
|     |- bmaControls                        /* not used */
|     |- guidFormat
|     | [v4l2-ctl -d /dev/videoX ...]       fourcc = "YUY2";
|     |
|     '- 360p                               360p {
|        |- bmCapabilities
|        |- dwDefaultFrameInterval
|        |- dwFrameInterval                     max-fps = <30 60>;
|        |- dwMaxBitRate
|        |- dwMaxVideoFrameBufferSize
|        |- dwMinBitRate
|        |- wHeight                             size = <480 360>;
|        '- wWidth
|                                           };
|                                       };
'- control/
   |- h
   |- ct.0                              uvc0_ct0: camera_terminal {
   |  |- bmControls [1]                     control-exposure-time-absolute;
   |  |                                     control-ae-mode;
   |  |                                     ...
   |  |  [v4l2 event]                       control-target = <&imx219>;
   |  '- baSourceID
   |                                    };
   |- pu.0                              uvc0_pu0: processing_unit {
   |  |- bmControls [1]                     control-gain;
   |  |                                     control-hue;
   |  |                                     control-saturation;
   |  |                                     control-brightness;
   |  |                                     ...
   |  '- baSourceID                         source-entity = <&uvc0_ct0>;
   |     [v4l2 event]                       control-target = <&imx219>;
   |                                    };
   |
   '- ot.0                              uvc0_ot0: processing_unit {
      |
      '- baSourceID                         source-entity = <&uvc0_pu0>;
         [v4l2 event]                       control-target = <&mipi0>;
                                        };

[1]: The USB standard is tricky about it: the control IDs and the control enable flags are shuffled for just a few items, so easy to get mistaken. In the current PR, the bitmap is decoded, every field described, and WIP mapped to native Zephyr video controls.

[2]: UVC controls (i.e. tune the exposure) are not yet supported in the current Linux user-space utility, but are still possible with a custom format.

The current Zephyr implementation brings a good leap in reducing the amount of duplicated configuration between the drivers and the protocol. It is possible to reduce it further if using run-time configuration:

• Using video_get_caps() to populate the list of formats
• Using video_enum_frmival() to populate the FPS
• Using video_get_ctrl(VIDEO_CTRL_GET_DEF) to populate the list of controls available
• Using compatible = to try to automatically configure UVC controls according to the real devicetree pipeline topology (probably error prone)

[EDIT: wording, typos]

[tmon-nordic]

I'd be inclined to have those properties (lowercased and as closely modeled around the standard UVC data structures) in DT for the moment being as we'll have to migrate several DT properties to software settings anyway once the config infrastructure is in place. This will be easier if we migrate from DT rather than from macros.

@decsny In the end it is of course entirely your call. Just thinking out loudly. I'll be presenting my software settings poc in the upcoming arch WG probably. This would be one of the first things I'd write a configuration handler for.

My first impression to something called "software settings" is that the component would be handling the non-volatile settings. But here the UVC bindings, just like UAC2 bindings are pretty much read-only descriptions. These do describe what is possible and can offer a choice, but the choice is volatile. The choice (e.g. selected sample rate in UAC2 case) lifetime is entirely controlled by host (can be abruptly terminated by user pulling the USB cable) and essentially is no longer valid after USB bus reset.

[ghost]

I'd be inclined to have those properties (lowercased and as closely modeled around the standard UVC data structures) in DT for the moment being as we'll have to migrate several DT properties to software settings anyway once the config infrastructure is in place.

My first impression to something called "software settings" is that the component would be handling the non-volatile settings.

Not exactly. Some consensus seems to build in the arch WG around a combination of read-only (build-time), provisioning (end-of-production-line/OTA) and runtime configuration closely related to macros (for build-time) and the settings subsystem (for provisioning/runtime and optionally build-time, too). The three use cases tied together through a common data model. The best metaphor I encountered so far would be a combination of sysfs and configfs which is more or less what the settings subsys API already exposes.

But here the UVC bindings, just like UAC2 bindings are pretty much read-only descriptions. These do describe what is possible and can offer a choice, but the choice is volatile.

Still seems to be a match for the planned config system (see Linux' use of configfs to define USB gadgets on-the-run) but I'm not deep enough into USB to know for sure. I'd say, though, that keeping the same compromise as the one found for UAC2 would be ok from my perspective. I looked at those bindings and I didn't see anything especially problematic with those if we ensure that we re-use every bit of existing structure that we can find in Linux and elsewhere.

[josuah]

force-push: several features, still WIP, and a few known bugs (cannot play again after closing the client application)

• Only a single compatible = "zephyr,uvc-device"; introduced.
• Commpletely zero-conf: query video devices to generate the format descriptors, simple topology instead of full graph.
• Introduction virtual drivers for an image sensor (imager) and MIPI RX peripheral, which provide a test pattern image source, replicating the real driver configuration of Zephyr.
  • video: emul: virtual driver for an imager and RX engine #79482
• Splitting video-formats..h from video.h so that they can be #included on the devicetree as well.
• Introduction of video-controls devicetree bindings: optional, default value on the bindings)
• Introduction of endpoint { direction = "in"/"out"; };, needed to configure the USB descriptors.
• It is now single-instance only. To handle multiple streams, the individual streams are declared inside of the unique instance.

The devicetree bindings addition are not USB-specific but USB is the first to need them.

New configuration syntax:

uvc.0                           uvc: uvc {
|                                   compatible = "zephyr,uvc-device";
|
|- streaming/                       port {
|  |- class                             uvc_in: endpoint {
|  |  |                                     remote-endpoint-label = "mipi0_ep_out";
|  |  |- h
|  |  |- hs
|  |  |  '- h
|  |  |- fs
|  |  |  '- h
|  |  |- ss                                                               /* empty */
|  |  |  '- h
|  |  '- ...
|  |- color_matching
|  |  '- ...
|  |- header
|  |  '- h
|  |- mjpeg
|  |  '- ...
|  '- uncompressed
|     |- u
|     |- bAspectRatioX
|     |- bAspectRatioY
|     |- bBitsPerPixel
|     |- bDefaultFrameIndex
|     |- bmInterfaceFlags
|     |- bmaControls
|     |- guidFormat
|     | [v4l2-ctl -d /dev/videoX ...]
|     '- 360p
|        |- bmCapabilities
|        |- dwDefaultFrameInterval
|        |- dwFrameInterval
|        |- dwMaxBitRate
|        |- dwMaxVideoFrameBufferSize
|        |- dwMinBitRate
|        |- wHeight
|        '- wWidth
'- control/
   |- h
   |- ct.0                                                                /* empty */
   |  |- bmControls [1]
   |  |  [v4l2 event]
   |  '- baSourceID
   |- pu.0
   |  |- bmControls [1]
   |  '- baSourceID
   |     [v4l2 event]
   '- ot.0
      '- baSourceID
         [v4l2 event]
                                        };
                                    };

New feature comparison table

	Linux	Current Zephyr PR	Ideal
Configuration time	❌ run-time	⭕ build-time	build-time
Descriptor configuration	❌ manual	⭕ none	nothing (guessed from drivers)
Configuration storage	❌ RAM	❌ flash/RAM	flash
Use of devicetree	⭕ hardware-only	⭕ hardware-only	hardware-only
USB knowledge required	❌ medium	⭕ none	none
Video topology supported	❌ limited	⭕ any (WIP)	any
Video interface	⭕ native /dev/video	⭕ native Video API	native

I hope I could have been introducing this the first time, to save everyone review time.

[josuah]

The intended usage is now:

1. Build a native Zephyr video pipeline, i.e. using a frame counter as output.
2. Enable UVC, it generates the descriptors during init, such as USB controls, formats, frame rates.
3. Connect the output of the existing pipeline (with remote-endpoint) to the UVC video sink like done for any video sink.

[EDIT: updated as per latest commit]

[josuah]

force-push:

• No more devicetree configuration except the standard video-interfaces.yaml include
• A single compatible = "zephyr,uvc-device" used
• Video devices are queried at init to generate USB descriptors.
• Kconfig used to allocate the resources (number of descriptrors, size of variable fields)
• Added Frame Rate control.
• Simpler macros, as Linux only supports flat topologies.

Tested on STM32H743, Raspberry Pi Pico, and WIP FRDM-MCXN947 (UDC error messages... maybe buffer alignment). The only requirement is USB device_next support, as it uses emulated/fake video devices.

west build -p -b mini_stm32h743             samples/subsys/usb/uvc
west build -p -b rpi_pico                   samples/subsys/usb/uvc
west build -p -b frdm_mcxn947/mcxn947/cpu0  samples/subsys/usb/uvc

Known quirks:

• Some deviations taken from clang-format to fit the rest of USB classes
• Previous versions were passing USB3CV, tested on Linux, Windows, Mac OSX, Android. This version not yet.
• On some platforms, opening the feed a 2nd/3rd time does not work.
• BULK endpoints only supported for now, ISOCHRONOUS endpoints will come in future PR.

In hope this version goes in the right direction from previous review, and that it costs few time to reviewers in this form.

[uLipe]

@josuah could you share your test setup for this PR? I'm intending to use my Arduino NIcla Vision for that.

[josuah]

@josuah could you share your test setup for this PR? I'm intending to use my Arduino NIcla Vision for that.

Yes sure! The only requirement is having an USB driver compatible with device-next.
This currently uses a software-based pattern generator, so no external camera needed to reproduce it.
UVC asks the driver what formats it supports and generate the USB descriptors from that, no extra config.

This board uses a similar chip as the Nicla.

west build -p -b mini_stm32h743 samples/subsys/usb/uvc

This shows how to convert an STM32H743 board from using device to using device_next, log through UART rather than USB (to avoid recursive logging). This currently goes over USB2 FullSpeed, I am probably missing something.

Do feel free to discuss this on Discord or this discussion.

[josuah]

force-push:

• address documentation build failure
• experimental support for MJPEG

[josuah]

force-push:

• Fix CI (doc and twister)
• More reasonable documentation
• DNM as waiting two parent PRs

[josuah]

force-push:

• Fix MJPEG support a bit more
• Validation of descriptors with USB3CV (3 minor errors left)
• Now also supports Android
• Rebase on top of post 4.0 main

P.S.: support for other operating system is planned but WIP.