druid-shell roadmap

[raphlinus]

This issue presents the roadmap for druid-shell development over the next few months, and a bit of vision of the longer term plans. A goal of this issue is to clearly define what is in scope and out of scope for this crate.

Current state

Right now, druid-shell is druid-win-shell. It provides window creation, basic mouse and keyboard input, and a small amount of native platform UI services: file dialogs and menus. The drawing surface for a window is (as of recently) piet, which wraps the Direct2D context.

For historical context, druid-win-shell used to be xi-win-shell, which was factored out of xi-win at some point. The original xi-win codebase was monolithic and not factored into layers.

There is basic support for high-dpi, but not dynamic per window dpi switching (the APIs for this are tricky and only exist in recent Windows 10 versions). Getting high-dpi right is a goal.

One particular feature that is perhaps over-developed compared with the maturity of the rest of the crate is smooth window resize. There is a sophisticated hybrid presentation scheme that switches between a swap chain for steady state and drawing to the redirection surface during live resize. There's more discussion of this in rust-windowing/winit#786, one of the issues blocking the use of winit, about which more later.

Near future

We want to move druid to platform independence, and most of the Windows dependencies are in the shell layer. Much of the immediate work is tracked in #13.

A major choice is whether to use winit or to do our own window creation. For Windows at least, I'd like to preserve the existing window creation logic so as not to regress smooth resize, and also so that we can do quantitative measurements. For other platforms, I'm open to the idea of winit, but I'm also open to doing our own. We may have two window creation back-end (probably controlled by feature) for a while until all this sorts out.

Aside from platform independence, some of the features coming up include:

• Incremental present. One of the goals of druid is high performance, and an important piece of that is to only draw what's actually changed on the screen. An important but special case is blinking the cursor in a text editor. On Windows, we want to use IDXGISwapChain1::Present1 for this, and in general provide plumbing for incremental present. Note that incremental present is probably going to be tricky on other platforms; it's not clear that macOS has native support at all (see chromium#474299 for one way to fake this).

• Low latency present. Right now, when animating, we block in the present call until the next frame is available. During that blocking, we can't take input, so it adds to the overall latency. DXGI 1.3 has latency wait objects that can decrease latency.

• IME. The keyboard handling is primitive, and there are already issues like Doesn’t handle astral plane characters xi-editor/xi-win#67. It would be great to have a cross-platform abstraction for IME. Note that this is a hard problem, but also one I've investigated a bit.

• A gfx-hal context. The primary 2D drawing interface is piet, but I'd like to support 3D apps as well. It seems to me sensible to standardize on gfx-hal for this.

I'd like to focus on Windows, macOS, and Linux for the near term. I think it will also be interesting to target web, one of the reasons there is a web back-end for piet.

Longer term future

Longer term, the scope of druid-shell is to provide an abstraction for the interface between applications and operating system services for UI. I can see this including:

• Access to preferences. When the OS provides a preferences mechanism, it's better to use that than inventing our own. This also includes being sensitive to system-wide preferences, including locale.

• Accessibility. Getting accessibility right requires hooks to the native platform, not least of which is respecting preferences (see above). This probably also includes features such as speech.

• Paste buffer. This is particularly important for xi-win, but is generally useful.

• Drag and drop. Another basic and nontrivial UI function.

• More platforms. In addition to the platforms listed above, Android and iOS are very interesting. There is a longer tail of platforms, including Fuchsia, but it's likely serious work on those shouldn't start until there is a body of applications people want to run. A personal goal of mine is Nintendo Switch; it's likely that this work can't be completely open source, but I'd like as much of at as possible to be.

What to do about winit?

Long term, it would be good to merge the window creation and input with winit, so there is one common create for these services. Right now, I'm reluctant to do so, as it would regress performance (and performance is a major goal of druid), and we can probably be nimbler in experimenting and getting changes landed faster.

It's also going to be interesting to see how much the scope of winit increases. There are issues for drag and drop and IME (see also rust-windowing/winit#753), but apparently not preferences, accessibility, speech, or pasteboard. There's a bit of discusssion of native widgets for menus and file dialog but not movement.

Non-druid layers

The broader goal of building cross-platform UI is deliberately factored into smaller crates, in large part to enable a mix-and-match approach. I can see people wanting to experiment with other UI approaches than druid. As druid-shell matures, it will probably become an appealing layer. I want to support, that but not at the cost of bringing in too much premature generality. The main goal of druid-shell is to support the needs of druid.

Things not in scope

The scope of druid-shell includes presenting a drawing surface to the higher layers, but nothing about what's drawn in that surface. So widgets, layout, etc., are not in scope. This boundary may become fuzzy if we want to support embedded native widgets, but this is not foreseen as a major use case. In particular, on macOS and iOS, I see building the UI with druid widgets, rather than native Cocoa.

I've investigated the use of the compositor (see discussion of planeshift in pcwalton's 2019 plans blog. This is a very complex tradeoff, and for druid I'm resolving it in the direction of not integrating with the compositor. It certainly is true, though, that the compositor is the lowest power solution for cursor blinking and scrolling on most current systems. Part of my thinking is that I want to enable hardware overlays, as these are lower latency and lower power on forward-looking systems (DirectFlip). See servo/webrender#3115 for deeper discussion. (I can see why Servo is going this direction, and thus druid-shell likely won't be a good solution for Servo).

Similarly, I'm not planning on exposing the scroll functionality in Present1. This seems to be a fairly narrow Windows feature (though it could be faked on macOS/iOS using the compositor).

If the OS does not directly support menus (Linux doesn't, macOS and Windows do), then it is not in scope for druid-shell to polyfill that. Rather, it reports the lack of menu support to the higher level (druid), and it is up to that level to implement its own menu widgets if not available.

A great many OS services not directly tied to the UI are out of scope, including audio, MIDI, networking, etc. Speech is an interesting question.

It is a goal of druid-shell to be a cross platform abstraction, but that abstraction is sometimes leaky. Instead of trying to be heroic, the general approach is to provide access to platform-specific APIs, so that applications can call in directly (we can use any and #cfg as appropriate to gate this access).

Note: not all these decisions are set in stone, and we can reconsider them based on the needs of apps built on top. But I see deliberately restricting scope (at least for now) as a good way to maintain momentum.

Discussion is welcome.