Support smooth window resizing on Windows

[raphlinus]

One of the goals I set in druid was to smoothly resize windows, meaning that the window resizes at 60fps (or higher, if the monitor supports it) and that on each frame the contents of the window are drawn to match the size of the window. One way to verify this is to resize by dragging the left window frame and looking at whether the right side of the window judders. Ideally the lag is low as well. This turns out to be shockingly difficult to do, as I'll try to explain a bit in this issue. Now that we're making druid cross-platform, I'm reconsidering whether to base window creation on winit, and this question is a bit of a showstopper. I'm filing this issue largely to determine whether winit can be improved to accommodate this requirement, or whether we should continue using our own codebase.

The crux of the issue is that to get smooth resizing, you need to draw updated content to the redirection surface within the WM_SIZE handler, ie before it returns. Looking at winit's code for this, it seems like it doesn't do any drawing there, but rather sends a message to another thread, which may do the drawing asynchronously. It seems like this architecture is fundamentally incapable of achieving smooth resize; there is no guarantee when the drawing will happen, and even if it's fast, there's fundamentally a race between when drawing happens and when Windows internally processes the completion of WM_SIZE.

The rathole goes much deeper. I said "redirection surface" above, but drawing into a redirection surface is not recommended, rather it's recommended to use a swap chain with a flip-sequential present model. I did a lot of experimentation and basically came to the conclusion that flip modes are incompatible with smooth resizing. However, I found that a hybrid approach does work, and pretty well - you dynamically switch between using a swapchain most of the time, and drawing to the redirection surface while in an active resize. Switching between the two requires synchronous handling in WM_ENTERRESIZEMOVE and WM_EXITRESIZEMOVE, similar to the synchronous handling in WM_SIZE above. Feel free to look at the code I ended up with.

I'm very willing to work with winit on this, as I'd prefer the rust ecosystem to be on one common codebase, but I'm not sure how to do it without major architectural rework. Thus, a main goal in filing this issue is to open a conversation. There are other tricky issues such as how to accommodate a latency wait object (not yet done in druid), but I think it's useful to start here.

[raphlinus]

Update: I've been reading through existing issues, and I think this has nontrivial overlap with #459. That looks like it's in progress, an associated PR is waiting to be merged. So maybe this isn't as intractable as I was fearing.

[Osspial]

I can confirm that this will get fixed when #459 #638 lands on master. Smooth resizing actually used to work fine on Windows (see #250), but we had to revert that PR because it was causing some Winit programs to straight-up freeze - a problem much worse than having resize artifacts.

[goddessfreya]

PRs have landed, so closing. Reopen if not resolved.

[mitchmindtree]

Re-opening this as it seems #638 does not address @raphlinus' original issue (see this comment).

The crux of the issue is that to get smooth resizing, you need to draw updated content to the redirection surface within the WM_SIZE handler, ie before it returns. Looking at winit's code for this, it seems like it doesn't do any drawing there, but rather sends a message to another thread, which may do the drawing asynchronously. It seems like this architecture is fundamentally incapable of achieving smooth resize; there is no guarantee when the drawing will happen, and even if it's fast, there's fundamentally a race between when drawing happens and when Windows internally processes the completion of WM_SIZE.

@Osspial is it still the case that resizes are sent to another thread? At a glance, I can see here that it does look like the event is queued and is not handled before the event returns, but I'm not familiar enough with the windows backend to know what goes on after. Is event handling still happening on another thread? Or is the event processed synchronously later on in the same thread?

[Osspial]

@mitchmindtree The Windows backend no longer has any threads, so no. send_event will always dispatch the event immediately unless doing so would invoke undefined behavior (in which case the event will get queued and dispatched later). However, I'm pretty sure a downstream user needs to directly call into the Windows API themselves to trigger the queuing behavior, and as such smooth resizing should happen most of the time.

Also, @raphlinus' assessment of the solution here wasn't entirely correct. You don't want to do the redrawing in the WM_SIZE handler - instead, Windows will dispatch a WM_PAINT message before rendering the resized window and you want to handle redrawing there. That's exposed in the Winit API through the RedrawRequested event, and handling that properly should lead to smooth resizing.

[raphlinus]

There are two issues here, one with threading, which I am satisfied has been resolved, and one being synchronization between swapchain presentation and window resizing. The latter is a very deep and complicated issue, and I will post a blog today describing it in more detail, though xi-editor/xi-win#17 does set out the essential point. I'd love to be proved wrong, but I'm pretty sure winit is incapable of smooth resizing while using a swapchain. No need to reopen the issue, though, I'm fine with druid being the only UI toolkit that gets this right :)

Edited to add what is hopefully less snarky and more constructive commentary: I would of course be thrilled for winit to get this right, but I am not comfortable depending on it for druid, and the interaction in this issue makes me more confident in that decision. Trying to get stuff perfectly synchronized with the underlying window manager is, as I've mentioned, a very tricky problem. Experimenting with it in xi-win (now druid) has required accessing the platform at the lowest levels. My plan going forward is to try to communicate what I've learned (through both blogging and having working code), then it's up to other projects, winit included, to apply that or not.

[Osspial]

@raphlinus It looks like you're using Direct2D, which I have no experience with and cannot comment on. However, using OpenGL, tests on my machine resize smoothly with no client-area artifacts if drawing is done in RedrawRequested. This example in glutin redraws in that event, and it's the example I just tested, so I'd love to see if you run into any artifacts I haven't noticed when you try it (diagonal tearing especially, since I personally haven't noticed it on my computer).

[raphlinus]

Right, I should probably clarify that this is an issue with DirectX (I'm doing Direct2D, but same issues with Direct3D) and one of the newer flip-based presentation strategies, which is strongly recommended for performance.

I haven't dug too deeply into OpenGL, but suspect they're doing the equivalent of DXGI_SWAP_EFFECT_SEQUENTIAL. This goes to the redirection surface (so is an extra blit) and has other performance problems - notably the swap_buffers call is blocking.

Clearly my use case is a bit different than the mainstream of winit users, and that in and of itself is not a problem.

[raphlinus]

The blog post is now up: https://raphlinus.github.io/rust/gui/2019/06/21/smooth-resize-test.html

[Osspial]

Hmm, that's pretty interesting. I wasn't aware of those flip strategies!

I just looked into the Windows OpenGL functions myself, and frustratingly it doesn't seem like they give the same level control over how stuff gets presented to the screen as DXGI does. That's probably at least somewhat intentional on Microsoft's part, but it doesn't make me any less unhappy.

However, that isn't Winit's problem, since Winit explicitly doesn't handle rendering context or swapchain creation. I see no reason why changing the render target to the redirection buffer on resizes (as you describe in your blog post) couldn't be done with a DirectX context rendering to a Winit window - it's just that doing that is the job of higher-level libraries that build atop Winit. Indeed, it might be possible to implement in Glutin, if we implement a DXGI-based OpenGL context creation method, which is an avenue that's worth exploring especially if it can bring significant performance improvements at essentially no cost.

That approach may be possible on NVIDIA systems, since NVIDIA exposes a DirectX/OpenGL interop extension. I can take a look into seeing if that's possible, since you've managed to hardcore nerd-snipe me on this :P.

[fu5ha]

Note that vulkan does expose what it calls 'present modes' but they are essentially what the swap modes of DX are, I believe.

[Osspial]

An update on where DXGI interop ended up going: DirectX context creation works entirely OK on Winit windows, so I'm confident that the smooth resizing mechanics @raphlinus described will work on Winit windows. OpenGL interop for rendering onto the swapchain would likely work if the relevant APIs did what they said they did, but I've spent a day trying to wrangle them into behaving and I don't see that work going anywhere. If anyone wants to take a crack at it, I'll post my WIP source code when I get home.

As for smooth resizing without DirectX - the Vulkan solution @termhn described is probably your best bet.

[Osspial]

Here's the link, if anyone wants to take a stab at it. It's based on this C++ example.

[mgood7123]

i am also suffering with this problem in DX12 Win11, its annoying how MS window manager doesnt sync with DX swap chain internally which is what i was suspecting

(WM may be rendering signifigantly higher than our draw rate which may be causing the laggy resize aka black bars, thus it can resize the window at a signifigantly higher rate than we are otherwise able to draw it)