[d3d11] Baldur's Gate 3 Modding Toolkit Blank Sub-Windows

[Cr-nch]

Although 3D rendering works correctly for some sub-windows in the toolkit, the main game window remains blank, using the previous frames' buffer information to fill it in, resulting in a blank sub-window. The game also does not take mouse input when attempting to interact with it.

On my hardware, a black window appears in the top left of the screen which has the same size as the main game viewport. This box strangely seems to capture input and focus of the main game window, or be completely ignored depending on which application I place underneath it.

Software information

Baldur's Gate 3 Modding Toolkit, running by instructions found here as the program does not launch otherwise: https://discord.com/channels/98922182746329088/1282031634954326066/1282031634954326066

System information

• GPU: Radeon 7700S
• Driver: amdgpu
• Wine version: Using Proton 9.0-3, and hence wine-9.0.
• DXVK version: Proton 9.0-3, d0ea5a4 dxvk (v2.4-40-gd0ea5a4a)
• Proton Version: 9.0-3
  Note: Multiple versions of Proton, including various proton-tkg and proton-GE builds, and Proton Experimental have the same issue.

Apitrace

I couldn't get an Apitrace as it entirely broke the application, not even making it to the loading screen, I can try to get them again if needed.

Log files

steam-2934770.log
Please ignore the crash at the end, it happens every time I quit the application.

Screenshots

The black box does seem to actually be there to render something, as this showed up once while I was doing some experimentation:

[Cr-nch]

I decided to see if RenderDoc could help diagnose any issues, it basically has just confirmed that, yes, the game is rendering correctly at some point, and it also takes only keyboard input (mouse is necessary to actually do anything meaningful).

[doitsujin]

Does this work on current master? We implement sequential swap effects there which might be part of the problem.

Could also be that the thing needs partial presentation implemented, which we do not support currently.

Could also be an entirely different problem.

[xStiven]

For what is worth dude from BG3 tool thread said "I used the wine-tkg version that has the child window rendering fix (or whatever its called), then used winetricks and set the renderer as vulkan". I tried tkg but the app wont even launch for me.

[Cr-nch]

For what is worth dude from BG3 tool thread said "I used the wine-tkg version that has the child window rendering fix (or whatever its called), then used winetricks and set the renderer as vulkan". I tried tkg but the app wont even launch for me.

I had issues with getting it to work, if you get to the loading screen and it crashes after "flushing dispatcher" it's an issue related to installed fonts because of some Microsoft runtime that looks for some specific default fonts.

I've found running winetricks fontfix in the prefix tends to fix that issue.

[Cr-nch]

Does this work on current master? We implement sequential swap effects there which might be part of the problem.

Could also be that the thing needs partial presentation implemented, which we do not support currently.

Could also be an entirely different problem.

To a very very weird extent. With the tooltips fix I tried building from the current commit and it still won't display anything in the main game window, the black box is still present in the top left corner of my screen. Other 3D rendering works:

Without the tooltips fix (the one where you disable hardware rendering for the UI, re-enabling this runs everything through dxvk) I managed to get the game to work with mouse input after closing the other windows and dragging the editor around on my screen which cleared some of the garbage data. Huge progress but I'm also so confused. I'll try get a screen recording of what happened.

[Cr-nch]

Apologies for the fairly grotesque capture, but I think it illustrates my point well enough. Moving the window (seems more erratically the better) somehow manages to clear some of the screen and makes it interactable.

Screencast_20241018_091056.webm

[xStiven]

Does this work on current master? We implement sequential swap effects there which might be part of the problem.
Could also be that the thing needs partial presentation implemented, which we do not support currently.
Could also be an entirely different problem.

To a very very weird extent. With the tooltips fix I tried building from the current commit and it still won't display anything in the main game window, the black box is still present in the top left corner of my screen. Other 3D rendering works:

Without the tooltips fix (the one where you disable hardware rendering for the UI, re-enabling this runs everything through dxvk) I managed to get the game to work with mouse input after closing the other windows and dragging the editor around on my screen which cleared some of the garbage data. Huge progress but I'm also so confused. I'll try get a screen recording of what happened.

Did you tried to follow suggestion from the dude on the discord? I see in your OP post that you tried all sorts of wine flavors but did you tried to set render to vulkan for the BG3 Toolkit prefix? Im asking because, im trying to figure it out, too. If you tried then we can cross-out, that possibility.

[Cr-nch]

Did you tried to follow suggestion from the dude on the discord? I see in your OP post that you tried all sorts of wine flavors but did you tried to set render to vulkan for the BG3 Toolkit prefix? Im asking because, im trying to figure it out, too. If you tried then we can cross-out, that possibility.

I already do have it set to Vulkan, yes

[Blisto91]

I am not 100% certain that this is a dxvk issue, especially since it behaves the same with wined3d.
On Windows the game view can show fine with dxvk

Screenshot

An extra note about dxvk on Windows is that if you minimize and open the toolkit Window again then the game view will for some reason switch to D3D9Ex instead of D3D11 and showing black before crashing.
Though i wouldn't necessarily chain that together with the issue on Linux as with such a complex app there is bound to be some dxvk issues on Windows. Also on Linux i can never even get the hud to show in the game viewport.

[eszlari]

Maybe this Wine MR (merged in 9.18) fixes this: https://gitlab.winehq.org/wine/wine/-/merge_requests/6467

[Blisto91]

Vulkan child Window rendering is already implemented in Proton so i doubt it

[eszlari]

Maybe the upstream implementation is more robust?

[Cr-nch]

For what it's worth, keeping hardware acceleration on (which means the UI is rendered with dxvk) does cause other issues seemingly unrelated to the game not showing. But it's still hard to confirm what exactly is causing this

I have found some other weird rendering bugs related to the 3D windows, that could probably be related to the wine issue (at the end of the clip, moving the windows to the left of the screen causes them to move away). This video also better showcases the workaround.
https://github.com/user-attachments/assets/0f4748fa-ced5-4129-8a02-1bbcc11cdfc7

I'll see what I can do about building wine from source, though it's probably good to note that I have tried proton--tkg which does (afaik) use more up to date Wine versions.

[eszlari]

I'll see what I can do about building wine from source, though it's probably good to note that I have tried proton--tkg which does (afaik) use more up to date Wine versions.

They merged this about a month ago: Frogging-Family/wine-tkg-git#1257

You could also try the Wayland driver, Wine 9.19 has this MR: https://gitlab.winehq.org/wine/wine/-/merge_requests/6323

[Cr-nch]

They merged this about a month ago: Frogging-Family/wine-tkg-git#1257

Tried a proton-tkg version built 2 hours ago, still has the bug. Had no luck getting the wayland driver working on it either so I'm now going to try running the toolkit through pure Wine and dxvk.

[Cr-nch]

Should probably just mention, in relation to Blisto's comment that it also happens in wined3d

While that is true, running the toolkit in dxvk allows for the window shaking workaround (while having other rendering errors), while wined3d (with a more stable general UI) doesn't seem to have any way to fix it. I'll try experiment with wined3d to see if I can get it working there or not to get something more conclusive though.

I also think it's quite strange that the black box moves when the window is moved in specific ways. In fact, I was able to get the box completely off screen and the game window was rendering perfectly normally, and was functional. It seems that whatever portion of the top left of the screen (an invisible rectangle bounded by the game window's size) is covered by the box shows nothing.

Also update on trying to run it through wine separately, I'll admit I'm fairly new to linux and the translation layer scene and have no idea what I'm doing beyond getting wine 9.19 installed and running. I'm a bit low on free time these next few days but I'll keep at it though. Worse comes to worst I can still use the workaround.

Edit: Got it loading with the x11 driver, Wayland just outright crashes. It's quite slow. But the black box now occasionally displays the game editor, but it doesn't update the displayed frame, input is taken to the main game window, but only keyboard, and it seems to have been a one-off of sorts (?). The new wine child-window rendering code is definitely having an effect, so it likely is a wine issue of sorts. Probably an important note that I've now found, the workaround to get the game to work does not seem to work with this set-up.

Edit 2: Okay after removing overrides for dxvk on the wayland drivers, I get a window to open, it locks up immediately, so that's a non-starter.