Library Forwarding: Add support for 32-bit Vulkan

[neobrain]

Overview

This brings together virtually every feature added to thunkgen over the past year to add preliminary support for running 32-bit applications with Vulkan calls forwarded to the 64-bit host driver. This moves us to Phase 4 of the Library Forwarding Roadmap:

	X11 (64-bit)	Wayland (64-bit)	Wayland (32-bit)	X11 (32-bit)
no forwarding	☑️ 2022	☑️ 2022	☑️ 2022	☑️ 2022
only Wayland forwarding	N/A	☑️ Phase 1	☑️ Phase 3	N/A
Vulkan	☑️ 2022	☑️ Phase 1	✅ Phase 4	Phase 7
zink on Vulkan	☑️ Phase 2	☑️ Phase 2	Phase 4	Phase 7
OpenGL	☑️ 2022	Phase 5	Phase 6	Phase 7

The incomplete but large subset of the Vulkan API covered here is sufficient to run zink:

Since X11 cannot be forwarded in 32-bit applications yet, this support is limited to Wayland only.

Implementation

Object handles

Vulkan structs that only contain object handles usually have compatible layout between 32-bit and 64-bit, but sometimes they need special treatment. This is because Vulkan distinguishes between "dispatchable" handles and "non-dispatchable" handles.

Most handles are non-dispatchable (e.g. VkPipeline, VkFramebuffer): They're represented as uint64_t and hence don't need any special handling. Dispatchable handles (e.g. VkDevice, VkQueue) are pointer-sized and hence need to be zero-extended. For structs containing dispatchable handles, this is done via custom repacking.

Furthermore, each dispatchable handle type is annotated as fexgen::opaque_type, which allows functions that take handle parameters directly to be forwarded automatically.

Vulkan's extension mechanism: pNext

The vast majority of struct types in Vulkan are extensible, since they contain a pNext pointer to other Vulkan structs (which in turn may also contain such a pNext pointer). This poses a major challenge for 32-bit support, since this pointer immediately breaks data layout compatibility even in otherwise trivially compatible member arrangements. FEX's forwarding code for Vulkan hence makes extensive use of custom repacking, and it's combined with a simple macro to automate it to some extent.

// interface definition
template<> struct fex_gen_config<&VkFenceCreateInfo::pNext> : fexgen::custom_repack {};

// Host.cpp
// Expands to fex_custom_repack_entry/exit
VULKAN_DEFAULT_CUSTOM_REPACK(VkFenceCreateInfo)

Arrays of structs

The Vulkan API uses pointers-to-arrays in some places:

// Reads an array of VkSubmitInfo
VkResult vkQueueSubmit(VkQueue queue, uint32_t submitCount, const VkSubmitInfo* pSubmits, VkFence fence);

// A struct containing an array of bindingCount bindings
struct VkDescriptorSetLayoutCreateInfo {
    VkStructureType                        sType;
    const void*                            pNext;
    VkDescriptorSetLayoutCreateFlags       flags;
    uint32_t                               bindingCount;
    const VkDescriptorSetLayoutBinding*    pBindings;
};

If the 32-bit data layout of the element type matches the one on 64-bit, there's nothing we need to do. However if it does differ, we have to manually repack each element individually. thunkgen won't directly help here, since it cannot distinguish a pointer-to-array from a simple pointer-to-object.

For function parameters, this is done using fexgen::ptr_passthrough:

VkResult fexfn_impl_libvulkan_vkQueueSubmit(VkQueue queue, uint32_t submit_count,
                                            guest_layout<const VkSubmitInfo*> submit_infos, VkFence fence) {
  auto HostSubmitInfos = RepackStructArray(submit_count, submit_infos);
  auto ret = fexldr_ptr_libvulkan_vkQueueSubmit(queue, submit_count, HostSubmitInfos.data(), fence);
  delete[] HostSubmitInfos.data();
  return ret;
}

For structs, custom repacking is used instead:

void fex_custom_repack_entry(host_layout<VkDescriptorSetLayoutCreateInfo>& into, const guest_layout<VkDescriptorSetLayoutCreateInfo>& from) {
  default_fex_custom_repack_entry(into, from);
  into.data.pBindings = RepackStructArray(from.data.bindingCount.data, from.data.pBindings).data();
}

bool fex_custom_repack_exit(guest_layout<VkDescriptorSetLayoutCreateInfo>& into, const host_layout<VkDescriptorSetLayoutCreateInfo>& from) {
  delete[] from.data.pBindings;
  return false;
}

In both cases, the helper function RepackStructArray does the actual legwork of repacking (also considering recursing into any potential pNext pointers).

Testing

Two programs have been verified to work with the lavapipe Vulkan driver:

• vkcube
• Super Meat Boy running via zink: SDL_VIDEODRIVER=wayland EGL_PLATFORM=wayland MESA_LOADER_DRIVER_OVERRIDE=zink FEXInterpreter ./SuperMeatBoy

TODO

[ ] Verify forwarding works on hardware drivers, too

[Sonicadvance1]

Looks reasonable to me even with the bit of debug logging and fetching of symbols each time an entrypoint is called.
Needs #3500 merged first to ensure we don't regress user's behaviour.