Implementing Mouse Picking in RAYX-UI: Preference for Color Picking Method

[Atraxus]

We are considering enhancing RAYX-ui's interaction capabilities by introducing a mouse picking feature to allow users to select and interact with objects in the 3D scene. After evaluating two prominent methods for mouse picking - geometric ray casting and color picking using a second frame buffer - I am leaning towards implementing the color picking method. This decision is based on several factors that seem to align well with our application's requirements and the Vulkan-based architecture of RAYX-ui.

Why Color Picking?

Color picking involves rendering the scene to an off-screen frame buffer where each object is drawn with a unique color that encodes its ID. This method offers several advantages:

• Performance: Leveraging the GPU's parallel processing capabilities provides fast and efficient object identification, which is crucial for maintaining high performance in complex scenes.
• Simplicity: The implementation process is generally more straightforward, avoiding the complex mathematical operations required for geometric ray casting.
• Scalability: It scales well with the complexity of the scene, as the performance impact is primarily on the GPU's rendering capabilities.

Considerations for Implementation in RAYX-UI:

• Vulkan Setup: Implementing color picking will require some modifications to our Vulkan setup, including the creation of an additional frame buffer and render pass specifically for object ID rendering.
• Shader Adjustments: Our shaders will need to be modified to output a unique color based on the object's ID, requiring changes to shader files and possibly the shader management system.
• Input Handling: The existing input handling system will need to be extended to read from the off-screen buffer upon detecting mouse clicks, translating pixel colors back into object IDs.

While color picking seems to offer a more straightforward path for Vulkan code changes and aligns with our performance goals, it's important to recognize its limitations, such as the dependency on the off-screen buffer's resolution and the additional resources required. However, given the nature of RAYX-ui and its rendering pipeline, these trade-offs appear manageable and justified by the benefits in performance and implementation simplicity.

[Atraxus]

Transforming RAYX-UI into a beamline design tool is currently blocked by a refactor in our backend. There is a refactor happening to work on the inner representation of beamline objects as DesignObjects.