Issues working with FilamentCanvas vs FilamentPanel

[kanawish]

Hello, I'm running into an issue trying to get FilamentCanvas to work on MacOS + JVM. Basically, I have a working example where FilamentPanel works fine, but FilamentCanvas, while hitting all the right breakpoints, doesn't show the rendering on screen.

Here's my sample, using Filament mac dist v1.9.9

fun main() {
    println("barKot: Hello world!")

    diagnostics()

    // AWT/OpenGL workarounds
    System.setProperty("sun.java2d.d3d", "false")
    System.setProperty("sun.java2d.noddraw", "true")
    System.setProperty("sun.awt.noerasebackground", "true")
    System.setProperty("sun.awt.erasebackgroundonresize", "false")

    SwingUtilities.invokeLater {
        try {
            awtSandbox()
        } catch (ex: Exception) {
            ex.printStackTrace()
        }
    }
}

private fun awtSandbox() {
    val f = JFrame()
    f.defaultCloseOperation = JFrame.EXIT_ON_CLOSE
    f.layout = BorderLayout()

    f.size = Dimension(800, 600)
    f.isVisible = true

    val t = Thread {
        Filament.init()
        val entityManager = EntityManager.get()
        val engine = Engine.create()
        val renderer = engine.createRenderer()
        val camera = engine.createCamera(entityManager.create())
        val view = engine.createView()
        val scene = engine.createScene()
        val updateSize = AtomicBoolean(true)

        //com.google.android.filament.FilamentPanel canvas = new com.google.android.filament.FilamentPanel();
        val canvas = FilamentCanvas()
//        val canvas = FilamentPanel()
        try {
            SwingUtilities.invokeAndWait {
                f.add(canvas, BorderLayout.CENTER)
                f.background = Color.MAGENTA
                canvas.background = null
                f.addComponentListener(object : ComponentAdapter() {
                    override fun componentResized(e: ComponentEvent) {
                        updateSize.set(true)
                    }
                })
                f.revalidate()
            }
            Thread.sleep(1000)
        } catch (e: InterruptedException) {
            e.printStackTrace()
        } catch (e: InvocationTargetException) {
            e.printStackTrace()
        }

        try {
            val skybox = Skybox.Builder().color(0.1f, 0.125f, 0.25f, 1.0f).build(engine)
            scene.skybox = skybox

            view.isPostProcessingEnabled = false

            view.scene = scene

            // Material work
            val material = engine.loadMaterial("/materials/baked_color.filamat")

            // Mesh work
            val vertexBuffer = engine.setVertexBuffer(triangleVertices)
            val indexBuffer = engine.setIndexBuffer(triangleVertices)

            // To create a renderable we first create a generic entity
            val renderable = entityManager.create()

            // We then create a renderable component on that entity
            // A renderable is made of several primitives; in this case we declare only 1
            RenderableManager.Builder(1)
                // Overall bounding box of the renderable
                .boundingBox(Box(0.0f, 0.0f, 0.0f, 1.0f, 1.0f, 0.01f))
                // Sets the mesh data of the first primitive
                .geometry(0, RenderableManager.PrimitiveType.TRIANGLES, vertexBuffer, indexBuffer, 0, 3)
                // Sets the material of the first primitive
                .material(0, material.defaultInstance)
                .build(engine, renderable)

            // Add the entity to the scene to render it
            scene.addEntity(renderable)
            view.camera = camera

            println("Starting rendering loop...")
            while (true) {
                if (updateSize.compareAndSet(true, false)) {
                    val width = canvas.width
                    val height = canvas.height
                    view.viewport = Viewport(0, 0, width, height)
                    val zoom = 1.5
                    val aspect = width.toDouble() / height.toDouble()
                    camera.setProjection(
                        Camera.Projection.ORTHO,
                        -aspect * zoom, aspect * zoom, -zoom, zoom, 0.0, 10.0)
                    println("VP: " + width + "x" + height)
                }

                if (canvas.beginFrame(engine, renderer)) {
                    renderer.render(view)
                    canvas.endFrame(renderer)
                }
                try {
                    Thread.sleep(50)
                } catch (e: InterruptedException) {
                    e.printStackTrace()
                }
            }
        } catch (ex: Exception) {
            ex.printStackTrace()
        } finally {
            canvas.destroy(engine)
        }
    }
    t.isDaemon = true
    t.priority = Thread.MAX_PRIORITY
    t.start()
}

I might be doing something silly with regards to AWT Canvas usage, I haven't managed to find a working sample using FilamentCanvas so far.

@romainguy pointed me this way, and recommended I ping @bejado

I can share the gradle build files / helper functions as needed. Thank you!

[bejado]

If you could share the gradle files / project, that would be helpful.

[kanawish]

Happy to help, here's a somewhat cleaned-up version of my test code. Left a few notes in the README.md regarding project setup and my system configuration in case it helps.

Let me know if you have any questions.

Thank you.

[bejado]

I'm able to reproduce the issue, but don't have any insights at the moment unfortunately.

[kanawish]

I've tried to look a bit more seriously at what bits of the Filament libraries are being called on underneath. [@romainguy originally pointed out JAWTUtils.cpp and Darwin.mm as of interest]

I tried understanding a bit how the swapChain for FilamentCanvas gets created and is used on the CPP side of Filament. I mostly focused on the use of public SwapChain createSwapChain(@NonNull Object surface, long flags)

I understand the call sequence (in my setup) to roughly be something like

• Engine.cpp - Java_com_google_android_filament_Engine_nCreateSwapChainHeadless
• Darwin.mm - void *getNativeWindow(JNIEnv *env, jclass klass, jobject surface)

and that this should setup the swapChain for future use in the render calls.

FWIW, I've not managed to find a single working example/test of FilamentCanvas so far for v1.9.9, I'm still not sure if I'm goofing up on initialization on the java side, but, looking at all the validation/checking going on in the C++ side, I'm starting to think it would break if that was the issue. So right now I assume there's something broken with the AWT window/swap-chain setup (on the C++ side).

[kanawish]

Had a couple of questions regarding this issue.

• If I'd like to explore/understand the underlying problem, any suggested reading/references I should be looking at?
  • Basically, I'm not sure where I should start. Read up on Apple platform docs, but then anything I should pay attention to on the JVM side?
• How high (or low) does this type of issue rank overall?

Mostly asking this from a "where should I put my energy" outlook, as a third party that would like to use Filament alongside some of the interesting work going on with Kotlin's Compose-for-desktop efforts. Is it reasonable to imagine using Filament for high-FPS scenarios in that setup, or should I only consider that possible from a C++ native desktop app perspective?

[romainguy]

FilamentPanel could be made more efficient. It currently reads the pixel back from the surface and swizzles them to be in a format that Java2D understands. I see two possible solutions to improve this:

• Figure out a way to make Java2D accept an RGBA image instead of ARGB. I believe this is possible by describing a custom color model
• Add an option in Filament to swizzle the output of the rendering on the GPU. Basically somewhere we'd just add: fragColor.rgba = fragColor.argb;. This should probably be done as an additional and optional pass in our FrameGraph

Fixing FilamentCanvas would be even better but I'm not sure where to start. There isn't much documentation about the AWT/JNI functions we use unfortunately. You could try by removing all the #ifndef NDEBUG in JAWTUtils.cpp to see if there is an error when the swap chain is created.

[kanawish]

Awesome info Romain, thank you!