Integrator for optimizing vertex positions of glass

[matthiasschwartz]

Hi, i'm currently trying to optimize the vertex positions of a rectangle with a dieletric material. I want to optimize it, such that an object behind it is obscured. I thought it might be possible, where the view rays are being split away from the object because of the refraction of the rectangle in front of it.
Is that possible? And if yes, which integrator is the best suited for the task? I could not get any good results, and i think it is because of the integrator. I've been using prb_reparam, but i dont get good results.
I thought it was possible, as it is kind of similar to the caustics_optimization tutorial, where a glass slab is also optimized.

[njroussel]

Hi @matthiasschwartz

prb_reparam is your only option here. We don't have any other integrators that can differentiate changing shapes.

It's not immediately clear to me why this would not work. What/How did you define as your target image? In theory, you should be able to optimize the glass slab up until it only produces total internal reflection.

[matthiasschwartz]

Hi, i put an answer to the original thread, but it was supposed to be an answer to your question, im not that used to github

[matthiasschwartz]

Hi @njroussel,

sorry for not having my question not be as clear as it should have been.
The scene i have set up is the following:

The first image is the reference picture. The second picture is the scene with the lens and the object behind it (the bunny).
I want to optimize the lens in such a way, where the lens refracts the light in such a way that it looks like there is no object at all.
The lens is the same lens as in the "caustic_optimization" tutorial with the same BSDF. I also zoom in to the part of the image with the lens, as to only take that area for the error.
I have the optimization loop also set up similarly to "caustic_optimization" (like the heightmap and loss, etc.)

Using "prb" the lens does not change at all.
Using "prb_reparam" the lens just gets very distorted, and the error does not decrease.
This is the result:

I had modified the "caustic_optimization" tutorial with "ptracer", where the emitter emitted a pre-rendered image of the scene onto the caustic. I then put the bunny between the emitter and the lens. Using this setup i got really good results with the ptracer, but i could not transfer them to the actualy scene with much success.

Thank you very much for your help! I hope this post made my question clearer

[ziyi-zhang]

Hi @matthiasschwartz

Could you try to replace the dielectric material with a rough dielectric one with small roughness (so it look like smooth dielectric)?
It might also be helpful to increase the kappa parameter in the reparam integrator.

---

I am also very curious about your ptracer experiment. Did you setup the scene like this (four things along one line?):

camera ~~ glass ~~ bunny ~~ area_emitter_with_image_tex

Would you mind sharing the optimized result?
Does it hide the bunny by (1) trapping the rays inside the glass plate (for example with total internal reflection), or (2) refract the ray so that we see a background region which matches the reference image?

[ziyi-zhang]

My guess about why ptracer works and prb-reparam fails:

• Essentially the caustic design tutorial is a continuous optimization. When the ray hits the Bayer-textured emitter, the radiance is computed through a bilinear interpolation (in a differentiable fashion). In order for this pixel to better match the reference image's corresponding pixel color, gradient descent would push the final hit position towards one of the four corners (RGBG). Since we only optimize the glass plate, so the height map receives the gradients to achieve the goal.
• However, what you want to achieve here is a discontinuous optimization. For a ray that currently hits the bunny to be refracted to hit the background (or whatever position that matches the target color), that ray needs to cross the bunny's visibility silhouette. This time we do not have the continuous bilinear texture interpolation, so the ray does not receive any gradient that pushes it across the boundary.
  • This discontinuous optimization can still work, if our system is aware that a slight change in the height field would make the visibility boundary move (illustration here Fig 5). In which case it will push the boundary inward because the background better matches the reference color. This implicitly pushes more rays across the visibility silhouette.
  • Unfortunately, currently no integrator in mitsuba can handle this combination of visibility discontinuity and smooth Delta BSDFs, which is why using a rough dielectric material might produce better results.
  • The ptracer experiment is interesting here. For the light paths that are occluded by the bunny, they would never carry any gradient because they are not connected with the sensor. But the optimization could still work if you use a Gaussian reconstruction filter: light paths that happen to miss the visibility silhouette would be dragged by interior pixels. What i wrote is purely speculative. I hope it helps.

[matthiasschwartz]

Hi, thanks for your answer!
Changing the material to roughdielectric definetly helped. I set up a simple scene with the bunny and the lens to try and optimize it, and the results look really good.
Here is the initial setup:

And here is the optimized scene:

It is not perfect yet, but i probably just have to tweak some parameters.
I also tried your suggestions in the more complex staircase scene and it still yielded not very good results, but i didn't experiment that much yet.

Regarding the ptracer experiment, the scene setup was like this (nearly identical to the tutorial with exception of the bunny:
caustic ~~ camera looking at caustic ~~ glass ~~ bunny ~~ area_emitter_with_image_tex

The initial scene looked like this:

And the results:

So the lens refracts the rays such that they resemble the reference image.

Again, thank you very much for your help!

[ziyi-zhang]

wow very impressive result!
Judging from the heightmap, what i wrote above might be correct:

• In the heightmap, we see the shape of a bunny with almost zero displacement. So light paths that hit the bunny are all wasted due to discontinuities.
• We still see the occluded chair in the optimized image. Their radiance comes from light paths that interact with the surrounding area of the glass plate.

This also explains why the original optimization fails:

• First, the correct derivative is hard to compute due to discontinuities.
  • We can alleviate the damage by using rough dielectric materials.
• Second, if the camera directly looks at the glass (this is a key difference), then we can no longer "see through" the bunny with the help of the light paths in the ptracer case. We must rely on the camera rays that hit the bunny to "jump across" the visibility silhouette.
  • This would be challenging since gradient descent is a local strategy. One solution is to initialize the heightfield such that no camera ray hits the bunny in the first place. Or we can use a very rough dielectric.