Compute local shading

[07hyx06]

Hi, I find some features in Mitsuba are very useful for computing the shading value of a surface point under an environment map, e.g. the BSDF importance sampling and the emitter importance sampling technique.

In my inverse rendering use case, the geometry is fixed and only the texture maps are needed to be optimized. I wonder is there an integrator or some demo code to support the local shading model (compute each surface point's shading value independently, ignore any visibility and inter-reflection)?

[flaribeiro]

sure

[07hyx06]

Thanks! Could you kindly provide more details to support local shading model in mitsuba?

[shinyoung-yi]

It will depend on what exactly you want.

1. Existing integrator plugins

Direct illumination integrator or Path tracer with the parameter max_depth = 2 only compute direct illumination ignoring inter-reflection, but it compute visibility (not ignoring).

You may want the differentiability feature for your inverse rendering use case. Then Direct illumination projective sampling or Path Replay Backpropagation with max_depth=2 will be useful.

2. Custom integrator ignoring visibility.

If you actually want such local shading ignoring visibility test, like a simple rasterization renderer, then you may need to write a custom integrator plugin. For example:

import numpy as np
import matplotlib.pyplot as plt
import drjit as dr
import mitsuba as mi
mi.set_variant('cuda_ad_rgb', "llvm_ad_rgb")
print(f"{dr.__version__ = }")
print(f"{mi.__version__ = }")
# dr.__version__ = '0.4.6'
# mi.__version__ = '3.5.2'

class LocalShadingIntegrator(mi.ad.integrators.common.ADIntegrator):
    def __init__(self, props = mi.Properties()):
        super().__init__(props)
    
    def sample(self,
               mode: dr.ADMode, # NOT USED NOW (differentiable rendering)
               scene: mi.Scene,
               sampler: mi.Sampler,
               ray: mi.Ray3f,
               active: mi.Bool,
               **kwargs # Absorbs unused arguments
               ):
        bsdf_ctx = mi.BSDFContext()
        L = mi.Spectrum(0)

        # ---------- Direct emission ----------
        si = scene.ray_intersect(ray)
        L += si.emitter(scene).eval(si)

        # ---------- BSDF sampling ----------
        # active_next = active & si.is_valid() # Better to introduce after Russian Roulete
        bsdf = si.bsdf()

        bsdf_sample, bsdf_weight = bsdf.sample(bsdf_ctx, si, sampler.next_1d(), sampler.next_2d())
        ray_bsdf = si.spawn_ray(si.to_world(bsdf_sample.wo))
        # active_bsdf = active_next & dr.any(dr.neq(bsdf_weight, 0.0)) # Better to introduce after Russian Roulete

        si_bsdf = scene.ray_intersect(ray_bsdf)
        L_bsdf = si_bsdf.emitter(scene).eval(si_bsdf)

        ds = mi.DirectionSample3f(scene, si_bsdf, si)
        emitter_pdf = scene.pdf_emitter_direction(si, ds)
        mis_bsdf = self.mis_weight(bsdf_asample.pdf, emitter_pdf)
        
        L += bsdf_weight * L_bsdf * mis_bsdf
        
        # ---------- Emitter sampling ----------
        ds, em_weight = scene.sample_emitter_direction(si, sampler.next_2d(), test_visibility=False)

        wo = si.to_local(ds.d)
        bsdf_value_em, bsdf_pdf_em = bsdf.eval_pdf(bsdf_ctx, si, wo)

        mis_em = self.mis_weight(ds.pdf, bsdf_pdf_em)

        L += bsdf_value_em * em_weight * mis_em

        return L, active, [], None
    
    def mis_weight(self, pdf_a, pdf_b):
        w = pdf_a / (pdf_a + pdf_b)
        return dr.select(dr.isfinite(w), w, 0)

mi.register_integrator("localshade", lambda props: LocalShadingIntegrator(props))

scene = mi.load_dict(mi.cornell_box())
int_local = mi.load_dict({'type': 'localshade'})
int_direct = mi.load_dict({'type': 'direct'})

fig, axes = plt.subplots(1, 2)
for ax, integrator in zip(axes, [int_local, int_direct]):
    img = mi.render(scene, integrator=integrator).numpy()
    ax.imshow(np.clip(img, 0, 1) ** (1/2.2))
    ax.set_title(str(integrator).split('[')[0])
    ax.set_axis_off()

Overall logic is just a simplified version of the path tracer, but the key difference is the following line:

ds, em_weight = scene.sample_emitter_direction(si, sampler.next_2d(), test_visibility=False)

The default parameter (used in most of existing integrator plugins in Mitsuba 3) of scene.sample_emitter_direct is test_visibility=True, which returns zero values of em_weight: mi.Spectrum whenever the sampled point on an emitter is not directly visible from the reference point (si: mi.SurfaceInteraction).

However note that this implementation is just a draft where the ways to ignore visibility are inconsistent for BSDF and emitter samplings. To correct this issue there will be several options, which makes implementation more complicated than the direct illuminator (w/ visibility test)
In addition, this draft of the local shading integrator may not provide differentiability properly. It needs to make the implementation more complicated.

My suggestion

In my opinion, in physically-based ray tracing system, taking visibility into account would provide more natural implementation of integrators rather than ignoring visibility. Thus, I suggest to use existing integrator plugins, which compute visibility.
You may also need differentiable rendering, but not differentiating w.r.t. object geometry. Then you may not need projective sampling integrators, which requires to decide more parameters and additional cost to compute geometry derivative.
So I suggest to use Path Replay Backpropagation integrator with setting the parameter max_depth=2
(I'm not sure whether path tracer or direct integrator support differentiation properly or not, in my knowledge. However, the PRB integrator will support)

[07hyx06]

Thanks a lot!!