IndirectLight in an AR context

[ThomasGorisse]

Hi,

I'm working on moving the Scene LightProbe inside Scenform from to the actual home made sfb definition (using PixelBufferDescriptor) to a ktx IndirectLight and on integrating ARCore Environmental Hdr Spherical Harmonics.

I have 3 use cases

1. No ARCore
   • Cubemap comes from an hdr asset with an IBL ktx from cmgen
   • KTXLoader is used to load IndirectLight
   • IndirectLight is directly applied to the scene
2. ARCore without Environmental Hdr Spherical Harmonics
   • Cubemap comes from an hdr asset with an IBL ktx from cmgen
   • KTXLoader is used to load IndirectLight
   • A new IndirectLight is applied to the scene every time light estimation changes with:
     • reflections = ktxIndirectLight.getReflectionsTexture()
     • irradiance = ktxIndirectLight.ambientColor * arCore.colorCorrection
     • intensity = ARCore.lightEstimate
3. ARCore with Environmental Hdr Spherical Harmonics
   • Cubemap comes from ARCore in an android.media.Image[] form.
   • PixelBufferDescriptor and generatePrefilterMipmap() is used to build the cube map Texture
   • A new IndirectLight is applied to the scene every time light estimation changes with:
     • reflections = new Texture(arCore.cubemapImageArray)
     • irradiance = arCore.ambientColor
     • intensity = ARCore.lightEstimate

Questions

1. In case 2, is there any way to directly use the KTX IndirectLight and apply the actual irradiance instead of building a new one based on it (IndirectLight.setIntensity() is possible but there is no IndirectLight.setIrradiance()) ?

2. In case 2, how to extract the ambientColor coming from the IndirectLight generated by KTXLoader in order to apply the AR environement colorCorrection to the instant generated IndirectLight ?

3. In case3, Sceneform developers leaved this comment // TODO: Update once Filament updates past v1.3.0.
   Do you think there could be a better way to generate the cubemapTexture from an Image[] than this ?

  public void setCubeMap(Image[] cubemapImageArray) {
    // TODO: Update once Filament updates past v1.3.0.
    if (cubemapImageArray.length != CUBEMAP_FACE_COUNT) {
      throw new IllegalArgumentException(
          "Unexpected cubemap array length: " + cubemapImageArray.length);
    }

    int width = cubemapImageArray[0].getWidth();
    int height = cubemapImageArray[0].getHeight();
    int bufferCapacity =
        width * height * CUBEMAP_FACE_COUNT * RGB_CHANNEL_COUNT * BYTES_PER_FLOAT16;
    if (cubemapBuffer.capacity() < bufferCapacity) {
      cubemapBuffer = ByteBuffer.allocate(bufferCapacity);
    } else {
      cubemapBuffer.clear();
    }

    int[] faceOffsets = new int[CUBEMAP_FACE_COUNT];
    for (int i = 0; i < CUBEMAP_FACE_COUNT; i++) {
      faceOffsets[i] = cubemapBuffer.position();
      Image.Plane[] planes = cubemapImageArray[i].getPlanes();
      if (planes.length != 1) {
        throw new IllegalArgumentException(
            "Unexpected number of Planes in cubemap Image array: " + planes.length);
      }
      Image.Plane currentPlane = planes[0];
      if (currentPlane.getPixelStride() != RGBA_BYTES_PER_PIXEL) {
        throw new IllegalArgumentException(
            "Unexpected pixel stride in cubemap data: expected "
                + RGBA_BYTES_PER_PIXEL
                + ", got "
                + currentPlane.getPixelStride());
      }
      if (currentPlane.getRowStride() != width * RGBA_BYTES_PER_PIXEL) {
        throw new IllegalArgumentException(
            "Unexpected row stride in cubemap data: expected "
                + (width * RGBA_BYTES_PER_PIXEL)
                + ", got "
                + currentPlane.getRowStride());
      }
      ByteBuffer rgbaBuffer = currentPlane.getBuffer();
      while (rgbaBuffer.hasRemaining()) {
        for (int byt = 0; byt < RGBA_BYTES_PER_PIXEL; byt++) {
          byte b = rgbaBuffer.get();
          if (byt < RGB_BYTES_PER_PIXEL) {
            cubemapBuffer.put(b);
          }
        }
      }
    }
    cubemapBuffer.flip();

    IEngine engine = EngineInstance.getEngine();
    int levels = (int) (1 + Math.log(width) / Math.log(2.0));
    Texture cubemapTexture =
        new com.google.android.filament.Texture.Builder()
            .width(width)
            .height(height)
            .levels(levels)
            .sampler(com.google.android.filament.Texture.Sampler.SAMPLER_CUBEMAP)
            .format(com.google.android.filament.Texture.InternalFormat.R11F_G11F_B10F)
            .build(engine.getFilamentEngine());
    Texture.PixelBufferDescriptor pixelBuf =
        new com.google.android.filament.Texture.PixelBufferDescriptor(
            cubemapBuffer,
            com.google.android.filament.Texture.Format.RGB,
            com.google.android.filament.Texture.Type.HALF);
    com.google.android.filament.Texture.PrefilterOptions options =
        new com.google.android.filament.Texture.PrefilterOptions();
    options.mirror = false;
    cubemapTexture.generatePrefilterMipmap(
        engine.getFilamentEngine(), pixelBuf, faceOffsets, options);
    setCubeMapFromTexture(cubemapTexture);
  }

Thanks

[romainguy]

In case 2, is there any way to directly use the KTX IndirectLight and apply the actual irradiance instead of building a new one based on it (IndirectLight.setIntensity() is possible but there is no IndirectLight.setIrradiance()) ?

Building a new IndirectLight is cheap, it's just a C++ object and we prefer to rely on builders than mutable objects. We add setters when they are truly necessary.

In case 2, how to extract the ambientColor coming from the IndirectLight generated by KTXLoader in order to apply the AR environement colorCorrection to the instant generated IndirectLight ?

The ambient color will be the first band (first 3 floats) of the spherical harmonics loaded from the KTX file.

In case3, Sceneform developers leaved this comment // TODO: Update once Filament updates past v1.3.0.
Do you think there could be a better way to generate the cubemapTexture from an Image[] than this ?

I'm not sure what Sceneform was waiting for in 1.3.0. Sceneform seems to be copying from one buffer to another, I don't see why they can't use the source buffer directly.

[ThomasGorisse]

@romainguy Thanks a lot for your time.

I'm not sure what Sceneform was waiting for in 1.3.0. Sceneform seems to be copying from one buffer to another, I don't see why they can't use the source buffer directly.

Maybe related to a crash when using flatbuffer buffers directly. I'll check if it's resolved since it seems to be the case elsewere even if it's in sfb part.

I have to say that it's a heartbreak to delete so much great work done by the Sceneform team specially the sfb part which is well thought and huge. I take my hat off to them.

Here is a quick result of Ambient Intensity vs Environmental HDR applied to this material :

Ambient Intensity (without KTX)	Environmental HDR (IBL generated by ARCore)

The real question for now is Spectacular vs Realistic

Should ask a cinema noise maker to know witch is the best 😄

[romainguy]

By Environmental HDR you mean the IBL generated by ARCore? If so, that IBL is very low resolution (16x16 for each cubemap face) which means you can't get nice looking glossy reflections. Which on to use will depend on your content :/

[ThomasGorisse]

Yes, I meant
Environmental HDR = ARCore LightEstimationMode.ENVIRONMENTAL_HDR = ARCore CubeMap + ARCore Converted irradiance + ARCore Converted intensity = IndirectLight
Is there any chance for one day seeing ARCore using the face camera to generate an high resolution IBL ?

I will make one test with KTX and one with ARCore Environmental HDR irradiance and intensity but the static same cubemap asset:

[ThomasGorisse]

or StreetView datas

[romainguy]

I unfortunately don't know what the ARCore plans are. I believe the current IBL is generate via ML to fill the gaps which might be why it has to be low resolution. One thing that would help with a static env map is to adapt the white balance using what ARCore estimates (or at least the first SH band).