Android camera preview using Hardware Buffers.

This repository showcases how to grab camera image stream and display the preview on the SurfaceView. Almost all the logic is written in C++.

Instructions and requirements

Requires Android SDK >= 26 and NDK 25.1.8937393. Open the project in Android Studio, make sure NDK is installed and run.

Overview and technology stack

• Using NDK Native Hardware Buffer along with EGL and Vulkan extensions to work with HW buffers and convert them to an OpenGL ES external texture or Vulkan image backed by external memory.
• Supporting both OpenGL ES 3 and Vulkan 1.3 rendering backends for Android CameraX.
  • Noting that CameraX hardware buffer is provided with AHARDWAREBUFFER_USAGE_CPU* flags so I have to re-allocate buffers internally so that they could be used as Vulkan external memory.
• Supporting OpenGL ES 3 rendering backend for Android Camera2.
  • Noting ImageReader is configured to generate images with AHARDWAREBUFFER_USAGE_GPU_SAMPLED_IMAGE flag so no extra copy operations happen, hardware buffer is mapped directly to OpenGL external texture.
  • There is no Vulkan support as actual underlying GPU sampled image has YUV format and Vulkan extension adaptation is required (use YUV sampler).
• Using Jetpack Compose to build comprehensive UI.
• Android CameraX is used to configure Android camera. Pretty interesting remark is that we are not even binding preview use case and make use only of image analysis.
• Using dedicated background thread to obtain camera images represented as ImageProxy.
• Using dedicated render thread in C++ backed up by NDK Looper.
• Using NDK Choreographer for effective rendering.

Next steps / tasks

• Investigate CameraX to provide Hardware Buffers with AHARDWAREBUFFER_USAGE_GPU_SAMPLED_IMAGE usage flag.
• Add support to take a photo by using AHardwareBuffer_lock functionality and compare it with glReadPixels approach.
• Gather some metrics to check Hardware Buffers performance in comparison with more classic approaches.
• Add support for ImageAnalysis.Builder().setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888).