Awesome Deep Optics/End-to-end Optical Design

A curated list of awesome deep optics papers, inspired by awesome-computer-vision.

Deep optics/end-to-end optical design learns optical elements simutaneously with the image processing network, with the goal to:

• find the best match between optics and network.
• encode more information from the physical world.
• minimize the cost and complexity.

Contribution

Please feel free to open pull requests or email ([email protected]) to contibute to this repo.

Courses

• Siggraph 2022. Differentiable cameras and displays. link
• Siggraph 2020. Deep optics: joint design of optics and image recovery algorithms for domain specific cameras. link
• Siggraph 2012. Computational displays: combining optical fabrication, computational processing, and perceptual tricks to build the displays of the future. link

Papers

A differentiable image formation model enables us to optimize the optics with the network. So in this part I classify papers according to the image formation model. The classical methods treat the diffraction and aberration separately, either simplifying the optical system as a series of thin elements to capture the wave optical effects, or doing ray tracing to capture the full geometry of the system.

1. Hybrid diffraction-aberration model

• 2019 Ray-based methods for simulating aberrations and cascaded diffraction in imaging systems. thesis

2. Wave propagation model

The wave propagation model is easy to describe the perporty of diffractive elements, but less accurate for spherical/spherical lenses. Therefore, people usually optimize a single DOE or DOE with a singlet lens.

Diffreactive optical element (DOE)

• 2016 Encoded diffractive optics for full-spectrum computational imaging. paper
• 2016 The diffractive achromat full spectrum computational imaging with diffractive optics. paper, code, slides, video.
• 2018 End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. paper, project, code.
• 2019 Compact Snapshot Hyperspectral Imaging with Diffracted Rotation. paper, project
• 2020 Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging. paper, project.
• 2021 Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics. paper, project, video
• 2022 Quantization-aware Deep Optics for Diffractive Snapshot Hyperspectral Imaging. paper, code

DOE + Thin lens (not optimizable)

• 2020 Learning Rank-1 Diffractive Optics for Single-shot High Dynamic Range Imaging. paper, project
• 2020 Deep Optics for Single-shot High-dynamic-range Imaging. paper, project, code
• 2020 End-to-end Learned, Optically Coded Super-resolution SPAD Camera. paper, project
• 2021 Depth from Defocus with Learned Optics for Imaging and Occlusion-aware Depth Estimation. paper, project, code
• 2022 Seeing Through Obstructions with Diffractive Cloaking. paper, project, code
• 2022 Hybrid diffractive optics design via hardware-in-the-loop methodology for achromatic extended-depth-of-field imaging. paper

MetaLens

• 2020 Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array. paper, code
• 2021 Neural nano-optics for high-quality thin lens imaging. paper, project, code

Freeform lens

• 2019 Deep optics for monocular depth estimation and 3D object detection. paper, project

Coded Aperture

• 2021 Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging. paper, project, codes
• 2021 Shift-variant color-coded diffractive spectral imaging system. paper, video, code

3. Ray tracing model

Conventional ray tracing focuses on PSF calculations, but recent works start to directly render and optimize sensor images (differentiable ray tracing). Differential ray tracing also has some similarities to differentiable rendering.

Singlet Lens

• 2019 Learned large field-of-view imaging with thin-plate optics. project, video, code
• 2021 End-to-end computational optics with a singlet lens for large depth-of-field imaging. paper
• 2021 End-to-end learned single lens design using fast differentiable ray tracing. paper

Compound Lens

• 2021 End-to-end complex lens design with differentiate ray tracing. paper, project
• 2021 Differentiable Compound Optics and Processing Pipeline Optimization for End-To-end Camera Design. paper, project
• 2021 Deep learning-enabled framework for automatic lens design starting point generation. paper, project
• 2021 dO: A differentiable engine for Deep Lens design of computational imaging systems. paper, project, code
• 2022 Automatic Lens Design based on Differentiable Ray-tracing. paper, project, video.
• 2022 Computational Optics for Mobile Terminals in Mass Production. paper
• 2022 The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection. paper, code
• 2023 Curriculum Learning for ab initio Deep Learned Refractive Optics. paper, video, code

Others (Sensor, Glass)

• 2021 Towards self-calibrated lens metrology by differentiable refractive deflectometry. paper, project, code
• 2021 End-to-end sensor and neural network design using differential ray tracing. paper
• 2022 Adjoint Nonlinear Ray Tracing. paper

Licenses

License

To the extent possible under law, Xinge Yang has waived all copyright and related or neighboring rights to this work.