Differentiable imaging

Short description: Technique and research method within computational imaging

Differentiable imaging is a method within computational imaging that incorporates differentiable programming to design imaging systems. It treats the entire imaging process - from light passing through optical components to the numerical reconstruction—as a differentiable programming problem. This approach links optical hardware with numerical reconstruction, enabling joint optimization of both parts through differentiable programming. Differentiable imaging additionally extends the scope of computational imaging beyond image reconstruction, such as by aiding in characterization of optical components.

Background

Main article: Computational imaging

Computational imaging combines optical hardware and computational algorithms to capture and reconstruct information that conventional imaging system cannot. This is achieved from a combination of the imaging system and the software used in the image reconstruction. Since the captured information may not directly show the image of the target, these systems often rely on numerical models that describe how light encodes the target. In practice, such models may deviate from the physical systems due to uncertainties such as noise, misalignments, manufacturing imperfections, environmental variations, and etc. These uncertainties can cause a mismatch between the physical system and its numerical model, which may degrade reconstruction quality and limit the effectiveness of the hardware–software co-design. Uncertainty quantification is also studied in other hybrid physical–numerical systems, such as digital twin. While numerical modeling imaging systems date back to the several decades, such as the multislice method in electron microscopy or X-Ray nanotomography, differentiable imaging emphasizes jointly modeling uncertainties and solving inverse problems with image reconstruction simultaneously.

Differentiable Imaging and Computational Imaging

Differentiable imaging transforms the traditional encoding model $y=f(x)$ into a more comprehensive formulation $y=f(x,\theta )$, where $\theta$ represents a parameter set of mismatches between physical systems and numerical models.

The forward model captures the entire imaging pipeline through a series of interconnected component functions:

$y=f(x,\theta ),f=f_{noise}\circ f_c\circ f_{oc}\circ f_x\circ f_{oi}\circ f_i,$

where the function composition operator $\circ$ connects each system component, and $\theta =\{{\theta }_c,{\theta }_{oc},\dots \}$ encompasses uncertainty system parameters. Each component corresponds to specific physical processes within the imaging system, from illumination through object interactions to sensor behavior and noises.

This forward model enables the formulation of an inverse problem that simultaneously optimizes system parameters while reconstructing images:

$x^{*},{\theta }^{*}={\text{argmin}}_{x,\theta }{ℒ}(f(x,\theta ),y)+\sum _{n=1}^N{\beta }_n{{ℛ}}_n(x)$

$s.t.x\in {\mathrm{\Omega }}_x,\theta \in {\mathrm{\Omega }}_{\theta }$

Here, ${ℒ}(f(x,\theta ),y)$ represents the fidelity term that quantifies the discrepancy between the model predictions and measured data.

The whole process of the $y=f(x,\theta )$ is constructed as a computer graph based on differentiable programming, and the inverse problem is solved with gradient based algorithm, while the gradient is calculated with automatic differentiation.

Applications

One application of differentiable imaging is uncertainty management, which seeks to quantify and mitigate the impact of factors induce reality-numerical mismatch. Explicitly accounting for uncertainties can improve reconstruction accuracy and system robustness. Examples include:

• Model-related uncertainties: unknown or unmeasurable variables—for instance, optical system quantities that differ from the design specifications
• Data and system uncertainties: artifacts introduced during image acquisition, such as low-quality data, noise, or hardware imperfections
• Manufacturing uncertainties: variability in the production of imaging hardware—such as slight deviations in lens curvature or sensor alignment—that alters the physical system's behavior