The images captured by a camera are usually processed using algorithms which are more and more often based on neural networks. The lens parameters must then be optimized in order to increase the performance of this processing, sometimes regardless of the image quality at the output of the optics. This raises the question of the end-to-end optimization of a camera dedicated to a given neural network. This question has recently led to a new field of research on the joint design of lens and neural network [1-5]. The general idea is to model the image, as captured by a given sensor, with a differentiable model with respect to the lens parameters. Thus, gradient descent optimization tools can be used to jointly optimize the optical and the neural network parameters. In this context, several fields of application have been investigated in the literature, such as depth of focus extension [1,4], depth estimation [2], pose estimation while preserving privacy[3] or object detection[2].

ONERA has been working on the end-to-end design of lenses and neural networks for several years. Our work is based on the use of a differentiable optical model based on ray tracing (Formidable). This model takes as input an optical system defined by a set of lenses and simulates its impulse response as well as its Jacobian with respect to the lens parameters. Using this tool, we have performed the joint design of the lens parameters for a single task such as image restoration [4]. However this work only considered optimization of refractive lenses with a reduce field of view.

The aim of this internship is to go further in the investigation of the end-to-end design of a camera with an increased field of view (FOV) and a differentiable optical model based on ray tracing. The first step will be to develop image simulation tools of large FOV image from an ideal image database, taking into account optical aberrations simulated using Formidable, then to develop co-design methods for camera and neural network optimization. To handle this challenging task, curriculum learning could be investigated as proposed in the literature [5]. Application of the proposed method will be conducted first on image restoration task then on a higher scene analysis task such as image segmentation.

This internship will be held in collaboration with Upciti, a private company that proposes sensors for smart city applications.

To apply to this offer, please send an email with your resume and your cover letter at pauline.trouve_at_onera.fr

Here is the link to the internship subject with more information.

Bibliography

[1] S. Elmalem et al., “Learned phase coded aperture for the benefit of depth of field extension,” Opt. Express 26, 2018.

[2] J. Chang and G. Wetzstein, “Deep optics for monocular depth estimation and 3d object detection,” ECCV 2019.

[3] C. Hinojosa et al., “Learning privacy-preserving optics for human pose estimation”, ICCV, 2021.

[4] M. Dufraisse et al., (2023). Deblur or denoise: the role of an aperture in lens and neural network co-design. Optics Letters, 48(2), 231-234.

[5] X. Yang et al., “Curriculum learning for ab initio deep learned refractive optics “, Nature communications, 2024.