This end-to-end camera design has recently led to a new field of research on the joint design of lens and neural network [1-5]. The general idea is to model the image, as captured by a given sensor, with a differentiable model with respect to the lens parameters. Thus, gradient descent optimization tools can be used to jointly optimize the optical and the neural network parameters. In this context, several fields of application have been investigated in the literature, such as depth of focus extension [1,4,5], depth estimation [2], pose estimation while preserving privacy[3] or object detection[2]. ONERA has been working on the end-to-end design of lenses and neural networks for several years. Our work is based on the use of a differentiable optical model based on ray tracing (Formidable). This model takes as input an optical system defined by a set of lenses and simulates its impulse response as well as its Jacobian with respect to the lens parameters. Using this tool, we have performed the joint design of the lens parameters for a single task such as image restoration [4,5]. However this work only considered optimization of refractive lenses. On the other hand, previous works of the literature on end-to-end joint design of optics and processing have proposed to use unconventional lenses such as phase mask or a free-form lens to encode the lens impulse response for a dedicated unsupervised or supervised processing [1,2,6,7]. These unconventional lenses have been introduced in a joint optical/neural network joint design, but only with a based on Fourier’s optics[1,2]. Therefore the first aim of this thesis is to integrate the optimization of unconventional lenses such as phase masks or freeform lenses into the end-to-end joint optical and neural network optimization framework, using a ray tracing optical model such as Formidable. In particular, an important work will be to investigate the interaction between the choice of the neural network architecture and the unconventional surface type. On the other hand, multi-tasks approaches are increasingly being developed in the computer vision literature, e.g. for RGB-D cameras producing bot h depth maps and extended depth-of-field images[8], or for aerial image processing for depth maps and image segmentation [9]. Furthermore, in the field of joint optical and neural processing joint design, two tasks can be trained in an adversarial mode, as in camera designed to perform image analysis while preserving privacy [3]. The joint optimization of an unconventional lens with two neural networks, possibly in a competitive mode, or in interaction with each other, is another important issue to be investigated in the thesis. This objective raises several questions, such as how to deal with the necessary balance between the two tasks during the optimization and how to avoid the local optima for the lens parameters associated with each task. It also questions the chosen starting points for each part of the system.

The PhD will be organized as follows : after studying the bibliography on end-to-end joint design of optics and processing, including neural networks, the PhD student will develop new optimization tools to perform the joint design of unconventional lenses and neural networks using Formidable. This work will be done in collaboration with the ONERA/DOTA department. Secondly, the addition of another task to the joint optimization framework will be considered, either in a multi-task mode, or in an adversarial mode. The last part of the thesis will be dedicated to the realization of a demonstrator for a selected application.

Bibliography

[1] S. Elmalem et al., “Learned phase coded aperture for the benefit of depth of field extension,” Opt. Express 26, 2018.
[2] J. Chang and G. Wetzstein, “Deep optics for monocular depth estimation and 3d object detection,” ECCV 2019.
[3] C. Hinojosa et al., “Learning privacy-preserving optics for human pose estimation”, ICCV, 2021.
[4] A. Halé et al., “End-to-end sensor and neural network design using differential ray tracing. Optics express, 29(21), 2021.
[5] M. Dufraisse et al., (2023). Deblur or denoise: the role of an aperture in lens and neural network co- design. Optics Letters, 48(2), 231-234.
[6] E. R. Dowski et W. T. Cathey, “Extended depth of field through wave-front coding,” Appl. Opt. 34(11), 1859–1866 (1995).
[7] F.Diaz, F. Goudail, B. Loiseaux, J.-P. Huignard, “Comparison of depth of focus enhancing pupil masks based on a signal-to-noise ratio criterion after the deconvolution”, J. Opt. Soc. Am. A 27, 2123-2131 (2010)
[8] S. Nazir et al.”2HDED: Net for joint depth estimation and image deblurring from a single out-of-focus image”, ICIP 2022.
[9] Carvalho, M., Le Saux, B., Trouvé-Peloux, P., Champagnat, F., & Almansa, A. (2019). Multitask learning of height and semantics from aerial images. IEEE Geoscience and Remote Sensing Letters, 17(8), 1391-1395.

To apply please send a resume and a cover letter to pauline.trouve@onera.fr

Link to the PhD description : here.