End-of-study Engineer or 2nd year Master internship offer

Title: Deep learning 3D infant pose estimation from monocular images.

keywords: pose estimation, infant 3D pose, deep learning, model finetuning, RGB images.

Context: Preterm birth is a risk factor for developmental disabilities, such as behavioral difficulties, cognitive impairments, or cerebral palsy (CP). In many cases, these developmental disabilities cannot be identified before two years of age [1]. However, early identification of preterm infants with abnormal developmental trajectories is critical to initiate early developmental intervention to prevent the occurrence of such developmental disabilities [2]. The General Movement Assessment (GMA) analyses the complexity, variability, and fluidity of preterm spontaneous movements using video recordings [3]. The GMA is a reliable assessment of brain maturation and is able to identify preterm infants with abnormal developmental trajectories [4]. However, this method remains subjective and time-consuming. This motivates the need for automated GMA, especially with the continuous progress in computer vision and artificial intelligence tools. To achieve this goal, we developed a specific pose estimation framework to track preterm infant movements and estimate their poses in the three dimensions of space [5]. Then, we proposed a quantitative approach for GMA with the mean 3D dispersion [11]. However, these tools require a stereoscopic camera, which is rarely accessible in clinical practice.

Successful completion of the internship may lead to a PhD position.

Objectives: To develop a method for estimating 3D infant poses from 2D RGB images without complex hardware or specialized protocols. Utilizing an existing manually annotated dataset with over 88,000 2D images and corresponding 3D poses, you will:

• Fine-tune state-of-the-art human pose estimation models [8,9,10] on the infant image dataset.
• Evaluate the performance of both retrained and non-retrained model versions compared to existing 3D infant pose estimation methods [5,6,7].
• Draft a scientific article summarizing the experiments conducted and the results obtained.

Host laboratory: Laboratoire Hubert Curien UMR CNRS 5516, Saint-Etienne, France

https://laboratoirehubertcurien.univ-st-etienne.fr/en/index.html

Candidate profile: Master or engineering school in computer science, preferably with knowledge of neural networks and deep learning. Experience in Python programming is mandatory.

Application: Send the following information to agma.st.etienne@gmail.com before the 1st of December 2024:

• Curriculum Vitae,
• Cover letter,
• Github repository of any previous projects (optional).

Internship allowances: 609€ for 20 working days (about one month).

Bibliography

[1] Pierrat V, Marchand-Martin L, Marret S, Arnaud C, Benhammou V, Cambonie G, Debillon T, Dufourg M-N, Gire C, Goffinet F, Kaminski M, Lapillonne A, Morgan AS, Rozé J-C, Twilhaar S, Charles M-A, Ancel P-Y (2021) Neurodevelopmental outcomes at age 5 among children born preterm: Epipage-2 cohort study. BMJ, 373

[2] Spittle A, Orton J, Anderson P, Boyd R, Doyle L (2015) Early developmental intervention programmes provided post hospital discharge to prevent motor and cognitive impairment in preterm infants. Cochrane Database of Systematic Reviews

[3] Hadders-Algra, Mijna. “Neural substrate and clinical significance of general movements: an update.” Developmental medicine and child neurology vol. 60,1 (2018): 39-46. doi:10.1111/dmcn.13540

[4] Olsen JE, Cheong JLY, Eeles AL, FitzGerald TL, Cameron KL, Albesher RA, Anderson PJ, Doyle LW, Spittle AJ (2020) Early general movements are associated with developmental outcomes at 4.5-5 years. Early Human Develop 148:105115

[5] Soualmi, A., Ducottet, C., Patural, H. et al. A 3D pose estimation framework for preterm infants hospitalized in the Neonatal Unit. Multimed Tools Appl 83, 24383–24400 (2024). https://doi.org/10.1007/s11042-023- 16333-6

[6] Qingqiang Wu, Guanghua Xu, Fan Wei, Chen Longting, and Sicong Zhang. Rgb-d videos-based early prediction of infant cerebral palsy via general movements complexity. IEEE Access, PP:1–1, 03 2021.

[7] Min Li, Fan Wei, Yu Li, Sicong Zhang, and Guanghua Xu. Threedimensional pose estimation of infants lying supine using data from a kinect sensor with low training cost. IEEE Sensors Journal, 21(5):6904–6913, 2021.

[8] W. Zhu, et al., "MotionBERT: A Unified Perspective on Learning Human Motion Representations," in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023 pp. 15039-15053. doi: 10.1109/ICCV51070.2023.01385

[9] Zheng, Ce & Zhu, Sijie & Mendieta, Matias & Yang, Taojiannan & Chen, Chen & Ding, Zhengming. (2021). 3D Human Pose Estimation with Spatial and Temporal Transformers. 11636-11645. 10.1109/ICCV48922.2021.01145.

[10] Zhang, Jinlu et al. “MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video.” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 13222- 13232.

[11] A. Soualmi, O. Alata, C. Ducottet, H. Patural and A. Giraud, "Mean 3D Dispersion for Automatic General Movement Assessment of Preterm Infants," 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Sydney, Australia, 2023, pp. 1-5, doi: 10.1109/EMBC40787.2023.10340961