1 Context and application

Earth observation has become a crucial issue in recent years, particularly as it enables us to measure the effects of climate change. One way of doing this is by analyzing a time series of multivariate images (MITS) obtained mainly from remote sensing satellites such as Sentinel, TerraSARX, ... These images are obtained using different types of sensor: radar, optical, mutlispectral and hyperspectral, etc. To cite an example of application thanks to this analysis, it is possible to classify crops thanks to the temporal, spatial and spectral dimensions of a multispectral MITS (see [Rußwurm et al., 2020] with the sentinel 2 mission). The main analysis algorithms in this field obviously come from the machine learning community, with more and more deep learning approaches, although the lack of labeled data remains a major problem. Even though these MITS are made up of different modalities with their own advantages and disadvantages, there are not many works that propose to use all this information because of the complexity of the problem. The classic solution would be to develop a model based on a classical CNN, such as Resnet, and to train it using all the data with all the modalities (RADAR, multispectral, hyperspectral, ...). However, this strategy has several major drawbacks. Firstly, it requires the transfer of all data to a single server, which is not only time-consuming but also very costly in terms

of memory space. Secondly, the learning phase will be complicated as the data properties for each mode are very different, and a merged model will therefore require a large number of labeled data in all modalities, which is not often the case, especially for RADAR. A popular strategy is to use federated learning. In our case, this will allow us to separately train a model for each modality, then transfer its parmeters to the central server. Finally, an algorithm merging all these parameters will

build a model adapted to all these modalities. This strategy is therefore lighter in terms of transfer costs and hardware, while still delivering good performance.

But for remote sensing data, this strategy will be insufficient for two main reasons: the lack of labeled data and the high variability of the data in the class, especially over time. Indeed, it is common to observe different data mismatches between training and test sets on both optical and RADAR data. To address this problem, we have recently shown that the combination of the use of second-order statistics (through the computation of a covariance matrix) and Riemannian geometry provides an excellent framework for building Machine Learning models robust to data shifts, uncertain labels and a small amount of training data.

2 Objectives

The main objective of this internship is therefore to build a robust federated learning model adapted to remote sensing MITS using covariance matrices as main features and the SPD (symmetric positive definite) architectures [Li et al., 2017, Huang and Gool, 2017] (CNNs with layers adapted to covariance matrices). We will call this new model Fed-SPD. Drawing on previous work in the field [Li and Ma, 2022], we will build this new model which will mainly consist in proposing a stochastic gradient descent (SGD) algorithm adapted to covariance matrices and SPD architecture [Zhao et al., 2024].

We propose to test our approach on the [Sainte Fare Garnot et al., 2022] dataset, which contains time series of multispectral and SAR images over the same period and region. This new model, FedSPD, will make use of all available information (spectral, spatial, polarimetric and temporal) from this dual modality. In addition, SAR data are very interesting, as they provide better periodicity and relevant information even at night or in the presence of clouds, and will therefore resolve the problem of missing data encountered with multispectral images.

In case of a good advancements of the work, we will extend our Fed-SPD model to an online procedure where the arrival of a new image does not need a new complete computation of the SGD. A recent work [Huang et al., 2024] has proposed a solution and we could also use our recent work to compute efficient Frechet (Karcher) mean of SPD matrices [Bouchard et al., 2024] to propose an innotive solution.

3 Requirements

Master/Engineering student with knowledge in signal/image processing, statistics and/or machine learning (having followed courses on these topics is strongly advised). Good coding skills in Python.

4 Internship details:

Location: LISTIC laboratory in Univ. Savoie Mont-Blanc, Annecy.

Duration: 4-6 months

Context: This work will constitute the following of a strong collaboration between Univ. Savoie Mont-Blanc and CentraleSupelec

Supervision team contacts:

• Ammar Mian, LISTIC, Univ. Savoie-Mont Blanc webpage.

email: ammar.mian@univ-smb.fr

• Florent Bouchard, L2S, CentraleSupelec webpage.

email: florent.bouchard@centralesupelec.fr

• Guillaume Ginolhac, LISTIC, Univ. Savoie-Mont-Blanc

email: guillaume.ginolhac@univ-smb.fr

References

[Bouchard et al., 2024] Bouchard, F., Mian, A., Tiomoko, M., Ginolhac, G., and Pascal, F. (2024). Random matrix theory improved Frechet mean of symmetric positive definite matrices. In International Conference on Machine Learning. PMLR.

[Huang and Gool, 2017] Huang, Z. and Gool, L. V. (2017). A riemannian network for SPD matrix learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 2036–2042. AAAI Press.

[Huang et al., 2024] Huang, Z., Huang, W., Jawanpuria, P., and Mishra, B. (2024). Riemannian federated learning via averaging gradient stream.

[Li and Ma, 2022] Li, J. and Ma, S. (2022). Federated learning on Riemannian manifolds.

[Li et al., 2017] Li, P., Xie, J., Wang, Q., and Zuo, W. (2017). Is second-order information helpful for large-scale visual recognition? In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2089–2097.

[Rußwurm et al., 2020] Rußwurm, M., Pelletier, C., Zollner, M., Lefèvre, S., and Körner, M. (2020). Breizhcrops: A time series dataset for crop type mapping. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ISPRS (2020).

[Sainte Fare Garnot et al., 2022] Sainte Fare Garnot, V., Landrieu, L., and Chehata, N. (2022). Multi-modal temporal attention models for crop mapping from satellite time series. ISPRS Journal of Photogrammetry and Remote Sensing.

[Zhao et al., 2024] Zhao, J., Wang, X., and Lei, J. (2024). Distributed Riemannian stochastic gradient tracking algorithm on the Stiefel manifold.