About

One of the funding schemes in the Horizon Europe framework programme is the MSCA – Staff Exchanges. It is predominantly a mobility scheme and the projects include beneficiaries from Europe and associated partners from non-EU countries. The main activity comes in the form of secondments, with research visits of duration 1-12 months between the partners. The project REMODEL – Research Exchanges in the Mathematics of Deep Learning with Applications had kickoff on February 1, 2024. The partners in this project are: NTNU, Norway, TU Eindhoven, The Netherlands, Bath University and Cambridge University, UK, Emory University, USA, Simon Fraser University, Canada, and Kobe University, Japan. 

The proposal was submitted in 2023 with NTNU as coordinator and TU/e, the University of Bath and the University of Cambridge as partner beneficiaries. However, during the summer of 2023 and after the proposal was approved, it become clear that the EU and the UK weren’t going to finalise an agreement about research funding after Brexit. As a consequence, the UK partners changed their status from beneficiaries to associated partners, meaning that they would receive no funding from the EU. Fortunately, the UK funding agency, UKRI, came to our rescue and provided the funding that allows the full participation of the UK participants to this programme, ensuring that all the research plans and secondments can be pursued and fulfilled exactly as originally planned.

The subject of this exchange programme is “mathematical aspects of deep learning algorithms and their applications”. We will address several questions related to the mathematical foundations of neural networks and our interdisciplinary and inter-sectoral team will design test problems and validate the research results obtained.

The impact of neural networks and deep learning in recent years has been profound and unprecedented. The number and variety of applications of the methods have skyrocketed over just a few years, and this technology has become an almost indispensable ingredient in a steadily increasing number of products that are important in intelligent industrial automation and in the everyday life of people.

But in the wake of the vast progress in this area, several questions and concerns have been raised about the robustness, reliability, accuracy, reproducibility and feasibility of neural networks. For example, in classification problems it is well-known that small perturbations to an image can result in dramatic changes to the output [4] with hazardous consequences for the user of the system [5]. Similarly, if a neural network has been trained to classify objects with a given fixed orientation, it may generalise poorly when executed on objects which have been rotated compared to the training data. Such types of problems have recently been addressed successfully by using principles from numerical analysis and geometry. The first example can be studied as a problem of stability [6], an important insight is that a neural network can be understood and analysed by means of an associated dynamical system. The second example has been tackled by means of equivariant neural networks [1], where the key is to design the network in such a way that its behaviour is not influenced by certain transformations to the data. Improvements to robustness and reliability of neural networks usually come at a price, such as reduced expressivity, accuracy or efficiency. Thus, the use of advanced mathematical methods for the design and analysis of neural networks is still in its infancy.

It is widely recognised that the mathematical sciences, including statistics, are a key enabling technology in many aspects of machine learning [2], not the least to resolve some of the above mentioned concerns. Mathematics and mathematical language and formalism can bring more rigour and precision to the understanding and analysis of the deep learning methodology.

In the last few years, there has been a significantly increased interest in applying deep learning methods to physical simulations, and to discover the underlying mathematical model [7]. However, most of the work in this area has been limited to “proof of concept” and has not been applied to practical problems. Deep learning methods for dynamics discovery are not well-developed for partial differential equations. Existing deep learning methods for model discovery typically assume that the state of the system can be fully observed. Often, only partially observed data are available in practice, and in some cases the data domain is not known a priori. An alternative to replacing physical models by neural networks to lower the computational cost is to make use of reduced order modelling, and reduced basis methods, and this can also be combined with machine learning methods [3].

The overarching objective of this proposal is to understand, study, prove, and test the properties of deep learning algorithms using principles from dynamical systems, geometry and optimisation theory. 

References

[1] CELLEDONI E, EHRHARDT MJ, ETMANN C, R. I. MCLACHLAN, B. OWREN,C.-B. SCHONLIEB and F. SHERRY, Structure-preserving deep learning. European Journal of Applied Mathematics. 2021;32(5):888-936. doi:10.1017/S0956792521000139

[2] Wil Schilders, Key enabling technology for scientific machine learning, link

[3] Theron Guo, Ondřej Rokoš, Karen Veroy, Learning constitutive models from microstructural simulations via a non-intrusive reduced basis method, Computer Methods in Applied Mechanics and Engineering, 2021. https://doi.org/10.1016/j.cma.2021.113924

[4] V Antun, F Renna, C Poon, B Adcock, AC Hansen, On instabilities of deep learning in image reconstruction and the potential costs of AI, Proceedings of the National Academy of Sciences, 2020. https://doi.org/10.1073/pnas.190737711

[5] N Papernot, P McDaniel, I Goodfellow, S Jha, ZB Celik, and A Swami. 2017. Practical Black-Box Attacks against Machine Learning. In Proc of the 2017 ACM on Asia Conf. on Comput. Comm. Security. Assoc. Comput. Mach., New York, NY, USA, 506–519.

[6] E Haber and L Ruthotto, Stable architectures for deep neural networks, 2018, Inverse Problems 34 014004 DOI 10.1088/1361-6420/aa9a90

[7] Takashi Matsubara, Ai Ishikawa, Takaharu Yaguchi, Deep Energy-based Modeling of Discrete-Time Physics, Part of Advances in Neural Information Processing Systems 33 (NeurIPS 2020).