Details of Grant 

EPSRC Reference:	EP/Y029089/1
Title:	DMS-EPSRC: Asymptotic Analysis of Online Training Algorithms in Machine Learning: Recurrent, Graphical, and Deep Neural Networks
Principal Investigator:	Sirignano, Professor J
Other Investigators:
Researcher Co-Investigators:
Project Partners:	Boston University
Department:	Mathematical Institute
Organisation:	University of Oxford
Scheme:	Standard Research - NR1
Starts:	01 March 2024	Ends:	28 February 2027	Value (£):	409,469
EPSRC Research Topic Classifications:	Artificial Intelligence Mathematical Analysis Statistics & Appl. Probability
EPSRC Industrial Sector Classifications:	Information Technologies
Related Grants:
Panel History:
Summary on Grant Application Form
Neural network models in machine learning have achieved immense practical success over the past decade, revolutionizing fields such as image, text, and speech recognition. Neural networks have also become widely-used in science, engineering, medicine, and finance. In particular, deep learning, which uses multi-layer neural networks, has transformed the field of machine learning. The training algorithms used for these complex machine learning problems -- although successful in practice -- are often ad hoc. Mathematical theory is yet to be established in many cases, and there is the potential to improve training algorithms and models via rigorous mathematical analysis. The primary purpose of this research is to develop new mathematical theory for the training algorithms and neural network models used in several key areas of machine learning. The problems in this project are motivated by both fundamental mathematical questions and questions highly relevant to applications. Developing and testing mathematical theory for widely-used training algorithms is crucial for ensuring their reliability and guaranteeing their performance in practice. The successful conclusion of this proposal's research program will contribute to the machine learning community by developing new mathematical methods which can be broadly used for the analysis of neural networks and machine learning algorithms. This research project is integrated with an educational programme which will contribute to the training of students in the mathematical foundations of machine learning and deep learning. Our proposed research will develop a rigorous mathematical analysis for the training algorithms and neural network models used in several important areas of machine learning, including: recurrent neural networks, reinforcement learning, graph neural networks, and deep neural networks. Our analysis will characterize the asymptotic behavior of these algorithms and neural network models as the number of training steps and the number of hidden units in the neural network go to infinity. The researchers will develop new mathematical methods designed for the analysis of neural networks by leveraging methods from stochastic analysis and weak convergence theory to study the asymptotics of online, stochastic training algorithms and neural network models as the number of hidden units becomes large. Applications of the mathematical results to parameter initialization, hyperparameter selection, design of optimization/training algorithms, and the selection of model architectures will be investigated. The research project is highly interdisciplinary, leveraging methods from probability, partial differential equations, large deviations theory, stochastic analysis, optimization, and machine learning. In addition to proving convergence theory for important neural network training algorithms, the research will be of broader interest outside of machine learning as it will study a new set of mean-field problems with novel and mathematically challenging features.
Key Findings
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
Potential use in non-academic contexts
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
Impacts
Description This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk Summary Date Materialised
Sectors submitted by the Researcher
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk
Project URL:
Further Information:
Organisation Website:	http://www.ox.ac.uk