EPSRC Reference: 
EP/Y029089/1 
Title: 
DMSEPSRC: Asymptotic Analysis of Online Training Algorithms in Machine Learning: Recurrent, Graphical, and Deep Neural Networks 
Principal Investigator: 
Sirignano, Professor J 
Other Investigators: 

Researcher CoInvestigators: 

Project Partners: 

Department: 
Mathematical Institute 
Organisation: 
University of Oxford 
Scheme: 
Standard Research  NR1 
Starts: 
01 March 2024 
Ends: 
28 February 2027 
Value (£): 
409,469

EPSRC Research Topic Classifications: 
Artificial Intelligence 
Mathematical Analysis 
Statistics & Appl. Probability 


EPSRC Industrial Sector Classifications: 

Related Grants: 

Panel History: 

Summary on Grant Application Form 
Neural network models in machine learning have achieved immense practical success over the past decade, revolutionizing fields such as image, text, and speech recognition. Neural networks have also become widelyused in science, engineering, medicine, and finance. In particular, deep learning, which uses multilayer neural networks, has transformed the field of machine learning. The training algorithms used for these complex machine learning problems  although successful in practice  are often ad hoc. Mathematical theory is yet to be established in many cases, and there is the potential to improve training algorithms and models via rigorous mathematical analysis. The primary purpose of this research is to develop new mathematical theory for the training algorithms and neural network models used in several key areas of machine learning. The problems in this project are motivated by both fundamental mathematical questions and questions highly relevant to applications. Developing and testing mathematical theory for widelyused training algorithms is crucial for ensuring their reliability and guaranteeing their performance in practice. The successful conclusion of this proposal's research program will contribute to the machine learning community by developing new mathematical methods which can be broadly used for the analysis of neural networks and machine learning algorithms. This research project is integrated with an educational programme which will contribute to the training of students in the mathematical foundations of machine learning and deep learning.
Our proposed research will develop a rigorous mathematical analysis for the training algorithms and neural network models used in several important areas of machine learning, including: recurrent neural networks, reinforcement learning, graph neural networks, and deep neural networks. Our analysis will characterize the asymptotic behavior of these algorithms and neural network models as the number of training steps and the number of hidden units in the neural network go to infinity. The researchers will develop new mathematical methods designed for the analysis of neural networks by leveraging methods from stochastic analysis and weak convergence theory to study the asymptotics of online, stochastic training algorithms and neural network models as the number of hidden units becomes large. Applications of the mathematical results to parameter initialization, hyperparameter selection, design of optimization/training algorithms, and the selection of model architectures will be investigated. The research project is highly interdisciplinary, leveraging methods from probability, partial differential equations, large deviations theory, stochastic analysis, optimization, and machine learning. In addition to proving convergence theory for important neural network training algorithms, the research will be of broader interest outside of machine learning as it will study a new set of meanfield problems with novel and mathematically challenging features.

Key Findings 
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk

Potential use in nonacademic contexts 
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk

Impacts 
Description 
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk 
Summary 

Date Materialised 


Sectors submitted by the Researcher 
This information can now be found on Gateway to Research (GtR) http://gtr.rcuk.ac.uk

Project URL: 

Further Information: 

Organisation Website: 
http://www.ox.ac.uk 