The Mathematics of Deep Neural Networks

Abstract

Machine learning models built with deep neural networks (DNN) have gained immense popularity in recent years. However successful they might seem, there is a substantial collective blind spot when it comes to a rigorous understanding of these models, their practical limitations, and the explainability and credibility of the results they generate. Establishing a theoretical framework to ensure the robustness of deep learning algorithms is one of the most active research areas in applied mathematics. The first step along this research path is understanding how these models "think" and "reason", which -unlike how humans think- can entirely be described in mathematical terms. The aim of this thesis is to present the mathematics underlying neural network models and deep learning algorithms. It covers the mathematical foundation of artificial neurons and feed-forward neural networks and presents important mathematical results that support their expressive power and approximation capabilities. The thesis then focuses on the learning algorithm used to train these models and covers the main challenges these algorithms face. The thesis also presents the foundation of more advanced DNNs, highlighting the diversity of current architectures. Finally the thesis examines what is meant by the "black box" nature of these models and their vulnerability to adversarial attacks. By examining how these models process data and make decisions, this thesis aims to emphasize the need for rigorous scrutiny and expert involvement when developing, testing and employing DNN models. Keywords: Machine Learning, Deep Neural Network, Learning Algorithm, Stochastic Gradient Descent, Backpropagation, Interpretability, Adversarial Attack.