Image Credits: O'Reilly Media
Deep Learning, to a large extent, is really about solving massive nasty optimization problems. A Neural Network is merely a very complicated function, consisting of millions of parameters, that represents a mathematical solution to a problem. Consider the task of image classification. AlexNet is a mathematical function that takes an array representing RGB values of an image, and produces the output as a bunch of class scores.
This is a companion discussion topic for the original entry at https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/