Implementing gradient descent in Python

Automatic differentiation is one of the core functions of deep Learning libraries such as Tensorflow and Pytorch. To facilitate the training of the neural networks, it is required to memorize the evaluation of all the neurons in the forward pass, and in the backward pass, it allows the computation of the gradients and updates of model parameters recursively.

In this video, we show how to write the gradient descent algorithm from scratch in Tensorflow. We fit a linear model from a constructed dataset with this algorithm. In the end, we introduce the high-level API provided in Tensorflow that concisely packs this procedure.