A cost function is a scalar function that quantifies the error factor of the neural network. Lower the cost function better the neural network. For example, while classifying the image in the MNIST dataset, the input image is digit 2, but the neural network wrongly predicts it to be 3.
