In my previous article, Neural Network Backpropagation, I explained the basics of how an artificial neural network learns. The cost function calculates how wrong the network's answer is, and then backpropagation goes back through the various layers of the neural network changing weights and biases to adjust the strengths of the connections between neurons and the calculations within certain neurons. I compare the process to that of turning the dials on a complex sound system to optimize the quality of sound being emitted by the speakers.

In that article, I focus mostly on how fine tuning a neural network with back propagation improves its accuracy — to arrive at the correct answer. Although coming up with the correct answer is certainly the top priority, backpropagation must also adjust the weights and biases to reduce the output that’s driving the wrong answers:

• First, it adjusts the connections and neurons that feed output to the neurons that are most confident in their wrong answers.
• Then, it moves on to adjust the inputs to neurons that are less confident in their wrong answers.

Learning as an Iterative Process

Keep in mind that the artificial neural network makes adjustments for every item in the training data set. In my previous post, I used an example of an artificial neural network for identifying dog breeds. To properly train the network to identify ten different dog breeds, the initial training data set would need at least one picture of a dog of each breed: German shepherd, Labrador retriever, Rottweiler, beagle, bulldog, golden retriever, Great Dane, poodle, Doberman, and dachshund. 

This is a very small dataset, but what is important is that it contains at least one picture of each breed. If you fed the network only the picture of the beagle, you would end up with a neural network that classifies every image of a dog as a beagle. You need to feed the network more pictures — pictures of dogs of different breeds and pictures of different dogs of the same breed.

You would follow up the initial training with a test data set — a separate collection of at least one picture of a dog of each breed. During this test, you would feed a picture of a dog to the neural network without telling it the dog's breed and then check its answer. If the answer was wrong, you would correct the neural network, and it would use backpropagation to make additional adjustments. You would then feed it the next picture, and continue the process, working through the entire set of test data.

Even after testing, the neural network continues to learn as you feed it larger volumes of data. With every picture you feed into the network, it uses backpropagation to make tiny adjustments to weights and biases, fine-tuning its ability to distinguish between different breeds.

Overall Improvement

While the example from my previous post focused on improving the network's ability to identify a picture of a beagle, you want the network to achieve the same degree of accuracy for each and every breed. To achieve this feat, the neural network needs to make some trade-offs. 

As you feed the network a diversity of pictures, it becomes a little less accurate in identifying beagles, so that it can do a better job identifying pictures of Labrador retrievers, Rottweilers, German shepherds, poodles, and so on. Your network tries to find the optimal weights and biases to minimize the cost regardless of the breed shown in the picture. The settings may not be the best for any one dog, but having well-balanced settings enables the network to make fewer mistakes, resulting in more accurate classification among different breeds of dogs overall.

The cost function, gradient descent, and backpropagation all work together to make this magic happen.

Creating a Model

Although machine learning is a continuous process of optimizing accuracy, the goal of training and testing a neural network is to create a model — the product of the entire machine learning process that can be used to make predictions. Creating a model is actually a joint effort of humans and machines. A human being builds the artificial neural network and establishes hyperparameters to set the stage for the learning process; the machine then takes over, adjusting parameters, including weights and biases, to develop the desired skill.

As you begin to build your own neural networks, keep in mind that the process often involves some experimentation. Your first attempts may involve small experiments to test outcomes followed by adjustments to hyperparameters. In a sense, both you and the machine are involved in a learning process.