Artificial Neural Networks

Neuron

Neural networks: historical background

In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts portrayed the workings of a neural network using a simple electrical circuit. They drew inspiration from the operation of biological neural networks in the brains of humans and animals. Their model of the neuron took various "inputs" and turned them into a weighted sum. As output, the neuron gave a 0 if the result was below a certain threshold, a 1 if above it.

In his book The Organization of Behavior (1949), Donald Hebb elaborated on the idea. He argued that neural pathways strengthen with each successive use, especially between neurons "firing" at the same time. Thus began the long start toward trying to understand and model the complex processes in the brain. Hebb's famous statement, "Cells that fire together, wire together," formed the basis of so-called "Hebbian learning. Threshold logic ('treshold logic'), which converts continuous input into 'discrete' (in distinct packets) output, was also an important precursor to today's artificial neural networks.

The Massachusetts Institute of Technology better known as MIT first successfully implemented a Hebbian network in a computer system in 1954. Around that time, Frank Rosenblatt, a research psychologist at the Aeronautical Laboratory at Buffalo, was trying to figure out which "input" in a fly's eye determined in which direction the insect would fly. In his attempt to quantify that "decision system" mathematically, he presented the Mark I Perceptron in 1958. He based it on the neuron of McCulloch and Pitts. In the perceptron, the weights of each input are "learned" by passing the "inputs" in succession. Rosenblatt connected his machine to a camera with 20x20 cadmium sulfide photocells, allowing him to produce an image of four hundred pixels. In a Navy demonstration for journalists, the perceptron learned to distinguish left from right after fifty attempts.

The New York Times reported that this "brain" was able to remember images and information that it had observed for itself. Ordinary computers, the journalist claimed, were only capable of "remembering" what they were fed via punch cards or magnetic tapes. In an interview with the same newspaper, the enthusiastic Rosenblatt dreamed aloud about perceptrons that would be sent to other planets as "mechanical space explorers. 'At first,' Rosenblatt stated, 'the Perceptron will make wrong decisions, but it will become wiser as it gains more experience.' The laboratory staff dreamed even further ahead:

'The service said it will use this principle to build the first of its Perceptron thinking machines that can read and write, expected to be ready in about a year for a hundred thousand dollars.'

In 1959, Bernard Widrow and Marcian Hoff at Stanford developed the first neural network to be successfully applied to a real-world problem. ADALINE and MADALINE, so named because of their use of Multiple ADAptive LINear Elements, eliminated noise in telephone lines and are still in use today. Compared to the earlier perceptrons, that neural network worked with "weighted" input data.

The principles underlying the perceptron have contributed to the modern revolution in artificial intelligence. 'Deep learning' and neural networks classify online images and translate texts from one language to another.

Artificial neural networks

Artificial intelligence means that computer systems learn by themselves to arrive at solutions. We program software to learn on its own through experience or by example, just as humans and even animals do. Computer scientists use a variety of learning algorithms to do this. For example, they try to simulate the functioning of neurons in the human brain with software. We speak of a "neural network" in this case. Most things in our lives we too learn by examples from others. Even in school, we have to learn from examples and mistakes.

By programming neural networks into software, computers can learn on their own ("machine learning"). We first feed the AI software with training sets (or data sets). For example, these could be a series of pictures of dogs. By feeding a mass of pictures of dogs as training sets to the software, the neural network gradually learns to recognize dogs in other pictures by itself. A key difference is that a human needs fewer examples than an AI algorithm. A child who has seen a cat a few times will quickly recognize other cats as well. AI needs a mass of examples. After all, a cat can look very different, as there are dozens of breeds, for example. In addition, you also need to be able to recognize a cat from all possible "angles" (top view, back view...) and among other objects in an image. For this, an AI must also be able to segment or distinguish various objects from each other. After all, when you show an image or webcam image to an AI, that input data consists only of a series of pixels with RGB color values. So the AI has to look for patterns in pixel data.

Weights

In our lives, we must constantly make decisions. In doing so, we weigh numerous factors. What clothes will we wear today? What gift will I buy? What am I going to eat tonight? Who will I vote for in the elections? Here you make a lot of trade-offs and some factors weigh more heavily in your final decision.

In the end, you add up all those factors. Some, however, will weigh more heavily. In this way, a neural network (whether a bunch of neurons in our brains or an artificial neural network) gets a lot of 'inputs' (trade-offs), each of which gets its own weight. The network "calculates" a decision and sends it out as an "output.

Just because you feed an extensive data set/training set to your AI algorithm does not mean that it will work flawlessly from then on. An AI algorithm can learn from its mistakes. At this stage, human intervention still matters. Humans can still check for themselves whether the algorithm's decision is correct. We speak of supervised learning or "supervised learning" or ... supervised learning.

An example: a spam filter

A spam filter is a well-known example of AI software based on a neural network. A lot of emails that contain the word "casino" and "penis enlargement" will be spam messages. But suppose your friend sends you an email saying "Are you going to the casino with me tonight?", you don't want this message to end up in spam. Or if you have real problems and your doctor sends you an email about your reproductive organ with the phrase "We're going to extend your sick penis" (Fortunately, I do not receive this kind of message from my doctor).  you don't want the combination of "extend" and "penis" in the same message to lead to spam. The neural network has to make a lot of tradeoffs to decide whether or not a message should be marked as spam.

You build such a spam filter by compiling a string of suspicious words. Messages containing one or more of these words must be recognized as spam. Those suspicious words are classified as 'true negative' (no spam) in one message, and 'true positive' (definitely spam) in another. Other terms may lead to the classification 'false positive' or 'false negative'. This happens when a spam message unexpectedly ends up in your regular mailbox anyway, or a good message ends up in your spam folder. In that case, you can self-classify a message. At that point, the spam filter learns that it made a mistake. It will take that into account in the future. Spam filters get better and better that way and make fewer and fewer mistakes.

Therefore, you can say that AI software literally makes predictions based on what the software has previously learned. Those predictions can be okay ("true positive," "true negative"), but equally very wrong ("false positive" or "true negative"). In the latter case, the algorithm must be adjusted.

In supervised learning, the algorithm learns from examples. In reinforcement learning, you empower the algorithm by rewarding or punishing it. If the predicted output is wrong or incomplete or leads to the wrong result, the AI software can 'learn' from its own mistakes. Suppose a robot is driving around in a room that has only one exit, that robot may try to get out by randomly bumping into the wall until it finds the opening. But once the robot has discovered the door, then it can 'remember' this.

In unsupervised learning ("unsupervised learning"), the algorithm is not given examples, but must classify (categorize) elements on its own based on common characteristics. Thus, in unsupervised learning, the algorithm is not given examples or labels in advance. The algorithm must classify the information itself based on features. Suppose you have to do laundry for the first time in your life. You see a pile of unwashed clothes in front of you. You now have to classify the laundry based on a number of characteristics. Classifying by color and textile type would be the desired output. But that is what the "algorithm" now needs to discover. If you use the K-means algorithm, you can only specify in advance how many groups you want. The rest the algorithm has to discover for itself.

In humans, a combination of supervised and unsupervised learning often occurs. A child learning to walk does get help from adults (they supervise or give a 'helping hand'), but which muscles the toddler has to use, the toddler literally learns that 'by himself' without supervision.

Often programmers try to recreate a piece of the human brain. They use codes to recreate a brain cell (neuron), a so-called "artificial neural network. Sometimes artificial intelligence learns by comparing. When you watch a series on Netflix, Netflix compares your preferences and selections with those of other people who watched the same series. Netflix then recommends series that those other people have also watched. Artificial intelligence learns by calculating probabilities. For example: you have a 50 percent chance that the first student you meet at school will be a boy and a 50 percent chance that it will be a girl. But in a nunnery, there is almost a hundred percent chance that you will run into a woman. A computer wouldn't know that because it doesn't know that context. If you give that information to the AI software, it will make a more correct prediction.

ANNs - artificial neural networks

Artificial Neural Networks or ANNs are statistical models inspired and "modeled" after biological neural networks.

Artificial Neural Networks or ANNs are statistical models inspired and "modeled" after biological neural networks. Human brains are estimated to contain about one hundred billion neurons. Those neurons form networks that exchange information and can also 'learn' from it. Exactly how it all works is not yet entirely clear, but 'learning' things would happen by strengthening connections between neurons. Decisions (from 'slapping someone in the face' to 'choosing between fries and shoarma') are made based on the inputs neurons receive. The output from the "network" forms the final decision.

ANNs are capable of making nonlinear connections between input data and output (the decision).

ANNs are capable of making nonlinear connections between input data and output (the decision). Roughly speaking, this means that the result is not always a choice for or against (a binary decision), but a weighted estimate. The final decision is the result of summing multiple inputs and the weight the network gives to each input. ANNs use a variety of algorithms and can also take multiple forms ("architectures"). To list a few: Feed-forward neural networks, Recurrent neural network, Multi-layer perceptrons (MLP), Convolutional neural networks, Recursive neural networks, Deep belief networks, Convolutional deep belief networks, Self-Organizing Maps, Deep Boltzmann machines, Stacked de-noising auto-encoders, Backpropagation, Gradient descent algorithm, Hebbian learning Rule, Self - Organizing Kohonen Rule, Hopfield Network Law, LMS algorithm (Least Mean Square), Competitive Learning...

Neural networks can contain multiple layers. We then speak of a "multilayer perceptron.

Neural networks can contain multiple layers. We then speak of a "multilayer perceptron. Each layer in such a network tries, for example, to recognize patterns in an image. When a pattern is found, the algorithm activates the next "hidden" layer. For example, the first layer recognizes edges in the image. The next layer combines the found edges, and so on. The more layers such a network contains (the deeper the network), the better the pattern recognition. Each layer in the ANN assigns weights (or enlarges or reduces them) each time new data (in this case, new images) enters the network, and that helps determine the input of the next layer.

Deep learning

When multiple "multilayer perceptrons" are combined, it is called "deep learning. 'Deep learning' breaks down a problem into multiple layers.

When multiple "multilayer perceptrons" are combined, it is called "deep learning. 'Deep learning' splits a problem into multiple layers. For example, the first layer recognizes certain shapes (circles, triangles...). The second layer then identifies, for example, the 'eyes' (two round or oval shapes next to each other). The third layer then recognizes the face.... Eventually, the algorithm can recognize the image of a human being.