The mathematical background

A commonly used mathematical function to describe an artificial neuron is the activation function. The activation function determines the output of an artificial neuron based on the weighted sum of the inputs. There are several types of activation functions used in practice. 

Some examples of activation functions:   

  1. Sigmoid function:The sigmoid function is a widely used activation function and was commonly used in early neuron models. It is given by the formula: f(x) = 1 / (1 + e^(-x)) The sigmoid function is S-shaped and produces values between 0 and 1. It is often used to apply non-linear transformations to the weighted sum of the inputs.

  2. Rectified Linear Unit (ReLU) function:The ReLU function is a popular activation function in deep neural networks. It is given by the formula: f(x) = max(0, x) The ReLU function simply returns the input value if it is positive and 0 if it is negative. It has the property of fast computation and can help avoid the vanishing gradient problem.

  3. Tanh function:The hyperbolic tangent (tanh) function is similar to the sigmoid function but produces values between -1 and 1. It is given by the formula: f(x) = (e^x - e^(-x)) / (e^x + e^(-x)) The tanh function is S-shaped and symmetric around the origin. It can be used in neuron models where both positive and negative values are important.

Sigmoid function

ReLU function

Tanh function

The choice of the activation function depends on the specific problem and the desired behavior of the artificial neuron. There are also other activation functions available, such as the softmax function, Leaky ReLU, and ELU (Exponential Linear Unit). Selecting the appropriate activation function is an important part of designing and training neural networks.

Next page