By Kardi Teknomo, PhD .

NN

Share this: Google+
< Previous | Next | Contents >

Activation Function of Neural Network

Once a computational neuron receive the inputs from other neurons the neuron aggregates the input values and the synapses weights values into a single valueActivation Function.  After that, they neuron utilize this aggregated value Activation Function into the output value of the neuron Activation Function through activation function. Thus, activation function is the way a neuron utilize the aggregation values from other neurons.

Similar to aggregation function, you can practically define any function to be your activation function.

If we use a threshold function type such as threshold step or binary step or bipolar step function, a neuron cell will be activated only if the value of the sum product is greater than a threshold value. For generality, we can write the output as a non-linear function of the sum product of the weights and the inputs.

Activation Function

(5)

When the inputs of a neuron are binary values, the most common activation function of perceptron is a binary step function as in equation (6) or threshold step function as in equation (7).

Activation Function

(6)

For binary step function, when the value of the sum product is positive or zero, the result of the activation function is one. When the value of the sum product is negative, the result of the activation function is zero. For threshold step function, when the value of the sum product is positive, the result of the activation function is one. When the value of the sum product is zero or negative, the result of the activation function is zero.

Activation Function

(7)

The graph of binary step function and threshold step function are shown below.

Activation Function

Figure Binary Step and Threshold Step Function

Example

Suppose we the sum product of the synapses weights and the value of the sensory cells of a neuron yieldsActivation Function. What would be the output value of the neuron given that the activation function is a binary step function?

Answer:

Activation Function

The value of Activation Function, thus Activation Function (positive) and therefore the output of the neuron is theActivation Function.

Example

A single layer neural network has two neuron cells and one input cell as shown in the figure below. The value of the weights areActivation Function and the bias weights areActivation Function. What are the outputs of the neural network when the input value is 1? What are the output values when the input value is 0? Use threshold step as the activation function.

Activation Function

Answer:

First get the sum product and then compute the activation function of each neuron to get the output.

When the input value is 1, we setActivation Function

Activation Function

Activation Function

Activation Function

Activation Function

When the input value is 0, we setActivation Function

Activation Function

Activation Function

Activation Function

Activation Function

The following table is summarized the input and output of this neural network.

input

Output (activation results)

Activation Function

Activation Function

Activation Function

Activation Function

You can see that a neural network is defined by its weights and its network architecture.

When the inputs of a neuron are continuous values between [0, 1] as in probability values the most common activation function is a sigmoid function type.  Sigmoid function has shape of an S curve, as you have learned in table 1, it can be logistic, inverse tangent or hyperbolic tangent function.

Activation Function

(8)

The graph of the sigmoid function is similar to S-curve as shown below. The parameterActivation Function is often called the temperature of a neuron. The higher the temperature, the more steep the values of the sigmoid function. The lower the temperature, the curve moves gentler as the value of s changes. In practice, we set the temperature parameter by looking at the range value ofActivation Function. For instance if the range of values of s is between -1 to +1, setting the value of beta to 10 will give smoother S-curve while setting the value of beta to 0.1 or 0.5 will give linear functions with different slope.

SigmoidSigmoid

Sigmoid Logistic Function

< Previous | Next | Contents >

Give your feedback and rate this tutorial

Read it off line on any device. Click here to purchase the complete E-book of this tutorial

See Also :
K means clustering , Similarity Measurement , Reinforcement Learning (Q-Learning) , Discriminant Analysis , Kernel Regression , Clustering , Decision Tree

This tutorial is copyrighted .

Preferable reference for this tutorial is

Teknomo, Kardi (2017). Neural NetworkTutorial. http:\\people.revoledu.com\kardi\tutorial\NeuralNetwork\