Share this:
Google+

< Previous | Next | Contents >

## Activation Function of Neural Network

Once a computational neuron
receive the inputs from other neurons the neuron aggregates the input values
and the synapses weights values into a single value \( s \). After that, they neuron utilize this
aggregated value \( s \) into the output value of the neuron \( y \) through *activation
function*. Thus, activation function is the way a neuron utilize the
aggregation values from other neurons.

Similar to aggregation function, you can practically define any function to be your activation function.

If we use a threshold function type such as threshold step or binary step or bipolar step function, a neuron cell will be activated only if the value of the sum product is greater than a threshold value. For generality, we can write the output as a non-linear function of the sum product of the weights and the inputs.

$$ y = f(s) $$ | (5) |

When the inputs of a neuron are
binary values, the most common activation function of perceptron is a *binary step function* as in equation (6)
or *threshold step function* as in
equation (7).

$$ f(s)=\left\{\begin{matrix} 1 \quad if s\geq 0 \\ 0 \quad if s< 0 \end{matrix}\right. $$ | (6) |

For binary step function, when the value of the sum product is positive or zero, the result of the activation function is one. When the value of the sum product is negative, the result of the activation function is zero. For threshold step function, when the value of the sum product is positive, the result of the activation function is one. When the value of the sum product is zero or negative, the result of the activation function is zero.

$$ f(s)=\left\{\begin{matrix} 1 \quad if s > 0 \\ 0 \quad if s\leq 0 \end{matrix}\right. $$ | (7) |

The graph of binary step function and threshold step function are shown below.

Figure Binary Step and Threshold Step Function

### Example

Suppose we the sum product of the synapses weights and the value of the sensory cells of a neuron yields \( s=0.25 \). What would be the output value of the neuron given that the activation function is a binary step function?

*Answer*:

The value of \( s=0.25 \), thus \( s>0 \) (positive) and therefore the output of the neuron is \( y=f(s)=f(0.25)=1 \).

### Example

A single layer neural network has two neuron cells and one input cell as shown in the figure below. The value of the weights are \( w_{12}=-2,w_{13}=-0.5 \) and the bias weights are \( w_{02}=1,w_{03}=1.5 \). What are the outputs of the neural network when the input value is 1? What are the output values when the input value is 0? Use threshold step as the activation function.

*Answer*:

First get the sum product and then compute the activation function of each neuron to get the output.

When the input value is 1, we set \( x_{1}=1 \)

$$ s_{2}=w_{02} x_{0} + w_{12} x_{1} = 1\cdot 1+ (-2)\cdot 1 = 1-2 =-1 $$ $$ y_{2}=f(s_{2})=f(-1)=0 $$ $$ s_{3}=w_{03} x_{0} + w_{13} x_{1} = 1.5 \cdot 1 + (-0.5) \cdot 1 =1.5-0.5=1 $$ $$ y_{3}=f(s_{3})=f(1)=1 $$When the input value is 0, we set \( x_{1}=0 \)

$$ s_{2}=w_{02} x_{0} + w_{12} x_{1} = 1 \cdot 1 + (-2)\cdot 0 = 1-0 = 1 $$ $$ y_{2}=f(s_{2})=f(1)=1 $$ $$ s_{3}=w_{03} x_{0} + w_{13} x_{1} = 1.5 \cdot 1 + (-0.5) \cdot 0 =1.5-0=1.5 $$ $$ y_{3}=f(s_{3})=f(1.5)=1 $$The following table is summarized the input and output of this neural network.

input |
Output (activation results) |

$$ x_{1}=1 $$ | $$ y_{2}=0, y_{3}=1 $$ |

$$ x_{1}=0 $$ | $$ y_{2}=1, y_{3}=1 $$ |

You can see that a neural network is defined by its weights and its network architecture.

When the inputs of a neuron are
continuous values between [0, 1] as in probability values the most common
activation function is a *sigmoid function
type*. Sigmoid function has shape of
an S curve, as you have learned in table 1, it can be logistic, inverse tangent
or hyperbolic tangent function.

$$ f(s)=\frac{1}{1+exp(-\beta s)},\beta >0 $$ | (8) |

The graph of the sigmoid function
is similar to S-curve as shown below. The parameter \( \beta \) is often called the *temperature* of a neuron. The higher the temperature, the more steep
the values of the sigmoid function. The lower the temperature, the curve moves
gentler as the value of s changes. In practice, we set the temperature
parameter by looking at the range value of \( s \).
For instance if the range of values of s is between -1 to +1, setting the value
of beta to 10 will give smoother S-curve while setting the value of beta to 0.1
or 0.5 will give linear functions with different slope.

Sigmoid Logistic Function

< Previous | Next | Contents >

Read it off line on any device. Click here to purchase the complete E-book of this tutorial

**
See Also
**
:

K means clustering
,
Similarity Measurement
,
Reinforcement Learning (Q-Learning)
,
Discriminant Analysis
,
Kernel Regression
,
Clustering
,
Decision Tree

This tutorial is copyrighted .

**
Preferable reference for this tutorial is
**

Teknomo, Kardi (2019). Neural NetworkTutorial. https:\\people.revoledu.com\kardi\tutorial\NeuralNetwork\