Practices of Neural Network in Python

by Kardi Teknomo

This is the simple practice session of Neural Network in Python. By the end of this lesson, you should be able to reveal some of the limitation of the built-in modules in scikit-learn.

Let us start with a simple example of Boolean OR

In [1]:
from sklearn.neural_network import MLPClassifier
from sklearn.linear_model import Perceptron
import sklearn.metrics as metric
import numpy as np
In [2]:
X_training=[[1, 1], 
            [1, 0],
            [0, 1],
            [0, 0]
           ]
y_training=[1, 
            1,
            1,
            0
           ]
X_testing=X_training
y_true=y_training

Perceptron

The following code is the example of how you will use Perceptron Neural Network to train, predict and measure the accuracy of your prediction. You can also get the weights of the Neural Network.

In [3]:
ptn = Perceptron(max_iter=500)                     # set the method
ptn.fit(X_training, y_training)                    # training
y_pred=ptn.predict(X_testing)                      # prediction
print(y_pred)                                      # show the output
accuracy=metric.accuracy_score(y_true, y_pred, normalize=True)
print('acuracy=',accuracy)                         # show accracy score
[1 1 1 0]
acuracy= 1.0
In [4]:
print(ptn.intercept_, ptn.coef_)  # show the synapsis weights w0, w1, w2, ...
[-1.] [[ 2.  2.]]

Multi-Layer Perceptron (MLP)

The following code is the example of how you will use Multi-Layer Perceptron (MLP) Neural Network to train, predict and measure the accuracy of your prediction. You can also get the weights of the Neural Network.

In [5]:
mlp = MLPClassifier(solver='lbfgs', hidden_layer_sizes=(1,1), activation='logistic') # set the method
mlp.fit(X_training, y_training)                    # training
y_pred=mlp.predict(X_testing)                      # prediction
print(y_pred)                                      # show the output
accuracy=metric.accuracy_score(np.array(y_true).flatten(), np.array(y_pred).flatten(), normalize=True)
print('acuracy=',accuracy)                         # show accracy score
[1 1 1 0]
acuracy= 1.0
In [6]:
print([coef.shape for coef in mlp.coefs_])  # size of synapsis weights
mlp.coefs_                                  # synapsis weights
[(2, 1), (1, 1), (1, 1)]
Out[6]:
[array([[ 4.1486074 ],
        [ 4.14636493]]), array([[ 7.43468985]]), array([[ 15.53567128]])]

Experiments

  • Change the data (X_training, y_training, X_testing)
    • Change binary into bipolar {-1, +1}
  • Change the hidden layer of MLP: number of layers (int) or size (n,m)
  • Change solver of MLP: {'lbfgs', 'sgd', 'adam'}
  • Change activation function of MLP: {'identity', 'logistic', 'tanh', 'relu'}, default 'relu'

Tips

  • Check accuracy of prediction

Practice-1: Bipolar OR

Instead of {0, 1}, we can also change the value into bipolar {-1, +1}. Try the following OR gate using Perceptron and MLP.

In [7]:
X_training=[[1, 1], 
            [1, -1],
            [-1, 1],
            [-1, -1]
           ]
y_training=[1, 
            1,
            1,
            -1
           ]
X_testing=X_training
y_true=y_training

Practice-2: AND gate

Train and predict back the AND gate training set below using Perceptron and MLP.

In [8]:
X_training=[[1, 1], 
            [1, 0],
            [0, 1],
            [0, 0]
           ]
y_training=[1, 
            0,
            0,
            0
           ]
X_testing=X_training
y_true=y_training

Practice-3: Bipolar AND gate

Modify the above Boolean AND training data into bipolar. Use Perceptron and MLP to train and predict back this gate.

Practice-4: two output neurons

Try to use perceptron and MLP to train and predict the following dataset. What happen? Why do you think the error happen?

In [9]:
X_training=[[1, 1], 
            [1, 0],
            [0, 1],
            [0, 0]
           ]
y_training=[[1, 1], 
            [1, 1],
            [1, 1],
            [0, 1]
           ]
X_testing=X_training
y_true=y_training

Practice-5: XOR

Use Perceptron and MLP to solve the following Boolean Exclusive Or (XOR) problem. How to improve the accuracy? How many minimum hidden layer is needed to make it 100% accuracy?

In [10]:
X_training=[[1, 1], 
            [1, 0],
            [0, 1],
            [0, 0]
           ]
y_training=[1, 
            0,
            0,
            1
           ]
X_testing=X_training
y_true=y_training

Practice-6: Tautology

Train the Perceptron and MLP to solve the following Boolean Tautology (all 1). What happen?

In [11]:
X_training=[[1, 1], 
            [1, 0],
            [0, 1],
            [0, 0]
           ]
y_training=[1, 
            1,
            1,
            1
           ]
X_testing=X_training
y_true=y_training

Practice-7: Boolean XNOR

Use Perceptron and MLP to solve the following Boolean Exclusive Not Or (XNOR) problem. How to improve the accuracy? How many minimum hidden layer is needed to make it 100% accuracy?

In [12]:
X_training=[[1, 1], 
            [1, 0],
            [0, 1],
            [0, 0]
           ]
y_training=[0, 
            1,
            1,
            0
           ]
X_testing=X_training
y_true=y_training

Practice-8: Neural Network with 3 input and 2 output

Try to train the following data using neural network. Experiment by changing the solver, hidden layers and activation function to improve the accuracy.

In [13]:
X_training=[[ 1,  1,  0], 
            [ 1, -1, -1],
            [-1,  1,  1],
            [-1, -1,  1],
            [ 0,  1, -1],
            [ 0, -1, -1],
            [ 1,  1,  1]
           ]
y_training=[[1, 0], 
            [0, 1],
            [1, 1],
            [1, 0],
            [1, 0],
            [1, 1],
            [1, 1]
           ]
X_testing=X_training
y_true=y_training

Practice-9: Neural Network for Addition

Train the Perceptron and MLP to solve neural network addition problem. What happen? Why you cannot solve this kind of simple problem?

In [14]:
X_training=[[ 1,  1], 
            [ 1, 2.5],
            [-1,  1.5],
            [-1, -1],
            [ 0,  1],
            [ 0, -1],
            [ 1,  1]
           ]
y_training=[2, 
            3.5,
            0.5,
            0,
            1,
            -1,
            2
           ]
X_testing=X_training
y_true=y_training

Practice-10: Transportation Mode Choice

The following practice session comes from my Neural Network book. Suppose we have the following 10 rows of training data. The training data is supposed to be part of a transportation study regarding the mode choice to select bus, car or train among commuters along a major route in a city, gathered through a questionnaire study. For simplicity and clarity, we selected only 4 attributes. Attribute ‘gender’ is a binary type, while ‘car ownership’ is a quantitative integer. ‘Travel cost/km’ is a quantitative of ratio type but here it was converted into an ordinal type. ‘Income level’ is also an ordinal type. Train a neural network to predict the transport mode of a person, given the four attributes: gender, car ownership, travel cost, and income level. After training the neural network using the data above, try to predict the Transportation Mode Choice of the following instance of data: Female without car ownership, willing to pay expensive travel cost and having medium income level.

Training Data

Gender Car ownership Travel Cost Income Level Transportation mode
Male 0 Cheap Low Bus
Male 1 Cheap Medium Bus
Female 1 Cheap Medium Train
Female 0 Cheap Low Bus
Male 1 Cheap Medium Bus
Male 0 Standard Medium Train
Female 1 Standard Medium Train
Female 1 Expensive High Car
Male 2 Expensive Medium Car
Female 2 Expensive High Car

Notes

  • Perceptron in scikit-learn can only accept one output neuron
  • MLP can accept multiple output neurons
  • MLP in scikit-learn must have at least 1 hidden layer
  • Neural network in scikit-learn does not have any option to change the aggregation function aside from sum product.
  • MLP is used for classification problem. Use MLPRegressor if your problem is actually a regression problem.

last update: Nov 2017

Cite this tutorial as [Teknomo (2017) Practices Neural Network in Python] (http://people.revoledu.com/kardi/tutorial/NeuralNetwork/)

See Also: Python for Data Science

Visit www.Revoledu.com for more tutorials in Data Science

Copyright © 2017 Kardi Teknomo

Permission is granted to share this notebook as long as the copyright notice is intact.