Practices of Neural Network in Python

This is the simple practice session of Neural Network in Python. By the end of this lesson, you should be able to reveal some of the limitation of the built-in modules in scikit-learn.

Let us start with a simple example of Boolean OR

In [1]:
from sklearn.neural_network import MLPClassifier
from sklearn.linear_model import Perceptron
import sklearn.metrics as metric
import numpy as np

In [2]:
X_training=[[1, 1],
[1, 0],
[0, 1],
[0, 0]
]
y_training=[1,
1,
1,
0
]
X_testing=X_training
y_true=y_training


Perceptron¶

The following code is the example of how you will use Perceptron Neural Network to train, predict and measure the accuracy of your prediction. You can also get the weights of the Neural Network.

In [3]:
ptn = Perceptron(max_iter=500)                     # set the method
ptn.fit(X_training, y_training)                    # training
y_pred=ptn.predict(X_testing)                      # prediction
print(y_pred)                                      # show the output
accuracy=metric.accuracy_score(y_true, y_pred, normalize=True)
print('acuracy=',accuracy)                         # show accracy score

[1 1 1 0]
acuracy= 1.0

In [4]:
print(ptn.intercept_, ptn.coef_)  # show the synapsis weights w0, w1, w2, ...

[-1.] [[ 2.  2.]]


Multi-Layer Perceptron (MLP)¶

The following code is the example of how you will use Multi-Layer Perceptron (MLP) Neural Network to train, predict and measure the accuracy of your prediction. You can also get the weights of the Neural Network.

In [5]:
mlp = MLPClassifier(solver='lbfgs', hidden_layer_sizes=(1,1), activation='logistic') # set the method
mlp.fit(X_training, y_training)                    # training
y_pred=mlp.predict(X_testing)                      # prediction
print(y_pred)                                      # show the output
accuracy=metric.accuracy_score(np.array(y_true).flatten(), np.array(y_pred).flatten(), normalize=True)
print('acuracy=',accuracy)                         # show accracy score

[1 1 1 0]
acuracy= 1.0

In [6]:
print([coef.shape for coef in mlp.coefs_])  # size of synapsis weights
mlp.coefs_                                  # synapsis weights

[(2, 1), (1, 1), (1, 1)]

Out[6]:
[array([[ 4.1486074 ],
[ 4.14636493]]), array([[ 7.43468985]]), array([[ 15.53567128]])]

Experiments¶

• Change the data (X_training, y_training, X_testing)
• Change binary into bipolar {-1, +1}
• Change the hidden layer of MLP: number of layers (int) or size (n,m)
• Change solver of MLP: {'lbfgs', 'sgd', 'adam'}
• Change activation function of MLP: {'identity', 'logistic', 'tanh', 'relu'}, default 'relu'

Tips

• Check accuracy of prediction

Practice-1: Bipolar OR¶

Instead of {0, 1}, we can also change the value into bipolar {-1, +1}. Try the following OR gate using Perceptron and MLP.

In [7]:
X_training=[[1, 1],
[1, -1],
[-1, 1],
[-1, -1]
]
y_training=[1,
1,
1,
-1
]
X_testing=X_training
y_true=y_training


Practice-2: AND gate¶

Train and predict back the AND gate training set below using Perceptron and MLP.

In [8]:
X_training=[[1, 1],
[1, 0],
[0, 1],
[0, 0]
]
y_training=[1,
0,
0,
0
]
X_testing=X_training
y_true=y_training


Practice-3: Bipolar AND gate¶

Modify the above Boolean AND training data into bipolar. Use Perceptron and MLP to train and predict back this gate.

Practice-4: two output neurons¶

Try to use perceptron and MLP to train and predict the following dataset. What happen? Why do you think the error happen?

In [9]:
X_training=[[1, 1],
[1, 0],
[0, 1],
[0, 0]
]
y_training=[[1, 1],
[1, 1],
[1, 1],
[0, 1]
]
X_testing=X_training
y_true=y_training


Practice-5: XOR¶

Use Perceptron and MLP to solve the following Boolean Exclusive Or (XOR) problem. How to improve the accuracy? How many minimum hidden layer is needed to make it 100% accuracy?

In [10]:
X_training=[[1, 1],
[1, 0],
[0, 1],
[0, 0]
]
y_training=[1,
0,
0,
1
]
X_testing=X_training
y_true=y_training


Practice-6: Tautology¶

Train the Perceptron and MLP to solve the following Boolean Tautology (all 1). What happen?

In [11]:
X_training=[[1, 1],
[1, 0],
[0, 1],
[0, 0]
]
y_training=[1,
1,
1,
1
]
X_testing=X_training
y_true=y_training


Practice-7: Boolean XNOR¶

Use Perceptron and MLP to solve the following Boolean Exclusive Not Or (XNOR) problem. How to improve the accuracy? How many minimum hidden layer is needed to make it 100% accuracy?

In [12]:
X_training=[[1, 1],
[1, 0],
[0, 1],
[0, 0]
]
y_training=[0,
1,
1,
0
]
X_testing=X_training
y_true=y_training


Practice-8: Neural Network with 3 input and 2 output¶

Try to train the following data using neural network. Experiment by changing the solver, hidden layers and activation function to improve the accuracy.

In [13]:
X_training=[[ 1,  1,  0],
[ 1, -1, -1],
[-1,  1,  1],
[-1, -1,  1],
[ 0,  1, -1],
[ 0, -1, -1],
[ 1,  1,  1]
]
y_training=[[1, 0],
[0, 1],
[1, 1],
[1, 0],
[1, 0],
[1, 1],
[1, 1]
]
X_testing=X_training
y_true=y_training


Practice-9: Neural Network for Addition¶

Train the Perceptron and MLP to solve neural network addition problem. What happen? Why you cannot solve this kind of simple problem?

In [14]:
X_training=[[ 1,  1],
[ 1, 2.5],
[-1,  1.5],
[-1, -1],
[ 0,  1],
[ 0, -1],
[ 1,  1]
]
y_training=[2,
3.5,
0.5,
0,
1,
-1,
2
]
X_testing=X_training
y_true=y_training


Practice-10: Transportation Mode Choice¶

The following practice session comes from my Neural Network book. Suppose we have the following 10 rows of training data. The training data is supposed to be part of a transportation study regarding the mode choice to select bus, car or train among commuters along a major route in a city, gathered through a questionnaire study. For simplicity and clarity, we selected only 4 attributes. Attribute ‘gender’ is a binary type, while ‘car ownership’ is a quantitative integer. ‘Travel cost/km’ is a quantitative of ratio type but here it was converted into an ordinal type. ‘Income level’ is also an ordinal type. Train a neural network to predict the transport mode of a person, given the four attributes: gender, car ownership, travel cost, and income level. After training the neural network using the data above, try to predict the Transportation Mode Choice of the following instance of data: Female without car ownership, willing to pay expensive travel cost and having medium income level.

Training Data¶

Gender Car ownership Travel Cost Income Level Transportation mode
Male 0 Cheap Low Bus
Male 1 Cheap Medium Bus
Female 1 Cheap Medium Train
Female 0 Cheap Low Bus
Male 1 Cheap Medium Bus
Male 0 Standard Medium Train
Female 1 Standard Medium Train
Female 1 Expensive High Car
Male 2 Expensive Medium Car
Female 2 Expensive High Car

Notes¶

• Perceptron in scikit-learn can only accept one output neuron
• MLP can accept multiple output neurons
• MLP in scikit-learn must have at least 1 hidden layer
• Neural network in scikit-learn does not have any option to change the aggregation function aside from sum product.
• MLP is used for classification problem. Use MLPRegressor if your problem is actually a regression problem.

last update: Nov 2017

Cite this tutorial as [Teknomo (2017) Practices Neural Network in Python] (http://people.revoledu.com/kardi/tutorial/NeuralNetwork/)