Neural Networks

Neural networks are a category of machine learning models which have seen a resurgence since 2006. Deep learning is the recent area of machine learning which combines many neuron layers (e.g. 20, 50, or more) to form a "deep" neural network. In doing so, a deep neural network can accomplish sophisticated classification tasks that classical machine learning models would find difficult.


Keras is a Python package for deep learning which provides an easy-to-use layer of abstraction on top of Theano and Tensorflow.

Import Keras objects:

In [3]:
from keras.models import Sequential
from keras.layers.core import Dense
import keras.optimizers

Create a neural network architecture by layering neurons. Define the number of neurons in each layer and their activation functions:

In [4]:
model = Sequential()
model.add(Dense(4, activation='relu', input_dim=2))
model.add(Dense(4, activation='relu'))
model.add(Dense(2, activation='softmax'))

Choose the optimizer, i.e. the update rule that the neural network will use to train:

In [5]:
optimizer = keras.optimizers.SGD(decay=0.001, momentum=0.99)

Compile the model, i.e. create the low-level code that the CPU or GPU will actually use for its calculations during training and testing:

In [6]:
model.compile(loss='binary_crossentropy', optimizer=optimizer)

Example: XOR

The operation XOR is defined as: XOR(x, y) = 1 if x != y else 0

Synthesize training data for the XOR problem.

In [7]:
X_train = numpy.random.randn(10000, 2)
(10000, 2)
In [8]:
[[ 0.42128999  0.41729839]
 [ 0.41792333  1.47336136]
 [ 0.42125759 -2.28450677]
 [ 1.08729453  0.51543248]
 [-1.11614025  0.97745119]]

Create target labels for the training data.

In [9]:
y_train = numpy.array([
    [float(x[0]*x[1] > 0), float(x[0]*x[1] <= 0)]
    for x in X_train
(10000, 2)
In [10]:
array([[1., 0.],
       [1., 0.],
       [0., 1.],
       [1., 0.],
       [0., 1.]])

Plot the training data:

<matplotlib.collections.PathCollection at 0x11b9c6588>

Finally, train the model!

In [ ]:
results =, y_train, epochs=200, batch_size=100)

Plot the loss function as a function of the training iteration number:

In [13]:
[<matplotlib.lines.Line2D at 0x11c00a518>]

Create test data:

In [14]:
X_test = numpy.random.randn(5000, 2)

Use the trained neural network to make predictions from the test data:

In [15]:
y_test = model.predict(X_test)
In [16]:
(5000, 2)

Let's see if it worked:

<matplotlib.collections.PathCollection at 0x11be77d68>