No activation function in multilayer_perceptron.py

[wrzadkow]

Hi,
there seem to be no activation functions in multilayer_perceptron.py This results in a poor performance of the network - after addition of additional ReLU at each layer (code change below), the cost almost immediately drops to ~2.3, contrary to slow convergence to ~21 for 15 batches in the current version.

current version:

def multilayer_perceptron(x):
    # Hidden fully connected layer with 256 neurons
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    # Hidden fully connected layer with 256 neurons
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
    # Output fully connected layer with a neuron for each class
    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
    return out_layer

changed into:

def multilayer_perceptron(x):
    # Hidden fully connected layer with 256 neurons
    layer_1 = tf.maximum(0.,tf.add(tf.matmul(x, weights['h1']), biases['b1']))
    # Hidden fully connected layer with 256 neurons
    layer_2 = tf.maximum(0.,tf.add(tf.matmul(layer_1, weights['h2']), biases['b2']))
    # Output fully connected layer with a neuron for each class
    out_layer = tf.maximum(0.,tf.matmul(layer_2, weights['out']) + biases['out'])
    return out_layer

[mlundyCO]

I agree it is odd there is not activation function in the hidden layers, but is tf.maximum a better way to implement the ReLU? Why not use tf.nn.relu? I've also found better performance with a single hidden layer with approximately 75 nodes.

[wrzadkow]

I agree it is odd there is not activation function in the hidden layers, but is tf.maximum a better way to implement the ReLU? Why not use tf.nn.relu? I've also found better performance with a single hidden layer with approximately 75 nodes.

Thanks for your comment! I used tf.maximum instead of tf.nn.relu to keep the spirit of the code - for example tf.matmul is used in the existing code instead of tf.contrib.layers.fully_connected, so I decided to use a lower level primitive.

Also nice that you observed an improvement with changed architecture - I didn't experiment beyond adding the activation function.