Deep Learning  Exercise 06
Learning goal:
Experiment with different network
topologies and transfer functions for a MLP
Experiment with MLP network topologies
Goto this excellent implementation of a MLP which runs in your browser!
Now step through the sequence of the following experiments:
I. Experiments with circle dataset
I.1 tanh

Network topology: 221
Using a MLP with inputs X1,X2 only and only one hidden layer with 2 neurons (network topology: 221) with tanh as transfer function: can it learn to classify the circle dataset (default dataset) correctly (near to 100%)? 
Network topology: 231
Now add a third neuron in the hidden layer and see whether it can learn to classify the circle dataset. 
Network topology: 2221
Instead of adding a third neuron in the hidden layer, repeat the experiment with two hidden layers two neurons each. 
Network topology: 22221
Also try three hidden layers with 2 neurons each. What is special for this case? (keyword: oscillations)
I.2 linear transfer function
Network topology: 2XYZ1Try to find a MLP with a network topology of your choice that uses only the linear transfer function. Could you find one?
I.3 rectified linear unit

Network topology: 221
Instead of using tanh or the linear transfer function, switch now to "ReLU" and try to learn the circle dataset classification task with a 221 network. Does it work? 
Network topology: 231
Seems we need one neuron more here as well as in the case of tanh, or? So add another neuron and repeat the experiment. Compare the form of the "borders" that the MLP classifier learns when using ReLU vs. using tanh as transfer function.
II. Experiments with Gaussian dataset
First think about whether a MLP with no hidden layers (i.e., a Perceptron) that uses linear transfer functions can learn to classify this dataset.Then try it out!
Does the classifiation rate improve when adding neurons or hidden layers?
III. Experiments with eXclusive OR (XOR) dataset
III.1 linear transfer function
Try to find a MLP with a topology of your choice that can correctly classify this dataset.III.2 tanh
Network topology: 2N1Try out tanh as transfer function and this topology with N=1,2,3,4,5,6 and observe how the classification error decreases in each new experiment as you add another neuron to the hidden layer.
III.3 ReLU
Network topology: 2N1Try out ReLU as transfer function and this topology with N=1,2,3,4,5,6 and observe how the classificaition error decreases in each new experiment as you add another neuron to the hidden layer.
Note: test topology 231 with different new initialized weights! Does Backpropagation always find the same solution?
IV. Experiments with spiral dataset
IV.1 linear transfer function
Try to find any MLP with a topology of your choice than can classify this dataset.IV.2 tanh
First try out one hidden layer and increase the number of neurons in this layer. Then add more hidden layers with some neurons in it and observe what happens.IV.3 ReLU
Conduct the same experiments with ReLU as transfer function.Can you reach a solution similar to the one depecited below using just X1 and X2 as input? (in my case it was a 2541 MLP)
Also try out a 28888881 MLP with ReLU and all features as input. Here is a good solution I could observe for the spiral dataset: