basic testing of new tconvnet API

[yamins81]

First, let's run some small-scale tests to ensure that the code acts as promised:

(1) Tests for small scale architectures like MNIST - make this a basic test.
This should test that standard MNIST works as promised and correctly trains to the end, and compares favorably to existing trained MNIST.

(2) Test MNIST + bypasses. This is primarily to ensure that the code works in a nontrivial circumstance, but should also achieve non-terrible numbers of possible.

(3) Test MNIST + feedbacks.

(3.5) Intermediate-sized ConvNet for testing and debugging

(4) Train on AlexNet and compare

(5) Train on DenseNet and compare

(6) Train on VGG and compare

For all things, make sure that:

-- code runs in a reasonable amount of time
-- using a reasonable amount of memory
-- appropriate training losses are achieved

NB: Items (1)-(3) above should be in the form of tests in the test suite

[snickerdudle]

Forward MNIST Benchmark metrics: (use architecture from tfutils models, no recurrence (state update variable is 0))

1. Code must run (duh)
2. Empirical performance of >90%
3. --Pseudo-- Time tests
4. Memory use with GPU stats??? -> look up the TF dynamic "memory use" flag

Do the above for MNIST Bypass (1 each for several arbitrary feedback configurations)

Do the above for MNIST Feedback (1 each for several arbitrary feedback configurations)

[snickerdudle]

Unicycle AlexNet updates:

1. Code runs and trains
2. Empirical performance
   Loss quickly goes down to 6.9, stays there throughout training, sometimes dips into high 6.8s. Dropout implemented and custom initializers implemented, doesn't help much.
3. Time tests
   Takes around 900 ms per step in the beginning, goes up to around 4.8 sec by step 6000
4. Memory use with GPU stats??? -> look up the TF dynamic "memory use" flag
   Uses 2.3 GB of GPU memory

[snickerdudle]

1. Finish DenseNet and make sure it trains
   Trains, but loss is capped at 6.9, doesn't go below this.
2. Go back to MNIST and check values and gradients using fully connected 2 layer from tfutils - Push through by hand in Jupyter using fully connected. Generate random inputs, push through, get outputs and values and gradients.
   3) Play around with different zero state values (for initial explosions)
3. Run some images through, compare the filters in learning
4. Try to get TensorBoard visualizations up
5. Pray to the spaghetti gods that it works out

[yamins81]

1. the notebooks for fc and conv mnist testing should be put into the form actual tests
2. a similar formal test for alexnet should be written
3. we then need to see that alexnet is training properly -- this doesn't need to be a formal test, but we need to have an ipython notebook referencing a DB illustrating that loss is going down as predicted
4. a formal test that skip connections are working
5. a formal test that feedback connections are working
6. a formal test that time decay is working