Interesting results for Highway Nets on ImageNet with A2

[flukeskywalker]

Thanks to timm and @rwightman's very nicely written paper on training recipes, I was able to directly compare Residual connections to Highways on ImageNet-val over the last couple of weeks, and got the results below. I find it interesting that just switching to Highway boosts the acc by 0.5% in each case, while the learned representations are smaller. Moreover, with Highway+A2 I get equal or better acc. than ResNet+A1. I'm curious to know what others think. Are these results interesting, unexpected, or useful to you?

	A1 (600 epochs) top-1 accuracy	A2 (300 epochs) top-1 accuracy	Parameters	Num. of Features
ResNet50	80.4	79.8	25.5 M	64, 128, 256, 512
Highway50		80.3	26.4 M	48, 96, 192, 384
ResNet101	81.5	81.3	44.5 M	64, 128, 256, 512
Highway101		81.8	44.2 M	48, 96, 180, 384
ResNet152	82.0	81.8	60.2 M	64, 128, 256, 512
Highway152		82.3	59.9 M	48, 96, 180, 384

[rwightman]

@flukeskywalker any code to look at re the model changes for the highway variant? I assume based on resnet models for depths / stages / but the widths above and 'highway network' style gating?

It'd be of interest to include, especially if the runtime (throughput, memory) is competitive with ResNet and similar

[flukeskywalker]

@rwightman Yes it is a quick mod of the ResNet code. The key file is here. In summary:

• Both H (regular transformation) and T (transform gate) use the same 3-layer bottleneck transformations.
• Initialize biases of T gate, which is the BN bias in this case, to -1, -2 or -3 depending on depth --- approximatelyln(depth) --- which comes from work on RNNs.
• Reduce stage widths (to approx. match parameters).

I trained on 4 x RTX3090, so had to reduce total batch size to 1024 and 512 (for 101/152 layers) and increase lr accordingly. Based on discussion here it appeared that this works fine for ResNets with rest of A2. The throughput could be improved with a better implementation that combines the computation of the two non-linear transformations (H and T) into single Conv2D calls. I can do that.

I'm interested to know your thoughts on the consistent improvement of 0.5%. I had expected it to perform the same as ResNets (within +- 0.1%). Is the improvement I got significant in your experience?

[flukeskywalker]

I used grouped convolutions to implement the Highway blocks (with groups=2) but using the benchmark.py script I'm still seeing slower inference speed (around 60%) for Highway vs Resnet at the same network depth. It appears that there are lots of reports that grouped convolutions are not as fast as they should be considering the number of ops. Any suggestions?