What is the training recipe of EfficientVit (MIT)?

[chjej202]

I found that the EfficientVit was uploaded yesterday. What is the training recipe of EfficientVit (MIT)?

I tried with several hyperparameters but failed to train with EfficientVit (MIT)

[rwightman]

@chjej202 yeah, I've been having some doubts about this model and may end up removing, I didn't train the weighs, they're from the official paper authors, but it seems the model won't train in AMP (it's unstable). So not really that practical for a model that's supposed to be 'efficient' ... if I can't find a way to improve stability in AMP I may get rid of it...

[rwightman]

FYI it should at least start to converge if you use float32 and hparams similar to vit models... adamw or nadamw, gradient clipping, wd .05, etc

[rwightman]

this seems to help dc18cda ... let me know if you have any luck...

[andravin]

Any news on EfficientViT training? I am deciding whether to include this model as a baseline in my paper. Its model efficiency is impressive. But I don't want to include it if the weights are not reproducible.

[andravin]

Here are the training configurations from the official repo: https://github.com/mit-han-lab/efficientvit/tree/master/configs/cls/imagenet
.. and the training README: https://github.com/mit-han-lab/efficientvit/blob/master/TRAINING.md

[rwightman]

@andravin for the exact recipe reproduction, I'm not sure. They use 'mesa' for their L models, otherwise I think most aspects can be matched here. I did some basic train tests on small data and training was not stable unless you used float32 or disabled autocast for the attn calculation when using AMP...

[rwightman]

yeah, it trains reasonably, reconfirmed with small-data (resisc45) runs from random init, actually fairly easy to beat resnet50 and efficientnet_b0 with faster/smaller efficientvit_mx, was a bit surprised. Didn't do much of a hparam search but still...

[rwightman]

Oh wait, that's the 'other' efficientvit, hah, m != mit :D will run same on b and l...

[rwightman]

So yeah the m* ones are easier to train from scratch than the b* and l*, although the latter do train fine, wouldn't describe them as being noteworthy relative to familiar convnets at comparable train speeds

[andravin]

Hmm, the paper reports 83.5% ImageNet-1K accuracy with 4.0G MACs for EfficientViT-B3 (r224). That is significantly better model efficiency than ConvNeXt or EfficientNet.

Are you evaluating them in terms of accuracy versus training speed?

[rwightman]

So yeah, may well be the case, but that result is for a highly tuned imagenet-1k pretrain, I was doing some checks with relatively simple hparams on small data, so in terms of 'ease' of train EfficientViT (MIT) is more fussy than EfficientViT (MSRA). But they both train without blowing up so if you like the end efficiency tradeoffs with the MIT variant than giver' and tune your hparams..