Language: 🇺🇸 🇨🇳

«KnowledgeReview» re-implements the paper Distilling Knowledge via Knowledge Review

arch_s	top1	top5	arch_t	top1	top5	dataset	lambda	top1	top5
MobileNetv2	80.620	95.820	ResNet50	83.540	96.820	CIFAR100	7.0	83.370	96.810
MobileNetv2	80.620	95.820	ResNet152	85.490	97.590	CIFAR100	8.0	84.530	97.470
MobileNetv2	80.620	95.820	ResNeXt_32x8d	85.720	97.650	CIFAR100	6.0	84.520	97.470
ResNet18	80.540	96.040	ResNet50	83.540	96.820	CIFAR100	10.0	83.130	96.350
ResNet50	83.540	96.820	ResNet152	85.490	97.590	CIFAR100	6.0	86.240	97.610
ResNet50	83.540	96.820	ResNeXt_32x8d	85.720	97.650	CIFAR100	6.0	86.220	97.490

more see docs

Table of Contents

• Table of Contents
• Background
• Installation
• Usage
• Maintainers
• Thanks
• Contributing
• License

Background

Unlike overhaul and other knowledge transfer algorithms，knowledge review use cross stage's teacher features to train student features. Meanwhile, it designed a new residual learning framework to simplify student transfer operation, and use ABF (attention based fusion) and HCL (hierarchical context loss) to improve feature distillation.

Current project implementation is based on ZJCV/overhaul and dvlab-research/ReviewKD.

Installation

$ pip install -r requirements.txt

Usage

• Train

$ CUDA_VISIBLE_DEVICES=0 python tools/train.py -cfg=configs/rfd/resnet/rfd_6_0_r152_pret_r50_c100_224_e100_sgd_mslr.yaml

• Test

$ CUDA_VISIBLE_DEVICES=0 python tools/test.py -cfg=configs/rfd/resnet/rfd_6_0_r152_pret_r50_c100_224_e100_sgd_mslr.yaml

Maintainers

• zhujian - Initial work - zjykzj

Thanks

@misc{chen2021distilling,
      title={Distilling Knowledge via Knowledge Review}, 
      author={Pengguang Chen and Shu Liu and Hengshuang Zhao and Jiaya Jia},
      year={2021},
      eprint={2104.09044},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Contributing

Anyone's participation is welcome! Open an issue or submit PRs.

Small note:

• Git submission specifications should be complied with Conventional Commits
• If versioned, please conform to the Semantic Versioning 2.0.0 specification
• If editing the README, please conform to the standard-readme specification.

License

Apache License 2.0 © 2021 zjykzj