This repository contains an implementation of 4 methods for semi-supervised representation learning:
- CrossEntropy: supervised baseline
- InfoNCE: self-supervised baseline (SimCLR without projection head)
- SuNCEt: InfoNCE + supervised contrastive learning
- PAWS: negative-free method with non-parametric pseudo-labels (can be seen as DINO + pseudo-labels)
Try it out in this Colab Notebook:
The trained encoders do not have a classification head except CrossEntropy. All methods are trained on the STL10 dataset, and the representations are evaluated using the accuracy of a k-nearest neighbour classifier. The encoder uses a simple convolutional architecture.
The codebase follows modern Tensorflow2 + Keras best practices and the implementation seeks to be as concise and readable as possible. This implementation is intended to be used as an easy-to-use baseline instead of as a line-by-line reproduction of the papers.
The image augmentation pipeline is an important component of all these methods. You can find implementations of other custom Keras image augmentation layers in this repository.
In CrossEntropy, SuNCEt and PAWS the labeled part of the dataset is repeated 20 times so that the labeled and unlabeled batch sizes can be the same size (this is not necessary, just a design choice). Therefore for these methods 1 epoch means 20 epochs over the labeled part of the dataset, and 1 epoch over the unlabeled part.