Can not reproduce the results

[sjiajia]

Hello, I cannot reproduce the results of the paper on the miniImagenet dataset. Is there any solution?（thanks very much）
I used the pretrained and finetuned model data（backbone-wrn） you provided in another issue. The test results were as follows：miniImagenet: 1shot 74.12% and 5shot 85.21%
tieredImageNet: 1shot 84.22% and 5shot 89.40%
The results given in the paper are:
miniImagenet: 1shot 79.22% and 5shot 88.05%.
tieredImageNet: 1shot 83.69% and 5shot 89.34%
The performance on the mini dataset is much worse. I pretrained and finetuned the model myself, and I tested it again, the performance was as follows：
miniImagenet（wrn）: 1shot 77.12% and 5shot 87.90%.
There is still a certain gap between the performance and the result in paper, and I found that in the 1shot experiment on the miniImageNet data set, the performance did not improve after the 8th epoch. Is there any problem here?
Finally, I still have a question. According to the results in the article, the performance of epnet-ssl is 9% higher than that of epnet in the case of 1 shot. Is this increase so much? When I did the experiment, the performance of epnetssl was 4% higher than that of epnet. Can you provide a solution? thank you very much!

[prlz77]

Hi @sjiajia,

First of all, just clarify that the results you are comparing with (79.22, 88.05, 83.69, 89.34) are for the semi-supervised learning experiment (Table 4 https://arxiv.org/pdf/2003.04151.pdf).

In order to debug your SSL set-up. Could you put a breakpoint in the SSL model (https://github.com/ElementAI/embedding-propagation/blob/master/src/models/ssl_wrapper.py#L28), and check if you are using it? Also, could tell me the value of self.exp_dict['unlabeled_size_test']?

Thanks

[sjiajia]

hello @prlz77 ,
I am sure that I am runing the semi-supervised learning experiment (Table 4). I use the pretrained and finetuned model that you provided in another issue (episodic_miniimagenet_wrn_1-shot/checkpoint_best.pth and episodic_miniimagenet_wrn_5-shot/checkpoint_best.pth) and get the results(74.12 85.21 84.22 89.40). I pretrain and finetune the model on miniImageNet by myself and get the results (77.12 87.90). The exp_dict of mini-wrn-1shot-ssl experiment is as follows:
{
"dataset_train_root": "mini-imagenet/",
"dataset_val_root": "mini-imagenet/",
"dataset_test_root": "mini-imagenet/",
"model": {
"name": "ssl",
"backbone": "wrn",
"depth": 28,
"width": 10,
"transform_train": "wrn_finetune_train",
"transform_val": "wrn_val",
"transform_test": "wrn_val"
},
"ngpu": 1,
"random_seed": 42,
"batch_size": 1,
"train_iters": 10,
"test_iters": 600,
"tasks_per_batch": 1,
"dropout": 0.1,
"avgpool": true,
"n_classes": 64,
"collate_fn": "identity",
"transform_train": "wrn_finetune_train",
"transform_val": "wrn_val",
"transform_test": "wrn_val",
"dataset_train": "episodic_miniimagenet_pkl",
"classes_train": 5,
"support_size_train": 1,
"query_size_train": 15,
"unlabeled_size_train": 0,
"dataset_val": "episodic_miniimagenet_pkl",
"classes_val": 5,
"support_size_val": 1,
"query_size_val": 15,
"unlabeled_size_val": 0,
"dataset_test": "episodic_miniimagenet_pkl",
"classes_test": 5,
"support_size_test": 1,
"query_size_test": 15,
"unlabeled_size_test": 100,
"predict_method": "labelprop",
"finetuned_weights_root": "./logs/finetuning",
"embedding_prop": true
}
Is there any other solutions? Thank you very much!

[prlz77]

Thanks for the clarification! At first glance I cannot see what is wrong. It may be related to this commit: a841b3f

I will investigate it but I am currently busy with the ICML deadline, so I may find a solution after then. If meanwhile, you find the problem, please feel free to make a pull request!

[Yoke-L]

Hello, I cannot reproduce the results of the paper on the miniImagenet dataset. Is there any solution?（thanks very much）
I used the pretrained and finetuned model data（backbone-wrn） you provided in another issue. The test results were as follows：miniImagenet: 1shot 74.12% and 5shot 85.21%
tieredImageNet: 1shot 84.22% and 5shot 89.40%
The results given in the paper are:
miniImagenet: 1shot 79.22% and 5shot 88.05%.
tieredImageNet: 1shot 83.69% and 5shot 89.34%
The performance on the mini dataset is much worse. I pretrained and finetuned the model myself, and I tested it again, the performance was as follows：
miniImagenet（wrn）: 1shot 77.12% and 5shot 87.90%.
There is still a certain gap between the performance and the result in paper, and I found that in the 1shot experiment on the miniImageNet data set, the performance did not improve after the 8th epoch. Is there any problem here?
Finally, I still have a question. According to the results in the article, the performance of epnet-ssl is 9% higher than that of epnet in the case of 1 shot. Is this increase so much? When I did the experiment, the performance of epnetssl was 4% higher than that of epnet. Can you provide a solution? thank you very much!

Hello, do you mind providing the miniImageNet pickle files for me?Thank you very much.