RuntimeError: Error(s) in loading state_dict for RecognitionPredictor:

[theadrianbao]

Hello, I've trained a custom recognizer, but I am getting an issue with RuntimeError: Error(s) in loading state_dict for RecognitionPredictor: Unexpected key(s) in state_dict: "feat_extractor.0.weight", "feat_extractor.0.bias", "feat_extractor.1.weight", "feat_extractor.1.bias", "feat_extractor.1.running_mean", "feat_extractor.1.running_var", "feat_extractor.1. num_batches_tracked", "feat_extractor.3.weight", "feat_extractor.3.bias", "feat_extractor.4.weight", "feat_extractor.4.bias", "feat_extractor.4.running_mean", "feat_extractor.4.running_var", "feat_extractor.4.num_batches_tracked ", "feat_extractor.7.weight", "feat_extractor.7.bias", "feat_extractor.8.weight", "feat_extractor.8.bias", "feat_extractor.8.running_mean", "feat_extractor.8.running_var", "feat_extractor.8.num_batches_tracked", "feat_extractor. 10.weight", "feat_extractor.10.bias", "feat_extractor.11.weight", "feat_extractor.11.bias", "feat_extractor.11.running_mean", "feat_extractor.11.running_var", "feat_extractor.11.num_batches_tracked", "feat_extractor.14.weight", "feat_extractor.14.bias", "feat_extractor.15.weight", "feat_extractor.15.bias", "feat_extractor.15.running_mean", "feat_extractor.15.running_var", "feat_extractor.15.num_batches_tracked", "feat_extractor.17.weight", "feat_extrac tor.17.bias", "feat_extractor.18.weight", "feat_extractor.18.bias", "feat_extractor.18.running_mean", "feat_extractor.18.running_var", "feat_extractor.18.num_batches_tracked", "feat_extractor.20.weight", "feat_extractor.20.bias" , "feat_extractor.21.weight", "feat_extractor.21.bias", "feat_extractor.21.running_mean", "feat_extractor.21.running_var", "feat_extractor.21.num_batches_tracked", "feat_extractor.24.weight", "feat_extractor.24.bias", "feat_extr actor.25.weight", "feat_extractor.25.bias", "feat_extractor.25.running_mean", "feat_extractor.25.running_var", "feat_extractor.25.num_batches_tracked", "feat_extractor.27.weight", "feat_extractor.27.bias", "feat_extractor.28.wei ght", "feat_extractor.28.bias", "feat_extractor.28.running_mean", "feat_extractor.28.running_var", "feat_extractor.28.num_batches_tracked", "feat_extractor.30.weight", "feat_extractor.30.bias", "feat_extractor.31.weight", "feat_ extractor.31.bias", "feat_extractor.31.running_mean", "feat_extractor.31.running_var", "feat_extractor.31.num_batches_tracked", "feat_extractor.34.weight", "feat_extractor.34.bias", "feat_extractor.35.weight", "feat_extractor.35 .bias", "feat_extractor.35.running_mean", "feat_extractor.35.running_var", "feat_extractor.35.num_batches_tracked", "feat_extractor.37.weight", "feat_extractor.37.bias", "feat_extractor.38.weight", "feat_extractor.38.bias", "fea t_extractor.38.running_mean", "feat_extractor.38.running_var", "feat_extractor.38.num_batches_tracked", "feat_extractor.40.weight", "feat_extractor.40.bias", "feat_extractor.41.weight", "feat_extractor.41.bias", "feat_extractor. 41.running_mean", "feat_extractor.41.running_var", "feat_extractor.41.num_batches_tracked", "decoder.weight_ih_l0", "decoder.weight_hh_l0", "decoder.bias_ih_l0", "decoder.bias_hh_l0", "decoder.weight_ih_l0_reverse", "decoder.wei ght_hh_l0_reverse", "decoder.bias_ih_l0_reverse", "decoder.bias_hh_l0_reverse", "decoder.weight_ih_l1", "decoder.weight_hh_l1", "decoder.bias_ih_l1", "decoder.bias_hh_l1", "decoder.weight_ih_l1_reverse", "decoder.weight_hh_l1_reverse", "decoder.bias_ih_l1_reverse", "decoder.bias_hh_l1_reverse", "linear.weight", "linear.bias".

I've successfully finetuned my model with this tutorial and its path is reco_model_path.

My implementation is like this:

    if reco_model_path is not None and os.path.exists(reco_model_path):
        finetuned_rec_model = torch.load(reco_model_path, map_location=device)
        # finetuned_rec_model = {k[7:]: v for k, v in finetuned_rec_model.items() if k[7:] in model_dict}

        model_dict.update(finetuned_rec_model)

        model.load_state_dict(model_dict)

The line finetuned_rec_model = {k[7:]: v for k, v in finetuned_rec_model.items() if k[7:] in model_dict} prevents the RuntimeError from being thrown, but I get garbled outputs.

python3 inference.py 2b827.png --reco_model_path crnn_vgg16_bn_official.pt (2b827 is the transcription)
Output: [('à€GàZîT;nNQU', 0.00903913751244545)]

Similarly, this is my finetuned model:

python3 inference.py 2b827.png --reco_model_path crnn_vgg16_bn_2.pt
Output: [('h*hû*û+l)ç)', 0.008850879967212677)]

It seems the finetuned model's keys and pretrained=True keys are identical after doing finetuned_rec_model = {k[7:]: v for k, v in finetuned_rec_model.items() if k[7:] in model_dict} that prevents the RuntimeError from being thrown.

Any help would be greatly appreciated!

[theadrianbao]

Hello, just a quick update. Just realized the keys don't have "model." in front of them, causing this issue. My previous approach of finetuned_rec_model = {k[7:]: v for k, v in finetuned_rec_model.items() if k[7:] in model_dict} caused the finetuned_rec_model to be empty, causing no errors, but causing this garbled text.

I did finetuned_rec_model = {f"model.{key}": value for key, value in finetuned_rec_model.items()} instead, but there seems to be a mismatch with the shape of the model that I've fine tuned from this tutorial.

Please see the error:

RuntimeError: Error(s) in loading state_dict for RecognitionPredictor:
        size mismatch for model.linear.weight: copying a param with shape torch.Size([127, 256]) from checkpoint, the shape in current model is torch.Size([124, 256]).
        size mismatch for model.linear.bias: copying a param with shape torch.Size([127]) from checkpoint, the shape in current model is torch.Size([124]).

[theadrianbao]

Just resolved this issue and posting this for future reference:
I realized I had to use this guide and changed my code:

model = crnn_vgg16_bn(pretrained=False, pretrained_backbone=False)

This line I mentioned in my previous answer is no longer necessary:

finetuned_rec_model = {f"model.{key}": value for key, value in finetuned_rec_model.items()}