Mismatch diffusers version & Wrong VAE checkpoint loading on 22.09.2023

[yitongx]

Hi, first of all thanks for sharing your improved codes!

While I was testing your jupyter notebook on CoLab, I have the following problems:

Firstly, I would like to know if you are certain that diffusers==0.11.0 can successfully run the codes? Since from diffusers.models.attention_processor import AttentionProcessor, AttnProcessor here is requiring version >= 0.14.0 at least. Currently I'm using diffusers==0.21.0 instead and only need to modify one import from diffusers.utils.torch_utils import is_compiled_module here. And after installing some missed packages like wandb, compel, ffmpeg, I can successfully finish the training part.

[yitongx]

Secondly, under diffusers==0.21.0, I failed at final inference step. I got an unexpected VAE checkpoint loading error:

===================================================================
`RuntimeError Traceback (most recent call last)
in <cell line: 63>()
111 # vae
112 converted_vae_checkpoint = convert_ldm_vae_checkpoint(base_state_dict, pipeline.vae.config)
--> 113 pipeline.vae.load_state_dict(converted_vae_checkpoint)
114 # unet
115 converted_unet_checkpoint = convert_ldm_unet_checkpoint(base_state_dict, pipeline.unet.config)

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
2039
2040 if len(error_msgs) > 0:
-> 2041 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
2042 self.class.name, "\n\t".join(error_msgs)))
2043 return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for AutoencoderKL:
Missing key(s) in state_dict: "encoder.mid_block.attentions.0.to_q.weight", "encoder.mid_block.attentions.0.to_q.bias", "encoder.mid_block.attentions.0.to_k.weight", "encoder.mid_block.attentions.0.to_k.bias", "encoder.mid_block.attentions.0.to_v.weight", "encoder.mid_block.attentions.0.to_v.bias", "encoder.mid_block.attentions.0.to_out.0.weight", "encoder.mid_block.attentions.0.to_out.0.bias", "decoder.mid_block.attentions.0.to_q.weight", "decoder.mid_block.attentions.0.to_q.bias", "decoder.mid_block.attentions.0.to_k.weight", "decoder.mid_block.attentions.0.to_k.bias", "decoder.mid_block.attentions.0.to_v.weight", "decoder.mid_block.attentions.0.to_v.bias", "decoder.mid_block.attentions.0.to_out.0.weight", "decoder.mid_block.attentions.0.to_out.0.bias".
Unexpected key(s) in state_dict: "encoder.mid_block.attentions.0.key.bias", "encoder.mid_block.attentions.0.key.weight", "encoder.mid_block.attentions.0.proj_attn.bias", "encoder.mid_block.attentions.0.proj_attn.weight", "encoder.mid_block.attentions.0.query.bias", "encoder.mid_block.attentions.0.query.weight", "encoder.mid_block.attentions.0.value.bias", "encoder.mid_block.attentions.0.value.weight", "decoder.mid_block.attentions.0.key.bias", "decoder.mid_block.attentions.0.key.weight", "decoder.mid_block.attentions.0.proj_attn.bias", "decoder.mid_block.attentions.0.proj_attn.weight", "decoder.mid_block.attentions.0.query.bias", "decoder.mid_block.attentions.0.query.weight", "decoder.mid_block.attentions.0.value.bias", "decoder.mid_block.attentions.0.value.weight".

===================================================================

This also happened when I tried to reproduce official inference results using the yaml configs.

I'm not familiar with your codes. Compare to original AnimateDiff, did you modify the VAE model structure somewhere? Or maybe I overlooked something.

Thanks in advance and best regards.