Why is lora training so slow?

[ArlenCHEN]

Dear author @yunkchen

Thanks for your awesome work!

I tried to run the lora training using my data, but the speed is very slow --- ~40s/it.

Training details:
512x512 model
2 GPUs - batch size: 1 for each

Is there anything I missed? Please give some hints on this. Thanks!

[yunkchen]

I had trained lora of 768x768 model with 144 frames on A100, using bf16.
And the speed is ~10s/it, FYI.

[ArlenCHEN]

thanks for the info!

ya. that's something I expect to see. but sadly. anyway I will check and update here on anything I will get.

[gulucaptain]

I met the same problem too. I fine-tune the model with lora on V100 machines. Its speed is about 40s/it. When I don't use the lora, the speed is 26s/it.

[gulucaptain]

thanks for the info!

ya. that's something I expect to see. but sadly. anyway I will check and update here on anything I will get.

do you find any solution to speed up the lora-training?

[ArlenCHEN]

@gulucaptain Not yet. Did you use --enable_xformers_memory_efficient_attention?

[huangjch526]

My situation: 8 A100 80G, batchsize 1, 19.35s/it, I feel it very slow. Is that normal?
I use the default train.sh as
accelerate launch --mixed_precision='bf16' scripts/train.py
--pretrained_model_name_or_path=$MODEL_NAME
--train_data_dir=$DATASET_NAME
--train_data_meta=$DATASET_META_NAME
--config_path "config/easyanimate_video_slicevae_multi_text_encoder_v4.yaml"
--image_sample_size=512
--video_sample_size=512
--token_sample_size=512
--video_sample_stride=1
--video_sample_n_frames=144
--train_batch_size=1
--video_repeat=1
--gradient_accumulation_steps=1
--dataloader_num_workers=8
--num_train_epochs=100
--checkpointing_steps=500
--learning_rate=2e-05
--lr_scheduler="constant_with_warmup"
--lr_warmup_steps=100
--seed=42
--output_dir="output_dir/ft_0.1Mv"
--enable_xformers_memory_efficient_attention
--gradient_checkpointing
--mixed_precision="bf16"
--adam_weight_decay=3e-2
--adam_epsilon=1e-10
--vae_mini_batch=1
--max_grad_norm=0.05
--random_hw_adapt
--training_with_video_token_length
--motion_sub_loss
--not_sigma_loss
--random_frame_crop
--enable_bucket
--train_mode="inpaint"
--trainable_modules "."

[huangjch526]

help me and help each other, please