Porting Stable Video Diffusion ControNet to HPU #1037

wenbinc-Bin · 2024-06-04T04:50:28Z

Enable Stable-Video-Diffusion ControNet on Gaudi

optimum/habana/diffusers/pipelines/controlnet/pipeline_stable_video_diffusion_controlnet.py

emascarenhas · 2024-09-03T14:56:08Z

Please sync your PR with main/upstream and fix any merge conflicts. Thank you.

yafshar · 2024-09-06T19:04:05Z

@dsocek have you reviewed this PR? If you are done, I can start

yafshar · 2024-09-06T19:12:41Z

examples/stable-diffusion/image_to_video_generation.py

+    if isinstance(args.image_path, str):
+        args.image_path = [args.image_path]
+    for image_path in args.image_path:
+        print(image_path)


Suggested change

print(image_path)

Done, Thanks.

yafshar · 2024-09-06T19:24:52Z

examples/stable-diffusion/image_to_video_generation.py

+        type=int,
+        default=25,
+        help="The number of video frames to generate."
+    )
    args = parser.parse_args()

    from optimum.habana.diffusers import GaudiStableVideoDiffusionPipeline


@wenbinc-Bin can you move the import to the top?

Done, Thanks.

yafshar · 2024-09-06T19:27:06Z

examples/stable-diffusion/image_to_video_generation.py

+        from optimum.habana.diffusers import GaudiStableVideoDiffusionPipelineControlNet
+        from optimum.habana.diffusers.models import ControlNetSDVModel
+        from optimum.habana.diffusers.models import UNetSpatioTemporalConditionControlNetModel
+        controlnet = controlnet = ControlNetSDVModel.from_pretrained(


Suggested change

controlnet = controlnet = ControlNetSDVModel.from_pretrained(

controlnet = ControlNetSDVModel.from_pretrained(

Done, Thanks.

yafshar · 2024-09-06T19:32:48Z

examples/stable-diffusion/image_to_video_generation.py

+        # Set seed before running the model
+        set_seed(args.seed)


Suggested change

# Set seed before running the model

set_seed(args.seed)

Isn't it better to set the random seed before loading the models before the conditional?
Personal suggestion! I think this ensures that any randomness involved in the model loading process (such as weight initialization for certain layers) is controlled, leading to reproducible results.

Done, Thanks.

yafshar · 2024-09-06T19:33:25Z

examples/stable-diffusion/image_to_video_generation.py

+        # Set seed before running the model
+        set_seed(args.seed)


Suggested change

# Set seed before running the model

set_seed(args.seed)

same as above!

Done, Thanks.

examples/stable-diffusion/image_to_video_generation.py

yafshar · 2024-09-06T19:37:20Z

examples/stable-diffusion/image_to_video_generation.py

+        "--controlnet_model_name_or_path",
+        default="CiaraRowles/temporal-controlnet-depth-svd-v1",
+        type=str,
+        help="Path to pre-trained controlnet model",


Suggested change

help="Path to pre-trained controlnet model",

help="Path to pre-trained controlnet model.",

Done, Thanks.

yafshar · 2024-09-06T19:37:31Z

examples/stable-diffusion/image_to_video_generation.py

+        type=str,
+        default=None,
+        nargs="*",
+        help="Path to controlnet input image(s) to guide video generation",


Suggested change

help="Path to controlnet input image(s) to guide video generation",

help="Path to controlnet input image(s) to guide video generation.",

Done, Thanks.

yafshar · 2024-09-06T19:53:16Z

@wenbinc-Bin can you check your example in the README. I am unable to run the command and am getting ImportError

>>> python -c "from optimum.habana.diffusers import GaudiEulerDiscreteScheduler"
/usr/local/lib/python3.10/dist-packages/diffusers/models/vq_model.py:20: FutureWarning: `VQEncoderOutput` is deprecated and will be removed in version 0.31. Importing `VQEncoderOutput` from `diffusers.models.vq_model` is deprecated and this will be removed in a future version. Please use `from diffusers.models.autoencoders.vq_model import VQEncoderOutput`, instead.
  deprecate("VQEncoderOutput", "0.31", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/vq_model.py:25: FutureWarning: `VQModel` is deprecated and will be removed in version 0.31. Importing `VQModel` from `diffusers.models.vq_model` is deprecated and this will be removed in a future version. Please use `from diffusers.models.autoencoders.vq_model import VQModel`, instead.
  deprecate("VQModel", "0.31", deprecation_message)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/root/optimum-habana/optimum/habana/diffusers/__init__.py", line 21, in <module>
    from .pipelines.controlnet.pipeline_stable_video_diffusion_controlnet import GaudiStableVideoDiffusionPipelineControlNet
  File "/root/optimum-habana/optimum/habana/diffusers/pipelines/controlnet/pipeline_stable_video_diffusion_controlnet.py", line 25, in <module>
    from diffusers.pipelines.stable_video_diffusion.pipeline_stable_video_diffusion import (
ImportError: cannot import name 'tensor2vid' from 'diffusers.pipelines.stable_video_diffusion.pipeline_stable_video_diffusion' (/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_video_diffusion/pipeline_stable_video_diffusion.py)

It sounds like tensor2vid being replaced. See huggingface/diffusers#9254

yafshar · 2024-09-06T20:13:29Z

@wenbinc-Bin, please also run make style and fix the issues!

optimum/habana/diffusers/models/controlnet_sdv.py

yafshar · 2024-09-06T20:24:25Z

@wenbinc-Bin please fix the port, README and style, so I can continue reviewing the PR

wenbinc-Bin · 2024-09-09T05:30:34Z

It works on diffusers==0.29.2 now.

wenbinc-Bin · 2024-09-09T05:31:11Z

Also fixed the issue reported by 'make style'.

yafshar · 2024-09-09T22:48:57Z

optimum/habana/diffusers/models/unet_spatio_temporal_condition_controlnet.py

+        down_block_additional_residuals: Optional[Tuple[torch.Tensor]] = None,
+        mid_block_additional_residual: Optional[torch.Tensor] = None,
+        return_dict: bool = True,
+        added_time_ids: torch.Tensor = None,


Suggested change

added_time_ids: torch.Tensor = None,

added_time_ids: Optional[torch.Tensor] = None,

Done, Thanks

yafshar · 2024-09-09T23:13:21Z

optimum/habana/diffusers/pipelines/controlnet/pipeline_stable_video_diffusion_controlnet.py

+    def __call__(
+        self,
+        image: Union[PIL.Image.Image, List[PIL.Image.Image], torch.FloatTensor],
+        controlnet_condition: [torch.FloatTensor] = None,


Suggested change

controlnet_condition: [torch.FloatTensor] = None,

controlnet_condition: Optional[torch.FloatTensor] = None,

Done, Thanks.

yafshar · 2024-09-10T16:22:09Z

@wenbinc-Bin is there a reason you have removed some functionalities in pipeline_stable_video_diffusion_controlnet.py compare to the original version https://github.com/CiaraStrawberry/svd-temporal-controlnet/blob/765cd95c3659c54593ae36a9616121f00b3d7c29/pipeline/pipeline_stable_video_diffusion_controlnet.py#L99

I see some differences, and appreciate it if you can clarify those.

yafshar · 2024-09-10T17:01:40Z

@wenbinc-Bin thanks for this contribution. Would you please add a test for this PR? If you can add tests, we can wrap up this PR.

yafshar · 2024-09-10T17:40:23Z

optimum/habana/diffusers/__init__.py

@@ -1,5 +1,8 @@
 from .pipelines.auto_pipeline import AutoPipelineForInpainting, AutoPipelineForText2Image
 from .pipelines.controlnet.pipeline_controlnet import GaudiStableDiffusionControlNetPipeline
+from .pipelines.controlnet.pipeline_stable_video_diffusion_controlnet import (
+    GaudiStableVideoDiffusionPipelineControlNet,


Suggested change

GaudiStableVideoDiffusionPipelineControlNet,

GaudiStableVideoDiffusionControlNetPipeline,

just a personal suggestion to be compliant with other naming like GaudiStableDiffusionControlNetPipeline

Done, thanks.

yafshar · 2024-09-10T17:41:30Z

optimum/habana/diffusers/pipelines/controlnet/pipeline_stable_video_diffusion_controlnet.py

+logger = logging.get_logger(__name__)  # pylint: disable=invalid-name
+
+
+class GaudiStableVideoDiffusionPipelineControlNet(GaudiStableVideoDiffusionPipeline):


Suggested change

class GaudiStableVideoDiffusionPipelineControlNet(GaudiStableVideoDiffusionPipeline):

class GaudiStableVideoDiffusionControlNetPipeline(GaudiStableVideoDiffusionPipeline):

just a personal suggestion to be compliant with other naming like GaudiStableDiffusionControlNetPipeline

Done, thanks.

yafshar · 2024-09-10T17:43:00Z

examples/stable-diffusion/image_to_video_generation.py

-        **kwargs,
-    )
+    if args.control_image_path is not None:
+        from optimum.habana.diffusers import GaudiStableVideoDiffusionPipelineControlNet


Suggested change

from optimum.habana.diffusers import GaudiStableVideoDiffusionPipelineControlNet

from optimum.habana.diffusers import GaudiStableVideoDiffusionControlNetPipeline

just a personal suggestion to be compliant with other naming like GaudiStableDiffusionControlNetPipeline

Done, thanks.

yafshar · 2024-09-10T17:43:18Z

examples/stable-diffusion/image_to_video_generation.py

-    set_seed(args.seed)
+        controlnet = ControlNetSDVModel.from_pretrained(args.controlnet_model_name_or_path, subfolder="controlnet")
+        unet = UNetSpatioTemporalConditionControlNetModel.from_pretrained(args.model_name_or_path, subfolder="unet")
+        pipeline = GaudiStableVideoDiffusionPipelineControlNet.from_pretrained(


Suggested change

pipeline = GaudiStableVideoDiffusionPipelineControlNet.from_pretrained(

pipeline = GaudiStableVideoDiffusionControlNetPipeline.from_pretrained(

Done, thanks.

yafshar · 2024-09-10T17:46:11Z

@wenbinc-Bin the naming change is just a personal suggestion to be compliant with other naming like GaudiStableDiffusionControlNetPipeline, if you do not agree, please ignore the changes! thanks

yafshar · 2024-09-11T15:40:01Z

@wenbinc-Bin would you please respond to the comments so we can finish this PR faster.

Signed-off-by: Wenbin Chen <[email protected]>

wenbinc-Bin · 2024-09-12T04:02:51Z

@wenbinc-Bin is there a reason you have removed some functionalities in pipeline_stable_video_diffusion_controlnet.py compare to the original version https://github.com/CiaraStrawberry/svd-temporal-controlnet/blob/765cd95c3659c54593ae36a9616121f00b3d7c29/pipeline/pipeline_stable_video_diffusion_controlnet.py#L99

I see some differences, and appreciate it if you can clarify those.

These functions are also in base class "StableVideoDiffusionPipeline" and they are basically same. I remove these functions to reduce redundant code.

wenbinc-Bin · 2024-09-12T04:03:55Z

@wenbinc-Bin the naming change is just a personal suggestion to be compliant with other naming like GaudiStableDiffusionControlNetPipeline, if you do not agree, please ignore the changes! thanks

I agree to change the name. Thanks for your advice.

wenbinc-Bin · 2024-09-12T04:05:10Z

@wenbinc-Bin would you please respond to the comments so we can finish this PR faster.

Sorry, I take some time to add test case. I am not familiar with this part before.

wenbinc-Bin · 2024-09-12T04:06:34Z

@wenbinc-Bin thanks for this contribution. Would you please add a test for this PR? If you can add tests, we can wrap up this PR.

I add test case and update the PR.

yafshar

Thanks for the nice contribution.

LGTM!

@regisss this PR is ready, please check it.

yafshar · 2024-09-18T18:02:18Z

@libinta would you please label this PR

github-actions · 2024-09-25T16:18:21Z

The code quality check failed, please run make style.

HuggingFaceDocBuilderDev · 2024-09-25T16:21:50Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

wenbinc-Bin requested a review from regisss as a code owner June 4, 2024 04:50

wenbinc-Bin marked this pull request as draft June 4, 2024 04:50

dsocek reviewed Jun 18, 2024

View reviewed changes

optimum/habana/diffusers/pipelines/controlnet/pipeline_stable_video_diffusion_controlnet.py Outdated Show resolved Hide resolved

optimum/habana/diffusers/pipelines/controlnet/pipeline_stable_video_diffusion_controlnet.py Outdated Show resolved Hide resolved

wenbinc-Bin force-pushed the port_svd_cn branch from e505e65 to b6b55af Compare June 19, 2024 01:24

wenbinc-Bin force-pushed the port_svd_cn branch from b6b55af to 3a49066 Compare September 4, 2024 02:45

wenbinc-Bin marked this pull request as ready for review September 4, 2024 02:45

yafshar reviewed Sep 6, 2024

View reviewed changes

examples/stable-diffusion/image_to_video_generation.py Show resolved Hide resolved

yafshar reviewed Sep 6, 2024

View reviewed changes

optimum/habana/diffusers/models/controlnet_sdv.py Show resolved Hide resolved

wenbinc-Bin force-pushed the port_svd_cn branch from 3a49066 to 6f72b7c Compare September 9, 2024 05:23

yafshar reviewed Sep 9, 2024

View reviewed changes

wenbinc-Bin force-pushed the port_svd_cn branch from 6f72b7c to 09b9001 Compare September 10, 2024 08:14

yafshar reviewed Sep 10, 2024

View reviewed changes

Porting Stable Video Diffusion ControNet to HPU

28d280d

Signed-off-by: Wenbin Chen <[email protected]>

wenbinc-Bin force-pushed the port_svd_cn branch from 09b9001 to 28d280d Compare September 12, 2024 03:58

yafshar approved these changes Sep 12, 2024

View reviewed changes

libinta added the run-test Run CI for PRs from external contributors label Sep 18, 2024

regisss added 2 commits October 3, 2024 16:09

Merge remote-tracking branch 'optimum-habana/main' into port_svd_cn

3ca2e39

Fix example command in README

def5d24

regisss approved these changes Oct 3, 2024

View reviewed changes

yafshar approved these changes Oct 3, 2024

View reviewed changes

regisss merged commit d613e06 into huggingface:main Oct 3, 2024
4 checks passed

	controlnet = controlnet = ControlNetSDVModel.from_pretrained(
	controlnet = ControlNetSDVModel.from_pretrained(

	help="Path to pre-trained controlnet model",
	help="Path to pre-trained controlnet model.",

	help="Path to controlnet input image(s) to guide video generation",
	help="Path to controlnet input image(s) to guide video generation.",

	added_time_ids: torch.Tensor = None,
	added_time_ids: Optional[torch.Tensor] = None,

	controlnet_condition: [torch.FloatTensor] = None,
	controlnet_condition: Optional[torch.FloatTensor] = None,

	GaudiStableVideoDiffusionPipelineControlNet,
	GaudiStableVideoDiffusionControlNetPipeline,

		logger = logging.get_logger(__name__) # pylint: disable=invalid-name


		class GaudiStableVideoDiffusionPipelineControlNet(GaudiStableVideoDiffusionPipeline):

	class GaudiStableVideoDiffusionPipelineControlNet(GaudiStableVideoDiffusionPipeline):
	class GaudiStableVideoDiffusionControlNetPipeline(GaudiStableVideoDiffusionPipeline):

	from optimum.habana.diffusers import GaudiStableVideoDiffusionPipelineControlNet
	from optimum.habana.diffusers import GaudiStableVideoDiffusionControlNetPipeline

	pipeline = GaudiStableVideoDiffusionPipelineControlNet.from_pretrained(
	pipeline = GaudiStableVideoDiffusionControlNetPipeline.from_pretrained(

Porting Stable Video Diffusion ControNet to HPU #1037

Porting Stable Video Diffusion ControNet to HPU #1037

Conversation

wenbinc-Bin commented Jun 4, 2024

emascarenhas commented Sep 3, 2024

yafshar commented Sep 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yafshar Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yafshar commented Sep 6, 2024 • edited Loading

yafshar commented Sep 6, 2024

yafshar commented Sep 6, 2024

wenbinc-Bin commented Sep 9, 2024

wenbinc-Bin commented Sep 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yafshar commented Sep 10, 2024 • edited Loading

yafshar commented Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yafshar commented Sep 10, 2024 • edited Loading

yafshar commented Sep 11, 2024

wenbinc-Bin commented Sep 12, 2024 • edited Loading

wenbinc-Bin commented Sep 12, 2024

wenbinc-Bin commented Sep 12, 2024

wenbinc-Bin commented Sep 12, 2024

yafshar left a comment

Choose a reason for hiding this comment

yafshar commented Sep 18, 2024

github-actions bot commented Sep 25, 2024

HuggingFaceDocBuilderDev commented Sep 25, 2024

yafshar Sep 6, 2024 •

edited

Loading

yafshar commented Sep 6, 2024 •

edited

Loading

yafshar commented Sep 10, 2024 •

edited

Loading

yafshar commented Sep 10, 2024 •

edited

Loading

yafshar commented Sep 10, 2024 •

edited

Loading

wenbinc-Bin commented Sep 12, 2024 •

edited

Loading