Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add PAG support for SD Img2Img #9463

Merged
merged 24 commits into from
Oct 9, 2024
Merged

add PAG support for SD Img2Img #9463

merged 24 commits into from
Oct 9, 2024

Conversation

SahilCarterr
Copy link
Contributor

@SahilCarterr SahilCarterr commented Sep 18, 2024

What does this PR do?

Adds PAG (Perturbed-Attention Guidance) support for SD models (StableDiffusionPAGImg2ImgPipeline).Continuation of #7944

Fixes #8710 (partially)

Before submitting

Who can review?

@yiyixuxu
Anyone in the community is free to review the PR once the tests have passed.


Usage

from diffusers import AutoPipelineForImage2Image
from diffusers.utils import load_image, make_image_grid
import torch

# Set device to CUDA for GPU processing
device = "cuda"

# Load the image-to-image pipeline with pre-trained weights
pipe = AutoPipelineForImage2Image.from_pretrained("Lykon/dreamshaper-8", enable_pag=True)
pipe.to(device)

# Load an initial image
init_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"
)

# Set a random generator with a specific seed
generator = torch.Generator("cuda").manual_seed(31)

# Define the prompt
prompt = "cat wizard"

# Generate the output image using the pipeline
image = pipe(prompt, image=init_image, generator=generator, pag_scale=0.4).images[0]

# Create a grid of the initial and generated images
make_image_grid([init_image, image], rows=1, cols=2)

cat_wizrd

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! I left some comments
can we see some outputs too? (saw it)!

src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated Show resolved Hide resolved
src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated Show resolved Hide resolved
src/diffusers/pipelines/pag/pipeline_pag_sd_img2img.py Outdated Show resolved Hide resolved
tests/pipelines/pag/test_pag_sd_img2img.py Outdated Show resolved Hide resolved
tests/pipelines/pag/test_pag_sd_img2img.py Outdated Show resolved Hide resolved
@yiyixuxu yiyixuxu added the PAG label Sep 18, 2024
@yiyixuxu
Copy link
Collaborator

don't forget to call make style and make fix-copies too

@SahilCarterr
Copy link
Contributor Author

added "# copied from" , removed depreciated methods and fix test

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! I think we still have one test needs to be removed, and then can merge!

), f"output is different from expected, {image_slice.flatten()}"

@slow
@require_torch_gpu
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test is not needed? (same as the one above it?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should i remove StableDiffusionPAGImg2ImgPipelineIntegrationTests(unittest.TestCase): also ?

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yiyixuxu yiyixuxu mentioned this pull request Sep 20, 2024
6 tasks
@yiyixuxu
Copy link
Collaborator

@SahilCarterr can you look into failing tests?

@SahilCarterr
Copy link
Contributor Author

Screenshot 2024-09-25 at 10 47 58 PM

3 errors remaining to be fix

@SahilCarterr
Copy link
Contributor Author

Help Needed to fix the remaining errors

@SahilCarterr
Copy link
Contributor Author

The errors are now fixed @yiyixuxu
Screenshot 2024-10-06 at 1 25 56 PM

@yiyixuxu
Copy link
Collaborator

yiyixuxu commented Oct 9, 2024

can you run make style and make fix-copies?

@SahilCarterr
Copy link
Contributor Author

fixed make error

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!

@yiyixuxu yiyixuxu merged commit af28ae2 into huggingface:main Oct 9, 2024
15 checks passed
leisuzz pushed a commit to leisuzz/diffusers that referenced this pull request Oct 11, 2024
* added pag to sd img2img pipeline


---------

Co-authored-by: YiYi Xu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add PAG support to SD1.5
3 participants