Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stable-diffusion model wrapper #438

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Add stable-diffusion model wrapper #438

wants to merge 15 commits into from

Conversation

cmgzn
Copy link

@cmgzn cmgzn commented Sep 11, 2024


name: Pull Request
about: Create a pull request

Description

add stable-diffusion model wrapper
Integrate https://github.com/AUTOMATIC1111/stable-diffusion-webui

Checklist

Please check the following items before code is ready to be reviewed.

  • Code has passed all tests
  • Docstrings have been added/updated in Google Style
  • Documentation has been updated
  • Code is ready for review

@cmgzn cmgzn changed the title add stable-diffusion model wrapper Add stable-diffusion model wrapper Sep 12, 2024
@cmgzn cmgzn marked this pull request as draft September 13, 2024 01:36
@cmgzn cmgzn marked this pull request as ready for review September 13, 2024 01:36
Copy link
Collaborator

@rayrayraykk rayrayraykk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please refer to the inline comments. Thank you!
Additionally, is it possible to launch the SD API server using a single, (interactive) bash script?

examples/conversation_with_stablediffusion_model/README.md Outdated Show resolved Hide resolved
@@ -64,6 +67,7 @@
"ZhipuAIEmbeddingWrapper",
"LiteLLMChatWrapper",
"YiChatWrapper",
"StableDiffusionTxt2imgWrapper",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change name to StableDiffusionImageSynthesisWrapper

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change name to StableDiffusionImageSynthesisWrapper

Since the SD API also provides an img2img interface, I think that both txt2img and img2img represent image synthesis, I'm concerned that naming the wrapper StableDiffusionImageSynthesisWrapper to represent text-to-image functionality might lead to confusion in the future if we decide to add an image-to-image wrapper.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DavdGao Please hv a look into this, thx.

src/agentscope/models/stablediffusion_model.py Outdated Show resolved Hide resolved
@rayrayraykk
Copy link
Collaborator

And please pass the UT, thx!

@cmgzn
Copy link
Author

cmgzn commented Sep 27, 2024

I've made some updates to the PR:

  1. Replaced the custom request with the third-party package webuiapi. Let me know if this is suitable for the project or if you prefer a different approach.
  2. I've wrapped the SD model as a callable agent service (similar to 'dashscope_services'), as I realized that directly conversing with the text2img model was not a good idea. But the dialogue agent for direct conversation with the SD model is still available and can be modified based on your feedback.

@rayrayraykk
Copy link
Collaborator

rayrayraykk commented Sep 29, 2024

Could you please further provide a minimal hardware requirement for running this demo on README? And the other part looks good to me. @DavdGao Please have a look at this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants