Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update image handling and add image processing options #706

Merged
merged 5 commits into from
Sep 11, 2024
Merged

Conversation

pelikhan
Copy link
Member

@pelikhan pelikhan commented Sep 11, 2024

This pull request includes several commits that improve the image handling and processing capabilities of the software. The commits update the image handling for buffers and blobs, add the "jimp" dependency for image processing, refactor the image options and clean up imports for image processing functions, and refactor the image processing and resolution functions for clarity and efficiency. Additionally, the commits add detail, scaling, and auto-cropping options to the images documentation. These changes enhance the software's ability to handle and process images, providing more flexibility and control to the users.

  • Updated documentation related to image scripting feature 📄, specifically a reference to defImages function in images.md. This introduced several options for image manipulation like detail, scaling, and auto cropping.
  • Added new jimp package in both core and cli to handle image manipulation such as resize and auto-crop 💻.
  • Refactored file.ts and runpromptcontext.ts in the core package to support the image manipulation features using jimp package 🖼️.
  • New file image.ts was introduced in the core package, which includes the implementation of the imageEncodeForLLM function using jimp for image manipulation 🖌️.
  • The public API in prompt_template.d.ts is updated to include maxWidth, maxHeight, and autoCrop relating to image manipulation options under DefImagesOptions 🧑‍💻.
  • A sample in describe-image.genai.js in the sample package was updated to use new image manipulation features of defImages function 🎨.

generated by pr-describe

You can automatically remove uniform color on the edges of the image.

```js "autoCrop"
defImages(img, { autoCrop: true })

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The defImages function call is missing the img argument in the example code snippet.

generated by pr-docs-review-commit missing_argument

Copy link

The changes in this pull request include:

  1. Refactoring of the resolveFileDataUri function in file.ts to accept a filename as a string, instead of a WorkspaceFile object. This makes the function more versatile.
  2. A new imageEncodeForLLM function in a new image.ts file, which processes images (resizing, autocrop, and encoding to base64) for use in chat renderings.
  3. Changes in runpromptcontext.ts to use the new imageEncodeForLLM function for image processing instead of the previous method of directly encoding image Buffers and Blobs to base64.
  4. New options added to DefImagesOptions in prompt_template.d.ts to support the new image manipulation capabilities introduced in imageEncodeForLLM.

I do not see any functional issues with these changes, and they seem to make the code cleaner and easier to maintain. The new image processing function adds capabilities for working with images that were not previously available.

Therefore, my response is "LGTM 🚀".

generated by pr-review

@pelikhan pelikhan merged commit f452c3f into main Sep 11, 2024
12 checks passed
@pelikhan pelikhan deleted the autocrop branch September 11, 2024 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant