MultimodalQnA image query, pdf, dynamic ports, and UI updates #1381

mhbuehler · 2025-01-10T22:31:20Z

Description

Per the proposed changes in this RFC's Phase 2 plan, this PR adds support for image queries, PDF ingestion and display, and dynamic ports. There are also some bug fixes. This PR goes with this one in GenAIComps.

Issues

RFC

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Others (enhancement, documentation, validation, etc.)

Dependencies

gradio-pdf is a new dependency

Tests

New tests were added to:

MultimodalQnA/tests/test_compose_on_gaudi.sh
MultimodalQnA/tests/test_compose_on_xeon.sh

* Add var for LVM model Signed-off-by: dmsuehir <[email protected]> * README updates and tests Signed-off-by: dmsuehir <[email protected]> * Build, compose, and doc updates for audio query (#12) Signed-off-by: okhleif-IL <[email protected]> * validated, updated tests Signed-off-by: okhleif-IL <[email protected]> * added one more curl test for audio Signed-off-by: okhleif-IL <[email protected]> * fixed typo Signed-off-by: okhleif-IL <[email protected]> * reverted git clone command Signed-off-by: okhleif-IL <[email protected]> * added ASR test Signed-off-by: okhleif-IL <[email protected]> * fixed command with backslashes Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]> Signed-off-by: okhleif-IL <[email protected]> Signed-off-by: dmsuehir <[email protected]> * MMQnA doc update correcting ASR and whisper image names (#24) * MMQnA doc update correcting ASR and whisper image names Signed-off-by: dmsuehir <[email protected]> * Add image tags Signed-off-by: dmsuehir <[email protected]> --------- Signed-off-by: dmsuehir <[email protected]> * Integrate audio query into UI (#22) * Enabled audio query functionality in the MultimodalQnA UI Signed-off-by: Melanie Buehler <[email protected]> * Temporarily redirect clones for tests Signed-off-by: Melanie Buehler <[email protected]> * Doc update and set MAX_IMAGES to default to 1 in compose.yaml Signed-off-by: dmsuehir <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * README update to add a note about MAX_IMAGES Signed-off-by: dmsuehir <[email protected]> * Add services to tests and correct small text error (#26) * Add services to tests and correct small text error Signed-off-by: Melanie Buehler <[email protected]> * Revert unintended changes Signed-off-by: Melanie Buehler <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]> * Fixed build.yaml inconsistency Signed-off-by: Melanie Buehler <[email protected]> * Moving note about MAX_IMAGES due to review feedback: Signed-off-by: dmsuehir <[email protected]> * Update repo clones for E2E tests Signed-off-by: Melanie Buehler <[email protected]> * Moved Audio Query Gateway changes to multimodalqna.py (#29) * Moved gateway changes to multimodalqna.py Signed-off-by: okhleif-IL <[email protected]> * reverted port changes Signed-off-by: okhleif-IL <[email protected]> * addressed review comments Signed-off-by: okhleif-IL <[email protected]> * reverted print statement Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]> * Removed proxies arg from translation request (#31) * Moved gateway changes to multimodalqna.py Signed-off-by: okhleif-IL <[email protected]> * reverted port changes Signed-off-by: okhleif-IL <[email protected]> * addressed review comments Signed-off-by: okhleif-IL <[email protected]> * reverted print statement Signed-off-by: okhleif-IL <[email protected]> * removed proxies Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]> * Add MAX_IMAGES to docs Signed-off-by: dmsuehir <[email protected]> * Moved gateway updates for image query to GenAIExamples Signed-off-by: dmsuehir <[email protected]> * Fix initial input format for TextImageDoc Signed-off-by: dmsuehir <[email protected]> * Fix merge conflict Signed-off-by: dmsuehir <[email protected]> * Fix merge conflict Signed-off-by: dmsuehir <[email protected]> * Add Gaudi tests Signed-off-by: dmsuehir <[email protected]> * Temporary update to use dev branch for test Signed-off-by: dmsuehir <[email protected]> * Temporary change for dockerfile to use dev fork/branch for GenAIComps Signed-off-by: dmsuehir <[email protected]> --------- Signed-off-by: dmsuehir <[email protected]> Signed-off-by: Melanie Buehler <[email protected]> Signed-off-by: okhleif-IL <[email protected]> Co-authored-by: Omar Khleif <[email protected]> Co-authored-by: Melanie Hart Buehler <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <[email protected]>

…image-query Signed-off-by: dmsuehir <[email protected]>

Signed-off-by: Harsha Ramayanam <[email protected]>

* Adds dynamic wait to MMQnA tests Signed-off-by: Melanie Buehler <[email protected]> * Fix for empty grep result Signed-off-by: Melanie Buehler <[email protected]> * Adjusted test for gaudi Signed-off-by: Melanie Buehler <[email protected]> * Fixed for Gaudi service check Signed-off-by: Melanie Buehler <[email protected]> * Added warning to check_service_ready function Signed-off-by: Melanie Buehler <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]>

Signed-off-by: dmsuehir <[email protected]>

Signed-off-by: Melanie Buehler <[email protected]>

* refactored compose and set env files Signed-off-by: okhleif-IL <[email protected]> * updated readmes Signed-off-by: okhleif-IL <[email protected]> * addressed review comments Signed-off-by: okhleif-IL <[email protected]> * added env vars to compose env Signed-off-by: okhleif-IL <[email protected]> * reverted port numbers back to defaults Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: Harsha Ramayanam <[email protected]>

Added fix for video upload format

* Add caption to chat history Signed-off-by: Melanie Buehler <[email protected]> * Fixed some issues with conversation history Signed-off-by: Melanie Buehler <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]>

* Initial implementation for PDF upload Signed-off-by: Melanie Buehler <[email protected]> * Make temp pdf from response Signed-off-by: Melanie Buehler <[email protected]> * Adds PDF display functionality to MMQnA UI with gradio_pdf Signed-off-by: Melanie Buehler <[email protected]> * Update tests and docs for PDF support Signed-off-by: Melanie Buehler <[email protected]> * Updated per review suggestions Signed-off-by: Melanie Buehler <[email protected]> * Fixed two more pdf references Signed-off-by: Melanie Buehler <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]>

Signed-off-by: okhleif-IL <[email protected]>

…IExamples into mmqna-image-query

…image-query

Signed-off-by: okhleif-IL <[email protected]>

* fix whisper_server_port issue Signed-off-by: okhleif-IL <[email protected]> * removed unneeded line Signed-off-by: okhleif-IL <[email protected]> * fix for megaservice curls Signed-off-by: okhleif-IL <[email protected]> * removed v1/encode and reverted accidental change Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: Harsha Ramayanam <[email protected]>

github-actions · 2025-01-10T22:31:35Z

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

MultimodalQnA/ui/gradio/requirements.txt

for more information, see https://pre-commit.ci

mhbuehler · 2025-01-10T22:52:03Z

MultimodalQnA/Dockerfile

@@ -13,7 +13,8 @@ RUN useradd -m -s /bin/bash user && \
    chown -R user /home/user/

 WORKDIR /home/user/
-RUN git clone https://github.com/opea-project/GenAIComps.git
+#RUN git clone https://github.com/opea-project/GenAIComps.git
+RUN git clone https://github.com/mhbuehler/GenAIComps.git --single-branch --branch mmqna-image-query


This needs to be reverted before merge.

mhbuehler · 2025-01-10T22:53:53Z

MultimodalQnA/tests/test_compose_on_gaudi.sh


 function build_docker_images() {
    cd $WORKPATH/docker_image_build
-    git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
+    git clone https://github.com/mhbuehler/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"mmqna-image-query"}" && cd ../


This needs to be reverted before merge.

mhbuehler · 2025-01-10T22:54:30Z

MultimodalQnA/tests/test_compose_on_xeon.sh


 function build_docker_images() {
    cd $WORKPATH/docker_image_build
-    git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
+    git clone https://github.com/mhbuehler/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"mmqna-image-query"}" && cd ../


This needs to be reverted before merge.

Signed-off-by: dmsuehir <[email protected]>

dmsuehir and others added 19 commits December 16, 2024 10:01

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

6ccced2

…image-query Signed-off-by: dmsuehir <[email protected]>

Added logic for image_query

7e05f57

Signed-off-by: Harsha Ramayanam <[email protected]>

Merge branch 'hramayan/img_query' into mmqna-image-query

e25ada0

Update branch used for testing (#35)

480bb4a

Signed-off-by: dmsuehir <[email protected]>

Merge branch 'main' into mmqna-image-query

a0487bb

Fix comps clone so it points to our branch (#36)

d2f8f64

Signed-off-by: Melanie Buehler <[email protected]>

Added fix for video upload format

c4d569c

Signed-off-by: Harsha Ramayanam <[email protected]>

Merge pull request #38 from mhbuehler/hramayan/vid_upload_bug_fix

8360618

Added fix for video upload format

Adds transcript to MMQnA conversation (#37)

8b54159

* Add caption to chat history Signed-off-by: Melanie Buehler <[email protected]> * Fixed some issues with conversation history Signed-off-by: Melanie Buehler <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]>

merging main

72c0709

Signed-off-by: okhleif-IL <[email protected]>

Merge branch 'mmqna-image-query' of https://github.com/mhbuehler/GenA…

d14ab65

…IExamples into mmqna-image-query

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into mmqna-…

b67d3ce

…image-query

removed merge conflicts

05812b5

Signed-off-by: okhleif-IL <[email protected]>

Fixes bug with image query (#40)

675735e

Signed-off-by: Harsha Ramayanam <[email protected]>

ashahba self-requested a review January 10, 2025 22:32

ashahba added this to the v1.2 milestone Jan 10, 2025

ashahba added WIP r1.2 OPEA 1.2 RELEASE TAG labels Jan 10, 2025

mhbuehler and others added 2 commits January 10, 2025 14:46

Merge branch 'main' into mmqna-image-query

21c3f88

[pre-commit.ci] auto fixes from pre-commit.com hooks

da1e6d2

for more information, see https://pre-commit.ci

mhbuehler commented Jan 10, 2025

View reviewed changes

mhbuehler mentioned this pull request Jan 10, 2025

MultimodalQnA image query, pdf, and dynamic ports opea-project/GenAIComps#1134

Open

3 tasks

Add asr to list of images to build in tests (#41)

5961795

Signed-off-by: dmsuehir <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MultimodalQnA image query, pdf, dynamic ports, and UI updates #1381

MultimodalQnA image query, pdf, dynamic ports, and UI updates #1381

mhbuehler commented Jan 10, 2025

github-actions bot commented Jan 10, 2025 •

edited

Loading

mhbuehler Jan 10, 2025

mhbuehler Jan 10, 2025

mhbuehler Jan 10, 2025

MultimodalQnA image query, pdf, dynamic ports, and UI updates #1381

Are you sure you want to change the base?

MultimodalQnA image query, pdf, dynamic ports, and UI updates #1381

Conversation

mhbuehler commented Jan 10, 2025

Description

Issues

Type of change

Dependencies

Tests

github-actions bot commented Jan 10, 2025 • edited Loading

Dependency Review

Scanned Files

mhbuehler Jan 10, 2025

Choose a reason for hiding this comment

mhbuehler Jan 10, 2025

Choose a reason for hiding this comment

mhbuehler Jan 10, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 10, 2025 •

edited

Loading