Merge remote-tracking branch 'upstream/main' into add_openvoice

.ci/ignore_treon_docker.txt

@@ -1,6 +1,5 @@
 107-speech-recognition-quantization
 108-gpu-device
-111-yolov5-quantization-migration
 117-model-server
 122-quantizing-model-with-accuracy-control
 204-segmenter-semantic-segmentation
@@ -45,6 +44,7 @@
 272-paint-by-example
 273-stable-zephyr-3b-chatbot
 276-stable-diffusion-torchdynamo-backend
+281-kosmos2-multimodal-large-language-model
 301-tensorflow-training-openvino
 305-tensorflow-quantization-aware-training
 404-style-transfer-webcam

.ci/ignore_treon_linux.txt

@@ -1,6 +1,5 @@
 107-speech-recognition-quantization
 108-gpu-device
-111-yolov5-quantization-migration
 112-pytorch-post-training-quantization-nncf
 117-model-server
 122-quantizing-model-with-accuracy-control
@@ -48,4 +47,5 @@
 272-paint-by-example
 273-stable-zephyr-3b-chatbot
 276-stable-diffusion-torchdynamo-backend
+281-kosmos2-multimodal-large-language-model
 404-style-transfer-webcam

.ci/ignore_treon_mac.txt

@@ -1,6 +1,5 @@
 107-speech-recognition-quantization
 108-gpu-device
-111-yolov5-quantization-migration
 117-model-server
 122-quantizing-model-with-accuracy-control
 204-segmenter-semantic-segmentation
@@ -45,5 +44,6 @@
 272-paint-by-example
 273-stable-zephyr-3b-chatbot
 276-stable-diffusion-torchdynamo-backend
+281-kosmos2-multimodal-large-language-model
 279-mobilevlm-language-assistant
 404-style-transfer-webcam

.ci/ignore_treon_win.txt

@@ -1,6 +1,5 @@
 107-speech-recognition-quantization
 108-gpu-device
-111-yolov5-quantization-migration
 117-model-server
 122-quantizing-model-with-accuracy-control
 204-segmenter-semantic-segmentation
@@ -47,3 +46,4 @@
 272-paint-by-example
 273-stable-zephyr-3b-chatbot
 276-stable-diffusion-torchdynamo-backend
+281-kosmos2-multimodal-large-language-model

.ci/spellcheck/.pyspelling.wordlist.txt

@@ -37,6 +37,7 @@ backend
 backends
 Baevski
 BasicUNet
+bboxes
 BEiT
 Belrose
 Benchmarking
@@ -87,6 +88,7 @@ configs
 Connectionist
 ContentVec
 Contrastive
+contrastive
 ControlNet
 controlnet
 ConvE
@@ -98,6 +100,7 @@ ConvNeXt
 ConvNeXts
 Convolutional
 convolutional
+coreference
 CoSENT
 CPUs
 cpu
@@ -298,6 +301,9 @@ KiTS
 Koltun
 Kondate
 Kosaraju
+kosmos
+Kosmos
+KOSMOS
 KServe
 Kubernetes
 Kupyn
@@ -363,6 +369,8 @@ mistralai
 MLS
 mms
 MMS
+MLLM
+MLLMs
 MMVLM
 MLP
 MobileLLaMA
@@ -601,6 +609,8 @@ Shazeer
 Shutterstock
 siggraph
 sigmoid
+SigLIP
+siglip
 SISR
 SlimOrca
 SlowFast

README.md

@@ -53,6 +53,7 @@ Check out the latest notebooks that show how to optimize and deploy popular mode
 |[Stable Diffusion with IP-Adapter](notebooks/278-stable-diffusion-ip-adapter)<br> | Image conditioning in Stable Diffusion pipeline using IP-Adapter | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/182657d9-2aa3-40b3-9fc4-a90b803419fe width=300> |
 | [MobileVLM](notebooks/279-mobilevlm-language-assistant)<br> | Mobile language assistant with MobileVLM and OpenVINO | |
 | [DepthAnything](notebooks/280-depth-anything)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F280-depth-anythingh%2F280-depth-anything.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/280-depth-anything/280-depth-anything.ipynb) | Monocular Depth estimation with DepthAnything and OpenVINO | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/a9a16658-512f-470c-a33c-0e1f9d0ae72c width=300> |
+| [Kosmos-2: Grounding Multimodal Large Language Models](notebooks/281-kosmos2-multimodal-large-language-model)<br> | Kosmos-2: Grounding Multimodal Large Language Model and OpenVINO™ | <img src=https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/annotated_snowman.jpg width=225> |
 | [OpenVoice](notebooks/284-depth-)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F284-openvoice%2F284-openvoice.ipynb)[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/284-openvoice/284-openvoice.ipynb) | OpenVoice a versatile instant voice tone transferring and generating speech in various languages. |<img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/5703039/ca7eab80-148d-45b0-84e8-a5a279846b51 width=300> |
 
 ## Table of Contents
@@ -127,7 +128,6 @@ Tutorials that explain how to optimize and quantize models with OpenVINO tools.
 | [108-gpu-device](notebooks/108-gpu-device/) | Working with GPUs in OpenVINO™                                                                                              |
 | [109-performance-tricks](notebooks/109-performance-tricks/)| Performance tricks in OpenVINO™                                                                                             |
 | [110-ct-segmentation-quantize](notebooks/110-ct-segmentation-quantize/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F110-ct-segmentation-quantize%2F110-ct-scan-live-inference.ipynb) | Quantize a kidney segmentation model and show live inference                                                                | 
-| [111-yolov5-quantization-migration](notebooks/111-yolov5-quantization-migration)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/111-yolov5-quantization-migration/111-yolov5-quantization-migration.ipynb) | Migrate YOLOv5 POT API based quantization pipeline on Neural Network Compression Framework (NNCF)                           | <img src = "https://user-images.githubusercontent.com/44352144/177097174-cfe78939-e946-445e-9fce-d8897417ef8e.png"  width=225> |
 | [112-pytorch-post-training-quantization-nncf](notebooks/112-pytorch-post-training-quantization-nncf/) | Use Neural Network Compression Framework (NNCF) to quantize PyTorch model in post-training mode (without model fine-tuning) | 
 | [113-image-classification-quantization](notebooks/113-image-classification-quantization/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?labpath=notebooks%2F113-image-classification-quantization%2F113-image-classification-quantization.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/113-image-classification-quantization/113-image-classification-quantization.ipynb) | Quantize Image Classification model                                                                                     |
 | [115-async-api](notebooks/115-async-api/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?labpath=notebooks%2F115-async-api%2F115-async-api.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/115-async-api/115-async-api.ipynb) | Use Asynchronous Execution to Improve Data Pipelining                                                                       | |
@@ -234,6 +234,8 @@ Demos that demonstrate inference on a particular model.
 | [278-stable-diffusion-ip-adapter](notebooks/278-stable-diffusion-ip-adapter)<br> | Image conditioning in Stable Diffusion pipeline using IP-Adapter |  <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/182657d9-2aa3-40b3-9fc4-a90b803419fe width=300> |
 | [279-mobilevlm-language-assistant](notebooks/279-mobilevlm-language-assistant)<br> | Mobile language assistant with MobileVLM and OpenVINO | |
 | [280-depth-anything](notebooks/280-depth-anything)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F280-depth-anythingh%2F280-depth-anything.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/280-depth-anything/280-depth-anything.ipynb) | Monocular Depth Estimation with DepthAnything and OpenVINO |  <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/a9a16658-512f-470c-a33c-0e1f9d0ae72c width=225> |
+| [281-kosmos2-multimodal-large-language-model](notebooks/281-kosmos2-multimodal-large-language-model)<br> | Kosmos-2: Multimodal Large Language Model and OpenVINO™ | <img src=https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/annotated_snowman.jpg width=225> |
+| [282-siglip-zero-shot-image-classification](notebooks/282-siglip-zero-shot-image-classification)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/282-siglip-zero-shot-image-classification/282-siglip-zero-shot-image-classification.ipynb) | Zero-shot Image Classification with SigLIP | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/67365453/c4eb782c-0fef-4a89-a5c6-5cc43518490b width=500> |
 
 <div id='-model-training'></div>
 

README_cn.md

@@ -111,7 +111,6 @@ Jupyter notebooks 分为四个大类，选择一个跟你需求相关的开始
 | [108-gpu-device](notebooks/108-gpu-device/) | 在GPU上使用OpenVINO™                                                                                              |
 | [109-performance-tricks](notebooks/109-performance-tricks/)|  OpenVINO™ 的优化技巧                                                                                            |
 | [110-ct-segmentation-quantize](notebooks/110-ct-segmentation-quantize/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F110-ct-segmentation-quantize%2F110-ct-scan-live-inference.ipynb) | 量化肾脏分割模型并展示实时推理                                                                | 
-| [111-yolov5-quantization-migration](notebooks/111-yolov5-quantization-migration)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/111-yolov5-quantization-migration/111-yolov5-quantization-migration.ipynb) | 在神经网络压缩框架(NNCF)上迁移基于YOLOv5 POT API的量化管道                           | <img src = "https://user-images.githubusercontent.com/44352144/177097174-cfe78939-e946-445e-9fce-d8897417ef8e.png"  width=225> |
 | [112-pytorch-post-training-quantization-nncf](notebooks/112-pytorch-post-training-quantization-nncf/) | 利用神经网络压缩框架(NNCF)在后训练模式下来量化PyTorch模型(无需模型微调) | 
 | [113-image-classification-quantization](notebooks/113-image-classification-quantization/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?labpath=notebooks%2F113-image-classification-quantization%2F113-image-classification-quantization.ipynb) | 量化mobilenet图片分类模型                                                                                      |
 | [115-async-api](notebooks/115-async-api/)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?labpath=notebooks%2F115-async-api%2F115-async-api.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/115-async-api/115-async-api.ipynb) | 使用异步执行改进数据流水线                                                                        | |