Merge branch 'main' into kosmos2

.ci/ignore_pip_conflicts.txt

@@ -13,3 +13,5 @@ notebooks/256-bark-text-to-audio/256-bark-text-to-audio.ipynb  # torch==1.13
 notebooks/257-llava-multimodal-chatbot/257-llava-multimodal-chatbot.ipynb # transformers<4.35
 notebooks/257-llava-multimodal-chatbot/257-videollava-multimodal-chatbot.ipynb # transformers<4.35
 notebooks/273-stable-zephyr-3b-chatbot/273-stable-zephyr-3b-chatbot.ipynb # install requirements.txt after clone repo
+notebooks/279-mobilevlm-language-assistant/279-mobilevlm-language-assistant.ipynb # transformers<4.35
+notebooks/280-depth-anything/280-depth-anything.ipynb # install requirements.txt after clone repo

.ci/ignore_treon_mac.txt

@@ -46,4 +46,5 @@
 273-stable-zephyr-3b-chatbot
 276-stable-diffusion-torchdynamo-backend
 278-kosmos2-multimodal-large-language-model
+279-mobilevlm-language-assistant
 404-style-transfer-webcam

.ci/spellcheck/.pyspelling.wordlist.txt

@@ -101,6 +101,7 @@ convolutional
 coreference
 CoSENT
 CPUs
+cpu
 CRNN
 CSV
 CTC
@@ -141,6 +142,7 @@ denoising
 denormalization
 denormalized
 depainting
+DepthAnything
 detections
 Dettmers
 dev
@@ -168,6 +170,7 @@ Dollár
 donut
 Downloader
 downloader
+downsample
 downsampled
 DPO
 dpredictor
@@ -272,6 +275,7 @@ intel
 invertible
 intervaling
 im
+ip
 IRs
 iteratively
 JFLEG
@@ -315,6 +319,7 @@ LCMs
 LCMScheduler
 LDM
 LDMs
+LDP
 learnable
 LeViT
 LibriSpeech
@@ -345,6 +350,8 @@ Mapillary
 Markovian
 Martyniuk
 maskrcnn
+mathbf
+MBs
 MediaPipe
 mel
 Mels
@@ -362,10 +369,14 @@ mms
 MMS
 MLLM
 MLLMs
+MMVLM
 MLP
+MobileLLaMA
 mobilenet
 MobileNet
 MobileNetV
+mobilevlm
+MobileVLM
 modelled
 Modelscope
 Mody
@@ -461,6 +472,7 @@ parametrize
 parsers
 perceptron
 Patil
+PEFT
 performant
 PersonaGPT
 photorealism
@@ -762,3 +774,4 @@ ZavyChromaXL
 Zongyuan
 ZeroScope
 zeroscope
+xformers

.docker/Pipfile

@@ -24,11 +24,11 @@ matplotlib = ">=3.4"
 monai = ">=0.9.1,<1.0.0"
 nbval = "*"
 notebook = "<7.0.0"
-nncf = "==2.7.0"
+nncf = "==2.8.0"
 numpy = ">=1.21.0"
 onnx = ">=1.11.0"
 opencv-python = "*"
-openvino-dev = {version = "==2023.2.0"}
+openvino-dev = {version = "==2023.3.0"}
 openvino-telemetry = "==2023.2.1"
 ovmsclient = "*"
 Pillow = ">=8.3.2"

README.md

@@ -49,7 +49,10 @@ Check out the latest notebooks that show how to optimize and deploy popular mode
 | [Segmind-VegaRT](notebooks/248-stable-diffusion-xl/248-segmind-vegart.ipynb)<br> | High-resolution image generation with Segmind-VegaRT and OpenVINO | <img src="https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/66bfe823-01c8-4749-a8aa-419a1d78a070" width=300> |
 | [Stable-Zephyr chatbot](notebooks/273-stable-zephyr-3b-chatbot/)<br> | Use Stable-Zephyr as chatbot assistant with OpenVINO | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/cfac6ddb-6f22-4343-855c-e513269cf2bf width=300> |
 | [Efficient-SAM](notebooks/274-efficient-sam)<br> | Object segmentation with EfficientSAM and OpenVINO | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/15d0a23a-0550-43c6-9ca9-f689e772a79a width=300> |
-| [LLM Instruction following pipeline](notebooks/275-llm-question-answering)<br> | Usage variety of LLM models for answering questions using OpenVINO | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/daafd702-5a42-4f54-ae72-2e4480d73501 width=300> | 
+| [LLM Instruction following pipeline](notebooks/275-llm-question-answering)<br> | Usage variety of LLM models for answering questions using OpenVINO | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/daafd702-5a42-4f54-ae72-2e4480d73501 width=300> |
+|[Stable Diffusion with IP-Adapter](notebooks/278-stable-diffusion-ip-adapter)<br> | Image conditioning in Stable Diffusion pipeline using IP-Adapter | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/182657d9-2aa3-40b3-9fc4-a90b803419fe width=300> |
+| [MobileVLM](notebooks/279-mobilevlm-language-assistant)<br> | Mobile language assistant with MobileVLM and OpenVINO | |
+| [DepthAnything](notebooks/280-depth-anything)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F280-depth-anythingh%2F280-depth-anything.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/280-depth-anything/280-depth-anything.ipynb) | Monocular Depth estimation with DepthAnything and OpenVINO | <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/a9a16658-512f-470c-a33c-0e1f9d0ae72c width=300> |
 
 ## Table of Contents
 
@@ -90,7 +93,7 @@ OpenVINO Notebooks require Python and Git. To get started, select the guide for
 
 The Jupyter notebooks are categorized into four classes, select one related to your needs or give them all a try. Good Luck!
 
-**NOTE: The main branch of this repository was updated to support the new OpenVINO 2023.2 release.** To upgrade to the new release version, please run `pip install --upgrade -r requirements.txt` in your `openvino_env` virtual environment. If you need to install for the first time, see the [Installation Guide](#-installation-guide) section below. If you wish to use the previous release version of OpenVINO, please checkout the [2023.1 branch](https://github.com/openvinotoolkit/openvino_notebooks/tree/2023.1). If you wish to use the previous Long Term Support (LTS) version of OpenVINO check out the [2022.3 branch](https://github.com/openvinotoolkit/openvino_notebooks/tree/2022.3). 
+**NOTE: The main branch of this repository was updated to support the new OpenVINO 2023.3 release.** To upgrade to the new release version, please run `pip install --upgrade -r requirements.txt` in your `openvino_env` virtual environment. If you need to install for the first time, see the [Installation Guide](#-installation-guide) section below. If you wish to use the previous release version of OpenVINO, please checkout the [2023.2 branch](https://github.com/openvinotoolkit/openvino_notebooks/tree/2023.2). If you wish to use the previous Long Term Support (LTS) version of OpenVINO check out the [2022.3 branch](https://github.com/openvinotoolkit/openvino_notebooks/tree/2022.3).
 
 If you need help, please start a GitHub [Discussion](https://github.com/openvinotoolkit/openvino_notebooks/discussions).  
 
@@ -228,6 +231,10 @@ Demos that demonstrate inference on a particular model.
 | [276-stable-diffusion-torchdynamo-backend](notebooks/276-stable-diffusion-torchdynamo-backend/)<br> | Image generation with Stable Diffusion and OpenVINO™ `torch.compile` feature | <img src=https://user-images.githubusercontent.com/32199725/270969617-ee88eb42-3b0d-42f4-ac71-992c40451849.png width=225> |
 | [277-amused-lightweight-text-to-image](notebooks/277-amused-lightweight-text-to-image)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/277-amused-lightweight-text-to-image/277-amused-lightweight-text-to-image.ipynb)<br>| Lightweight image generation with aMUSEd and OpenVINO™ | <img src=https://huggingface.co/amused/amused-256/resolve/main/assets/collage_small.png width=225> | 
 | [278-kosmos2-multimodal-large-language-model](notebooks/278-kosmos2-multimodal-large-language-model)<br> | Kosmos-2: Multimodal Large Language Model and OpenVINO™ | <img src="[https://github.com/googlestaging/frame-interpolation/raw/main/moment.gif](https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/annotated_snowman.jpg)" width=225> |
+| [278-stable-diffusion-ip-adapter](notebooks/278-stable-diffusion-ip-adapter)<br> | Image conditioning in Stable Diffusion pipeline using IP-Adapter |  <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/182657d9-2aa3-40b3-9fc4-a90b803419fe width=300> |
+| [279-mobilevlm-language-assistant](notebooks/279-mobilevlm-language-assistant)<br> | Mobile language assistant with MobileVLM and OpenVINO | |
+| [280-depth-anything](notebooks/280-depth-anything)<br>[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/openvinotoolkit/openvino_notebooks/HEAD?filepath=notebooks%2F280-depth-anythingh%2F280-depth-anything.ipynb)<br>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openvinotoolkit/openvino_notebooks/blob/main/notebooks/280-depth-anything/280-depth-anything.ipynb) | Monocular Depth Estimation with DepthAnything and OpenVINO |  <img src=https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/a9a16658-512f-470c-a33c-0e1f9d0ae72c width=225> |
+
 
 <div id='-model-training'></div>
 

notebooks/002-openvino-api/002-openvino-api.ipynb

@@ -117,14 +117,14 @@
     "\n",
     "After initializing OpenVINO Runtime, first read the model file with `read_model()`, then compile it to the specified device with the `compile_model()` method. \n",
     "\n",
-    "[OpenVINO™ supports several model formats](https://docs.openvino.ai/2023.0/Supported_Model_Formats.html#doxid-supported-model-formats) and enables developers to convert them to its own OpenVINO IR format using a tool dedicated to this task.\n",
+    "[OpenVINO™ supports several model formats](https://docs.openvino.ai/2023.3/Supported_Model_Formats.html) and enables developers to convert them to its own OpenVINO IR format using a tool dedicated to this task.\n",
     "\n",
     "### OpenVINO IR Model\n",
     "[back to top ⬆️](#Table-of-contents:)\n",
     "\n",
     "An OpenVINO IR (Intermediate Representation) model consists of an `.xml` file, containing information about network topology, and a `.bin` file, containing the weights and biases binary data. Models in OpenVINO IR format are obtained by using model conversion API. The `read_model()` function expects the `.bin` weights file to have the same filename and be located in the same directory as the `.xml` file: `model_weights_file == Path(model_xml).with_suffix(\".bin\")`. If this is the case, specifying the weights file is optional. If the weights file has a different filename, it can be specified using the `weights` parameter in `read_model()`.\n",
     "\n",
-    "The OpenVINO [Model Conversion API](https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html) tool is used to convert models to OpenVINO IR format. Model conversion API reads the original model and creates an OpenVINO IR model (`.xml` and `.bin` files) so inference can be performed without delays due to format conversion. Optionally, model conversion API can adjust the model to be more suitable for inference, for example, by alternating input shapes, embedding preprocessing and cutting training parts off.\n",
+    "The OpenVINO [Model Conversion API](https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html) tool is used to convert models to OpenVINO IR format. Model conversion API reads the original model and creates an OpenVINO IR model (`.xml` and `.bin` files) so inference can be performed without delays due to format conversion. Optionally, model conversion API can adjust the model to be more suitable for inference, for example, by alternating input shapes, embedding preprocessing and cutting training parts off.\n",
     "For information on how to convert your existing TensorFlow, PyTorch or ONNX model to OpenVINO IR format with model conversion API, refer to the [tensorflow-to-openvino](../101-tensorflow-classification-to-openvino/101-tensorflow-classification-to-openvino.ipynb) and [pytorch-onnx-to-openvino](../102-pytorch-onnx-to-openvino/102-pytorch-onnx-to-openvino.ipynb) notebooks. "
    ]
   },

notebooks/101-tensorflow-classification-to-openvino/101-tensorflow-classification-to-openvino.ipynb

@@ -8,7 +8,7 @@
    "source": [
     "# Convert a TensorFlow Model to OpenVINO™\n",
     "\n",
-    "This short tutorial shows how to convert a TensorFlow [MobileNetV3](https://docs.openvino.ai/2023.0/omz_models_model_mobilenet_v3_small_1_0_224_tf.html) image classification model to OpenVINO [Intermediate Representation](https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_IR_and_opsets.html) (OpenVINO IR) format, using [Model Conversion API](https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html). After creating the OpenVINO IR, load the model in [OpenVINO Runtime](https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html) and do inference with a sample image.  \n",
+    "This short tutorial shows how to convert a TensorFlow [MobileNetV3](https://docs.openvino.ai/2023.0/omz_models_model_mobilenet_v3_small_1_0_224_tf.html) image classification model to OpenVINO [Intermediate Representation](https://docs.openvino.ai/2023.3/openvino_docs_MO_DG_IR_and_opsets.html) (OpenVINO IR) format, using [Model Conversion API](https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html). After creating the OpenVINO IR, load the model in [OpenVINO Runtime](https://docs.openvino.ai/nightly/openvino_docs_OV_UG_OV_Runtime_User_Guide.html) and do inference with a sample image.  \n",
     "\n",
     "#### Table of contents:\n",
     "- [Imports](#Imports)\n",
@@ -187,7 +187,7 @@
     "[back to top ⬆️](#Table-of-contents:)\n",
     "\n",
     "Use the model conversion Python API to convert the TensorFlow model to OpenVINO IR. The `ov.convert_model` function accept path to saved model directory and returns OpenVINO Model class instance which represents this model. Obtained model is ready to use and to be loaded on a device using `ov.compile_model` or can be saved on a disk using the `ov.save_model` function.\n",
-    "See the [tutorial](https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html) for more information about using model conversion API with TensorFlow models."
+    "See the [tutorial](https://docs.openvino.ai/2023.3/openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html) for more information about using model conversion API with TensorFlow models."
    ]
   },
   {
@@ -418,7 +418,7 @@
     "## Timing\n",
     "[back to top ⬆️](#Table-of-contents:)\n",
     "\n",
-    "Measure the time it takes to do inference on thousand images. This gives an indication of performance. For more accurate benchmarking, use the [Benchmark Tool](https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html) in OpenVINO. Note that many optimizations are possible to improve the performance. "
+    "Measure the time it takes to do inference on thousand images. This gives an indication of performance. For more accurate benchmarking, use the [Benchmark Tool](https://docs.openvino.ai/2023.3/openvino_sample_benchmark_tool.html) in OpenVINO. Note that many optimizations are possible to improve the performance. "
    ]
   },
   {

notebooks/102-pytorch-to-openvino/102-pytorch-onnx-to-openvino.ipynb

@@ -250,7 +250,7 @@
     "### Convert ONNX Model to OpenVINO IR Format\n",
     "[back to top ⬆️](#Table-of-contents:)\n",
     "\n",
-    "To convert the ONNX model to OpenVINO IR with `FP16` precision, use model conversion API. The models are saved inside the current directory. For more information on how to convert models, see this [page](https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html)."
+    "To convert the ONNX model to OpenVINO IR with `FP16` precision, use model conversion API. The models are saved inside the current directory. For more information on how to convert models, see this [page](https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html)."
    ]
   },
   {
@@ -700,7 +700,7 @@
     "## Performance Comparison\n",
     "[back to top ⬆️](#Table-of-contents:)\n",
     "\n",
-    "Measure the time it takes to do inference on twenty images. This gives an indication of performance. For more accurate benchmarking, use the [Benchmark Tool](https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html). Keep in mind that many optimizations are possible to improve the performance. "
+    "Measure the time it takes to do inference on twenty images. This gives an indication of performance. For more accurate benchmarking, use the [Benchmark Tool](https://docs.openvino.ai/2023.3/openvino_sample_benchmark_tool.html). Keep in mind that many optimizations are possible to improve the performance. "
    ]
   },
   {
@@ -863,8 +863,8 @@
     "* [Pytorch ONNX Documentation](https://pytorch.org/docs/stable/onnx.html)\n",
     "* [PIP install openvino-dev](https://pypi.org/project/openvino-dev/)\n",
     "* [OpenVINO ONNX support](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_ONNX_Support.html)\n",
-    "* [Model Conversion API documentation](https://docs.openvino.ai/2023.0/openvino_docs_model_processing_introduction.html)\n",
-    "* [Converting Pytorch model](https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_PyTorch.html)\n"
+    "* [Model Conversion API documentation](https://docs.openvino.ai/2023.3/openvino_docs_model_processing_introduction.html)\n",
+    "* [Converting Pytorch model](https://docs.openvino.ai/2023.3/openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_PyTorch.html)\n"
    ]
   }
  ],