PromptClip_visual.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "This notebook walks you through the process of creating clips with LLM prompts using visual information.\n",
    "\n",
    "The sample video provided in the walkthrough is an episode of Mr. Bean, which relies heavily on visual comedy rather than spoken words. Therefore, visual information is the most effective way to create the clip.\n",
    "\n",
    "We will create two clips:\n",
    "\n",
    "<ol>\n",
    "  <p></p>\n",
    "  <li> The famous Mr. Bean copying in an exam meme.</li>\n",
    "  <p></p>\n",
    "  <li>A compilation of all the car gags from the episode.</li>\n",
    "</ol>\n",
    "\n",
    "After the walkthrough, you can pick your own video, decide on your prompt, and generate a new clip ⚡️\n",
    "\n",
    "It's as simple as it sounds.\n",
    "\n",
    "If you want to go extra mile you can add Image Overlays or Audio overlays on these clips.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# But first, let's install the dependecies.\n",
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading The Video\n",
    "\n",
    "Before proceeding, ensure access to [VideoDB](https://videodb.io) API key. If not, sign up for API access on the respective platforms.\n",
    "\n",
    "> Get your API key from [VideoDB Console](https://console.videodb.io). ( Free for first 50 uploads, **No credit card required** ) 🎉\n",
    "\n",
    "You can either source a new video from YouTube or select an existing one from your VideoDB collection.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import videodb\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "\n",
    "# TODO: setup .env file\n",
    "load_dotenv()\n",
    "\n",
    "# Connect to VideoDB\n",
    "conn = videodb.connect()\n",
    "coll = conn.get_collection()\n",
    "\n",
    "# NOTE: Add VISUAL_DEMO_VIDEO_ID in .env if video is already present in your collection\n",
    "video_id = os.getenv(\"VISUAL_DEMO_VIDEO_ID\")\n",
    "video_url = \"https://www.youtube.com/watch?v=7Im2I6STbms\"\n",
    "\n",
    "if not video_id:\n",
    "    video = coll.upload(url=video_url)\n",
    "else:\n",
    "    video = coll.get_video(video_id)\n",
    "\n",
    "print(f\"video_id: {video.id}, name: {video.name}\")\n",
    "video.play()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing The Visual Information\n",
    "\n",
    "Here, you can either set the `scene_index_id` in cell or `VISUAL_DEMO_SCENE_INDEX_ID` in the .env, if already indexed, or leave it blank to index the video for visual retrieval.\n",
    "\n",
    "To know more about scene indexing click [here](https://docs.videodb.io/scene-index-guide-80).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#NOTE: Add VISUAL_DEMO_SCENE_INDEX_ID in the .env if already indexed.\n",
    "scene_index_id = os.getenv(\"VISUAL_DEMO_SCENE_INDEX_ID\")\n",
    "\n",
    "if not scene_index_id:\n",
    "    scene_index_id = video.index_scenes(\n",
    "        extraction_config={\n",
    "            \"threshold\": 20, \n",
    "            \"frame_count\": 3\n",
    "        },\n",
    "        prompt=\"Summarize the essence of the scene in one or two concise sentences without focusing on individual images.\"\n",
    "    )\n",
    "scenes = video.get_scene_index(scene_index_id)\n",
    "print(f\"Video is indexed with scene_index_id {scene_index_id} with {len(scenes)} scenes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting The Indexed Scenes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for scene in scenes:\n",
    "    print(f\"{scene['start']}-{scene['end']}: {scene['description']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Results From Your Prompt\n",
    "\n",
    "The `scene_prompter` function in `video_prompter.py` takes the indexed scenes, chunks them, and then parallelly calls the LLM with user and system prompts to retrieve the desired matching scenes.\n",
    "\n",
    "To create a clip using the `scene_prompter` function from a video, it's crucial to craft a specific prompt that will help identify the most relevant segments for your use case. This prompt should highlight the themes, activity, or specific visual cues you're interested in. \n",
    "\n",
    "The `scene_prompter` will return sentences or segments from the video that match your prompt. Review these to ensure they align with your needs. You can then use these segments to create your clip, focusing on the content that's most relevant to your use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from video_prompter import scene_prompter\n",
    "\n",
    "# This prompt is for finding the iconic copying in examination scene of Mr. Bean\n",
    "user_prompt = \"find the moment where mr.bean is attempting to cheat peeking over at the answer sheet of man beside him, find it with high accuracy.\"\n",
    "result = scene_prompter(scenes, user_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate The Clip\n",
    "To generate a clip, first we'll call `get_result_timestamps` from `video_prompter.py` it uses VideoDB's `keyword search` feature. We already leveraged the power of the LLM (Large Language Model) to identify relevant sentences. We'll use the search results to create a programmable video stream. Here's how you can approach this process:\n",
    "\n",
    "We have the descriptions in the `results` variable. We input these keywords into VideoDB's keyword search feature. This function will search through the indexed scenes of your videos to find matches.\n",
    "\n",
    "The search will return a SearchResult object, which contains detailed information about the found segments, including their timestamps, the text of the scene description.\n",
    "\n",
    "**Create a Programmable Video Stream with Timeline**: With the specific segments identified, you can now use `build_video_timeline` from `video_prompter.py` to get the Timeline to create a new programmable video stream. The Timeline tool allows you to stitch together video segments based on the timestamps provided in the SearchResult object. You can arrange, cut, or combine these segments to craft a fresh video stream that focuses on your topic of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from videodb import play_stream\n",
    "from videodb.timeline import Timeline\n",
    "from video_prompter import get_result_timestamps, build_video_timeline\n",
    "\n",
    "timeline = Timeline(conn)\n",
    "\n",
    "result_timestamps = get_result_timestamps(video, result, scene_index_id=scene_index_id)\n",
    "print(f\"We have got {len(result_timestamps)} segments matching with the user prompt.\")\n",
    "# Since we are only interested in one segment for the meme, we've hardcoded the timestamp to filter it out.\n",
    "# In an actual scenario, you can inspect all the segments and select the ones you're interested in.\n",
    "# Alternatively, you can skip the filtering if you want a clip of all the segments involving cheating.\n",
    "meme_start_time = 370.4\n",
    "meme_result = [next((item for item in result_timestamps if item[0] == meme_start_time), None)]\n",
    "if meme_result:\n",
    "    print(\"Selecting the segment with the meme.\")\n",
    "else:\n",
    "    print(f\"Segment with start {meme_start_time} not found.\")\n",
    "    print(\"Taking the first segment instead, please inspect the user_prompt and result_timestamps if you are intrested only in the meme.\")\n",
    "    meme_result = result_timestamps[:1]\n",
    "result_timestamps = meme_result\n",
    "timeline, duration = build_video_timeline(video, meme_result, timeline)\n",
    "stream = timeline.generate_stream()\n",
    "play_stream(stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stream With Text Overlay\n",
    "\n",
    "You can add custom text to the meme for further personalization.\n",
    "\n",
    "For more customization options, refer to the [TextAsset Styling Guide](https://docs.videodb.io/guide-textasset-75)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from videodb.asset import TextStyle\n",
    "from videodb.timeline import TextAsset\n",
    "\n",
    "timeline = Timeline(conn)\n",
    "timeline, duration = build_video_timeline(video, result_timestamps, timeline, top_n=1)\n",
    "left = TextAsset(\n",
    "    text=\"XXXX\",\n",
    "    duration=duration,\n",
    "    style=TextStyle(\n",
    "        x=190,\n",
    "        y=15,\n",
    "        font = \"Inter\",\n",
    "        fontsize = 25,\n",
    "        fontcolor = \"#002869\",\n",
    "    )\n",
    ")\n",
    "right = TextAsset(\n",
    "    text=\"YYYY\",\n",
    "    duration=duration,\n",
    "    style=TextStyle(\n",
    "        x=420,\n",
    "        y=15,\n",
    "        font = \"Inter\",\n",
    "        fontsize = 25, \n",
    "        fontcolor = \"#00692c\",\n",
    "    )\n",
    ")\n",
    "timeline.add_overlay(0, left)\n",
    "timeline.add_overlay(0, right)\n",
    "stream = timeline.generate_stream()\n",
    "play_stream(stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stream With Image Overlay \n",
    "\n",
    "You can also add image overlays if needed. For more details, refer to the [Dynamic Video Stream Guide](https://docs.videodb.io/dynamic-video-stream-guide-44)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from videodb.asset import TextStyle, ImageAsset\n",
    "from videodb import MediaType\n",
    "\n",
    "timeline = Timeline(conn)\n",
    "timeline, duration = build_video_timeline(video, result_timestamps, timeline, top_n=1)\n",
    "\n",
    "image1_id = os.getenv(\"VISUAL_DEMO_IMAGE_1\")\n",
    "if not image1_id:\n",
    "    image1_url = \"https://upload.wikimedia.org/wikipedia/sco/thumb/d/d1/Ferrari-Logo.svg/344px-Ferrari-Logo.svg.png\"\n",
    "    image1 = coll.upload(url=image1_url, media_type=MediaType.image)\n",
    "    image1_id = image1.id\n",
    "    print(f\"image1_id: {image1_id}\")\n",
    "\n",
    "image2_id = os.getenv(\"VISUAL_DEMO_IMAGE_2\")\n",
    "if not image2_id:\n",
    "    image2_url = \"https://upload.wikimedia.org/wikipedia/en/thumb/6/66/McLaren_Racing_logo.svg/512px-McLaren_Racing_logo.svg.png\"\n",
    "    image2 = coll.upload(url=image2_url, media_type=MediaType.image)\n",
    "    image2_id = image2.id\n",
    "    print(f\"image2_id: {image2_id}\")\n",
    "\n",
    "left = ImageAsset(\n",
    "    asset_id=image1_id,\n",
    "    duration=duration,\n",
    "    width=70,\n",
    "    height=124,\n",
    "    x=150,\n",
    "    y=200,\n",
    ")\n",
    "right = ImageAsset(\n",
    "    asset_id=image2_id,\n",
    "    duration=duration,\n",
    "    width=128,\n",
    "    height=40,\n",
    "    x=400,\n",
    "    y=240\n",
    ")\n",
    "\n",
    "timeline.add_overlay(0, left)\n",
    "timeline.add_overlay(0, right)\n",
    "stream = timeline.generate_stream()\n",
    "play_stream(stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Another Prompt With Simple Compilation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_prompt = \"find all the car gags with high accuracy\"\n",
    "result = scene_prompter(scenes, user_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timeline = Timeline(conn)\n",
    "result_timestamps = get_result_timestamps(video, result, scene_index_id=scene_index_id)\n",
    "timeline, duration  = build_video_timeline(video, result_timestamps, timeline)\n",
    "stream = timeline.generate_stream()\n",
    "play_stream(stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have any questions or feedback. Feel free to reach out to us 🙌🏼\n",
    "\n",
    "* [Discord](https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fdiscord.gg%2Fpy9P639jGz)\n",
    "* [GitHub](https://github.com/video-db)\n",
    "* [Website](https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fvideodb.io)\n",
    "* [Email](ashu@videodb.io)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Playground For You To Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "video_url = \"\"\n",
    "video_id = \"\"\n",
    "\n",
    "if not video_id:\n",
    "    video = coll.upload(url=video_url)\n",
    "else:\n",
    "    video = coll.get_video(video_id)\n",
    "\n",
    "print(f\"video_id: {video.id}, name: {video.name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scene_index_id = \"\"\n",
    "\n",
    "if not scene_index_id:\n",
    "    scene_index_id = video.index_scenes(\n",
    "        extraction_config={\n",
    "            \"threshold\": 20, \n",
    "            \"frame_count\": 3\n",
    "        },\n",
    "        prompt=\"Summarize the essence of the scene in one or two concise sentences without focusing on individual images.\"\n",
    "    )\n",
    "scenes = video.get_scene_index(scene_index_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for scene in scenes:\n",
    "    print(f\"{scene['start']}-{scene['end']}: {scene['description']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from video_prompter import scene_prompter\n",
    "\n",
    "user_prompt = \"\"\n",
    "result = scene_prompter(scenes, user_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timeline = Timeline(conn)\n",
    "result_timestamps = get_result_timestamps(video, result, scene_index_id=scene_index_id)\n",
    "timeline, duration  = build_video_timeline(video, result_timestamps, timeline)\n",
    "stream = timeline.generate_stream()\n",
    "play_stream(stream)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}