GeminiAIP_Colab_VideoAudio_Img.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "35afe990",
   "metadata": {},
   "source": [
    "# Google Colab using LLM & Webscrape Fun\n",
    "\n",
    "# <font color=red>Mr Fugu Data Science</font>\n",
    "\n",
    "# (◕‿◕✿)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78cb0985",
   "metadata": {},
   "source": [
    "# `Google Gemini 1.5 Flash & Pro`\n",
    "\n",
    "+ We will go to the link connected in order to see the baisc landing page allowing us to read documentation, obtain an API key and related materials to start learning.\n",
    "\n",
    "https://deepmind.google/technologies/gemini/flash/\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b7d7161",
   "metadata": {},
   "source": [
    "# `Gemini Flash 1.5 Pricing`\n",
    "\n",
    "https://ai.google.dev/pricing\n",
    "\n",
    "+ What countries can access free usage: https://ai.google.dev/gemini-api/docs/available-regions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d9ab1639",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q -U google-generativeai # Installing Gemini AI for use with API\n",
    "\n",
    "# Google Gemini AI (API)\n",
    "import google.generativeai as genai\n",
    "\n",
    "# File management\n",
    "import os\n",
    "\n",
    "# Ff we call urls\n",
    "import requests\n",
    "\n",
    "# Handle images\n",
    "from IPython.display import Image\n",
    "\n",
    "# Use this if you set the environment variable for the api key: (preferred route when not using notebook)\n",
    "# genai.configure(api_key=os.environ[\"API_KEY\"])\n",
    "\n",
    "# If using File From Colab Notebook Directly, use stored API key \n",
    "from google.colab import userdata\n",
    "api=userdata.get('Google_Gemini_API')\n",
    "\n",
    "# Initializing your key for use\n",
    "genai.configure(api_key=api)\n",
    "\n",
    "model = genai.GenerativeModel('models/gemini-1.5-pro-001')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87266b0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of Available Models: NOT ALL WORK because of deprecation\n",
    "for model_name in genai.list_models():\n",
    "    if 'generateContent'  in model_name.supported_generation_methods:\n",
    "        print(model_name.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42dce1ed",
   "metadata": {},
   "source": [
    "# `Ex 1) Read Text from Image`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2354aa2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Used with Google Colab to collect files:\n",
    "from google.colab import files\n",
    "\n",
    "uploaded = files.upload() # prompt will open and you can choose files, you can also create a path "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbf442ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is an image I used, you will need to put your image here\n",
    "img_=Image('IMG_7850.jpg') # my file for example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca26a69f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is another model we can use, I will use here\n",
    "model = genai.GenerativeModel('gemini-1.5-pro-001')\n",
    "\n",
    "# Text prompt for Gemini AI\n",
    "prompt='analyze and extract all information from image, including best by date and all other information'\n",
    "\n",
    "#Gemini takes our img, prompt and generate a response string from it\n",
    "result=model.generate_content([img_,prompt])\n",
    "\n",
    "\n",
    "# Output text \n",
    "print(result.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fc4e997",
   "metadata": {},
   "source": [
    "# `Document Processing:` \n",
    "\n",
    "+ Setup: https://ai.google.dev/gemini-api/docs/document-processing?lang=python\n",
    "    \n",
    "+ Improve Prompting: https://ai.google.dev/gemini-api/docs/file-prompting-strategies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9681910",
   "metadata": {},
   "source": [
    "# `Ex 2) PDF File Summarize `\n",
    "\n",
    "https://ai.google.dev/gemini-api/docs/document-processing?lang=python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d528ceb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download Article:\n",
    "!curl -o txt2vidDiffModel.pdf https://arxiv.org/html/2403.06098v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76caa481",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \n",
    "sample_file = genai.upload_file(path=\"txt2vidDiffModel.pdf\",\n",
    "                                display_name=\"text2Vid_DiffisionModel PDF\")\n",
    "\n",
    "print(f\"Uploaded file '{sample_file.display_name}' as: {sample_file.uri}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6beea8c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose a Gemini model.\n",
    "model = genai.GenerativeModel(model_name=\"gemini-1.5-pro-001\")\n",
    "\n",
    "# Prompt the model with text and the previously uploaded image.\n",
    "response = model.generate_content([sample_file, \"Can you summarize this document as a bulleted list?\"])\n",
    "\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4107b2fa",
   "metadata": {},
   "source": [
    "# `Ex3) Analyze video`\n",
    "\n",
    "\n",
    "+ There is a caveat here. You can take videos and analyze them but, if you are interested in the audio there needs to be captions/subtitles from the video you are using.\n",
    "+ Also, the video will need to be downloaded somewhere, at this time I am unsure how to stream the data directly without downloading. \n",
    "\n",
    "Now, with this problem I have found a work around to get this done! It may not be the best way but it is a proof of concept at the very least. \n",
    "\n",
    "https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/video-understanding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d84d0522",
   "metadata": {},
   "outputs": [],
   "source": [
    "import vertexai\n",
    "\n",
    "from vertexai.generative_models import GenerativeModel, Part\n",
    "\n",
    "# TODO(developer): Update project_id and location\n",
    "PROJECT_ID='Generative Language API Key'\n",
    "vertexai.init(project=PROJECT_ID, location=\"us-central1\")\n",
    "\n",
    "vision_model = GenerativeModel(\"gemini-1.5-pro-001\")\n",
    "\n",
    "# Generate text\n",
    "response = vision_model.generate_content(\n",
    "    [\n",
    "        Part.from_uri(\n",
    "            \"https://youtu.be/CNbqLm7uqaA\", mime_type=\"video/mp4\"\n",
    "        ),\n",
    "        \"What is in the video?, give a summary with bullet points\",\n",
    "    ]\n",
    ")\n",
    "print(response.text)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dbadb49",
   "metadata": {},
   "source": [
    "# `Ex4) Audio Summarizing or Transcriptions`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fac8ccbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install pytube\n",
    "!pip install pytubefix\n",
    "from pytubefix import YouTube\n",
    "import os\n",
    "\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive/')\n",
    "\n",
    "\n",
    "def sanitize_filename(filename):\n",
    "    return \"\".join(c if c.isalnum() or c in \" ._-\" else \"_\" for c in filename)\n",
    "\n",
    "\n",
    "def audio(thelink, path):\n",
    "    try:\n",
    "        yt = YouTube(thelink)\n",
    "        print('Title:', yt.title)\n",
    "        print('Views:', yt.views)\n",
    "        yd = yt.streams.get_audio_only()\n",
    "        yt_title = sanitize_filename(yt.title)\n",
    "        yd.download(output_path=path, filename=f'{yt_title}.mp3')\n",
    "        print('Finished downloading audio')\n",
    "    except Exception as e:\n",
    "        print(f\"Error: {e}\")\n",
    "\n",
    "\n",
    "def high(thelink, path):\n",
    "    try:\n",
    "        yt = YouTube(thelink)\n",
    "        print('Title:', yt.title)\n",
    "        print('Views:', yt.views)\n",
    "        yt_title = sanitize_filename(yt.title)\n",
    "\n",
    "        # Download the highest resolution video with a specified filename\n",
    "        video_stream = yt.streams.filter().order_by(\"resolution\").last()\n",
    "        audio_stream = yt.streams.get_audio_only()\n",
    "\n",
    "        video_filename = f'{yt_title}.mp4'\n",
    "        audio_filename = f'{yt_title}.mp3'\n",
    "\n",
    "        video_stream.download(output_path=path, filename=video_filename)\n",
    "        audio_stream.download(output_path=path, filename=audio_filename)\n",
    "\n",
    "        print('Finished downloading high resolution video and audio')\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"Error: {e}\")\n",
    "\n",
    "\n",
    "def low(thelink, path):\n",
    "    try:\n",
    "        yt = YouTube(thelink)\n",
    "        print('Title:', yt.title)\n",
    "        print('Views:', yt.views)\n",
    "        yd = yt.streams.get_lowest_resolution()\n",
    "        yt_title = sanitize_filename(yt.title)\n",
    "        yd.download(output_path=path, filename=f'{yt_title}.mp4')\n",
    "        print('Finished downloading low resolution video')\n",
    "    except Exception as e:\n",
    "        print(f\"Error: {e}\")\n",
    "\n",
    "\n",
    "link_inp = input(\"Please Enter the link of the video: \")\n",
    "path_inp = input(\"Please Enter the download path: \")\n",
    "menu_inp = input('Select:\\n1- Audio\\n2- Highest Resolution\\n3- Lowest Resolution\\n')\n",
    "\n",
    "if menu_inp == '1':\n",
    "    audio(link_inp, path_inp)\n",
    "elif menu_inp == '2':\n",
    "    high(link_inp, path_inp)\n",
    "elif menu_inp == '3':\n",
    "    low(link_inp, path_inp)\n",
    "else:\n",
    "    print('Invalid input')\n",
    "    \n",
    "  \n",
    "\n",
    "\n",
    " # ----------------------------------\n",
    "# NOT MY CODE, Came from reddit\n",
    "# https://www.reddit.com/r/learnpython/comments/156p9ju/pytube_errors_help_fix_or_recommend_other_library/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c4cd85b",
   "metadata": {},
   "source": [
    "# `Summarize Audio`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "492c7b0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a Gemini model appropriate for your use case.\n",
    "import google.generativeai as genai\n",
    "import pathlib\n",
    "\n",
    "\n",
    "from google.colab import userdata\n",
    "api=userdata.get('Google_Gemini_API')\n",
    "\n",
    "genai.configure(api_key=api)\n",
    "model = genai.GenerativeModel('models/gemini-1.5-pro-001')\n",
    "\n",
    "# Create the prompt.\n",
    "prompt = \"Please summarize the audio with bullet points and timestamps.\"\n",
    "\n",
    "# Load the samplesmall.mp3 file into a Python Blob object containing the audio\n",
    "# file's bytes and then pass the prompt and the audio to Gemini.\n",
    "response = model.generate_content([\n",
    "    prompt,\n",
    "    {\n",
    "        \"mime_type\": \"audio/mp3\",\n",
    "        \"data\": pathlib.Path('/content/drive/FRUSTRATING Runway ML _ Kling AI Free Tier_s NEED WORK.mp3').read_bytes()\n",
    "    }\n",
    "])\n",
    "\n",
    "# Output Gemini's response to the prompt and the inline audio.\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d572e7c",
   "metadata": {},
   "source": [
    "# `Transcribe with timestamps:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "957ab948",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a Gemini model appropriate for your use case.\n",
    "import google.generativeai as genai\n",
    "import pathlib\n",
    "\n",
    "\n",
    "from google.colab import userdata\n",
    "api=userdata.get('Google_Gemini_API')\n",
    "\n",
    "genai.configure(api_key=api)\n",
    "model = genai.GenerativeModel('models/gemini-1.5-pro-001')\n",
    "\n",
    "# Create the prompt.\n",
    "prompt = \"\"\"Can you transcribe this audio, in the format of timecode,caption and separate each section by a new line\"\"\"\n",
    "\n",
    "# Load the samplesmall.mp3 file into a Python Blob object containing the audio\n",
    "# file's bytes and then pass the prompt and the audio to Gemini.\n",
    "response = model.generate_content([\n",
    "    prompt,\n",
    "    {\n",
    "        \"mime_type\": \"audio/mp3\",\n",
    "        \"data\": pathlib.Path('/content/drive/FRUSTRATING Runway ML _ Kling AI Free Tier_s NEED WORK.mp3').read_bytes()\n",
    "    }\n",
    "])\n",
    "\n",
    "# Output Gemini's response to the prompt and the inline audio.\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dd4009ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ex) Reading a Schema for a database: ER Diagram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5ccf381",
   "metadata": {},
   "outputs": [],
   "source": [
    "uploaded = files.upload()\n",
    "Image('Screen Shot 2024-09-19 at 5.42.39 PM.png')\n",
    "\n",
    "# https://www.csub.edu/~ychoi2/MIS/Assignment/ERD_Practice.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88f9cfda",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import files \n",
    "from IPython.display import Image\n",
    "uploaded = files.upload()\n",
    "\n",
    "img_e=Image('Screen Shot 2024-09-19 at 5.42.39 PM.png')\n",
    "\n",
    "model = genai.GenerativeModel('gemini-1.5-pro-001')\n",
    "\n",
    "prompt = \"Document the entities and relationships in this ER diagram.\"\n",
    "\n",
    "contents = [prompt, img_e]\n",
    "\n",
    "# Use a more deterministic configuration with a low temperature\n",
    "generation_config = model.generate_content(contents)\n",
    "   \n",
    "\n",
    "\n",
    "\n",
    "print(\"-------Prompt--------\")\n",
    "# print_multimodal_prompt(contents)\n",
    "\n",
    "print(\"\\n-------Response--------\")\n",
    "for response in generation_config:\n",
    "    print(response.text, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f843d1df",
   "metadata": {},
   "source": [
    "# Caching Large requests to lower token usage and context, like short term memory..\n",
    "\n",
    "https://ai.google.dev/gemini-api/docs/caching?lang=python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e06fd254",
   "metadata": {},
   "outputs": [],
   "source": [
    "#idea for connecting to webscraping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4598fd53",
   "metadata": {},
   "outputs": [],
   "source": [
    "# audio analysis or segmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28714e65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# video: you can summarize a video or find a specific image within a video and extra infomration from it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "984b3ff0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "dddd5e0d",
   "metadata": {},
   "source": [
    "# `Citations & Help:`\n",
    "\n",
    "# ◔̯◔\n",
    "\n",
    "https://deepmind.google/technologies/gemini/flash/\n",
    "\n",
    "https://medium.com/rahasak/build-rag-application-using-a-llm-running-on-local-computer-with-ollama-and-langchain-e6513853fda0\n",
    "    \n",
    "https://cloud.google.com/vertex-ai/docs/tutorials/jupyter-notebooks (AI_ML Notebooks to practice)\n",
    "\n",
    "https://www.youtube.com/watch?v=mA1K1tT56jo\n",
    "\n",
    "https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_1_5_pro.ipynb\n",
    "\n",
    "https://medium.com/@punya8147_26846/understanding-prompt-templates-in-langchain-f714cd7ab380\n",
    "\n",
    "https://www.datacamp.com/tutorial/prompt-chaining-llm\n",
    "\n",
    "https://medium.com/vectrix-ai/image-extraction-with-langchain-and-gemini-a-step-by-step-guide-02c79abcd679\n",
    "\n",
    "https://medium.com/@ibrahimmukherjee/ai-for-drug-discovery-with-python-code-47e1fe3a8233\n",
    "\n",
    "https://diverger.medium.com/building-asynchronous-llm-applications-in-python-f775da7b15d1\n",
    "\n",
    "https://developers.google.com/youtube/v3/guides/uploading_a_video\n",
    "\n",
    "https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/intro_multimodal_use_cases.ipynb\n",
    "\n",
    "https://medium.com/pythoneers/building-a-video-insights-generator-using-gemini-flash-e4ee4fefd3ab\n",
    "\n",
    "https://dev.to/deepgram/live-transcription-with-python-and-django-4aj2\n",
    "\n",
    "https://www.assemblyai.com/blog/real-time-transcription-in-python/\n",
    "\n",
    "https://www.reddit.com/r/learnpython/comments/156p9ju/pytube_errors_help_fix_or_recommend_other_library/\n",
    "\n",
    "https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_python.ipynb\n",
    "\n",
    "https://medium.com/vectrix-ai/image-extraction-with-langchain-and-gemini-a-step-by-step-guide-02c79abcd679\n",
    "\n",
    "https://medium.com/@mohammed97ashraf/revolutionizing-image-data-extraction-a-comprehensive-guide-to-gemini-pro-vision-and-langchain-200bbc60b949\n",
    "\n",
    "https://rohitraj-iit.medium.com/part-10-voice-chat-with-gemini-484514580033\n",
    "\n",
    "https://blogs.jollytoday.com/how-to-use-llm-such-as-gemini-and-chatgpt-for-video-translation-e22ff076e885\n",
    "\n",
    "https://medium.com/featurepreneur/extracting-audio-from-video-using-pythons-moviepy-library-e351cd652ab8\n",
    "\n",
    "https://medium.com/featurepreneur/extracting-audio-from-video-using-pythons-moviepy-library-e351cd652ab8"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}