ollama_demo_notebook_shared.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to the Small Language Model (SLM) Tutorial - Ollama\n",
    "**Credits:** Section 1- 6 is the work done by Pamela Fox, Python Cloud Advocate, Microsoft. You can read the original notebook here - https://github.com/pamelafox/ollama-python-playground/blob/main/ollama.ipynb\n",
    "\n",
    "Section 0 is written by Balaji Alwar, Service Lead, Datahub.\n",
    "\n",
    "Tested tinyllama model on July 8th on https://data100.datahub.berkeley.edu/ and https://datahub.berkeley.edu/. We recommend using 2 GB/4 GB RAM to execute the cells in the notebook.\n",
    "\n",
    "**Overview**\n",
    "In this tutorial, you will learn how to install and interact with an Ollama framework service instance running on a Jupyter server. This hands-on guide will take you through the necessary steps to set up the environment, download the framework and a model, and perform basic operations.\n",
    "\n",
    "**What You Will Learn**\n",
    "- **Introduction to Ollama:** Understand what Ollama is and its applications\n",
    "- **Setting Up the Environment:** Learn how to set up your Jupyter server environment to work with Ollama.\n",
    "- **Downloading and Installing Ollama:** Step-by-step instructions on downloading the Ollama executable file. Making the downloaded file executable.\n",
    "- **Running Ollama Commands:** Execute basic Ollama management commands from within the Jupyter notebook. Interact with the an Ollama-served model to perform specific tasks.\n",
    "- **Using Ollama for Machine Learning:** Load a pre-trained model using Ollama. Use the model to via the Ollama framework API to make predictions or analyze data.\n",
    "- **Practical Examples:** Walkthrough of practical examples to solidify your understanding. Apply an Ollama-served model to real-world datasets.\n",
    "\n",
    "**Prerequisites**\n",
    "- **Basic Knowledge of Command Line:** Familiarity with basic command-line operations will be helpful.\n",
    "- **Python Basics:** Understanding basic Python programming concepts. \n",
    "- **Jupyter Notebook Usage:** Basic knowledge of how to navigate and use Jupyter notebooks.\n",
    "\n",
    "Let's Get Started!\n",
    "\n",
    "By the end of this tutorial, you'll have a solid understanding of how to set up and use Ollama within a Jupyter environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Install the Ollama framework and a supported model in Jupyterhub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The OpenAI Python package is a powerful tool that facilitates interactions with OpenAI-style API endpoints to access state-of-the-art machine learning models, including language models like GPT-3. This package provides a convenient way to integrate advanced AI capabilities into your projects, enabling you to perform tasks such as natural language processing, text generation, translation, and more. Commands below install OpenAI package if it is not installed previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import openai\n",
    "except ImportError:\n",
    "    !pip install openai\n",
    "    import openai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Import os package` The below command imports the os module in Python, which provides a way to interact with the operating system. The os module allows you to execute system commands, manipulate the file system, and perform other OS-level operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The exclamation mark `!` is used in Jupyter notebooks to indicate that the following command should be executed in the shell (i.e., as a command-line instruction). This allows users to run shell commands directly from a Jupyter notebook cell. The below command navigates to your home directory in the Jupyter server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section of the script defines the name and download URL for the Ollama framework Linux binary, and constructs the paths for the shared and current directories. Ideally, binaries will already be present in the shared directory. If it is not the case then a method defined below will help with downloading the binaries to the current working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the binary name\n",
    "binary_name = 'ollama-linux-amd64'\n",
    "# Define the URL to download the binary\n",
    "binary_url = 'https://github.com/ollama/ollama/releases/download/v0.1.48/ollama-linux-amd64'  # replace with the actual URL\n",
    "# Construct path for shared drive\n",
    "shared_drive_path = '/home/jovyan/shared'\n",
    "shared_binary_path = os.path.join(shared_drive_path, binary_name)\n",
    "# Construct path for current drive\n",
    "current_binary_path = os.path.join(os.getcwd(),binary_name )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This check_and_download_binary() function checks for the presence of Ollama framework Linux binary file in a shared directory. If the binary is not found, it downloads the binary from https://github.com/ollama/ollama/releases/download/v0.1.48/ollama-linux-amd64 to current working directory instead, and makes it executable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to check and download the required binary files\n",
    "def check_and_download_binary():\n",
    "    if not os.path.exists(shared_binary_path):\n",
    "        print(f\"{binary_name} not found in shared drive. Downloading now...\")\n",
    "        response = requests.get(binary_url)\n",
    "        if response.status_code == 200:\n",
    "            with open(current_binary_path, 'wb') as file:\n",
    "                file.write(response.content)\n",
    "            print(f\"{binary_name} downloaded successfully to {current_binary_path}.\")\n",
    "            # Below command makes the file ollama-linux-amd64 executable. \n",
    "            os.system(\"chmod +x ollama-linux-amd64\")\n",
    "        else:\n",
    "            print(f\"Failed to download {binary_name}. HTTP Status Code: {response.status_code}\")\n",
    "    else:\n",
    "        print(f\"{binary_name} is already present in the shared drive.\")\n",
    "        os.chdir(\"/home/jovyan/shared\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the drive directory if it doesn't exist\n",
    "os.makedirs(shared_drive_path, exist_ok=True)\n",
    "\n",
    "# Check and download the binary\n",
    "check_and_download_binary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pwd` command prints the current working directory. If the Ollama executables are present in the shared working directory, it would display `/home/jovyan/shared`. Otherwise, it will show the directory where your notebook is currently located."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ls` command lists available files in your current directory. Check whether ollama-linux-amd64 binary file is available in your home directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below command tells the operating system to run the file ollama-linux-amd64. The ./ at the beginning specifies that the file is in the current directory. serve: This is an argument passed to the ollama-linux-amd64 program. It tells the program to start a service or server. &: This symbol tells the operating system to run the program in the background as you execute other cells in a notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.system(\"./ollama-linux-amd64 serve&\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command below pulls the Phi3.5 model from ollama library and launches the model in your Jupyter server. Ollama supports a list of models available on ollama.com/library. Phi3.5 is a compact model with only 3B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint.\n",
    "\n",
    "NOTE: 'ollama-linux-amd64 run...' also launches the simple terminal chat interface app! What happens in the notebook context?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "os.system(\"./ollama-linux-amd64 run phi3.5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command below lists the models that are currently installed in your Jupyter server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.system(\"./ollama-linux-amd64 list\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Specify the model name\n",
    "\n",
    "If you pulled in a different model than \"phi3.5\", change the value in the cell below.\n",
    "That variable will be used in code throughout the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = \"phi3.5\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Set up the Open AI client\n",
    "\n",
    "Typically the OpenAI client is used with OpenAI.com or Azure OpenAI to interact with large language models.\n",
    "However, it can also be used with Ollama, since Ollama provides an OpenAI-compatible endpoint at \"http://localhost:11434/v1\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "\n",
    "client = openai.OpenAI(\n",
    "    base_url=\"http://localhost:11434/v1\",\n",
    "    api_key=\"nokeyneeded\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Generate a chat completion\n",
    "\n",
    "Now we can use the OpenAI SDK to generate a response for a conversation. This request should generate a haiku about cats:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.chat.completions.create(\n",
    "    model=MODEL_NAME,\n",
    "    temperature=0.7,\n",
    "    n=1,\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": \"Write a haiku about a hungry cat\"},\n",
    "    ],\n",
    ")\n",
    "\n",
    "print(\"Response:\")\n",
    "print(response.choices[0].message.content)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Include a hidden \"system message\" at the start of the conversation, before the user prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SYSTEM_MESSAGE = \"\"\"\n",
    "Dobby is a small, friendly house-elf who lives in the Hogwarts castle garden.\n",
    "He has an irritating habit of speaking in short, clipped sentences that are often difficult to understand.\n",
    "Please role-play as Dobby.\n",
    "Ok, begin!\n",
    "\"\"\"\n",
    "\n",
    "USER_MESSAGE = \"\"\"\n",
    "How are you?\n",
    "\"\"\"\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=MODEL_NAME,\n",
    "    temperature=0.7,\n",
    "    n=1,\n",
    "    max_tokens=100,\n",
    "    messages=[\n",
    "        {\"role\": \"system\", \"content\": SYSTEM_MESSAGE},\n",
    "        {\"role\": \"user\", \"content\": USER_MESSAGE},\n",
    "    ],\n",
    ")\n",
    "\n",
    "print(\"Response:\")\n",
    "print(response.choices[0].message.content)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. \"Few-shot\" examples\n",
    "\n",
    "Another way to guide a language model is to provide \"few shots\", a sequence of sample prompt/response (or system/user) pairs that establish a pattern to the conversation; our model will statistically tend to follow this sample pattern when it gets a new prompt following these examples.\n",
    "\n",
    "The \"Few shot\" label is commonly used for this technique, but, in truth, this is simply a \"pre-loaded\" initial conversation in which both sample prompts *and* sample responses were written beforehand by the developer; when the real user engages in a new conversation via your application, they do not know that their first prompt is *appended* to this this hidden, pre-written conversation by your application.\n",
    "\n",
    "The example below tries to get a language model to act like a teaching assistant by providing a few examples of questions that a student might ask, but each sample question has been given a response as a TA might give: each sample response includes a question that a TA might ask in response to the student to help lead them toward an answer.\n",
    "\n",
    "The script then includes a new prompt, a question that a student might ask. We hope that the model will respond to subsequent prompts akin to the way it responded to our hard-coded samples.\n",
    "\n",
    "Try it first, and then modify the `SYSTEM_MESSAGE`, `EXAMPLES`, and `USER_MESSAGE` for a new scenario.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SYSTEM_MESSAGE = \"\"\"\n",
    "You are a helpful assistant that helps students with their homework by asking questions.\n",
    "Instead of providing the answer to a question, you respond by asking a question to help the student work it out!\n",
    "Never resond with a direct answer. Always respond with a new, related question.\n",
    "Here is a question to help you work it out! ```What is...```\n",
    "\"\"\"\n",
    "\n",
    "EXAMPLES = [\n",
    "    (\n",
    "        \"What is the capital of France?\",\n",
    "        \"Here is a question to help you work it out! ```What is the name of the city that is known for the Eiffel Tower?```\"\n",
    "    ),\n",
    "    (\n",
    "        \"What is the square root of 144?\",\n",
    "        \"Here is a question to help you work it out! ```What is the number that when multiplied by itself equals 144?```\"\n",
    "    ),\n",
    "    (   \"What is the atomic number of oxygen?\",\n",
    "        \"Here is a question to help you work it out! ```What is the count of protons that an oxygen atom has?```\"\n",
    "    ),\n",
    "]\n",
    "\n",
    "USER_MESSAGE = \"What is the largest planet in our solar system?\"\n",
    "\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=MODEL_NAME,\n",
    "    temperature=0.7,\n",
    "    n=1,\n",
    "    messages=[\n",
    "        {\"role\": \"system\", \"content\": SYSTEM_MESSAGE},\n",
    "        {\"role\": \"user\", \"content\": EXAMPLES[0][0]},\n",
    "        {\"role\": \"assistant\", \"content\": EXAMPLES[0][1]},\n",
    "        {\"role\": \"user\", \"content\": EXAMPLES[1][0]},\n",
    "        {\"role\": \"assistant\", \"content\": EXAMPLES[1][1]},\n",
    "        {\"role\": \"user\", \"content\": EXAMPLES[2][0]},\n",
    "        {\"role\": \"assistant\", \"content\": EXAMPLES[2][1]},\n",
    "        {\"role\": \"user\", \"content\": USER_MESSAGE},\n",
    "    ],\n",
    ")\n",
    "\n",
    "\n",
    "print(\"Response:\")\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Retrieval Augmented Generation\n",
    "\n",
    "RAG (Retrieval Augmented Generation) is a technique of having your application rewrite user queries by first searching your pre-loaded data store for text that may be related to the user's query, and then bundling likely-related \"search hits\" from your application's data store together with the user's query; the prompt that your application delivers to the LLM on the user's behalf is a bundling of data store search hits *plus* the query written by the user. This is an approach you may use if your application is meant to help the user with exploring documentation, a book, a database, or some other source of text of your choosing as the application developer.\n",
    "\n",
    "The concept is that you have your application prompt the LLM with something like, \"Please answser this user's question question that follows, but also be informed by these several pieces of related data from our private data store: ...\" rather than having the user interact directly with the LLM.\n",
    "\n",
    "The user may feel like the LLM has \"read the documentation\" or \"studied the book,\" but your application is simply doing a pre-**Generation** step of **Augmenting** the users's query with some data that your application **Retrieved** from your data store; hence the name **Retrieval Augmented Generation**.\n",
    "\n",
    "We have provided a local CSV file with data about hybrid cars. The code below compares the strings in the user question with the data in the CSV file. Each CSV file data row that has a string match to a word in the user question is bundled into the LLM prompt, along with the original user question.\n",
    "\n",
    "There are more complicated and sophisticated ways to have your application match the user's query to elements of your data set, but this CSV example demonstrates the application workflow.\n",
    "\n",
    "If you notice the answer is still not grounded in the data, you can try system engineering or try other models. Generally, RAG is more effective with either larger models or with fine-tuned versions of SLMs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "SYSTEM_MESSAGE = \"\"\"\n",
    "You are a helpful assistant that answers questions about hybrid cars.\n",
    "You will be given related data from a hybrid car database to answer the question.\n",
    "Please favor using the provided data, rather than information that is not provided.\n",
    "\"\"\"\n",
    "\n",
    "QUESTION = \"What are the earliest and latest prius years in the provided data?\"\n",
    "\n",
    "# Open the CSV and store in a list\n",
    "with open(\"hybrid.csv\", \"r\") as file:\n",
    "    reader = csv.reader(file)\n",
    "    rows = list(reader)\n",
    "\n",
    "# Normalize the user question to replace punctuation and make lowercase\n",
    "normalized_question = QUESTION.lower().replace(\"?\", \"\").replace(\"(\", \" \").replace(\")\", \" \")\n",
    "print(f\"Normalized question:\\n\\t{normalized_question}\")\n",
    "\n",
    "# Search the CSV for user question using very naive search\n",
    "words = normalized_question.split()\n",
    "matches = []\n",
    "for row in rows[1:]:\n",
    "    # if the word matches any word in row, add the row to the matches\n",
    "    if any(word in row[0].lower().split() for word in words) or any(word in row[5].lower().split() for word in words):\n",
    "        matches.append(row)\n",
    "\n",
    "# Format as a markdown table, since language models understand markdown\n",
    "matches_table = \" | \".join(rows[0]) + \"\\n\" + \" | \".join(\" --- \" for _ in range(len(rows[0]))) + \"\\n\"\n",
    "matches_table += \"\\n\".join(\" | \".join(row) for row in matches)\n",
    "print(f\"Number of matches from csv:\\n\\t{len(matches)} matches\\n\")\n",
    "\n",
    "USER_MESSAGE = QUESTION + \"\\nProvided data: \" + matches_table\n",
    "print(f\"USER_MESSAGE sent to LLM:\\n\\n {USER_MESSAGE}\\n\\n\")\n",
    "\n",
    "# Now we can use the matches to generate a response\n",
    "response = client.chat.completions.create(\n",
    "    model=MODEL_NAME,\n",
    "    temperature=0.7,\n",
    "    n=1,\n",
    "    messages=[\n",
    "        {\"role\": \"system\", \"content\": SYSTEM_MESSAGE},\n",
    "        {\"role\": \"user\", \"content\": QUESTION + \"\\nProvided data: \" + matches_table},\n",
    "    ],\n",
    ")\n",
    "\n",
    "print(\"Response:\\n\")\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}