notebooks/tutorial4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Legal Analytics\n",
    "# LAW 3027 Tutorial 4: Large Language Models for Legal Q&A and Information Retrieval\n",
    "\n",
    "\n",
    "#### Intended Learning Outcomes:\n",
    "This notebook provides an introduction to the use of large language models.\n",
    "By the end of this notebook you will know how to:\n",
    "- Use the OpenAI API to access large language models and generate (legal) text.\n",
    "- Writing effective prompts to acomplish tasks, such as legal Q&A.\n",
    "- Iteratively optimizing prompts to improve the quality of generated (legal) text.\n",
    "- Use advanced techniques such as Retrieval Augmented Generation (RAG) to improve the factuality of the generated (legal) text.\n",
    "\n",
    "#### Libraries to be used:\n",
    "You can activate your previously used environment. For this notebook, you will additionally need the following packages:\n",
    "- requests\n",
    "- openai\n",
    "- chromadb\n",
    "- scikit-learn\n",
    "- chevron\n",
    "- pandas\n",
    "- datasets\n",
    "\n",
    "#### Reading Material:\n",
    "- [OpenAI API](https://platform.openai.com/docs/api-reference/introduction)\n",
    "- [OpenAI prompt engineering guidelines](https://platform.openai.com/docs/guides/prompt-engineering)\n",
    "- [Lessons after a half-billion GPT tokens](https://kenkantzer.com/lessons-after-a-half-billion-gpt-tokens/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "Run this command to install the required packages for this lesson:\n",
    "\n",
    "```bash\n",
    "pip install requests openai scipy scikit-learn pandas datasets chevron\n",
    "```\n",
    "\n",
    "If you are on colab, you can do this by running the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install requests openai scipy scikit-learn pandas datasets chevron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Large Language Models (LLMs) have taken the world by storm, and are today some of the most exciting and powerful tools to use in artificial intelligence. Today, we will explore how they can be used in the legal field. We will use the OpenAI API connected to our own custom server. You have received a code, which allows you to access two models, one for text completion and one for embedding:\n",
    "- Text Generation: [Mixtral-8x7B-Instruct-v0.1](https://mistral.ai/news/mixtral-of-experts/)\n",
    "- Text Embedding: [text-embedding-3-small](https://platform.openai.com/docs/guides/embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your key and initialize the client in this cell\n",
    "import openai\n",
    "\n",
    "MODEL_MIXTRAL = \"Mixtral-8x7B-Instruct-v0.1\"\n",
    "MODEL_EMBEDDING = \"text-embedding-3-small\"\n",
    "\n",
    "key = \"sk-...\" # your key here\n",
    "base_url = \"https://llm.wstrmnn.com\"\n",
    "\n",
    "client = openai.OpenAI(\n",
    "    api_key=key,\n",
    "    base_url=base_url,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We have assigned you 0.1 USD to spend on the API, which should easily cover the exercises in this notebook. Run this function to check how much you have spent so far.\n",
    "import requests\n",
    "\n",
    "def check_spend():\n",
    "    url = f\"{base_url}/key/info\"\n",
    "    headers = {\n",
    "        \"Authorization\": f\"Bearer {key}\",\n",
    "    }\n",
    "    response = requests.get(url, headers=headers)\n",
    "    decoded = response.json()\n",
    "    spent = round(decoded[\"info\"][\"spend\"],5)\n",
    "    max_budget = decoded[\"info\"][\"max_budget\"]\n",
    "    percent = round(spent / max_budget * 100,1)\n",
    "    print(f\"Spent: {spent} USD. Max Budget: {max_budget} USD. Percent used: {percent}%\")\n",
    "check_spend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1 - Exploring Large Language Models\n",
    "\n",
    "First, let us learn how to us the OpenAI API. We will use the Chat Completions API - read the documentation [here](https://platform.openai.com/docs/guides/text-generation/chat-completions-api). The API is called with messages. Each message can have the following roles:\n",
    "- `system`: Instructions to the model, defining how it should act in general.\n",
    "- `user`: A message from the user, like if you sent a message to ChatGPT. Can be used to ask questions or provide data.\n",
    "- `assistant`: A previous response from the model. Can be used to continue a conversation. Not necessary for us.\n",
    "\n",
    "\n",
    "The API call also takes a number of parameters, including `max_tokens` and `temperature`. `max_tokens` defines the maximum number of tokens the model should generate, and `temperature` defines how creative the model should be. A temperature of 0 will always return the most likely token, while a temperature of 1 will return a more random token."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1.1 - Write a poem\n",
    "\n",
    "Look at the code below, which uses the OpenAI client to call the API and generate a poem about the law. Can you get it to generate a poem about another topic? How does changing the max_tokens and temperature affect the poem?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_tokens = 500\n",
    "temperature = 0.7\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "  model=MODEL_MIXTRAL,\n",
    "  messages=[\n",
    "    {\"role\": \"system\", \"content\": \"Write a beautiful poem about the subject I will send to you.\"},\n",
    "    {\"role\": \"user\", \"content\": \"The Law\"},\n",
    "  ],\n",
    "  max_tokens=max_tokens,\n",
    "  temperature=temperature,\n",
    ")\n",
    "\n",
    "print (response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1.2 - Ask a question\n",
    "\n",
    "Write code to use the OpenAI client to call the API and ask what the notice required is for quitting a job in the netherlands. Use a system prompt telling the bot to be a lawyer, and a user prompt asking the question: What is the notice period to quit my job in the Netherlands?\n",
    "\n",
    "Then, verify whether the answer is correct. For your consideration: [Think of language models like ChatGPT as a “calculator for words”](https://simonwillison.net/2023/Apr/2/calculator-for-words/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.chat.completions.create(\n",
    "  model=MODEL_MIXTRAL,\n",
    "  messages=[] # <- Add your messages here   \n",
    ")\n",
    "\n",
    "print (response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1.3 - Drafting via function\n",
    "\n",
    "To integrate the OpenAI API into our workflow, it can be useful to write a function that does the API call for us. Write a function that acts as a lawyer drafting letters for their client. The function takes as argument a short sentence the user wants to write, call the API with a system prompt and user prompt, and return the response. \n",
    "\n",
    "Watch this video to learn more about how to use functions in python: https://youtu.be/NSbOtYzIQI0 . You can also follow the tutorial here: https://www.geeksforgeeks.org/python-functions/ \n",
    "\n",
    "- Test the function with the sentences below.\n",
    "- Change the prompt to see how it impacts the generated letter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_letter(description):\n",
    "    #Your code here\n",
    "    return letter_text\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(write_letter(\"I want to write a letter to my boss, informing them that I want to quit on the 17th of April.\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(write_letter(\"I want to write a notice to my landlord, informing them that there is an issue with the heating and I want it fixed.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2 - Legal Q&A\n",
    "\n",
    "Now that we have learned how to use the OpenAI API, let us use it to answer legal questions. We will use the Contract NLI dataset, which contains annotated contracts. We will analyze a series of Non-Disclosure Agreements to investigate whether the following is true: `Receiving Party shall not disclose the fact that Agreement was agreed or negotiated.`. More information about this dataset and task is available here: https://hazyresearch.stanford.edu/legalbench/tasks/contract_nli_confidentiality_of_agreement.html\n",
    "\n",
    "Run the cell below to load the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 0)\n",
    "\n",
    "import datasets\n",
    "dataset = datasets.load_dataset(\"nguha/legalbench\", \"contract_nli_confidentiality_of_agreement\")\n",
    "train_df = dataset[\"train\"].to_pandas()\n",
    "test_df = dataset[\"test\"].to_pandas()\n",
    "\n",
    "#For train and test, only keep the columns \"answer\" and \"text\"\n",
    "train_df = train_df[[\"text\", \"answer\"]]\n",
    "test_df = test_df[[\"text\", \"answer\"]]\n",
    "\n",
    "\n",
    "test_df = test_df.sample(frac=1, random_state=42).reset_index(drop=True)\n",
    "test_df = test_df.head(30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Print the number of rows in the train and test set\n",
    "print(f\"Train set has {train_df.shape[0]} rows.\")\n",
    "print(f\"Test set has {test_df.shape[0]} rows.\")\n",
    "\n",
    "#Is this surprising? How does it compare to the ratio (size of train and test) in more traditional machine learning text classification which you did in the last tutorial?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In the dataset, what is the predictor variable and what is the target variable?\n",
    "train_df.head(8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2.1 - Zero-shot classification\n",
    "\n",
    "Next, let us try whether the model can predict whether a clause prohibits the disclosure of the agreement itself, or the fact that the parties are negotiating an agreement. Write a function that takes a clause as input, and returns `Yes` or `No`.\n",
    "\n",
    "Zero-shot means that we will not give _any_ prior examples to the model - instead, we will explain the task to the model and let it figure out the answer by itself. How does this compare to traditional machine learning?\n",
    "\n",
    "**_Note_: Since we evaluate this model on multiple clauses, it can use a lot of your tokens if you are not careful. Make sure to set the max_tokens parameter to a low value.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your system prompt below:\n",
    "system_prompt = ''' Write the system prompt here.  '''\n",
    "\n",
    "def predict_confidentiality(text):\n",
    "    # Your code here\n",
    "    return prediction #Note that the function should return a 'Yes' or 'No'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the below cells to run the prediction on the clauses in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the function on the test set\n",
    "# Store the predictions in a new column called 'predicted_answer'\n",
    "test_df[\"predicted_answer\"] = test_df[\"text\"].apply(predict_confidentiality)\n",
    "test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "Run the below cell to compare the `predicted_answer` with the actual `answer`. Evaluate the performance of the model. Does the model perform well?\n",
    "\n",
    "You will also see a list of clauses where the model failed. Can you identify why the model failed on these clauses? Can you improve the prompt to make the model perform better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Use the scikit learn classification report to evaluate the model\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(test_df[\"answer\"], test_df[\"predicted_answer\"]))\n",
    "\n",
    "#Print the rows where the answer is different from the predicted answer\n",
    "test_df[test_df[\"answer\"] != test_df[\"predicted_answer\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3 - Retrieval Augmented Generation\n",
    "We have now seen how good these models are at answering questions about legal texts. However, models are also prone to hallucinate - if they are asked specific questions about legislation or cases, they may invent the answer. To mitigate this, we can use a technique called Retrieval Augmented Generation (RAG). This technique uses a retriever to find relevant pieces of texts (such as articles from legislation) and then generates text based on this information. First, let us see what happens when we ask the model to generate text about a specific topic.\n",
    "\n",
    "You can see the slides about LLMs and RAG [here](https://canvas.maastrichtuniversity.nl/courses/14858/files/3886649?wrap=1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3.1 - Unaugmented generation\n",
    "\n",
    "Run the code below. What happens when you ask the model to generate text about a specific topic? Can you identify any factual errors? DOes the model e.g. mention [article V](https://www.unoosa.org/oosa/en/ourwork/spacelaw/treaties/outerspacetreaty.html#:~:text=Article%20V-,States%20Parties%20to%20the,of%20their%20space%20vehicle.,-In%20carrying%20on) of the outer space treaty?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_answer(question):\n",
    "    response = client.chat.completions.create(\n",
    "        model=MODEL_MIXTRAL,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": \"You are a helpful legal chatbot. Answer the question of the user, referencing relevant legal sources.\"},\n",
    "            {\"role\": \"user\", \"content\": f\"{question}\"},\n",
    "        ],\n",
    "        temperature=0.7,\n",
    "        max_tokens=500,\n",
    "        )\n",
    "    value = response.choices[0].message.content\n",
    "    return value\n",
    "\n",
    "print(generate_answer(\"What does the Outer Space Treaty say about accidents in space?\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3.2 - Retrieval augmented generation\n",
    "Now, let us implement a pipeline to use RAG to generate text. First, let us load a library of articles from the Outer Space Treaty (OST.json). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "url = 'https://raw.githubusercontent.com/maastrichtlawtech/law3027-advanced-legal-analytics/main/data/OST.json'\n",
    "\n",
    "# Read JSON data from the URL directly into a DataFrame\n",
    "df = pd.read_json(url)\n",
    "\n",
    "article_name_list = df[\"name\"].tolist()\n",
    "article_text_list = df[\"text\"].tolist()\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subtask 3.2.1 - Embed the articles in a vector space\n",
    "First, let us create an embedding of the articles of the Outer Space Treaty. For this, we will use the text-embedding-3-small model. You can read more about embeddings and how to create them [here](https://platform.openai.com/docs/guides/embeddings)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A function that takes a list of text strings and returns a list of embeddings.\n",
    "def get_embeddings(text_list):\n",
    "    response = client.embeddings.create(\n",
    "        model=MODEL_EMBEDDING,\n",
    "        input=text_list\n",
    "    )\n",
    "    embeddings = [item.embedding for item in response.data]\n",
    "    return embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the function to get the embeddings\n",
    "embeddings = get_embeddings(article_text_list)\n",
    "print (embeddings[0][:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subtask 3.2.2 - Create an index to find similar sentences\n",
    "Next, let us create an index to find similar sentences. We will use the [BallTree](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.BallTree.html) algorithm to create an index of the embeddings, which can then be used to find the most similar sentences to a given query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a BallTree from the embeddings\n",
    "from sklearn.neighbors import BallTree\n",
    "\n",
    "tree = BallTree(embeddings)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#This should return the indices of the closest 3 articles to the first article.\n",
    "dist, ind = tree.query([embeddings[0]], k=3)\n",
    "print(ind)\n",
    "\n",
    "# Also note that the BallTree returns the indices of the articles in the original list of articles. So the article I has an index of 0, Article II has an index of 1 etc.\n",
    "# Note that the first article is the closest (most similar) to itself, so the first index is 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subtask 3.2.2 - Retrieve relevant articles\n",
    "Create a function that takes a query string, embeds the query, finds the closest N articles, and returns the articles and their content.\n",
    "Return a list of dictionaries, where each dictionary has the keys \"name\" and \"text\".\n",
    "\n",
    "You might need to use a for loop for this. Learn more about the for loop here: https://www.youtube.com/watch?v=OnDr4J2UXSA&ab_channel=CSDojo . You may also need to use the `append()` function as you need to return the articles and their content in a list of dictionaries. So you can watch this video to understand the `append()` function: https://youtu.be/5IEhquZghp0\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retrieve_articles(query, n=3):\n",
    "    query_embedding = get_embeddings([query])[0]\n",
    "    dist, ind = tree.query([query_embedding], k=n)\n",
    "    articles = []\n",
    "\n",
    "    #your code here\n",
    "    return articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run this code to test your function. Does it return relevant articles for the questions?\n",
    "\n",
    "questions = [\n",
    "    \"According to the Outer Space Treaty, what happens in case of an accident in space?\",\n",
    "    \"Can I capture planets?\",\n",
    "    \"Can I place weapons on the moon?\"\n",
    "]\n",
    "\n",
    "for question in questions:\n",
    "    articles = retrieve_articles(question, n=1)\n",
    "    print (\"==============================\")\n",
    "    print(f\"Question: {question}\")\n",
    "    print (\"==============================\")\n",
    "    for article in articles:\n",
    "        print(f\"Article: {article['name']}\")\n",
    "        print(f\"Content: {article['text']}\")\n",
    "    print(\"\\n\")\n",
    "    print (\"==============================\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us evaluate the results you received above. Fill in the table below, indicating whether the articles retrieved are relevant to the question asked.\n",
    "\n",
    "| Query     | Article Retrieved | Relevant? |\n",
    "|-----------|-------------------|-----------|\n",
    "| According to the Outer Space Treaty, what happens in case of an accident in space? |                   |           |\n",
    "| Can I capture planets?   |                   |           |\n",
    "| Can I place weapons on the moon?   |                   |           |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us use precision to evaluate the results. Precision is defined as the number of relevant articles retrieved divided by the total number of articles retrieved. Fill in the table with the precision of the articles retrieved.\n",
    "\n",
    "We can evaluate the precision at different numbers of retrieved articles. What is the precision if we retrieve 1 article?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Calculate the precision@1 and precision@3\n",
    "total_queries = 3\n",
    "correct_queries_at_first_spot = #Fill in. How many times was the retrieved article correct?\n",
    "\n",
    "precision_at_1 = correct_queries_at_first_spot / total_queries\n",
    "\n",
    "print(f\"Precision@1: {precision_at_1}\")\n",
    "\n",
    "# Are there any other metrics you could calculate to evaluate the performance of the system?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subtask 3.2.3 - Assemble the prompt\n",
    "Write a function that takes as input a question and a list of articles, and writes prompt messages that instructs the model to answer the questions based on the articles.\n",
    "\n",
    "*Note*: In the example below, we use the `chevron` library to format the prompt. You can install it by running `pip install chevron`. Chevron uses \"mustache\" to format strings. If you are interested, read more [here](https://mustache.github.io/mustache.5.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import chevron\n",
    "\n",
    "system_prompt = \"\"\" \"\"\" #<- Write your system prompt here. Tell the model to answer the question using the relevant articles that will be provided.\n",
    "\n",
    "user_prompt_template = \"\"\"Articles:\n",
    "{{#articles}}\n",
    "{{name}}: {{text}}\n",
    "{{/articles}}\n",
    "\n",
    "Question: {{question}}\n",
    "\"\"\"\n",
    "\n",
    "def build_user_prompt(question, articles):\n",
    "    user_prompt = chevron.render(user_prompt_template, {\n",
    "        \"articles\": articles,\n",
    "        \"question\": question\n",
    "    })\n",
    "    return user_prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run this code to test your function. Does it generate a relevant prompt for the question and articles?\n",
    "question = \"According to the Outer Space Treaty, what happens in case of an accident in space?\"\n",
    "\n",
    "articles = retrieve_articles(question)\n",
    "user_prompt = build_user_prompt(question, articles)\n",
    "\n",
    "print (\"This is what the LLM will see:\")\n",
    "print (\"============SYSTEM===========\")\n",
    "print (system_prompt)\n",
    "print (\"============USER=============\")\n",
    "print(user_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subtask 3.2.4 - Putting it all together\n",
    "Write a function that takes a query as input, retrieves the relevant articles, puts them into the prompt, sends the prompt to the model, and returns the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_answer(question):\n",
    "    #Your code here\n",
    "    return answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run this code to test your function. Does it generate a relevant answer for the question?\n",
    "questions = [\n",
    "    \"According to the Outer Space Treaty, what happens in case of an accident in space?\",\n",
    "    \"Can I capture planets?\",\n",
    "    \"Can I place weapons on the moon?\"\n",
    "]\n",
    "\n",
    "for question in questions:\n",
    "    print (\"==============================\")\n",
    "    print(f\"Question: {question}\")\n",
    "    print (\"==============================\")\n",
    "    print(generate_answer(question))\n",
    "    print (\"\\n\\n\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}