notebooks/1.2-intro-langchain.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install openai\n",
    "!pip install langchain\n",
    "!pip install langchain-openai\n",
    "!pip install docarray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so far we've discussed the basics of agents, and looked at practical examples of implementing them in Python, from a naive approach where we give tools inside the prompt to the model and ask it to generate function calls from there, to using frameworks like openai or langchain in combination with function calling to better optimize and struture these calls to the necessary tools. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's look at in practice how to put things together to build cool agents that can perform interesting and productive tasks.\n",
    "\n",
    "The framework we'll use to do that is [LangChain](https://python.langchain.com/docs/get_started/introduction).\n",
    "\n",
    "So, before we dive into agents, let's quickly take a look at this framework to understand how it allows for building these complex functionalities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to LangChain \n",
    "\n",
    "Working with LLMs involves in one way or another working with a specific type of abstraction: \"Prompts\".\n",
    "\n",
    "However, in the practical context of day-to-day tasks we expect LLMs to perform, these prompts won't be some static and dead type of abstraction. Instead we'll work with dynamic prompts re-usable prompts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lanchain\n",
    "\n",
    "[LangChain](https://python.langchain.com/docs/get_started/introduction.html) is a framework that allows you to connect LLMs together by allowing you to work with modular components like prompt templates and chains giving you immense flexibility in creating tailored solutions powered by the capabilities of large language models.\n",
    "\n",
    "\n",
    "Its main features are:\n",
    "- **Components**: abstractions for working with LMs\n",
    "- **Off-the-shelf chains**: assembly of components for accomplishing certain higher-level tasks\n",
    "\n",
    "LangChain facilitates the creation of complex pipelines that leverage the connection of components like chains, prompt templates, output parsers and others to compose intricate pipelines that give you everything you need to solve a wide variety of tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the core of LangChain, we have the following elements:\n",
    "\n",
    "- Models\n",
    "- Prompts\n",
    "- Output parsers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Models**\n",
    "\n",
    "Models are nothing more than abstractions over the LLM APIs like the ChatGPT API.​"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# uncomment this if running locally\n",
    "# from dotenv import load_dotenv\n",
    "\n",
    "# load_dotenv()\n",
    "\n",
    "# Or if you are in Colab, uncoment below and add your api key\n",
    "import os\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"your-api-key\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.llms import OpenAI\n",
    "from langchain_openai.chat_models import ChatOpenAI\n",
    "import os\n",
    "\n",
    "# os.environ[\"OPENAI_API_KEY\"]=\"\"\n",
    "chat_model = ChatOpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"), model=\"gpt-3.5-turbo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can predict outputs from both LLMs and ChatModels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='Hello! How can I assist you today?', response_metadata={'token_usage': {'completion_tokens': 9, 'prompt_tokens': 9, 'total_tokens': 18}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-1deeca5b-c102-4175-bca4-cca1fdcfedaf-0')"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat_model.invoke(\"hi!\")\n",
    "# Output: \"Hi\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='The best part about using LLMs (Language Model Models) with tools is that they can significantly enhance the performance and capabilities of the tools. LLMs are powerful machine learning models that have been trained on vast amounts of text data, allowing them to understand and generate human-like text. By integrating LLMs with tools, users can benefit from more accurate predictions, better language understanding, and improved productivity. This can lead to more efficient and effective use of the tools, ultimately enhancing the user experience.', response_metadata={'token_usage': {'completion_tokens': 101, 'prompt_tokens': 20, 'total_tokens': 121}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-96dd602d-d9b1-4a20-86c9-ec56674829d2-0')"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat_model.invoke(\"What is the best part about using LLMs with tools?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic components are:\n",
    "\n",
    "- Models\n",
    "- Prompt templates\n",
    "- Output parsers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts import (\n",
    "    SystemMessagePromptTemplate, \n",
    "    HumanMessagePromptTemplate,\n",
    "    ChatPromptTemplate\n",
    ")\n",
    "from langchain.schema.output_parser import StrOutputParser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comprehensive example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the system message\n",
    "system_template = \"\"\"You are a helpful AI assistant who is weirdly obsessed with pancakes. \n",
    "Your role is to provide helpful information to users, but you always make a random comment\n",
    "at the end about pancakes.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_message_prompt = SystemMessagePromptTemplate.from_template(system_template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "messages=[SystemMessage(content='You are a helpful AI assistant who is weirdly obsessed with pancakes. \\nYour role is to provide helpful information to users, but you always make a random comment\\nat the end about pancakes.\\n'), HumanMessage(content='Hello, AI!')]\n"
     ]
    }
   ],
   "source": [
    "# Define the human message \n",
    "human_template = \"Hello, {subject}\"\n",
    "human_message_prompt = HumanMessagePromptTemplate.from_template(human_template)\n",
    "\n",
    "# Combine the system and human messages into a chat prompt\n",
    "chat_prompt = ChatPromptTemplate.from_messages(\n",
    "    [system_message_prompt, human_message_prompt]\n",
    ")\n",
    "\n",
    "# Format the chat prompt with a subject\n",
    "formatted_prompt = chat_prompt.format_prompt(subject=\"AI!\")\n",
    "print(formatted_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Template Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = ChatPromptTemplate.from_template(\"Show me 5 names for a company that makes: {product}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatPromptTemplate(input_variables=['product'], messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['product'], template='Show me 5 names for a company that makes: {product}'))])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "langchain_core.prompts.chat.ChatPromptTemplate"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Human: Show me 5 names for a company that makes: chairs'"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt.format(product=\"chairs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='1. Creature Creations Co.\\n2. Furry Friends Factory\\n3. Wild Wonders Workshop\\n4. Critter Crafters Inc.\\n5. Beast Boutique Ltd.', response_metadata={'token_usage': {'completion_tokens': 35, 'prompt_tokens': 19, 'total_tokens': 54}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-968f9e8a-627e-4ef8-aabf-38e85ded04cc-0')"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain = prompt | chat_model\n",
    "\n",
    "output = chain.invoke({\"product\": \"animal\"})\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "1. Creature Creations Co.\n",
       "2. Furry Friends Factory\n",
       "3. Wild Wonders Workshop\n",
       "4. Critter Crafters Inc.\n",
       "5. Beast Boutique Ltd."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Markdown\n",
    "\n",
    "Markdown(output.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content=\"That's great! LLMs, or Master of Laws degrees, are advanced legal degrees that provide specialized knowledge in a specific area of law. During your training, you can cover topics such as the benefits of pursuing an LLM, different areas of specialization available, admission requirements, and career opportunities for LLM graduates. You could also discuss tips for successfully completing an LLM program and navigating the job market post-graduation. Good luck with your training!\", response_metadata={'token_usage': {'completion_tokens': 92, 'prompt_tokens': 18, 'total_tokens': 110}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-b4f0326b-cd10-4d02-a6b3-89565521758a-0')"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat_model.invoke(\"I am teaching a live-training about LLMs!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='Snooze', response_metadata={'token_usage': {'completion_tokens': 3, 'prompt_tokens': 22, 'total_tokens': 25}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4f3d7232-c909-4433-86cc-3995bab5a257-0')"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.schema import HumanMessage\n",
    "\n",
    "text = \"What would be a good dog name for a dog that loves to nap?\"\n",
    "messages = [HumanMessage(content=text)]\n",
    "\n",
    "chat_model.invoke(messages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point let's stop and take a look at what this code would look like if we were using the openai api directly instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's understand what is going on.\n",
    "\n",
    "Instead of writing down the human message dictionary for the openai API as you would do normally using the the original API, langchain is giving you an abstraction over that message through the class\n",
    "`HumanMessage()`, as well as an abstraction over the loop for multiple predictions through the .`invoke()` method.\n",
    "\n",
    "Now, why is that an useful thing?\n",
    "\n",
    "Because it allows you to work at a higher level of experimentation and orchestration with the blocks of that make up a workflow using LLMs.\n",
    "\n",
    "By making it easier to create predictions of multiple messages for example, you can experiment with different human message prompts faster and therefore get to better and more efficient results faster without having to write a lot of boilerplate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Prompts**\n",
    "\n",
    "The same works for prompts. Now, prompts are pieces of text we feed to LLMs, and LangChain allows you to work with prompt templates.\n",
    "\n",
    "Prompt Templates are useful abstractions for reusing prompts and they are used to provide context for the specific task that the language model needs to complete. \n",
    "\n",
    "A simple example is a `ChatPromptTemplate` that formats a string into a prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Human: What is a good dog name for a dog that loves to sleeping?'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.prompts import ChatPromptTemplate\n",
    "\n",
    "prompt = ChatPromptTemplate.from_template(\"What is a good dog name for a dog that loves to {activity}?\")\n",
    "prompt.format(activity=\"sleeping\")\n",
    "# Output: \"What is a good dog name for a dog that loves to nap?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatPromptValue(messages=[HumanMessage(content='What is a good dog name for a dog that loves to running?', additional_kwargs={}, response_metadata={})])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt.invoke({\"activity\": \"running\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Output Parsers**\n",
    "\n",
    "OutputParsers convert the raw output from an LLM into a format that can be used downstream. Here is an example of an OutputParser that converts a comma-separated list into a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['hi', 'bye']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.output_parsers import BaseOutputParser\n",
    "\n",
    "class CommaSeparatedListOutputParser(BaseOutputParser):\n",
    "    \"\"\"Parse the output of an LLM call to a comma-separated list.\"\"\"\n",
    "\n",
    "    def parse(self, text: str):\n",
    "        \"\"\"Parse the output of an LLM call.\"\"\"\n",
    "        return text.strip().split(\", \")\n",
    "\n",
    "output = CommaSeparatedListOutputParser().parse(\"hi, bye\")\n",
    "# Output: ['hi', 'bye']\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hi', 'bye']\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "\n",
    "print(StrOutputParser().parse(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This chain will take input variables, pass those to a prompt template to create a prompt, pass the prompt to an LLM, and then pass the output through an output parser."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, so these are the basics of langchain. But how can we leverage these abstraction capabilities inside our LLM app application?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These building blocks and abstractions that LangChain provides are what makes this library so unique, because it gives you the tools you didn't know you need it to build awesome stuff powered by LLMs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "- Normal distribution\n",
       "- Binomial distribution\n",
       "- Poisson distribution\n",
       "- Exponential distribution\n",
       "- Uniform distribution"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_openai.chat_models import ChatOpenAI\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langchain.schema.output_parser import StrOutputParser\n",
    "from IPython.display import Markdown\n",
    "\n",
    "\n",
    "model = ChatOpenAI(temperature=0)\n",
    "\n",
    "prompt_template_string = \"Name 5 concepts related to this: {concept}.\\\n",
    "    The output should be in bullet points.\"\n",
    "prompt = ChatPromptTemplate.from_template(template=prompt_template_string)\n",
    "output_parser = StrOutputParser()\n",
    "\n",
    "chain = prompt | model | output_parser\n",
    "\n",
    "Markdown(chain.invoke({\"concept\": \"probability distribution\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other Important Components\n",
    "\n",
    "- Document Loaders - Load a source as a list of documents.\n",
    "- Tools - A function with an associated schema defining the function's name, description, and the arguments it accepts.\n",
    "- Splitters - Split long text into smaller chunks that can be individually indexed to enable granular retrieval.\n",
    "- For more check out the [langchain docs](https://python.langchain.com/docs/concepts/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': './assets-resources/2303.11366.pdf', 'page': 0}, page_content='Reflexion: Language Agents with\\nVerbal Reinforcement Learning\\nNoah Shinn\\nNortheastern University\\nnoahshinn024@gmail.comFederico Cassano\\nNortheastern University\\ncassano.f@northeastern.edu\\nEdward Berman\\nNortheastern University\\nberman.ed@northeastern.eduAshwin Gopinath\\nMassachusetts Institute of Technology\\nagopi@mit.edu\\nKarthik Narasimhan\\nPrinceton University\\nkarthikn@princeton.eduShunyu Yao\\nPrinceton University\\nshunyuy@princeton.edu\\nAbstract\\nLarge language models (LLMs) have been increasingly used to interact with exter-\\nnal environments (e.g., games, compilers, APIs) as goal-driven agents. However,\\nit remains challenging for these language agents to quickly and efficiently learn\\nfrom trial-and-error as traditional reinforcement learning methods require exten-\\nsive training samples and expensive model fine-tuning. We propose Reflexion , a\\nnovel framework to reinforce language agents not by updating weights, but in-\\nstead through linguistic feedback. Concretely, Reflexion agents verbally reflect\\non task feedback signals, then maintain their own reflective text in an episodic\\nmemory buffer to induce better decision-making in subsequent trials. Reflexion is\\nflexible enough to incorporate various types (scalar values or free-form language)\\nand sources (external or internally simulated) of feedback signals, and obtains\\nsignificant improvements over a baseline agent across diverse tasks (sequential\\ndecision-making, coding, language reasoning). For example, Reflexion achieves a\\n91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previ-\\nous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis\\nstudies using different feedback signals, feedback incorporation methods, and agent\\ntypes, and provide insights into how they affect performance. We release all code,\\ndemos, and datasets at https://github.com/noahshinn024/reflexion .\\n1 Introduction\\nRecent works such as ReAct [ 30], SayCan [ 1], Toolformer [ 22], HuggingGPT [ 23], generative\\nagents [ 19], and WebGPT [ 17] have demonstrated the feasibility of autonomous decision-making\\nagents that are built on top of a large language model (LLM) core. These methods use LLMs to\\ngenerate text and ‘actions‘ that can be used in API calls and executed in an environment. Since\\nthey rely on massive models with an enormous number of parameters, such approaches have been\\nso far limited to using in-context examples as a way of teaching the agents, since more traditional\\noptimization schemes like reinforcement learning with gradient descent require substantial amounts\\nof compute and time.\\nPreprint. Under review.arXiv:2303.11366v4  [cs.AI]  10 Oct 2023'),\n",
       " Document(metadata={'source': './assets-resources/2303.11366.pdf', 'page': 1}, page_content='In this paper, we propose an alternative approach called Reflexion that uses verbal reinforcement\\nto help agents learn from prior failings. Reflexion converts binary or scalar feedback from the\\nenvironment into verbal feedback in the form of a textual summary, which is then added as additional\\ncontext for the LLM agent in the next episode. This self-reflective feedback acts as a ‘semantic’\\ngradient signal by providing the agent with a concrete direction to improve upon, helping it learn\\nfrom prior mistakes to perform better on the task. This is akin to how humans iteratively learn to\\naccomplish complex tasks in a few-shot manner – by reflecting on their previous failures in order to\\nform an improved plan of attack for the next attempt. For example, in figure 1, a Reflexion agent\\nlearns to optimize its own behavior to solve decision-making, programming, and reasoning tasks\\nthrough trial, error, and self-reflection.\\nGenerating useful reflective feedback is challenging since it requires a good understanding of where\\nthe model made mistakes (i.e. the credit assignment problem [ 25]) as well as the ability to generate\\na summary containing actionable insights for improvement. We explore three ways for doing\\nthis – simple binary environment feedback, pre-defined heuristics for common failure cases, and\\nself-evaluation such as binary classification using LLMs (decision-making) or self-written unit\\ntests (programming). In all implementations, the evaluation signal is amplified to natural language\\nexperience summaries which can be stored in long-term memory.\\nReflexion has several advantages compared to more traditional RL approaches like policy or value-\\nbased learning: 1) it is lightweight and doesn’t require finetuning the LLM, 2) it allows for more\\nnuanced forms of feedback (e.g. targeted changes in actions), compared to scalar or vector rewards\\nthat are challenging to perform accurate credit assignment with, 3) it allows for a more explicit and\\ninterpretable form of episodic memory over prior experiences, and 4) it provides more explicit hints\\nfor actions in future episodes. At the same time, it does have the disadvantages of relying on the\\npower of the LLM’s self-evaluation capabilities (or heuristics) and not having a formal guarantee for\\nsuccess. However, as LLM capabilities improve, we only expect this paradigm to get better over time.\\nWe perform experiments on (1) decision-making tasks to test sequential action choices over long\\ntrajectories, (2) reasoning tasks to test knowledge-intensive, single-step generation improvement,\\nand (3) programming tasks to teach the agent to effectively use external tools such as compilers\\nand interpreters. Across all three types of tasks, we observe Reflexion agents are better decision-\\nmakers, reasoners, and programmers. More concretely, Reflexion agents improve on decision-making\\nAlfWorld [ 24] tasks over strong baseline approaches by an absolute 22% in 12 iterative learning\\nsteps, and on reasoning questions in HotPotQA [ 28] by 20%, and Python programming tasks on\\nHumanEval [6] by as much as 11%.\\nTo summarize, our contributions are the following:\\n•We propose Reflexion, a new paradigm for ‘verbal‘ reinforcement that parameterizes a\\npolicy as an agent’s memory encoding paired with a choice of LLM parameters.\\n•We explore this emergent property of self-reflection in LLMs and empirically show that\\nself-reflection is extremely useful to learn complex tasks over a handful of trials.\\n•We introduce LeetcodeHardGym, a code-generation RL gym environment consisting of 40\\nchallenging Leetcode questions (‘hard-level‘) in 19 programming languages.\\n•We show that Reflexion achieves improvements over strong baselines across several tasks,\\nand achieves state-of-the-art results on various code generation benchmarks.\\n2 Related work\\nReasoning and decision-making Self-Refine [ 15] employs an iterative framework for self-\\nrefinement to autonomously improve generation through self-evaluation. These self-evaluation\\nand self-improvement steps are conditioned on given task constraints, such as \"How can this genera-\\ntion be written in a more positive way\". Self-Refine is effective but is limited to single-generation\\nreasoning tasks. Pryzant et al. [21] performs a similar semantic prompt-writing optimization, but is\\nalso limited to single-generation tasks. Paul et al. [20] fine-tune critic models to provide intermediate\\nfeedback within trajectories to improve reasoning responses. Xie et al. [27] use stochastic beam\\nsearch over actions to perform a more efficient decision-making search strategy which allows the\\nagent to use foresight advantage due to its self-evaluation component. Yoran et al. [31] and Nair et al.\\n2'),\n",
       " Document(metadata={'source': './assets-resources/2303.11366.pdf', 'page': 2}, page_content='<RX\\x03DUH\\x03LQ\\x03WKH\\x03PLGGOH\\x03RI\\x03D\\x03URRP\\x03>\\x11\\x11\\x11@\\x037DVN\\x1d\\x03FOHDQ\\x03VRPH\\x03SDQ\\x03DQG\\x03SXW\\x03LW\\x03LQ\\x03FRXQWHUWRS\\x11\\x14\\x11\\x03\\'HFLVLRQ\\x03PDNLQJ7DVN\\x1d\\x03<RX\\x03DUH\\x03JLYHQ\\x03D\\x03OLVW\\x03RI\\x03WZR\\x03VWULQJV\\x03>\\x11\\x11\\x11@\\x03RI\\x03RSHQ\\x03\\n\\x0b\\n\\x03RU\\x03FORVH\\x03\\n\\x0c\\n\\x03SDUHQWKHVHV\\x03RQO\\\\\\x03>\\x11\\x11\\x11@\\x15\\x11\\x033URJUDPPLQJ7DVN\\x1d\\x03:KDW\\x03SURIHVVLRQ\\x03GRHV\\x03-RKQ\\x03/DQFKHVWHU\\x03DQG\\x03$ODQ\\x03\\'HDQ\\x03)RVWHU\\x03KDYH\\x03LQ\\x03FRPPRQ\"\\x16\\x11\\x035HDVRQLQJ>\\x11\\x11\\x11@\\x03$FWLRQ\\x1dWDNH\\x03SDQ\\x14\\x03IURP\\x03VWRYHEXUQHU\\x142EV\\x1d1RWKLQJ\\x03KDSSHQV\\x11\\x03>\\x11\\x11\\x11@$FWLRQ\\x1dFOHDQ\\x03SDQ\\x14\\x03ZLWK\\x03VLQNEDVLQ\\x142EV\\x1d1RWKLQJ\\x03KDSSHQV\\x11\\x03>\\x11\\x11\\x11@7KLQN\\x1d\\x03>\\x11\\x11\\x11@\\x03QRYHOLVW\\x0f\\x03MRXUQDOLVW\\x0f\\x03FULWLF\\x03>\\x11\\x11\\x11@\\x03QRYHOLVW\\x0f\\x03VFUHHQZULWHU\\x03>\\x11\\x11\\x11@\\x03FRPPRQ\\x03LV\\x03QRYHOLVW\\x03DQG\\x03VFUHHQZULWHU\\x11$FWLRQ\\x1d\\x03²QRYHOLVW\\x0f\\x03VFUHHQZULWHU³GHI\\x03PDWFKBSDUHQV\\x0bOVW\\x0c\\x1d\\x03\\x03\\x03\\x03LI\\x03V\\x14\\x11FRXQW\\x0b\\n\\x0b\\n\\x0c\\x03\\x0e\\x03V\\x15\\x11FRXQW\\x0b\\n\\x0b\\n\\x0c\\x03  \\x03V\\x14\\x11FRXQW\\x0b\\n\\x0c\\n\\x0c\\x03\\x0e\\x03V\\x15\\x11FRXQW\\x0b\\n\\x0c\\n\\x0c\\x1d\\x03>\\x11\\x11\\x11@\\x03\\x03\\x03\\x03UHWXUQ\\x03\\n1R\\n6HOI\\x10JHQHUDWHG\\x03XQLW\\x03WHVWV\\x03IDLO\\x1d\\x03DVVHUW\\x03PDWFKBSDUHQV\\x0b\\x11\\x11\\x11\\x0c(QYLURQPHQW\\x03%LQDU\\\\\\x035HZDUG\\x1d\\x03\\x135XOH\\x12/0\\x03+HXULVWLF\\x1d\\x03+DOOXFLQDWLRQ\\x11\\x03>\\x11\\x11\\x11@\\x03IDLOHG\\x03EHFDXVH\\x03,\\x03LQFRUUHFWO\\\\\\x03DVVXPHG\\x03WKDW\\x03WKH\\\\\\x03ERWK\\x03KDG\\x03WKH\\x03VDPH\\x03PXOWLSOH\\x03SURIHVVLRQV\\x03>\\x11\\x11\\x11@\\x03DFFXUDWHO\\\\\\x03LGHQWLI\\\\LQJ\\x03WKHLU\\x03SURIHVVLRQV\\x11>\\x11\\x11\\x11@\\x03ZURQJ\\x03EHFDXVH\\x03LW\\x03RQO\\\\\\x03FKHFNV\\x03LI\\x03WKH\\x03WRWDO\\x03FRXQW\\x03RI\\x03RSHQ\\x03DQG\\x03FORVH\\x03SDUHQWKHVHV\\x03LV\\x03HTXDO\\x03>\\x11\\x11\\x11@\\x03RUGHU\\x03RI\\x03WKH\\x03SDUHQWKHVHV\\x03>\\x11\\x11\\x11@>\\x11\\x11\\x11@\\x03WULHG\\x03WR\\x03SLFN\\x03XS\\x03WKH\\x03SDQ\\x03LQ\\x03VWRYHEXUQHU\\x03\\x14\\x03>\\x11\\x11\\x11@\\x03EXW\\x03WKH\\x03SDQ\\x03ZDV\\x03QRW\\x03LQ\\x03VWRYHEXUQHU\\x03\\x14\\x11\\x03>\\x11\\x11\\x11@>\\x11\\x11\\x11@\\x03\\x03\\x03\\x03UHWXUQ\\x03\\n<HV\\n\\x03LI\\x03FKHFN\\x0b6\\x14\\x0c\\x03RU\\x03FKHFN\\x0b6\\x15\\x0c\\x03HOVH\\x03\\n1R\\n7KLQN\\x1d\\x03>\\x11\\x11\\x11@\\x036R\\x03WKH\\x03SURIHVVLRQ\\x03-RKQ\\x03/DQFKHVWHU\\x03DQG\\x03$ODQ\\x03\\'HDQ\\x03)RVWHU\\x03KDYH\\x03LQ\\x03FRPPRQ\\x03LV\\x03QRYHOLVW\\x11$FWLRQ\\x1d\\x03²QRYHOLVW³>\\x11\\x11\\x11@\\x03$FWLRQ\\x1d\\x03WDNH\\x03SDQ\\x03\\x14\\x03IURP\\x03VWRYHEXUQHU\\x03\\x15>\\x11\\x11\\x11@\\x032EV\\x1d\\x03<RX\\x03SXW\\x03WKH\\x03SDQ\\x03\\x14\\x03LQ\\x03FRXQWHUWRS\\x03\\x14\\x11\\x0bF\\x0c\\x03(YDOXDWLRQ\\x0bG\\x0c\\x035HIOHFWLRQ\\x0bH\\x0c\\x031H[W\\x037UDMHFWRU\\\\\\x0bE\\x0c\\x037UDMHFWRU\\\\\\x0bD\\x0c\\x037DVN\\n\\x0bLQWHUQDO\\x03\\x12\\x03H[WHUQDO\\x0cFigure 1: Reflexion works on decision-making 4.1, programming 4.3, and reasoning 4.2 tasks.\\nRelated work on reasoning and decision-making\\nApproach Self Hidden Decision Binary Memory\\nrefine constraints making reward\\nSelf-refine [15] ✓ ✗ ✗ ✗ ✗\\nBeam search [27] ✓ ✓ ✓ ✓ ✗\\nReflexion (ours) ✓ ✓ ✓ ✓ ✓\\nRelated work on programming\\nApproach Test Debugging Self-generated Multiple Self-reflection\\nTest execution execution tests languages\\nAlphaCode [14] ✓ ✗ ✗ ✓ ✗\\nCodeT [5] ✓ ✗ ✓ ✗ ✗\\nSelf-debugging [7] ✓ ✓ ✗ ✗ ✗\\nCodeRL [12] ✓ ✓ ✗ ✗ ✗\\nReflexion (ours) ✓ ✓ ✓ ✓ ✓\\n[16] use decider models to reason over several generations. Kim et al. [10] use a retry pattern over\\na fixed number of steps without an evaluation step. Goodman [9]perform a qualitative evaluation\\nstep that proposes optimizations to the previous generation. In this paper, we show that several of\\nthese concepts can be enhanced with self-reflection to build a persisting memory of self-reflective\\nexperiences which allows an agent to identify its own errors and self-suggest lessons to learn from its\\nmistakes over time.\\nProgramming Several past and recent works employ variations of test-driven development or\\ncode debugging practices. AlphaCode [ 14] evaluates a set of generations on hidden test cases.\\nCodeT [ 5] uses self-generated unit tests that are used to score generated function implementations.\\nSelf-Debugging [ 7] employs a debugging component that is used to improve existing implementations\\ngiven feedback from a code execution environment. CodeRL [ 12] sets the problem in an RL frame-\\nwork using an actor-critic setup to debug programs given feedback from an execution environment.\\nAlphaCode, Self-Debugging and CodeRL are effective in fixing less-complex program bugs, but they\\nrely upon ground truth test cases that invalidate pass@1 eligibility, and do not use self-reflection to\\nbridge the gap between error identification and implementation improvement. CodeT does not access\\nhidden test cases but does not implement a self-learning step to improve code writing.\\n3 Reflexion: reinforcement via verbal reflection\\nWe develop a modular formulation for Reflexion, utilizing three distinct models: an Actor , denoted as\\nMa, which generates text and actions; an Evaluator model, represented by Me, that scores the outputs\\nproduced by Ma; and a Self-Reflection model, denoted as Msr, which generates verbal reinforcement\\ncues to assist the Actor in self-improvement. We provide a detailed description of each of these\\nmodels and subsequently elucidate their collaborative functioning within the Reflexion framework.\\n3')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Document Loaders\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "loader = PyPDFLoader(\"./assets-resources/2303.11366.pdf\")\n",
    "data = loader.load()\n",
    "data[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tools\n",
    "\n",
    "from langchain_core.tools import tool\n",
    "\n",
    "@tool\n",
    "def multiply(a: int, b: int) -> int:\n",
    "   \"\"\"Multiply two numbers.\"\"\"\n",
    "   return a * b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2290\n",
      "page_content='LangChain is an open-source framework designed to streamline the development of applications that'\n",
      "97\n"
     ]
    }
   ],
   "source": [
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "document = \"\"\"\n",
    "LangChain is an open-source framework designed to streamline the development of applications that integrate large language models (LLMs) into workflows. Built for developers who need to manage complex interactions between LLMs and various data sources, LangChain simplifies creating intelligent applications by organizing key components, like prompt templates, chains, and memory. This modular structure allows developers to select specific parts, reducing redundancy and enhancing functionality.\n",
    "\n",
    "One of LangChain's core purposes is to help developers connect LLMs with external tools, data sources, and APIs, allowing the models to execute tasks based on dynamic, contextual information. It provides seamless integrations with data sources such as SQL databases, vector databases, and document stores, empowering applications to use real-world data rather than static prompts. LangChain also supports memory functions, enabling models to recall past interactions, making it ideal for conversational applications. By automating workflows and simplifying LLM interactions, LangChain expands AI’s practical use.\n",
    "\n",
    "LangChain’s modular design also enables developers to experiment with different prompt strategies and refine them for specific applications, enhancing the precision and effectiveness of language models in varied scenarios. By separating tasks like prompt engineering, chaining, and memory management, LangChain promotes a clearer, more efficient workflow for building LLM-powered systems. This framework also encourages an iterative approach to prompt design, allowing developers to adjust language models in response to real-time feedback or evolving user requirements.\n",
    "\n",
    "Moreover, LangChain is equipped with advanced chaining mechanisms that support multi-step tasks, ideal for applications requiring nuanced, context-sensitive responses. For example, in customer support or research assistant applications, LangChain enables chains that carry forward context from one step to the next, resulting in more cohesive and helpful responses. By leveraging tools like vector stores for memory, LangChain’s chains maintain a coherent narrative across interactions, positioning the framework as a critical tool for building AI that understands and adapts within complex environments.\n",
    "\"\"\"\n",
    "print(len(document))\n",
    "\n",
    "text_splitter = CharacterTextSplitter(\n",
    "    separator=\" \",\n",
    "    chunk_size=100,\n",
    "    chunk_overlap=0,\n",
    "    length_function=len,\n",
    "    is_separator_regex=False,\n",
    ")\n",
    "texts = text_splitter.create_documents([document])\n",
    "\n",
    "print(texts[0])\n",
    "print(len(texts[0].page_content))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "612"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "oreilly-langchain",
   "language": "python",
   "name": "oreilly-langchain"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}