sdk/python/endpoints/online/llm/langchain/1_langchain_basic_deploy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unleashing the new possibilities: Deploy your LangChain application on managed online endpoint \n",
    "\n",
    "In rapidly evolving AI landscape, the need for swift and scalable AI application deployment is essential. Recognizing this, we are excited to unveil a new capability that supports LangChain applications- the leading tool for building LLM apps to be deployed quickly and effortlessly on managed online endpoints and enjoy the perks of managed online endpoints.\n",
    "\n",
    "In this notebook, you will learn how to deploy rapidly your LangChain app to a managed online endpoint for use in real-time inference. You can follow below steps to easily deploy your LangCahin app on managed online endpoints. We’re providing you with a QuickStart image that you can reuse for your development purpose along with example below to follow along."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Langchain Integration into Azure Machine Learning (AzureML)\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "* A basic understanding of Machine Learning and Large Language Models\n",
    "* An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "* An Azure Machine Learning Workspace, Azure OpenAI Service Endpoint and Deployed model.\n",
    "* A working GPT deployment in a Azure AI services.\n",
    "\n",
    "**Motivations** - The Langchain framework allows for rapid development of applications powered by large language models. This sample creates a chat bot application backed by a large language model and deploys the application to AzureML.\n",
    "\n",
    "**Outline** - \n",
    "1. Prepare the required resources\n",
    "2. Deploy the app to an **AzureML Managed Online Endpoint**\n",
    "3. Test\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: OPENAI_API_VERSION=2023-03-15-preview\n",
      "env: AZURE_OPENAI_API_ENDPOINT=https://aoai-ibisckmf5skye.openai.azure.com\n",
      "env: AZURE_OPENAI_DEPLOYMENT=gpt-35-turbo\n"
     ]
    }
   ],
   "source": [
    "# required for Azure OpenAI API\n",
    "AZURE_OPENAI_ENDPOINT = \"<AOAI endpoint>\"\n",
    "AZURE_OPENAI_DEPLOYMENT = \"<deployment-name>\"\n",
    "OPENAI_API_VERSION = \"2023-03-15-preview\"\n",
    "\n",
    "# set to env var for the langchain code to consume\n",
    "%env OPENAI_API_VERSION=$OPENAI_API_VERSION\n",
    "%env AZURE_OPENAI_API_ENDPOINT=$AZURE_OPENAI_API_ENDPOINT\n",
    "%env AZURE_OPENAI_DEPLOYMENT=$AZURE_OPENAI_DEPLOYMENT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Install required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r requirements.txt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Set workspace details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enter details of your AML workspace\n",
    "SUBSCRIPTION_ID = \"<SUBSCRIPTION_ID>\"\n",
    "RESOURCE_GROUP = \"<RESOURCE_GROUP>\"\n",
    "AML_WORKSPACE_NAME = \"<AML_WORKSPACE_NAME>\"\n",
    "AZURE_AI_SERVICES_NAME = \"<AZURE_AI_SERVICES_NAME>\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Login to your Azure account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Authenticate clients\n",
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    "    AzureCliCredential,\n",
    ")\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential(additionally_allowed_tenants=[\"*\"])\n",
    "except Exception as ex:\n",
    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
    "    credential = InteractiveBrowserCredential(additionally_allowed_tenants=[\"*\"])\n",
    "\n",
    "# If login doesn't work above, uncomment the code below and login using device code\n",
    "# !az login --use-device-code"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Managed Online Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a endpoint\n",
    "import datetime\n",
    "\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    ")\n",
    "\n",
    "from azure.ai.ml import (\n",
    "    MLClient,\n",
    ")\n",
    "\n",
    "time = str(datetime.datetime.now().strftime(\"%m%d%H%M%f\"))\n",
    "online_endpoint_name = f\"aml-llm-lc-demo-{time}\"\n",
    "\n",
    "# get a handle to the workspace\n",
    "ml_client = MLClient(credential, SUBSCRIPTION_ID, RESOURCE_GROUP, AML_WORKSPACE_NAME)\n",
    "\n",
    "# create an online endpoint\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=\"online endpoint for Langchain server\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "\n",
    "endpoint = ml_client.begin_create_or_update(endpoint).result()\n",
    "\n",
    "print(endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# assign the Cognitive Services User role to the endpoint\n",
    "endpoint_principal_id = endpoint.identity.principal_id\n",
    "!az role assignment create --assignee-principal-type ServicePrincipal --assignee-object-id {endpoint_principal_id} --role \"Cognitive Services User\" --scope /subscriptions/{SUBSCRIPTION_ID}/resourceGroups/{RESOURCE_GROUP}/providers/Microsoft.CognitiveServices/accounts/{AZURE_AI_SERVICES_NAME}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Deploy to Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineDeployment,\n",
    "    OnlineRequestSettings,\n",
    "    Model,\n",
    ")\n",
    "\n",
    "environment_uri = (\n",
    "    \"azureml://registries/azureml/environments/minimal-app-quickstart/labels/latest\"\n",
    ")\n",
    "\n",
    "deployment_name = f\"deploy-{time}-4\"\n",
    "lc_deployment = ManagedOnlineDeployment(\n",
    "    name=deployment_name,\n",
    "    environment=environment_uri,\n",
    "    model=Model(path=\"../src/langchain\"),\n",
    "    request_settings=OnlineRequestSettings(request_timeout_ms=60000),\n",
    "    environment_variables={\n",
    "        \"OPENAI_API_VERSION\": OPENAI_API_VERSION,\n",
    "        \"AZURE_OPENAI_ENDPOINT\": AZURE_OPENAI_ENDPOINT,\n",
    "        \"AZURE_OPENAI_DEPLOYMENT\": AZURE_OPENAI_DEPLOYMENT,\n",
    "    },\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    instance_type=\"Standard_F2s_v2\",\n",
    "    instance_count=1,\n",
    ")\n",
    "ml_client.online_deployments.begin_create_or_update(lc_deployment).result()\n",
    "\n",
    "endpoint.traffic = {deployment_name: 100}\n",
    "ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Test\n",
    "Now endpoint has been deployed, let's test it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can customize your inference experience following Langserve instruction. The below code is just a simple example.\n",
    "\"\"\"\n",
    "\n",
    "uncomment the code below to test the endpoint\n",
    "\n",
    "from langserve import RemoteRunnable\n",
    "\n",
    "token = ml_client.online_endpoints.get_keys(name=endpoint.name).primary_key\n",
    "url = endpoint.scoring_uri\n",
    "url = url.replace(\"/score\", \"\")\n",
    "runnable_az = RemoteRunnable(\n",
    "    f\"{url}/openai-functions-agent\", headers={\"Authorization\": \"Bearer \" + token}\n",
    ")\n",
    "async for msg in runnable_az.astream({\"chat_history\": [], \"input\": \"Holle?\"}):\n",
    "    print(msg, end=\"\", flush=True)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Clean up resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.begin_delete(name=online_endpoint_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}