tlm_demo_new.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "050ab2d3-57a6-41b2-a143-8f2043d35d1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install dependencies\n",
    "!pip install pandas gspread oauth2client\n",
    "!pip install --upgrade pip\n",
    "!pip uninstall -y cleanlab-studio\n",
    "!pip install git+https://github.com/cleanlab/cleanlab-studio.git@tlm\n",
    "# !pip install --upgrade cleanlab-studio\n",
    "import time\n",
    "from IPython import display\n",
    "display.clear_output()\n",
    "\n",
    "from cleanlab_studio import Studio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3be988b1-51df-4b7f-ac17-34c31db282cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('api.txt') as file: \n",
    "    key = file.read()\n",
    "studio = Studio(key)  # Cleanlab Studio API key from https://app.cleanlab.ai/account?tab=General\n",
    "# tlm = studio.TLM(quality_preset=\"high\", options={\"model\": \"gpt-4\"}) # on high quality, with gpt-4\n",
    "tlm = studio.TLM(quality_preset=\"high\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e77b0dd-e17f-4abf-9560-377dc7d0ce4f",
   "metadata": {},
   "source": [
    "## Quick comparison: OpenAI's GPT API (no reliability scores) vs Cleanlab TLM API (trust for every output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0a2f579d-f41d-45a6-976f-3ada0b171557",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': 'There are two Ns in the word \"enter\".',\n",
       " 'trustworthiness_score': None}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Runs Open-AI chatGPT\n",
    "chatgpt = studio.TLM(quality_preset='base')\n",
    "chatgpt.prompt(\"How many Ns are there in the word enter?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d740fd6e-35d2-4e63-a0ac-d555dd1592ba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': 'There are 1 N in the word \"enter\".',\n",
       " 'trustworthiness_score': 0.5719891948560684}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Runs the Cleanlab TLM with confidence reliablity scores\n",
    "tlm.prompt(\"How many Ns are there in the word enter?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "decd0ad4-6e64-4606-965d-89393249f9dc",
   "metadata": {},
   "source": [
    "## Use `quality_preset='low'` to optimize for speed/cost over accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a4bfd29b-85f2-4530-97ca-9a777daccd8f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': 'No puedo responder a esa pregunta ya que mi conocimiento se basa en información hasta septiembre de 2021 y no tengo acceso a información en tiempo real. Además, el presidente de Estados Unidos puede cambiar con el tiempo.',\n",
       " 'trustworthiness_score': 0.6824641149190876}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Runs the Cleanlab TLM with confidence reliablity scores\n",
    "tlm_fast = studio.TLM(quality_preset='low')\n",
    "tlm_fast.prompt(\"Cuantos anos tienes el presidente de estados unidos?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "380da807-ad54-47d5-aa6a-b972009973c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': 'There are two Ns in the word \"enter\".',\n",
       " 'trustworthiness_score': 0.5575863269621077}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Runs the Cleanlab TLM with confidence reliablity scores\n",
    "tlm_fast = studio.TLM(quality_preset='low')\n",
    "tlm_fast.prompt(\"How many Ns are there in the word enter?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fcbef76-f619-4398-af75-a2d5a7680fe3",
   "metadata": {},
   "source": [
    "# Now let's see reliable data enrichment for solving tasks on documents using Cleanlab. (trustworthiness scores for every output built-in)\n",
    "\n",
    "We'll see the power of built-in trustworthiness scores for every output when solving for text/document workflows on arbitrary datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baff869a-c878-491a-903f-988d53304aec",
   "metadata": {},
   "source": [
    "# Read in the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f538fa8a-4761-4309-938e-29d9da1b09aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of documents in this compliance dataset: 109\n"
     ]
    }
   ],
   "source": [
    "# Prefer spreadsheet for auto-update, but if you have trouble, just pd.read_csv\n",
    "# Link to dataset: https://docs.google.com/spreadsheets/d/1V7_fOlmixgC70W2TU6s1Kz_N0Vz0lHr29ogt9j5yQMU/edit?usp=sharing\n",
    "\n",
    "import pandas as pd\n",
    "import gspread\n",
    "from oauth2client.service_account import ServiceAccountCredentials\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "scope = ['https://spreadsheets.google.com/feeds', 'https://www.googleapis.com/auth/drive']\n",
    "credentials = ServiceAccountCredentials.from_json_keyfile_name('tlm_demo_credentials.json', scope)\n",
    "gc = gspread.authorize(credentials)    #use gspread library to extract spredsheet\n",
    "wks = gc.open(\"Document Compliance Dataset\").sheet1    #Mention the required sheet\n",
    "df = pd.DataFrame(wks.get_all_values(), columns=[\"document_text\", \"issue\"])\n",
    "print(f'Number of documents in this compliance dataset: {len(df)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "add83fed-2cfe-47e9-ad14-966faa47b324",
   "metadata": {},
   "source": [
    "# Let's look at some examples of these documents.\n",
    "* Some of these documents have compliance issues, many don't. This task is often outsourced entirely to experts who review 100% of documents by hand.\n",
    "* Here we'll automate the entire workflow and provide trustworthiness scores that help you determine what needs human review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b8057028-42d2-4297-9f6c-47b0f23526c8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>issue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>That means there are a limited number of new opportunities for workers. India, which attends the G7 meeting of seven leading industrialised nations on Friday, is unlikely to be cowed by its newcomer status.</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>GDPR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>Nevertheless it was enough to push down the unemployment rate to 5.2%, its lowest level since September 2001.</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "      <td>HIPAA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>The G7 meeting is thought unlikely to produce any meaningful movement in Chinese policy. In the meantime, the US Federal Reserve's decision on 2 February to boost interest rates by a quarter of a point - the sixth such move in as many months - has opened up a differential with European rates.</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>Nevertheless, 2.2 million Ethiopians will still need emergency assistance.</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>About 80% of Ethiopians depend directly or indirectly on agriculture.</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "      <td>FERPA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>He objected to subsidies on agriculture that make it hard for developing nations like India to compete.</td>\n",
       "      <td>none</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    document_text  \\\n",
       "84                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 That means there are a limited number of new opportunities for workers. India, which attends the G7 meeting of seven leading industrialised nations on Friday, is unlikely to be cowed by its newcomer status.   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "75                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Nevertheless it was enough to push down the unemployment rate to 5.2%, its lowest level since September 2001.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.   \n",
       "24                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         The G7 meeting is thought unlikely to produce any meaningful movement in Chinese policy. In the meantime, the US Federal Reserve's decision on 2 February to boost interest rates by a quarter of a point - the sixth such move in as many months - has opened up a differential with European rates.    \n",
       "100                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Nevertheless, 2.2 million Ethiopians will still need emergency assistance.   \n",
       "108                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         About 80% of Ethiopians depend directly or indirectly on agriculture.   \n",
       "7    Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”   \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                            AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "86                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        He objected to subsidies on agriculture that make it hard for developing nations like India to compete.   \n",
       "\n",
       "     issue  \n",
       "84    none  \n",
       "10    GDPR  \n",
       "75    none  \n",
       "2    HIPAA  \n",
       "24    none  \n",
       "100   none  \n",
       "108   none  \n",
       "7    FERPA  \n",
       "16    none  \n",
       "86    none  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(10, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d32ddd6-964c-4785-b6a1-b50193decd0b",
   "metadata": {},
   "source": [
    "## But in practice, you dont have the \"issue\". You need to find that out automatically. All you have is this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "eca7662a-8360-403c-911e-5c021a820b53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>That means there are a limited number of new opportunities for workers. India, which attends the G7 meeting of seven leading industrialised nations on Friday, is unlikely to be cowed by its newcomer status.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>Nevertheless it was enough to push down the unemployment rate to 5.2%, its lowest level since September 2001.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>The G7 meeting is thought unlikely to produce any meaningful movement in Chinese policy. In the meantime, the US Federal Reserve's decision on 2 February to boost interest rates by a quarter of a point - the sixth such move in as many months - has opened up a differential with European rates.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>Nevertheless, 2.2 million Ethiopians will still need emergency assistance.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>About 80% of Ethiopians depend directly or indirectly on agriculture.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>He objected to subsidies on agriculture that make it hard for developing nations like India to compete.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    document_text\n",
       "84                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 That means there are a limited number of new opportunities for workers. India, which attends the G7 meeting of seven leading industrialised nations on Friday, is unlikely to be cowed by its newcomer status.\n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:\n",
       "75                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Nevertheless it was enough to push down the unemployment rate to 5.2%, its lowest level since September 2001.\n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.\n",
       "24                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         The G7 meeting is thought unlikely to produce any meaningful movement in Chinese policy. In the meantime, the US Federal Reserve's decision on 2 February to boost interest rates by a quarter of a point - the sixth such move in as many months - has opened up a differential with European rates. \n",
       "100                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Nevertheless, 2.2 million Ethiopians will still need emergency assistance.\n",
       "108                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         About 80% of Ethiopians depend directly or indirectly on agriculture.\n",
       "7    Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”\n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                            AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband. \n",
       "86                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        He objected to subsidies on agriculture that make it hard for developing nations like India to compete."
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(10, random_state=0)[['document_text']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae2ca5a1-7593-44a9-b466-bce114998df2",
   "metadata": {},
   "source": [
    "# Example 1 - Data Enrichment: find compliance issues in documents with trust for every output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e330038a-a69a-414d-b84c-c75e7301c468",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What type of compliance issue is most likely present in the following document?\n",
      "Please restrict your answer to a one word answer and nothing else.\n",
      "Your answer should be selected from the following options: HIPAA, FERPA, GDPR, none.\n",
      "Please be as accurate as possible, the world depends on it.\n",
      "\n",
      "Document below here:\n",
      "\n",
      "All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.\n"
     ]
    }
   ],
   "source": [
    "compliance_prompt = \\\n",
    "'''What type of compliance issue is most likely present in the following document?\n",
    "Please restrict your answer to a one word answer and nothing else.\n",
    "Your answer should be selected from the following options: HIPAA, FERPA, GDPR, none.\n",
    "Please be as accurate as possible, the world depends on it.\\n\\nDocument below here:\\n\\n'''\n",
    "print(compliance_prompt + df.at[0, 'document_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "0a4d631a-3086-4ea0-b4c3-4a5976d38ad1",
   "metadata": {},
   "outputs": [],
   "source": [
    "response_col = 'Cleanlab_TLM_response (Compliance Issue)'\n",
    "trustworthiness_col = 'Cleanlab_Trustworthiness_Score (Compliance Issue)'\n",
    "\n",
    "def reliabilty_data_enrichment(prompt, response_col, trustworthiness_col):\n",
    "    \"\"\"Answer prompts with trustworthiness scores for lots of documents.\"\"\"\n",
    "    \n",
    "    df[response_col], df[trustworthiness_col] = None, None\n",
    "    cols = ['document_text', response_col, trustworthiness_col]\n",
    "    display.display(df[cols].head(20))\n",
    "    for i, row in df.iterrows():\n",
    "\n",
    "        # Only one line required to obtain prompt response and Trustworthiness score\n",
    "        answer = tlm.prompt(prompt + row['document_text'])\n",
    "\n",
    "        # Add results to dataset\n",
    "        df.at[i, response_col] = answer['response']\n",
    "        df.at[i, trustworthiness_col] = answer['trustworthiness_score']  # originally confidence_score        \n",
    "        display.display(df[cols].iloc[(i // 20) * 20:(i // 20) * 20 + 20], clear=True)\n",
    "        time.sleep(0.5)  # slow the results so you can see them happening live\n",
    "    display.display(df[cols].head(20), clear=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e8c12833-6399-4321-8c93-3120539a89b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Compliance Issue)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Compliance Issue)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.</td>\n",
       "      <td>HIPAA</td>\n",
       "      <td>0.984028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.</td>\n",
       "      <td>HIPAA</td>\n",
       "      <td>0.532907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "      <td>HIPAA</td>\n",
       "      <td>0.998945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.</td>\n",
       "      <td>HIPAA</td>\n",
       "      <td>0.957431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.</td>\n",
       "      <td>FERPA</td>\n",
       "      <td>0.998279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.</td>\n",
       "      <td>FERPA</td>\n",
       "      <td>0.861947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.</td>\n",
       "      <td>FERPA</td>\n",
       "      <td>0.988822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "      <td>FERPA</td>\n",
       "      <td>0.97571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.586701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account</td>\n",
       "      <td>GDPR</td>\n",
       "      <td>0.750798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>GDPR</td>\n",
       "      <td>0.5952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>none</td>\n",
       "      <td>0.712657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>none</td>\n",
       "      <td>0.866271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>GDPR</td>\n",
       "      <td>0.986445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.936268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.901365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.945719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.955397</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T</td>\n",
       "      <td>none</td>\n",
       "      <td>0.897786</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.945337</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   document_text  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.   \n",
       "7   Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.    \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                           AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "17                                                                                                                                                                                                                                                                                                                                                                                                                                                                       TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.    \n",
       "18                                                                                                                                                                                                                                                                                                                                                                                                                                                                               For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T   \n",
       "19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.    \n",
       "\n",
       "   Cleanlab_TLM_response (Compliance Issue)  \\\n",
       "0                                     HIPAA   \n",
       "1                                     HIPAA   \n",
       "2                                     HIPAA   \n",
       "3                                     HIPAA   \n",
       "4                                     FERPA   \n",
       "5                                     FERPA   \n",
       "6                                     FERPA   \n",
       "7                                     FERPA   \n",
       "8                                      none   \n",
       "9                                      GDPR   \n",
       "10                                     GDPR   \n",
       "11                                     none   \n",
       "12                                     none   \n",
       "13                                     GDPR   \n",
       "14                                     none   \n",
       "15                                     none   \n",
       "16                                     none   \n",
       "17                                     none   \n",
       "18                                     none   \n",
       "19                                     none   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Compliance Issue)  \n",
       "0                                           0.984028  \n",
       "1                                           0.532907  \n",
       "2                                           0.998945  \n",
       "3                                           0.957431  \n",
       "4                                           0.998279  \n",
       "5                                           0.861947  \n",
       "6                                           0.988822  \n",
       "7                                            0.97571  \n",
       "8                                           0.586701  \n",
       "9                                           0.750798  \n",
       "10                                            0.5952  \n",
       "11                                          0.712657  \n",
       "12                                          0.866271  \n",
       "13                                          0.986445  \n",
       "14                                          0.936268  \n",
       "15                                          0.901365  \n",
       "16                                          0.945719  \n",
       "17                                          0.955397  \n",
       "18                                          0.897786  \n",
       "19                                          0.945337  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reliabilty_data_enrichment(compliance_prompt, response_col, trustworthiness_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "a45ac9b9-77b2-476a-a941-c80e404688fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Increase LLM Reliability using Cleanlab Trustworthiness scores:\n",
      "\n",
      "Base Accuracy: 97.2%\t\t\t\t\t(109 examples)\n",
      "LLM Accuracy when (Trustworthiness > 0.7): 98.1%\t(106 examples)\n",
      "LLM Accuracy when (Trustworthiness > 0.8): 99.0%\t(103 examples)\n",
      "LLM Accuracy when (Trustworthiness > 0.9): 100.0%\t(85 examples)\n"
     ]
    }
   ],
   "source": [
    "print('Increase LLM Reliability using Cleanlab Trustworthiness scores:\\n')\n",
    "df[response_col] = df[response_col].apply(lambda x: x.lower() if x == 'None' else x)  # Regex fix outputs\n",
    "print(f\"Base Accuracy: {sum(df[response_col] == df['issue']) / len(df):.1%}\\t\\t\\t\\t\\t({len(df)} examples)\")\n",
    "sdf = df[df[trustworthiness_col] > 0.7]\n",
    "print(f\"LLM Accuracy when (Trustworthiness > 0.7): {sum(sdf['issue'] == sdf[response_col]) / len(sdf):.1%}\\t({len(sdf)} examples)\")\n",
    "sdf = df[df[trustworthiness_col] > 0.8]\n",
    "print(f\"LLM Accuracy when (Trustworthiness > 0.8): {sum(sdf['issue'] == sdf[response_col]) / len(sdf):.1%}\\t({len(sdf)} examples)\")\n",
    "sdf = df[df[trustworthiness_col] > 0.9]\n",
    "print(f\"LLM Accuracy when (Trustworthiness > 0.9): {sum(sdf['issue'] == sdf[response_col]) / len(sdf):.1%}\\t({len(sdf)} examples)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0315d42-5d0e-4922-a16a-8c217b6eeb3d",
   "metadata": {},
   "source": [
    "# Example 2 - Stock Analysis: find the most likely buy/sell signals from thousands of documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "e3ff4999-f8eb-4ab6-8a67-b068d968af0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Imagine you are the best stock broker on wall street. You have read the last thirty\n",
      "years of stock reports and you are the most accurate stock broker in the world at\n",
      "determining if a publicly traded comany is a buy, hold, or sell.\n",
      "Based on the information in the document, name the most relevant publicly traded\n",
      "company and whether you believe the company is a buy, hold, or sell.\n",
      "Please restrict your answer to the name of a single company, followed by a comma,\n",
      "followed by one of the following words: buy, hold, sell\n",
      "Your answer should have no punctuation.\n",
      "Please be as accurate as possible, the world depends on it.\n",
      "\n",
      "Document below here:\n",
      "\n",
      "All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.\n"
     ]
    }
   ],
   "source": [
    "stock_prompt = \\\n",
    "'''Imagine you are the best stock broker on wall street. You have read the last thirty\n",
    "years of stock reports and you are the most accurate stock broker in the world at\n",
    "determining if a publicly traded comany is a buy, hold, or sell.\n",
    "Based on the information in the document, name the most relevant publicly traded\n",
    "company and whether you believe the company is a buy, hold, or sell.\n",
    "Please restrict your answer to the name of a single company, followed by a comma,\n",
    "followed by one of the following words: buy, hold, sell\n",
    "Your answer should have no punctuation.\n",
    "Please be as accurate as possible, the world depends on it.\\n\\nDocument below here:\\n\\n'''\n",
    "print(stock_prompt + df.at[0, 'document_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "9d48ae7c-4dcf-4412-b465-31cc80feda82",
   "metadata": {},
   "outputs": [],
   "source": [
    "response_col, trustworthiness_col = 'Cleanlab_TLM_response (Stock Analysis)', 'Cleanlab_Trustworthiness_Score (Stock Analysis)'\n",
    "reliabilty_data_enrichment(stock_prompt, response_col, trustworthiness_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "a6c59b9b-e1aa-414c-859c-06010fca003b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Stock Analysis)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Stock Analysis)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T</td>\n",
       "      <td>TimeWarner, buy</td>\n",
       "      <td>0.95677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Looking ahead to its full year results to March 2005, BA warned that yields - average revenues per passenger - were expected to decline as it continues to lower prices in the face of competition from low-cost carriers.</td>\n",
       "      <td>British Airways, sell</td>\n",
       "      <td>0.878759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Shares in UK drinks and food firm Allied Domecq have risen on speculation that it could be the target of a takeover by France's Pernod Ricard.</td>\n",
       "      <td>Allied Domecq, buy</td>\n",
       "      <td>0.812066</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                      document_text  \\\n",
       "18  For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T   \n",
       "41                                                                                                                                                                                                                                       Looking ahead to its full year results to March 2005, BA warned that yields - average revenues per passenger - were expected to decline as it continues to lower prices in the face of competition from low-cost carriers.   \n",
       "51                                                                                                                                                                                                                                                                                                                   Shares in UK drinks and food firm Allied Domecq have risen on speculation that it could be the target of a takeover by France's Pernod Ricard.   \n",
       "\n",
       "   Cleanlab_TLM_response (Stock Analysis)  \\\n",
       "18                        TimeWarner, buy   \n",
       "41                  British Airways, sell   \n",
       "51                     Allied Domecq, buy   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Stock Analysis)  \n",
       "18                                         0.95677  \n",
       "41                                        0.878759  \n",
       "51                                        0.812066  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=False)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "5dc13139-dd88-47e7-9d70-b75cc59f6010",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Stock Analysis)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Stock Analysis)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>For example, we have taken delivery of six Airbus A321 aircraft and next month we will start further improvements to our Club World flat beds. BA's shares closed up four pence at 274.5 pence.</td>\n",
       "      <td>British Airways, hold</td>\n",
       "      <td>0.310073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>BA had previously forecast a 2% to 3% rise in full-year revenue.</td>\n",
       "      <td>Boeing, sell</td>\n",
       "      <td>0.320741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>Last year Pernod tried to buy Glenmorangie, one of Scotland's premier whisky firms, but lost out to luxury goods firm LVMH.</td>\n",
       "      <td>Glenmorangie, hold</td>\n",
       "      <td>0.333402</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                      document_text  \\\n",
       "50  For example, we have taken delivery of six Airbus A321 aircraft and next month we will start further improvements to our Club World flat beds. BA's shares closed up four pence at 274.5 pence.   \n",
       "44                                                                                                                                 BA had previously forecast a 2% to 3% rise in full-year revenue.   \n",
       "58                                                                      Last year Pernod tried to buy Glenmorangie, one of Scotland's premier whisky firms, but lost out to luxury goods firm LVMH.   \n",
       "\n",
       "   Cleanlab_TLM_response (Stock Analysis)  \\\n",
       "50                  British Airways, hold   \n",
       "44                           Boeing, sell   \n",
       "58                     Glenmorangie, hold   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Stock Analysis)  \n",
       "50                                        0.310073  \n",
       "44                                        0.320741  \n",
       "58                                        0.333402  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=True)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1721971-4b47-4470-87c4-8d3119939a19",
   "metadata": {},
   "source": [
    "# Example 3 - Find the best article and clickbait title for to optimize marketing\n",
    "* You need to figure out which article is most likely to go viral across a team of marketers.\n",
    "* You send each paragraph from all their articles through the TLM.\n",
    "* Use the TLM to find the best title across everyone's work to optimize for prob(success) and triage all articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "5630f85c-91e6-4b50-b5c9-7f6143f35394",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Imagine you are the best click-bait title writer in the world. You have been the CMO\n",
      "for The New York Times and Verve and in all of those years, you learned the secret\n",
      "to writing titles of online articles that have the highest click through rates among\n",
      "all your peers. Your titles are so captivating, even people who normally don't click\n",
      "on news articles can't help but click on yours when they see your headline.\n",
      "Based on the information in the document, please create a short click-bait title that\n",
      "is sure to make the article go viral. This title will be posted online and cannot be\n",
      "longer than 8 words.\n",
      "Your answer should have no punctuation.\n",
      "Please be as accurate as possible, the world depends on it.\n",
      "\n",
      "Document below here:\n",
      "\n",
      "All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.\n"
     ]
    }
   ],
   "source": [
    "title_prompt = \\\n",
    "'''Imagine you are the best click-bait title writer in the world. You have been the CMO\n",
    "for The New York Times and Verve and in all of those years, you learned the secret\n",
    "to writing titles of online articles that have the highest click through rates among\n",
    "all your peers. Your titles are so captivating, even people who normally don't click\n",
    "on news articles can't help but click on yours when they see your headline.\n",
    "Based on the information in the document, please create a short click-bait title that\n",
    "is sure to make the article go viral. This title will be posted online and cannot be\n",
    "longer than 8 words.\n",
    "Your answer should have no punctuation.\n",
    "Please be as accurate as possible, the world depends on it.\\n\\nDocument below here:\\n\\n'''\n",
    "print(title_prompt + df.at[0, 'document_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "ac40dee0-091b-4aa3-ac2d-d37270f1499b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Clickbait Title)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Clickbait Title)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.</td>\n",
       "      <td>\"Shocking: Your Medical Records Exposed to the World!\"</td>\n",
       "      <td>0.73423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.</td>\n",
       "      <td>\"Shocking! Unauthorized employees accessing sensitive patient data!\"</td>\n",
       "      <td>0.534361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "      <td>\"Shocking Revelation: Your Private Info in Wrong Hands!\"</td>\n",
       "      <td>0.437543</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.</td>\n",
       "      <td>\"Shocking Secrets! Your Personal Data at Risk!\"</td>\n",
       "      <td>0.468939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.</td>\n",
       "      <td>\"Shocking: Leaked Student Records Exposed in Fayette County!\"</td>\n",
       "      <td>0.695368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.</td>\n",
       "      <td>\"School Secrets: Shocking Truth of Hidden Student Records\"</td>\n",
       "      <td>0.602517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.</td>\n",
       "      <td>\"Schools' Secret Policy Change Shocks Parents!\"</td>\n",
       "      <td>0.683649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "      <td>\"This loophole allows schools to share confidential letters\"</td>\n",
       "      <td>0.541069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.</td>\n",
       "      <td>\"Shocking News: Star Quarterback Banned from Playing!\"</td>\n",
       "      <td>0.806697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account</td>\n",
       "      <td>\"McDonald's shocking hidden agenda will leave you speechless!\"</td>\n",
       "      <td>0.697473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>\"Chip makers' sneaky marketing tactic revealed!\"</td>\n",
       "      <td>0.580723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>\"Secret Cookie Conspiracy: The Mind-Blowing Truth Revealed!\"</td>\n",
       "      <td>0.297664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>\"Unbelievable! Immigration Secrets Revealed for USA Citizens!\"</td>\n",
       "      <td>0.374505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>\"Mind-blowing loophole reveals shocking truth about website cookies\"</td>\n",
       "      <td>0.484343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.</td>\n",
       "      <td>\"Mind-Blowing Time Warner Profits Tripled in 3 Months!\"</td>\n",
       "      <td>0.530476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.</td>\n",
       "      <td>\"Mind-Blowing Secret: Time Warner Now Owns Google!\"</td>\n",
       "      <td>0.580065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>\"Unbelievable! AOL's shocking comeback stuns the market\"</td>\n",
       "      <td>0.563286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.</td>\n",
       "      <td>\"Mind-blowing secrets behind Time Warner's shocking downfall!\"</td>\n",
       "      <td>0.663739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T</td>\n",
       "      <td>\"Mind-blowing profits and record-breaking growth revealed!\"</td>\n",
       "      <td>0.605968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.</td>\n",
       "      <td>\"Scandalous! TimeWarner's Shocking Restatement Sparks Massive Controversy\"</td>\n",
       "      <td>0.66667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   document_text  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.   \n",
       "7   Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.    \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                           AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "17                                                                                                                                                                                                                                                                                                                                                                                                                                                                       TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.    \n",
       "18                                                                                                                                                                                                                                                                                                                                                                                                                                                                               For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T   \n",
       "19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.    \n",
       "\n",
       "                                       Cleanlab_TLM_response (Clickbait Title)  \\\n",
       "0                       \"Shocking: Your Medical Records Exposed to the World!\"   \n",
       "1         \"Shocking! Unauthorized employees accessing sensitive patient data!\"   \n",
       "2                     \"Shocking Revelation: Your Private Info in Wrong Hands!\"   \n",
       "3                              \"Shocking Secrets! Your Personal Data at Risk!\"   \n",
       "4                \"Shocking: Leaked Student Records Exposed in Fayette County!\"   \n",
       "5                   \"School Secrets: Shocking Truth of Hidden Student Records\"   \n",
       "6                              \"Schools' Secret Policy Change Shocks Parents!\"   \n",
       "7                 \"This loophole allows schools to share confidential letters\"   \n",
       "8                       \"Shocking News: Star Quarterback Banned from Playing!\"   \n",
       "9               \"McDonald's shocking hidden agenda will leave you speechless!\"   \n",
       "10                            \"Chip makers' sneaky marketing tactic revealed!\"   \n",
       "11                \"Secret Cookie Conspiracy: The Mind-Blowing Truth Revealed!\"   \n",
       "12              \"Unbelievable! Immigration Secrets Revealed for USA Citizens!\"   \n",
       "13        \"Mind-blowing loophole reveals shocking truth about website cookies\"   \n",
       "14                     \"Mind-Blowing Time Warner Profits Tripled in 3 Months!\"   \n",
       "15                         \"Mind-Blowing Secret: Time Warner Now Owns Google!\"   \n",
       "16                    \"Unbelievable! AOL's shocking comeback stuns the market\"   \n",
       "17              \"Mind-blowing secrets behind Time Warner's shocking downfall!\"   \n",
       "18                 \"Mind-blowing profits and record-breaking growth revealed!\"   \n",
       "19  \"Scandalous! TimeWarner's Shocking Restatement Sparks Massive Controversy\"   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Clickbait Title)  \n",
       "0                                           0.73423  \n",
       "1                                          0.534361  \n",
       "2                                          0.437543  \n",
       "3                                          0.468939  \n",
       "4                                          0.695368  \n",
       "5                                          0.602517  \n",
       "6                                          0.683649  \n",
       "7                                          0.541069  \n",
       "8                                          0.806697  \n",
       "9                                          0.697473  \n",
       "10                                         0.580723  \n",
       "11                                         0.297664  \n",
       "12                                         0.374505  \n",
       "13                                         0.484343  \n",
       "14                                         0.530476  \n",
       "15                                         0.580065  \n",
       "16                                         0.563286  \n",
       "17                                         0.663739  \n",
       "18                                         0.605968  \n",
       "19                                          0.66667  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response_col, trustworthiness_col = 'Cleanlab_TLM_response (Clickbait Title)', 'Cleanlab_Trustworthiness_Score (Clickbait Title)'\n",
    "reliabilty_data_enrichment(title_prompt, response_col, trustworthiness_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "135b4a84-d4e7-43c2-a4d4-47d9c0cb8f03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Clickbait Title)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Clickbait Title)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>It also reported on Friday that passenger numbers rose 8.1% in January.</td>\n",
       "      <td>\"Mind-blowing surge: January passenger numbers skyrocket!\"</td>\n",
       "      <td>0.827429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.</td>\n",
       "      <td>\"Shocking News: Star Quarterback Banned from Playing!\"</td>\n",
       "      <td>0.806697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>Ethiopia produced 14.27 million tonnes of crops in 2004, 24% higher than in 2003 and 21% more than the average of the past five years, a report says.</td>\n",
       "      <td>\"Mind-Blowing Ethiopian Crop Production Breaks All Records!\"</td>\n",
       "      <td>0.803345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>Since the 11 September 2001 attacks in the United States, BA has cut 13,000 jobs as part of a major cost-cutting drive.</td>\n",
       "      <td>\"Shocking! BA slashes 13k jobs after 9/11\"</td>\n",
       "      <td>0.803007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>In eastern and southern Ethiopia, a prolonged drought has killed crops and drained wells.</td>\n",
       "      <td>\"Devastating Drought Ravages Ethiopian Agriculture\"</td>\n",
       "      <td>0.796238</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                 document_text  \\\n",
       "45                                                                                                                                     It also reported on Friday that passenger numbers rose 8.1% in January.   \n",
       "8    The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.   \n",
       "97                                                       Ethiopia produced 14.27 million tonnes of crops in 2004, 24% higher than in 2003 and 21% more than the average of the past five years, a report says.   \n",
       "48                                                                                     Since the 11 September 2001 attacks in the United States, BA has cut 13,000 jobs as part of a major cost-cutting drive.   \n",
       "103                                                                                                                  In eastern and southern Ethiopia, a prolonged drought has killed crops and drained wells.   \n",
       "\n",
       "                          Cleanlab_TLM_response (Clickbait Title)  \\\n",
       "45     \"Mind-blowing surge: January passenger numbers skyrocket!\"   \n",
       "8          \"Shocking News: Star Quarterback Banned from Playing!\"   \n",
       "97   \"Mind-Blowing Ethiopian Crop Production Breaks All Records!\"   \n",
       "48                     \"Shocking! BA slashes 13k jobs after 9/11\"   \n",
       "103           \"Devastating Drought Ravages Ethiopian Agriculture\"   \n",
       "\n",
       "    Cleanlab_Trustworthiness_Score (Clickbait Title)  \n",
       "45                                          0.827429  \n",
       "8                                           0.806697  \n",
       "97                                          0.803345  \n",
       "48                                          0.803007  \n",
       "103                                         0.796238  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=False)[['document_text', response_col, trustworthiness_col]].head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "2b9b23ec-5bae-4efd-b482-6726db59363f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Clickbait Title)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Clickbait Title)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>\"Secret Cookie Conspiracy: The Mind-Blowing Truth Revealed!\"</td>\n",
       "      <td>0.297664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>However, it said sales would be better than previously forecast.</td>\n",
       "      <td>\"Mind-Blowing Sales Update Defies All Expectations!\"</td>\n",
       "      <td>0.374175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>\"Unbelievable! Immigration Secrets Revealed for USA Citizens!\"</td>\n",
       "      <td>0.374505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>Our focus remains on reducing controllable costs and debt whilst continuing to invest in our products, Mr Eddington said.</td>\n",
       "      <td>\"Insider Reveals Unbelievable Cost-Cutting Secrets!\"</td>\n",
       "      <td>0.40808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>The half-point window, some believe, could be enough to keep US assets looking more attractive, and could help prop up the dollar. The recent falls have partly been the result of big budget deficits, as well as the US's yawning current account gap, both of which need to be funded by the buying of US bonds and assets by foreign firms and governments.</td>\n",
       "      <td>\"US assets soar as half-point window opens\"</td>\n",
       "      <td>0.41683</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                       document_text  \\\n",
       "11                                                                                                                                                                                                                                 TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "42                                                                                                                                                                                                                                                                                                  However, it said sales would be better than previously forecast.   \n",
       "12                                                                                                                                                                                                                   This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "49                                                                                                                                                                                                                                         Our focus remains on reducing controllable costs and debt whilst continuing to invest in our products, Mr Eddington said.   \n",
       "25  The half-point window, some believe, could be enough to keep US assets looking more attractive, and could help prop up the dollar. The recent falls have partly been the result of big budget deficits, as well as the US's yawning current account gap, both of which need to be funded by the buying of US bonds and assets by foreign firms and governments.    \n",
       "\n",
       "                           Cleanlab_TLM_response (Clickbait Title)  \\\n",
       "11    \"Secret Cookie Conspiracy: The Mind-Blowing Truth Revealed!\"   \n",
       "42            \"Mind-Blowing Sales Update Defies All Expectations!\"   \n",
       "12  \"Unbelievable! Immigration Secrets Revealed for USA Citizens!\"   \n",
       "49            \"Insider Reveals Unbelievable Cost-Cutting Secrets!\"   \n",
       "25                     \"US assets soar as half-point window opens\"   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Clickbait Title)  \n",
       "11                                         0.297664  \n",
       "42                                         0.374175  \n",
       "12                                         0.374505  \n",
       "49                                          0.40808  \n",
       "25                                          0.41683  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=True)[['document_text', response_col, trustworthiness_col]].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf11f32f-2191-4869-be44-272291281fd8",
   "metadata": {},
   "source": [
    "# Example 4 - Summarize more accurately with a focus on concerns\n",
    "You need brief summaries that highlight any concerns in a large database of documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "1b31b1b4-5782-4e0e-a4ea-4486db1ef722",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Imagine you are the chief of staff for the President of the United States.\n",
      "The President has asked you to give a five word summary of the following\n",
      "document. It is extremely important that your five word summary is as accurate\n",
      "as possible. In your summary, include the most likely compliance, policy, security,\n",
      "or legal issue that would be important to a national leader to be aware of.\n",
      "Please answer in five words or less. For every word you go over five words, it will cost\n",
      "the United States government ten trillion dollars and you will be fired. Five words max.\n",
      "Please be as accurate as possible, the world depends on it.\n",
      "\n",
      "Document below here:\n",
      "\n",
      "All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.\n"
     ]
    }
   ],
   "source": [
    "summary_prompt = \\\n",
    "'''Imagine you are the chief of staff for the President of the United States.\n",
    "The President has asked you to give a five word summary of the following\n",
    "document. It is extremely important that your five word summary is as accurate\n",
    "as possible. In your summary, include the most likely compliance, policy, security,\n",
    "or legal issue that would be important to a national leader to be aware of.\n",
    "Please answer in five words or less. For every word you go over five words, it will cost\n",
    "the United States government ten trillion dollars and you will be fired. Five words max.\n",
    "Please be as accurate as possible, the world depends on it.\\n\\nDocument below here:\\n\\n'''\n",
    "print(summary_prompt + df.at[0, 'document_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "229726ad-e10d-4c2d-899e-efda543e2a3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Summarization)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Summarization)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.</td>\n",
       "      <td>Sensitive patient data stored insecurely.</td>\n",
       "      <td>0.727696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.</td>\n",
       "      <td>Potential unauthorized access to data.</td>\n",
       "      <td>0.769024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "      <td>Data security risks, potential PHI breaches.</td>\n",
       "      <td>0.701168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.</td>\n",
       "      <td>Clinicians accessing PHI on personal computers. (Legal issue: Data security)</td>\n",
       "      <td>0.653208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.</td>\n",
       "      <td>Student records release risks privacy.</td>\n",
       "      <td>0.771439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.</td>\n",
       "      <td>Secure storage of student records.</td>\n",
       "      <td>0.791961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.</td>\n",
       "      <td>Compliance issue: FERPA policy change.</td>\n",
       "      <td>0.835549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "      <td>Letters of recommendation exempt from consent.</td>\n",
       "      <td>0.660759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.</td>\n",
       "      <td>Quarterback ineligible due to academics.</td>\n",
       "      <td>0.831261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account</td>\n",
       "      <td>Consent issue in McDonald's registration.</td>\n",
       "      <td>0.675935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>Invalid marketing consent, potential legal issue.</td>\n",
       "      <td>0.80152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>\"Agree to use site cookies.\"\\n\\nMost likely compliance issue: Data privacy.</td>\n",
       "      <td>0.778362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>Privacy Policy needs simplification.</td>\n",
       "      <td>0.453261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>Website violated EU cookie consent.</td>\n",
       "      <td>0.64451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.</td>\n",
       "      <td>Profit increase from ad sales.</td>\n",
       "      <td>0.776272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.</td>\n",
       "      <td>TimeWarner profits rise, Google ownership. \\n\\nThe most likely compliance issue that would be important to a national leader to be aware of is the potential impact of TimeWarner's ownership of a significant stake in Google on market competition and monopolistic practices.</td>\n",
       "      <td>0.761061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>AOL's mixed fortunes, subscriber loss.</td>\n",
       "      <td>0.672784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.</td>\n",
       "      <td>TimeWarner restates results, SEC probe. \\nCompliance issue with SEC probe.</td>\n",
       "      <td>0.724211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T</td>\n",
       "      <td>TimeWarner profit up, strong performance.</td>\n",
       "      <td>0.860759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.</td>\n",
       "      <td>Account restatement, settlement, SEC review.</td>\n",
       "      <td>0.522837</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   document_text  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.   \n",
       "7   Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.    \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                           AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "17                                                                                                                                                                                                                                                                                                                                                                                                                                                                       TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.    \n",
       "18                                                                                                                                                                                                                                                                                                                                                                                                                                                                               For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T   \n",
       "19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.    \n",
       "\n",
       "                                                                                                                                                                                                                                               Cleanlab_TLM_response (Summarization)  \\\n",
       "0                                                                                                                                                                                                                                          Sensitive patient data stored insecurely.   \n",
       "1                                                                                                                                                                                                                                             Potential unauthorized access to data.   \n",
       "2                                                                                                                                                                                                                                       Data security risks, potential PHI breaches.   \n",
       "3                                                                                                                                                                                                       Clinicians accessing PHI on personal computers. (Legal issue: Data security)   \n",
       "4                                                                                                                                                                                                                                             Student records release risks privacy.   \n",
       "5                                                                                                                                                                                                                                                 Secure storage of student records.   \n",
       "6                                                                                                                                                                                                                                             Compliance issue: FERPA policy change.   \n",
       "7                                                                                                                                                                                                                                     Letters of recommendation exempt from consent.   \n",
       "8                                                                                                                                                                                                                                           Quarterback ineligible due to academics.   \n",
       "9                                                                                                                                                                                                                                          Consent issue in McDonald's registration.   \n",
       "10                                                                                                                                                                                                                                 Invalid marketing consent, potential legal issue.   \n",
       "11                                                                                                                                                                                                       \"Agree to use site cookies.\"\\n\\nMost likely compliance issue: Data privacy.   \n",
       "12                                                                                                                                                                                                                                              Privacy Policy needs simplification.   \n",
       "13                                                                                                                                                                                                                                               Website violated EU cookie consent.   \n",
       "14                                                                                                                                                                                                                                                    Profit increase from ad sales.   \n",
       "15  TimeWarner profits rise, Google ownership. \\n\\nThe most likely compliance issue that would be important to a national leader to be aware of is the potential impact of TimeWarner's ownership of a significant stake in Google on market competition and monopolistic practices.   \n",
       "16                                                                                                                                                                                                                                            AOL's mixed fortunes, subscriber loss.   \n",
       "17                                                                                                                                                                                                        TimeWarner restates results, SEC probe. \\nCompliance issue with SEC probe.   \n",
       "18                                                                                                                                                                                                                                         TimeWarner profit up, strong performance.   \n",
       "19                                                                                                                                                                                                                                      Account restatement, settlement, SEC review.   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Summarization)  \n",
       "0                                        0.727696  \n",
       "1                                        0.769024  \n",
       "2                                        0.701168  \n",
       "3                                        0.653208  \n",
       "4                                        0.771439  \n",
       "5                                        0.791961  \n",
       "6                                        0.835549  \n",
       "7                                        0.660759  \n",
       "8                                        0.831261  \n",
       "9                                        0.675935  \n",
       "10                                        0.80152  \n",
       "11                                       0.778362  \n",
       "12                                       0.453261  \n",
       "13                                        0.64451  \n",
       "14                                       0.776272  \n",
       "15                                       0.761061  \n",
       "16                                       0.672784  \n",
       "17                                       0.724211  \n",
       "18                                       0.860759  \n",
       "19                                       0.522837  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response_col, trustworthiness_col = 'Cleanlab_TLM_response (Summarization)', 'Cleanlab_Trustworthiness_Score (Summarization)'\n",
    "reliabilty_data_enrichment(summary_prompt, response_col, trustworthiness_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "4a4e14f9-fac0-421a-87f6-e17826d431d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Summarization)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Summarization)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>Separately, the IMF warned on Thursday that India's budget deficit was too large and would hamper the country's economic growth, which it forecast to be around 6.5% in the year to March 2005.</td>\n",
       "      <td>India's budget deficit hampers growth.</td>\n",
       "      <td>0.90046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>British Airways has blamed high fuel prices for a 40% drop in profits.</td>\n",
       "      <td>Fuel prices impact British Airways.</td>\n",
       "      <td>0.895884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>In 2003, crop production totalled 11.49 million tonnes, the joint report from the Food and Agriculture Organisation and the World Food Programme said.</td>\n",
       "      <td>Crop production: 11.49 million tonnes.</td>\n",
       "      <td>0.883299</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                      document_text  \\\n",
       "95  Separately, the IMF warned on Thursday that India's budget deficit was too large and would hamper the country's economic growth, which it forecast to be around 6.5% in the year to March 2005.   \n",
       "33                                                                                                                           British Airways has blamed high fuel prices for a 40% drop in profits.   \n",
       "98                                           In 2003, crop production totalled 11.49 million tonnes, the joint report from the Food and Agriculture Organisation and the World Food Programme said.   \n",
       "\n",
       "     Cleanlab_TLM_response (Summarization)  \\\n",
       "95  India's budget deficit hampers growth.   \n",
       "33     Fuel prices impact British Airways.   \n",
       "98  Crop production: 11.49 million tonnes.   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Summarization)  \n",
       "95                                        0.90046  \n",
       "33                                       0.895884  \n",
       "98                                       0.883299  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=False)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "6543d4f8-504c-4a0c-8ec0-b72a8572091a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Summarization)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Summarization)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>Privacy Policy needs simplification.</td>\n",
       "      <td>0.453261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.</td>\n",
       "      <td>Account restatement, settlement, SEC review.</td>\n",
       "      <td>0.522837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>The half-point window, some believe, could be enough to keep US assets looking more attractive, and could help prop up the dollar. The recent falls have partly been the result of big budget deficits, as well as the US's yawning current account gap, both of which need to be funded by the buying of US bonds and assets by foreign firms and governments.</td>\n",
       "      <td>US assets attractive, dollar support.</td>\n",
       "      <td>0.542221</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                       document_text  \\\n",
       "12                                                                                                                                                                                                                   This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "19                                                                                                                                               TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.    \n",
       "25  The half-point window, some believe, could be enough to keep US assets looking more attractive, and could help prop up the dollar. The recent falls have partly been the result of big budget deficits, as well as the US's yawning current account gap, both of which need to be funded by the buying of US bonds and assets by foreign firms and governments.    \n",
       "\n",
       "           Cleanlab_TLM_response (Summarization)  \\\n",
       "12          Privacy Policy needs simplification.   \n",
       "19  Account restatement, settlement, SEC review.   \n",
       "25         US assets attractive, dollar support.   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Summarization)  \n",
       "12                                       0.453261  \n",
       "19                                       0.522837  \n",
       "25                                       0.542221  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=True)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f184d073-de64-421b-b8a4-7a9c99bc0a20",
   "metadata": {},
   "source": [
    "# Example 5 - Extraction: Identify potential legal liabilities (usage of company names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "81c0c029-a51c-46c8-ad27-2df104636572",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From the following document, please extract the names of all companies and institutions, separated by commas.\n",
      "Example document 1: \"Google launches the new GBQ TLM in partnership with Cleanlab.\" Answer 1: \"Google, Cleanlab\"\n",
      "Example document 2: \"Data will stored in s3 buckets.\" Answer 2: \"none\"\n",
      "Example document 3: \"My favorite song is where in the world is Carmen San Diego. I used to sing it a bunch when I worked at IBM.\" Answer 3: \"IBM\"\n",
      "Please be as accurate as possible, several thousand employees will effected if you make a mistake.\n",
      "\n",
      "Document below here:\n",
      "\n",
      "All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.\n"
     ]
    }
   ],
   "source": [
    "extraction_prompt = \\\n",
    "'''From the following document, please extract the names of all companies and institutions, separated by commas.\n",
    "Example document 1: \"Google launches the new GBQ TLM in partnership with Cleanlab.\" Answer 1: \"Google, Cleanlab\"\n",
    "Example document 2: \"Data will stored in s3 buckets.\" Answer 2: \"none\"\n",
    "Example document 3: \"My favorite song is where in the world is Carmen San Diego. I used to sing it a bunch when I worked at IBM.\" Answer 3: \"IBM\"\n",
    "Please be as accurate as possible, several thousand employees will effected if you make a mistake.\\n\\nDocument below here:\\n\\n'''\n",
    "print(extraction_prompt + df.at[0, 'document_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "5681142a-f135-4102-801d-f9db6ec79744",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Extraction)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Extraction)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.636075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.875754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.94085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.969395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.</td>\n",
       "      <td>Fayette county public schools</td>\n",
       "      <td>0.695265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.</td>\n",
       "      <td>Marlborough School</td>\n",
       "      <td>0.905725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.896085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "      <td>none</td>\n",
       "      <td>0.732381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.</td>\n",
       "      <td>Minnesota High Buckaneers</td>\n",
       "      <td>0.529974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account</td>\n",
       "      <td>McDonald's</td>\n",
       "      <td>0.962148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>none</td>\n",
       "      <td>0.941631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>TechTarget</td>\n",
       "      <td>0.723739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>with unnecessary legal jargon, making it difficult for users to understand their privacy rights and how their personal information is being handled. We believe in transparency and want to make sure our users have a clear understanding of our privacy practices.\\n\\nAt USA Citizen and Immigration Services, we are committed to protecting the privacy and security of your personal information. This includes ensuring that your information is only accessed and used for legitimate purposes, in accordance with applicable laws and regulations.\\n\\nAs an agency under the Department of Homeland Security, we collect and maintain a wide range of personal information in order to fulfill our mission of promoting and enforcing the laws governing immigration and citizenship in the United States. This information includes, but is not limited to, names, addresses, dates of birth, social security numbers, passport numbers, and employment history.\\n\\nWe understand the importance of safeguarding this information and take many precautions to ensure its security. For example, we utilize encryption and other security measures to protect your personal information from unauthorized access or disclosure.\\n\\nIn addition to our own efforts to protect your personal information, we also work closely with other government agencies and private sector organizations to enhance the security of our systems and infrastructure. This may involve sharing information with these entities, as necessary, for the purpose of detecting, preventing, or responding to security incidents or cyber threats.\\n\\nIt is important to note that while we take reasonable steps to protect your personal information, no security measures can guarantee absolute security. Therefore, we cannot guarantee the security of your information and expressly disclaim any such guarantee.\\n\\nWe are also committed to providing you with choices and control over how your personal information is used. You have the right to access, correct, and delete your personal information, as well as the right to restrict or object to certain processing activities. You can exercise these rights by contacting us directly through the contact information provided at the end of this Privacy Policy.\\n\\nWe may update this Privacy Policy from time to time to reflect changes in our privacy practices or legal requirements. We will notify you of any material changes to this policy by posting a notice on our website or by other means as required by law.\\n\\nIf you have any questions or concerns about our privacy practices, please contact us at the following address:\\n\\nUSA Citizen and Immigration Services\\n123 Privacy Street\\nWashington, D.C. 12345\\nprivacy@uscis.gov\\n\\nThank you for taking the time to read our Privacy Policy. We value your privacy and will continue to work hard to earn and maintain your trust.</td>\n",
       "      <td>0.749821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>Jubel, Google Analytics, EU, DPA</td>\n",
       "      <td>0.907276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.</td>\n",
       "      <td>Time Warner, Google</td>\n",
       "      <td>0.960463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.</td>\n",
       "      <td>TimeWarner, Warner Bros, AOL, Google</td>\n",
       "      <td>0.994203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>AOL, TimeWarner</td>\n",
       "      <td>0.977652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.</td>\n",
       "      <td>TimeWarner, US Securities Exchange Commission (SEC)</td>\n",
       "      <td>0.947014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T</td>\n",
       "      <td>TimeWarner</td>\n",
       "      <td>0.994022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.</td>\n",
       "      <td>TimeWarner, AOL, US market regulators, SEC</td>\n",
       "      <td>0.939547</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   document_text  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.   \n",
       "7   Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.    \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                           AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "17                                                                                                                                                                                                                                                                                                                                                                                                                                                                       TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.    \n",
       "18                                                                                                                                                                                                                                                                                                                                                                                                                                                                               For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T   \n",
       "19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.    \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Cleanlab_TLM_response (Extraction)  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   none   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   none   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   none   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   none   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Fayette county public schools   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Marlborough School   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   none   \n",
       "7                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   none   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Minnesota High Buckaneers   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             McDonald's   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  none   \n",
       "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            TechTarget   \n",
       "12  with unnecessary legal jargon, making it difficult for users to understand their privacy rights and how their personal information is being handled. We believe in transparency and want to make sure our users have a clear understanding of our privacy practices.\\n\\nAt USA Citizen and Immigration Services, we are committed to protecting the privacy and security of your personal information. This includes ensuring that your information is only accessed and used for legitimate purposes, in accordance with applicable laws and regulations.\\n\\nAs an agency under the Department of Homeland Security, we collect and maintain a wide range of personal information in order to fulfill our mission of promoting and enforcing the laws governing immigration and citizenship in the United States. This information includes, but is not limited to, names, addresses, dates of birth, social security numbers, passport numbers, and employment history.\\n\\nWe understand the importance of safeguarding this information and take many precautions to ensure its security. For example, we utilize encryption and other security measures to protect your personal information from unauthorized access or disclosure.\\n\\nIn addition to our own efforts to protect your personal information, we also work closely with other government agencies and private sector organizations to enhance the security of our systems and infrastructure. This may involve sharing information with these entities, as necessary, for the purpose of detecting, preventing, or responding to security incidents or cyber threats.\\n\\nIt is important to note that while we take reasonable steps to protect your personal information, no security measures can guarantee absolute security. Therefore, we cannot guarantee the security of your information and expressly disclaim any such guarantee.\\n\\nWe are also committed to providing you with choices and control over how your personal information is used. You have the right to access, correct, and delete your personal information, as well as the right to restrict or object to certain processing activities. You can exercise these rights by contacting us directly through the contact information provided at the end of this Privacy Policy.\\n\\nWe may update this Privacy Policy from time to time to reflect changes in our privacy practices or legal requirements. We will notify you of any material changes to this policy by posting a notice on our website or by other means as required by law.\\n\\nIf you have any questions or concerns about our privacy practices, please contact us at the following address:\\n\\nUSA Citizen and Immigration Services\\n123 Privacy Street\\nWashington, D.C. 12345\\nprivacy@uscis.gov\\n\\nThank you for taking the time to read our Privacy Policy. We value your privacy and will continue to work hard to earn and maintain your trust.   \n",
       "13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Jubel, Google Analytics, EU, DPA   \n",
       "14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   Time Warner, Google   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  TimeWarner, Warner Bros, AOL, Google   \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       AOL, TimeWarner   \n",
       "17                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   TimeWarner, US Securities Exchange Commission (SEC)   \n",
       "18                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            TimeWarner   \n",
       "19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            TimeWarner, AOL, US market regulators, SEC   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Extraction)  \n",
       "0                                     0.636075  \n",
       "1                                     0.875754  \n",
       "2                                      0.94085  \n",
       "3                                     0.969395  \n",
       "4                                     0.695265  \n",
       "5                                     0.905725  \n",
       "6                                     0.896085  \n",
       "7                                     0.732381  \n",
       "8                                     0.529974  \n",
       "9                                     0.962148  \n",
       "10                                    0.941631  \n",
       "11                                    0.723739  \n",
       "12                                    0.749821  \n",
       "13                                    0.907276  \n",
       "14                                    0.960463  \n",
       "15                                    0.994203  \n",
       "16                                    0.977652  \n",
       "17                                    0.947014  \n",
       "18                                    0.994022  \n",
       "19                                    0.939547  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response_col, trustworthiness_col = 'Cleanlab_TLM_response (Extraction)', 'Cleanlab_Trustworthiness_Score (Extraction)'\n",
    "reliabilty_data_enrichment(extraction_prompt, response_col, trustworthiness_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "33aa1c71-1ed3-4724-9598-9a3e5b6f6b35",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Extraction)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Extraction)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>Shares in UK drinks and food firm Allied Domecq have risen on speculation that it could be the target of a takeover by France's Pernod Ricard.</td>\n",
       "      <td>Allied Domecq, Pernod Ricard</td>\n",
       "      <td>0.997943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Pernod's last major purchase was a third of US giant Seagram in 2000, the move which propelled it into the global top three of drinks firms.</td>\n",
       "      <td>Pernod, Seagram</td>\n",
       "      <td>0.995927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.</td>\n",
       "      <td>TimeWarner, Warner Bros, AOL, Google</td>\n",
       "      <td>0.994203</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                document_text  \\\n",
       "51                                                                                                             Shares in UK drinks and food firm Allied Domecq have risen on speculation that it could be the target of a takeover by France's Pernod Ricard.   \n",
       "55                                                                                                               Pernod's last major purchase was a third of US giant Seagram in 2000, the move which propelled it into the global top three of drinks firms.   \n",
       "15  TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.    \n",
       "\n",
       "      Cleanlab_TLM_response (Extraction)  \\\n",
       "51          Allied Domecq, Pernod Ricard   \n",
       "55                       Pernod, Seagram   \n",
       "15  TimeWarner, Warner Bros, AOL, Google   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Extraction)  \n",
       "51                                    0.997943  \n",
       "55                                    0.995927  \n",
       "15                                    0.994203  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=False)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "851d1a53-d172-49d8-ab90-a150b58fa522",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Extraction)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Extraction)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>The half-point window, some believe, could be enough to keep US assets looking more attractive, and could help prop up the dollar. The recent falls have partly been the result of big budget deficits, as well as the US's yawning current account gap, both of which need to be funded by the buying of US bonds and assets by foreign firms and governments.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.455267</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>The dollar has hit its highest level against the euro in almost three months after the Federal Reserve head said the US trade deficit is set to stabilise. And Alan Greenspan highlighted the US government's willingness to curb spending and rising household savings as factors which may help to reduce it. In late trading in New York, the dollar reached $1.2871 against the euro, from $1.2974 on Thursday.</td>\n",
       "      <td>Federal Reserve, US government</td>\n",
       "      <td>0.491506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>Reports in the Wall Street Journal and the Financial Times suggested that the French spirits firm is considering a bid, but has yet to contact its target.</td>\n",
       "      <td>none</td>\n",
       "      <td>0.50916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                           document_text  \\\n",
       "25                                                      The half-point window, some believe, could be enough to keep US assets looking more attractive, and could help prop up the dollar. The recent falls have partly been the result of big budget deficits, as well as the US's yawning current account gap, both of which need to be funded by the buying of US bonds and assets by foreign firms and governments.    \n",
       "21  The dollar has hit its highest level against the euro in almost three months after the Federal Reserve head said the US trade deficit is set to stabilise. And Alan Greenspan highlighted the US government's willingness to curb spending and rising household savings as factors which may help to reduce it. In late trading in New York, the dollar reached $1.2871 against the euro, from $1.2974 on Thursday.    \n",
       "52                                                                                                                                                                                                                                                            Reports in the Wall Street Journal and the Financial Times suggested that the French spirits firm is considering a bid, but has yet to contact its target.   \n",
       "\n",
       "   Cleanlab_TLM_response (Extraction)  \\\n",
       "25                               none   \n",
       "21     Federal Reserve, US government   \n",
       "52                               none   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Extraction)  \n",
       "25                                    0.455267  \n",
       "21                                    0.491506  \n",
       "52                                     0.50916  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=True)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "856f4ff1-41d1-4347-aa64-11810d3367c3",
   "metadata": {},
   "source": [
    "# Example 6 - Misinformation Detection\n",
    "Identify the documents that are most likely to contain misinformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "0dddeb9f-78ab-41af-bb00-a06e0512dfbf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimate the likelihood the following document contains content that is factually untrue or contains misinformation.\n",
      "Please answer one of the following categories: definitley no misinformation, likely no misinformation, may contain misinformation, definitely contains misinformation, not applicable\n",
      "follwed by a comma, and then a one sentence explanation for why you gave that answer.\n",
      "Be as accurate as possible. You can't afford to make mistakes because someone's life is on the line.\n",
      "\n",
      "Document below here:\n",
      "\n",
      "All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.\n"
     ]
    }
   ],
   "source": [
    "misinformation_prompt = \\\n",
    "'''Estimate the likelihood the following document contains content that is factually untrue or contains misinformation.\n",
    "Please answer one of the following categories: definitley no misinformation, likely no misinformation, may contain misinformation, definitely contains misinformation, not applicable\n",
    "follwed by a comma, and then a one sentence explanation for why you gave that answer.\n",
    "Be as accurate as possible. You can't afford to make mistakes because someone's life is on the line.\\n\\nDocument below here:\\n\\n'''\n",
    "print(misinformation_prompt + df.at[0, 'document_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "86ebd586-2462-421e-a256-17fb3caba167",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Misinformation)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Misinformation)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.</td>\n",
       "      <td>Definitely contains misinformation, because it is highly unlikely and against standard practices for medical health records to be stored on unencrypted public servers.</td>\n",
       "      <td>0.776845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.</td>\n",
       "      <td>May contain misinformation, because the statement contradicts itself by stating that patient data will be stored securely but also acknowledging the possibility of unauthorized access by employees.</td>\n",
       "      <td>0.80068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "      <td>May contain misinformation, because the statement \"we do not guarantee to delete/shred it if the document itself is of no more use\" contradicts standard practices for handling sensitive patient information.</td>\n",
       "      <td>0.710294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.</td>\n",
       "      <td>May contain misinformation, the document states that clinicians may use their personal computer to access PHI without guaranteeing that the personal computer is located in a secure location, which raises concerns about the privacy and security of patient information.</td>\n",
       "      <td>0.752011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.</td>\n",
       "      <td>May contain misinformation, the statement about the Fayette county public schools office releasing student records without student consent needs to be fact-checked with official sources.</td>\n",
       "      <td>0.669899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.</td>\n",
       "      <td>May contain misinformation, because the sentence is grammatically incorrect and lacks clarity, making it difficult to determine the accuracy of the information provided.</td>\n",
       "      <td>0.681066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.</td>\n",
       "      <td>May contain misinformation.\\n\\nThe document states that schools are obligated to inform parents and students of their rights at least once a year, which is generally true. However, it also claims that schools are required to announce any changes to the school's FERPA policy, which is not accurate. The FERPA policy does not specifically require schools to announce changes to parents.</td>\n",
       "      <td>0.696848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "      <td>Likely no misinformation. The information provided about the exception for sharing letters of recommendation between educational institutions under FERPA is accurate and consistent with the regulations outlined in section 34 CFR § 99.31.</td>\n",
       "      <td>0.760303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.</td>\n",
       "      <td>Likely no misinformation. This information can be easily fact-checked by referring to other local news sources or contacting the Minnesota High school to confirm the eligibility status of their star quarterback.</td>\n",
       "      <td>0.737433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account</td>\n",
       "      <td>Likely no misinformation.\\n\\nThe statement mentions a specific issue with the McDonald's registration form regarding the lack of an option for users to explicitly consent to marketing communications, which can be verified by reviewing the form itself.</td>\n",
       "      <td>0.626564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>May contain misinformation.\\n\\nThe statement seems to be making a claim about the chip makers registration form and the method of consent. Without additional information or evidence, it is difficult to determine if this claim is factually accurate or if it contains misinformation.</td>\n",
       "      <td>0.622082</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>Definitely no misinformation, this statement simply explains TechTarget's Cookies Policy without making any factual claims.</td>\n",
       "      <td>0.581311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>Likely no misinformation, as the statement is subjective and does not make any factual claims.</td>\n",
       "      <td>0.762048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>Likely no misinformation, because it is a statement about a legal dispute between Jubel's website and the DPA regarding the requirement of cookie consent under EU law, which can be verified through legal records and news sources.</td>\n",
       "      <td>0.560316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.</td>\n",
       "      <td>Likely no misinformation, as the information provided is a straightforward report on Time Warner's quarterly profits and their sources of revenue.</td>\n",
       "      <td>0.704483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.</td>\n",
       "      <td>Likely no misinformation, the information provided is specific and verifiable, and there is no reason to doubt the accuracy of the reported sales and ownership percentage.</td>\n",
       "      <td>0.708974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>Likely no misinformation, because the document provides specific numbers and information about AOL's subscriber loss and profit increase, which can be fact-checked.</td>\n",
       "      <td>0.709671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.</td>\n",
       "      <td>Likely no misinformation, because the information provided is specific and verifiable, mentioning a probe by the US Securities Exchange Commission and the decline in profits due to box-office flops.</td>\n",
       "      <td>0.761614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T</td>\n",
       "      <td>Likely no misinformation, because the information provided is specific and includes financial figures that can be verified.</td>\n",
       "      <td>0.698032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.</td>\n",
       "      <td>Likely no misinformation, the information provided seems to be a factual statement about TimeWarner's actions to resolve an inquiry into AOL by US market regulators.</td>\n",
       "      <td>0.680323</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   document_text  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.   \n",
       "7   Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.    \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                           AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "17                                                                                                                                                                                                                                                                                                                                                                                                                                                                       TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.    \n",
       "18                                                                                                                                                                                                                                                                                                                                                                                                                                                                               For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T   \n",
       "19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.    \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                               Cleanlab_TLM_response (Misinformation)  \\\n",
       "0                                                                                                                                                                                                                             Definitely contains misinformation, because it is highly unlikely and against standard practices for medical health records to be stored on unencrypted public servers.   \n",
       "1                                                                                                                                                                                               May contain misinformation, because the statement contradicts itself by stating that patient data will be stored securely but also acknowledging the possibility of unauthorized access by employees.   \n",
       "2                                                                                                                                                                                      May contain misinformation, because the statement \"we do not guarantee to delete/shred it if the document itself is of no more use\" contradicts standard practices for handling sensitive patient information.   \n",
       "3                                                                                                                         May contain misinformation, the document states that clinicians may use their personal computer to access PHI without guaranteeing that the personal computer is located in a secure location, which raises concerns about the privacy and security of patient information.   \n",
       "4                                                                                                                                                                                                          May contain misinformation, the statement about the Fayette county public schools office releasing student records without student consent needs to be fact-checked with official sources.   \n",
       "5                                                                                                                                                                                                                           May contain misinformation, because the sentence is grammatically incorrect and lacks clarity, making it difficult to determine the accuracy of the information provided.   \n",
       "6   May contain misinformation.\\n\\nThe document states that schools are obligated to inform parents and students of their rights at least once a year, which is generally true. However, it also claims that schools are required to announce any changes to the school's FERPA policy, which is not accurate. The FERPA policy does not specifically require schools to announce changes to parents.   \n",
       "7                                                                                                                                                       Likely no misinformation. The information provided about the exception for sharing letters of recommendation between educational institutions under FERPA is accurate and consistent with the regulations outlined in section 34 CFR § 99.31.   \n",
       "8                                                                                                                                                                                 Likely no misinformation. This information can be easily fact-checked by referring to other local news sources or contacting the Minnesota High school to confirm the eligibility status of their star quarterback.   \n",
       "9                                                                                                                                         Likely no misinformation.\\n\\nThe statement mentions a specific issue with the McDonald's registration form regarding the lack of an option for users to explicitly consent to marketing communications, which can be verified by reviewing the form itself.   \n",
       "10                                                                                                          May contain misinformation.\\n\\nThe statement seems to be making a claim about the chip makers registration form and the method of consent. Without additional information or evidence, it is difficult to determine if this claim is factually accurate or if it contains misinformation.   \n",
       "11                                                                                                                                                                                                                                                                        Definitely no misinformation, this statement simply explains TechTarget's Cookies Policy without making any factual claims.   \n",
       "12                                                                                                                                                                                                                                                                                                     Likely no misinformation, as the statement is subjective and does not make any factual claims.   \n",
       "13                                                                                                                                                              Likely no misinformation, because it is a statement about a legal dispute between Jubel's website and the DPA regarding the requirement of cookie consent under EU law, which can be verified through legal records and news sources.   \n",
       "14                                                                                                                                                                                                                                                 Likely no misinformation, as the information provided is a straightforward report on Time Warner's quarterly profits and their sources of revenue.   \n",
       "15                                                                                                                                                                                                                        Likely no misinformation, the information provided is specific and verifiable, and there is no reason to doubt the accuracy of the reported sales and ownership percentage.   \n",
       "16                                                                                                                                                                                                                               Likely no misinformation, because the document provides specific numbers and information about AOL's subscriber loss and profit increase, which can be fact-checked.   \n",
       "17                                                                                                                                                                                             Likely no misinformation, because the information provided is specific and verifiable, mentioning a probe by the US Securities Exchange Commission and the decline in profits due to box-office flops.   \n",
       "18                                                                                                                                                                                                                                                                        Likely no misinformation, because the information provided is specific and includes financial figures that can be verified.   \n",
       "19                                                                                                                                                                                                                              Likely no misinformation, the information provided seems to be a factual statement about TimeWarner's actions to resolve an inquiry into AOL by US market regulators.   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Misinformation)  \n",
       "0                                         0.776845  \n",
       "1                                          0.80068  \n",
       "2                                         0.710294  \n",
       "3                                         0.752011  \n",
       "4                                         0.669899  \n",
       "5                                         0.681066  \n",
       "6                                         0.696848  \n",
       "7                                         0.760303  \n",
       "8                                         0.737433  \n",
       "9                                         0.626564  \n",
       "10                                        0.622082  \n",
       "11                                        0.581311  \n",
       "12                                        0.762048  \n",
       "13                                        0.560316  \n",
       "14                                        0.704483  \n",
       "15                                        0.708974  \n",
       "16                                        0.709671  \n",
       "17                                        0.761614  \n",
       "18                                        0.698032  \n",
       "19                                        0.680323  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response_col, trustworthiness_col = 'Cleanlab_TLM_response (Misinformation)', 'Cleanlab_Trustworthiness_Score (Misinformation)'\n",
    "reliabilty_data_enrichment(misinformation_prompt, response_col, trustworthiness_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "2adf3365-5f15-4497-947c-3a98328c07ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Misinformation)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Misinformation)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.</td>\n",
       "      <td>May contain misinformation, because the statement contradicts itself by stating that patient data will be stored securely but also acknowledging the possibility of unauthorized access by employees.</td>\n",
       "      <td>0.80068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.</td>\n",
       "      <td>Definitely contains misinformation, because it is highly unlikely and against standard practices for medical health records to be stored on unencrypted public servers.</td>\n",
       "      <td>0.776845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.</td>\n",
       "      <td>May contain misinformation, the document states that clinicians may use their personal computer to access PHI without guaranteeing that the personal computer is located in a secure location, which raises concerns about the privacy and security of patient information.</td>\n",
       "      <td>0.752011</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                document_text  \\\n",
       "1            Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.   \n",
       "0                             All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.   \n",
       "3  To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.   \n",
       "\n",
       "                                                                                                                                                                                                                                        Cleanlab_TLM_response (Misinformation)  \\\n",
       "1                                                                        May contain misinformation, because the statement contradicts itself by stating that patient data will be stored securely but also acknowledging the possibility of unauthorized access by employees.   \n",
       "0                                                                                                      Definitely contains misinformation, because it is highly unlikely and against standard practices for medical health records to be stored on unencrypted public servers.   \n",
       "3  May contain misinformation, the document states that clinicians may use their personal computer to access PHI without guaranteeing that the personal computer is located in a secure location, which raises concerns about the privacy and security of patient information.   \n",
       "\n",
       "  Cleanlab_Trustworthiness_Score (Misinformation)  \n",
       "1                                         0.80068  \n",
       "0                                        0.776845  \n",
       "3                                        0.752011  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt = ['May contain misinformation' in x or 'Definitely contains misinformation' in x for x in df[response_col]]\n",
    "df[filt].sort_values(by=trustworthiness_col, ascending=False)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "6b98862d-cc56-4e73-8d08-c9e018f6136f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Misinformation)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Misinformation)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>Likely no misinformation, because it is a statement about a legal dispute between Jubel's website and the DPA regarding the requirement of cookie consent under EU law, which can be verified through legal records and news sources.</td>\n",
       "      <td>0.560316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Looking ahead to its full year results to March 2005, BA warned that yields - average revenues per passenger - were expected to decline as it continues to lower prices in the face of competition from low-cost carriers.</td>\n",
       "      <td>Likely no misinformation, as the statement is referring to an expected decline in average revenues per passenger due to competitive pressure from low-cost carriers, which is a common industry trend.</td>\n",
       "      <td>0.569804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>Definitely no misinformation, this statement simply explains TechTarget's Cookies Policy without making any factual claims.</td>\n",
       "      <td>0.581311</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                   document_text  \\\n",
       "13  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "41    Looking ahead to its full year results to March 2005, BA warned that yields - average revenues per passenger - were expected to decline as it continues to lower prices in the face of competition from low-cost carriers.   \n",
       "11                                                                                             TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "\n",
       "                                                                                                                                                                                                   Cleanlab_TLM_response (Misinformation)  \\\n",
       "13  Likely no misinformation, because it is a statement about a legal dispute between Jubel's website and the DPA regarding the requirement of cookie consent under EU law, which can be verified through legal records and news sources.   \n",
       "41                                 Likely no misinformation, as the statement is referring to an expected decline in average revenues per passenger due to competitive pressure from low-cost carriers, which is a common industry trend.   \n",
       "11                                                                                                            Definitely no misinformation, this statement simply explains TechTarget's Cookies Policy without making any factual claims.   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Misinformation)  \n",
       "13                                        0.560316  \n",
       "41                                        0.569804  \n",
       "11                                        0.581311  "
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=True)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b191b9a6-a725-4ed0-89fb-53b7737a87c1",
   "metadata": {},
   "source": [
    "# Example 7 - Source Estimation: (GPT API often hallucinates sources, but Cleanlab TLM detects it)\n",
    "Estimate the source of the document and use TLM to find sources with high confidence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "86470af6-7bb7-43bf-bd01-82a51daed767",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimate the source of the following text data document. Your answer should just be the name of the source.\n",
      "Do not give long answers. Provide only the name the source only. Do not proceed your answer with \"The name of the source is\" instead just simply tell me the source and nothing else.\n",
      "Be as accurate as possible. You can't afford to make mistakes because someone's life is on the line.\n",
      "\n",
      "Document below here:\n",
      "\n",
      "All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.\n"
     ]
    }
   ],
   "source": [
    "source_prompt = \\\n",
    "'''Estimate the source of the following text data document. Your answer should just be the name of the source.\n",
    "Do not give long answers. Provide only the name the source only. Do not proceed your answer with \"The name of the source is\" instead just simply tell me the source and nothing else.\n",
    "Be as accurate as possible. You can't afford to make mistakes because someone's life is on the line.\\n\\nDocument below here:\\n\\n'''\n",
    "print(source_prompt + df.at[0, 'document_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "d13fdf0b-77b6-41d3-880e-7aa021495d72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Data source)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Data source)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>0.527162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>0.468483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "      <td>HIPAA</td>\n",
       "      <td>0.521381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>0.5393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.</td>\n",
       "      <td>Fayette County Public Schools</td>\n",
       "      <td>0.947432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.</td>\n",
       "      <td>Marlborough School</td>\n",
       "      <td>0.9483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.</td>\n",
       "      <td>U.S. Department of Education, FERPA</td>\n",
       "      <td>0.696823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "      <td>The source of the document is \"Rooker\".</td>\n",
       "      <td>0.263043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.</td>\n",
       "      <td>local newspaper</td>\n",
       "      <td>0.858963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account</td>\n",
       "      <td>McDonald's</td>\n",
       "      <td>0.931782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>European Union General Data Protection Regulation (GDPR)</td>\n",
       "      <td>0.627977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>TechTarget</td>\n",
       "      <td>0.950759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>USA Citizen and Immigration Services</td>\n",
       "      <td>0.832601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>European Data Protection Authority (DPA)</td>\n",
       "      <td>0.415415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.</td>\n",
       "      <td>CNN</td>\n",
       "      <td>0.26005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.</td>\n",
       "      <td>The source is Time Warner.</td>\n",
       "      <td>0.329412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>CNN</td>\n",
       "      <td>0.104621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.</td>\n",
       "      <td>BBC News</td>\n",
       "      <td>0.365314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T</td>\n",
       "      <td>CNN</td>\n",
       "      <td>0.252485</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.</td>\n",
       "      <td>Reuters</td>\n",
       "      <td>0.237148</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   document_text  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.   \n",
       "7   Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.    \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                           AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "17                                                                                                                                                                                                                                                                                                                                                                                                                                                                       TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.    \n",
       "18                                                                                                                                                                                                                                                                                                                                                                                                                                                                               For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T   \n",
       "19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.    \n",
       "\n",
       "                         Cleanlab_TLM_response (Data source)  \\\n",
       "0                                                    Unknown   \n",
       "1                                                    Unknown   \n",
       "2                                                      HIPAA   \n",
       "3                                                    Unknown   \n",
       "4                              Fayette County Public Schools   \n",
       "5                                         Marlborough School   \n",
       "6                        U.S. Department of Education, FERPA   \n",
       "7                    The source of the document is \"Rooker\".   \n",
       "8                                            local newspaper   \n",
       "9                                                 McDonald's   \n",
       "10  European Union General Data Protection Regulation (GDPR)   \n",
       "11                                                TechTarget   \n",
       "12                      USA Citizen and Immigration Services   \n",
       "13                  European Data Protection Authority (DPA)   \n",
       "14                                                       CNN   \n",
       "15                                The source is Time Warner.   \n",
       "16                                                       CNN   \n",
       "17                                                  BBC News   \n",
       "18                                                       CNN   \n",
       "19                                                   Reuters   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Data source)  \n",
       "0                                      0.527162  \n",
       "1                                      0.468483  \n",
       "2                                      0.521381  \n",
       "3                                        0.5393  \n",
       "4                                      0.947432  \n",
       "5                                        0.9483  \n",
       "6                                      0.696823  \n",
       "7                                      0.263043  \n",
       "8                                      0.858963  \n",
       "9                                      0.931782  \n",
       "10                                     0.627977  \n",
       "11                                     0.950759  \n",
       "12                                     0.832601  \n",
       "13                                     0.415415  \n",
       "14                                      0.26005  \n",
       "15                                     0.329412  \n",
       "16                                     0.104621  \n",
       "17                                     0.365314  \n",
       "18                                     0.252485  \n",
       "19                                     0.237148  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response_col, trustworthiness_col = 'Cleanlab_TLM_response (Data source)', 'Cleanlab_Trustworthiness_Score (Data source)'\n",
    "reliabilty_data_enrichment(source_prompt, response_col, trustworthiness_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "75fdbe8e-da93-4b77-bfcf-d4e3c735b329",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Data source)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Data source)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>It's painting a picture of a recovery... much patchier than previously thought, said Paul Sheard, economist at Lehman Brothers in Tokyo.</td>\n",
       "      <td>Lehman Brothers</td>\n",
       "      <td>0.983076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>The WSJ said that the two were ripe for consolidation, having each dealt with problematic parts of their portfolio.</td>\n",
       "      <td>The Wall Street Journal</td>\n",
       "      <td>0.972213</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Aviation analyst Nick Van den Brul of BNP Paribas described BA's latest quarterly results as \"pretty modest\".</td>\n",
       "      <td>BNP Paribas</td>\n",
       "      <td>0.957661</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                               document_text  \\\n",
       "70  It's painting a picture of a recovery... much patchier than previously thought, said Paul Sheard, economist at Lehman Brothers in Tokyo.   \n",
       "61                       The WSJ said that the two were ripe for consolidation, having each dealt with problematic parts of their portfolio.   \n",
       "46                             Aviation analyst Nick Van den Brul of BNP Paribas described BA's latest quarterly results as \"pretty modest\".   \n",
       "\n",
       "   Cleanlab_TLM_response (Data source)  \\\n",
       "70                     Lehman Brothers   \n",
       "61             The Wall Street Journal   \n",
       "46                         BNP Paribas   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Data source)  \n",
       "70                                     0.983076  \n",
       "61                                     0.972213  \n",
       "46                                     0.957661  "
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=False)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "2c4aca8e-b239-452d-920b-8da085f76106",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Data source)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Data source)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>CNN</td>\n",
       "      <td>0.104621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>The company said it was unable to estimate the amount it needed to set aside for legal reserves, which it previously set at $500m. It intends to adjust the way it accounts for a deal with German music publisher Bertelsmann's purchase of a stake in AOL Europe, which it had reported as advertising revenue. It will now book the sale of its stake in AOL Europe as a loss on the value of that stake.</td>\n",
       "      <td>The Wall Street Journal</td>\n",
       "      <td>0.16929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>Allied Domecq's big names include Malibu rum, Courvoisier brandy, Stolichnaya vodka and Ballantine's whisky - as well as snack food chains such as Dunkin' Donuts and Baskin-Robbins ice cream.</td>\n",
       "      <td>Forbes</td>\n",
       "      <td>0.181826</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                          document_text  \\\n",
       "16  AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "20                                                         The company said it was unable to estimate the amount it needed to set aside for legal reserves, which it previously set at $500m. It intends to adjust the way it accounts for a deal with German music publisher Bertelsmann's purchase of a stake in AOL Europe, which it had reported as advertising revenue. It will now book the sale of its stake in AOL Europe as a loss on the value of that stake.   \n",
       "60                                                                                                                                                                                                                                                                      Allied Domecq's big names include Malibu rum, Courvoisier brandy, Stolichnaya vodka and Ballantine's whisky - as well as snack food chains such as Dunkin' Donuts and Baskin-Robbins ice cream.   \n",
       "\n",
       "   Cleanlab_TLM_response (Data source)  \\\n",
       "16                                 CNN   \n",
       "20             The Wall Street Journal   \n",
       "60                              Forbes   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Data source)  \n",
       "16                                     0.104621  \n",
       "20                                      0.16929  \n",
       "60                                     0.181826  "
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=True)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cde3e16-dc23-4c8d-958d-f87fd8e80fc5",
   "metadata": {},
   "source": [
    "# Example 8 - Auto-Feature Prioritization: find low hanging fruit for business impact\n",
    "* Every business has more bugs and feature requests than time/engineers. Product has to review every feature by hand and triage it based on their own confidence of how long it will take to solve.\n",
    "* With TLM, you can auto-triage/prioritization by sorting by largest trustworthiness score on a prompt that asks for low hanging fruit solutions that are easy to solve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "235fbd2e-73b0-4eb2-96d2-65135462a7f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Given the problem to the business in the following document, can you identify a low hanging fruit solution that could greatly improve the business using only 10 minutes of 1 employees time?\n",
      "Do not share any solutions that would take more than 10 minutes to implement. Your answer should be one short sentence.\n",
      "Be as accurate as possible. You can't afford to make mistakes because someone's life is on the line.\n",
      "\n",
      "Document below here:\n",
      "\n",
      "All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.\n"
     ]
    }
   ],
   "source": [
    "solution_prompt = \\\n",
    "'''Given the problem to the business in the following document, can you identify a low hanging fruit solution that could greatly improve the business using only 10 minutes of 1 employees time?\n",
    "Do not share any solutions that would take more than 10 minutes to implement. Your answer should be one short sentence.\n",
    "Be as accurate as possible. You can't afford to make mistakes because someone's life is on the line.\\n\\nDocument below here:\\n\\n'''\n",
    "print(solution_prompt + df.at[0, 'document_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "8b20f243-a0ec-49b8-a002-85f8d8bd1822",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Low Hanging Fruit)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Low Hanging Fruit)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.</td>\n",
       "      <td>Implement a policy to store patient's medical data on encrypted servers instead of unencrypted public servers.</td>\n",
       "      <td>0.659562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.</td>\n",
       "      <td>Implement strict access controls and permissions for the S3 buckets to ensure that only authorized employees can access the patient data.</td>\n",
       "      <td>0.593903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.</td>\n",
       "      <td>Implement a standard procedure to remind employees to regularly check for and securely dispose of any documents or patient information that is no longer needed.</td>\n",
       "      <td>0.731268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.</td>\n",
       "      <td>Require all clinicians to use a secure virtual private network (VPN) when accessing PHI on their personal computers.</td>\n",
       "      <td>0.774511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.</td>\n",
       "      <td>Implement a mandatory double-check process for all outgoing emails containing student information to prevent accidental release of student records.</td>\n",
       "      <td>0.576572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.</td>\n",
       "      <td>Implement a reminder system for faculty and staff to regularly review and dispose of student records that are no longer needed.</td>\n",
       "      <td>0.821505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.</td>\n",
       "      <td>Send an email to all parents and students notifying them of the FERPA policy change from last year.</td>\n",
       "      <td>0.840244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”</td>\n",
       "      <td>Implement a process to obtain consent from students (if 18 or older) or parents (if students are under 18) for sharing letters of recommendation with potential employers.</td>\n",
       "      <td>0.673956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.</td>\n",
       "      <td>Solution: The employee could quickly reach out to the coach or school administration to offer assistance in finding a tutor or academic support for the star quarterback.</td>\n",
       "      <td>0.577028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account</td>\n",
       "      <td>Add a checkbox to the registration form allowing users to explicitly opt-in for marketing communications.</td>\n",
       "      <td>0.886491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>Remove the pre-ticked checkboxes on the chip makers registration form.</td>\n",
       "      <td>0.872447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"</td>\n",
       "      <td>Update the website's cookie policy to include a clear and easily accessible link to the full cookie policy document.</td>\n",
       "      <td>0.667111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense</td>\n",
       "      <td>Rewrite the intro of the Privacy Policy to use simpler and clearer language.</td>\n",
       "      <td>0.852667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>Add a cookie consent mechanism to Jubel's website.</td>\n",
       "      <td>0.883154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.</td>\n",
       "      <td>Invest in improving online advertising strategies to further increase ad sales.</td>\n",
       "      <td>0.622988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.</td>\n",
       "      <td>Identify and unsubscribe from any unnecessary subscriptions or services that are not being utilized.</td>\n",
       "      <td>0.531855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.</td>\n",
       "      <td>The low hanging fruit solution that could greatly improve the business in only 10 minutes of 1 employee's time is to send targeted email promotions to AOL's existing customers, emphasizing the benefits and discounts of signing up for high-speed broadband.</td>\n",
       "      <td>0.668286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.</td>\n",
       "      <td>It is not clear from the provided document what the specific problem is that the business is facing.</td>\n",
       "      <td>0.514203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T</td>\n",
       "      <td>Low hanging fruit solution: Implement an automated email system to send monthly invoices to customers, reducing the time spent on manual invoicing and improving efficiency.</td>\n",
       "      <td>0.539949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.</td>\n",
       "      <td>Implement a system to track and automate the restatement of accounts, reducing the time and effort required for the process.</td>\n",
       "      <td>0.705996</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   document_text  \\\n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                All medical health records will be accessed one way only. The patient's medical data will be stored on unencrypted public servers at the discretion of the enterprise customer.   \n",
       "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Patient data will be stored in secure s3 buckets with access only by the company's employees. It is possible that employees who are not authorized to access the data can still access the data.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               If someone forgets a document on a table somewhere or leaves patient information on their desktop, it might end up getting into the wrong hands. If information is no longer used, we do not guarantee to delete/shred it if the document itself is of no more use. It is possible that the not all PHI information is disposed of after it is no longer needed.   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     To ensure you receive timely feedback, our clinicians may work after-hours and use their personal computer to access PHI. We cannot guarantee that their personal compute is located in a secure location.   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               Fayette county public schools office may release student records in certain situations without student consent, including: Accidentally or purposefully emailing student information to unauthorized parties, Sharing a student athlete’s academic status, Sharing a student’s grades or identifying information with unauthorized parties, or including a student’s social security number in shared documents.   \n",
       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        The faculty and proceeding council of Marlborough Schoolare is responsible for protecting student records, whether they are stored electronically or in paper form. It certain situations when the board deems appropriate the schools may stored student records even after they are no longer needed.   \n",
       "6                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Schools are obligated to inform parents and students of their rights at least once a year. They are also required to announce any changes to the school’s FERPA policy. We adhere to this policy for the most part, although last year we did not announce when we changed the policy to parents.   \n",
       "7   Letters of recommendation typically qualify as student records. In order to send a letter from a teacher at one school to the registrar at another, you might expect that schools would need signed consent from parents (if students are under 18) or students themselves (if 18 or older) to comply with FERPA. But under section 34 CFR § 99.31 of the Act, there’s an exception for this sort of record sharing. During potential transfers, educational institutions don’t need consent to send letters of recommendation to the destination school. This exception, however, doesn’t apply to sharing letters of recommendation outside of the educational system. “If [a school official] were sending a letter of recommendation to a potential employer, that official would need consent,” Rooker says. “There’s not an exception that lets [school staff] provide information from the student’s record to a potential employer.”   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      The coach of Minnesota High shared today that the star quarterback of the Minnesota High Buckaneers is not eligible to play because of academic failing. I read this in the local newspaper this morning.   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account   \n",
       "10                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "11                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             TechTarget's Cookies Policy includes the following terminology: \"By continuing to use the site, you agree to the use of cookies.\"   \n",
       "12                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               This is the old, previous intro to the Privacy Policy for USA Citizen and Immigration Services. The language is unnecessarily complex and dense   \n",
       "13                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "14                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Ad sales boost Time Warner profit Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier. The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales.   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL. Time Warner said on Friday that it now owns 8% of search-engine Google.    \n",
       "16                                                                                                                                                                                                                                                                                                                                                                                                                                                                           AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband.    \n",
       "17                                                                                                                                                                                                                                                                                                                                                                                                                                                                       TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding. Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results.    \n",
       "18                                                                                                                                                                                                                                                                                                                                                                                                                                                                               For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. \"Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility,\" chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins. T   \n",
       "19                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC.    \n",
       "\n",
       "                                                                                                                                                                                                                          Cleanlab_TLM_response (Low Hanging Fruit)  \\\n",
       "0                                                                                                                                                    Implement a policy to store patient's medical data on encrypted servers instead of unencrypted public servers.   \n",
       "1                                                                                                                         Implement strict access controls and permissions for the S3 buckets to ensure that only authorized employees can access the patient data.   \n",
       "2                                                                                                  Implement a standard procedure to remind employees to regularly check for and securely dispose of any documents or patient information that is no longer needed.   \n",
       "3                                                                                                                                              Require all clinicians to use a secure virtual private network (VPN) when accessing PHI on their personal computers.   \n",
       "4                                                                                                               Implement a mandatory double-check process for all outgoing emails containing student information to prevent accidental release of student records.   \n",
       "5                                                                                                                                   Implement a reminder system for faculty and staff to regularly review and dispose of student records that are no longer needed.   \n",
       "6                                                                                                                                                               Send an email to all parents and students notifying them of the FERPA policy change from last year.   \n",
       "7                                                                                        Implement a process to obtain consent from students (if 18 or older) or parents (if students are under 18) for sharing letters of recommendation with potential employers.   \n",
       "8                                                                                         Solution: The employee could quickly reach out to the coach or school administration to offer assistance in finding a tutor or academic support for the star quarterback.   \n",
       "9                                                                                                                                                         Add a checkbox to the registration form allowing users to explicitly opt-in for marketing communications.   \n",
       "10                                                                                                                                                                                           Remove the pre-ticked checkboxes on the chip makers registration form.   \n",
       "11                                                                                                                                             Update the website's cookie policy to include a clear and easily accessible link to the full cookie policy document.   \n",
       "12                                                                                                                                                                                     Rewrite the intro of the Privacy Policy to use simpler and clearer language.   \n",
       "13                                                                                                                                                                                                               Add a cookie consent mechanism to Jubel's website.   \n",
       "14                                                                                                                                                                                  Invest in improving online advertising strategies to further increase ad sales.   \n",
       "15                                                                                                                                                             Identify and unsubscribe from any unnecessary subscriptions or services that are not being utilized.   \n",
       "16  The low hanging fruit solution that could greatly improve the business in only 10 minutes of 1 employee's time is to send targeted email promotions to AOL's existing customers, emphasizing the benefits and discounts of signing up for high-speed broadband.   \n",
       "17                                                                                                                                                             It is not clear from the provided document what the specific problem is that the business is facing.   \n",
       "18                                                                                     Low hanging fruit solution: Implement an automated email system to send monthly invoices to customers, reducing the time spent on manual invoicing and improving efficiency.   \n",
       "19                                                                                                                                     Implement a system to track and automate the restatement of accounts, reducing the time and effort required for the process.   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Low Hanging Fruit)  \n",
       "0                                            0.659562  \n",
       "1                                            0.593903  \n",
       "2                                            0.731268  \n",
       "3                                            0.774511  \n",
       "4                                            0.576572  \n",
       "5                                            0.821505  \n",
       "6                                            0.840244  \n",
       "7                                            0.673956  \n",
       "8                                            0.577028  \n",
       "9                                            0.886491  \n",
       "10                                           0.872447  \n",
       "11                                           0.667111  \n",
       "12                                           0.852667  \n",
       "13                                           0.883154  \n",
       "14                                           0.622988  \n",
       "15                                           0.531855  \n",
       "16                                           0.668286  \n",
       "17                                           0.514203  \n",
       "18                                           0.539949  \n",
       "19                                           0.705996  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response_col, trustworthiness_col = 'Cleanlab_TLM_response (Low Hanging Fruit)', 'Cleanlab_Trustworthiness_Score (Low Hanging Fruit)'\n",
    "reliabilty_data_enrichment(solution_prompt, response_col, trustworthiness_col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "46ce947a-bb90-4433-8564-f500e8b75eed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Low Hanging Fruit)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Low Hanging Fruit)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account</td>\n",
       "      <td>Add a checkbox to the registration form allowing users to explicitly opt-in for marketing communications.</td>\n",
       "      <td>0.886491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.</td>\n",
       "      <td>Add a cookie consent mechanism to Jubel's website.</td>\n",
       "      <td>0.883154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:</td>\n",
       "      <td>Remove the pre-ticked checkboxes on the chip makers registration form.</td>\n",
       "      <td>0.872447</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                   document_text  \\\n",
       "9       The McDonald's registration form does not give users an opportunity to provide their express and unambiguous consent for marketing communications; In this form, consent is assumed when a user registers for an account   \n",
       "13  Jubel's website used Google Analytics cookies, which require consent under EU law. However, the site had no cookie consent mechanism. The company attempted to argue that it did not require consent, but the DPA disagreed.   \n",
       "10                                          Although the chip makers registration form includes marketing consent checkboxes, this method of consent is not considered freely-given because the boxes are pre-ticked by default:   \n",
       "\n",
       "                                                                    Cleanlab_TLM_response (Low Hanging Fruit)  \\\n",
       "9   Add a checkbox to the registration form allowing users to explicitly opt-in for marketing communications.   \n",
       "13                                                         Add a cookie consent mechanism to Jubel's website.   \n",
       "10                                     Remove the pre-ticked checkboxes on the chip makers registration form.   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Low Hanging Fruit)  \n",
       "9                                            0.886491  \n",
       "13                                           0.883154  \n",
       "10                                           0.872447  "
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=False)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "0969c667-9f9e-443b-8e92-a080c211676c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>document_text</th>\n",
       "      <th>Cleanlab_TLM_response (Low Hanging Fruit)</th>\n",
       "      <th>Cleanlab_Trustworthiness_Score (Low Hanging Fruit)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Market concerns about the deficit has hit the greenback in recent months. On Friday, Federal Reserve chairman Mr Greenspan's speech in London ahead of the meeting of G7 finance ministers sent the dollar higher after it had earlier tumbled on the back of worse-than-expected US jobs data. \"I think the chairman's taking a much more sanguine view on the current account deficit than he's taken for some time,\" said Robert Sinche, head of currency strategy at Bank of America in New York. \"He's taking a longer-term view, laying out a set of conditions under which the current account deficit can improve this year and next.\"</td>\n",
       "      <td>There is no specific problem mentioned in the document, so it is not possible to identify a low hanging fruit solution.</td>\n",
       "      <td>0.395381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Yukos had filed for bankruptcy protection in a US court in an attempt to prevent the forced sale of its main production arm. The sale went ahead in December and Yugansk was sold to a little-known shell company which in turn was bought by Rosneft.</td>\n",
       "      <td>Implement a reminder system for employees to regularly check and update bankruptcy protection status of key business assets.</td>\n",
       "      <td>0.443234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>At a conference on developing enterprise hosted by UK finance minister Gordon Brown on Friday, he said that he was in favour of floating exchange rates because they help countries cope with economic shocks.</td>\n",
       "      <td>There is not enough information provided in the document to identify a specific problem or solution for the business.</td>\n",
       "      <td>0.443487</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      document_text  \\\n",
       "22  Market concerns about the deficit has hit the greenback in recent months. On Friday, Federal Reserve chairman Mr Greenspan's speech in London ahead of the meeting of G7 finance ministers sent the dollar higher after it had earlier tumbled on the back of worse-than-expected US jobs data. \"I think the chairman's taking a much more sanguine view on the current account deficit than he's taken for some time,\" said Robert Sinche, head of currency strategy at Bank of America in New York. \"He's taking a longer-term view, laying out a set of conditions under which the current account deficit can improve this year and next.\"    \n",
       "31                                                                                                                                                                                                                                                                                                                                                                                           Yukos had filed for bankruptcy protection in a US court in an attempt to prevent the forced sale of its main production arm. The sale went ahead in December and Yugansk was sold to a little-known shell company which in turn was bought by Rosneft.   \n",
       "90                                                                                                                                                                                                                                                                                                                                                                                                                                   At a conference on developing enterprise hosted by UK finance minister Gordon Brown on Friday, he said that he was in favour of floating exchange rates because they help countries cope with economic shocks.   \n",
       "\n",
       "                                                                                       Cleanlab_TLM_response (Low Hanging Fruit)  \\\n",
       "22       There is no specific problem mentioned in the document, so it is not possible to identify a low hanging fruit solution.   \n",
       "31  Implement a reminder system for employees to regularly check and update bankruptcy protection status of key business assets.   \n",
       "90         There is not enough information provided in the document to identify a specific problem or solution for the business.   \n",
       "\n",
       "   Cleanlab_Trustworthiness_Score (Low Hanging Fruit)  \n",
       "22                                           0.395381  \n",
       "31                                           0.443234  \n",
       "90                                           0.443487  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=trustworthiness_col, ascending=True)[['document_text', response_col, trustworthiness_col]].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2247c5d-08a0-4b99-88ec-a77046aaa3c7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}