Lab 2 - final - (batch size 16).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1158b98f",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 429,
   "id": "7a6c9b38",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "from torch import Tensor\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "from sentence_transformers import SentenceTransformer, InputExample\n",
    "from sentence_transformers.models import Transformer, Pooling\n",
    "from sentence_transformers.losses import CosineSimilarityLoss\n",
    "from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator, SimilarityFunction\n",
    "\n",
    "# For debugging\n",
    "# from scipy.stats import spearmanr, pearsonr\n",
    "# from sklearn.metrics.pairwise import paired_cosine_distances, paired_euclidean_distances, paired_manhattan_distances"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e4b25a1",
   "metadata": {},
   "source": [
    "### Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 576,
   "id": "a8a93ab7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data and training\n",
    "batch_size = 16  # {16, 32}\n",
    "\n",
    "# Training\n",
    "epochs = 1\n",
    "warmup_steps = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a64cf881",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad19052a",
   "metadata": {},
   "source": [
    "Using the official STS Benchmark dataset:  \n",
    "https://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d4f658d",
   "metadata": {},
   "source": [
    "Information about the train/dev/test datasets are available under `data/stsbenchmark/readme.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6962ad33",
   "metadata": {},
   "source": [
    "## Data loading and cleanup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b899ad",
   "metadata": {},
   "source": [
    "There seem to be some rows with 7 columns and some with 9 columns (proabably due to additional licensing information), and so we'll try to make sure we don't miss the 3 important columns that we need: Score, Sentence 1, and Sentence 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 582,
   "id": "1fdb9764",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_csv = 'data/stsbenchmark/sts-train.csv'\n",
    "dev_csv   = 'data/stsbenchmark/sts-dev.csv'\n",
    "test_csv  = 'data/stsbenchmark/sts-test.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5d5416d",
   "metadata": {},
   "source": [
    "### Training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 515,
   "id": "9fb34c81",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(train_csv) as f:\n",
    "    lines = f.readlines()\n",
    "    lines = [l.strip() for l in lines]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 516,
   "id": "334a0ebb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5749"
      ]
     },
     "execution_count": 516,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Should be 5749\n",
    "len(lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 517,
   "id": "cc770f41",
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = [l.split('\\t') for l in lines]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 518,
   "id": "73283543",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{7: 5552, 9: 197}"
      ]
     },
     "execution_count": 518,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row_type_counts = {}\n",
    "\n",
    "for row in rows:\n",
    "    if len(row) not in row_type_counts.keys():\n",
    "        row_type_counts[len(row)] = 1\n",
    "    else:\n",
    "        row_type_counts[len(row)] += 1\n",
    "\n",
    "row_type_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87b77290",
   "metadata": {},
   "source": [
    "Assuming that the data is clean enough and the rows with 7 columns are all correctly filled, we will just quickly check the relevant columns of those with 9 fields, just to make sure our data is clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 519,
   "id": "456cc344",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4  |  Driver bac  |  Driver bac\n",
      "5  |  Spain Prin  |  Spain prin\n",
      "5  |  Senate con  |  Senate app\n",
      "5  |  U.N. right  |  UN Rights \n",
      "5  |  US Senate   |  Senate con\n",
      "5  |  Syrian Reb  |  Syrian reb\n",
      "3  |  Mayawati d  |  Mayawati d\n",
      "4  |  Uganda's p  |  Uganda's p\n",
      "5  |  Rocks, Tea  |  Rocks, tea\n",
      "5  |  Boston bom  |  Boston bom\n",
      "5  |  Six dead i  |  6 killed i\n",
      "5  |  China army  |  China army\n",
      "4  |  Silvio Ber  |  Silvio Ber\n",
      "5  |  Two killed  |  Two killed\n",
      "5  |  Russia war  |  Russia war\n",
      "4  |  Hosni Muba  |  Mubarak re\n",
      "4  |  Egypt prot  |  Egypt prot\n",
      "5  |  Couple mar  |  Couple get\n",
      "4  |  Qatar's em  |  Qatari emi\n",
      "3  |  Philippine  |  Philippine\n",
      "4  |  Egypt brac  |  Egypt brac\n",
      "4  |  Red Sox Be  |  Red Sox be\n",
      "4  |  China land  |  China land\n",
      "4  |  Ukrainian   |  Ukraine's \n",
      "5  |  Venezuela   |  Venezuela \n",
      "5  |  North Kore  |  North Kore\n",
      "4  |  Captain of  |  Captain of\n",
      "3  |  Cars, driv  |  Cars plung\n",
      "5  |  Death toll  |  Death toll\n",
      "4  |  Communist   |  Communist \n",
      "3  |  Egypt's Mo  |  Egypt's Mo\n",
      "5  |  Iran Nucle  |  Iran nucle\n",
      "3  |  Thai army   |  Thailand's\n",
      "4  |  Woman dies  |  Woman dies\n",
      "4  |  Nobel auth  |  Nobel Priz\n",
      "5  |  Over 100 d  |  At least 1\n",
      "4  |  First post  |  India's fi\n",
      "3  |  Israel agr  |  Israel agr\n",
      "3  |  Quake of 6  |  Earthquake\n",
      "4  |  British st  |  British st\n",
      "5  |  What makes  |  What makes\n",
      "4  |  White diam  |  White diam\n",
      "3  |  Chinese sa  |  Malaysia: \n",
      "5  |  Hamas Urge  |  Hamas call\n",
      "3  |  U.S. and R  |  China welc\n",
      "4  |  PSNI faile  |  13:23 PSNI\n",
      "5  |  Stocks edg  |  Stocks edg\n",
      "4  |  Police hel  |  Police hel\n",
      "4  |  US ups pre  |  US, UN up \n",
      "4  |  Australian  |  Australian\n",
      "1  |  Mandela's   |  Mandela's \n",
      "4  |  Philippe b  |  Philippe a\n",
      "4  |  India anti  |  India's An\n",
      "5  |  Spain appr  |  Spanish go\n",
      "2  |  Obama orde  |  UN urges S\n",
      "5  |  German amb  |  Shots fire\n",
      "2  |  Ukraine's   |  Ukraine's \n",
      "3  |  Oklahoma e  |  Okla. inma\n",
      "5  |  Napolitano  |  Italian Pr\n",
      "4  |  UN court c  |  UN War Cri\n",
      "4  |  Saudi man   |  Saudi Man \n",
      "5  |  North Kore  |  North Kore\n",
      "3  |  Thai prote  |  Thai prote\n",
      "3  |  Egypt impo  |  Egypt anno\n",
      "4  |  Rallies de  |  Across US,\n",
      "3  |  UN warns o  |  UNICEF cal\n",
      "4  |  Hagel, Dem  |  Hagel lame\n",
      "3  |  Sony to se  |  Sony to ax\n",
      "2  |  Man kills   |  Police: Gu\n",
      "2  |  N. Korea D  |  North Kore\n",
      "1  |  Search lea  |  Searchers \n",
      "1  |  Mandela re  |  Mandela's \n",
      "4  |  Ukrainian   |  Ukraine ri\n",
      "2  |  Obama mull  |  Obama urge\n",
      "4  |  Syria's PM  |  Syrian pri\n",
      "3  |  Jordan ele  |  Jordan ele\n",
      "0  |  At least 1  |  More than \n",
      "3  |  Police sur  |  Peaceful a\n",
      "1  |  George Zim  |  George Zim\n",
      "3  |  Ten rescue  |  Six rescue\n",
      "5  |  Mandela sp  |  Mandela in\n",
      "2  |  One killed  |  Egypt: Pro\n",
      "2  |  Saudi Wome  |  Saudi wome\n",
      "1  |  Russian pa  |  President \n",
      "2  |  Nelson Man  |  Nelson Man\n",
      "1  |  World stoc  |  Fed expect\n",
      "1  |  No winners  |  The impact\n",
      "3  |  World Cup   |  Worker die\n",
      "1  |  Tata Nano   |  How does w\n",
      "3  |  Greece rea  |  Debt inspe\n",
      "5  |  Murray end  |  Murray win\n",
      "0  |  Malaysia A  |  US: Russia\n",
      "2  |  US to Dete  |  What we kn\n",
      "0  |  The covert  |  New major \n",
      "2  |  Russia's P  |  Kremlin: P\n",
      "3  |  Senior CPC  |  North Kore\n",
      "1  |  Turkey unr  |  Turkish PM\n",
      "5  |  Pope calls  |  Pope urges\n",
      "2  |  Singapore   |  Singapore \n",
      "5  |  Who Is Jih  |  Who was 'J\n",
      "2  |  Headlines   |  Headlines \n",
      "4  |  US Attorne  |  US Attorne\n",
      "5  |  U.S., EU W  |  U.S., EU B\n",
      "1  |  Top Asian   |  Top Asian \n",
      "2  |  How the Do  |  How the Do\n",
      "4  |  Hajj to go  |  Hajj to go\n",
      "3  |  US House R  |  US House R\n",
      "3  |  Top Asian   |  Top Asian \n",
      "1  |  Singapore   |  Singapore \n",
      "1  |  China trea  |  China trea\n",
      "5  |  US Attorne  |  Attorney G\n",
      "3  |  Headlines   |  Headlines \n",
      "4  |  China on h  |  China issu\n",
      "3  |  Saudi-led   |  Saudi airs\n",
      "3  |  Mali hunts  |  Militants \n",
      "1  |  7.3 magnit  |  6.6-magnit\n",
      "4  |  Tokyo shar  |  Tokyo shar\n",
      "1  |  Iran's par  |  Iran hardl\n",
      "5  |  Hawaii pre  |  Hawaii bra\n",
      "2  |  Iraq bombi  |  Iraq suici\n",
      "3  |  Sierra Leo  |  Sierra Leo\n",
      "4  |  Tokyo shar  |  Tokyo shar\n",
      "0  |  4 killed,   |  Two killed\n",
      "3  |  More than   |  Over 2,000\n",
      "4  |  Lewis Hami  |  Lewis Hami\n",
      "2  |  UN Securit  |  UN Securit\n",
      "1  |  Israeli fo  |  Israeli se\n",
      "0  |  4 dead, 3   |  2 dead, 8 \n",
      "2  |  Markets Li  |  Markets Li\n",
      "2  |  Palestinia  |  Palestinia\n",
      "4  |  Suspected   |  Suspected \n",
      "2  |  The end of  |  UK 'concer\n",
      "2  |  Russian wa  |  Russian ro\n",
      "1  |  CCTV of gu  |  19 killed \n",
      "0  |  Live Blog:  |  Live Blog:\n",
      "2  |  Nigeria Vo  |  Guineans G\n",
      "2  |  Prince Wil  |  Prince Wil\n",
      "2  |  Micronesia  |  Vice Presi\n",
      "3  |  Islamic St  |  Islamic St\n",
      "2  |  Iran ends   |  Iran, Taji\n",
      "1  |  Teenager t  |  Texas hosp\n",
      "1  |  Texas sues  |  Turkey 'ex\n",
      "2  |  New US San  |  China disa\n",
      "0  |  5 killed i  |  38 IS mili\n",
      "2  |  Deadly Bla  |  12 Dead, D\n",
      "3  |  Greece bai  |  Greece con\n",
      "1  |  Rooney hel  |  West Ham b\n",
      "1  |  Hard-line   |  Israeli le\n",
      "2  |  Backstage   |  Mixed Prog\n",
      "1  |  Egypt: IPI  |  Egypt sent\n",
      "1  |  occupied P  |  occupied P\n",
      "0  |  Libya: UN   |  India's to\n",
      "2  |  Taiwan coa  |  Chinese mi\n",
      "1  |  Burundi po  |  Montenegri\n",
      "3  |  Nepal eart  |  Death toll\n",
      "1  |  Ashya's Pa  |  Ashya King\n",
      "1  |  Nigeria ad  |  US ramps u\n",
      "1  |  Thousands   |  Thousands \n",
      "0  |  Kranjcar l  |  Harry Kane\n",
      "0  |  Oscar Pist  |  'Bookkeepe\n",
      "2  |  Pro-Palest  |  Pro-Palest\n",
      "1  |  Benzema fa  |  Menendez f\n",
      "2  |  All 224 de  |  Heat flash\n",
      "0  |  Govt. to h  |  Cameron to\n",
      "0  |  Li Na to a  |  US Senator\n",
      "1  |  Gunners fi  |  EPL clubs \n",
      "1  |  Iranian Pr  |  Iran arres\n",
      "1  |  Boko Haram  |  Press awas\n",
      "1  |  South Kore  |  South Kore\n",
      "0  |  Plane Cras  |  Tour De Fr\n",
      "0  |  Obama quer  |  Spanish bu\n",
      "0  |  China's ge  |  China keep\n",
      "0  |  Hezbollah   |  Belarus ti\n",
      "1  |  South Kore  |  South Kore\n",
      "0  |  Obama call  |  Obama vows\n",
      "1  |  U.S. milit  |  Observator\n",
      "0  |  Who abando  |  Police aba\n",
      "2  |  North Kore  |  N Korea hi\n",
      "2  |  Hong Kong   |  Hong Kong \n",
      "0  |  Ebola UK:   |  UK says in\n",
      "1  |  UK Leader   |  Why Cyprus\n",
      "0  |  N. Korea a  |  Johor crow\n",
      "0  |  Iran, a la  |  Iran reite\n",
      "0  |  Taiwan's p  |  Poland pre\n",
      "0  |  Two Palest  |  Palestinia\n",
      "0  |  Thomas Coo  |  University\n",
      "0  |  Narendra M  |  PM Narendr\n",
      "1  |  UN chief w  |  UN chief c\n",
      "0  |  Pakistan e  |  Pakistan p\n",
      "0  |  Two Austra  |  Permalink \n",
      "2  |  France shu  |  Security t\n",
      "2  |  Russian pl  |  Pilot kill\n",
      "0  |  Severe Gal  |  Merkel ple\n",
      "0  |  Dozens of   |  Egyptian b\n",
      "0  |  President   |  President \n",
      "0  |  China, Ind  |  China Scra\n",
      "0  |  Putin spok  |  The Latest\n"
     ]
    }
   ],
   "source": [
    "for row in rows:\n",
    "    if len(row) == 9:\n",
    "        print(f'{row[4][:10]}  |  {row[5][:10]}  |  {row[6][:10]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e3eadd",
   "metadata": {},
   "source": [
    "Everything seems OK. Let's just check the first 20 rows with 7 columns also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 520,
   "id": "1aca36aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.000  |  A plane is  |  An air pla\n",
      "3.800  |  A man is p  |  A man is p\n",
      "3.800  |  A man is s  |  A man is s\n",
      "2.600  |  Three men   |  Two men ar\n",
      "4.250  |  A man is p  |  A man seat\n",
      "4.250  |  Some men a  |  Two men ar\n",
      "0.500  |  A man is s  |  A man is s\n",
      "1.600  |  The man is  |  The man is\n",
      "2.200  |  A man is p  |  A woman is\n",
      "5.000  |  A person i  |  A person t\n",
      "4.200  |  The man hi  |  The man sp\n",
      "4.600  |  A woman pi  |  A woman pi\n",
      "3.867  |  A man is p  |  A man is p\n",
      "4.667  |  A person i  |  Someone is\n",
      "1.667  |  A man is r  |  A panda do\n",
      "3.750  |  A dog is t  |  A dog is t\n",
      "5.000  |  The polar   |  A polar be\n",
      "0.500  |  A woman is  |  A woman is\n",
      "3.800  |  A cat is r  |  A cat is r\n",
      "5.000  |  The man is  |  A man is r\n"
     ]
    }
   ],
   "source": [
    "for row in rows[:20]:\n",
    "    if len(row) == 7:\n",
    "        print(f'{row[4][:10]}  |  {row[5][:10]}  |  {row[6][:10]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b993b1",
   "metadata": {},
   "source": [
    "All good!\n",
    "\n",
    "Making SentenceTransformer-specific DataLoaders out of them then."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 521,
   "id": "02263db9",
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = []\n",
    "\n",
    "for row in rows:\n",
    "    examples.append(\n",
    "        InputExample(\n",
    "            texts=[row[5].strip(), row[6].strip()],\n",
    "            label=float(row[4])))\n",
    "\n",
    "train_dl = DataLoader(examples, shuffle=True, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 522,
   "id": "f8bbae5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5749"
      ]
     },
     "execution_count": 522,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Should be 5749\n",
    "len(examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce72cdae",
   "metadata": {},
   "source": [
    "### Development set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 539,
   "id": "6bf16f2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(dev_csv) as f:\n",
    "    lines = f.readlines()\n",
    "    lines = [l.strip() for l in lines]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 540,
   "id": "70d83455",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1500"
      ]
     },
     "execution_count": 540,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Should be 1500\n",
    "len(lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 541,
   "id": "b33af8e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = [l.split('\\t') for l in lines]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 542,
   "id": "4b9ec600",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{7: 1478, 9: 22}"
      ]
     },
     "execution_count": 542,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row_type_counts = {}\n",
    "\n",
    "for row in rows:\n",
    "    if len(row) not in row_type_counts.keys():\n",
    "        row_type_counts[len(row)] = 1\n",
    "    else:\n",
    "        row_type_counts[len(row)] += 1\n",
    "\n",
    "row_type_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ea4144d",
   "metadata": {},
   "source": [
    "Assuming that the data is clean enough and the rows with 7 columns are all correctly filled, we will just quickly check the relevant columns of those with 9 fields, just to make sure our data is clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 543,
   "id": "a0809c5d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5  |  US Supreme  |  High court\n",
      "4  |  Euro crisi  |  Eurozone d\n",
      "4  |  Treasury p  |  Treasury p\n",
      "3  |  EU Ministe  |  EU Ministe\n",
      "5  |  Chinese lu  |  China land\n",
      "2  |  Boy, 14, a  |  Two teenag\n",
      "3  |  Protests a  |  Zimmerman \n",
      "3  |  Maldives b  |  Crisis-hit\n",
      "1  |  Obama, Hol  |  Hollande a\n",
      "5  |  Iran, IAEA  |  Iran, IAEA\n",
      "2  |  Bombings k  |  Bombings k\n",
      "2  |  10 Things   |  10 Things \n",
      "2  |  New UN pea  |  UN takes o\n",
      "5  |  Oil falls   |  Oil prices\n",
      "3  |  Israeli fo  |  Israeli fo\n",
      "3  |  Israeli po  |  Israel Pol\n",
      "0  |  3 killed,   |  Five kille\n",
      "2  |  Scientists  |  Has Nasa d\n",
      "0  |  Pranab str  |  WTO: India\n",
      "2  |  Volkswagen  |  Volkswagen\n",
      "0  |  Obama is r  |  Obama wait\n",
      "0  |  New video   |  New York p\n"
     ]
    }
   ],
   "source": [
    "for row in rows:\n",
    "    if len(row) == 9:\n",
    "        print(f'{row[4][:10]}  |  {row[5][:10]}  |  {row[6][:10]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d015e9",
   "metadata": {},
   "source": [
    "Everything seems OK. Let's just check the first 20 rows with 7 columns also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 544,
   "id": "c58dd4b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.000  |  A man with  |  A man wear\n",
      "4.750  |  A young ch  |  A child is\n",
      "5.000  |  A man is f  |  The man is\n",
      "2.400  |  A woman is  |  A man is p\n",
      "2.750  |  A woman is  |  A man is p\n",
      "2.615  |  A woman is  |  A man is c\n",
      "5.000  |  A man is e  |  The man is\n",
      "2.333  |  A woman is  |  A woman is\n",
      "3.750  |  Three men   |  Three men \n",
      "5.000  |  A woman pe  |  A woman is\n",
      "3.200  |  People are  |  Men are pl\n",
      "1.583  |  A man is p  |  A man is p\n",
      "5.000  |  The cougar  |  A cougar i\n",
      "5.000  |  The man cu  |  A man chop\n",
      "4.909  |  The man is  |  A man is p\n",
      "0.800  |  A man is f  |  A woman is\n",
      "2.400  |  The girl s  |  The lady s\n",
      "5.000  |  A man is c  |  A man clim\n",
      "4.000  |  Kittens ar  |  Kittens ar\n",
      "0.636  |  A man is s  |  A man is s\n",
      "3.000  |  A woman is  |  A woman is\n",
      "1.714  |  A man is p  |  A man is p\n",
      "3.200  |  An animal   |  An animal \n",
      "2.167  |  A man is p  |  A man is p\n",
      "1.000  |  A man is p  |  A man is p\n",
      "1.917  |  A girl is   |  A girl is \n",
      "4.250  |  A man is l  |  A man is l\n",
      "3.000  |  An animal   |  A slow lor\n",
      "1.000  |  A woman is  |  A woman is\n",
      "0.600  |  A man is c  |  A woman is\n",
      "2.600  |  The lady c  |  The man is\n",
      "5.000  |  A band is   |  A band is \n",
      "4.600  |  Elephants   |  A herd of \n",
      "5.000  |  A man is p  |  A man play\n",
      "4.800  |  A man sitt  |  A man sitt\n",
      "3.800  |  A puppy pl  |  The dog is\n",
      "5.000  |  A man play  |  A man is p\n",
      "5.000  |  A man play  |  A man is p\n",
      "4.200  |  A man is e  |  A man is e\n",
      "1.400  |  A man is p  |  A man is p\n",
      "3.600  |  A toddler   |  A little g\n",
      "2.800  |  A plane is  |  A animated\n",
      "1.600  |  A panda is  |  A man is c\n",
      "3.000  |  Kids are d  |  Some peopl\n",
      "1.400  |  A man play  |  A man play\n",
      "0.250  |  A woman is  |  A woman is\n",
      "0.250  |  A woman is  |  A man is p\n",
      "0.000  |  The man is  |  A woman is\n",
      "4.000  |  A man is d  |  Someone is\n",
      "4.500  |  A man is f  |  A man is a\n",
      "0.500  |  Two little  |  A little g\n",
      "3.800  |  Someone is  |  Someone is\n",
      "4.800  |  The woman   |  The lady b\n",
      "5.000  |  A woman pu  |  A woman is\n",
      "0.250  |  A man is t  |  A man is s\n",
      "1.200  |  A man is p  |  A man is p\n",
      "0.600  |  A man is p  |  A man is e\n",
      "0.800  |  A man is j  |  A man is c\n",
      "3.800  |  A man is p  |  A guy is p\n",
      "0.000  |  A young ma  |  A woman is\n",
      "3.500  |  A woman is  |  A woman is\n",
      "4.500  |  Someone is  |  Someone is\n",
      "2.800  |  A slow lor  |  A animal i\n",
      "3.800  |  A little b  |  A boy is v\n",
      "3.800  |  A woman is  |  A woman is\n",
      "0.000  |  A man is m  |  A woman is\n",
      "4.000  |  A man play  |  A person i\n",
      "4.250  |  A rabbit i  |  A bunny is\n",
      "2.812  |  A plane ri  |  An airplan\n",
      "4.250  |  A man shoo  |  A man with\n",
      "3.000  |  A man play  |  The man sa\n",
      "1.000  |  A guy is p  |  A man is p\n",
      "3.750  |  One man is  |  A man brea\n",
      "0.000  |  A man is t  |  A man is w\n",
      "0.400  |  The woman   |  A woman is\n",
      "4.000  |  A chef is   |  A chef pre\n",
      "2.800  |  The man ta  |  A teenager\n",
      "3.750  |  A turtle w  |  A large tu\n",
      "1.154  |  The lady c  |  The lady s\n",
      "2.750  |  Five kitte  |  Kittens ar\n",
      "2.800  |  A woman is  |  A woman is\n",
      "0.000  |  A man play  |  A woman is\n",
      "0.000  |  A man is p  |  A dog is b\n",
      "3.400  |  A baby rhi  |  A baby rhi\n",
      "5.000  |  The man us  |  A man slic\n",
      "0.800  |  A man is c  |  A man is p\n",
      "2.250  |  A person i  |  A woman is\n",
      "2.750  |  A man is p  |  A man turn\n",
      "4.500  |  A girl is   |  The girl t\n",
      "2.600  |  Batman and  |  A helicopt\n",
      "3.800  |  A woman po  |  An older w\n",
      "2.800  |  A woman is  |  A woman is\n",
      "1.600  |  The lady p  |  The lady f\n",
      "0.000  |  A woman is  |  A man is s\n",
      "1.333  |  The boy is  |  A band is \n",
      "2.600  |  The cook i  |  A woman is\n",
      "0.600  |  A woman is  |  A dog is r\n",
      "0.000  |  A woman is  |  The woman \n",
      "0.000  |  A cat is p  |  A man is s\n",
      "3.000  |  A man and   |  A couple a\n",
      "0.000  |  A woman is  |  A man is t\n",
      "0.200  |  A man is s  |  A man is g\n",
      "3.200  |  A girl is   |  A girl pla\n",
      "2.000  |  A small ba  |  A boy sits\n",
      "0.000  |  A woman is  |  A man is d\n",
      "3.000  |  A man and   |  The man an\n",
      "3.091  |  A child is  |  A boy is p\n",
      "2.750  |  Men are ju  |  Men jump i\n",
      "1.200  |  A woman is  |  A woman is\n",
      "0.500  |  People are  |  The cricke\n",
      "0.250  |  A person i  |  A woman is\n",
      "0.000  |  A woman is  |  A man brea\n",
      "0.000  |  A ballet d  |  The beaver\n",
      "0.500  |  A man shoo  |  A man is p\n",
      "0.000  |  The elepha  |  Two women \n",
      "0.600  |  The man cu  |  A man is d\n",
      "4.000  |  A woman pu  |  The woman \n",
      "3.400  |  A woman an  |  A woman dr\n",
      "0.000  |  A girl is   |  A onion is\n",
      "0.000  |  A cat is o  |  A man is e\n",
      "1.000  |  The lady s  |  A woman is\n",
      "0.000  |  A cow is e  |  A dog is p\n",
      "2.400  |  The man st  |  The man is\n",
      "0.400  |  A man jump  |  A man is t\n",
      "1.800  |  A woman is  |  A woman cu\n",
      "2.500  |  Someone is  |  A woman st\n",
      "0.083  |  A man is c  |  A woman pl\n",
      "0.000  |  A man is b  |  A slow lor\n",
      "1.000  |  A woman cu  |  A woman is\n",
      "4.500  |  The lady c  |  A woman is\n",
      "0.000  |  A man play  |  A woman an\n",
      "3.600  |  A person i  |  A person i\n",
      "5.000  |  A woman is  |  A woman is\n",
      "5.000  |  A small do  |  A small do\n",
      "5.000  |  The man is  |  A man is s\n",
      "5.000  |  The man is  |  A man is s\n",
      "3.800  |  A person i  |  A man is p\n",
      "5.000  |  A woman is  |  A woman pu\n",
      "2.500  |  A woman is  |  A woman is\n",
      "3.824  |  Women are   |  Two women \n",
      "1.600  |  A man is p  |  A woman is\n",
      "5.000  |  A man dive  |  A man dive\n",
      "5.000  |  The man is  |  A man is s\n",
      "2.375  |  A man is c  |  A man is c\n",
      "5.000  |  A man is s  |  A man slic\n",
      "1.250  |  A man is d  |  A man is s\n",
      "1.250  |  A man is r  |  A man is s\n",
      "3.000  |  The man is  |  A man is t\n",
      "2.167  |  A man is p  |  A man is p\n",
      "1.400  |  A man is r  |  A woman is\n",
      "4.000  |  A woman is  |  A person i\n",
      "3.600  |  A man is e  |  A man is e\n",
      "4.857  |  A man is p  |  A man play\n",
      "0.400  |  The man is  |  The man is\n",
      "0.200  |  A woman is  |  A woman is\n",
      "1.300  |  Three men   |  Women are \n",
      "1.200  |  A man is s  |  A man is d\n",
      "1.200  |  A man is d  |  A man is s\n",
      "4.400  |  A person i  |  A person i\n",
      "1.800  |  A man is p  |  A man is p\n",
      "1.400  |  A man is p  |  A woman is\n",
      "0.400  |  A boy is r  |  A monkey i\n",
      "4.400  |  The puppy   |  A puppy is\n",
      "3.250  |  A woman is  |  The woman \n",
      "0.778  |  A person i  |  A person i\n",
      "2.250  |  A man is p  |  The girl i\n",
      "2.750  |  A lion cub  |  A lion tri\n",
      "2.000  |  A man is t  |  A guy is t\n",
      "0.833  |  A man is p  |  A man is c\n",
      "3.750  |  Someone is  |  The man is\n",
      "1.800  |  A woman is  |  A woman is\n",
      "4.333  |  The man is  |  A man is s\n",
      "4.214  |  A target i  |  The target\n",
      "0.000  |  A woman is  |  A man is p\n",
      "3.400  |  A woman is  |  A person i\n",
      "4.200  |  A man puts  |  A man is s\n",
      "0.800  |  A man is s  |  A man is s\n",
      "4.200  |  A person i  |  A person i\n",
      "2.750  |  A person d  |  A man toss\n",
      "2.200  |  The man is  |  A man is p\n",
      "1.200  |  A woman is  |  A woman is\n",
      "1.000  |  A woman is  |  A man is p\n",
      "1.533  |  A man is r  |  A woman is\n",
      "0.200  |  A man is p  |  A man is d\n",
      "3.000  |  A tiger is  |  A tiger wa\n",
      "3.800  |  A person p  |  A person i\n",
      "0.750  |  A woman is  |  A woman is\n",
      "3.000  |  A parrot i  |  A parrot i\n",
      "0.400  |  A man is s  |  A man is d\n",
      "3.400  |  A woman is  |  A woman li\n",
      "3.800  |  A child ru  |  A young bo\n",
      "1.400  |  A man is p  |  A man is p\n",
      "3.400  |  A guy is r  |  A man is r\n",
      "3.200  |  A dog is s  |  The dog to\n",
      "2.333  |  A person u  |  A person i\n",
      "0.800  |  A man is p  |  A man is g\n",
      "0.000  |  A woman is  |  A man is s\n",
      "3.000  |  A puppy is  |  The puppy \n",
      "3.750  |  A person i  |  A person f\n",
      "0.000  |  The boy is  |  The woman \n",
      "0.000  |  A man is c  |  A woman is\n",
      "0.000  |  A woman is  |  A man is a\n",
      "0.500  |  A man is p  |  A woman is\n",
      "0.500  |  A woman is  |  The woman \n",
      "0.400  |  A man is r  |  A man is t\n",
      "0.750  |  A man is s  |  A man seas\n",
      "2.500  |  Someone is  |  A woman is\n",
      "1.800  |  A man is s  |  A man is o\n",
      "2.500  |  A monkey i  |  A monkey p\n",
      "0.500  |  A man shap  |  A man carr\n",
      "0.400  |  A woman is  |  A woman is\n",
      "1.200  |  A woman is  |  A woman is\n",
      "2.750  |  A man is s  |  A man is b\n",
      "0.400  |  A woman is  |  A woman is\n",
      "0.600  |  A woman is  |  The woman \n",
      "0.800  |  A woman is  |  A woman is\n",
      "3.000  |  A baby eat  |  A baby is \n",
      "0.400  |  A man is s  |  A man is s\n",
      "3.200  |  A man is p  |  A man is p\n",
      "1.250  |  A boy is p  |  A man is s\n",
      "5.000  |  The person  |  A person m\n",
      "0.200  |  A woman is  |  A woman is\n",
      "3.000  |  A person i  |  A man is f\n",
      "1.400  |  A woman is  |  A boy is f\n",
      "3.800  |  A person i  |  A woman is\n",
      "2.750  |  The dog co  |  A dog is t\n",
      "0.000  |  A woman is  |  A man is d\n",
      "2.800  |  The lady m  |  A chef is \n",
      "0.000  |  Someone is  |  A man is p\n",
      "0.000  |  A woman is  |  A man is p\n",
      "2.600  |  The animal  |  A slow lor\n",
      "0.100  |  A baby is   |  A cat is w\n",
      "0.000  |  A tiger cl  |  A person i\n",
      "2.000  |  A woman is  |  Two women \n",
      "3.750  |  The flag w  |  A flag is \n",
      "3.100  |  The baby l  |  A baby is \n",
      "2.692  |  A dog knoc  |  A dog push\n",
      "4.429  |  The bird i  |  A bird is \n",
      "5.000  |  A man play  |  A man is s\n",
      "0.000  |  Women are   |  Two men ar\n",
      "0.800  |  The man is  |  A man is w\n",
      "1.333  |  A tiger is  |  A tiger at\n",
      "0.000  |  A cat is p  |  A man is p\n",
      "0.600  |  A cat retr  |  The cat tr\n",
      "0.000  |  A chef is   |  A person i\n",
      "0.000  |  A woman is  |  A young ma\n",
      "0.000  |  A man is p  |  A woman is\n",
      "3.400  |  Someone dr  |  A person d\n",
      "2.200  |  The girl i  |  A woman is\n",
      "0.000  |  The man is  |  A boy is p\n",
      "3.6  |  A cat stan  |  A black an\n",
      "0  |  The lamb i  |  A small bi\n",
      "1.4  |  A bull sta  |  A brown ho\n",
      "2.2  |  The black   |  A black co\n",
      "3.8  |  A group of  |  Four peopl\n",
      "2.6  |  A group of  |  A group of\n",
      "0.4  |  Older man   |  A sheep in\n",
      "4  |  The cows f  |  The brown \n",
      "2.25  |  A small re  |  A red and \n",
      "3.2  |  A black an  |  Black and \n",
      "0  |  A bag of c  |  Calico cat\n",
      "0.4  |  Two horses  |  A black an\n",
      "3.8  |  Red double  |  A red two \n",
      "0.8  |  The lamb i  |  A cat look\n",
      "4.2  |  A red doub  |  A double d\n",
      "3.6  |  A jockey r  |  A jockey r\n",
      "2  |  The udders  |  The black \n",
      "4.2  |  A woman po  |  a woman is\n",
      "3.4  |  One Asian   |  Two women \n",
      "3  |  A black tr  |  A mostly b\n",
      "3.4  |  The brown   |  A tan dog \n",
      "3.6  |  The black   |  The black \n",
      "3  |  A smiling   |  Two smilin\n",
      "2.4  |  Two airpla  |  A plane in\n",
      "2  |  A coal tra  |  An old tra\n",
      "4  |  A cat look  |  A grey, bl\n",
      "0  |  World news  |  A man dres\n",
      "4  |  Two women   |  Two smilin\n",
      "4.2  |  A red and   |  Red and wh\n",
      "0.8  |  A black an  |  A black an\n",
      "2.2  |  A bicycle   |  A pink bic\n",
      "3.8  |  Red double  |  A red doub\n",
      "3.2  |  A bus driv  |  A green bu\n",
      "4.4  |  An elderly  |  An older A\n",
      "0  |  Brown and   |  A red and \n",
      "1.4  |  People rid  |  People rid\n",
      "2.6  |  Large silv  |  The silver\n",
      "0.4  |  Two people  |  Two eagles\n",
      "0  |  A scantily  |  A yellow a\n",
      "3.2  |  Several te  |  Three tele\n",
      "4.6  |  Sheep are   |  Sheep are \n",
      "0  |  The small   |  The back o\n",
      "0.6  |  Children l  |  Child milk\n",
      "1.2  |  A black-an  |  A black an\n",
      "3.8  |  the back o  |  The back o\n",
      "3.2  |  Living roo  |  Living roo\n",
      "2.6  |  Two women   |  Two women \n",
      "5  |  A doubly d  |  A red doub\n",
      "0  |  The black   |  The back o\n",
      "3.2  |  A Das Air   |  A passenge\n",
      "4  |  A cat slee  |  A black an\n",
      "1.2  |  A close-up  |  A close-up\n",
      "4.4  |  A kitchen   |  A kitchen \n",
      "3.4  |  A motorcyc  |  A motorcyc\n",
      "4.4  |  Two boys s  |  Two people\n",
      "0.8  |  Two white   |  The birds \n",
      "4.6  |  Large whit  |  Large crui\n",
      "4.2  |  Two bald e  |  Two eagles\n",
      "0  |  A backpack  |  A social g\n",
      "3.2  |  The bird i  |  A red and \n",
      "0  |  A group of  |  A group of\n",
      "1.4  |  Black dome  |  A grey cat\n",
      "3.2  |  Four child  |  Children l\n",
      "1.8  |  A black an  |  A black an\n",
      "1.8  |  Two men on  |  A wooden y\n",
      "0  |  A man doin  |  A woman po\n",
      "3.8  |  A woman eq  |  The woman \n",
      "3.8  |  Train goin  |  A black st\n",
      "4  |  A white pl  |  A white Sw\n",
      "0  |  The white   |  A woman in\n",
      "3.2  |  A group of  |  A group of\n",
      "1.6  |  Young baby  |  A woman wi\n",
      "3.6  |  Two cats a  |  Two baby k\n",
      "1  |  A group of  |  A group of\n",
      "3.6  |  A man jump  |  A man on a\n",
      "2  |  A black an  |  A bottle o\n",
      "3.6  |  A black an  |  Black and \n",
      "4  |  A black an  |  A large bl\n",
      "3.6  |  A long-hai  |  A man with\n",
      "3.2  |  An AeroMex  |  A silver A\n",
      "2.8  |  A red bird  |  Five birds\n",
      "2.4  |  An older A  |  A group of\n",
      "3.6  |  Domestic c  |  A white ca\n",
      "1.8  |  Two cows g  |  A brown ho\n",
      "0.2  |  A person i  |  A person w\n",
      "3.4  |  A woman an  |  A women an\n",
      "1.2  |  A white bi  |  A white ca\n",
      "1.4  |  A woman at  |  Four women\n",
      "2.6  |  A laptop c  |  A laptop a\n",
      "2.4  |  A white ca  |  A white ca\n",
      "0  |  A small ye  |  A small gi\n",
      "4  |  Two women   |  Two women \n",
      "3  |  A man with  |  A man talk\n",
      "4  |  A close-up  |  A close-up\n",
      "4  |  An office   |  Small offi\n",
      "2  |  A woman le  |  A man in a\n",
      "4  |  A man on a  |  Motorcycli\n",
      "2.2  |  Left side   |  A close-up\n",
      "2.2  |  Blue couch  |  A leather \n",
      "3.4  |  A white bi  |  Old style \n",
      "3.2  |  A bird eat  |  A yellow b\n",
      "4  |  Black bird  |  Black bird\n",
      "3.2  |  Dark color  |  A grey dog\n",
      "3.8  |  A young bl  |  A young gi\n",
      "3  |  Two brown   |  Brown hors\n",
      "4  |  Boy lying   |  A small bo\n",
      "2.2  |  A woman is  |  A man is l\n",
      "3.4  |  A grey hou  |  House with\n",
      "3.8  |  A woman wi  |  A woman le\n",
      "0  |  Depressed   |  Older woma\n",
      "0  |  An elderly  |  The boats \n",
      "4.8  |  The brown   |  Brown and \n",
      "3  |  A young wo  |  TThe girl \n",
      "0.8  |  Black and   |  A black an\n",
      "4.2  |  A group of  |  A group of\n",
      "4.2  |  Two cats s  |  Two cats s\n",
      "0  |  A man sitt  |  A women ho\n",
      "4.2  |  A blue sco  |  A blue mop\n",
      "4.60  |  Two gray d  |  Two grey d\n",
      "5.00  |  Two dogs a  |  Two dogs r\n",
      "0.00  |  The couple  |  The boy ho\n",
      "3.20  |  German She  |  A dog jump\n",
      "2.60  |  A group of  |  A woman ad\n",
      "2.20  |  A dog runs  |  A wet dog \n",
      "4.60  |  The man an  |  A man and \n",
      "5.00  |  A man eati  |  A man sits\n",
      "4.60  |  two young   |  two young \n",
      "4.60  |  Two white   |  Two white \n",
      "4.80  |  Four girls  |  Four women\n",
      "0.60  |  Kids playi  |  Two dogs p\n",
      "0.20  |  A man in a  |  A man is u\n",
      "2.20  |  A black an  |  A black an\n",
      "4.80  |  The group   |  Several hi\n",
      "1.20  |  Two girls   |  A woman, a\n",
      "2.20  |  A woman is  |  A man who \n",
      "4.60  |  A young bo  |  The boy is\n",
      "2.60  |  A black do  |  A dog is r\n",
      "3.60  |  A man and   |  A man and \n",
      "0.80  |  The brown   |  A brown do\n",
      "2.40  |  Two dogs f  |  Two dogs s\n",
      "4.80  |  The man an  |  A man and \n",
      "5.00  |  A man prac  |  A man prac\n",
      "0.40  |  The girl w  |  The girl i\n",
      "3.20  |  A group of  |  A group of\n",
      "1.80  |  A black do  |  Black dog \n",
      "3.20  |  A small wh  |  A small wh\n",
      "3.40  |  The white   |  The big wh\n",
      "4.80  |  A black do  |  The black \n",
      "4.20  |  A black an  |  A small bl\n",
      "2.60  |  Black and   |  A black an\n",
      "2.20  |  two dogs r  |  A dog layi\n",
      "1.40  |  A golden d  |  A pack of \n",
      "2.60  |  A skateboa  |  A skateboa\n",
      "1.20  |  A girl bou  |  A girl dan\n",
      "1.80  |  A brown an  |  the brown \n",
      "4.00  |  a wet blac  |  A black do\n",
      "0.40  |  Two people  |  Two men in\n",
      "0.00  |  Two little  |  Two ladies\n",
      "0.40  |  A woman in  |  A man and \n",
      "3.00  |  Two people  |  Two people\n",
      "4.00  |  A black do  |  The black \n",
      "1.40  |  A woman in  |  A woman in\n",
      "0.40  |  Three peop  |  A girl sta\n",
      "2.40  |  A group of  |  A crowd of\n",
      "2.80  |  A boy skat  |  A skateboa\n",
      "4.60  |  A white do  |  A white do\n",
      "4.20  |  A man and   |  A man and \n",
      "3.20  |  a skateboa  |  A skateboa\n",
      "1.20  |  A woman an  |  A man and \n",
      "1.80  |  Two girls   |  two girls \n",
      "2.60  |  Boat sails  |  A man is k\n",
      "1.60  |  Three dogs  |  Two dogs a\n",
      "0.00  |  A man in r  |  A man and \n",
      "3.80  |  Two people  |  Two childr\n",
      "4.00  |  A little b  |  A young gi\n",
      "4.80  |  A boy ridi  |  A young bo\n",
      "2.40  |  A black do  |  A black do\n",
      "5.00  |  A black do  |  A black do\n",
      "0.00  |  The man is  |  A man is l\n",
      "3.60  |  A man and   |  A man and \n",
      "2.40  |  Two dogs i  |  Two dogs p\n",
      "4.20  |  Three dogs  |  Three dogs\n",
      "0.20  |  The boy in  |  The black \n",
      "4.20  |  Several pe  |  A group of\n",
      "3.20  |  Two dogs f  |  Two dogs p\n",
      "0.60  |  A group of  |  A group of\n",
      "1.60  |  Two dogs a  |  Two dogs r\n",
      "0.00  |  A black do  |  Kids play \n",
      "1.40  |  A man dres  |  A woman in\n",
      "4.80  |  A man help  |  The man he\n",
      "4.00  |  The two do  |  The dogs p\n",
      "3.00  |  Two dogs a  |  Two black \n",
      "0.20  |  Two tan an  |  Lady in a \n",
      "1.60  |  A man is r  |  Two dogs r\n",
      "4.40  |  A seated y  |  a woman en\n",
      "4.80  |  A black an  |  The black \n",
      "0.40  |  A woman st  |  A man stan\n",
      "2.40  |  Many peopl  |  A busy bea\n",
      "3.00  |  Black and   |  A tan and \n",
      "2.00  |  Three clow  |  Three wome\n",
      "4.60  |  Two beige   |  two tan do\n",
      "1.00  |  A dog and   |  A dog digs\n",
      "5.00  |  A dog jogs  |  a dog trot\n",
      "1.40  |  Group of l  |  a group of\n",
      "0.00  |  A man in a  |  A woman in\n",
      "3.00  |  A brown an  |  a dog with\n",
      "0.60  |  A dog and   |  Two grey d\n",
      "0.00  |  A family i  |  A naked li\n",
      "2.60  |  a dog runs  |  A dog lays\n",
      "1.20  |  A group of  |  A group of\n",
      "5.00  |  There is a  |  A black do\n",
      "3.40  |  Three dogs  |  Three dogs\n",
      "0.80  |  A group of  |  A group of\n",
      "0.80  |  A crowd of  |  A bunch of\n",
      "1.00  |  A Japanese  |  A woman is\n",
      "0.00  |  A man wear  |  A woman we\n",
      "0.00  |  A group of  |  A group of\n",
      "2.00  |  A man with  |  A man with\n",
      "2.40  |  The black   |  The black \n",
      "2.20  |  A boy is j  |  A little b\n",
      "3.80  |  A person w  |  A person i\n",
      "2.20  |  This dog i  |  Two dogs r\n",
      "4.60  |  a young gi  |  A girl pos\n",
      "2.80  |  Two women   |  Two ladies\n",
      "2.00  |  A hockey p  |  A hockey p\n",
      "4.20  |  Two brown   |  Two brown \n",
      "4.50  |  Children a  |  Several ch\n",
      "0.00  |  A dog with  |  A four whe\n",
      "1.80  |  A black an  |  A black an\n",
      "4.20  |  Two black   |  Two dogs p\n",
      "4.20  |  Two boys a  |  Two boys a\n",
      "1.40  |  A group of  |  A group of\n",
      "3.60  |  A black pe  |  A man sitt\n",
      "0.60  |  A man with  |  A spotted \n",
      "5.00  |  A Golden R  |  A golden r\n",
      "0.80  |  A man is s  |  The man is\n",
      "0.00  |  A lady wit  |  A dog with\n",
      "0.00  |  A skier is  |  A woman is\n",
      "1.80  |  The dogs a  |  Two dogs p\n",
      "0.80  |  A young bo  |  A dog is b\n",
      "2.80  |  Two dogs a  |  A brown do\n",
      "0.00  |  a young bo  |  Five greyh\n",
      "0.80  |  A skateboa  |  A dog jump\n",
      "1.80  |  Two dogs r  |  a black an\n",
      "1.40  |  The black   |  A black an\n",
      "1.80  |  A woman ru  |  a dog runs\n",
      "2.40  |  A golden c  |  A black do\n",
      "1.60  |  A white do  |  A black an\n",
      "3.80  |  Woman in b  |  Woman in a\n",
      "4.80  |  Two dogs a  |  Two dogs f\n",
      "0.200000  |  Our curren  |  The car ne\n",
      "1.000000  |  A woman su  |  A woman is\n",
      "0.400000  |  The sky ha  |  This Lady \n",
      "0.400000  |  The young   |  The teen r\n",
      "4.000000  |  There are   |  The dogs a\n",
      "3.400000  |  Some cycli  |  Two men st\n",
      "3.000000  |  These cook  |  The women \n",
      "1.800000  |  A man is c  |  A dog is c\n",
      "2.200000  |  a woman wa  |  An audienc\n",
      "3.600000  |  A girl dan  |  A girl is \n",
      "0.400000  |  The woman   |  The man is\n",
      "0.200000  |  The young   |  Asian chil\n",
      "3.800000  |  A sad man   |  A man is j\n",
      "4.200000  |  A woman is  |  A girl is \n",
      "1.200000  |  A clown pe  |  A blond-ha\n",
      "3.400000  |  Four girls  |  Three youn\n",
      "2.400000  |  a group of  |  A group of\n",
      "3.200000  |  A man play  |  A man play\n",
      "2.200000  |  The people  |  The people\n",
      "4.400000  |  The dogs a  |  The dogs a\n",
      "2.600000  |  the tan do  |  A yellow d\n",
      "0.200000  |  a girl wit  |  The girl i\n",
      "3.200000  |  A person s  |  A woman is\n",
      "2.200000  |  People roc  |  Two people\n",
      "5.000000  |  A blonde w  |  The blond \n",
      "5.000000  |  A boy is a  |  The boy is\n",
      "2.200000  |  An old man  |  a man in a\n",
      "0.400000  |  The carria  |  A man ridi\n",
      "1.000000  |  A woman lo  |  A woman is\n",
      "2.200000  |  Man sittin  |  A man is s\n",
      "2.000000  |  A small bl  |  A young bo\n",
      "1.200000  |  The four w  |  The women \n",
      "4.000000  |  man on ste  |  A man sits\n",
      "1.000000  |  There is a  |  There is a\n",
      "4.200000  |  A woman pa  |  A person p\n",
      "3.400000  |  Two kids a  |  The kids a\n",
      "4.000000  |  The man is  |  The man is\n",
      "4.800000  |  A little b  |  A boy is d\n",
      "0.400000  |  A man has   |  A man on a\n",
      "4.400000  |  The biker   |  A biker is\n",
      "0.400000  |  A musician  |  Trombonist\n",
      "0.000000  |  A pair of   |  Two smilin\n",
      "3.000000  |  A group of  |  A group of\n",
      "4.600000  |  A man is e  |  A man is w\n",
      "0.400000  |  A woman wa  |  A man is s\n",
      "1.400000  |  A man call  |  A man is o\n",
      "3.000000  |  A young bo  |  A young bo\n",
      "1.600000  |  A player b  |  A player c\n",
      "2.400000  |  Men compet  |  Two women \n",
      "0.000000  |  The puppy   |  A man in p\n",
      "1.600000  |  A young gi  |  A boy in a\n",
      "1.600000  |  A smiling   |  A man in a\n",
      "0.200000  |  A person i  |  A person a\n",
      "2.200000  |  Two men ar  |  The two wo\n",
      "2.000000  |  The girl i  |  The man is\n",
      "0.400000  |  Many guard  |  Two men in\n",
      "4.800000  |  A man at a  |  A man is a\n",
      "2.000000  |  four peopl  |  Two men wa\n",
      "0.200000  |  A little g  |  An older c\n",
      "3.800000  |  The child   |  The childr\n",
      "2.200000  |  Men are fi  |  The men ar\n",
      "2.000000  |  People are  |  People are\n",
      "4.000000  |  A dog is c  |  A white do\n",
      "4.600000  |  One footba  |  A football\n",
      "1.400000  |  Three men   |  Two men ar\n",
      "0.800000  |  A woman do  |  A woman is\n",
      "0.600000  |  Five kids   |  People are\n",
      "1.200000  |  Female in   |  A girl in \n",
      "1.600000  |  Baseball p  |  A basketba\n",
      "0.000000  |  They are p  |  Two medica\n",
      "0.600000  |  Some smili  |  People wal\n",
      "3.000000  |  Two fisher  |  Several me\n",
      "2.400000  |  The woman   |  The woman \n",
      "0.600000  |  Black and   |  a small bl\n",
      "4.600000  |  A lady and  |  A girl and\n",
      "0.600000  |  A boy does  |  a boy is r\n",
      "4.800000  |  Three huma  |  Three peop\n",
      "5.000000  |  There are   |  People are\n",
      "0.200000  |  Two women   |  Two Asian \n",
      "0.600000  |  A man is t  |  A little b\n",
      "0.800000  |  Two girls   |  The two gi\n",
      "3.600000  |  A crowd of  |  Several me\n",
      "0.000000  |  The fruits  |  three dogs\n",
      "4.400000  |  A doctor p  |  A doctor p\n",
      "4.600000  |  A woman is  |  The woman \n",
      "2.800000  |  Spectators  |  A crowd of\n",
      "4.000000  |  The man an  |  A man and \n",
      "1.000000  |  The boy in  |  The team i\n",
      "5.000000  |  People are  |  People are\n",
      "0.400000  |  Friends wa  |  a man walk\n",
      "0.800000  |  The roller  |  Two vehicl\n",
      "0.400000  |  There is a  |  A boy is n\n",
      "0.000000  |  A guy and   |  Two boys a\n",
      "1.600000  |  In Abu Dha  |  A busy cit\n",
      "1.600000  |  A pregnant  |  a woman ha\n",
      "3.600000  |  The religi  |  The group \n",
      "3.000000  |  A girl in   |  A girl in \n",
      "2.800000  |  A man is p  |  A young ma\n",
      "0.200000  |  There was   |  The couple\n",
      "1.000000  |  There are   |  Two racers\n",
      "1.400000  |  A basketba  |  A tennis p\n",
      "2.800000  |  A helicopt  |  The plane \n",
      "3.000000  |  A man doin  |  A boy ridi\n",
      "1.600000  |  Three guys  |  A few men \n",
      "2.800000  |  A group wa  |  A group wa\n",
      "4.000000  |  A woman ju  |  A woman po\n",
      "0.800000  |  A man is o  |  A man is h\n",
      "2.000000  |  There are   |  Three wome\n",
      "4.400000  |  The two me  |  The two me\n",
      "4.600000  |  The man wi  |  The man ju\n",
      "2.400000  |  The woman   |  The woman \n",
      "4.400000  |  A man walk  |  A man with\n",
      "0.800000  |  The child   |  There is a\n",
      "3.800000  |  A soccer p  |  The soccer\n",
      "1.400000  |  Two boys i  |  Man in whi\n",
      "2.800000  |  The player  |  The basket\n",
      "3.000000  |  A motorcro  |  A race car\n",
      "1.600000  |  The gate i  |  The gate i\n",
      "1.000000  |  A dog walk  |  A boy with\n",
      "2.600000  |  A pair of   |  Two men st\n",
      "1.600000  |  Blond woma  |  The couple\n",
      "0.200000  |  The raven   |  The skinny\n",
      "3.000000  |  the man is  |  A man is k\n",
      "3.000000  |  Two men ar  |  Two men ar\n",
      "3.200000  |  The woman   |  The woman \n",
      "3.00  |  You'll nee  |  If you nee\n",
      "0.60  |  I am not a  |  At the uni\n",
      "1.00  |  Apart from  |  Age is not\n",
      "0.60  |  This post   |  This is ab\n",
      "0.80  |  If you don  |  If you don\n",
      "2.40  |  I don't th  |  You're goi\n",
      "1.00  |  My answer   |  The proble\n",
      "0.20  |  If you hav  |  You can ha\n",
      "0.60  |  In general  |  For comple\n",
      "3.60  |  It is perf  |  You can al\n",
      "0.20  |  The academ  |  I would no\n",
      "0.20  |  I will adv  |  You could \n",
      "0.00  |  Edit, agai  |  I can thin\n",
      "0.00  |  I agree wi  |  I would sa\n",
      "1.20  |  Supporting  |  I think fo\n",
      "0.00  |  This does   |  I am a PhD\n",
      "0.40  |  A few poss  |  In mathema\n",
      "0.40  |  As an comp  |  The other \n",
      "3.00  |  Stars are   |  Stars rota\n",
      "4.40  |  It indeed   |  In a few b\n",
      "3.20  |  The short   |  It is high\n",
      "3.40  |  All we kno  |  The expans\n",
      "3.40  |  Like Chris  |  Stars are \n",
      "3.60  |  According   |  The AAVSO \n",
      "0.80  |  All of the  |  I will be \n",
      "0.80  |  We are mov  |  There is n\n",
      "1.00  |  Yes, it's   |  When you g\n",
      "1.60  |  Let us ass  |  Globular c\n",
      "1.00  |  Why is the  |  Speed of l\n",
      "3.20  |  As per you  |  The main f\n",
      "2.00  |  A method u  |  You only n\n",
      "0.00  |  The farthe  |  Here is my\n",
      "0.60  |  Stars form  |  It may be \n",
      "0.00  |  A black ho  |  I think yo\n",
      "1.40  |  Yes, but i  |  Constellat\n",
      "2.00  |  Asteroids   |  The object\n",
      "2.60  |  Here is a   |  Just for r\n",
      "4.20  |  It's prett  |  On the sur\n",
      "2.80  |  It is hard  |  Technicall\n",
      "0.00  |  I've found  |  According \n",
      "0.00  |  There are   |  There are \n",
      "2.00  |  Spike trig  |  There's th\n",
      "0.00  |  As mention  |  I like to \n",
      "2.60  |  Q: What is  |  One human-\n",
      "2.60  |  Intrinsic   |  According \n",
      "0.00  |  Presumably  |  This short\n",
      "2.60  |  When study  |  Basically,\n",
      "0.00  |  Some ancie  |  This is in\n",
      "1.60  |  These are   |  I'm not su\n",
      "0.00  |  Since this  |  I'll addre\n",
      "0.60  |  If non-hum  |  Your own t\n",
      "1.80  |  The Hare P  |  From an ar\n",
      "3.80  |  I wrap ste  |  Wrap well \n",
      "1.60  |  In additio  |  I'd been u\n",
      "3.20  |  The best w  |  Soak whole\n",
      "2.60  |  There are   |  The differ\n",
      "3.00  |  I've never  |  I don't bo\n",
      "1.40  |  I just hol  |  By the wor\n",
      "1.60  |  I've store  |  Its going \n",
      "1.60  |  Break the   |  lay down a\n",
      "1.00  |  I managed   |  I've only \n",
      "1.80  |  I  believe  |  In my expe\n",
      "1.80  |  I grill sa  |  Salmon is \n",
      "1.40  |  The recipe  |  Starting w\n",
      "2.60  |  They don't  |  Black pepp\n",
      "2.00  |  This depen  |  Tomatoes w\n",
      "1.00  |  Coagulated  |  There are \n",
      "1.40  |  A common o  |  White fish\n",
      "0.60  |  Good chedd  |  I have two\n",
      "1.40  |  Copper can  |  Probably s\n",
      "0.40  |  She obviou  |  The first \n",
      "1.80  |  In some ca  |  There is n\n",
      "4.00  |  I agree wi  |  I have nev\n",
      "0.60  |  The assert  |  \"I had mai\n",
      "2.80  |  I'd sugges  |  As bikeboy\n",
      "2.60  |  I see some  |  There are \n",
      "2.60  |  If I under  |  Inclusion \n",
      "0.60  |  Firstly, y  |  Semicolon \n",
      "4.40  |  Pardon the  |  Named is p\n",
      "2.20  |  In standar  |  I'd say th\n",
      "1.00  |  The \"more   |  As Shinto \n",
      "0.00  |  I'm Americ  |  In situati\n",
      "1.60  |  The room u  |  I'm Englis\n",
      "1.80  |  I can only  |  The form i\n",
      "3.80  |  As others   |  \"Ten of si\n",
      "3.00  |  Certainly   |  Second one\n",
      "1.40  |  This may b  |  I'm not su\n",
      "3.00  |  Back in th  |  The word \"\n",
      "0.40  |  Those expr  |  I suspect \n",
      "3.60  |  In the Uni  |  Jail is a \n",
      "3.40  |  Yes, your   |  Most style\n",
      "2.80  |  If you und  |  Another me\n",
      "4.20  |  There are   |  There are \n",
      "4.20  |  You defini  |  No, don't \n",
      "1.80  |  Yes, you s  |  Neuromuscu\n",
      "1.00  |  I feel you  |  Sweat rate\n",
      "2.00  |  Running do  |  Additional\n",
      "1.00  |  I recently  |  Low reps w\n",
      "1.60  |  Rotate thr  |  Air squats\n",
      "2.60  |  If you're   |  There are \n",
      "3.20  |  I've notic  |  A treadmil\n",
      "0.40  |  Google sea  |  As @Sarge \n",
      "0.00  |  Since you'  |  In general\n",
      "0.00  |  First of a  |  You can mi\n",
      "2.20  |  Properly e  |  I've seen \n",
      "1.50  |  I think it  |  I haven't \n",
      "0.60  |  It is not   |  Adding soa\n",
      "4.00  |  Yes, you c  |  Absolutely\n",
      "1.80  |  There are   |  I brought \n",
      "0.60  |  Pick them   |  I live in \n",
      "1.60  |  Thistle is  |  I've got t\n",
      "2.40  |  Support is  |  Some suppo\n",
      "1.00  |  To paraphr  |  First, let\n",
      "2.00  |  Just about  |  We have qu\n",
      "1.60  |  Chances ar  |  The longer\n",
      "0.20  |  So you kno  |  Since you'\n",
      "0.20  |  In the pas  |  Additional\n",
      "2.00  |  There is a  |  It is mean\n",
      "0.00  |  I found it  |  I reccomen\n",
      "3.20  |  Simple ans  |  It appears\n",
      "1.00  |  Dirt can m  |  Use coco p\n",
      "2.00  |  If you can  |  The NPK nu\n",
      "2.40  |  The point   |  Your pile \n",
      "0.00  |  I bought a  |  Well I wou\n",
      "1.60  |  Aloes, lik  |  Some of th\n",
      "3.80  |  This looks  |  I concur w\n",
      "1.20  |  Some villa  |  Many towns\n",
      "2.00  |  The USA is  |  The United\n",
      "1.00  |  Nearly all  |  The USSR g\n",
      "2.00  |  While Quee  |  There is a\n",
      "0.60  |  To no exte  |  They had a\n",
      "0.00  |  \"What I am  |  The follow\n",
      "0.00  |  In Hinduis  |  This relat\n",
      "3.40  |  I guess th  |  That is in\n",
      "2.60  |  These are   |  I guess th\n",
      "0.40  |  Looks like  |  There was \n",
      "2.80  |  Yes, I can  |  It is true\n",
      "2.00  |  By the tim  |  The Han Dy\n",
      "1.60  |  There were  |  As per you\n",
      "1.40  |  \"The grass  |  As noted o\n",
      "1.20  |  http://en.  |  I am not e\n",
      "2.20  |  Dominus, p  |  A greek-sp\n",
      "0.60  |  The bulk o  |  I believe \n",
      "2.60  |  Yes, this   |  Indefinite\n",
      "3.00  |  Preserving  |  In fact, y\n",
      "2.20  |  In some no  |  Modern Eng\n",
      "1.80  |  The diffic  |  It depends\n",
      "0.60  |  In English  |  In traditi\n",
      "0.00  |  A caveat:   |  I agree wi\n",
      "1.60  |  The reason  |  Although I\n",
      "0.80  |  Palatalisa  |  Sinhala, o\n",
      "0.20  |  All the sa  |  One idea t\n",
      "1.00  |  Newspeak,   |  Every lang\n",
      "2.80  |  In Russian  |  As in Russ\n",
      "0.60  |  As you can  |  This is an\n",
      "2.80  |  Zipf’s law  |  As with al\n",
      "1.60  |  A part of   |  \"Phoneme\" \n",
      "1.40  |  If you are  |  This is wh\n",
      "1.40  |  From what   |  According \n",
      "3.00  |  A tent may  |  No, a tent\n",
      "3.20  |  A few poin  |  In general\n",
      "2.80  |  Nylon drie  |  Nylon, amo\n",
      "3.40  |  GPS (wikip  |  GPS system\n",
      "2.00  |  Different   |  American b\n",
      "2.80  |  So it's im  |  Skiing is \n",
      "0.00  |  In a gym,   |  Alright, t\n",
      "0.20  |  I visit AZ  |  Apart from\n",
      "0.80  |  A wonderfu  |  If you can\n",
      "0.00  |  An embolis  |  Here is an\n",
      "4.20  |  It would b  |  I'm no her\n",
      "0.20  |  As I wrote  |  Unlike oth\n",
      "3.20  |  Once you o  |  Boiling el\n",
      "0.00  |  Different   |  You can ge\n",
      "0.60  |  If you wea  |  The above \n",
      "0.00  |  Oh friend,  |  Yes, it is\n",
      "4.80  |  My opinion  |  My opinion\n",
      "3.80  |  You are sa  |  I would sa\n",
      "1.80  |  He is taki  |  You can tr\n",
      "0.40  |  I would re  |  Would you \n",
      "2.60  |  Yes, twins  |  Here's wha\n",
      "1.60  |  Whatever y  |  We always \n",
      "1.40  |  A rule of   |  Any breed \n",
      "2.00  |  I know of   |  Baby talk,\n",
      "1.80  |  A possibil  |  Many child\n",
      "1.20  |  The point   |  I've found\n",
      "0.80  |  You should  |  It is alwa\n",
      "2.20  |  You will b  |  The topics\n",
      "3.20  |  Although c  |  As was men\n",
      "1.40  |  Both of ou  |  Yours look\n",
      "1.40  |  I have nev  |  We used a \n",
      "0.00  |  I'm facing  |  An alterna\n",
      "1.60  |  My son did  |  Here are s\n",
      "0.80  |  Show her h  |  You can pu\n",
      "2.80  |  Our daught  |  My wife an\n",
      "1.80  |  Playing do  |  I have bee\n",
      "0.67  |  Most docto  |  Eating off\n",
      "2.60  |  Most of th  |  In my expe\n",
      "0.40  |  Here is my  |  I understa\n",
      "2.40  |  It is poss  |  We tried p\n",
      "2.80  |  Turtles lo  |  First, the\n",
      "2.00  |  Given that  |  To reduce \n",
      "3.00  |  My experie  |  Breed Zebr\n",
      "2.60  |  I have see  |  I think Ke\n",
      "1.80  |  Cats (and   |  They don't\n",
      "3.80  |  It does so  |  Something \n",
      "3.00  |  I have not  |  Cats may o\n",
      "1.40  |  How could   |  In my 200 \n",
      "1.60  |  First, dou  |  First, you\n",
      "0.00  |  The approp  |  The info y\n",
      "0.60  |  My guess i  |  Some resea\n",
      "0.00  |  Cover the   |  I started \n",
      "3.40  |  Try switch  |  As mention\n",
      "3.60  |  You should  |  As covered\n",
      "2.20  |  The main c  |  Allergies \n",
      "3.00  |  There are   |  John Cavan\n",
      "1.80  |  Stocking i  |  You'll alw\n",
      "2.40  |  The best a  |  A trick I \n",
      "2.60  |  I don't ha  |  Bettas are\n",
      "1.40  |  We always   |  We had a c\n",
      "1.60  |  The bigges  |  Cats don't\n",
      "1.60  |  Breed Zebr  |  Supposedly\n",
      "1.40  |  Clean the   |  Others hav\n",
      "2.80  |  Aristotle   |  Why did Ar\n",
      "3.80  |  I don't th  |  I am not c\n",
      "3.40  |  Well for o  |  The differ\n",
      "0.00  |  People who  |  My motivat\n",
      "2.00  |  The differ  |  sufficient\n",
      "0.00  |  I have no   |  It's a tau\n",
      "3.80  |  It's true   |  Accessibil\n",
      "0.20  |  I once met  |  Although I\n",
      "0.60  |  If a state  |  Should arr\n",
      "1.80  |  The Questi  |  The least \n",
      "1.80  |  I think th  |  Wittgenste\n",
      "1.20  |  Forget for  |  As others \n",
      "0.00  |  The questi  |  wikipedia,\n",
      "1.00  |  Not all ne  |  I don't th\n",
      "0.40  |  As I under  |  Our intera\n",
      "0.80  |  I can imag  |  It is the \n",
      "3.00  |  When I nee  |  An alterna\n",
      "3.80  |  Fast Pictu  |  Fast Pictu\n",
      "3.20  |  There are   |  There are \n",
      "2.60  |  Apart from  |  The wider \n",
      "3.00  |  When print  |  It refers \n",
      "2.60  |  You need t  |  In my expe\n",
      "1.80  |  I have a f  |  If your in\n",
      "2.60  |  I have fou  |  There are \n",
      "0.80  |  I've used   |  In Linux I\n",
      "1.60  |  A class 6   |  Some cards\n",
      "0.60  |  Here are m  |  It's a 1:1\n",
      "2.60  |  Same here,  |  I have the\n",
      "1.20  |  I use Elin  |  Or just go\n",
      "0.00  |  The short   |  Speaking f\n",
      "2.40  |  The termin  |  Lossless /\n",
      "0.40  |  For what i  |  Regular di\n",
      "0.60  |  Large-form  |  If I had t\n",
      "1.00  |  Anil, ther  |  If you wan\n",
      "2.60  |  Freezing m  |  In additio\n",
      "4.00  |  Federal DO  |  DOMA as a \n",
      "3.60  |  Public emp  |  The legal \n",
      "2.60  |  The U.S. a  |  There are \n",
      "1.60  |  Leaving as  |  Healthcare\n",
      "4.00  |  The presid  |  The functi\n",
      "3.00  |  In a \"firs  |  While the \n",
      "0.60  |  The part o  |  Side note \n",
      "0.00  |  Let me sha  |  I am sure \n",
      "0.40  |  Most peopl  |  To start, \n",
      "0.40  |  I'm not ye  |  There are \n",
      "2.00  |  No, nothin  |  There is a\n",
      "0.60  |  Russian Fe  |  The last r\n",
      "0.00  |  This quest  |  Minimum wa\n",
      "3.20  |  According   |  As with th\n",
      "2.80  |  Because th  |  They do no\n",
      "2.60  |  Absolute m  |  In countri\n",
      "2.80  |  Professor   |  Minimum wa\n",
      "0.20  |  The Univer  |  This is st\n",
      "3.20  |  Before you  |  As Eugene \n",
      "0.60  |  Eclipse Be  |  To train y\n",
      "3.60  |  As others   |  I've been \n",
      "0.00  |  The import  |  I have bee\n",
      "0.00  |  I think yo  |  One genera\n",
      "0.00  |  A quick re  |  The issue \n",
      "0.00  |  Here's a t  |  I agree wi\n",
      "2.20  |  I have a s  |  As mention\n",
      "0.00  |  Even thoug  |  I don't kn\n",
      "0.40  |  Your brain  |  I'm writin\n",
      "0.00  |  To stick w  |  There is a\n",
      "2.60  |  I think 90  |  Definitely\n",
      "0.80  |  The user n  |  Instead of\n",
      "0.40  |  I like the  |  Buy a 3 pa\n",
      "3.00  |  I'd start   |  I'm consid\n",
      "2.20  |  The best w  |  What works\n",
      "0.60  |  I disagree  |  Research d\n",
      "3.40  |  Currently   |  My GTD sys\n",
      "2.20  |  Feel the f  |  13 Tips To\n",
      "1.80  |  I've recen  |  http://www\n",
      "0.80  |  I would sa  |  There's al\n",
      "0.00  |  Some studi  |  There's an\n",
      "1.60  |  Experience  |  I think a \n",
      "2.80  |  There is a  |  The capita\n",
      "1.67  |  I disagree  |  The most e\n",
      "1.80  |  I'd be mor  |  It is not \n",
      "2.80  |  On the one  |  Time Turne\n",
      "0.00  |  I was invi  |  It's obvio\n",
      "0.60  |  First of a  |  I always t\n",
      "1.80  |  I started   |  Without de\n",
      "0.40  |  I voted fo  |  Allen Stee\n",
      "0.20  |  It was bef  |  The exact \n",
      "0.00  |  We have tw  |  Our intrep\n",
      "1.40  |  Dick was a  |  Most of Ph\n",
      "0.00  |  First, we   |  \"Fins\" on \n",
      "0.60  |  According   |  In real-wo\n",
      "0.80  |  I don't th  |  The short \n",
      "0.20  |  Here is th  |  Wikipedia'\n",
      "2.20  |  Sign langu  |  It is safe\n",
      "1.80  |  Remember,   |  Well, it r\n",
      "0.20  |  This idea   |  The scienc\n",
      "4.00  |  Is it poss  |  Yes, it's \n",
      "4.20  |  Yes a team  |  There's no\n",
      "4.40  |  It is simp  |  Bowling st\n",
      "3.00  |  Yes, absol  |  Yes, the r\n",
      "2.60  |  We only us  |  Fartlek is\n",
      "0.60  |  If the num  |  This is th\n",
      "3.20  |  Women firs  |  In 1900, t\n",
      "2.60  |  It is hard  |  Drafting h\n",
      "1.20  |  To play go  |  Having fai\n",
      "3.00  |  Yes, dew (  |  Another di\n",
      "0.00  |  Reading yo  |  I'd say yo\n",
      "2.60  |  There is N  |  There is a\n",
      "1.20  |  Basically,  |  It is fair\n",
      "2.80  |  At the hig  |  From perso\n",
      "0.20  |  I can thin  |  There have\n",
      "1.80  |  The moveme  |  Both of th\n",
      "0.00  |  Actually,   |  That is al\n",
      "0.00  |  Red ball u  |  I am too a\n",
      "3.00  |  No, there   |  As per the\n",
      "0.00  |  I can't th  |  The Laws o\n",
      "2.40  |  As per FIF  |  Although h\n",
      "0.40  |  Glad to he  |  A general \n",
      "2.60  |  I prefer t  |  I would de\n",
      "0.00  |  It was ran  |  This was t\n",
      "3.60  |  For the mo  |  Generally \n",
      "1.80  |  There is o  |  There are \n",
      "1.60  |  I'd sugges  |  I am not s\n",
      "2.00  |  As I remem  |  This page \n",
      "2.00  |  My experie  |  When using\n",
      "0.00  |  I had a si  |  I'm actual\n",
      "2.60  |  A technica  |  Usually it\n",
      "0.40  |  Kyoto, at   |  Coin locke\n",
      "0.00  |  There's a   |  It's prett\n",
      "0.20  |  I visited   |  Cusco: Hig\n",
      "1.80  |  Generally,  |  There are \n",
      "1.60  |  The most o  |  Look at th\n",
      "1.00  |  The answer  |  As someone\n",
      "0.60  |  There are   |  As others \n",
      "1.00  |  Would goin  |  Ozbus offe\n",
      "2.80  |  English wo  |  I traveled\n",
      "2.60  |  I think th  |  I think it\n",
      "3.40  |  There is n  |  The minima\n",
      "1.60  |  I've start  |  I've tried\n",
      "4.00  |  Yes, this   |  I believe \n",
      "1.80  |  For fictio  |  To answer \n",
      "0.20  |  David Webe  |  One thing \n",
      "0.00  |  You have t  |  Whenever I\n",
      "1.40  |  Your worry  |  I don't be\n",
      "0.00  |  No, you do  |  I have don\n",
      "1.20  |  One thing   |  I noticed \n",
      "3.60  |  We just ad  |  guillemets\n",
      "3.00  |  The villai  |  A villain \n",
      "1.20  |  You can wr  |  Sequence a\n",
      "1.80  |  Yes, dull   |  I agree wi\n",
      "0.80  |  The answer  |  At first, \n",
      "1.20  |  Webster's   |  http://www\n",
      "0.00  |  Keep in mi  |  I think St\n",
      "3.20  |  In these d  |  I agree wi\n",
      "0.20  |  I have bee  |  Am I allow\n",
      "0.800  |  The techno  |  The broad \n",
      "3.600  |  \"It's a hu  |  \"It's a hu\n",
      "5.000  |  He later l  |  He later f\n",
      "1.400  |  RT Jones a  |  He has a v\n",
      "2.600  |  A key ques  |  France, wh\n",
      "2.000  |  Mel Gibson  |  Mel Gibson\n",
      "3.750  |  Executive   |  Executive \n",
      "4.000  |  The decade  |  More than \n",
      "3.750  |  Tom Kasmer  |  The 14-yea\n",
      "3.200  |  The budget  |  Budget neg\n",
      "4.000  |  Also at in  |  Also at in\n",
      "2.400  |  Shares of   |  In morning\n",
      "4.400  |  \"I feel th  |  The people\n",
      "4.200  |  \"Generally  |  Generally \n",
      "3.200  |  The retail  |  The decisi\n",
      "3.000  |  Lord Falco  |  \"It's a ne\n",
      "0.800  |  The techno  |  The broad \n",
      "3.200  |  Business W  |  BusinessWe\n",
      "3.800  |  \"I started  |  Gulping fo\n",
      "3.000  |  It was the  |  For the th\n",
      "2.400  |  GM, the wo  |  They cover\n",
      "5.000  |  The compan  |  The compan\n",
      "3.800  |  North Amer  |  North Amer\n",
      "3.200  |  The Standa  |  The Standa\n",
      "3.800  |  \"I know of  |  \"I know of\n",
      "3.800  |  \"She was c  |  \"She was c\n",
      "3.750  |  \"Frank Qua  |  Quattrone \n",
      "3.200  |  The dollar  |  The dollar\n",
      "3.400  |  Germans ac  |  Germans re\n",
      "4.600  |  Also in Mo  |  Near a mos\n",
      "3.333  |  Mr. Bush h  |  In Texas, \n",
      "3.500  |  This integ  |  IBM said t\n",
      "1.750  |  Freeman's   |  Freeman sa\n",
      "3.000  |  Protesters  |  The protes\n",
      "4.400  |  Another mi  |  A Spanish \n",
      "3.800  |  Singapore   |  Although a\n",
      "3.000  |  After 9/11  |  Connolly s\n",
      "4.500  |  Calvin Hol  |  The lawyer\n",
      "2.200  |  PG&E Corp.  |  PG&E's sha\n",
      "4.000  |  \"This deci  |  The decisi\n",
      "4.600  |  Paul Themb  |  MDC spokes\n",
      "3.500  |  He claims   |  \"This prop\n",
      "3.000  |  YES filed   |  The suit w\n",
      "4.000  |  He will be  |  Dynes will\n",
      "4.333  |  His daught  |  Axelrod di\n",
      "3.250  |  Deaths in   |  Fatalities\n",
      "3.250  |  The pound   |  The Britis\n",
      "1.750  |  The Record  |  That is, i\n",
      "0.800  |  The broade  |  The techno\n",
      "4.400  |  Earlier Th  |  Thursday m\n",
      "3.000  |  Both paper  |  The study \n",
      "3.000  |  The WHO ex  |  Hebei has \n",
      "3.250  |  \"I do not   |  \"Yes, I do\n",
      "4.400  |  They are b  |  Both men a\n",
      "5.000  |  The report  |  The report\n",
      "5.000  |  Who will s  |  There is n\n",
      "3.200  |  Under the   |  The U.S. H\n",
      "2.200  |  The Securi  |  That feder\n",
      "3.250  |  Lee Janzen  |  Lee Janzen\n",
      "3.750  |  Tenet has   |  In Novembe\n",
      "3.200  |  Dotson was  |  Dotson, 21\n",
      "4.400  |  The bank a  |  The offer \n",
      "2.600  |  \"They've b  |  The quarte\n",
      "3.500  |  The world'  |  The world'\n",
      "3.750  |  But price   |  McArthur t\n",
      "3.500  |  Putin hail  |  Before the\n",
      "4.250  |  Appellate   |  Lower cour\n",
      "2.400  |  He was arr  |  He was arr\n",
      "3.250  |  The effort  |  July 10 - \n",
      "3.800  |  The shares  |  The sale r\n",
      "3.400  |  Sirius car  |  Sirius rec\n",
      "3.500  |  Mahmud con  |  Mahmud con\n",
      "2.400  |  According   |  According \n",
      "3.800  |  Actor Noah  |  Noah Wyle,\n",
      "3.800  |  The WHO sa  |  The WHO ha\n",
      "4.000  |  U.S. troop  |  U.S. soldi\n",
      "3.400  |  It got up   |  \"It got up\n",
      "3.250  |  A former c  |  Mr. McCall\n",
      "4.600  |  FBI agents  |  A former p\n",
      "3.714  |  Yeager sai  |  Yeager sai\n",
      "3.333  |  The 30-yea  |  The 30-yea\n",
      "3.400  |  \"If that a  |  And if tha\n",
      "2.800  |  \"It's a ma  |  The Maine \n",
      "3.000  |  The Embrae  |  The Bombar\n",
      "3.000  |  \"I can say  |  \"I am bein\n",
      "3.750  |  Investigat  |  Investigat\n",
      "4.400  |  Revenue ro  |  Revenue wa\n",
      "1.000  |  The broad   |  The techno\n",
      "2.800  |  Neither Io  |  Iowa State\n",
      "3.250  |  The male e  |  The 21-yea\n",
      "1.750  |  Agnew said  |  Still, Pin\n",
      "1.750  |  Marc Garbe  |  Whitley's \n",
      "5.000  |  BioRelianc  |  Shares of \n",
      "3.600  |  Just five   |  Just five \n",
      "3.000  |  Artists ar  |  The artist\n",
      "2.800  |  At his req  |  District S\n",
      "3.600  |  A federal   |  A federal \n",
      "3.000  |  The time w  |  The seizur\n",
      "1.000  |  The techno  |  The broade\n",
      "5.000  |  EU ministe  |  Gerry Kiel\n",
      "3.500  |  \"Now, the   |  \"The other\n",
      "4.500  |  In his new  |  Dynes will\n",
      "4.800  |  But in the  |  But Vincen\n",
      "3.500  |  Justices s  |  In a 6-3 r\n",
      "3.692  |  The vetera  |  Razali Ism\n",
      "4.000  |  At first b  |  At first b\n",
      "3.500  |  Ukrainian   |  The disput\n",
      "4.750  |  Chechen of  |  Officials \n",
      "3.200  |  About 25 g  |  According \n",
      "2.600  |  The panel   |  It called \n",
      "3.000  |  The compan  |  The servic\n",
      "3.250  |  Gold's ris  |  Gold's bou\n",
      "2.200  |  Shares of   |  Shares of \n",
      "3.500  |  It will co  |  It will co\n",
      "4.750  |  Teams of i  |  Teams of i\n",
      "3.200  |  The identi  |  The rovers\n",
      "3.250  |  Representa  |  Puretunes \n",
      "3.750  |  I've still  |  \"I'm still\n",
      "3.000  |  Two astron  |  Astronomer\n",
      "2.600  |  \"What they  |  The church\n",
      "3.250  |  The woman   |  The woman \n",
      "3.400  |  The Americ  |  The Americ\n",
      "1.333  |  Amgen shar  |  Shares of \n",
      "5.000  |  The last t  |  In 1995, t\n",
      "4.000  |  Higher cou  |  The federa\n",
      "3.600  |  In two wee  |  Frankel sa\n",
      "0.800  |  The techno  |  The broade\n",
      "3.500  |  He allowed  |  Zambrano p\n",
      "3.000  |  Still, he   |  \"I'm absol\n",
      "1.667  |  Nationally  |  There were\n",
      "3.750  |  On Thursda  |  On Thursda\n",
      "2.250  |  Mr. Heatle  |  Heatley un\n",
      "4.200  |  Former com  |  Last week,\n",
      "3.600  |  U.S. Distr  |  FirstEnerg\n",
      "3.400  |  Mr Pollard  |  Nick Polla\n",
      "3.000  |  \"They were  |  \"The concr\n",
      "3.250  |  \"IT is the  |  \"IT is the\n",
      "3.600  |  The new sy  |  The system\n",
      "3.000  |  The router  |  Netgear pr\n",
      "1.750  |  On Monday,  |  Last Thurs\n",
      "3.250  |  The first   |  Emile Laro\n",
      "4.400  |  \"I'm not g  |  \"I'm not g\n",
      "2.800  |  Dixon was   |  Scott Dixo\n",
      "2.750  |  By Sunday   |  Major fire\n",
      "3.000  |  U.S. force  |  U.S. force\n",
      "3.600  |  They came   |  BA said th\n",
      "3.000  |  The author  |  Spitz is e\n",
      "2.000  |  The two-ye  |  The benchm\n",
      "3.000  |  As part of  |  Peregrine \n",
      "4.400  |  Part of th  |  A key port\n",
      "2.400  |  Shares in   |  Shares in \n",
      "4.400  |  The compan  |  Chiron and\n",
      "4.400  |  Investigat  |  Investigat\n",
      "2.600  |  Joining St  |  The Intern\n",
      "3.000  |  The Winsto  |  Six new Kr\n",
      "4.500  |  The report  |  There will\n",
      "3.000  |  Families s  |  Families s\n",
      "1.750  |  For the en  |  \"The avera\n",
      "2.000  |  Orange sha  |  France Tel\n",
      "2.400  |  A search f  |  Authoritie\n",
      "4.800  |  Beleaguere  |  Two federa\n",
      "4.000  |  He claimed  |  In his let\n",
      "4.000  |  Police the  |  While wait\n",
      "4.400  |  At 12 mont  |  At 12 mont\n",
      "4.200  |  \"It just s  |  \"It just s\n",
      "3.600  |  \"City-grow  |  \"I know th\n",
      "2.500  |  The Standa  |  The broad \n",
      "1.500  |  The Dow Jo  |  The Dow Jo\n",
      "5.000  |  \"We hope a  |  China's fo\n",
      "3.750  |  School off  |  Van-Vliet,\n",
      "1.750  |  Board Chan  |  Mr. Mills \n",
      "4.400  |  Its former  |  Chief Exec\n",
      "3.000  |  Police bel  |  Police bel\n",
      "3.800  |  According   |  The Telegr\n",
      "3.250  |  Bulger's b  |  Bulger's b\n",
      "2.750  |  Florida's   |  On Tuesday\n",
      "3.400  |  \"The Europ  |  Tito Barbi\n",
      "3.200  |  \"It was a   |  \"We're in \n",
      "2.000  |  From Broad  |  Playwright\n",
      "3.000  |  Nine secon  |  Nine secon\n",
      "3.250  |  The plane   |  US Airways\n",
      "5.000  |  Gainer sai  |  The securi\n",
      "3.333  |  He also sa  |  The academ\n",
      "3.600  |  Gartner's   |  Total ship\n",
      "3.800  |  Roy Moore,  |  Moore, the\n",
      "3.000  |  The blue-c  |  The blue-c\n",
      "4.250  |  An arrest   |  According \n",
      "3.500  |  FRIENDS of  |  Hollywood \n",
      "3.200  |  But JT was  |  \"However, \n",
      "2.400  |  Shares of   |  Halliburto\n",
      "4.000  |  Tail waggi  |  On Monday,\n",
      "4.667  |  At midnigh  |  With 97 pe\n",
      "3.333  |  Graham is   |  Later Tues\n",
      "3.000  |  \"These cha  |  Still, cha\n",
      "3.250  |  Barbini sa  |  Barbini sa\n",
      "4.000  |  The blaze   |  The fire s\n",
      "3.500  |  Sources sa  |  Agents con\n",
      "2.000  |  The helico  |  The helico\n",
      "3.600  |  \"I am advi  |  \"I am advi\n",
      "3.400  |  The new re  |  It will ap\n",
      "3.500  |  Two convic  |  A convicte\n",
      "2.400  |  John Hicke  |  Hickenloop\n",
      "3.500  |  Young has   |  Under NASD\n",
      "2.800  |  Stocks dip  |  Wall Stree\n",
      "4.800  |  From the s  |  But the Un\n",
      "3.000  |  Selenski d  |  Selenski u\n",
      "3.400  |  Common sid  |  The most c\n",
      "3.250  |  Announcing  |  Together, \n",
      "4.750  |  Supermarke  |  Supermarke\n",
      "3.600  |  INTEL TODA  |  Intel on W\n",
      "4.200  |  The 6th U.  |  An Ohio la\n",
      "2.000  |  The Nasdaq  |  The techno\n",
      "3.800  |  \"We're goi  |  \"There's v\n",
      "1.600  |  The compan  |  The compan\n",
      "3.250  |  Shares of   |  Shares of \n",
      "5.000  |  The upcomi  |  They say s\n",
      "3.417  |  That took   |  That saw t\n",
      "2.750  |  The bank's  |  Shares of \n",
      "4.111  |  In the lat  |  Early Mond\n",
      "3.600  |  Consumer g  |  Consumer g\n",
      "3.600  |  Every Thur  |  A grains m\n",
      "4.333  |  Colin Powe  |  Secretary \n",
      "3.200  |  Stout prev  |  Stout come\n",
      "2.250  |  Scrimshaw,  |  Best Minis\n",
      "4.500  |  Former Sou  |  South Afri\n",
      "2.750  |  In Septemb  |  Four month\n",
      "4.000  |  Consumers   |  To watch p\n",
      "3.400  |  On Friday,  |  Yesterday,\n",
      "2.200  |  A Stage 1   |  The federa\n",
      "3.250  |  While wait  |  The bomb e\n",
      "3.667  |  \"It seems   |  \"Leaving a\n",
      "4.750  |  The Americ  |  The U.S. w\n",
      "3.500  |  Its maker,  |  MedImmune \n",
      "3.000  |  Kaichen ap  |  She appear\n",
      "3.250  |  An increme  |  The increm\n",
      "3.600  |  Bush turne  |  Mr. Bush d\n",
      "3.800  |  The govern  |  The govern\n",
      "1.000  |  The broade  |  The techno\n",
      "3.333  |  Crews work  |  Crews work\n",
      "4.250  |  Buoyed by   |  Buoyed by \n",
      "1.750  |  Net revenu  |  That is up\n",
      "2.750  |  These men   |  \"The petit\n",
      "3.250  |  The mock e  |  The mock e\n",
      "2.500  |  The commis  |  Patton als\n",
      "3.091  |  Aiken's st  |  The findin\n",
      "4.400  |  A Merrill   |  \"We are ve\n",
      "4.800  |  Trans fats  |  Trans fats\n",
      "5.000  |  But skepti  |  But skepti\n",
      "3.000  |  Crohn's di  |  Symptoms i\n",
      "3.200  |  Authoritie  |  Powell is \n",
      "4.4  |  Netanyahu   |  Israel's N\n",
      "1.8  |  Annan, UN   |  Russia, Ch\n",
      "4  |  Green Lant  |  Green Lant\n",
      "1  |  Military p  |  Military p\n",
      "0.6  |  Saudi gas   |  Nigeria ch\n",
      "4.6  |  Coty launc  |  Coty bids \n",
      "1.8  |  Greece vot  |  Greece loo\n",
      "1.2  |  Iran warns  |  Iranians “\n",
      "4.6  |  Mohamed Mo  |  Mohamed Mo\n",
      "1.4  |  1 person k  |  Nine kille\n",
      "3.2  |  IAF strike  |  IAF strike\n",
      "2.8  |  US senator  |  US decries\n",
      "0.8  |  Syrian env  |  Syria depl\n",
      "3.8  |  Dali paint  |  Salvador D\n",
      "3.8  |  Annan urge  |  Annan urge\n",
      "4.2  |  Colorado s  |  Lawyers: C\n",
      "1.6  |  Suicide bo  |  Suicide bo\n",
      "3.8  |  PM in Tehr  |  Manmohan i\n",
      "1.4  |  Senators c  |  Messi scor\n",
      "3.6  |  South Afri  |  South Afri\n",
      "1.8  |  Civilians   |  Taliban la\n",
      "3.8  |  Russia, Ch  |  Russia, Ch\n",
      "0.2  |  Syrians tr  |  Israel bra\n",
      "1.6  |  Israeli Le  |  Israeli ar\n",
      "0.4  |  Government  |  Finance mi\n",
      "2.33333333  |  Egyptian p  |  Egypt's Pr\n",
      "0.4  |  Japan fall  |  Japan than\n",
      "3.6  |  Syrian tro  |  Activists:\n",
      "3.4  |  Suspicious  |  Police tre\n",
      "2.6  |  At least 1  |  At least 1\n",
      "2.2  |  US drone s  |  US drone s\n",
      "2.6  |  Obama endo  |  Obama's de\n",
      "4.8  |  Jordan Ope  |  Jordan Ope\n",
      "4.8  |  Eurozone b  |  Eurozone a\n",
      "4.8  |  Two blasts  |  Two blasts\n",
      "2.6  |  Barack Oba  |  Obama back\n",
      "1.6  |  Police dis  |  Police dis\n",
      "2.2  |  2 pro-Pale  |  Dozens of \n",
      "1  |  Spanish jo  |  US jobless\n",
      "1.2  |  Israeli PM  |  Israel bla\n",
      "3.4  |  Music Indu  |  Music indu\n",
      "0.6  |  Five dead   |  Over 120 d\n",
      "3.2  |  At CPAC, M  |  CPAC 2012:\n",
      "3.2  |  Glorious t  |  Britain sa\n",
      "1.8  |  Toulouse s  |  Toulouse s\n",
      "1.2  |  Wigan 3-2   |  Arsenal 0-\n",
      "1.6  |  UK spends   |  US spends \n",
      "4.8  |  Lance Arms  |  Lance Arms\n",
      "3.6  |  Prince Phi  |  Britain's \n",
      "3.8  |  'Desperate  |  'Housewive\n",
      "3  |  Gaza rocke  |  Rocket fir\n",
      "3.6  |  Egypt's Co  |  Egypt prep\n",
      "4.4  |  EU Set for  |  EU plans m\n",
      "2  |  Assad meet  |  Assad make\n",
      "2.4  |  Manchester  |  Manchester\n",
      "3.2  |  Soggy sols  |  Summer sol\n",
      "3.4  |  ICC lawyer  |  ICC lawyer\n",
      "2.8  |  Freight tr  |  Freight tr\n",
      "3.2  |  China yuan  |  China's yu\n",
      "1.8  |  Breivik co  |  Breivik: J\n",
      "2.8  |  Romney roc  |  Romney tap\n",
      "3.4  |  Eurozone a  |  Eurozone t\n",
      "4.6  |  South Kore  |  S. Korean \n",
      "3.8  |  Russia, Ch  |  Russia, Ch\n",
      "1.6  |  Juror: San  |  Jerry Sand\n",
      "0.8  |  Romney lea  |  Romney: I \n",
      "1  |  9/11 defen  |  Sept. 11 d\n",
      "2.6  |  Egyptians   |  Egyptians \n",
      "4.2  |  Sly Stallo  |  Sylvester \n",
      "4.4  |  Clinton: U  |  Clinton: U\n",
      "2.2  |  Romney, Ob  |  Obama and \n",
      "2.6  |  Roadside b  |  Roadside b\n",
      "2.8  |  Remarks by  |  The Presid\n",
      "4  |  Malawi con  |  Malawi con\n",
      "3.2  |  Venezuela'  |  Venezuelan\n",
      "0  |  Residents   |  Best’s gre\n",
      "1.8  |  Syria in c  |  Syrian cea\n",
      "0  |  Spurs get   |  Abduct Isr\n",
      "2.6  |  China's CP  |  China's in\n",
      "2.75  |  Gaza milit  |  Gazans fir\n",
      "3.2  |  Iraq Relea  |  Iraq Said \n",
      "1.8  |  Suicide bo  |  Suicide bo\n",
      "4.2  |  7 detained  |  China deta\n",
      "4.2  |  China Yuan  |  China yuan\n",
      "2.6  |  Texas resi  |  Residents \n",
      "4.2  |  Shinzo Abe  |  Shinzo Abe\n",
      "2  |  2 dead in   |  Bangladesh\n",
      "5  |  De Blasio   |  Bill De Bl\n",
      "0.8  |  China to r  |  China and \n",
      "3.8  |  'Around 10  |  Hundreds d\n",
      "1.6  |  Nelson Man  |  Nelson Man\n",
      "3.4  |  Cardinals   |  Cardinals \n",
      "0.6  |  AU leaders  |  Egypt Yout\n",
      "1.4  |  Today in H  |  Today in H\n",
      "3.8  |  UK alert o  |  West raise\n",
      "1.2  |  Israelis,   |  Israeli la\n",
      "3  |  Moderate e  |  Moderate e\n",
      "2.2  |  32 die in   |  1 Killed i\n",
      "1  |  Thai polic  |  Brazil lea\n",
      "2.2  |  Calif man   |  Calif. man\n",
      "1.8  |  Motorists   |  Cars trapp\n",
      "0  |  Chinese sh  |  Chinese st\n",
      "4.2  |  Red grapes  |  Red grapes\n",
      "0.4  |  2 police k  |  At least 2\n",
      "2.6  |  The Note's  |  The Note's\n",
      "0.6  |  Lebanon bu  |  Tunisia pr\n",
      "3.8  |  Indian pol  |  Mumbai pol\n",
      "0  |  Security t  |  Security a\n",
      "5  |  Google inv  |  Google inv\n",
      "1  |  China stri  |  Insurance \n",
      "3.2  |  Demolition  |  Demolition\n",
      "2.8  |  US confirm  |  US capture\n",
      "4.6  |  London Mar  |  London Mar\n",
      "3.4  |  Messi nets  |  Football: \n",
      "1.2  |  Ukrainian   |  Ukrainian \n",
      "4  |  Two NATO s  |  NATO soldi\n",
      "1.8  |  The Note's  |  The Note's\n",
      "4  |  Deaths con  |  Scottish p\n",
      "2.4  |  North Kore  |  North Kore\n",
      "1.8  |  Stocks clo  |  Stocks clo\n",
      "3.8  |  Afghan leg  |  Afghan pre\n",
      "1.4  |  Almagor bl  |  Yacimovich\n",
      "2.6  |  Hospital:   |  Dozens rep\n",
      "1.4  |  Clashes er  |  Coptic pri\n",
      "1.6  |  Queen pays  |  US preside\n",
      "4.8  |  Iran says   |  Iran says \n",
      "0.8  |  No radiati  |  US Congres\n",
      "3.4  |  Bangladesh  |  Bangladesh\n",
      "3.2  |  Iran leade  |  Iran Presi\n",
      "2.2  |  Singapore   |  Singapore \n",
      "1.2  |  2 Pussy Ri  |  Freed Puss\n",
      "3  |  Israel lau  |  Israel lau\n",
      "2.6  |  Floods lea  |  Heavy rain\n",
      "0.4  |  Iran Moder  |  Iran threa\n",
      "4.6  |  New World   |  One World \n",
      "4  |  Fashion de  |  Designer, \n",
      "2.8  |  New Japan   |  Japan, US \n",
      "3.8  |  Websites b  |  Websites f\n",
      "0  |  Turkish po  |  British po\n",
      "3.4  |  Guatemala   |  Held in Gu\n",
      "4.6  |  Biden call  |  Biden call\n",
      "0.8  |  4 dead aft  |  8 dead aft\n",
      "3.8  |  Cannes thi  |  Robber ste\n",
      "2.2  |  Nelson Man  |  ‘Nelson Ma\n",
      "3.2  |  US Militar  |  US militar\n",
      "5  |  Kenya Supr  |  Kenya SC u\n",
      "1.6  |  Death toll  |  Thousands \n",
      "2.2  |  Nicaragua,  |  Bolivia's \n",
      "0.8  |  5 killed,   |  3 killed, \n",
      "3.6  |  At least 3  |  Dozens of \n",
      "2.5  |  Chinese sh  |  China stoc\n",
      "3  |  Miss New Y  |  Miss New Y\n",
      "2.2  |  Four kille  |  Egypt viol\n",
      "0  |  China deta  |  Napthine t\n",
      "0.2  |  US Senator  |  SFG meetin\n",
      "3.20  |  One dead i  |  Six confir\n",
      "5.00  |  Matt Smith  |  Matt Smith\n",
      "2.20  |  Rescuers s  |  Select ima\n",
      "4.40  |  19 hurt in  |  Police: 19\n",
      "3.80  |  Another mi  |  Another mi\n",
      "1.20  |  Scores Kil  |  Turkey's P\n",
      "4.60  |  Michelle O  |  Michelle O\n",
      "3.80  |  Rosberg em  |  FORMULA 1:\n",
      "1.40  |  Brooks ple  |  Dave Lee T\n",
      "3.80  |  Protesters  |  Protesters\n",
      "0.00  |  France cle  |  Manila to \n",
      "1.60  |  Egypt arre  |  Egypt brac\n",
      "5.00  |  Israel exp  |  Israel wid\n",
      "0.40  |  Syrian for  |  Hunting fo\n",
      "2.40  |  Libya asks  |  Kerry Call\n",
      "3.20  |  Tornadoes   |  Tornadoes \n",
      "3.80  |  Margaret T  |  Margaret T\n",
      "2.60  |  One dead i  |  Six killed\n",
      "1.00  |  6.4-magnit  |  6.9-magnit\n",
      "4.00  |  Police hel  |  Helicopter\n",
      "2.00  |  Quotes fro  |  Obama shak\n",
      "2.20  |  Gunmen 'ki  |  Gunmen kil\n",
      "4.60  |  Major fire  |  Fire break\n",
      "4.40  |  China-EU 2  |  Full text:\n",
      "1.80  |  Chinese sh  |  Chinese sh\n",
      "0.00  |  Global pro  |  Iran diplo\n",
      "0.00  |  Greek poli  |  Israel pre\n",
      "2.00  |  UN Chemica  |  US Says Ch\n",
      "3.40  |  Syria agre  |  Syria FM: \n",
      "0.20  |  Fatah endo  |  Pakistan D\n",
      "2.40  |  Attack kil  |  NATO: 3 tr\n",
      "0.00  |  Food poiso  |  Baghdad ca\n",
      "0.40  |  Zoo worker  |  Indian-ori\n",
      "1.80  |  Six killed  |  12 killed \n",
      "0.80  |  17 govt em  |  At least 7\n",
      "0.40  |  At least 1  |  At least 1\n",
      "2.20  |  Israel 'bo  |  Fierce cla\n",
      "1.80  |  US, Russia  |  US and Rus\n",
      "4.60  |  Australia   |  Australia \n",
      "2.20  |  Syria oppo  |  Syria divi\n",
      "1.00  |  Internatio  |  Arrested B\n",
      "1.00  |  Obama sign  |  Obama urge\n",
      "4.20  |  Fire in Ru  |  38 feared \n",
      "0.40  |  DPRK leade  |  North Kore\n",
      "0.20  |  Chinese, R  |  Chinese pr\n",
      "0.00  |  Poll: Cory  |  Nigella La\n",
      "0.00  |  Blast on I  |  State: No \n",
      "0.00  |  Iran threa  |  Man admits\n",
      "0.60  |  Manchester  |  Manchester\n",
      "0.60  |  Foreign co  |  More carri\n",
      "4.00  |  U.S. fast-  |  Fast-food \n",
      "0.00  |  All eyes o  |  Flood aler\n",
      "0.80  |  One killed  |  Three kill\n",
      "3.20  |  Hassoun to  |  Mufti : co\n",
      "4.80  |  Hong Kong   |  Hong Kong \n",
      "4.20  |  Senate con  |  US Senate \n",
      "2.20  |  Obama to m  |  Obama to m\n",
      "1.00  |  Protests f  |  Scores 'ki\n",
      "3.60  |  China yuan  |  China yuan\n",
      "1.60  |  Israelis a  |  Israeli ar\n",
      "3.00  |  Japanese p  |  Japan mayo\n",
      "2.40  |  4 killed,   |  43 killed,\n",
      "0.40  |  Iran backs  |  Iran says \n",
      "0.00  |  Thai gover  |  Philippine\n",
      "5.00  |  Colorado G  |  Colorado g\n",
      "1.20  |  Pakistani   |  Pakistani \n",
      "1.40  |  House blaz  |  US drone s\n",
      "4.00  |  FAA lifts   |  FAA lifts \n",
      "4.60  |  World's ol  |  Japan: Wor\n",
      "3.40  |  Pakistanis  |  Pakistan h\n",
      "0.40  |  Algeria pr  |  Bulgarian \n",
      "0.00  |  Yemen: Fig  |  Janet Yell\n",
      "4.80  |  Pakistan's  |  Pakistan's\n"
     ]
    }
   ],
   "source": [
    "for row in rows:\n",
    "    if len(row) == 7:\n",
    "        print(f'{row[4][:10]}  |  {row[5][:10]}  |  {row[6][:10]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63e2c4b6",
   "metadata": {},
   "source": [
    "All good!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 545,
   "id": "4141b0ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = []\n",
    "\n",
    "dev_s1 = []\n",
    "dev_s2 = []\n",
    "dev_scores = []\n",
    "\n",
    "for row in rows:\n",
    "    examples.append(\n",
    "        InputExample(\n",
    "            texts=[row[5].strip(), row[6].strip()],\n",
    "            label=float(row[4])))\n",
    "    \n",
    "    dev_s1.append(row[5].strip())\n",
    "    dev_s2.append(row[6].strip())\n",
    "    dev_scores.append(float(row[4].strip()))\n",
    "\n",
    "dev_dl = DataLoader(examples, shuffle=True, batch_size=batch_size)\n",
    "\n",
    "# Because EmbeddingSimilarityEvaluator won't accept a DataLoader\n",
    "dev_examples = {\n",
    "      'sentences1': dev_s1\n",
    "    , 'sentences2': dev_s2\n",
    "    , 'scores': dev_scores }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 546,
   "id": "fc0fdbe4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1500"
      ]
     },
     "execution_count": 546,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Should be 1500\n",
    "len(examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22fa5d19",
   "metadata": {},
   "source": [
    "### Test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 547,
   "id": "d644f4cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(test_csv) as f:\n",
    "    lines = f.readlines()\n",
    "    lines = [l.strip() for l in lines]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 548,
   "id": "7f150894",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1379"
      ]
     },
     "execution_count": 548,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Should be 1379\n",
    "len(lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 549,
   "id": "e85be2e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = [l.split('\\t') for l in lines]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 550,
   "id": "94f59b03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{7: 1095, 9: 284}"
      ]
     },
     "execution_count": 550,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row_type_counts = {}\n",
    "\n",
    "for row in rows:\n",
    "    if len(row) not in row_type_counts.keys():\n",
    "        row_type_counts[len(row)] = 1\n",
    "    else:\n",
    "        row_type_counts[len(row)] += 1\n",
    "\n",
    "row_type_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47bbac02",
   "metadata": {},
   "source": [
    "Assuming that the data is clean enough and the rows with 7 columns are all correctly filled, we will just quickly check the relevant columns of those with 9 fields, just to make sure our data is clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 551,
   "id": "4a65c977",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3  |  I remained  |  I remained\n",
      "3  |  In the US,  |  It really \n",
      "0  |  There's al  |  There is a\n",
      "0  |  You also i  |  You can do\n",
      "2  |  I did this  |  I have thi\n",
      "0  |  You just h  |  You may wa\n",
      "5  |  You do not  |  You don't \n",
      "3  |  You should  |  You should\n",
      "2  |  You should  |  You should\n",
      "0  |  You need t  |  You have t\n",
      "2  |  It depends  |  i think it\n",
      "5  |  You can do  |  Yes, you c\n",
      "1  |  You should  |  You can do\n",
      "4  |  You have t  |  You have t\n",
      "4  |  I have few  |  I have two\n",
      "1  |  You want t  |  You will h\n",
      "1  |  if you don  |  The key th\n",
      "1  |  Unfortunat  |  My answer \n",
      "4  |  As soon as  |  Start them\n",
      "1  |  You just h  |  It depends\n",
      "5  |  The answer  |  The answer\n",
      "4  |  To give th  |  I'll answe\n",
      "4  |  Unfortunat  |  Sorry, I d\n",
      "4  |  The rule -  |  I always g\n",
      "4  |  This is no  |  This sound\n",
      "3  |  Yes, it's   |  It's a goo\n",
      "4  |  It probabl  |  It depends\n",
      "0  |  It's not a  |  It's a goo\n",
      "0  |  It's prett  |  It's much \n",
      "1  |  Yes, there  |  Yes, that \n",
      "4  |  Have you t  |  Have you t\n",
      "5  |  You guys a  |  You are ma\n",
      "2  |  You don't   |  You have n\n",
      "4  |  There are   |  There are \n",
      "0  |  Work into   |  It seems t\n",
      "2  |  You can bu  |  You can bu\n",
      "1  |  The coffee  |  One idea i\n",
      "0  |  There are   |  There are \n",
      "2  |  You might   |  My advice \n",
      "5  |  It's not a  |  It is not \n",
      "1  |  You just h  |  They can, \n",
      "3  |  Yes, you s  |  Yes, you s\n",
      "2  |  From what   |  Can you do\n",
      "5  |  Take a loo  |  Take a loo\n",
      "2  |  I'd say it  |  I'd say it\n",
      "1  |  Some of wh  |  Not much y\n",
      "5  |  This is no  |  But it is \n",
      "0  |  The answer  |  You have a\n",
      "5  |  I was in a  |  I had a si\n",
      "5  |  I've had t  |  I had this\n",
      "0  |  There is n  |  There is n\n",
      "2  |  I am not s  |  I am not s\n",
      "4  |  It depends  |  In other w\n",
      "0  |  You need t  |  Yes, you s\n",
      "2  |  You are no  |  No you are\n",
      "3  |  It depends  |  I guess it\n",
      "0  |  You just h  |  It is his \n",
      "3  |  It depends  |  It depends\n",
      "1  |  This is a   |  This is a \n",
      "3  |  This is a   |  This sound\n",
      "2  |  It very mu  |  I think it\n",
      "4  |  I have the  |  I have had\n",
      "0  |  No it does  |  No it is n\n",
      "4  |  My answer   |  I think th\n",
      "4  |  I don't th  |  I don't th\n",
      "2  |  It depends  |  It depends\n",
      "2  |  There's no  |  There's no\n",
      "5  |  You answer  |  You have a\n",
      "2  |  You just h  |  Th answer \n",
      "2  |  This is no  |  This is pr\n",
      "0  |  The best t  |  My recomme\n",
      "2  |  I'd say it  |  I'd say it\n",
      "0  |  You don't   |  You don't \n",
      "2  |  I have the  |  I have the\n",
      "4  |  It's not a  |  I do not t\n",
      "4  |  You are on  |  You are ri\n",
      "1  |  This doesn  |  This is a \n",
      "4  |  How should  |  So how sho\n",
      "4  |  Does this   |  Does this \n",
      "2  |  You can us  |  You can st\n",
      "2  |  It really   |  It depends\n",
      "2  |  I've had t  |  I've had t\n",
      "0  |  You need t  |  You should\n",
      "3  |  It depends  |  It depends\n",
      "1  |  Yes, you h  |  You are no\n",
      "5  |  I don't se  |  I don't se\n",
      "1  |  Hope this   |  If what yo\n",
      "0  |  The best t  |  The best t\n",
      "2  |  It depends  |  It depends\n",
      "2  |  You just h  |  You have t\n",
      "0  |  You PROBAB  |  Saying \"th\n",
      "3  |  It really   |  It doesn't\n",
      "3  |  You don't   |  You don't \n",
      "3  |  I think yo  |  I think yo\n",
      "5  |  It makes a  |  No, it mak\n",
      "5  |  I think it  |  I think it\n",
      "4  |  I'm going   |  I'm going \n",
      "1  |  You just h  |  Yes, you c\n",
      "0  |  You should  |  You should\n",
      "5  |  There's no  |  I'm afraid\n",
      "0  |  What kind   |  What kind \n",
      "4  |  It depends  |  I guess it\n",
      "5  |  You answer  |  You've ans\n",
      "3  |  I don't th  |  I don't th\n",
      "3  |  There are   |  There are \n",
      "3  |  This is th  |  Something \n",
      "2  |  That's wha  |  Yes, I bel\n",
      "1  |  If you are  |  If not, do\n",
      "0  |  It's all i  |  It's all a\n",
      "2  |  If you are  |  If you do \n",
      "0  |  You need t  |  You have t\n",
      "2  |  I would sa  |  Personally\n",
      "5  |  Can you do  |  Can you do\n",
      "3  |  Sure, I've  |  I've done \n",
      "3  |  You can us  |  You can do\n",
      "5  |  How do you  |  How to do \n",
      "3  |  I've locat  |  I found so\n",
      "2  |  I would sa  |  I would sa\n",
      "1  |  Unfortunat  |  The answer\n",
      "1  |  If you are  |  You don't,\n",
      "5  |  It's also   |  It's defin\n",
      "1  |  It's not a  |  It's a goo\n",
      "2  |  This answe  |  This answe\n",
      "4  |  There are   |  There are \n",
      "0  |  You will w  |  You will a\n",
      "2  |  It depends  |  It depends\n",
      "1  |  There are   |  There are \n",
      "3  |  You don't   |  Since you \n",
      "1  |  Vietnamese  |  Nepalese c\n",
      "1  |  It depends  |  It depends\n",
      "5  |  I don't se  |  I don't se\n",
      "5  |  There's no  |  I really d\n",
      "5  |  You got it  |  Yes you go\n",
      "0  |  You just h  |  Remember t\n",
      "4  |  How do you  |  How should\n",
      "4  |  I was in a  |  I am in a \n",
      "0  |  What is yo  |  What is yo\n",
      "0  |  You don't   |  You don't \n",
      "2  |  That is yo  |  Work with \n",
      "4  |  It depends  |  It's up to\n",
      "5  |  There are   |  A couple t\n",
      "3  |  It depends  |  It depends\n",
      "3  |  The wood p  |  It's just \n",
      "4  |  Well, I wo  |  I wouldn't\n",
      "1  |  This is a   |  This is a \n",
      "0  |  The answer  |  To directl\n",
      "2  |  Yes, there  |  Valid reas\n",
      "2  |  I think it  |  It depends\n",
      "5  |  Can you do  |  So, CAN yo\n",
      "4  |  I think th  |  I don't th\n",
      "5  |  There are   |  There are \n",
      "1  |  The key is  |  You may wa\n",
      "0  |  You don't   |  You don't \n",
      "2  |  I think yo  |  You've pre\n",
      "0  |  There are   |  There are \n",
      "2  |  You burn m  |  The more y\n",
      "1  |  It depends  |  It depends\n",
      "5  |  You got it  |  You've got\n",
      "5  |  You answer  |  You pretty\n",
      "3  |  I have the  |  I had the \n",
      "0  |  Yes, there  |  Yes, there\n",
      "0  |  They can b  |  Yes, altho\n",
      "2  |  I would lo  |  I would lo\n",
      "5  |  You are on  |  You're on \n",
      "1  |  If you hav  |  You should\n",
      "3  |  Yes it is   |  The simple\n",
      "3  |  I don't th  |  I don't th\n",
      "2  |  It looks t  |  It is up t\n",
      "1  |  Problem is  |  The Joker \n",
      "1  |  In the fir  |  So I don't\n",
      "2  |  It is impo  |  It is diff\n",
      "2  |  You need t  |  This gives\n",
      "4  |  Work into   |  You work o\n",
      "0  |  No, you do  |  You do not\n",
      "1  |  Yes, it is  |  Yes, it is\n",
      "1  |  You just h  |  So if that\n",
      "2  |  Yes, you s  |  Yes, you s\n",
      "2  |  I think it  |  I think th\n",
      "0  |  You don't   |  Other than\n",
      "1  |  Yes, there  |  There's no\n",
      "5  |  What are y  |  What are y\n",
      "2  |  I'm going   |  I'm going \n",
      "2  |  If your lo  |  You do not\n",
      "5  |  A few more  |  A few thin\n",
      "2  |  Now you do  |  You don't \n",
      "5  |  I'm afraid  |  I think th\n",
      "0  |  It is more  |  Point is: \n",
      "4  |  It varies   |  This varie\n",
      "4  |  I have the  |  I have the\n",
      "4  |  I have the  |  I have the\n",
      "0  |  You are on  |  You're rig\n",
      "2  |  Primer/pai  |  Nope that \n",
      "2  |  The best t  |  The best t\n",
      "5  |  It's also   |  It's mostl\n",
      "2  |  I was in a  |  Personally\n",
      "3  |  You can us  |  Yes, you c\n",
      "4  |  You have t  |  You have t\n",
      "1  |  It depends  |  It depends\n",
      "5  |  I had the   |  I have jus\n",
      "2  |  Sometime i  |  You could \n",
      "1  |  You don't   |  They just \n",
      "2  |  Also it is  |  It is impo\n",
      "1  |  I think it  |  I think it\n",
      "4  |  It's not a  |  No, it's n\n",
      "5  |  I'm thinki  |  I think it\n",
      "1  |  It depends  |  You need t\n",
      "4  |  You have a  |  This is a \n",
      "2  |  Unfortunat  |  If the con\n",
      "4  |  Unfortunat  |  This canno\n",
      "2  |  I would pe  |  I would re\n",
      "4  |  I think th  |  I don't th\n",
      "5  |  It's not a  |  Not a good\n",
      "0  |  You don't   |  You don't \n",
      "0  |  You have t  |  You're pro\n",
      "5  |  I had the   |  I have exa\n",
      "4  |  My answer   |  The answer\n",
      "1  |  You should  |  You should\n",
      "4  |  I had the   |  I had the \n",
      "4  |  Hope this   |  Is this th\n",
      "5  |  There are   |  I think th\n",
      "5  |  It's prett  |  It is up t\n",
      "4  |  There are   |  There are \n",
      "1  |  The answer  |  The answer\n",
      "0  |  It is impo  |  This is a \n",
      "3  |  There are   |  There are \n",
      "4  |  It depends  |  It depends\n",
      "3  |  There are   |  There are \n",
      "5  |  There are   |  There are \n",
      "5  |  So think i  |  Think of i\n",
      "5  |  Maybe this  |  Maybe this\n",
      "1  |  It depends  |  They can, \n",
      "0  |  Microwave   |  Your best \n",
      "4  |  In the US,  |  This will \n",
      "1  |  It's not a  |  It's not j\n",
      "4  |  There are   |  There are \n",
      "3  |  I've had t  |  I had this\n",
      "5  |  This is no  |  Not a good\n",
      "5  |  I realized  |  I know thi\n",
      "0  |  Yes, there  |  Yes, there\n",
      "1  |  Yes, there  |  To my know\n",
      "0  |  You don't   |  You don't \n",
      "0  |  There is n  |  There is n\n",
      "0  |  Yes, you s  |  Yes, you s\n",
      "0  |  You should  |  You don't \n",
      "4  |  I've had t  |  I had this\n",
      "5  |  You may ha  |  You have t\n",
      "1  |  My answer   |  This is a \n",
      "1  |  The key th  |  It really \n",
      "4  |  Not a dire  |  Not a real\n",
      "1  |  There are   |  There are \n",
      "5  |  I agree wi  |  I agree wi\n",
      "3  |  The answer  |  If the ans\n",
      "5  |  It rather   |  That depen\n",
      "5  |  This is a   |  This is a \n",
      "2  |  US drone s  |  US drone s\n",
      "5  |  Russian op  |  Russian Op\n",
      "5  |  Senate con  |  Senate con\n",
      "5  |  Israel con  |  Israel off\n",
      "5  |  Jurors fin  |  Jodi Arias\n",
      "4  |  Twitter re  |  Twitter sa\n",
      "3  |  Goa buildi  |  Indian bui\n",
      "3  |  Pak religi  |  CII endors\n",
      "5  |  Mt Gox rev  |  Mt Gox fin\n",
      "4  |  More than   |  Hundreds b\n",
      "2  |  Redford to  |  Redford ad\n",
      "1  |  Obama hold  |  UK urges c\n",
      "4  |  South Kore  |  South Kore\n",
      "0  |  Tycoon cla  |  Polling st\n",
      "2  |  Syria agre  |  UK's Camer\n",
      "5  |  AbbVie Sou  |  AbbVie coo\n",
      "3  |  Militant a  |  Militant a\n",
      "2  |  13 killed   |  6 killed i\n",
      "2  |  Rev's Earl  |  Rev.'s Ear\n",
      "0  |  Pope canon  |  Sweden rec\n",
      "0  |  Funeral of  |  Funeral of\n",
      "1  |  Waba emerg  |  Waiting fo\n",
      "1  |  5 US airst  |  Kurds see \n",
      "1  |  US Senate   |  US scienti\n",
      "2  |  Editors fr  |  Media free\n",
      "0  |  Philippine  |  Philippine\n",
      "1  |  Israel bar  |  Two-state \n",
      "1  |  How much d  |  Lawmakers \n",
      "0  |  Obama Stru  |  Myanmar St\n",
      "0  |  South Kore  |  North Kore\n"
     ]
    }
   ],
   "source": [
    "for row in rows:\n",
    "    if len(row) == 9:\n",
    "        print(f'{row[4][:10]}  |  {row[5][:10]}  |  {row[6][:10]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29596fc4",
   "metadata": {},
   "source": [
    "Everything seems OK. Let's just check the first 20 rows with 7 columns also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 552,
   "id": "4db124e0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.500  |  A girl is   |  A girl is \n",
      "3.600  |  A group of  |  A group of\n",
      "5.000  |  One woman   |  A woman me\n",
      "4.200  |  A man is c  |  A man is s\n",
      "1.500  |  A man is p  |  A man is p\n",
      "1.800  |  A woman is  |  A woman is\n",
      "3.500  |  A man is r  |  A man is r\n",
      "2.200  |  A man is p  |  A man is p\n",
      "2.200  |  A man is p  |  A lady is \n",
      "1.714  |  A man is p  |  A man is p\n",
      "1.714  |  A man is p  |  A man is p\n",
      "5.000  |  A man is c  |  A man cuts\n",
      "0.600  |  A man is c  |  A man is t\n",
      "4.400  |  A man is s  |  A man is c\n",
      "2.000  |  A man is s  |  A man is s\n",
      "1.800  |  A man is p  |  A man is p\n",
      "4.400  |  A baby pan  |  A panda sl\n",
      "3.600  |  A man is s  |  A man is p\n",
      "3.600  |  A man atta  |  A man slap\n",
      "1.200  |  A man is d  |  A man is r\n"
     ]
    }
   ],
   "source": [
    "for row in rows[:20]:\n",
    "    if len(row) == 7:\n",
    "        print(f'{row[4][:10]}  |  {row[5][:10]}  |  {row[6][:10]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84066ebb",
   "metadata": {},
   "source": [
    "All good!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 553,
   "id": "7aae8523",
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = []\n",
    "\n",
    "test_s1 = []\n",
    "test_s2 = []\n",
    "test_scores = []\n",
    "\n",
    "for row in rows:\n",
    "    examples.append(\n",
    "        InputExample(\n",
    "            texts=[row[5].strip(), row[6].strip()],\n",
    "            label=float(row[4])))\n",
    "    \n",
    "    test_s1.append(row[5].strip())\n",
    "    test_s2.append(row[6].strip())\n",
    "    test_scores.append(float(row[4].strip()))\n",
    "\n",
    "test_dl = DataLoader(examples, shuffle=True, batch_size=batch_size)\n",
    "\n",
    "# Because EmbeddingSimilarityEvaluator won't accept a DataLoader\n",
    "test_examples = {\n",
    "      'sentences1': test_s1\n",
    "    , 'sentences2': test_s2\n",
    "    , 'scores': test_scores }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 554,
   "id": "832bf381",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1379"
      ]
     },
     "execution_count": 554,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Should be 1379\n",
    "len(examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d736bd40",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 580,
   "id": "a18b30b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = f'S-BERT-STS-Trained-batch_size-{batch_size}-epochs-{epochs}-warmup_steps-{warmup_steps}'\n",
    "model_path = 'models' + '/' + model_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79d8fbc4",
   "metadata": {},
   "source": [
    "Either build a model..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 571,
   "id": "07941ec9",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "# word_embedding_model = Transformer('bert-base-uncased')  # [optional] max_seq_length=256\n",
    "# mean_pooling = Pooling(word_embedding_model.get_word_embedding_dimension(), pooling_mode='mean')\n",
    "# model = SentenceTransformer(modules=[word_embedding_model, mean_pooling])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd6ab36",
   "metadata": {},
   "source": [
    "... or load a previously saved one from disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 578,
   "id": "5b8e0180",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = SentenceTransformer(model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6594fbe",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9774cfa",
   "metadata": {},
   "source": [
    "### S-BERT Regression Objective"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "248ddd55",
   "metadata": {},
   "source": [
    "Documentation for the Cosine Similarity Loss:  \n",
    "https://www.sbert.net/docs/package_reference/losses.html#sentence_transformers.losses.CosineSimilarityLoss\n",
    "\n",
    "The loss that will be minimized is the following:\n",
    "\n",
    "$$ \\mathrm{Loss} = \\lVert  \\,\\,  \\mathrm{input \\, label} - \\mathrm{cos\\_score\\_transformation} \\, ( \\, \\mathrm{cosine\\_sim} \\, (u,v) \\,)  \\,\\,  \\rVert_2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9ab41bb",
   "metadata": {},
   "source": [
    "The similarity score in the STS benchmark goes from 0 to 5, while cosine similarity goes from -1 to 1.  \n",
    "We need to therefore map the cosine similarity score back to the 0-5 range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 563,
   "id": "d9087795",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ScoreMap(nn.Module):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super(ScoreMap, self).__init__()\n",
    "\n",
    "    def forward(self, input: Tensor) -> Tensor:\n",
    "        \"\"\"Maps input from [-1, 1] to [0, 5].\"\"\"\n",
    "        return 5/2 * input + 5/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 564,
   "id": "a674bb2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loss = CosineSimilarityLoss(\n",
    "                 model=model,\n",
    "                 loss_fct=nn.MSELoss(),\n",
    "                 cos_score_transformation=ScoreMap())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea8fbe92",
   "metadata": {},
   "source": [
    "### Model Training (Fine-Tuning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e6975e2",
   "metadata": {},
   "source": [
    "Not necessary to call `fit()`, if you've loaded a fine-tuned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4aa783ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model.fit(train_objectives=[(train_dl, train_loss)],\n",
    "#           epochs=epochs,\n",
    "#           warmup_steps=warmup_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "483ae336",
   "metadata": {},
   "source": [
    "Save the model to disk for later use if you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 581,
   "id": "496c2c7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model.save(path=model_path, model_name=model_name, create_model_card=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 565,
   "id": "f3dad5e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_evaluator = EmbeddingSimilarityEvaluator(\n",
    "    sentences1=dev_examples['sentences1'],\n",
    "    sentences2=dev_examples['sentences2'],\n",
    "    scores=dev_examples['scores'],\n",
    "    batch_size=batch_size,\n",
    "    main_similarity=SimilarityFunction.COSINE,  # {COSINE, EUCLIDEAN, MANHATTAN, DOT_PRODUCT}\n",
    "    show_progress_bar=True,\n",
    "    write_csv=False)\n",
    "\n",
    "# The SimilarityFunction values come from:\n",
    "# https://github.com/UKPLab/sentence-transformers/blob/46a149433fe9af0851f7fa6f9bf37b5ffa2c891c/sentence_transformers/evaluation/EmbeddingSimilarityEvaluator.py#L120"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c38b2e9",
   "metadata": {},
   "source": [
    "### Evaluation: Spearman Rank Correlation on Dev Set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f09972f1",
   "metadata": {},
   "source": [
    "In order to obtain the embeddings necessary for the evaluation, the model will run over the data twice: once for the first sentences, and once for the second sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 569,
   "id": "9aa81cd2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spearman (dev set): 73.03830917146799\n"
     ]
    }
   ],
   "source": [
    "dev_spearman_rho = model.evaluate(dev_evaluator)\n",
    "print(f'Spearman (dev set): {dev_spearman_rho * 100}')  # For consistency with the paper"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "924da968",
   "metadata": {},
   "source": [
    "# Final Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be215b48",
   "metadata": {},
   "source": [
    "### Spearman Rank Correlation on Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 567,
   "id": "e25edc12",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_evaluator = EmbeddingSimilarityEvaluator(\n",
    "    sentences1=test_examples['sentences1'],\n",
    "    sentences2=test_examples['sentences2'],\n",
    "    scores=test_examples['scores'],\n",
    "    batch_size=batch_size,\n",
    "    main_similarity=SimilarityFunction.COSINE,  # {COSINE, EUCLIDEAN, MANHATTAN, DOT_PRODUCT}\n",
    "    show_progress_bar=True,\n",
    "    write_csv=False)\n",
    "\n",
    "# The SimilarityFunction values come from:\n",
    "# https://github.com/UKPLab/sentence-transformers/blob/46a149433fe9af0851f7fa6f9bf37b5ffa2c891c/sentence_transformers/evaluation/EmbeddingSimilarityEvaluator.py#L120"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 570,
   "id": "33e8dec0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spearman (test set): 65.57546300121835\n"
     ]
    }
   ],
   "source": [
    "test_spearman_rho = model.evaluate(test_evaluator)\n",
    "print(f'Spearman (test set): {test_spearman_rho * 100}')  # For consistency with the paper"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb991d35",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "599c70cf",
   "metadata": {},
   "source": [
    "All else being equal, for `batch_size` in {16, 32}, we have the following Spearman scores:\n",
    " - dev set: {73.03830917146799, 72.19737929737312}\n",
    " - test set: {65.57546300121835, 64.55123444928913}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}