Automatic text summarizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1b46cc9a",
   "metadata": {},
   "source": [
    "# Automatic Text Summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81a391d5",
   "metadata": {},
   "source": [
    "Summarization refers to the task of creating a short summary that captures the main ideas\n",
    "of an input text. For my project, we focus on abstractive summarization, which generates\n",
    "the summary through paraphrase. By comparison, extractive summarization works by\n",
    "extracting only words found in the input. Models for abstractive summarization fall under a larger deep learning category called\n",
    "sequence-to-sequence models, which map from an input sequence to a target sequence\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8673678",
   "metadata": {},
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef7cd3de",
   "metadata": {},
   "source": [
    "We trained our model on new summary dataset here is the [link](https://www.kaggle.com/datasets/sunnysai12345/news-summary). It has two files. One has 4100+ examples another one has 90,000+ examples to train along with labels. We trained our model to predict the headline given the first sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4c3c193",
   "metadata": {},
   "source": [
    "## Import the packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fb3b7110",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.keras\n",
    "import pandas as pd\n",
    "import re\n",
    "import numpy as np\n",
    "import string\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66027a19",
   "metadata": {},
   "source": [
    "## Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9e5a505e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.read_csv(r\"C:\\Users\\Sai Pavan Vulasi\\Desktop\\text_dataset\\news_summary.csv\",encoding='iso-8859-1',usecols=['headlines','text'])\n",
    "df2 = pd.read_csv(r\"C:\\Users\\Sai Pavan Vulasi\\Desktop\\text_dataset\\news_summary_more.csv\",encoding='iso-8859-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "413d6ec2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Daman &amp; Diu revokes mandatory Rakshabandhan in...</td>\n",
       "      <td>The Administration of Union Territory Daman an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Malaika slams user who trolled her for 'divorc...</td>\n",
       "      <td>Malaika Arora slammed an Instagram user who tr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>'Virgin' now corrected to 'Unmarried' in IGIMS...</td>\n",
       "      <td>The Indira Gandhi Institute of Medical Science...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aaj aapne pakad liya: LeT man Dujana before be...</td>\n",
       "      <td>Lashkar-e-Taiba's Kashmir commander Abu Dujana...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Hotel staff to get training to spot signs of s...</td>\n",
       "      <td>Hotels in Maharashtra will train their staff t...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           headlines                                               text\n",
       "0  Daman & Diu revokes mandatory Rakshabandhan in...  The Administration of Union Territory Daman an...\n",
       "1  Malaika slams user who trolled her for 'divorc...  Malaika Arora slammed an Instagram user who tr...\n",
       "2  'Virgin' now corrected to 'Unmarried' in IGIMS...  The Indira Gandhi Institute of Medical Science...\n",
       "3  Aaj aapne pakad liya: LeT man Dujana before be...  Lashkar-e-Taiba's Kashmir commander Abu Dujana...\n",
       "4  Hotel staff to get training to spot signs of s...  Hotels in Maharashtra will train their staff t..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "44443515",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4514, 2)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "00b1b126",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>upGrad learner switches to career in ML &amp; Al w...</td>\n",
       "      <td>Saurav Kant, an alumnus of upGrad and IIIT-B's...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Delhi techie wins free food from Swiggy for on...</td>\n",
       "      <td>Kunal Shah's credit card bill payment platform...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>New Zealand end Rohit Sharma-led India's 12-ma...</td>\n",
       "      <td>New Zealand defeated India by 8 wickets in the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aegon life iTerm insurance plan helps customer...</td>\n",
       "      <td>With Aegon Life iTerm Insurance plan, customer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Have known Hirani for yrs, what if MeToo claim...</td>\n",
       "      <td>Speaking about the sexual harassment allegatio...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           headlines                                               text\n",
       "0  upGrad learner switches to career in ML & Al w...  Saurav Kant, an alumnus of upGrad and IIIT-B's...\n",
       "1  Delhi techie wins free food from Swiggy for on...  Kunal Shah's credit card bill payment platform...\n",
       "2  New Zealand end Rohit Sharma-led India's 12-ma...  New Zealand defeated India by 8 wickets in the...\n",
       "3  Aegon life iTerm insurance plan helps customer...  With Aegon Life iTerm Insurance plan, customer...\n",
       "4  Have known Hirani for yrs, what if MeToo claim...  Speaking about the sexual harassment allegatio..."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cdfbacd0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(98401, 2)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2efd6c87",
   "metadata": {},
   "source": [
    "## Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c3815a7",
   "metadata": {},
   "source": [
    "The aim of the preprocessing step is to reduce the\n",
    "dimensionality of the representation space, and it normally includes: (i) stop-word\n",
    "elimination – common words with no semantics and which do not aggregate relevant\n",
    "information to the task (e.g., “the”, “a”) are eliminated;; (ii) case folding: consists of\n",
    "converting all the characters to the same kind of letter case - either upper case or\n",
    "lower case; (iii) stemming: syntactically-similar words, such as plurals, verbal\n",
    "variations, etc. are considered similar; the purpose of this procedure is to obtain the\n",
    "stem or radix of each word, which emphasize its semantics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a7ae0028",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Daman &amp; Diu revokes mandatory Rakshabandhan in...</td>\n",
       "      <td>The Administration of Union Territory Daman an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Malaika slams user who trolled her for 'divorc...</td>\n",
       "      <td>Malaika Arora slammed an Instagram user who tr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>'Virgin' now corrected to 'Unmarried' in IGIMS...</td>\n",
       "      <td>The Indira Gandhi Institute of Medical Science...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aaj aapne pakad liya: LeT man Dujana before be...</td>\n",
       "      <td>Lashkar-e-Taiba's Kashmir commander Abu Dujana...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Hotel staff to get training to spot signs of s...</td>\n",
       "      <td>Hotels in Maharashtra will train their staff t...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           headlines                                               text\n",
       "0  Daman & Diu revokes mandatory Rakshabandhan in...  The Administration of Union Territory Daman an...\n",
       "1  Malaika slams user who trolled her for 'divorc...  Malaika Arora slammed an Instagram user who tr...\n",
       "2  'Virgin' now corrected to 'Unmarried' in IGIMS...  The Indira Gandhi Institute of Medical Science...\n",
       "3  Aaj aapne pakad liya: LeT man Dujana before be...  Lashkar-e-Taiba's Kashmir commander Abu Dujana...\n",
       "4  Hotel staff to get training to spot signs of s...  Hotels in Maharashtra will train their staff t..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#concatenate two dataframes to get total data\n",
    "df = pd.concat([df1,df2],axis=0,ignore_index=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d71d0170",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(102915, 2)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8324b2c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop duplicates and null values\n",
    "df.dropna(inplace=True)\n",
    "df.drop_duplicates(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f1c8949d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100266, 2)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1b4c656a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset size : 100266\n"
     ]
    }
   ],
   "source": [
    "print(f'Dataset size : {df.shape[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "13cf5a98",
   "metadata": {},
   "outputs": [],
   "source": [
    "# text cleaning\n",
    "\n",
    "def cleantext(text):\n",
    "    text = str(text)\n",
    "    text=text.split()\n",
    "    words=[]\n",
    "    for t in text:\n",
    "        if t.isalpha():\n",
    "            words.append(t)\n",
    "    text=\" \".join(words)\n",
    "    text=text.lower()\n",
    "    text=re.sub(r\"what's\",\"what is \",text)\n",
    "    text=re.sub(r\"it's\",\"it is \",text)\n",
    "    text=re.sub(r\"\\'ve\",\" have \",text)\n",
    "    text=re.sub(r\"i'm\",\"i am \",text)\n",
    "    text=re.sub(r\"\\'re\",\" are \",text)\n",
    "    text=re.sub(r\"n't\",\" not \",text)\n",
    "    text=re.sub(r\"\\'d\",\" would \",text)\n",
    "    text=re.sub(r\"\\'s\",\"s\",text)\n",
    "    text=re.sub(r\"\\'ll\",\" will \",text)\n",
    "    text=re.sub(r\"can't\",\" cannot \",text)\n",
    "    text=re.sub(r\" e g \",\" eg \",text)\n",
    "    text=re.sub(r\"e-mail\",\"email\",text)\n",
    "    text=re.sub(r\"9\\\\/11\",\" 911 \",text)\n",
    "    text=re.sub(r\" u.s\",\" american \",text)\n",
    "    text=re.sub(r\" u.n\",\" united nations \",text)\n",
    "    text=re.sub(r\"\\n\",\" \",text)\n",
    "    text=re.sub(r\":\",\" \",text)\n",
    "    text=re.sub(r\"-\",\" \",text)\n",
    "    text=re.sub(r\"\\_\",\" \",text)\n",
    "    text=re.sub(r\"\\d+\",\" \",text)\n",
    "    text=re.sub(r\"[$#@%&*!~?%{}()]\",\" \",text)\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2d96dbc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in df.columns:\n",
    "    df[col] = df[col].apply(lambda x: cleantext(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "be8f477a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>daman diu revokes mandatory rakshabandhan in o...</td>\n",
       "      <td>the administration of union territory daman an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>malaika slams user who trolled her for rich</td>\n",
       "      <td>malaika arora slammed an instagram user who tr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>now corrected to in form</td>\n",
       "      <td>the indira gandhi institute of medical science...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aaj aapne pakad let man dujana before being ki...</td>\n",
       "      <td>kashmir commander abu who was killed by securi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>hotel staff to get training to spot signs of s...</td>\n",
       "      <td>hotels in maharashtra will train their staff t...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           headlines                                               text\n",
       "0  daman diu revokes mandatory rakshabandhan in o...  the administration of union territory daman an...\n",
       "1        malaika slams user who trolled her for rich  malaika arora slammed an instagram user who tr...\n",
       "2                           now corrected to in form  the indira gandhi institute of medical science...\n",
       "3  aaj aapne pakad let man dujana before being ki...  kashmir commander abu who was killed by securi...\n",
       "4  hotel staff to get training to spot signs of s...  hotels in maharashtra will train their staff t..."
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "928db258",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.DataFrame()\n",
    "data['text'] = df['text']\n",
    "data['summary'] = df['headlines']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f7c8330",
   "metadata": {},
   "source": [
    "Remember to add the START and END special tokens at the beginning and end of the summary. Here, I have chosen sostok and eostok as START and END tokens. The beginning-of-sentence token serves as an instruction to the decoder to start decoding (it needs a very first state to predict what the next first token is). The end-of-sentence token is an instruction for the decoding algorithm to stop generating more tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "177724a5",
   "metadata": {},
   "source": [
    "## Understanding distribution of sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9c09d22",
   "metadata": {},
   "source": [
    "Here, we will analyze the length of the reviews and the summary to get an overall idea about the distribution of length of the text. This will help us fix the maximum length of the sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "230b60d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9999900265294317\n"
     ]
    }
   ],
   "source": [
    "text_count = 0\n",
    "for text in data['text']:\n",
    "    if len(text.split())<60:\n",
    "        text_count += 1\n",
    "print(text_count/len(data['text']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3de49ace",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9999601061177269\n"
     ]
    }
   ],
   "source": [
    "summary_count = 0\n",
    "for summary in data['summary']:\n",
    "    if len(summary.split())<15:\n",
    "        summary_count+=1\n",
    "print(summary_count/len(data['summary']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f3126cac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEICAYAAAC9E5gJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAYJUlEQVR4nO3df5BlZZ3f8fdH8AdBERC3ZWdwh9UpLVYiPybCRrPbgsIIRkyVGi02jBQlSYm1WmFVsJJiVyWBPxRl1zVhZZbBVZFCUaK4OAG63FQCAqIgoMXIjpmZAkYdfji4aEa/+eM+rdee7p7uO933F+9X1a2+5zk/7nNPn+7PPc957nNSVUiSntqeNugKSJIGzzCQJBkGkiTDQJKEYSBJwjCQJGEYSJIwDEZOks1JXjMs25E0HgwDSZpDkn0HXYd+MQxGSJJPAy8E/keSnUnel+T4JP87yaNJvpNksi37L5P8OMlhbfrlSR5J8tLZtjOo96Txl+T9SbYl+WmS7yc5MckVST7ctcxkkq1d05uTvDfJXUmeSHJ5kokkX2vb+Z9JDmrLrkpSSc5MsqUd5/8hyb9o6z+a5K+6tv2iJDcl+Un7G/lMkgNnvPb7k9wFPNHq8YUZ7+nSJB9fzv3Wd1XlY4QewGbgNe35CuAnwCl0gv21bfr5bf6FwE3AfsDdwLtm244PH8v1AF4CbAF+t02vAl4EXAF8uGu5SWBr1/Rm4BZgoh3n24FvAUcDz2rH9QVd2yzgv7V5JwFPAl8Cfqdr/T9uy7+4/a08E3g+8A3gYzNe+9vAYe1v51DgCeDANn/ftr1jB71/l/LhmcFo+xPg+qq6vqp+VVUbgdvphAPAnwPPBb4JbAM+MZBa6qnsl3T+6R6R5OlVtbmqfrDAdf+yqh6uqm3APwC3VtWdVfUkcC2dYOj2oap6sqq+Tuef9+eqanvX+kcDVNWmqtpYVT+vqh8BHwX+eMa2Lq2qLVX1T1X1IJ3AeHObtxb4cVXdsag9MeQMg9H2e8Cb22nwo0keBV5F55MMVfX/6HwCexnwkWofa6R+qapNwHvofDDZnuSqJL+7wNUf7nr+T7NMP7uX5Vtz01Wt6epx4O+AQ2Zsa8uM6Q10PnzRfn56ge9hZBgGo6f7H/oW4NNVdWDXY/+qugggyQrgAuBvgY8keeYc25GWTVV9tqpeRefDSwEX0/nk/s+6FntBH6v0X1o9jqyqA+j8c8+MZWb+fXwJ+OdJXga8HvjMcley3wyD0fMw8Pvt+d8B/zrJyUn2SfKsdiFuZZLQOSu4HDgLeBD40BzbkZZFkpckOaF9EHmSzif0X9Fpkz8lycFJXkDn7KFfngPsBB5rH5jeu6cVWtPUNcBngW9W1f9d3ir2n2Ewev4r8J9ak9C/BU4DPgD8iM6Zwnvp/F7/lM7Fs//cmofOBM5M8q9mbifJn/X3Legp5JnARcCPgYfoHJPn02lm+Q6di7VfBz7fxzr9BXAM8BjwVeCLC1xvA3AkY9hEBBCbkSVpz5K8EPge8IKqenzQ9VlqnhlI0h4keRrwH4GrxjEIoNNfVpI0hyT707nG9kM63UrHks1EkiSbiSRJI9xMdMghh9SqVat+Pf3EE0+w//77D65CPbDO/TFXne+4444fV9XzB1Clnsw85ofdKB4r00a57tDjMT/o8TB6fRx77LHV7eabb65RY537Y646A7fXEBzLC33MPOaH3SgeK9NGue5VvR3zNhNJkgwDSZJhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJIkRHo5Co23VeV9d9DqbLzp1GWqiceJx1TvPDCRJhoEkyTCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSWGAYJNmc5O4k305yeys7OMnGJPe3nwe18iS5NMmmJHclOaZrO+va8vcnWddVfmzb/qa2bpb6jUqS5raYM4NXV9VRVbWmTZ8H3FhVq4Eb2zTA64DV7XE28EnohAdwAXAc8ArggukAacu8o2u9tT2/I0nSou1NM9FpwIb2fAPwxq7yK6vjFuDAJIcCJwMbq2pHVT0CbATWtnkHVNUtVVXAlV3bkiT1wULvZ1DA15MU8N+r6jJgoqoebPMfAiba8xXAlq51t7ay+cq3zlK+myRn0znbYGJigqmpqV/P27lz529Nj4Kncp3PPXLXotfp9XVHcT9L/bbQMHhVVW1L8jvAxiTf655ZVdWCYlm1ELoMYM2aNTU5OfnreVNTU3RPj4Kncp3f3stNSE7v7XVHcT9L/bagZqKq2tZ+bgeupdPm/3Br4qH93N4W3wYc1rX6ylY2X/nKWcolSX2yxzBIsn+S50w/B04CvgtcB0z3CFoHfLk9vw44o/UqOh54rDUn3QCclOSgduH4JOCGNu/xJMe3XkRndG1LktQHC2kmmgCubb099wU+W1V/n+Q24OokZwE/BN7Slr8eOAXYBPwMOBOgqnYk+RBwW1vug1W1oz1/J3AFsB/wtfaQJPXJHsOgqh4AXj5L+U+AE2cpL+CcOba1Hlg/S/ntwMsWUF9J0jLwG8iSJMNAkmQYSJIwDCRJGAbSnJLsk+TOJF9p04cnubUNqPj5JM9o5c9s05va/FVd2zi/lX8/ycld5Wtb2aYk5+324lKfGQbS3N4N3Nc1fTFwSVW9GHgEOKuVnwU80sovacuR5AjgrcAf0Bl88a9bwOwDfILOoI5HAG9ry0oDYxhIs0iyEjgV+FSbDnACcE1bZObgjNODNl4DnNiWPw24qqp+XlX/SOe7N69oj01V9UBV/QK4qi0rDcxCxyaSnmo+BrwPeE6bfh7waFVNj7DXPaDirwdhrKpdSR5ry68AbunaZvc6MwdtPG62Ssw3OOOwG8QAgUs1AOKoD27YS/0NA2mGJK8HtlfVHUkmB1mX+QZnHHaDGCBwqQZAHPXBDXupv2Eg7e6VwBuSnAI8CzgA+Dide3Ps284OugdUnB6EcWuSfYHnAj9h7sEZmadcGgivGUgzVNX5VbWyqlbRuQB8U1WdDtwMvKktNnNwxulBG9/Ulq9W/tbW2+hwOnfx+yad8blWt95Jz2ivcV0f3po0J88MpIV7P3BVkg8DdwKXt/LLgU8n2QTsoPPPnaq6J8nVwL3ALuCcqvolQJJ30RnJdx9gfVXd09d3Is1gGEjzqKopYKo9f4BOT6CZyzwJvHmO9S8ELpyl/Ho6I/xKQ8FmIkmSYSBJMgwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJYhFhkGSfJHcm+UqbPjzJrUk2Jfl8u30f7RZ/n2/ltyZZ1bWN81v595Oc3FW+tpVtSnLeEr4/SdICLObM4N3AfV3TFwOXVNWLgUeAs1r5WcAjrfySthxJjqBzO8A/ANYCf90CZh/gE8DrgCOAt7VlJUl9sqAwSLISOBX4VJsOcAJwTVtkA/DG9vy0Nk2bf2Jb/jTgqqr6eVX9I7CJzi0EXwFsqqoHquoXwFVtWUlSnyz0HsgfA94HPKdNPw94tKp2temtwIr2fAWwBaCqdiV5rC2/Arila5vd62yZUX7cbJVIcjZwNsDExARTU1O/nrdz587fmh4FT+U6n3vkrj0vNEOvrzuK+1nqtz2GQZLXA9ur6o4kk8teo3lU1WXAZQBr1qypycnfVGdqaoru6VHwVK7z28/76qLX2Xx6b687ivtZ6reFnBm8EnhDklOAZwEHAB8HDkyybzs7WAlsa8tvAw4DtibZF3gu8JOu8mnd68xVLknqgz1eM6iq86tqZVWtonMB+KaqOh24GXhTW2wd8OX2/Lo2TZt/U1VVK39r6210OLAa+CZwG7C69U56RnuN65bk3UmSFmSh1wxm837gqiQfBu4ELm/llwOfTrIJ2EHnnztVdU+Sq4F7gV3AOVX1S4Ak7wJuAPYB1lfVPXtRL0nSIi0qDKpqCphqzx+g0xNo5jJPAm+eY/0LgQtnKb8euH4xdZEkLR2/gSxJMgwkSYaBJAnDQJKEYSBJwjCQJGEYSJLYuy+dSQCs6mGcIUnDxTMDSZJhIEkyDCRJGAbSbpI8K8k3k3wnyT1J/qKVe99vjS3DQNrdz4ETqurlwFHA2iTH432/NcYMA2mG6tjZJp/eHoX3/dYYs2upNIv26f0O4MV0PsX/gCG77/ewG8S9p5fq3tqjft/sXupvGEizaDdeOirJgcC1wEsHVI857/s97AZx7+mlurf2qN83u5f620wkzaOqHqVzi9c/pN33u82a7b7fLPC+3/PdD1waCMNAmiHJ89sZAUn2A14L3If3/dYYs5lI2t2hwIZ23eBpwNVV9ZUk9+J9vzWmDANphqq6Czh6lnLv+62xZTORJMkwkCQZBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJJYQBh41ydJGn8LOTPwrk+SNOb2GAbe9UmSxt+CBqobhbs+jeKdicalzr3cXaoXve6rUdzPUr8tKAxG4a5Po3hnonGpcy93l+rFbHekWohR3M9Svy2qN5F3fZKk8bSQ3kTe9UmSxtxCmom865Mkjbk9hoF3fZKk8ec3kCVJhoEkyTCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIO0myWFJbk5yb5J7kry7lR+cZGOS+9vPg1p5klyaZFOSu5Ic07WtdW35+5Os6yo/NsndbZ1Lk6T/71T6DcNA2t0u4NyqOgI4HjgnyRHAecCNVbUauLFNA7wOWN0eZwOfhE54ABcAx9G5RewF0wHSlnlH13pr+/C+pDkZBtIMVfVgVX2rPf8pcB+wAjgN2NAW2wC8sT0/DbiyOm4BDkxyKHAysLGqdlTVI8BGYG2bd0BV3VJVBVzZtS1pIPYddAWkYZZkFXA0cCswUVUPtlkPARPt+QpgS9dqW1vZfOVbZymf7fXPpnO2wcTEBFNTU72/mT7buXNn3+t77pG7Fr3ObHUcRN2XUi/1NwykOSR5NvAF4D1V9Xh3s35VVZJa7jpU1WXAZQBr1qypycnJ5X7JJTM1NUW/6/v287666HU2nz65W9kg6r6Ueqm/zUTSLJI8nU4QfKaqvtiKH25NPLSf21v5NuCwrtVXtrL5ylfOUi4NjGEgzdB69lwO3FdVH+2adR0w3SNoHfDlrvIzWq+i44HHWnPSDcBJSQ5qF45PAm5o8x5Pcnx7rTO6tiUNhM1E0u5eCfw74O4k325lHwAuAq5OchbwQ+Atbd71wCnAJuBnwJkAVbUjyYeA29pyH6yqHe35O4ErgP2Ar7WHNDCGgTRDVf0vYK5+/yfOsnwB58yxrfXA+lnKbwdethfVlJaUzUSSJMNAkmQzkaQhtKqHLqLaO3s8M3CcFkkafwtpJnKcFkkac3sMA8dpkaTxt6gLyIMep0WStDwWfAF5GMZpmW/QrlEcWGpc6tzL4GC96HVfjeJ+lvptQWEw3zgtVfXgIsZpmZxRPsUixmmZb9CuURxYalzq3MvgYL2YbUCxhRjF/Sz120J6EzlOiySNuYWcGThOiySNuT2GgeO0SNL4czgKSZJhIEkyDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQWcXMbSRpHq2a5H8e5R+6a9z4dmy86dTmrNBCeGUiSDANJkmEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIM0qyfok25N8t6vs4CQbk9zffh7UypPk0iSbktyV5Jiudda15e9Psq6r/Ngkd7d1Lk2S/r5D6bcZBtLsrgDWzig7D7ixqlYDN7ZpgNcBq9vjbOCT0AkP4ALgOOAVwAXTAdKWeUfXejNfS+orw0CaRVV9A9gxo/g0YEN7vgF4Y1f5ldVxC3BgkkOBk4GNVbWjqh4BNgJr27wDquqWqirgyq5tSQPhqKXSwk1U1YPt+UPARHu+AtjStdzWVjZf+dZZyneT5Gw6ZxtMTEwwNTW1d++gj3bu3MnU1BR3b3ts0euee+QyVGgRJvbrjFw6l2H/PUzv+8UwDKQeVFUlqT68zmXAZQBr1qypycnJ5X7JJTM1NcXk5OS8Q0EPq3OP3MVH7p773+Pm0yf7V5keTO/7xbCZSFq4h1sTD+3n9la+DTisa7mVrWy+8pWzlEsDYxhIC3cdMN0jaB3w5a7yM1qvouOBx1pz0g3ASUkOaheOTwJuaPMeT3J860V0Rte2pIHYYxjYxU5PRUk+B/wf4CVJtiY5C7gIeG2S+4HXtGmA64EHgE3A3wDvBKiqHcCHgNva44OtjLbMp9o6PwC+1o/3Jc1lIdcMrgD+ik6Ph2nTXewuSnJem34/v93F7jg63eeO6+pitwYo4I4k17UeFtNd7G6l80e1Fv8wNGBV9bY5Zp04y7IFnDPHdtYD62cpvx142d7UUVpKezwzsIudJI2/XnsT9b2LHczfza6XrlSDNi51nq8L3lLqdV+N4n6W+m2vu5b2q4tde605u9n10pVq0Malzv3qOthrd75R3M9Sv/Xam8gudpI0RnoNA7vYSdIY2WMzUetiNwkckmQrnV5BFwFXt+52PwTe0ha/HjiFTne5nwFnQqeLXZLpLnawexe7K4D96PQisieRJPXZHsPALnaSNP78BrIkyTCQJBkGkiQMA0kS3s9AkhZtVQ9ftNx80anLUJOl45mBJMkwkCQZBpIkDANJEl5AlrQIi7lweu6Ru/o2oq32nmcGkiTDQJJkGEiS8JqBZthTm7DtwNJ48sxAkmQYSJJsJtII6WU8GIAr1u6/xDWRxo9nBpIkw0CSZBhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJDFEYJFmb5PtJNiU5b9D1kZabx7yGyVCEQZJ9gE8ArwOOAN6W5IjB1kpaPh7zGjbDMoT1K4BNVfUAQJKrgNOAewdaqxHW63DP6puBH/MeI/3V6/7efNGpS1yT2Q1LGKwAtnRNbwWOm7lQkrOBs9vkziTf75p9CPDjZavh8hi5Ov/pCNb51RfPWeff63dduizFMT/URvFYmTZMdc/FPa226GN+WMJgQarqMuCy2eYlub2q1vS5SnvFOvfHKNZ52nzH/LAb5f0+ynWH3uo/FNcMgG3AYV3TK1uZNK485jVUhiUMbgNWJzk8yTOAtwLXDbhO0nLymNdQGYpmoqraleRdwA3APsD6qrpnkZsZxVNp69wfQ1fnJTrmh93Q7fdFGOW6Qw/1T1UtR0UkSSNkWJqJJEkDZBhIkkY/DEbhK/1JDktyc5J7k9yT5N2t/OAkG5Pc334eNOi6zpRknyR3JvlKmz48ya1tf3++XfwcGkkOTHJNku8luS/JH47Cfh43STYnuTvJt5PcPuj6zCfJ+iTbk3y3q2xkjpk56v/nSba1/f/tJKfsaTsjHQYj9JX+XcC5VXUEcDxwTqvnecCNVbUauLFND5t3A/d1TV8MXFJVLwYeAc4aSK3m9nHg76vqpcDL6dR9FPbzOHp1VR01Av31rwDWzigbpWPmCnavP3T+To9qj+v3tJGRDgO6vtJfVb8Apr/SP1Sq6sGq+lZ7/lM6/6BW0KnrhrbYBuCNA6ngHJKsBE4FPtWmA5wAXNMWGao6J3ku8EfA5QBV9YuqepQh388arKr6BrBjRvHIHDNz1H/RRj0MZvtK/4oB1WVBkqwCjgZuBSaq6sE26yFgYlD1msPHgPcBv2rTzwMerapdbXrY9vfhwI+Av21NW59Ksj/Dv5/HUQFfT3JHG1Jj1IzDMfOuJHe1ZqQ9NnONehiMlCTPBr4AvKeqHu+eV50+vkPTzzfJ64HtVXXHoOuyCPsCxwCfrKqjgSeYcXo/bPt5jL2qqo6h04R7TpI/GnSFejWix8wngRcBRwEPAh/Z0wqjHgYj85X+JE+nEwSfqaovtuKHkxza5h8KbB9U/WbxSuANSTbTaX47gU57/IFJpr+sOGz7eyuwtapubdPX0AmHYd7PY6mqtrWf24Fr6TTpjpKRPmaq6uGq+mVV/Qr4Gxaw/0c9DEbiK/2trf1y4L6q+mjXrOuAde35OuDL/a7bXKrq/KpaWVWr6OzXm6rqdOBm4E1tsWGr80PAliQvaUUn0hkSemj38zhKsn+S50w/B04Cvjv/WkNnpI+Z6SBr/g0L2P8j/w3k1mXqY/zmK/0XDrZGu0vyKuAfgLv5Tfv7B+hcN7gaeCHwQ+AtVbXXF4KWWpJJ4M+q6vVJfp/OmcLBwJ3An1TVzwdYvd+S5Cg6F7yfATwAnEnnQ8/Q7+dx0Y6Ra9vkvsBnh/HvclqSzwGTdIZ9fhi4APgSI3LMzFH/STpNRAVsBv591zWQ2bcz6mEgSdp7o95MJElaAoaBJMkwkCQZBpIkDANJEoaBJAnDQJIE/H830tjfPRzXkgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "text_word_count = []\n",
    "summary_word_count = []\n",
    "\n",
    "# get number of words in text and summary\n",
    "\n",
    "for i in data['text']:\n",
    "    text_word_count.append(len(i.split()))\n",
    "    \n",
    "for i in data['summary']:\n",
    "    summary_word_count.append(len(str(i).split()))\n",
    "    \n",
    "length = pd.DataFrame()\n",
    "length['text'] = pd.Series(text_word_count)\n",
    "length['summary'] = pd.Series(summary_word_count)\n",
    "\n",
    "length.hist(bins=10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "74bd91f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_text_len=60\n",
    "max_summary_len=15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7de50c82",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "text =np.array(data['text'])\n",
    "summary=np.array(data['summary'])\n",
    "\n",
    "short_text=[]\n",
    "short_summary=[]\n",
    "\n",
    "for i in range(len(text)):\n",
    "    if(len(str(summary[i]).split())<=max_summary_len and len(text[i].split())<=max_text_len):\n",
    "        short_text.append(text[i])\n",
    "        short_summary.append(summary[i])\n",
    "        \n",
    "df=pd.DataFrame({'text':short_text,'summary':short_summary})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bfdf4211",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['summary'] = df['summary'].apply(lambda x : 'sostok '+ x + ' eostok')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d04e5ed2",
   "metadata": {},
   "source": [
    "## Split the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8187055c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Conda\\envs\\directml\\lib\\importlib\\_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "x_tr,x_val,y_tr,y_val=train_test_split(np.array(df['text']),np.array(df['summary']),test_size=0.1,random_state=0,shuffle=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96521441",
   "metadata": {},
   "source": [
    "## Tokenize: text_to_word_sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "28735145",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer \n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "#prepare a tokenizer for reviews on training data\n",
    "x_tokenizer = Tokenizer() \n",
    "x_tokenizer.fit_on_texts(list(x_tr))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "26da2c75",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "% of rare words in vocabulary: 56.57998805546139\n",
      "Total Coverage of rare words: 1.1902033035582427\n"
     ]
    }
   ],
   "source": [
    "thresh=4\n",
    "\n",
    "cnt=0\n",
    "tot_cnt=0\n",
    "freq=0\n",
    "tot_freq=0\n",
    "\n",
    "for key,value in x_tokenizer.word_counts.items():\n",
    "    tot_cnt=tot_cnt+1\n",
    "    tot_freq=tot_freq+value\n",
    "    if(value<thresh):\n",
    "        cnt=cnt+1\n",
    "        freq=freq+value\n",
    "    \n",
    "print(\"% of rare words in vocabulary:\",(cnt/tot_cnt)*100)\n",
    "print(\"Total Coverage of rare words:\",(freq/tot_freq)*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e77112b0",
   "metadata": {},
   "source": [
    "## Padding: pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c89c241",
   "metadata": {},
   "source": [
    "Because input and output sequence lengths vary, we standardized all inputs to be some\n",
    "max length by padding short sentences with PAD tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "2a2cf8b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "#prepare a tokenizer for reviews on training data\n",
    "x_tokenizer = Tokenizer(num_words=tot_cnt-cnt) \n",
    "x_tokenizer.fit_on_texts(list(x_tr))\n",
    "\n",
    "#convert text sequences into integer sequences\n",
    "x_tr_seq    =   x_tokenizer.texts_to_sequences(x_tr) \n",
    "x_val_seq   =   x_tokenizer.texts_to_sequences(x_val)\n",
    "\n",
    "#padding zero upto maximum length\n",
    "x_tr    =   pad_sequences(x_tr_seq,  maxlen=max_text_len, padding='post')\n",
    "x_val   =   pad_sequences(x_val_seq, maxlen=max_text_len, padding='post')\n",
    "\n",
    "#size of vocabulary ( +1 for padding token)\n",
    "x_voc   =  x_tokenizer.num_words + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "08fc31c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26901"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_voc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "e86bc86f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#prepare a tokenizer for reviews on training data\n",
    "y_tokenizer = Tokenizer()   \n",
    "y_tokenizer.fit_on_texts(list(y_tr))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "63703ff4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "% of rare words in vocabulary: 55.537201701050854\n",
      "Total Coverage of rare words: 2.6054205341817265\n"
     ]
    }
   ],
   "source": [
    "thresh=4\n",
    "\n",
    "cnt=0\n",
    "tot_cnt=0\n",
    "freq=0\n",
    "tot_freq=0\n",
    "\n",
    "for key,value in y_tokenizer.word_counts.items():\n",
    "    tot_cnt=tot_cnt+1\n",
    "    tot_freq=tot_freq+value\n",
    "    if(value<thresh):\n",
    "        cnt=cnt+1\n",
    "        freq=freq+value\n",
    "    \n",
    "print(\"% of rare words in vocabulary:\",(cnt/tot_cnt)*100)\n",
    "print(\"Total Coverage of rare words:\",(freq/tot_freq)*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c85184cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "#prepare a tokenizer for reviews on training data\n",
    "y_tokenizer = Tokenizer(num_words=tot_cnt-cnt) \n",
    "y_tokenizer.fit_on_texts(list(y_tr))\n",
    "\n",
    "#convert text sequences into integer sequences\n",
    "y_tr_seq    =   y_tokenizer.texts_to_sequences(y_tr) \n",
    "y_val_seq   =   y_tokenizer.texts_to_sequences(y_val) \n",
    "\n",
    "#padding zero upto maximum length\n",
    "y_tr    =   pad_sequences(y_tr_seq, maxlen=max_summary_len, padding='post')\n",
    "y_val   =   pad_sequences(y_val_seq, maxlen=max_summary_len, padding='post')\n",
    "\n",
    "#size of vocabulary\n",
    "y_voc  =   y_tokenizer.num_words +1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "ac81b365",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12652"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_voc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f845e6b8",
   "metadata": {},
   "source": [
    "We are using a custom attention layer because keras does not officially support attention layers. We will use third-party implementation of custom attention layer taken from https://github.com/thushv89/attention_keras/blob/master/src/layers/attention.py , This implements Bahdanau Attention.\n",
    "For encoder-decoder neural networks, the use of attention allows for the creation\n",
    "of a context vector at each timestep, given the decoder’s current hidden state and a subset\n",
    "of the encoder’s hidden states."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "0f53f38b",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import backend as K\n",
    "\n",
    "logger = tf.get_logger()\n",
    "\n",
    "class AttentionLayer(tf.keras.layers.Layer):\n",
    "    \"\"\"\n",
    "    This class implements Bahdanau attention (https://arxiv.org/pdf/1409.0473.pdf).\n",
    "    There are three sets of weights introduced W_a, U_a, and V_a\n",
    "     \"\"\"\n",
    "\n",
    "    def __init__(self, **kwargs):\n",
    "        super(AttentionLayer, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        assert isinstance(input_shape, list)\n",
    "        # Create a trainable weight variable for this layer.\n",
    "\n",
    "        self.W_a = self.add_weight(name='W_a',\n",
    "                                   shape=tf.TensorShape((input_shape[0][2], input_shape[0][2])),\n",
    "                                   initializer='uniform',\n",
    "                                   trainable=True)\n",
    "        self.U_a = self.add_weight(name='U_a',\n",
    "                                   shape=tf.TensorShape((input_shape[1][2], input_shape[0][2])),\n",
    "                                   initializer='uniform',\n",
    "                                   trainable=True)\n",
    "        self.V_a = self.add_weight(name='V_a',\n",
    "                                   shape=tf.TensorShape((input_shape[0][2], 1)),\n",
    "                                   initializer='uniform',\n",
    "                                   trainable=True)\n",
    "\n",
    "        super(AttentionLayer, self).build(input_shape)  # Be sure to call this at the end\n",
    "\n",
    "    def call(self, inputs):\n",
    "        \"\"\"\n",
    "        inputs: [encoder_output_sequence, decoder_output_sequence]\n",
    "        \"\"\"\n",
    "        assert type(inputs) == list\n",
    "        encoder_out_seq, decoder_out_seq = inputs\n",
    "\n",
    "        logger.debug(f\"encoder_out_seq.shape = {encoder_out_seq.shape}\")\n",
    "        logger.debug(f\"decoder_out_seq.shape = {decoder_out_seq.shape}\")\n",
    "\n",
    "        def energy_step(inputs, states):\n",
    "            \"\"\" Step function for computing energy for a single decoder state\n",
    "            inputs: (batchsize * 1 * de_in_dim)\n",
    "            states: (batchsize * 1 * de_latent_dim)\n",
    "            \"\"\"\n",
    "\n",
    "            logger.debug(\"Running energy computation step\")\n",
    "\n",
    "            if not isinstance(states, (list, tuple)):\n",
    "                raise TypeError(f\"States must be an iterable. Got {states} of type {type(states)}\")\n",
    "\n",
    "            encoder_full_seq = states[-1]\n",
    "\n",
    "            \"\"\" Computing S.Wa where S=[s0, s1, ..., si]\"\"\"\n",
    "            # <= batch size * en_seq_len * latent_dim\n",
    "            W_a_dot_s = K.dot(encoder_full_seq, self.W_a)\n",
    "\n",
    "            \"\"\" Computing hj.Ua \"\"\"\n",
    "            U_a_dot_h = K.expand_dims(K.dot(inputs, self.U_a), 1)  # <= batch_size, 1, latent_dim\n",
    "\n",
    "            logger.debug(f\"U_a_dot_h.shape = {U_a_dot_h.shape}\")\n",
    "\n",
    "            \"\"\" tanh(S.Wa + hj.Ua) \"\"\"\n",
    "            # <= batch_size*en_seq_len, latent_dim\n",
    "            Ws_plus_Uh = K.tanh(W_a_dot_s + U_a_dot_h)\n",
    "\n",
    "            logger.debug(f\"Ws_plus_Uh.shape = {Ws_plus_Uh.shape}\")\n",
    "\n",
    "            \"\"\" softmax(va.tanh(S.Wa + hj.Ua)) \"\"\"\n",
    "            # <= batch_size, en_seq_len\n",
    "            e_i = K.squeeze(K.dot(Ws_plus_Uh, self.V_a), axis=-1)\n",
    "            # <= batch_size, en_seq_len\n",
    "            e_i = K.softmax(e_i)\n",
    "\n",
    "            logger.debug(f\"ei.shape = {e_i.shape}\")\n",
    "\n",
    "            return e_i, [e_i]\n",
    "\n",
    "        def context_step(inputs, states):\n",
    "            \"\"\" Step function for computing ci using ei \"\"\"\n",
    "\n",
    "            logger.debug(\"Running attention vector computation step\")\n",
    "\n",
    "            if not isinstance(states, (list, tuple)):\n",
    "                raise TypeError(f\"States must be an iterable. Got {states} of type {type(states)}\")\n",
    "\n",
    "            encoder_full_seq = states[-1]\n",
    "\n",
    "            # <= batch_size, hidden_size\n",
    "            c_i = K.sum(encoder_full_seq * K.expand_dims(inputs, -1), axis=1)\n",
    "\n",
    "            logger.debug(f\"ci.shape = {c_i.shape}\")\n",
    "\n",
    "            return c_i, [c_i]\n",
    "\n",
    "        # we don't maintain states between steps when computing attention\n",
    "        # attention is stateless, so we're passing a fake state for RNN step function\n",
    "        fake_state_c = K.sum(encoder_out_seq, axis=1)\n",
    "        fake_state_e = K.sum(encoder_out_seq, axis=2)  # <= (batch_size, enc_seq_len, latent_dim\n",
    "\n",
    "        \"\"\" Computing energy outputs \"\"\"\n",
    "        # e_outputs => (batch_size, de_seq_len, en_seq_len)\n",
    "        last_out, e_outputs, _ = K.rnn(\n",
    "            energy_step, decoder_out_seq, [fake_state_e], constants=[encoder_out_seq]\n",
    "        )\n",
    "\n",
    "        \"\"\" Computing context vectors \"\"\"\n",
    "        last_out, c_outputs, _ = K.rnn(\n",
    "            context_step, e_outputs, [fake_state_c], constants=[encoder_out_seq]\n",
    "        )\n",
    "\n",
    "        return c_outputs, e_outputs\n",
    "\n",
    "    def compute_output_shape(self, input_shape):\n",
    "        \"\"\" Outputs produced by the layer \"\"\"\n",
    "        return [\n",
    "            tf.TensorShape((input_shape[1][0], input_shape[1][1], input_shape[1][2])),\n",
    "            tf.TensorShape((input_shape[1][0], input_shape[1][1], input_shape[0][1]))\n",
    "            ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f74dfb5",
   "metadata": {},
   "source": [
    "At training time, our model decoders take in article truth embeddings as input at each\n",
    "timestep. This allows the model to learn to output predictions given some inputs x, and\n",
    "prediction history context, yc. During testing, however, the model’s decoder feeds its\n",
    "output as a word embedding input into the next decoder cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "622e86af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_6\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_13 (InputLayer)           [(None, 60)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_12 (Embedding)        (None, 60, 200)      5380200     input_13[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "lstm_24 (LSTM)                  [(None, 60, 300), (N 601200      embedding_12[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "input_14 (InputLayer)           [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "lstm_25 (LSTM)                  [(None, 60, 300), (N 721200      lstm_24[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_13 (Embedding)        (None, None, 200)    2530400     input_14[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "lstm_26 (LSTM)                  [(None, 60, 300), (N 721200      lstm_25[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm_27 (LSTM)                  [(None, None, 300),  601200      embedding_13[0][0]               \n",
      "                                                                 lstm_26[0][1]                    \n",
      "                                                                 lstm_26[0][2]                    \n",
      "__________________________________________________________________________________________________\n",
      "attention_layer (AttentionLayer ((None, None, 300),  180300      lstm_26[0][0]                    \n",
      "                                                                 lstm_27[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "concat_layer (Concatenate)      (None, None, 600)    0           lstm_27[0][0]                    \n",
      "                                                                 attention_layer[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "time_distributed_6 (TimeDistrib (None, None, 12652)  7603852     concat_layer[0][0]               \n",
      "==================================================================================================\n",
      "Total params: 18,339,552\n",
      "Trainable params: 18,339,552\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from tensorflow.keras.layers import Embedding,LSTM,Input,TimeDistributed,Dense,Concatenate\n",
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "latent_dim = 300\n",
    "embedding_dim= 200\n",
    "\n",
    "# Encoder\n",
    "encoder_inputs = Input(shape=(max_text_len,))\n",
    "\n",
    "#embedding layer\n",
    "enc_emb =  Embedding(x_voc, embedding_dim,trainable=True)(encoder_inputs)\n",
    "\n",
    "#encoder lstm 1\n",
    "encoder_lstm1 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)\n",
    "encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)\n",
    "\n",
    "#encoder lstm 2\n",
    "encoder_lstm2 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)\n",
    "encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)\n",
    "\n",
    "#encoder lstm 3\n",
    "encoder_lstm3=LSTM(latent_dim, return_state=True, return_sequences=True,dropout=0.4,recurrent_dropout=0.4)\n",
    "encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)\n",
    "\n",
    "# Set up the decoder, using `encoder_states` as initial state.\n",
    "decoder_inputs = Input(shape=(None,))\n",
    "\n",
    "#embedding layer\n",
    "dec_emb_layer = Embedding(y_voc, embedding_dim,trainable=True)\n",
    "dec_emb = dec_emb_layer(decoder_inputs)\n",
    "\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True,dropout=0.4,recurrent_dropout=0.2)\n",
    "decoder_outputs,decoder_fwd_state, decoder_back_state = decoder_lstm(dec_emb,initial_state=[state_h, state_c])\n",
    "\n",
    "# Attention layer\n",
    "attn_layer = AttentionLayer(name='attention_layer')\n",
    "attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])\n",
    "\n",
    "# Concat attention input and decoder LSTM output\n",
    "decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])\n",
    "\n",
    "#dense layer\n",
    "decoder_dense =  TimeDistributed(Dense(y_voc, activation='softmax'))\n",
    "decoder_outputs = decoder_dense(decoder_concat_input)\n",
    "\n",
    "# Define the model \n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "a53b8d4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy',metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "10cba72e",
   "metadata": {},
   "outputs": [],
   "source": [
    "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1,patience=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "6d0a8bc3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 90239 samples, validate on 10027 samples\n",
      "Epoch 1/50\n",
      "90239/90239 [==============================] - 522s 6ms/sample - loss: 4.3398 - acc: 0.4487 - val_loss: 4.0127 - val_acc: 0.4690\n",
      "Epoch 2/50\n",
      "90239/90239 [==============================] - 529s 6ms/sample - loss: 3.8417 - acc: 0.4726 - val_loss: 3.5827 - val_acc: 0.4866\n",
      "Epoch 3/50\n",
      "90239/90239 [==============================] - 531s 6ms/sample - loss: 3.4435 - acc: 0.4916 - val_loss: 3.3079 - val_acc: 0.5037\n",
      "Epoch 4/50\n",
      "90239/90239 [==============================] - 532s 6ms/sample - loss: 3.1625 - acc: 0.5077 - val_loss: 3.1533 - val_acc: 0.5160\n",
      "Epoch 5/50\n",
      "90239/90239 [==============================] - 532s 6ms/sample - loss: 2.9445 - acc: 0.5208 - val_loss: 3.0454 - val_acc: 0.5245\n",
      "Epoch 6/50\n",
      "90239/90239 [==============================] - 533s 6ms/sample - loss: 2.7631 - acc: 0.5328 - val_loss: 2.9725 - val_acc: 0.5308\n",
      "Epoch 7/50\n",
      "90239/90239 [==============================] - 534s 6ms/sample - loss: 2.6086 - acc: 0.5438 - val_loss: 2.9209 - val_acc: 0.5372\n",
      "Epoch 8/50\n",
      "90239/90239 [==============================] - 534s 6ms/sample - loss: 2.4745 - acc: 0.5544 - val_loss: 2.8899 - val_acc: 0.5401\n",
      "Epoch 9/50\n",
      "90239/90239 [==============================] - 534s 6ms/sample - loss: 2.3555 - acc: 0.5655 - val_loss: 2.8645 - val_acc: 0.5432\n",
      "Epoch 10/50\n",
      "90239/90239 [==============================] - 533s 6ms/sample - loss: 2.2505 - acc: 0.5757 - val_loss: 2.8485 - val_acc: 0.5461\n",
      "Epoch 11/50\n",
      "90239/90239 [==============================] - 534s 6ms/sample - loss: 2.1559 - acc: 0.5863 - val_loss: 2.8344 - val_acc: 0.5486\n",
      "Epoch 12/50\n",
      "90239/90239 [==============================] - 540s 6ms/sample - loss: 2.0709 - acc: 0.5961 - val_loss: 2.8313 - val_acc: 0.5496\n",
      "Epoch 13/50\n",
      "90239/90239 [==============================] - 541s 6ms/sample - loss: 1.9947 - acc: 0.6056 - val_loss: 2.8219 - val_acc: 0.5520\n",
      "Epoch 14/50\n",
      "90239/90239 [==============================] - 540s 6ms/sample - loss: 1.9247 - acc: 0.6147 - val_loss: 2.8265 - val_acc: 0.5514\n",
      "Epoch 15/50\n",
      "90239/90239 [==============================] - 535s 6ms/sample - loss: 1.8623 - acc: 0.6227 - val_loss: 2.8238 - val_acc: 0.5548\n",
      "Epoch 00015: early stopping\n"
     ]
    }
   ],
   "source": [
    "history=model.fit([x_tr,y_tr[:,:-1]], y_tr.reshape(y_tr.shape[0],y_tr.shape[1], 1)[:,1:] ,epochs=50,callbacks=[es],batch_size=128, validation_data=([x_val,y_val[:,:-1]], y_val.reshape(y_val.shape[0],y_val.shape[1], 1)[:,1:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "0e6ae411",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAskElEQVR4nO3dd3xUVf7/8deZSSMhCSGVkISE0EINELo0UapiR1exrO5XXburuOhWd3+7q7t+bd+1oSJYlhVRd0FRAaUoPQRC7wRSICSBhCSkz/n9cQcIkIQhyeTOTD7Px2MeM3Pn3juf4UHeOTlz7jlKa40QQgj3ZzG7ACGEEM1DAl0IITyEBLoQQngICXQhhPAQEuhCCOEhvMx647CwMB0fH2/W2wshhFvatGlTvtY6vK7XTAv0+Ph4UlNTzXp7IYRwS0qpw/W9Jl0uQgjhISTQhRDCQ0igCyGEhzCtD10IIRqjqqqKrKwsysvLzS7Fqfz8/IiJicHb29vhYyTQhRBuJSsri8DAQOLj41FKmV2OU2itKSgoICsri4SEBIePky4XIYRbKS8vJzQ01GPDHEApRWho6GX/FSKBLoRwO54c5mc05jO6XaAfKTjN84t2UFVjM7sUIYRwKW4X6Htzi/lgdQb/3phpdilCiFaosLCQN99887KPmzx5MoWFhc1fUC1uF+jjkiIYnNCe15btpaSi2uxyhBCtTH2BXl3dcB4tXryYdu3aOakqg9sFulKK5yYnkV9SyaxVB80uRwjRysycOZMDBw6QnJzMoEGDGDlyJFOnTqVnz54AXH/99QwcOJBevXoxa9ass8fFx8eTn59PRkYGSUlJ/M///A+9evVi/PjxlJWVNUttbjlsMTm2HVP6duDdVQeZPiSOiCA/s0sSQpjg+UU72JlzqlnP2TM6iD9c26ve11944QW2b9/Oli1bWLFiBVOmTGH79u1nhxfOnj2b9u3bU1ZWxqBBg7jpppsIDQ097xz79u1j3rx5vPvuu0ybNo3PP/+c6dOnN7l2t2uhn/HMhO5U22y8smyf2aUIIVqxwYMHnzdW/PXXX6dfv34MHTqUzMxM9u27OKMSEhJITk4GYODAgWRkZDRLLW7ZQgfoFBrAHUM68eHaDO67Ip4uEYFmlySEaGENtaRbSkBAwNnHK1asYNmyZaxduxZ/f3/GjBlT51hyX1/fs4+tVmuzdbm4bQsd4NEruxDg48UL3+wxuxQhRCsRGBhIcXFxna8VFRUREhKCv78/u3fvZt26dS1am1sHemhbXx4ck8iyXbmsP1hgdjlCiFYgNDSUESNG0Lt3b2bMmHHeaxMnTqS6upqkpCRmzpzJ0KFDW7Q2pbVu0Tc8IyUlRTfHAhdllTWMfWkFkcF+/Oeh4a3iCjIhWrNdu3aRlJRkdhktoq7PqpTapLVOqWt/t26hA7TxsfLU+G6kZxayeNsxs8sRQgjTuH2gA9w4IIYeUYH8/bvdVFbLlABCiNbJIwLdalHMnNSDwwWn+df6epfbE0IIj+YRgQ4wuls4I7qE8voP+zlVXmV2OUII0eI8JtCVUjw7KYkTpZW8s/KA2eUIIUSL85hAB+jdMZjrk6N578dDHC1qnoH6QgjhLjwq0AGeGt8dreGVpXvNLkUI4YEaO30uwKuvvsrp06ebuaJzPC7QY9v7c/fwTizYlMXuY807aY8QQrhyoLvtXC4NeXhsFz7dmMkL3+xmzs8Hm12OEMKD1J4+9+qrryYiIoL58+dTUVHBDTfcwPPPP09paSnTpk0jKyuLmpoafve735Gbm0tOTg5jx44lLCyM5cuXN3ttHhno7fx9eOTKLvx18W5W789nRJcws0sSQjjDNzPh2LbmPWdUH5j0Qr0v154+d8mSJSxYsIANGzagtWbq1KmsWrWKvLw8oqOj+frrrwFjjpfg4GBefvllli9fTliYczLJ47pczrhrWDwd27Xhb9/swmYzZ3oDIYRnW7JkCUuWLKF///4MGDCA3bt3s2/fPvr06cPSpUv59a9/zY8//khwcHCL1OORLXQAP28rT0/oxpOfprNoaw7XJXc0uyQhRHNroCXdErTWPPvsszzwwAMXvZaWlsbixYv57W9/y7hx4/j973/v9Ho8toUOcF2/jvTsEMTfv91DRXWN2eUIITxA7elzJ0yYwOzZsykpKQEgOzub48ePk5OTg7+/P9OnT2fGjBmkpaVddKwzeGwLHcBiMdYfnf7+ej5ae5hfjOxsdklCCDdXe/rcSZMmcfvttzNs2DAA2rZty8cff8z+/fuZMWMGFosFb29v3nrrLQDuv/9+Jk6cSHR0tFO+FHX76XMdcdfsDaRnFrJqxliC/b1b5D2FEM4h0+c2w/S5SimrUmqzUuqrOl7zVUp9qpTar5Rar5SKv9zCnWnmxB6cKq/izRX7zS5FCCGc5nL60B8HdtXz2n3ASa11F+AV4MWmFlavspPw0ytgc3ya3J7RQdzYP4YP1mSQddJ5g/qFEMJMDgW6UioGmAK8V88u1wFz7Y8XAOOUs5YO2rsElv0RNn1wWYc9Nb4bAC8vkSkBhHB3ZnUVt6TGfEZHW+ivAs8A9TWLOwKZ9iKqgSIg9MKdlFL3K6VSlVKpeXl5l10sAH2nQcIoI9RPHXX4sOh2bbh3RAJfbslme3ZR495bCGE6Pz8/CgoKPDrUtdYUFBTg5+d3WcddcpSLUuoa4LjWepNSakzjyjNorWcBs8D4UrRRJ1EKrnkV3hoO3zwDt37k8KG/HJPIvzce4cVvd/PRfUMa9fZCCHPFxMSQlZVFoxuFbsLPz4+YmJjLOsaRYYsjgKlKqcmAHxCklPpYaz291j7ZQCyQpZTyAoKBgsuq5HKEJsLoZ+D7P8Hur6HHFIcOC27jzaNXduXPX+1k1d48RnULd1qJQgjn8Pb2JiEhwewyXNIlu1y01s9qrWO01vHAbcAPF4Q5wELgbvvjm+37OPfvoeGPQUQv+PppKHd8VsXpQ+OIbd+Gv32zmxqZEkAI4UEafaWoUupPSqmp9qfvA6FKqf3Ar4CZzVFcg6zeMPV1KD4KP/zZ4cN8vazMmNCDXUdP8Z/N2U4sUAghWtZlBbrWeoXW+hr7499rrRfaH5drrW/RWnfRWg/WWh90RrEXiUmBwffDhnchc6PDh13TpwN9Y4L53yV7KK+SKQGEEJ7B/edyGfc7CIqGRY9BjWOLQ1ssxvqjOUXlzFmT4dz6hBCihbh/oPsGwuSX4PhOWPO6w4cNSwxlXI8I3li+n5OllU4sUAghWob7BzpAj8nQ8zpY8SIUHHD4sF9P6kFpRTX/XC5TAggh3J9nBDrApL+Dlx989QQ4OMCmW2Qg01Ji+XBtBpknZEoAIYR785xAD4yCq/8Ih1bBln85fNiTV3fDalH847s9zqtNCCFagOcEOsCAeyB2KCz5DZQ4dhVZZJAf/zOyMwvTc9iaVejU8oQQwpk8K9AtFrj2Nagoge+ec/iw+0d1JjTAh//3taw/KoRwX54V6AARPWDkU7BtPuxf5tAhgX7ezJjQnQ2HTvDx+sNOLlAIIZzD8wIdYOSvILQrfPUkVJY6dMitg2IZ3S2cvy7exaF8x44RQghX4pmB7uVrdL0UHoEVf3PoEKUUL97UFx+rhafmb5F5XoQQbsczAx0gfgQMuBvWvgE5Wxw6JCrYjz9f35u0I4W8s8rx8exCCOEKPDfQAa5+HvzDYNHjUFPt0CFT+0UzpU8HXlm6l11HHZ/FUQghzObZgd4mBCa9CEe3wPq3HTpEKcWfr+9NcBsfnvx0CxXVMnmXEMI9eHagA/S6AbpNhOV/gZOOjWBpH+DDizf1YfexYl5bts/JBQohRPPw/EBXypi8CwVf/8rhaQHGJUUyLSWGt1ceYNPhk86tUQghmoHnBzpAu1hjmt39y2D75w4f9rtretIhuA1Pf5bO6UrH+uCFEMIsrSPQwVgII7o/fDsTTp9w6JBAP2/+cUtfDuWX8uI3u51coBBCNE3rCXSLFab+nxHmS3/n8GHDE8O4d0QCc9ce5qd9+U4sUAghmqb1BDpAVB8Y/ihs/tiYldFBz0zsTmJ4ADMWpFNU5tiqSEII0dJaV6ADjP41hMTDoiegqtyhQ/y8rbw8LZnjxRU8v2iHU8sTQojGan2B7uMP17wCJw7Ajy85fFi/2HY8PCaRL9Ky+W7HMScWKIQQjdP6Ah0g8Uroexv89Ark7nT4sEeu7Eqv6CCe+2Ib+SUVTixQCCEuX+sMdIAJfwHfIFj0GNhsDh3i42Xh5WnJFJdX85svt6EdHNMuhBAtofUGekAYTPwbZG2E1PcdPqx7VCBPT+jGdzty+SIt24kFCiHE5Wm9gQ7Q91boPAaWPQ+nchw+7L4rOjM4vj1/XLiDnMIy59UnhBCXoXUHulLGF6S2Klg8w+HDrBbFS7f0o0ZrnlmwVZatE0K4hNYd6ADtO8OYmbD7K9i1yOHD4kL9+c2UJH7any/L1gkhXIIEOsCwRyCyjzE2PW+Pw4fdPjhOlq0TQrgMCXQAqzfc8gEoC8y5BvL2OnTYmWXrfL2sPDV/C9U1jo2WEUIIZ5BAPyOsK9y9CNAw91rI3+/QYVHBfvzpul72ZesOOrdGIYRogAR6bRE9jFC3VcPca6DAsXVFzyxb9+qyvezMkWXrhBDmuGSgK6X8lFIblFLpSqkdSqnn69jnHqVUnlJqi/32C+eU2wIikoxQr6k0ul8cCPXay9b9ar4sWyeEMIcjLfQK4EqtdT8gGZiolBpax36faq2T7bf3mrPIFhfZE+5aCNXlRvfLiUt3pciydUIIs10y0LWhxP7U237z/IHXUb3h7oVQdRrmXAsnDl3yEFm2TghhJof60JVSVqXUFuA4sFRrvb6O3W5SSm1VSi1QSsXWc577lVKpSqnUvLy8xlfdUqL6GC31yhKjpX4y45KHnFm27qn5W2TZOiFEi3Io0LXWNVrrZCAGGKyU6n3BLouAeK11X2ApMLee88zSWqdorVPCw8ObUHYL6tAX7vovVBQbLfWTDV9EFOjnzUu39COj4DQvyLJ1QogWdFmjXLTWhcByYOIF2wu01mfmk30PGNgs1bmK6GS46z9QUWSMfinMbHD3YYmh3DsigQ9l2TohRAtyZJRLuFKqnf1xG+BqYPcF+3So9XQqsKsZa3QN0f3hzv9AWRHMmQJFWQ3uLsvWCSFamiMt9A7AcqXUVmAjRh/6V0qpPymlptr3ecw+pDEdeAy4xznlmqzjALjrSyg7aQ/1+qfPrb1s3dOfpVMlV5EKIZxMmbVIQ0pKik5NTTXlvZssKxU+ugH8Q+HniyEout5d567J4A8Ld3Btv2hevTUZq0W1YKFCCE+jlNqktU6p6zW5UrQxYlJg+hdQmm9cfNTAXOp3D49n5qQeLErPkal2hRBOJYHeWLGD4M4voCTXGNJ46mi9uz44OpEnr+rG52lZ/OY/22XpOiGEU0igN0XsYJj+uRHmc6+F4mP17vrYuC48PDaReRuO8PyinRLqQohmJ4HeVHFDYfoCo9tl7rVQcrzO3ZRSPD2+O7+4IoE5azL42ze7JdSFEM1KAr05dBoOd3xmDGW8RKj/ZkoSdw7txKxVB3llqWPzrgshhCMk0JtL/Agj1AuPwNypUFL31AZKKZ6f2ovbBsXy+g/7+ecPMpGXEKJ5SKA3p/gr4PZPjTlfPpxqjIKpg8Wi+MsNfbixf0deWrKXWascm3ddCCEaIoHe3BJGGaF+4qDRUi8tqHM3q0Xx95v7MqVvB/66eDdz12S0bJ1CCI8jge4MnUfDz/4NJw4YLfXTJ+rczctq4dVbkxnfM5I/LNzBvA1HWrhQIYQnkUB3lsSx8LN5kL8P3hwGmz8G28WX/3tbLfzf7f0Z2z2c577cxoJNDc8RI4QQ9ZFAd6bEK+Heb6FdLPz3YXh3DGSsvmg3Xy8rb00fyIjEMJ5ZkM7C9PqvPBVCiPpIoDtbxwFw31K46X2jP33OZPh0+kXL2vl5W5l110BS4tvz5Kdb+HZ7/RcpCSFEXSTQW4JS0OdmeDQVxv4W9v8AbwyBJb+D8qKzu/n7eDH7nkH0iwnm0XlpfL8r18SihRDuRgK9JXm3gdEz4NFN0GcarPk/eH0AbHwfaozl6tr6ejHn3sEkdQjilx+nsWqvGyzVJ4RwCRLoZgjqANe/AfevgPAe8PWv4J2RsP9742U/bz68dzCdwwO4/6NU1h6oe+ijEELUJoFupuhkuOcrmPYRVJ2Gj2+ET6ZB3l7a+fvwyS+GEBviz31zN7LpcN1DH4UQ4gwJdLMpBT2nwsMb4Oo/w5G18NYwWPwMoZZSPvnFECKD/Lhn9kbSMwvNrlYI4cIk0F2Fly+MeAweTYMBd8HGd+H1/kTs/IB/3TuAdgHe3Pn+enbkFF36XEKIVkkC3dW0DYdrXoEHVxsLU387kw6fjOWLccW09bEy/b317DlWbHaVQggXJIHuqiJ7wp1fwu3zAUX4orv4PvJVkiyZ3PHeeg7klZhdoRDCxUiguzKloNsEeGgtTPo7bfK28UnN0zxb8zYPz/qO/ccl1IUQ50iguwOrNwx5AB7bjBr8ADeq5SyoeoSf3niA1evWmF2dEMJFSKC7E//2MOkF1EPrsHa9iunqG0Z8O4nsl0dj2zIPqsrMrlAIYSIJdHcU1pU2d3xE9RM7WRTxSyoLj2L5z4Pol7rD4hmQu8PsCoUQJlBmLVSckpKiU1NTTXlvT6K1Zu7qQyz55gvua7OSK23rULZK6JgCA++GXjeCb1uzyxRCNBOl1CatdUqdr0mge4Z1Bwt4+JM02lQX8X7/A3TP+hzy94BPoDEx2MC7jWGQQgi31lCgS5eLhxjaOZSFj15BSFgUE9b15rXuH2O751tIuhbS58GsMfDOKGMisPJTZpcrhHACaaF7mPKqGp77chtfpGVzdc9IXp7Wj0BdCts+g01zIHc7ePtD7xthwD0Qk2IMjxRCuAXpcmlltNbMWZPB//t6F/Gh/sy6K4XE8LagNWSnQdoc2PY5VJVCRE8YeA/0nQZtQswuXQhxCRLordSZfvXKahuv3JrMVT0jz71YUQzbFkDaXMjZDF5+0PM66H8nxA0Dq5d5hQsh6iWB3oplF5bx4Eeb2JZdxJNXdePRK7tgsVzQxXI0HTbNNbplKk6BbxDEj4TOY4zFrkO7SLeMEC6iSYGulPIDVgG+gBewQGv9hwv28QU+BAYCBcCtWuuMhs4rgd5y6uxX9/O+eMfKUti3BA6ugAPLofCwsT0oxgj3M7e24S1XvBDiPE0NdAUEaK1LlFLewE/A41rrdbX2eQjoq7V+UCl1G3CD1vrWhs4rgd6y6u1Xb8iJQ3BwuRHwB1dCeaGxPbK3PdzHQqfh4OPv5OqFEGc0W5eLUsofI9B/qbVeX2v7d8AftdZrlVJewDEgXDdwcgl0czTYr94QW43RNXMm4I+sg5pKsPpA7JBzAR+dDBarEz+BEK1bkwNdKWUFNgFdgDe01r++4PXtwEStdZb9+QFgiNY6/4L97gfuB4iLixt4+PDhRnwc0VQO9atfSuVpY3Wlg8vhwArI3WZs92sHCaPOdc+07yz970I0o+ZsobcDvgQe1Vpvr7XdoUCvTVro5iqvquG5L7bxxeZL9Ks7qiQPDq08F/Cnsozt7eKMYI8fCVF9IayrtOCFaIJmHeWilPo9cFpr/VKtbdLl4oYa1a/u2Imh4MC57plDq4zRMwBebYzFO6L6GAEf1Rcie0k/vBAOauqXouFAlda6UCnVBlgCvKi1/qrWPg8DfWp9KXqj1npaQ+eVQHcdaw8U8PC/0qi63H51R9VUG/PKHN0Kx7bBsa3Grdy+PqqyGEMjz4Z8H+jQDwLCmrcOITxAUwO9LzAXsGLM/TJfa/0npdSfgFSt9UL70MaPgP7ACeA2rfXBhs4rge5asgvLeOCjVLZnn2L60DienZREgK8TLy7SGooya4W8PeiLMs/tE9jhXMBH9YEOfaFdPFhkCiLResmFRcIh5VU1/OO7PcxefYiO7drw95v6MrxLC7eST584P+CPbYO8PaBrjNd9As8FfFQfo2UfHGOEv1zdKloBCXRxWVIzTjBjwVYO5ZcyfWgcMycl0daZrfVLqSqH4zvPD/lj2425aM5QFmgbBcEdIaijEfLBMfbHHY2LowLCpXUv3J4EurhsZZU1/O+SPby/+hDRwW34+819GdHSrfWG2Gxw4iCczDBG1BRlw6lso8vmzOPq8vOPsfpAULQR7meD3x72wfZtfu1kmKVwaRLootE2HT7BjM+2cjC/lDuGxPHsZJNb647S2ui+ORP2RVkXBH82FOeArfr847wDjGBvG1nrFnHxvX+oDL8UppBAF01SXlXDS9+5cGu9sWw1UHK8jrDPMraX5Bq3qtMXH6ssRhfO2ZC/IPADaj32C5ZWv2g2EuiiWdRurd8+JI7n3KW13lQVJVB6vFbI1wr787YdB1vVxcdbfY1wDwgzwt0vyJjR0i/Yfh9Uz739dS+flv/MwmVJoItmU15l9K2/95PRWn/xpr5c0dUDWuvNQWsoO1l/8JfmGRdYlZ86d1/7i936ePk1EPzB4BtoLATuGwg+F9z7tjVGBvkGgpev/KXgASTQRbOr3Vr/2eA4npvco2lTB7RWNdVGuF8Y9Gfvi+rZXuu+ssSx97J41Qr6M6Hf1n4fVOux/d7LF5TV+K5AWY1fBmcen3ev6thmNUYUKcsF27yMvzi82oC3n3Evw00viwS6cIryqhpeXrqX9348SIfgNrxwUx9GdpW50lucrcaYy76i2Aj3ihKoLDaeV5TYt52q9bjk3C+CipKLj9O2lq3f4nV+wHv7GX+VeLcxfqlc9Fod91Zv4wtuW43R7WWrNm411ecen91WdYl9a72ubRf/MlMW+2PLudvZ57VfVxc8r/U4YRR0m9Cof66GAl1+NYpG8/O28tzkJCb0imLGgnTufH8DPxscy3OTk6S13pIsVqP7xS+o6efS2vgSuKLEmB5Z19iDTdd6XPveZoTeRa/Zzt1f+Fp1hTGktKrMsfuyQvsxZcY1CWdeq+v7irqc+cvA6m38W1m8wOJtv7fat3udfzuzTVmMz26rNmo483m07fzPfuHn1bqOz287d/P2b3SgN/hRpYUumkN5VQ2vLN3Luz8eJCrIjxdu6suobtJaF05kqzkX+jWV9pC2XhDIVo+7mKyhFrpnfVJhGj9vK89OTmLBL4fj52PlrtkbmPn5VorLHWxFCXG5LFajvz8gzLhgrG04+Lc3/lLxtnfDeFiYX0rr+rTC6QbEhbD4sZE8MLoz81MzmfDKKlbuzTO7LCFaBQl00ez8vK08OymJz385nDY+Vu6evYGnP0vneHH5pQ8WQjSaBLpwmv5xIXz92EgeHJ3IfzZnM/YfK3hj+X7Kq2rMLk0IjySBLpzKz9vKzEk9WPLkKIYlhvGP7/Yw7n9Xsig9B7O+kBfCU0mgixbRObwt792dwr9+MYRAPy8enbeZm99ey5bMQrNLE8JjSKCLFjW8SxhfPzaSF27sw+GC01z/xmqe/HQLR4vKzC5NCLcngS5anNWiuG1wHCtmjOGhMYl8ve0oY19awctL93K6svrSJxBC1EkCXZimra8Xz0zswfe/Gs24pEhe/34fY19awYJNWdhs0r8uxOWSQBemi23vzxu3D2DBg8OICvLj6c/Suf7N1WzMOGF2aUK4FQl04TJS4tvz5UMjeOXWfhw/VcEtb6/loU82kXmijgUmhBAXkcm5hEuxWBQ39I9hQq8oZq06yDsrD7Js53F+fkU8j4ztIpN+CdEAaaELl+Tv48UTV3Vj+dNjuKZfB95ZeZCxL63gX+uPUCP960LUSQJduLSoYD9enpbMwkdGkBAWwHNfbmPK6z/y0758s0sTwuVIoAu30DemHfMfGMabdwygpKKa6e+v5745G9mXW2x2aUK4DJkPXbid8qoaPlidwRvL91NaWc3kPh149Mou9IhqhgUehHBxsgSd8EgnSit578eDzF2TQWllDRN7RfHouC70ig42uzQhnEYCXXi0wtOVzP7pEB+szqC4opqrkiJ5fFxX+sRIsAvPI4EuWoWisirmrM7g/Z8Ocqq8mit7RPDolV3oHxdidmlCNBsJdNGqFJdX8eHaw7z740EKT1cxqls4j4/rwsBO7c0uTYgmk0AXrVJJRTUfrzvMu6sOUlBayYguoTx2ZVeGdA41uzQhGq1Ji0QrpWKVUsuVUjuVUjuUUo/Xsc8YpVSRUmqL/fb75ihciKZo6+vFg6MT+fHXY/ntlCT2HCvh1lnruPWdtazZny8LbAiPc8kWulKqA9BBa52mlAoENgHXa6131tpnDPC01voaR99YWuiipZVX1TBvwxHeXnmA3FMVpHQK4bFxXRnZNQyllNnlCeGQJrXQtdZHtdZp9sfFwC6gY/OWKITz+Xlb+fmIBFbOGMufr+tFdmEZd83ewA1vrmH57uPSYhdu77L60JVS8cAqoLfW+lSt7WOAz4EsIAejtb6jjuPvB+4HiIuLG3j48OEmlC5E01RU1/D5pmzeWL6f7MIy+nQM5rFxXbkqKUJa7MJlNcuXokqptsBK4C9a6y8ueC0IsGmtS5RSk4HXtNZdGzqfdLkIV1FZbePLzVn8c/l+Mk+U0bNDEA+M7syk3h3w8ZLZMYRraXKgK6W8ga+A77TWLzuwfwaQorWudwYlCXThaqpqbPx3Sw5vrtjPwbxSIoN8uWtYPD8bHEf7AB+zyxMCaGKgK+Nvz7nACa31E/XsEwXkaq21UmowsADopBs4uQS6cFU2m2blvjxm/3SIH/fl4+tl4Yb+Hfn5iAS6RwWaXZ5o5RoKdEcWuBgB3AlsU0ptsW97DogD0Fq/DdwM/FIpVQ2UAbc1FOZCuDKLRTG2ewRju0ewN7eYD1Zn8EVaFv/emMkVXcK494p4xnSLwGKRfnbhWuTCIiEccLK0knkbj/DhmsMcO1VOQlgA9wyP5+aBMQT4ysJfouXIlaJCNJOqGhvfbD/G7J8OsSWzkEA/L24bFMtdw+KJbe9vdnmiFZBAF8IJ0o6c5IPVGSzedhStNeN7RnHvFQkMig+RYY/CaSTQhXCinMIyPlp3mH+tP0JRWRW9OwZx74gEpvTtgK+X1ezyhIeRQBeiBZRV1vDl5mxmrz7E/uMlhAf6Mn1IJ+4YGkdYW1+zyxMeQgJdiBaktebHffnMXn2IFXvy8LFauC45mntGxMtqSqLJmjpsUQhxGZRSjOoWzqhu4RzIK2HO6gwWbMris01Z9IsJ5paUWKYmRxPk5212qcLDSAtdiBZQdLqKz9OymJ+aye5jxfh5W5jcuwO3pMQytHN7+RJVOEy6XIRwEVprtmUX8enGTBZuyaG4oppOof7cMjCGmwfGEhXsZ3aJwsVJoAvhgsoqa/h2x1E+3ZjJuoMnsCgY1S2cW1NiGZcUKRODiTpJoAvh4g4XlPJZahYLNmVx7FQ57QN8uKF/R24dFEu3SJk/RpwjgS6Em6ixaVbty2P+xkyW7cqlqkbTL7Ydt6bEcm2/DgTKF6mtngS6EG6ooKSCLzdnMz81k725JcYXqX06cGtKLIMT5IvU1koCXQg3prUmPcv4InVReg4lFdXEh/pzS0osNw+MITJIvkhtTSTQhfAQpyur+WbbMT5NzWTDoXNfpE7tF834XlG0lZkfPZ4EuhAe6FB+KZ+lZvLfLTlkF5bh62VhXFIEU/tFM6Z7BH7eMo+MJ5JAF8KD2WyatCMnWZiew9dbj1JQWkmgrxfje0UxNTmaEYmheFllCKSnkEAXopWorrGx5kABC9Nz+G77MYorqgkN8GFynw5MTY5mYFyIrLTk5iTQhWiFyqtqWLEnj0Vbc/h+Vy7lVTaig/24pl80U/tF0ys6SEbKuCEJdCFauZKKapbtzGVheg6r9uZRbdN0Dg/g2r7RTE2OJjG8rdklCgdJoAshzjpZWsk324+xMD2b9YdOoDX0ig5iar9oru0XTXS7NmaXKBoggS6EqNOxonK+2prDovQc0rOKABgUH8LUftFM7N2B8EBZmMPVSKALIS4pI7+URek5LEzPYd/xEpSCAXEhjO8ZyfheUSSEBZhdokACXQhxGbTW7Mkt5rvtuSzddYzt2acA6BLR9my49+0YLKNlTCKBLoRotKyTp1m2M5clO3NZf+gENTZNZJAvVyUZ4T6sc6hM9duCJNCFEM2i8HQlP+w+ztKduazYk0dZVQ2Bvl6M6RHB+J6RjOkeLjNCOpkEuhCi2ZVX1bB6fz5LduSybFcuBaWVeFsVwxLDGN8zkqt7RsrEYU4ggS6EcKoam2bzkZMs2ZnLdzuOcbjgNADJse24umckE3pFkhjeVi5kagYS6EKIFqO1Zt/xEpbsOMaSnblstQ+H7BwWwNU9IxnbI4IBcSHS795IEuhCCNMcLSo7+6Xq2gMFVNs0AT5WhncJY3S3cEZ3Cye2vb/ZZboNCXQhhEsoLq9izYECVu7NY+WePLILywCj9T7KHu5DO4fSxkem/q2PBLoQwuVorTmYX8rKPXms3JvHuoMFVFTb8PGyMCShPaO6hjO6ezhdI6TvvbYmBbpSKhb4EIgENDBLa/3aBfso4DVgMnAauEdrndbQeSXQhRC1lVfVsOHQCVbuzWPV3jz2HS8BoEOw39lwH9EljOA2rXtYZEOB7sh6VdXAU1rrNKVUILBJKbVUa72z1j6TgK722xDgLfu9EEI4xM/byqhu4YzqFg5ATmEZq/YarffF247yaWomVosiObbd2b73PnLF6nkuu8tFKfVf4J9a66W1tr0DrNBaz7M/3wOM0Vofre880kIXQjiqqsbGlszCswG/LbsIrSHE35uRXY1fAsMTQ1vFTJFNbaHXPlE80B9Yf8FLHYHMWs+z7NvOC3Sl1P3A/QBxcXGX89ZCiFbM22phUHx7BsW356nx3SkoqeCn/fms3JPHqn15LEzPASA+1J9hiWEMTwxlaOfQVjdbpMOBrpRqC3wOPKG1PtWYN9NazwJmgdFCb8w5hBAitK0v1yV35Lrkjthsmt3Hill7sIC1B/L5Kj2HeRuOANAtsi3DE8MYlhjK0IRQgv09u//doUBXSnljhPknWusv6tglG4it9TzGvk0IIZzKYlH0jA6iZ3QQ912RQHWNje05p1h7oIA1B/L598YjzFmTgVLGQh5nAn5QfHva+l5WJ4XLc2SUiwLmAie01k/Us88U4BGMUS5DgNe11oMbOq/0oQshWkJltY30rELW7DcCfvORQiprbFgtin4xwQy3d9EM6BSCn7frj39v6rDFK4AfgW2Azb75OSAOQGv9tj30/wlMxBi2+HOtdYNpLYEuhDBDWWUNaUdOsuZAPmsOFLA1q4gam8bHy8KAuHZnA75vTDuXnJ5ALiwSQoh6FJdXkZpxLuB3Hj2F1tDG28rATiEM7BRCSnwI/eNCXKKLptlGuQghhKcJ9PNmbI8IxvaIAIw539cdPMHaA/lsyDjJ6z/sQ2uwKOgeFUSKPeQHdgohJqSNS13FKi10IYRoQHF5FZuPFLLp8Ek2HT7J5iMnKa2sASAyyNce7u0Z2CmEXtFBeFud200jLXQhhGikQD/v865gra6xsSe3+GzAp2acZPG2YwD4eVvoF9PubDfNgLgQ2vn7tFit0kIXQogmOlZUboT74ROkHT7JjpxTVNuMbO0S0ZaBcSEMjA8hpVMICWEBTeqmkS9FhRCiBZ2urCY9s4i0IydJzTjBpsMnOVVeDUD7AB8eGpPIL0Z2btS5pctFCCFakL+PF8MSQxmWGAqAzaY5kFdCqr2bJsJJa61KoAshhJNZLIqukYF0jQzkZ4OdN4+V642aF0II0SgS6EII4SEk0IUQwkNIoAshhIeQQBdCCA8hgS6EEB5CAl0IITyEBLoQQngI0y79V0rlAYcbeXgYkN+M5TibO9XrTrWCe9XrTrWCe9XrTrVC0+rtpLUOr+sF0wK9KZRSqfXNZeCK3Kled6oV3Kted6oV3Kted6oVnFevdLkIIYSHkEAXQggP4a6BPsvsAi6TO9XrTrWCe9XrTrWCe9XrTrWCk+p1yz50IYQQF3PXFroQQogLSKALIYSHcLtAV0pNVErtUUrtV0rNNLue+iilYpVSy5VSO5VSO5RSj5tdkyOUUlal1Gal1Fdm19IQpVQ7pdQCpdRupdQupdQws2tqiFLqSfv/g+1KqXlKKecsWdNISqnZSqnjSqnttba1V0otVUrts9+HmFnjGfXU+g/7/4WtSqkvlVLtTCzxPHXVW+u1p5RSWikV1hzv5VaBrpSyAm8Ak4CewM+UUj3Nrape1cBTWuuewFDgYReutbbHgV1mF+GA14BvtdY9gH64cM1KqY7AY0CK1ro3YAVuM7eqi8wBJl6wbSbwvda6K/C9/bkrmMPFtS4Femut+wJ7gWdbuqgGzOHielFKxQLjgSPN9UZuFejAYGC/1vqg1roS+Ddwnck11UlrfVRrnWZ/XIwROB3NraphSqkYYArwntm1NEQpFQyMAt4H0FpXaq0LTS3q0ryANkopL8AfyDG5nvNorVcBJy7YfB0w1/54LnB9S9ZUn7pq1Vov0VpX25+uA2JavLB61PNvC/AK8AzQbCNT3C3QOwKZtZ5n4eIhCaCUigf6A+tNLuVSXsX4D2YzuY5LSQDygA/s3UPvKaUCzC6qPlrrbOAljJbYUaBIa73E3KocEqm1Pmp/fAyINLOYy3Av8I3ZRTREKXUdkK21Tm/O87pboLsdpVRb4HPgCa31KbPrqY9S6hrguNZ6k9m1OMALGAC8pbXuD5TiOt0BF7H3PV+H8YsoGghQSk03t6rLo43xzS4/xlkp9RuM7s5PzK6lPkopf+A54PfNfW53C/RsILbW8xj7NpeklPLGCPNPtNZfmF3PJYwApiqlMjC6sq5USn1sbkn1ygKytNZn/uJZgBHwruoq4JDWOk9rXQV8AQw3uSZH5CqlOgDY74+bXE+DlFL3ANcAd2jXvsAmEeOXe7r95y0GSFNKRTX1xO4W6BuBrkqpBKWUD8YXSwtNrqlOSimF0ce7S2v9stn1XIrW+lmtdYzWOh7j3/UHrbVLtiK11seATKVUd/umccBOE0u6lCPAUKWUv/3/xThc+EvcWhYCd9sf3w3818RaGqSUmojRXThVa33a7HoaorXeprWO0FrH23/esoAB9v/XTeJWgW7/0uMR4DuMH4j5Wusd5lZVrxHAnRgt3S3222Szi/IgjwKfKKW2AsnAX80tp372vyQWAGnANoyfO5e6VF0pNQ9YC3RXSmUppe4DXgCuVkrtw/gr4wUzazyjnlr/CQQCS+0/a2+bWmQt9dTrnPdy7b9MhBBCOMqtWuhCCCHqJ4EuhBAeQgJdCCE8hAS6EEJ4CAl0IYTwEBLoQgjhISTQhRDCQ/x/gpbmKuSiCHgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "plt.plot(history.history['loss'], label='train')\n",
    "plt.plot(history.history['val_loss'], label='test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "c35d7ab7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA0yklEQVR4nO3dd3hUVfrA8e+bEHpP6AESIPQqoSjiIgiCBWyLiA0buurqz1XXtuuuuq5ld10bFkBEBWXtoqAoCmIBJCAqnSSUJLSQSnp7f3/cCQwhIRNSZiZ5P88zz8w9956b90Jy3jvn3HuuqCrGGGPqngBvB2CMMcY7LAEYY0wdZQnAGGPqKEsAxhhTR1kCMMaYOqqetwOoiJCQEA0LC/N2GMYY41fWr19/WFXblCz3qwQQFhZGVFSUt8Mwxhi/IiJ7Siu3LiBjjKmjLAEYY0wd5VECEJGJIrJdRKJF5P4ytpkqIltEZLOIvO0qGywiq11lv4rI5W7bzxeRXSKy0fUaXCVHZIwxxiPljgGISCAwCxgPxAPrRGSxqm5x2yYCeAAYpaopItLWtSoLuEZVd4pIR2C9iCxT1VTX+ntV9f3KHEB+fj7x8fHk5ORUZjc+r2HDhoSGhhIUFOTtUIwxtYQng8DDgWhVjQUQkUXAFGCL2zY3AbNUNQVAVQ+53ncUb6Cq+0TkENAGSK2S6IH4+HiaNWtGWFgYIlJVu/UpqkpSUhLx8fGEh4d7OxxjTC3hSRdQJyDObTneVeauJ9BTRH4QkTUiMrHkTkRkOFAfiHErftzVNfRfEWlQwdgByMnJITg4uNY2/gAiQnBwcK3/lmOMqVlVNQhcD4gAxgBXAHNEpGXxShHpALwFXKeqRa7iB4DewDCgNXBfaTsWkZkiEiUiUYmJiaX+8Nrc+BerC8dojKlZniSABKCz23Koq8xdPLBYVfNVdRewAychICLNgSXAQ6q6priCqu5XRy7wOk5X0wlUdbaqRqpqZJs2J9zHYIwxtdqmhDQe+XQz+YVF5W9cQZ4kgHVAhIiEi0h9YBqwuMQ2H+Oc/SMiIThdQrGu7T8C3iw52Ov6VoA4p7YXAZtO+Si8KDU1lZdeeqnC9c477zxSU1OrPiBjjN/LyS/kww3xXPzSD1zwwvcs+imO7QeOVPnPKXcQWFULROR2YBkQCMxT1c0i8igQpaqLXesmiMgWoBDn6p4kEbkKOAsIFpEZrl3OUNWNwEIRaQMIsBG4pWoPrWYUJ4Bbb731uPKCggLq1Sv7n3fp0qXVHZoxxs/EJWexcO1e3o2KIzkzj24hTXj4gr5cOjSUFo2q/gpAj6aCUNWlwNISZQ+7fVbgT66X+zYLgAVl7HNsRYP1Rffffz8xMTEMHjyYoKAgGjZsSKtWrdi2bRs7duzgoosuIi4ujpycHO68805mzpwJHJvWIiMjg0mTJnHmmWfy448/0qlTJz755BMaNWrk5SMzxtSEoiJl1c5E3lq9h2+2H0KAc/q045rTwxjVo3ovcPGruYDK88inm9myL71K99m3Y3P+dmG/Mtc/+eSTbNq0iY0bN7Jy5UrOP/98Nm3adPRyzXnz5tG6dWuys7MZNmwYl156KcHBwcftY+fOnbzzzjvMmTOHqVOn8sEHH3DVVVdV6XEYY3xLalYe70XFs2DtHvYkZRHStD63jenB9BFd6NiyZk4Aa1UC8AXDhw8/7lr9559/no8++giAuLg4du7ceUICCA8PZ/DgwQAMHTqU3bt311S4xpga9lt8Gm+u3s3iX/aRW1DEsLBW/Gl8Tyb170D9ejU7O0+tSgAnO1OvKU2aNDn6eeXKlSxfvpzVq1fTuHFjxowZU+q1/A0aHLsFIjAwkOzs7BqJ1RhTM3LyC1ny637eXLOHX+JSaRQUyCWnhXL1yK707djca3HVqgTgDc2aNePIkdJH59PS0mjVqhWNGzdm27ZtrFmzptTtjDG1U1xyFgvW7uHddXGkZOXTrU0T/nahM6jbvKH3p3WxBFBJwcHBjBo1iv79+9OoUSPatWt3dN3EiRN55ZVX6NOnD7169WLkyJFejNQYUxOKipRvXYO6K7YfIkCE8X3acfXpXTmju2/NWiDOBTz+ITIyUks+EGbr1q306dPHSxHVrLp0rMb4myM5+Sz6KY631uxhb3IWIU0bMH14Z64Y0YUOLbx7VZ+IrFfVyJLl9g3AGGMqITkzj/k/7GL+j7tJzylgWFgr7jm3FxP7ta/xQd2KsgRgjDGn4EBaDnO+i+XttXvJzi/k3H7tuHVMDwZ1bunt0DxmCcAYYypg9+FMXl0Vw/vr4ylSmDKoI38Y052Ids28HVqFWQIwxhgPbN2fzssrY/js133UCwxg2rAuzDyrG51bN/Z2aKfMEoAxxpzEhr0pvLQimuVbD9GkfiA3ndWNG84Mp22zht4OrdIsARhjTAmqyvfRh3lpRQyrY5No2TiIP43vybWnh9Gisfev368qvj1E7QdOdTpogGeffZasrKwqjsgYc6qKipQvNh1gyqwfuPq1n4g9nMFfzu/DD/eN5Y5xEbWq8QdLAJVmCcAY/1dQWMSHG+I599lV3LJgPWnZ+TxxyQBW/flsbhzdjSYNamdnSe08qhrkPh30+PHjadu2Le+++y65ublcfPHFPPLII2RmZjJ16lTi4+MpLCzkr3/9KwcPHmTfvn2cffbZhISEsGLFCm8fijF1Tk5+Ie+tj+fVb2OIT8mmV7tmPDdtMOcP6EC9wNp/fly7EsDn98OB36p2n+0HwKQny1ztPh30l19+yfvvv89PP/2EqjJ58mRWrVpFYmIiHTt2ZMmSJYAzR1CLFi145plnWLFiBSEhIVUbszHmpDJyC1i4Zg9zv99F4pFchnRpyd8v7MfY3m0JCPCdqRqqW+1KAF725Zdf8uWXXzJkyBAAMjIy2LlzJ6NHj+buu+/mvvvu44ILLmD06NFejtSYuik9J583ftjN3O93kZadz+iIEJ6fNoSR3Vr71Bw9NaV2JYCTnKnXBFXlgQce4Oabbz5h3YYNG1i6dCl/+ctfGDduHA8//HApezDGVIe07Hxe/2EX877fRXpOAef0acvtYyMY7Ed37VaH2pUAvMB9Ouhzzz2Xv/71r1x55ZU0bdqUhIQEgoKCKCgooHXr1lx11VW0bNmSuXPnHlfXuoCMqR6pWXnM+34Xr/+wmyO5BUzo2447xkXQv1MLb4fmEywBVJL7dNCTJk1i+vTpnH766QA0bdqUBQsWEB0dzb333ktAQABBQUG8/PLLAMycOZOJEyfSsWNHGwQ2pgqlZOYx9/tY3vhxDxm5BUzq357bx/agX0dr+N15NB20iEwEngMCgbmqekJfi4hMBf4OKPCLqk53lV8L/MW12T9U9Q1X+VBgPtAI54Hzd2o5wdh00HXnWI05FUkZucz5bhdvrd5NVn4h5w3owB/H9qB3e+89dcsXnPJ00CISCMwCxgPxwDoRWayqW9y2iQAeAEapaoqItHWVtwb+BkTiJIb1rropwMvATcBanAQwEfi8codpjKmLEo/kMue7WN5avYecgkIuHNiR28f2oKcfTtBWkzzpAhoORKtqLICILAKmAFvctrkJmOVq2FHVQ67yc4GvVDXZVfcrYKKIrASaq+oaV/mbwEVYAjDGVMChIzm8+m0sC9fuIa+giCmDO3Hb2T3o0bapt0PzC54kgE5AnNtyPDCixDY9AUTkB5xuor+r6hdl1O3kesWXUn4CEZkJzATo0qVLqQGqaq2/hMufntxmTHU7mJ7DyytjeOenvRQUKRcN7sRtZ3enWxtr+CuiqgaB6wERwBggFFglIgOqYseqOhuYDc4YQMn1DRs2JCkpieBg33rWZlVSVZKSkmjY0P9nHzSmMvanZfPyyhgWrYujsEi59LRO3DqmB2EhTbwdml/yJAEkAJ3dlkNdZe7igbWqmg/sEpEdOAkhAScpuNdd6SoPLWefHgkNDSU+Pp7ExMRTqe43GjZsSGhoaPkbGlMLJaRm89KKaN6LiqdIlcuGhnLb2T38ei5+X+BJAlgHRIhIOE4jPQ2YXmKbj4ErgNdFJASnSygWiAH+KSKtXNtNAB5Q1WQRSReRkTiDwNcAL5zKAQQFBREeHn4qVY0xPi4+JYtZK2J4f73Tk/z7yM7cOqY7oa2s4a8K5SYAVS0QkduBZTj9+/NUdbOIPApEqepi17oJIrIFKATuVdUkABF5DCeJADxaPCAM3Mqxy0A/xwaAjTEuB9JyeHHFTv63Lg5BmDasC38Y052OLRt5O7RaxaP7AHxFafcBGGNqj8Qjuby8MoYFa/dQVKRcPqwzt4/tQYcW1vBXxinfB2CMMdUtJTOP2d/FMv+H3eQWFHLJaaHcOS7C+virmSUAY4zXpOfk89p3u3jt+11k5hVw4cCO3HlOBN3tcs4aYQnAGFPjMnMLmP/jbmaviiUtO5+J/dpz1/ie9Gpvd+7WJEsAxpgak5NfyII1e3h5ZQxJmXmM7d2WP43vabNzeoklAGNMtcstKOTddXG8uCKag+m5nNkjhLvG92Ro11blVzbVxhKAMaba5Lsetv7819EkpGYzLKwVz14+hNO7B3s7NIMlAGNMNSgsUhb/ksBzy3eyOymLQaEteOKSAYyOCKm1U7b4I0sAxpgqU1SkfLH5AM98tYPoQxn06dCcuddEMq5PW2v4fZAlAGNMpakqX289xH++2sHW/en0aNuUWdNPY1L/9gQEWMPvqywBGGMq5ee9Kfxz6VbW7U6ha3Bj/nv5ICYP6kSgNfw+zxKAMeaUxCVn8fSy7Xz6yz5Cmtbn8Yv7MzWyM0GBAd4OzXjIEoAxpkLSsvKZtTKa+T/sJiAA7hjbg5m/607TBtac+Bv7HzPGeCSvoIgFa/bw/Dc7ScvO57LTQrl7Qi/at7AHFfkrSwDGmJNSVZZtPsCTn29jd1IWo3oE8+B5fejX0e7e9XeWAIwxZfp5bwqPL9lK1J4UIto25fXrhjGmZxu7pLOWsARgjDlBXHIWT32xjc9+3U9I0wY8cckAfj80lHo2wFurWAIwxhxlA7x1i/2vGmNsgLeOsgRgTB2mqnyx6QBPfrGNPUlZnNkjhAfP60Pfjs29HZqpAR516InIRBHZLiLRInJ/KetniEiiiGx0vW50lZ/tVrZRRHJE5CLXuvkisstt3eCqPDBjzMlt2JvCZa+s5g8LN9CgXgDzrxvGWzcMt8a/Din3G4CIBAKzgPFAPLBORBar6pYSm/5PVW93L1DVFcBg135aA9HAl26b3Kuq7596+MaYitqblMVTy7axxAZ46zxPuoCGA9GqGgsgIouAKUDJBFCey4DPVTWrgvWMMVUg8Ugus1ZEs3DtHgIDhDvGRXDzWd1oYgO8dZYn//OdgDi35XhgRCnbXSoiZwE7gLtUNa7E+mnAMyXKHheRh4GvgftVNbfkTkVkJjAToEuXLh6Ea4xxl56Tz5xVsbz2/S5yC4qYGhnKneN62gCvqbJB4E+Bd1Q1V0RuBt4AxhavFJEOwABgmVudB4ADQH1gNnAf8GjJHavqbNd6IiMjtYriNabWy8kv5M3Vu3lpZQypWfmcP7ADd4/vSbc2Tb0dmvERniSABKCz23Koq+woVU1yW5wLPF1iH1OBj1Q1363OftfHXBF5HbjH06CNMWUrKCzivfXxPLd8JwfSczirZxv+fG4ve/C6OYEnCWAdECEi4TgN/zRguvsGItLBrUGfDGwtsY8rcM74T6gjzj3lFwGbKh6+MaZYUZHy+aYD/OfL7cQezmRIl5b89/LB9vxdU6ZyE4CqFojI7TjdN4HAPFXdLCKPAlGquhi4Q0QmAwVAMjCjuL6IhOF8g/i2xK4XikgbQICNwC2VPhpj6iBVZdXOw/xr2TY2JaTTs11T5lwTyTn2GEZTDlH1n271yMhIjYqK8nYYxviMDXtTePqLbayJTSa0VSP+NL4nUwbb07jM8URkvapGliy367+M8UM7Dh7hX8u289WWg4Q0rc8jk/sxbXhnGtQL9HZoxo9YAjDGj8QlZ/Hf5Tv46OcEmtavxz0TenLdqHC7lt+cEvutMcYPHM7I5cVvnJu4AkSYObobt/yuO62a1Pd2aMaPWQIwxoel5+Qzd1Usc4/exNWZO8dF2E1cpkpYAjDGB+UVFPHm6t28uCKa1Kx8LhjYgT/ZTVymilkCMMbHrNh+iMc+20JsYqbdxGWqlSUAY3xEbGIG/1iylW+2HaJbSBNenzGMs3u39XZYphazBGCMl6Xn5PPC1zuZ/+NuGtYL5KHz+nDtGWHUr2fTM5vqZQnAGC8pKlLeXx/P08u2kZSZx9Shnbnn3F60adbA26GZOsISgDFesH5PMn9fvIXfEtIY2rUVr88YzoBQ6+c3NcsSgDE1aH9aNk9+vo1PNu6jffOGPDdtMJMHdbQ5e4xXWAIwpgbk5Bcy97tYZq2IoVCVO8b24JYx3Wlc3/4EjffYb58x1UhV+WLTAR5fupX4lGwm9W/Pg+f1oXPrxt4OzRhLAMZUl20H0nlk8RZWxybRu30z3r5pBGd0D/F2WMYcZQnAmCqWkpnHM1/tYOHaPTRvFMRjF/XnimGdqRdol3Ua32IJwJgqUlBYxMK1e3nmqx1k5BZw9ciu3DW+Jy0b24RtxjdZAjCmCvwQfZhHPt3MjoMZjOoRzMMX9KNX+2beDsv4u5x0OLQVDm2GQdMhqGonAbQEYEwlxCRm8OTn2/hqy0E6t27Eq1cPZULfdnZZp6mYgjw4vAMObXFeB13vaXHHtukUCR0GVumPtQRgzCk4nJHLc8t38vZPe2kUFMi95/bihjPDaRhkT+QyJ1FUBGl7XQ38Ztf7VkjaCUUFzjYBQRDSEzqPgMjroG0/aNcXWnSu8nAsARhTATn5hbz2/S5eXhlDdn4hV47owh3jIghpatM3+AVVyEqG1D1QkAP1GkC9hq5Xg+PfA+pBZb7JZR6Gg5uPP6tP3AZ5Gce2adkV2vaF3uc57237QnAPqFcz40YeJQARmQg8BwQCc1X1yRLrZwD/AhJcRS+q6lzXukLgN1f5XlWd7CoPBxYBwcB64GpVzavU0RhTTYqKlI83JvDvZdvZl5bD+L7tuH9Sb7rb/Py+JzsVUvc6jXzqXkjZc+xz6t7jG+CTkYDSE0Op742c94BASN7lnNVnHjq2r8bBTuM+5Cpo28c5q2/bGxp4d5yo3AQgIoHALGA8EA+sE5HFqrqlxKb/U9XbS9lFtqoOLqX8KeC/qrpIRF4BbgBerlD0xtSAH6MP8/jSrWzel87A0BY8c/lgRnYL9nZYdVde5okNe8ruY41+Ttrx29dvBq26QqtwCP+d87llFwhqDAW5zjeBMt/LWFeY67znHimxPtfZd8QEp9umbV9o1w+atKnct4lq4sk3gOFAtKrGAojIImAKUDIBeEycEbKxwHRX0RvA37EEYHzIzoNHeOLzbXyz7RCdWjbiuWmDuXBgRwICfO8P2W8V5juN6AmvdOdMPfeI05WSusfV4O+FrMPH76NeI6fRbdXV6Tcv/tyyi9PF0qiVTza+vsCTBNAJcBuKJh4YUcp2l4rIWcAO4C5VLa7TUESigALgSVX9GKfbJ1VVC9z22am0Hy4iM4GZAF26dPEgXGMq59CRHJ5dvpNFP+2lSYN63D+pNzPOCLMB3rJkpzhn4GkJTsNd3IDnHoHcjLIb9twjzplzeQLrOwOgLbtAnwucRr1lF2gV5rz76Nm1P6iqQeBPgXdUNVdEbsY5ox/rWtdVVRNEpBvwjYj8BqSVtaOSVHU2MBsgMjJSqyheY06QlVfA3O928cq3MeQVFHHN6WHcMS6C1k3q+I1cBbnHulxSdrnOxne7lvdAbhl/zoENoEFTp5+7QTNo0Byadzy2XL+pU3Z0fRmvoCYQYHdRVwdPEkAC4H79USjHBnsBUNUkt8W5wNNu6xJc77EishIYAnwAtBSReq5vASfs05iaUlikfLAhnv98uZ2D6blM7Nee+yb1JjykibdDqxlFRZBxwNWg7y7RwO+GI/sBt3OvwAauLpau0Hm460y8K7TsDA1buhr1ps6gqPFpniSAdUCE66qdBGAax/ruARCRDqq637U4GdjqKm8FZLm+GYQAo4CnVVVFZAVwGc6VQNcCn1TFARlTEd/tTOTxJVvZduAIgzq35MXppzEsrLW3w6p6BXlOw54U7bzcG/jUvc6g5lHinKm37ArdxrgGUMOc5VZh0LSdnZHXEuUmAFUtEJHbgWU4l4HOU9XNIvIoEKWqi4E7RGQyTj9/MjDDVb0P8KqIFAEBOGMAxYPH9wGLROQfwM/Aa1V4XMac1LYD6TyxdBvf7kikc+tGvHDFEC4Y2MG/7+AtKoIj+4418kkxbg3+HtDCY9s2bOE05m37QK9Jbo18mHMmb2fvdYKo+k+3emRkpEZFRXk7DOPHDqbn8MyXO3hvfRxNG9TjjnERXH16VxrU86MB3qxkV+O+s0RjHwMF2ce2C2oMrbtDSA/n5qLiV+tu0LgWfssxZRKR9aoaWbLc7gQ2dUJOfiFzVsXy0soYCoqKuG5UOH8c28N3Z+osyHUa9sM7Tzybz04+tp0EOmfuwT2ca9yDux9r6Jt1sK4ac1KWAEytpqp8vukAjy/ZSkJqNhP7teeB83rTNdhHBnjzspwz+cTtzjQBxe/JsaBFx7Zr1tFp3PtOOf5svlVXCAzyXvzGr1kCMLXWln3pPPLpZtbuSvb+E7lyj0DiDlcjv82Z+TFxm9M3X3yFjQQ6jXzbPtDvYmjT+1hD38CmnDBVzxKAqXWSMnL5z1c7WPTTXlo0CuIfF/VnWk09kSs7xa2hdzurT48/tk1gfQiOgI6nOXO8t+nlNPatu9XYJGDGgCUAU4vkFxbx1uo9PLt8B5l5hVx7Rhj/N64nLRpXcRdJUaFz6aR7v/zhHU5Dn3Hg2Hb1GkGbnhA2ymnkQ1wNfaswCLQ/PeN99ltoaoWV2w/x2GdbiEnMZHRECA9f0JeIdpWYaVHVmYPm6FU2O481+MmxUOg2cW2DFs6VNj3GHTubb9MLWnSxQVjj0ywBGL8Wm5jBP5Zs5ZtthwgLbsxr10Yytndbz6/nz8s88br5w67G3n2Kg8D6ThdNcA/oea7ThVPcP98kxOaiMX7JEoDxS+k5+bz4TTSv/7CLBvUCefC83lx7RtjJr+dPioGdXzpdNcUN/pF9x2/TorPTqA+c6na1TXdn0rEAP7pXwBgPWAIwfqWwSHl/fRz/WradpMw8fj80lHvP7U2bZmXcuZqyGzZ/BJs+hAO/OmWNWjln8N3GOI17SMSxG6SCGtXUoRjjdZYAjN9YtzuZRz7dzKaEdIZ2bcW8GcMYGNryxA3T4o81+vs2OGWdImHC48519C2r/tmqxvgjSwDG5yWkZvPk59v49Jd9dGjRkOemDWbyoI7H9/On74ctn8DmDyFurVPWYTCc84hzTX2rrl6J3RhfZgnA+KzsvEJeXRXDK9/GoAp3jIvglt91o3F9169txiFXo/8R7PkRUGjXH8b+1Wn0g7t7NX5jfJ0lAONzVJUlv+3niaXbSEjN5vyBHXhgUm9CWzWGzCT4dbHT6O/+zpkuoU1vGPOA6+7Znt4O3xi/YQnA+JS9SVn85ZNNrNqRSN8OzXlm6iBGdAiArR/AZx9B7EpnWuPgHjD6HqfRb9fX22Eb45csARifkF9YxJzvYnlu+U6CAgN4fFIXrmixiYDVL0DMN1CU7zyQZNQd0O8SaD/Arr03ppIsARivW78nmQc/3MT+g/t5oHMM05puoOGqb527bVt0hpG3OI1+xyHW6BtThSwBGK9Jy8rnhSVrOLLxE/5eP4oRjX4jILEAckNh2I1O907oMGv0jakmlgBMjdMjB/h1+UJyfv2I+4s2Uy+oiKIWYQT0u825Tr/jadboG1MDLAGYmpGWAFs/JefXj6i/by2DUOIDQ0kedBttR0wlwPr0jalxHiUAEZkIPIfzUPi5qvpkifUzgH8BCa6iF1V1rogMBl4GmgOFwOOq+j9XnfnA74DiGbdmqOrGShyL8TUpe2DrYtiyGOJ/AmCPdmY5l9HpjMu58JxxBNbEHP3GmFKVmwBEJBCYBYwH4oF1IrJYVbeU2PR/qnp7ibIs4BpV3SkiHYH1IrJMVVNd6+9V1fcrdwjGpyTFODdnbV0M+34GILN1P/7X4CoWpA+mZ9/T+NvkvnRoYXPuGONtnnwDGA5Eq2osgIgsAqYAJRPACVR1h9vnfSJyCGgDpJ5StMY3JW53Gv0ti+Hgb05Zp6Fk/+5vvHCgLy/9UkjHFg159Or+nNO3nXdjNcYc5UkC6ATEuS3HAyNK2e5SETkL2AHcparudRCR4UB9IMat+HEReRj4GrhfVXNL7lREZgIzAbp06eJBuKZGFOQ5d+OumQX7f3HKOo+Ec59A+1zA4t2BPPbZVpIzc7nxzHDuGt+TJg1syMkYX1JVf5GfAu+oaq6I3Ay8AYwtXikiHYC3gGtVtchV/ABwACcpzAbuAx4tuWNVne1aT2RkpFZRvOZUZadA1Ovw02w4st95zOHEp6DvZGjekT1Jmfzlg018t/Mwg0JbMP+6YfTv1MLbURtjSuFJAkgA3OfPDeXYYC8AqprktjgXeLp4QUSaA0uAh1R1jVud/a6PuSLyOnBPxUI3NSopBta+Aj8vgPwsZy79yS9A93EQEEBeQRFzVkTz/NfOnbx/v7AvV58eRmCAXdljjK/yJAGsAyJEJByn4Z8GTHffQEQ6uDXok4GtrvL6wEfAmyUHe4vriDOn70XApsociKkGqrB3NayeBduWQEA950lZI2+F9v2Pbha1O5kHP/qNHQczmNivvQ3yGuMnyk0AqlogIrcDy3AuA52nqptF5FEgSlUXA3eIyGSgAEgGZriqTwXOAoJdl4rCscs9F4pIG0CAjcAtVXVQppIK851B3dUvOlfyNGoNZ93j3J3brP3RzTJzC3ji860sWLOXji0aMveaSBvkNcaPiKr/dKtHRkZqVFSUt8OovbJTYcMbsPZVSE9wZtwceSsMugLqNz5u03W7k7n73V+IS8niujPCuXuCDfIa46tEZL2qRpYst79YA8m7nP79DW9BfiaEjYbzn4GICRBw/I1aOfmF/PerHcz+LpbQVo1YdNNIRnQL9lLgxpjKsARQV6lC3E9ON8+2z0ACoP9lcPqt0GFQqVU2JaTxp3c3suNgBlcM78JD5/ehqZ31G+O37K+3rikscO7SXT0LEqKgYUsY9X8w/CZo3rHUKgWFRby0Mobnv95J6yb1ef26YZzdq22Nhm2MqXqWAOqKvEzn+v21r0BaHLTuBuf9GwZPh/pNyqwWfSiDu9/dyC/xaUwe1JFHp/SjZeP6NRi4Maa6WAKoC6KXw6d3Qdpe6DoKJj0FPSdCQGCZVYqKlPk/7uapL7bRqH4gL04fwgUDS/+GYIzxT5YAarOsZPjiAfh1EQRHwIylEDaq3GrxKVnc+96vrI5NYmzvtjx5yQDaNm9YAwEbY2qSJYDaSBU2fQCf3wc5qXDWvc4D1INO3oirKu+tj+fRT7egqjx16QCmRnZGbJ5+Y2olSwC1TVo8LLkbdnzhPFlr8ifH3bVblkNHcnjww99YvvUQI8Jb8+/fD6Jz68bl1jPG+C9LALVFURFEvQbLHwEthHP/CSNuOWk/f7Glv+3noY9+IzOvkL+c34frR4UTYHP4GFPrWQKoDRJ3wOI/Qtwa6HY2XPgstAort1paVj4PL97EJxv3MTC0Bc9MHUSPts2qPVxjjG+wBODPCvLgh+dg1dMQ1BguetmZtsGDPvtvdyTy5/d/ISkjj7vO6cmtZ3cnyB7PaEydYgnAXyWsh0/+CIc2Q7+LYdLT0LT8m7Mycwv459KtLFy7l4i2TZl7zTAGhNp8/cbURZYA/E1eJnzzOKx9GZq2g2nvQO/zPKrqPoHbTaPDuXtCLxoGlT9GYIypnSwB+JOYb+DT/4PUPRB5PZzzd2hY/tm7qvLiN9E8s3yHTeBmjDnKEoA/yEqGZQ/BL287UzR7eEMXOLN33vfBr3yycR8XDe7IPy4eYBO4GWMASwC+TdV58Prnf3aexTv6bjjrz+Xe0FUs8UguN78VxYa9qdx7bi9uHdPdbuoyxhxlCcBXpSW4buj6HDoMhqs/gvYDPK6+7UA6N8yPIikzl5evPI1JAzpUX6zGGL9kCcDXqELUPPjqb1BUAOMfc57KFej5f9U32w7yx7d/pmnDerx38xl2lY8xplSWAHxJYT58eidsXAjhZ8GFzznTNntIVZn3w24eX7KFPh2a89q1w2jfwiZxM8aUzqM7f0RkoohsF5FoEbm/lPUzRCRRRDa6Xje6rbtWRHa6Xte6lQ8Vkd9c+3xe6nrndG4GvDPNafx/dx9cs7hCjX9+YREPfbyJxz7bwjl92vHeLadb42+MOalyvwGISCAwCxgPxAPrRGSxqm4psen/VPX2EnVbA38DIgEF1rvqpgAvAzcBa4GlwETg80oej3/KSIS3fw/7f3HO+ofOqFD1tKx8bn17PT9EJ/GHMd25d0Ivm8vHGFMuT7qAhgPRqhoLICKLgClAyQRQmnOBr1Q12VX3K2CiiKwEmqvqGlf5m8BF1MUEkBQDCy6FIwecm7p6TaxQ9d2HM7n+jXXEJWfxr8sG8vvIztUUqDGmtvEkAXQC4tyW44ERpWx3qYicBewA7lLVuDLqdnK94kspr1vi1ztn/qow4zMIjaxQ9TWxSdyyYD0CLLhhhN3cZYypkKqa/etTIExVBwJfAW9U0X4RkZkiEiUiUYmJiVW1W+/bsQzeuADqN4Ubvqpw4//uujiufm0twU3q8/Fto6zxN8ZUmCcJIAFw71cIdZUdpapJqprrWpwLDC2nboLrc5n7dNv3bFWNVNXINm3aeBCuH9jwJrxzBYREOI1/SA+PqxYWKU8s3cqfP/iVkd2C+fDWUXQNLvuh7sYYUxZPEsA6IEJEwkWkPjANWOy+gYi432U0Gdjq+rwMmCAirUSkFTABWKaq+4F0ERnpuvrnGuCTSh6L71OFlU85c/d3GwMzlkCzdh5Xz8wt4JYF63l1VSxXj+zK6zOG0aJRUPXFa4yp1codA1DVAhG5HacxDwTmqepmEXkUiFLVxcAdIjIZKACSgRmuuski8hhOEgF4tHhAGLgVmA80whn8rd0DwIUFsPRuWD/fmbN/8gsQ6HnjvS81mxveiGL7gXQemdyPa88Iq7ZQjTF1g6iqt2PwWGRkpEZFRXk7jIrLy4L3r3emdRh9N4z9q0cPbSn2S1wqN74ZRXZeIS9OH8KYXuXP+2+MMcVEZL2qnjDQaHcCV7fMJHjncoiPgvP+DcNvqlD1Jb/u50/vbqRNswYsvHEEPdvZIxuNMVXDEkB1StntXOOfFg+XvwV9LvS4avEc/v/5agdDu7bi1auHEtK0QfXFaoypcywBVJd9G2Hh76EwD675BLqM9LhqTn4h93/wKx9v3MfFQzrxxCUD7MldxpgqZwmgOkR/De9eA41aOTd4tenlcdXUrDxueCOK9XtSuGdCT247u4fN4W+MqRaWAKraL4vgk9ugTW+48n1o7vk8/CmZeVw5dy3RiRnMmn4a5w+0OfyNMdXHEkBVUYUfnoXlf3emcr58gUfP6y2W7Gr8YxIzmHNNJL/rWUtuejPG+CxLAFWhqBC+uB9+mg39L4OLXoZ69T2unpyZx/Q5a9h1OJO510RyljX+xpgaYAmgsvJz4MObYOtiOOOPcM6jEOD5FEtJGblcOXet0/hfG8noCGv8jTE1wxJAZWSnwDvTYe9qOPcJOP3WClV3b/xfu3YYZ0aEVFOgxhhzIksApyotARZcAsmxcNk86H9Jhaofzsjlyjlr2ZOcybwZwxjVwxp/Y0zNsgRwKg5Hw1sXQXYqXPUhhI+uWPWMXKbPWcPe5CzmXTuMM6zxN8Z4gSWAitr3Myy4zJnL57ol0GFQhaonHnEa/7iULObNGMYZ3a3xN8Z4R1U9EKZu2LUK5l8IQY3h+mUVbvwPHcnhijlriE/J5vUZw63xN8Z4lSUAT239zJnXp0Uo3LAMgrtXqPqhIzlcMXsNCSnZvH7dME7vbk/wMsZ4lyUAT/y8AN692jnjv24pNO9YoeqH0p3Gf39aDvOvG8ZIe3yjMcYHWAIozw/POVM7dB/rTOrWuHWFqh9Kz2HanOLGf7g9u9cY4zNsELgsqrD8b04C6H8pXPRKhe7uBTjoOvM/mJ7DG9cPZ1hYxZKHMcZUJ0sApSksgM/+D35+C4bdCJP+VaG7e+HExj/SGn9jjI+xBFBSfg58cANs+wx+dx+MeaBCj28EOJDmXO1zyBp/Y4wPswTgLicdFk2H3d/BpKdhxM0V3sX+tGyumL2Gwxl5vHnDcIZ2tcbfGOObPOrXEJGJIrJdRKJF5P6TbHepiKiIRLqWrxSRjW6vIhEZ7Fq30rXP4nXefdJ5RiK8cYEzr88lc06p8d+Xms00V+P/xvXW+BtjfFu53wBEJBCYBYwH4oF1IrJYVbeU2K4ZcCewtrhMVRcCC13rBwAfq+pGt2pXqmpUZQ+i0lL3wlsXO/P7THsHek6o8C6KG/+UTOfM/7QuraohUGOMqTqefAMYDkSraqyq5gGLgCmlbPcY8BSQU8Z+rnDV9S2HtsFr50JmIlzz8Sk1/gnW+Btj/JAnCaATEOe2HO8qO0pETgM6q+qSk+zncuCdEmWvu7p//iplPPhWRGaKSJSIRCUmJnoQbgXER8HrE0GL4LrPK/Tg9qO7SMli2uzVpGTl8daNIxhijb8xxk9U+kYwEQkAngHuPsk2I4AsVd3kVnylqg4ARrteV5dWV1Vnq2qkqka2aVOFD0uJ+QbemAwNW8L1X0C7fhXehdP4ryE1K58FN4xgcOeWVRefMcZUM08SQALQ2W051FVWrBnQH1gpIruBkcDi4oFgl2mUOPtX1QTX+xHgbZyuppqx+SNYOBVad3MmdWsdXuFdOI9xXEt6dj4LbxzBIGv8jTF+xpMEsA6IEJFwEamP05gvLl6pqmmqGqKqYaoaBqwBJhcP7rq+IUzFrf9fROqJSIjrcxBwAeD+7aD6rHsN3rsOQiNhxmfQrF2Fd1FQWMTtb2/ggOs6/4GhLas+TmOMqWblXgWkqgUicjuwDAgE5qnqZhF5FIhS1cUn3wNnAXGqGutW1gBY5mr8A4HlwJxTOgJPqcJ3/4Zv/gE9J8Jlr0P9xqe0qyc/38aPMUn867KB1udvjPFbHt0IpqpLgaUlyh4uY9sxJZZX4nQLuZdlAkMrEGflFBXBlw/Bmpdg4DSY8iIEBp3Srj7+OYG53+9ixhlh/D6yc/kVjDHGR9X+O4FVndk8f3kbRt4KEx6v8Lw+xTYlpHHfB78yIrw1D53fp4oDNcaYmlX7E4AIhA6F4G4w+p4Kz+tTLCkjl5vfWk9wk/rMuvI0ggJtJm1jjH+r/QkAnBk9K8EZ9P2ZxIxc3r/ldEKaNqiiwIwxxnvsNNYD/1y6jdWxSTxx8QC74scYU2tYAijHhxvimffDLq4bFcalQ0O9HY4xxlQZSwAn8Vt8Gg98+Bsju7XmwfNs0NcYU7tYAijD4Yxcbn4ripCmDZg13QZ9jTG1T90YBK6g/MIiblu4gaTMPD74wxkE26CvMaYWsgRQiseXbGXtrmT+e/kg+ndq4e1wjDGmWli/RgkfrI9n/o+7uX5UOBcPsUFfY0ztZQnAza/xqTzw0W+c0T2YB8/r7e1wjDGmWlkCcEk84tzp26ZpA16cfhr1bNDXGFPL2RgArkHftzeQkpXH+7ecQesm9b0dkjHGVDtLAMA/PtvCT7uSeW7aYBv0NcbUGXW+n+O9qDjeWL2HG88MZ8rgTuVXMMaYWqJOJ4CNcak89PEmRvUI5v5JNuhrjKlb6mwCOHQkh1veWk/bZg144Qob9DXG1D11cgwgr8C50zc1O48P/zDKBn2NMXVSnUwAj322hXW7U3j+iiH07djc2+EYY4xXeNTvISITRWS7iESLyP0n2e5SEVERiXQth4lItohsdL1ecdt2qIj85trn8yKn+KiuCnp3XRxvrdnDzLO6MXlQx5r4kcYY45PK/QYgIoHALGA8EA+sE5HFqrqlxHbNgDuBtSV2EaOqg0vZ9cvATa7tlwITgc8regAV8fPeFP7y8SbO7BHCn8/tVZ0/yhhjfJ4n3wCGA9GqGquqecAiYEop2z0GPAXklLdDEekANFfVNaqqwJvARR5HfQoOHcnhlgXradeiAS9cMcQGfY0xdZ4nrWAnIM5tOd5VdpSInAZ0VtUlpdQPF5GfReRbERntts/4k+3Tbd8zRSRKRKISExM9CPdEeQVF3LpgA+nZBbx6VSStbNDXGGMqPwgsIgHAM8CMUlbvB7qoapKIDAU+FpF+Fdm/qs4GZgNERkbqqcT4yKebidqTwovTbdDXGGOKeZIAEoDObsuhrrJizYD+wErXOG57YLGITFbVKCAXQFXXi0gM0NNVP/Qk+6wyqkp4SBNuO7s7Fwy0QV9jjCnmSQJYB0SISDhOIz0NmF68UlXTgJDiZRFZCdyjqlEi0gZIVtVCEekGRACxqposIukiMhJnEPga4IWqOih3IsKNo7tVx66NMcavlZsAVLVARG4HlgGBwDxV3SwijwJRqrr4JNXPAh4VkXygCLhFVZNd624F5gONcK7+qdYrgIwxxhxPnItw/ENkZKRGRUV5OwxjjPErIrJeVSNLltu1kMYYU0dZAjDGmDrKEoAxxtRRlgCMMaaOsgRgjDF1lCUAY4ypo/zqMlARSQT2nGL1EOBwFYZT3fwpXou1+vhTvP4UK/hXvJWNtauqtilZ6FcJoDJEJKq062B9lT/Fa7FWH3+K159iBf+Kt7pitS4gY4ypoywBGGNMHVWXEsBsbwdQQf4Ur8VaffwpXn+KFfwr3mqJtc6MARhjjDleXfoGYIwxxo0lAGOMqaPqRAIQkYkisl1EokXkfm/HUxYR6SwiK0Rki4hsFpE7vR1TeUQk0PXM58+8HUt5RKSliLwvIttEZKuInO7tmMoiIne5fgc2icg7ItLQ2zG5E5F5InJIRDa5lbUWka9EZKfrvZU3Y3RXRrz/cv0u/CoiH4lISy+GeFRpsbqtu1tEVERCSqtbUbU+AYhIIDALmAT0Ba4Qkb7ejapMBcDdqtoXGAnc5sOxFrsT2OrtIDz0HPCFqvYGBuGjcYtIJ+AOIFJV++M8iGmad6M6wXxgYomy+4GvVTUC+Nq17Cvmc2K8XwH9VXUgsAN4oKaDKsN8TowVEekMTAD2VtUPqvUJABgORKtqrKrmAYuAKV6OqVSqul9VN7g+H8FpoDp5N6qyiUgocD4w19uxlEdEWuA8oe41AFXNU9VUrwZ1cvWARiJSD2gM7PNyPMdR1VVAconiKcAbrs9vABfVZEwnU1q8qvqlqha4Ftdw/HPKvaaMf1uA/wJ/Bqrsyp26kAA6AXFuy/H4cKNaTETCgCE4z0z2Vc/i/EIWeTkOT4QDicDrri6ruSLSxNtBlUZVE4B/45zp7QfSVPVL70blkXaqut/1+QDQzpvBVND1+PBjaUVkCpCgqr9U5X7rQgLwOyLSFPgA+D9VTfd2PKURkQuAQ6q63tuxeKgecBrwsqoOATLxrS6Ko1x951NwklZHoImIXOXdqCpGnevL/eIacxF5CKf7daG3YymNiDQGHgQerup914UEkAB0dlsOdZX5JBEJwmn8F6rqh96O5yRGAZNFZDdOt9pYEVng3ZBOKh6IV9Xib1Tv4yQEX3QOsEtVE1U1H/gQOMPLMXnioIh0AHC9H/JyPOUSkRnABcCV6rs3RXXHORn4xfX3FgpsEJH2ld1xXUgA64AIEQkXkfo4g2mLvRxTqUREcPqot6rqM96O52RU9QFVDVXVMJx/029U1WfPUlX1ABAnIr1cReOALV4M6WT2AiNFpLHrd2IcPjpgXcJi4FrX52uBT7wYS7lEZCJOF+ZkVc3ydjxlUdXfVLWtqoa5/t7igdNcv9OVUusTgGuQ53ZgGc4f0buqutm7UZVpFHA1ztn0RtfrPG8HVYv8EVgoIr8Cg4F/ejec0rm+pbwPbAB+w/k79alpC0TkHWA10EtE4kXkBuBJYLyI7MT5FvOkN2N0V0a8LwLNgK9cf2uveDVIlzJirZ6f5bvfeowxxlSnWv8NwBhjTOksARhjTB1lCcAYY+ooSwDGGFNHWQIwxpg6yhKAMcbUUZYAjDGmjvp/Hr0EiwBkcdgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "plt.plot(history.history['acc'], label='train')\n",
    "plt.plot(history.history['val_acc'], label='test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "9569db3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "reverse_target_word_index=y_tokenizer.index_word\n",
    "reverse_source_word_index=x_tokenizer.index_word\n",
    "target_word_index=y_tokenizer.word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2be1dbad",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34db6cdc",
   "metadata": {},
   "source": [
    "Set up the inference for the encoder and decoder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "c2088535",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode the input sequence to get the feature vector\n",
    "encoder_model = Model(inputs=encoder_inputs,outputs=[encoder_outputs, state_h, state_c])\n",
    "\n",
    "# Decoder setup\n",
    "# Below tensors will hold the states of the previous time step\n",
    "decoder_state_input_h = Input(shape=(latent_dim,))\n",
    "decoder_state_input_c = Input(shape=(latent_dim,))\n",
    "decoder_hidden_state_input = Input(shape=(max_text_len,latent_dim))\n",
    "\n",
    "# Get the embeddings of the decoder sequence\n",
    "dec_emb2= dec_emb_layer(decoder_inputs) \n",
    "# To predict the next word in the sequence, set the initial states to the states from the previous time step\n",
    "decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=[decoder_state_input_h, decoder_state_input_c])\n",
    "\n",
    "#attention inference\n",
    "attn_out_inf, attn_states_inf = attn_layer([decoder_hidden_state_input, decoder_outputs2])\n",
    "decoder_inf_concat = Concatenate(axis=-1, name='concat')([decoder_outputs2, attn_out_inf])\n",
    "\n",
    "# A dense softmax layer to generate prob dist. over the target vocabulary\n",
    "decoder_outputs2 = decoder_dense(decoder_inf_concat) \n",
    "\n",
    "# Final decoder model\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + [decoder_hidden_state_input,decoder_state_input_h, decoder_state_input_c],\n",
    "    [decoder_outputs2] + [state_h2, state_c2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "551e4cfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    # Encode the input as state vectors.\n",
    "    e_out, e_h, e_c = encoder_model.predict(input_seq)\n",
    "    \n",
    "    # Generate empty target sequence of length 1.\n",
    "    target_seq = np.zeros((1,1))\n",
    "    \n",
    "    # Populate the first word of target sequence with the start word.\n",
    "    target_seq[0, 0] = target_word_index['sostok']\n",
    "\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition:\n",
    "      \n",
    "        output_tokens, h, c = decoder_model.predict([target_seq] + [e_out, e_h, e_c])\n",
    "\n",
    "        # Sample a token\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_token = reverse_target_word_index[sampled_token_index]\n",
    "        \n",
    "        if(sampled_token!='eostok'):\n",
    "            decoded_sentence += ' '+sampled_token\n",
    "\n",
    "        # Exit condition: either hit max length or find stop word.\n",
    "        if (sampled_token == 'eostok'  or len(decoded_sentence.split()) >= (max_summary_len-1)):\n",
    "            stop_condition = True\n",
    "\n",
    "        # Update the target sequence (of length 1).\n",
    "        target_seq = np.zeros((1,1))\n",
    "        target_seq[0, 0] = sampled_token_index\n",
    "\n",
    "        # Update internal states\n",
    "        e_h, e_c = h, c\n",
    "\n",
    "    return decoded_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "1e5ba346",
   "metadata": {},
   "outputs": [],
   "source": [
    "# below functions converts sequence into texts\n",
    "\n",
    "def seq2summary(input_seq):\n",
    "    newString=''\n",
    "    for i in input_seq:\n",
    "        if((i!=0 and i!=target_word_index['sostok']) and i!=target_word_index['eostok']):\n",
    "            newString=newString+reverse_target_word_index[i]+' '\n",
    "    return newString\n",
    "\n",
    "def seq2text(input_seq):\n",
    "    newString=''\n",
    "    for i in input_seq:\n",
    "        if(i!=0):\n",
    "            newString=newString+reverse_source_word_index[i]+' '\n",
    "    return newString"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "5cac8062",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review: the wwe has confirmed the will return to the ring for the first time since wrestlemania in which he lost and left his iconic coat and hat in the sparking retirement he will return on flagship raw anniversary on january who retired in will also attend the \n",
      "Original summary: the to make wwe return on anniversary \n",
      "Predicted summary:  indian to get its first weight\n",
      "\n",
      "\n",
      "Review: as many as three people were killed in riots after dera sacha sauda chief gurmeet ram rahim was convicted of rape by a cbi court on railway station and a petrol pump were set on fire amid clashes between security forces and the reports said over cars were set on \n",
      "Original summary: dead in riots after dera chief ram rahim convicted of rape \n",
      "Predicted summary:  dera sacha sauda follower ram dera dera campus\n",
      "\n",
      "\n",
      "Review: bengaluru police has claimed they have solved cases after arresting a in the named had snatched gold chains of people in and around and had warrants pending against him from different police said they have recovered kg of gold ornaments worth crore and five motorbikes from \n",
      "Original summary: bengaluru police claims solving cases with one arrest \n",
      "Predicted summary:  police officers to get record in mumbai\n",
      "\n",
      "\n",
      "Review: around tonnes of meat sourced from animal carcasses has been seized from an ice factory in while three accused in the case said the meat was mostly of stray dogs and municipal authorities said it will be tested to confirm the the which was chemically treated at was to be sold to \n",
      "Original summary: meat from dog for sale seized in wb \n",
      "Predicted summary:  kg heroin worth seized seized in spain\n",
      "\n",
      "\n",
      "Review: australia gave away five runs as penalty after a ball from nathan lyon hit the helmet on the final day of the fifth test in sydney on the ball slipped past wicketkeeper tim paine before rolling on to hit his helmet placed right behind at the australia were one wicket away from the \n",
      "Original summary: australia handed penalty as ball hits helmet \n",
      "Predicted summary:  england lose test wickets in a test\n",
      "\n",
      "\n",
      "Review: sharing an old poem on twitter which sushant singh rajput had written for his late he day sushant had earlier revealed the poem was his first ever attempt at in another sushant wish with all my could just be able to have one more minute with \n",
      "Original summary: sushant shares old poem he had written for his late mother \n",
      "Predicted summary:  happy to see my abhishek on karvachauth\n",
      "\n",
      "\n",
      "Review: sports minister vijay goel conducted a surprise check at indira gandhi venue of the asian wrestling and cleaned the toilet himself after finding it visit came after international wrestlers reportedly complained about dirty saying they had to relieve themselves in the goel also pulled up sports authority of india officials at the \n",
      "Original summary: sports minister goel cleans toilet at wrestling venue \n",
      "Predicted summary:  minister throws sports minister rathore at mumbai airport\n",
      "\n",
      "\n",
      "Review: there are no and shiv sena members in the list of new ministers who were inducted into the union cabinet on this was because the bjp leadership was unable to come up with a formula for representation which could satisfy all according to shiv sena has three members in the rajya while has \n",
      "Original summary: no shiv sena names in list of new union ministers \n",
      "Predicted summary:  shiv sena launches portal for comparing to shiv sena\n",
      "\n",
      "\n",
      "Review: netflix on tuesday confirmed continue its association with anurag vikramaditya motwane and writer varun grover for season they added they carried out an independent investigation and are satisfied with the while varun was accused of sexual kashyap and motwane were criticised for their alleged inaction over allegations against their business partner vikas \n",
      "Original summary: netflix to continue working with anurag kashyap for sacred games \n",
      "Predicted summary:  netflix fires kevin spacey as he is anurag kashyap\n",
      "\n",
      "\n",
      "Review: a group of labourers living at a makeshift settlement near on friday travelled over km to reach their respective polling stations to cast the labourers who otherwise travel in bullock carts to arranged for private vehicles to reach the polling have to shows my one of the labourers \n",
      "Original summary: labourers travel km to vote for telangana elections \n",
      "Predicted summary:  labourers climb to build beer in telangana\n",
      "\n",
      "\n",
      "Review: mohammad wife hasin jahan has shared screenshots of the alleged chat with an african woman with the ki in the comment jahan ko v nhi tha this comes two days after shami was awarded a contract by the bcci after clearing him of corruption \n",
      "Original summary: wali ko bhi nahin wife hasin jahan \n",
      "Predicted summary:  wife accuses wife of wife wanting to marry him\n",
      "\n",
      "\n",
      "Review: the outgoing indian air force chief raha on wednesday said the worst moment during his tenure was when he heard about an plane going missing with personnel on citing that the iaf undertook over flying hours to search for the raha termed it as a mistake of not fitting with underwater \n",
      "Original summary: missing plane my worst moment in iaf chief \n",
      "Predicted summary:  iaf chief pilot cancels flight visit\n",
      "\n",
      "\n",
      "Review: the passport of billionaire jeweller nirav modi has been revoked in connection with the pnb lawyer vijay aggarwal has the ministry of external affairs had served a notice to nirav modi seeking response as to why his passport be nirav reportedly replied to the notice raising objections on the notice issued to \n",
      "Original summary: nirav passport revoked over pnb fraud \n",
      "Predicted summary:  nirav modi pnb pnb fraud in pnb fraud case\n",
      "\n",
      "\n",
      "Review: sonu who will be portraying the villain in said that his role is very grey character gets the job is added sood all here to entertain you and sure you will be he further produced by karan the rohit shetty directorial is scheduled to release on december \n",
      "Original summary: playing villain in is very sonu sood \n",
      "Predicted summary:  sonu sood to make comeback with sonu\n",
      "\n",
      "\n",
      "Review: at annual shareholder james asked the ceo mark zuckerberg to george not vladimir at the meeting where shareholders proposed to limit called current a shares have times the voting power of that available to regular \n",
      "Original summary: copy not shareholder to facebook ceo \n",
      "Predicted summary:  ceo sikka resigns from tata steel ceo\n",
      "\n",
      "\n",
      "Review: expressing displeasure over supreme recent ban on sale of firecrackers in delhi and tripura governor tathagata roy on tuesday remarked that the will soon file pleas against hindu roy added that as a practising he was not happy with the ban since it deprived the community of an important aspect of \n",
      "Original summary: soon there will be plea against hindu tripura guv \n",
      "Predicted summary:  sc order to ban on tobacco sale of unity\n",
      "\n",
      "\n",
      "Review: the vaishno devi cave shrine pilgrimage has been suspended from the katra base camp due to heavy rush of officials said the ticket counter had to be closed nearly two hours before scheduled closure due to a huge number of around pilgrims are stranded at the katra base officials further \n",
      "Original summary: vaishno devi pilgrimage suspended due to heavy crowd \n",
      "Predicted summary:  bus service connecting to be suspended\n",
      "\n",
      "\n",
      "Review: reacting to naseeruddin statement that he fears for his safety in paresh rawal not right to say not worth living i keep but not right to paresh to bulandshahr shah had said a death is more significant than that of a \n",
      "Original summary: not right to say india not worth living rawal on remark \n",
      "Predicted summary:  i am not a not a rajinikanth\n",
      "\n",
      "\n",
      "Review: flying car startup kitty ceo sebastian thrun in a recent interview admire indian people but i admire the traffic in adding that kitty vision is that people will be able to travel in the air daily in the he also that there be any traffic in the \n",
      "Original summary: i indians but not traffic in kitty hawk ceo \n",
      "Predicted summary:  startup ceo steps down after years\n",
      "\n",
      "\n",
      "Review: olympic gold winner usain bolt posted his first sub metre sprint time of the year at the monaco diamond league meeting on the who went to germany to have treatment for back issues he had for clocked winning the race by of a \n",
      "Original summary: bolt runs race in under sec for first time this year \n",
      "Predicted summary:  usain bolt breaks world record for the first time\n",
      "\n",
      "\n",
      "Review: comedian krushna who became a father to twin has revealed that salman khan was among those who counselled him about having was one of the first stars i told about my he krushna further said when he first saw the he know how to react and was scared to pick them \n",
      "Original summary: salman me about having krushna abhishek \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted summary:  salman khan is a salman khan\n",
      "\n",
      "\n",
      "Review: most valuable bank jpmorgan ceo jamie dimon the us truly an exceptional and is clear that something is in a letter to dimon wrote us has dumped trillions of dollars into wars and piled huge debt onto us has forced foreigners to leave after getting advanced he \n",
      "Original summary: clearly something is wrong with says jpmorgan ceo \n",
      "Predicted summary:  clearly something is a billionaire soros\n",
      "\n",
      "\n",
      "Review: tweeting his picture with singer and youtuber vidya iyer as vidya actor hrithik roshan vox is in the my what do i such a vidya is known for mixing western electronic dance music and indian classical she also shared the pictures while dreams come \n",
      "Original summary: such a hrithik on vidya \n",
      "Predicted summary:  hrithik is a hrithik on her roka\n",
      "\n",
      "\n",
      "Review: singer katy during her ongoing live stream on has said that she singer taylor swift and is sorry for anything she ever did while hope the same from while addressing her feud with further can be representatives of strong women that come together despite their \n",
      "Original summary: i forgive sorry for anything i says katy perry \n",
      "Predicted summary:  katy perry reveals she is miss universe\n",
      "\n",
      "\n",
      "Review: rape convict gurmeet ram adopted daughter honeypreet and others in their confessions revealed that honeypreet planned to settle down in a foreign country with ram rahim after his escape post they further revealed they wanted to indulge in after settling down and avenge ram arrest by out from the \n",
      "Original summary: honeypreet planned to settle abroad with ram rahim \n",
      "Predicted summary:  honeypreet panchkula honeypreet granted bail\n",
      "\n",
      "\n",
      "Review: latest update for its virtual assistant allows users to disable the dedicated bixby button in galaxy plus and note the which requires the latest version of bixby and bixby will stop bixby home from opening when the button is it will still be accessible by the \n",
      "Original summary: samsung lets users dedicated button \n",
      "Predicted summary:  oneplus to notify users to unlock ads from report\n",
      "\n",
      "\n",
      "Review: addressing a rally in prime minister narendra modi asked why former pm manmohan singh show the courage to order a surgical strike post the mumbai terror the air force had approached singh with a plan for surgical strikes post the attacks but the government did not show the courage to order pm modi \n",
      "Original summary: why order surgical strikes post asks pm \n",
      "Predicted summary:  surgical strikes emerge surgical strikes in pm modi\n",
      "\n",
      "\n",
      "Review: congratulating the indian cricket team after india became the first asian team to win a test series in anushka sharma posted pictures of team india and written and created by this farhan akhtar also to virat kohli and team india for finally breaking through the aussie historic tweeted anil \n",
      "Original summary: history anushka on test series win in australia \n",
      "Predicted summary:  india need to win a series with anushka\n",
      "\n",
      "\n",
      "Review: the supreme court on tuesday observed that hotels and restaurants were not bound by the maximum retail price for selling bottled earlier the same the centre had told the sc that selling bottled mineral water above mrp would attract monetary penalty and jail term for the management of it claimed overcharging would lead to tax \n",
      "Original summary: restaurants not bound by mrp in selling bottled sc \n",
      "Predicted summary:  sc orders plan to sell bottled water to sell packaged products\n",
      "\n",
      "\n",
      "Review: the indian air force has filed an fir against unknown persons for creating a fake iaf website and duping job wing commander singh had found and reported the which claimed that recruitment for iaf was on and had downloadable application were different options to make payment for the singh n \n",
      "Original summary: iaf files fir over fake website duping job seekers \n",
      "Predicted summary:  fir against iaf officials over fake fir\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(30):\n",
    "    print(\"Review:\",seq2text(x_tr[i]))\n",
    "    print(\"Original summary:\",seq2summary(y_tr[i]))\n",
    "    print(\"Predicted summary:\",decode_sequence(x_tr[i].reshape(1,max_text_len)))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9167e0c",
   "metadata": {},
   "source": [
    "## Limitations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a1a804d",
   "metadata": {},
   "source": [
    "Our biggest limitation was the lack of time left after finishing the implementation\n",
    "of our attentive encode-decoder RNN model. For lack of a better word, abstractive\n",
    "summarization is a messy task – training and truth data are not well-aligned, compared to the near one-to-one alignment for machine translation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ee5d6ea",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f216e00",
   "metadata": {},
   "source": [
    "We successfully managed to implement the attentive encoder-decoder RNN\n",
    "summarizer. As you can see the predicted summaries do not match in terms of words with actual summaries. But in most of the cases both will convey same meaning. The model has it flaws like it generates same words in a sequence."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "15982de9ef07d49ca26cdffe937ab9b96dac0fd348fa6aba314f0d9731c21e86"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}