W3-Transformers-EngToTamil-Translation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "* In this assignment you will be using the entire transformer architecture for a translation task.\n",
        "* we will just be using one encoder layer and one decoder layer\n",
        "* You can copy the encoder and decoder modules from the previous assignments. You are going to translate a few sentences from **English to Tamil**\n",
        "  * Source language: English\n",
        "  * Target language: Tamil\n",
        "\n",
        "* You may experiment with a target language of your choice for checking the impelementation. (You may use google translate for that)\n",
        "\n",
        "* We need to install torchdata and torchtext (which take about 3 minutes to finish installing) for tokenizing the text.\n",
        "* We already defined useful functions for the tokenization of texts\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "HuEecgDwXV4c"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install torchdata==0.6.0 # to be compatible with torch 2.0\n",
        "!pip install portalocker==2.0.0\n",
        "!pip install -U torchtext==0.15.1"
      ],
      "metadata": {
        "id": "m1TwVG-9cQmG"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* Let's import all required libraries"
      ],
      "metadata": {
        "id": "qiGXHBiwcvI1"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "w4nHBd0CHVX5"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from torch import Tensor\n",
        "\n",
        "import torch.nn as nn\n",
        "from torch.nn import Parameter\n",
        "import torch.nn.functional as F\n",
        "from torch.nn.functional import one_hot\n",
        "\n",
        "import torch.optim as optim\n",
        "\n",
        "#text lib\n",
        "import torchtext\n",
        "\n",
        "# tokenizer\n",
        "from torchtext.data.utils import get_tokenizer\n",
        "\n",
        "#build vocabulary\n",
        "from torchtext.vocab import vocab\n",
        "from torchtext.vocab import build_vocab_from_iterator\n",
        "\n",
        "# get input_ids (numericalization)\n",
        "from torchtext.transforms import VocabTransform, LabelToIndex\n",
        "\n",
        "# get embeddings\n",
        "from torch.nn import Embedding\n",
        "\n",
        "from  pprint import pprint\n",
        "from yaml import safe_load\n",
        "import copy\n",
        "import numpy as np\n",
        "import requests\n",
        "import math"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Preparing Data"
      ],
      "metadata": {
        "id": "1UbaTNbHac1g"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "* Source and target text"
      ],
      "metadata": {
        "id": "8O1Qu_Z8pBZF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "src_text = \"\"\"The most famous ruler of ancient India was Emperor Ashoka.\n",
        "It was during his period that Buddhism spread to different parts of Asia.\n",
        "Ashoka gave up war after seeing many people grieving death after the Kalinga war.\n",
        "He embraced Buddhism and then devoted his life to spread the message of peace and dharma.\n",
        "His service for the cause of public good was exemplary.\n",
        "He was the first ruler to give up war after victory.\n",
        "He was the first to build hospitals for animals.\n",
        "He was the first to lay roads.\"\"\""
      ],
      "metadata": {
        "id": "9rv6jV3yaf1U"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tar_text = \"\"\"பண்டைய இந்திய அரசர்களில் பேரும் புகழும் பெற்ற அரசர் அசோகர் ஆவார்.\n",
        "இவரது ஆட்சியில் தான் புத்த மதம் ஆசியாவின் பல்வேறு பகுதிகளுக்குப் பரவியது.\n",
        "கலிங்கப் போருக்குப் பின் பல உயிர்கள் மடிவதைக் கண்டு வருந்தி, போர் தொடுப்பதைக் கைவிட்டார்.\n",
        "அதற்குப் பிறகு புத்த சமயத்தைத் தழுவி, அமைதியையும் அறத்தையும் பரப்புவதற்காகத் தன் வாழ்வையே அர்ப்பணித்தார்.\n",
        "பொதுமக்களுக்கு அவர் ஆற்றிய சேவை முன் மாதிரியாக விளங்கியது.\n",
        "வெற்றிக்குப் பின் போரைத் துறந்த முதல் அரசர் அசோகர்தான்.\n",
        "உலகிலேயே முதன்முதலாக விலங்குகளுக்கும் தனியே மருத்துவமனை அமைத்துத் தந்தவரும் அசோகரே ஆவார்.\n",
        " இன்றும் அவர் உருவாக்கிய சாலைகளை நாம் பயன்படுத்திக்கொண்டு இருக்கிறோம்.\"\"\""
      ],
      "metadata": {
        "id": "fFXiagnva88E"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* Tokenize and build vocabulary using a simple tokenization algorithm"
      ],
      "metadata": {
        "id": "h7vjJPzJpNrB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "def seq_len(seq):\n",
        "  return len(seq.strip('').split(' '))\n",
        "\n",
        "# check the maximum leneght of the src and target seq to decide the context length of encdoer and decoder\n",
        "src_raw_seq = src_text.strip('').split('\\n')\n",
        "src_max_seq_len =max(list(map(seq_len,src_raw_seq)))\n",
        "print('Source max_seq_length:  ',src_max_seq_len)\n",
        "\n",
        "\n",
        "tar_raw_seq = tar_text.strip('').split('\\n')\n",
        "tar_max_seq_len =max(list(map(seq_len,tar_raw_seq)))\n",
        "print('Target max_seq_length: ',tar_max_seq_len)"
      ],
      "metadata": {
        "id": "3renrPd6fh01"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* We encourage you to go through the code given below to understand the typical functionalities of Tokenizer object (If you want, you can skip)"
      ],
      "metadata": {
        "id": "4xhQw4aM66_K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "class Tokenizer(object):\n",
        "\n",
        "  def __init__(self,text):\n",
        "    self.text = text\n",
        "    self.word_tokenizer = self.word_tokenizer\n",
        "    self.vocab_size = None\n",
        "    self.vocab = None\n",
        "\n",
        "  @staticmethod\n",
        "  def word_tokenizer(seq):\n",
        "    return seq.strip('').split(' ')\n",
        "\n",
        "  def get_tokens(self):\n",
        "    for sentence in self.text.strip().split('\\n'):\n",
        "      yield self.word_tokenizer(sentence)\n",
        "\n",
        "  def build_vocab(self):\n",
        "    self.vocab = build_vocab_from_iterator(self.get_tokens(),\n",
        "                                  min_freq=1,specials=['<pad>','<start>','<end>','<unk>'])\n",
        "    self.vocab.set_default_index(self.vocab['<unk>']) # index of OOV\n",
        "    self.vocab_size = len(self.vocab)\n",
        "    return self.vocab\n",
        "\n",
        "  def encode(self,sentence):\n",
        "    v = self.build_vocab()\n",
        "    vt = VocabTransform(v)\n",
        "    token_ids = vt(self.word_tokenizer(sentence))\n",
        "    # add special tokens\n",
        "    token_ids.insert(0,v.vocab.get_stoi()['<start>'])\n",
        "    token_ids.append(v.vocab.get_stoi()['<end>']) # <end>:2\n",
        "    return torch.tensor(token_ids,dtype=torch.int64)\n",
        "\n",
        "  def decode(self,ids):\n",
        "    v = self.build_vocab()\n",
        "    list_ids = ids.tolist()\n",
        "    tokens = [v.vocab.get_itos()[id] for id in list_ids]\n",
        "    return ' '.join(tokens)\n",
        "\n",
        "  def encode_batch(self,batch_size,max_seq_len):\n",
        "    batch_data = torch.zeros(size=(batch_size,max_seq_len+2)) # +2 for special tokens\n",
        "    for i,sentence in enumerate(self.text.strip('').split('\\n')):\n",
        "      token_ids = self.encode(sentence)\n",
        "      batch_data[i,0:len(token_ids)] = token_ids\n",
        "    return batch_data.type(dtype=torch.int64)\n",
        "\n"
      ],
      "metadata": {
        "id": "b2QLspwihNSB"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* It is always go to check the implementation"
      ],
      "metadata": {
        "id": "tIiCA9uppzMX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "batch_size = 8"
      ],
      "metadata": {
        "id": "BPWNsuaolNcz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# you can play with this\n",
        "src_tokenizer = Tokenizer(src_text)\n",
        "print(src_tokenizer.encode('The most famous ruler of ancient India was Emperor Ashoka.'))\n",
        "print(src_tokenizer.encode_batch(batch_size,src_max_seq_len))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sV0wYxrFkSzr",
        "outputId": "114ae1d6-d34a-4f78-b591-6e5250ae0c67"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([ 1, 27, 49, 39, 15,  8, 28, 24,  5, 22, 20,  2])\n",
            "tensor([[ 1, 27, 49, 39, 15,  8, 28, 24,  5, 22, 20,  2,  0,  0,  0,  0,  0,  0],\n",
            "        [ 1, 25,  5, 36, 14, 53, 58, 11, 16,  6, 35, 50,  8, 21,  2,  0,  0,  0],\n",
            "        [ 1, 19, 40, 17, 18,  9, 56, 47, 52, 43, 32,  9,  4, 26, 61,  2,  0,  0],\n",
            "        [ 1,  7, 37, 11, 12, 59, 33, 14, 46,  6, 16,  4, 48,  8, 51, 12, 34,  2],\n",
            "        [ 1, 23, 57, 13,  4, 31,  8, 54, 42,  5, 38,  2,  0,  0,  0,  0,  0,  0],\n",
            "        [ 1,  7,  5,  4, 10, 15,  6, 41, 17, 18,  9, 60,  2,  0,  0,  0,  0,  0],\n",
            "        [ 1,  7,  5,  4, 10,  6, 30, 44, 13, 29,  2,  0,  0,  0,  0,  0,  0,  0],\n",
            "        [ 1,  7,  5,  4, 10,  6, 45, 55,  2,  0,  0,  0,  0,  0,  0,  0,  0,  0]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# you can play with this\n",
        "tar_tokenizer = Tokenizer(tar_text)\n",
        "print(tar_tokenizer.encode('பண்டைய இந்திய அரசர்களில் பேரும் புகழும் பெற்ற அரசர் அசோகர் ஆவார்.'))\n",
        "print(tar_tokenizer.encode_batch(batch_size,tar_max_seq_len))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GZBhYD7MxzpR",
        "outputId": "75011385-f914-418c-9545-49431b42e952"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([ 1, 44, 22, 16, 53, 51, 52,  4, 11,  6,  2])\n",
            "tensor([[ 1, 44, 22, 16, 53, 51, 52,  4, 11,  6,  2,  0,  0],\n",
            "        [ 1, 25, 20, 39,  8, 59, 19, 49, 43, 47,  2,  0,  0],\n",
            "        [ 1, 30, 55,  7, 48, 26, 58, 29, 65, 57, 41, 31,  2],\n",
            "        [ 1, 13, 50,  8, 32, 38, 14, 18, 46, 37, 66, 17,  2],\n",
            "        [ 1, 54,  5, 21, 34, 64, 61, 68,  2,  0,  0,  0,  0],\n",
            "        [ 1, 69,  7, 56, 40, 63,  4, 12,  2,  0,  0,  0,  0],\n",
            "        [ 1, 28, 62, 67, 36, 60, 15, 35, 10,  6,  2,  0,  0],\n",
            "        [ 1,  9, 23,  5, 27, 33, 42, 45, 24,  2,  0,  0,  0]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "* Let's load the token ids of the words in the sentences of source and target languages"
      ],
      "metadata": {
        "id": "FNVpiIqDKp0P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "x = src_tokenizer.encode_batch(batch_size,src_max_seq_len)\n",
        "y = tar_tokenizer.encode_batch(batch_size,tar_max_seq_len)"
      ],
      "metadata": {
        "id": "wQKgFX1xLtC3"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* we have appended zeros to sentences that are shorter than max-seq-len\n",
        "* We have to ignore computing loss over those padded tokens\n",
        "* You have to take care of that in the cell below"
      ],
      "metadata": {
        "id": "DjE9mlNOxQlO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# your code goes here\n",
        "label = None"
      ],
      "metadata": {
        "id": "z-cIRfUeLLpN"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* Define the context lengths for encoder and decoder"
      ],
      "metadata": {
        "id": "k0i4jRhh5U95"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "enc_ctxt_len = src_max_seq_len+2\n",
        "dec_ctxt_len = tar_max_seq_len+2"
      ],
      "metadata": {
        "id": "Z6YT0qC-MEeG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Load configuration file"
      ],
      "metadata": {
        "id": "LxpOb7WXJuPS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "config_url = \"https://raw.githubusercontent.com/Arunprakash-A/LLM-from-scratch-PyTorch/main/config_files/enc_config.yml\"\n",
        "response = requests.get(config_url)\n",
        "config = response.content.decode(\"utf-8\")\n",
        "config = safe_load(config)\n",
        "pprint(config)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lwp2oxceUguz",
        "outputId": "14b2c9b4-0301-4e2c-c124-4d101a5c710f"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'input': {'batch_size': 10, 'embed_dim': 32, 'seq_len': 8, 'vocab_size': 10},\n",
            " 'model': {'d_ff': 128,\n",
            "           'd_model': 32,\n",
            "           'dk': 4,\n",
            "           'dq': 4,\n",
            "           'dv': 4,\n",
            "           'n_heads': 8,\n",
            "           'n_layers': 6}}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "src_vocab_size =src_tokenizer.vocab_size\n",
        "batch_size = x.shape[0]\n",
        "embed_dim = config['input']['embed_dim']"
      ],
      "metadata": {
        "id": "VdNVN3GUUqi4"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "dq = torch.tensor(config['model']['dq'])\n",
        "dk = torch.tensor(config['model']['dk'])\n",
        "dv = torch.tensor(config['model']['dv'])\n",
        "dmodel = embed_dim\n",
        "heads = torch.tensor(config['model']['n_heads'])\n",
        "d_ff = config['model']['d_ff']"
      ],
      "metadata": {
        "id": "UPKo8KPrXDIa"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "config_url = \"https://raw.githubusercontent.com/Arunprakash-A/LLM-from-scratch-PyTorch/main/config_files/dec_config.yml\"\n",
        "response = requests.get(config_url)\n",
        "config = response.content.decode(\"utf-8\")\n",
        "config = safe_load(config)\n",
        "pprint(config)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QFJh4tsEJva-",
        "outputId": "142d9aed-c1c7-4910-8d2a-bcbd4fa589fd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'input': {'batch_size': 10, 'embed_dim': 32, 'seq_len': 8, 'vocab_size': 12},\n",
            " 'model': {'d_ff': 128,\n",
            "           'd_model': 32,\n",
            "           'dk': 4,\n",
            "           'dq': 4,\n",
            "           'dv': 4,\n",
            "           'n_heads': 8,\n",
            "           'n_layers': 6}}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# do not edit this cell\n",
        "tar_vocab_size = tar_tokenizer.vocab_size"
      ],
      "metadata": {
        "id": "Ijfm3gfDPWs3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Encoder\n",
        "\n",
        " * You can copy paste the required code from the previous assignments"
      ],
      "metadata": {
        "id": "tjIYp4I177LY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class MHA(nn.Module):\n",
        "  pass\n",
        "\n",
        "class Prediction(nn.Module):\n",
        "  pass\n",
        "\n",
        "class EncoderLayer(nn.Module):\n",
        "  pass\n",
        "\n",
        "class Encoder(nn.Module):\n",
        "  pass\n"
      ],
      "metadata": {
        "id": "MVxV2ROQ8Aky"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Decoder"
      ],
      "metadata": {
        "id": "dpx5HW_bXHWX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class MHCA(nn.Module):\n",
        "  pass\n",
        "\n",
        "class MHMA(nn.Module):\n",
        "  pass\n",
        "\n",
        "class DecoderLayer(nn.Module):\n",
        "  pass\n",
        "\n",
        "class Decoder(nn.Module):\n",
        "  pass"
      ],
      "metadata": {
        "id": "jDZSDFHnX5MS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Positional Embedding\n",
        "\n",
        " * You may take the code directly from any source."
      ],
      "metadata": {
        "id": "dtqZMexg-Ov0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class PositionalEncoding(nn.Module):\n",
        "    pass"
      ],
      "metadata": {
        "id": "ehcB8MeD-Rbk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Generate target mask\n",
        "\n",
        "  * We will be passing the causal mask for the decoder layer as one of its arguments"
      ],
      "metadata": {
        "id": "Q5BDOOj_cJlJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mask = (torch.triu(torch.ones(dec_ctxt_len,dec_ctxt_len)) == 1).transpose(0,1)\n",
        "mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n",
        "print(mask)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ngAlmGaccFva",
        "outputId": "b7e81e68-3e78-420e-b049-0e0d3d655a28"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., 0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -inf, -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -inf, -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -inf, -inf],\n",
            "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -inf],\n",
            "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Transformer"
      ],
      "metadata": {
        "id": "DyXhnsw4SPeQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class Transformer(nn.Module):\n",
        "\n",
        "  def __init__(self,src_vocab_size,tar_vocab_szie,src_seq_len,tar_seq_len,dmodel,dq,dk,dv,d_ff,heads,target_mask,num_layers=1):\n",
        "    super(Transformer,self).__init__()\n",
        "    self.src_embeddings = nn.Embedding(src_vocab_size,embed_dim)\n",
        "    self.tar_embeddings = nn.Embedding(tar_vocab_size,embed_dim)\n",
        "    self.pos_embeddings = PositionalEncoding(dmodel)\n",
        "    self.encoder = Encoder(src_vocab_size,dmodel,dq,dk,dv,d_ff,heads,num_layers)\n",
        "    self.decoder = Decoder(tar_vocab_size,dmodel,dq,dk,dv,d_ff,heads,target_mask,num_layers)\n",
        "\n",
        "  def forward(self,src_token_ids,tar_token_ids):\n",
        "    out = self.encoder(self.pos_embeddings(self.src_embeddings(src_token_ids)))\n",
        "    out = self.decoder(out,self.tar_embeddings(tar_token_ids))\n",
        "    return out"
      ],
      "metadata": {
        "id": "yCug8q1GSIjU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model = Transformer(src_vocab_size,tar_vocab_size,enc_ctxt_len,dec_ctxt_len,dmodel,dq,dk,dv,d_ff,heads,mask)"
      ],
      "metadata": {
        "id": "awmAIEf7kaMA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.SGD(model.parameters(), lr=0.01)"
      ],
      "metadata": {
        "id": "A18BP8POk772"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def train(src_token_ids,tar_token_ids,labels,epochs=1000):\n",
        "  loss_trace = []\n",
        "  for epoch in range(epochs):\n",
        "    out = model(src_token_ids,tar_token_ids)\n",
        "    loss = criterion() # edit this\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    optimizer.zero_grad()"
      ],
      "metadata": {
        "id": "yWbFM5HnlXSC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train(x,y,label,1000)"
      ],
      "metadata": {
        "id": "HyslfJ1KZ51L"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Run the model AutoRegressively"
      ],
      "metadata": {
        "id": "rkRq8gYolAh-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "@torch.inference_mode()\n",
        "def inference(test_input):\n",
        "  '''\n",
        "  Run the model in autoregressive fashion and store the output at each time step in a list\n",
        "  '''\n",
        "\n",
        "  return None"
      ],
      "metadata": {
        "id": "psLM9R44lEMf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* Modify the code below to suit your implementation\n",
        "* Display the original and translated sentence (with all the spcial tokens)\n",
        "* Note that, the second half of the second sentence is poorly translated\n",
        "*  Same goes for 3rd and 4th sentence\n",
        "* All other sentences are properly translated"
      ],
      "metadata": {
        "id": "i3GljM0lkgzP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for token_ids in x:\n",
        "  print(src_tokenizer.decode(token_ids))\n",
        "  print(tar_tokenizer.decode(inference(token_ids)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hb04_GlqYaMj",
        "outputId": "741cf997-ac80-40bb-d09b-fba5ce973b94"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<start> The most famous ruler of ancient India was Emperor Ashoka. <end> <pad> <pad> <pad> <pad> <pad> <pad>\n",
            "<start> பண்டைய இந்திய அரசர்களில் பேரும் புகழும் பெற்ற அரசர் அசோகர் ஆவார். <pad> <pad> <pad>\n",
            "<start> It was during his period that Buddhism spread to different parts of Asia. <end> <pad> <pad> <pad>\n",
            "<start> இவரது ஆட்சியில் தான் புத்த மதம் ஆசியாவின் புத்த ஆற்றிய அசோகரே பின் <unk> பேரும்\n",
            "<start> Ashoka gave up war after seeing many people grieving death after the Kalinga war. <end> <pad> <pad>\n",
            "<start> கலிங்கப் தனியே மடிவதைக் பின் வாழ்வையே தழுவி, பெற்ற அரசர்களில் அரசர்களில் அரசர்களில் அரசர்களில் அரசர்களில்\n",
            "<start> He embraced Buddhism and then devoted his life to spread the message of peace and dharma. <end>\n",
            "<start> அதற்குப் பிறகு புத்த சமயத்தைத் தழுவி, அமைதியையும் அறத்தையும் பரப்புவதற்காகத் தன் வாழ்வையே பெற்ற அசோகர்\n",
            "<start> His service for the cause of public good was exemplary. <end> <pad> <pad> <pad> <pad> <pad> <pad>\n",
            "<start> பொதுமக்களுக்கு அவர் ஆற்றிய சேவை முன் மாதிரியாக விளங்கியது. <pad> <pad> <pad> <pad> <pad>\n",
            "<start> He was the first ruler to give up war after victory. <end> <pad> <pad> <pad> <pad> <pad>\n",
            "<start> வெற்றிக்குப் பின் போரைத் துறந்த முதல் அரசர் அசோகர்தான். <pad> <pad> <pad> <pad> <pad>\n",
            "<start> He was the first to build hospitals for animals. <end> <pad> <pad> <pad> <pad> <pad> <pad> <pad>\n",
            "<start> உலகிலேயே முதன்முதலாக விலங்குகளுக்கும் தனியே மருத்துவமனை அமைத்துத் தந்தவரும் அசோகரே ஆவார். <pad> <pad> <pad>\n",
            "<start> He was the first to lay roads. <end> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad>\n",
            "<start>  இன்றும் அவர் உருவாக்கிய சாலைகளை நாம் பயன்படுத்திக்கொண்டு இருக்கிறோம். <pad> <pad> <pad> <pad>\n"
          ]
        }
      ]
    }
  ]
}