distributional_models.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNijr+/Lfii7Uv3GWK838/J",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/pmadhyastha/INM434/blob/main/distributional_models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Distributional models, vector space representations and word embeddings"
      ],
      "metadata": {
        "id": "Gsl2dbU_geYH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "__author__ = \"Pranava Madhyastha\" \n",
        "__version__ = \"INM434/IN3045 City, University of London, Spring 2023\""
      ],
      "metadata": {
        "id": "dBegv3B2gkae"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Setup: english wiki corpus! \n",
        "\n",
        "We will begin by downloading a large wikipedia corpus and perform some cleaning and text normalisation. "
      ],
      "metadata": {
        "id": "pwULzRTjg2ka"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import urllib.request\n",
        "import re\n",
        "\n",
        "# Download the corpus\n",
        "url = \"http://mattmahoney.net/dc/enwik8.zip\"\n",
        "urllib.request.urlretrieve(url, \"enwik8.zip\")\n",
        "\n",
        "# Extract the corpus and clean it\n",
        "import zipfile\n",
        "\n",
        "with zipfile.ZipFile('enwik8.zip', 'r') as zip_ref:\n",
        "    zip_ref.extractall()\n",
        "\n",
        "with open('enwik8', 'r', encoding='utf-8') as f_in, open('enwik8_clean', 'w', encoding='utf-8') as f_out:\n",
        "    for line in f_in:\n",
        "        # Strip off HTML tags\n",
        "        line = re.sub(r'<.*?>', '', line)\n",
        "        # Normalize the text\n",
        "        line = line.lower()\n",
        "        # Write the cleaned line to the output file\n",
        "        f_out.write(line)\n",
        "\n",
        "# Tokenize the cleaned corpus\n",
        "with open('enwik8_clean', 'r', encoding='utf-8') as f:\n",
        "    corpus = f.read()"
      ],
      "metadata": {
        "id": "rsX-uX1jgqZV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The above code downloads an old version of English wikipedia corpus. \n",
        "\n",
        "### TODO: \n",
        "\n",
        "1.   How are we normalising text? Can you print out a part of the corpus and see what is being done? \n",
        "\n"
      ],
      "metadata": {
        "id": "i-pZifO0hTkf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Vector space represenation of words using co-occurrence information. \n",
        "\n",
        "We will now write code for getting out first vector space model by building very simple word co-occurrence dictionary (matrix and dictionaries and interchangeably used in this lab session). \n",
        "\n",
        "\n",
        "The program reads in a corpus of text stored in the file called 'enwik8_clean' (which we obtained using the process above). The program then tokenizes it using NLTK library. It then creates a vocabulary of *unique words* in the corpus and counts the number of occurrences of each word using a defaultdict object. \n",
        "\n",
        "Next, it counts the co-occurrences of words within a fixed window size of three words and stores the results in a nested defaultdict. \n",
        "\n",
        "The program then creates a vector space representation for each word by iterating through the vocabulary and creating a vector of co-occurrence counts with all other words in the vocabulary. \n",
        "\n",
        "The resulting vectors are stored in a dictionary object named 'vectors'. The code also filters out stop words from the corpus before counting co-occurrences."
      ],
      "metadata": {
        "id": "1JGPulrAiPsE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "from nltk.corpus import stopwords\n",
        "from collections import defaultdict\n",
        "import os \n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "nltk.download('punkt')\n",
        "\n",
        "\n",
        "# Constants\n",
        "WINDOW_SIZE = 3\n",
        "STOPWORDS = set(stopwords.words('english'))\n",
        "\n",
        "# Read in corpus\n",
        "with open('enwik8_clean', 'r', encoding='utf-8') as f:\n",
        "    filesize = os.path.getsize('enwik8_clean')\n",
        "    lines_limit = 1000\n",
        "    lines = []\n",
        "    line = 0\n",
        "    while line < lines_limit:\n",
        "        l = f.readline()\n",
        "        lines.append(l)\n",
        "        line += 1\n",
        "    corpus = ''.join(lines)\n",
        "\n",
        "# Create vocabulary\n",
        "corpus = word_tokenize(corpus)\n",
        "vocab = set(corpus)\n",
        "\n",
        "print(len(vocab))\n",
        "\n",
        "# Count occurrences of each word\n",
        "word_counts = defaultdict(int)\n",
        "for word in corpus:\n",
        "    word_counts[word] += 1\n",
        "\n",
        "# Count co-occurrences of words within a window\n",
        "context_counts = defaultdict(lambda: defaultdict(int))\n",
        "for i in range(len(corpus)):\n",
        "    if corpus[i] in STOPWORDS:\n",
        "        continue\n",
        "    for j in range(i - WINDOW_SIZE, i + WINDOW_SIZE + 1):\n",
        "        if j < 0 or j >= len(corpus) or i == j or corpus[j] in STOPWORDS:\n",
        "            continue\n",
        "        context_counts[corpus[i]][corpus[j]] += 1\n",
        "\n",
        "# Create vector space representation for each word\n",
        "vectors = {}\n",
        "for word in vocab:\n",
        "    vector = [context_counts[word][w] for w in vocab]\n",
        "    vectors[word] = vector"
      ],
      "metadata": {
        "id": "y4nS9rFci7xB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### TODO: \n",
        "\n",
        "1. How many lines of data are we considering the code? \n",
        "2. \"vectors\" is a dictionary, can you print the vocabulary? \n",
        "3. What is the dimensionality of the word vector? \n",
        "4. What are we storing in each one of the arrays corresponding to each word? \n",
        "5. Consider the word \"peaceful\", what is the colsum of this word? What does colsum signify here? \n",
        "6. Feel free to increase the number of lines, there is a point at which it is going to occupy a very large amount of memory and \"colab\" will stop functioning. What is causing this? \n",
        "7. What is a way to mitigate the memory explosion?  "
      ],
      "metadata": {
        "id": "WpsTCLSPjda-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Word similarity using metrics \n",
        "\n",
        "We will now play with vector similarity. We want to essentially understand how we can obtain the most similar and the least similar words. We will use distance metrics to operationalise this. "
      ],
      "metadata": {
        "id": "seOLOZXblWY5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "def top_n_words(vectors, n, distance_function):\n",
        "    distances = {}\n",
        "    for word, vector in vectors.items():\n",
        "        distance = distance_function(vector)\n",
        "        distances[word] = distance\n",
        "    sorted_distances = sorted(distances.items(), key=lambda x: x[1], reverse=True)\n",
        "    return sorted_distances[:n]\n",
        "\n",
        "# Example dictionary of vectors\n",
        "vectors = {\n",
        "    'apple': np.array([1, 2, 3]),\n",
        "    'banana': np.array([4, 5, 6]),\n",
        "    'orange': np.array([7, 8, 9]),\n",
        "    'grape': np.array([10, 11, 12])\n",
        "}\n",
        "\n",
        "# Euclidean distance function\n",
        "def euclidean_distance(vector):\n",
        "    return np.linalg.norm(vector)\n",
        "\n",
        "# Manhattan distance function\n",
        "def manhattan_distance(vector):\n",
        "    return np.sum(np.abs(vector))\n",
        "\n",
        "# Top-n words using Euclidean distance\n",
        "top_n_euclidean = top_n_words(vectors, 2, euclidean_distance)\n",
        "print('Top 2 words using Euclidean distance:', top_n_euclidean)\n",
        "\n",
        "# Top-n words using Manhattan distance\n",
        "top_n_manhattan = top_n_words(vectors, 2, manhattan_distance)\n",
        "print('Top 2 words using Manhattan distance:', top_n_manhattan)\n"
      ],
      "metadata": {
        "id": "-J0NuZvUmCBb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## TODO: \n",
        "\n",
        "1. Can you extend the code for cosine similarity and cosine distance? \n",
        "2. Have a look at https://docs.scipy.org/doc/scipy/reference/spatial.distance.html for additional distance metrics. Try different distance functions and see how top-2 words change! \n",
        "3. We are playing with toy data for this code, can you instead use the vectors from the example above and obtain the top-2 words for a few words in the vocab? (say start with the word \"peaceful\"). \n",
        "4. Can you try playing with a few of reweighting techniques? See how distances change? "
      ],
      "metadata": {
        "id": "zzKwhjc5mXcb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Latent semantic analysis (truncated SVD) \n",
        "\n",
        "We will now see the basic workings of latent semantic analysis. For this, we will use a readily available function from scikit-learn: CountVectorizer. We have used this in the previous sets of labs, we should be fairly familiar with the basic functionality of this method. "
      ],
      "metadata": {
        "id": "KUJmZzqInxMB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.decomposition import TruncatedSVD\n",
        "\n",
        "# Define some example text documents\n",
        "docs = [\n",
        "    \"The quick brown fox jumps over the lazy dog\",\n",
        "    \"The quick brown fox jumps over the quick dog\",\n",
        "    \"Dog is man's best friend\",\n",
        "    \"Cat is not a man's best friend\",\n",
        "    \"The brown fox is quick\"\n",
        "]\n",
        "\n",
        "# Convert the documents to a co-occurrence matrix\n",
        "vectorizer = CountVectorizer()\n",
        "cooc = vectorizer.fit_transform(docs)\n",
        "\n",
        "# print the vocabulary\n",
        "print(vectorizer.get_feature_names())\n",
        "\n",
        "# Perform singular value decomposition (SVD)\n",
        "svd = TruncatedSVD(n_components=5)\n",
        "svd.fit(cooc)\n",
        "\n",
        "\n",
        "# Extract left singular vectors and singular values as word embeddings\n",
        "word_embeddings = svd.components_.T * svd.singular_values_\n",
        "\n",
        "print(word_embeddings)\n"
      ],
      "metadata": {
        "id": "K8bPXIbvmEQH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Todo\n",
        "1. Can you associate the words to their corresponding vectors? \n",
        "2. What was the initial dimensionality? \n",
        "3. Can you change the CountVectorizer such that it now considers \"document information\" (here a document is a sentence) -- try looking at TfidfVectorizer. \n",
        "4. Can you print the top-2 words for each word in the vocabulary?"
      ],
      "metadata": {
        "id": "sHFbolM8pmYV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Word2Vev\n",
        "\n",
        "In the following code, we will implement a simplified version of word2vec model that trains a continuous bag-of-words model using PyTorch to learn word embeddings from a small corpus. The CBOW model predicts a target word given its surrounding context words, and the word embeddings are the weights of the model's embedding layer.\n",
        "\n",
        "\n",
        "The train_cbow function trains the CBOW model using the specified hyperparameters. It initializes the model, criterion, and optimizer, and then trains the model for the specified number of epochs. For each sentence in the corpus, it iterates over the positions in the sentence and creates a context tensor and target tensor for each position. It calculates the output of the model for the context tensor, computes the loss using the cross-entropy loss function, and backpropagates the loss to update the model parameters. At the end of each epoch, it prints the total loss for that epoch. \n",
        "\n",
        "This code looks very similar to the linear and non-linear models that we have explored before! "
      ],
      "metadata": {
        "id": "BHj6sEINqeIB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from collections import Counter\n",
        "import numpy as np\n",
        "\n",
        "# Toy data for training\n",
        "sentences = [[\"apple\", \"orange\", \"banana\", \"pear\"], [\"apple\", \"pear\"], [\"banana\", \"orange\"], [\"pear\", \"banana\", \"orange\"]]\n",
        "\n",
        "# Building vocabulary\n",
        "word_counts = Counter([word for sentence in sentences for word in sentence])\n",
        "vocabulary = list(word_counts.keys())\n",
        "word_to_index = {word: i for i, word in enumerate(vocabulary)}\n",
        "\n",
        "# Hyperparameters\n",
        "embedding_size = 100\n",
        "window_size = 2\n",
        "learning_rate = 0.001\n",
        "num_epochs = 100\n",
        "\n",
        "class CBOW(nn.Module):\n",
        "    def __init__(self, vocabulary_size, embedding_size):\n",
        "        super(CBOW, self).__init__()\n",
        "        self.embedding = nn.Embedding(vocabulary_size, embedding_size)\n",
        "        self.linear = nn.Linear(embedding_size, vocabulary_size)\n",
        "\n",
        "    def forward(self, context):\n",
        "        embedded_context = self.embedding(context)\n",
        "        sum_embedded_context = torch.sum(embedded_context, dim=1)\n",
        "        output = self.linear(sum_embedded_context)\n",
        "        return output\n",
        "\n",
        "def train_cbow(sentences, word_to_index, vocabulary, embedding_size, window_size, learning_rate, num_epochs):\n",
        "    model = CBOW(len(vocabulary), embedding_size)\n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "    optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n",
        "\n",
        "    for epoch in range(num_epochs):\n",
        "        total_loss = 0\n",
        "        for sentence in sentences:\n",
        "            for i in range(window_size, len(sentence) - window_size):\n",
        "                context = [word_to_index[word] for word in sentence[i-window_size:i] + sentence[i+1:i+window_size+1]]\n",
        "                target = torch.tensor(word_to_index[sentence[i]], dtype=torch.long)\n",
        "                optimizer.zero_grad()\n",
        "                output = model(torch.tensor(context, dtype=torch.long))\n",
        "                loss = criterion(output.unsqueeze(0), target)\n",
        "                total_loss += loss.item()\n",
        "                loss.backward()\n",
        "                optimizer.step()\n",
        "        print(\"Epoch %s, loss=%.4f\" % (epoch+1, total_loss))\n",
        "\n",
        "    return model\n",
        "\n",
        "# Train the CBOW model\n",
        "model = train_cbow(sentences, word_to_index, vocabulary, embedding_size, window_size, learning_rate, num_epochs)\n",
        "\n",
        "# Get the learned embeddings\n",
        "word_embeddings = model.embedding.weight.detach().numpy()\n",
        "\n",
        "# Test the embeddings\n",
        "print(\"Embedding for 'apple': \", word_embeddings[word_to_index['apple']])\n",
        "print(\"Embedding for 'orange': \", word_embeddings[word_to_index['orange']])\n",
        "print(\"Embedding for 'banana': \", word_embeddings[word_to_index['banana']])\n",
        "print(\"Embedding for 'pear': \", word_embeddings[word_to_index['pear']])\n"
      ],
      "metadata": {
        "id": "CGt2AxcGpMgb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Todo\n",
        "\n",
        "1. Is this a linear or a non-linear model? \n",
        "2. What does the window-size signify here? "
      ],
      "metadata": {
        "id": "UbLVmLvNsTS9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Typical use case: Pre-trained word representations. \n",
        "\n",
        "We will now use pre-trained word representations using GloVe (an alternative way of learning word representations similar to word2vec). GloVe uses a similar formalisation of the problem similar to skipgram version of word2vec but additionally adds information relating to \"ratios\" of co-occurrences with other words. For the purposes of this module, it is sufficient to know that GloVe is very similar to word2vec and shows similar behaviour. (if you are further interested, for more details, please refer: https://nlp.stanford.edu/projects/glove/). \n",
        "\n",
        "In the code below, we are simply going to load the pretrained word vectors (vectors that are previously trained over a large corpus). To do this, we are going to download GloVe based word vectors. Then we are going to load it up into a dictionary and obtain 10 most similar words for a given word. \n",
        "\n"
      ],
      "metadata": {
        "id": "EErFNaG6wqod"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import urllib.request\n",
        "import zipfile\n",
        "from pathlib import Path\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# Download and extract the GloVe embeddings\n",
        "url = 'http://nlp.stanford.edu/data/glove.6B.zip'\n",
        "filename = 'glove.6B.zip'\n",
        "infile = Path(filename)\n",
        "if not infile:\n",
        "    urllib.request.urlretrieve(url, filename)\n",
        "    with zipfile.ZipFile(filename, 'r') as zip_ref:\n",
        "        zip_ref.extractall('.')\n",
        "\n",
        "# Load the embeddings into a dictionary\n",
        "embeddings = {}\n",
        "with open('glove.6B.50d.txt', 'r', encoding='utf-8') as f:\n",
        "    for line in f:\n",
        "        values = line.split()\n",
        "        word = values[0]\n",
        "        vector = np.asarray(values[1:], dtype='float32')\n",
        "        embeddings[word] = vector\n",
        "\n",
        "# Function to obtain most similar words to a given word\n",
        "def get_most_similar_words(word, embeddings, n=10):\n",
        "    word_vector = embeddings.get(word)\n",
        "    if word_vector is None:\n",
        "        return []\n",
        "    similarities = {}\n",
        "    for w, v in embeddings.items():\n",
        "        sim = np.dot(word_vector, v) / (np.linalg.norm(word_vector) * np.linalg.norm(v))\n",
        "        similarities[w] = sim\n",
        "    sorted_similarities = sorted(similarities.items(), key=lambda x: x[1], reverse=True)\n",
        "    return [w for w, s in sorted_similarities[:n]]\n",
        "\n",
        "print(\"most similar words for  'apple': \", get_most_similar_words(\"apple\", embeddings))\n"
      ],
      "metadata": {
        "id": "F4_frRoAq9rC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### TODO: \n",
        "\n",
        "1. Can you try for a list of other words? \n",
        "2. **Advanced** A sentence is made of words, we can consider \"mean pooling\" word representations to form the sentence representation. Can you use this clue to construct a sentiment classifier for Lab 3 and 4 with mean pooled word representations as features?  is this better? What is different now? "
      ],
      "metadata": {
        "id": "Jun8VxjhyqvR"
      }
    }
  ]
}