Exercise_Sheet_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing\n",
    "\n",
    "## Exercise Sheet 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#imports for all exercises\n",
    "## import nltk\n",
    "# nltk.download('book')\n",
    "## from nltk.book import *\n",
    "## from nltk import FreqDist\n",
    "## import random\n",
    "\n",
    "from nltk.corpus import brown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "Rewrite the following loop as a list comprehension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('The', 3),\n",
       " ('dog', 3),\n",
       " ('gave', 4),\n",
       " ('John', 4),\n",
       " ('the', 3),\n",
       " ('newspaper', 9)]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = ['The', 'dog', 'gave', 'John', 'the', 'newspaper'] \n",
    "result = [] \n",
    "for word in sent: \n",
    "    word_len = (word, len(word)) \n",
    "    result.append(word_len) \n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the task a list of words 'sent' is given and then iteratie on its element by for loop and create tuple 'word_len' that contains each word in the first element and the length of it in the second element.\n",
    "than its appended to the 'result' list and then its printed out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# List Comprehension\n",
    "\"List comprehension offers a shorter syntax when you want to create a new list based on the values of an existing list.\"\n",
    "https://www.w3schools.com/python/python_lists_comprehension.asp#:~:text=List%20comprehension%20offers%20a%20shorter%20syntax%20when%20you%20want%20to%20create%20a%20new%20list%20based%20on%20the%20values%20of%20an%20existing%20list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('The', 3),\n",
       " ('dog', 3),\n",
       " ('gave', 4),\n",
       " ('John', 4),\n",
       " ('the', 3),\n",
       " ('newspaper', 9)]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = ['The', 'dog', 'gave', 'John', 'the', 'newspaper']\n",
    "result = [(word, len(word)) for word in sent]\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the above list comprehension iterates through the words within 'sent' list and in the same time creates the tuple (word, len(word)) which will be stored in the 'result' list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "\n",
    "Pig Latin is a simple transformation of English text. Each word of the text is converted as follows: move any consonant (or consonant cluster) that appears at the start of the word to the end, then append \"ay\", e.g. \"string\" $\\rightarrow$ \"ingstray\". If a word starts with a vowel, just add \"way\" to the end, e.g. \"idle\" $\\rightarrow$ \"idleway\". \n",
    "\n",
    "Write a function to convert a word to Pig Latin. Test it with the words \"pig\", \"cheers\", and \"omelet\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pig → igpay\n",
      "cheers → eerschay\n",
      "omelet → omeletway\n"
     ]
    }
   ],
   "source": [
    "def is_vowel(char):\n",
    "    \"\"\"Check if a character is a vowel.\"\"\"\n",
    "    return char.lower() in 'aeiou'\n",
    "\n",
    "def pig_latin(word):\n",
    "    \"\"\"Convert a word to Pig Latin.\"\"\"\n",
    "    if is_vowel(word[0]):\n",
    "        return word + \"way\"\n",
    "    \n",
    "    # Find index of the first vowel\n",
    "    index = next((i for i, char in enumerate(word) if is_vowel(char)), len(word))\n",
    "    \n",
    "    return word[index:] + word[:index] + \"ay\"\n",
    "\n",
    "# Test the function\n",
    "words = [\"pig\", \"cheers\", \"omelet\"]\n",
    "for word in words:\n",
    "    print(f\"{word} → {pig_latin(word)}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pig → igpay\n",
      "cheers → eerschay\n",
      "omelet → omeletway\n"
     ]
    }
   ],
   "source": [
    "# Check if a character is a vowel.\n",
    "def is_vowel(char):\n",
    "    return char.lower() in 'aeiou'\n",
    "\n",
    "# Convert a word to Pig Latin\n",
    "def pig_latin(word):\n",
    "    # we check if the first charachter is vowel then we can return the ward + \"way\"\n",
    "    if is_vowel(word[0]):\n",
    "        return word + \"way\"\n",
    "    \n",
    "    # oterwiese we need to check each charachter until we find vowel\n",
    "    index = 0\n",
    "    for char in word:\n",
    "        # we check if its vowel\n",
    "        if is_vowel(char):\n",
    "            break\n",
    "        index += 1\n",
    "    # we slicing the word into to parts, \n",
    "        # part starts from the first vowel to the end p1\n",
    "        # part starts from first string to the first vowel p2\n",
    "        # we add p1 + p1 + \"ay\" and return it\n",
    "    return word[index:] + word[:index] + \"ay\"\n",
    "\n",
    "# Test the function\n",
    "words = [\"pig\", \"cheers\", \"omelet\"]\n",
    "for word in words:\n",
    "    print(f\"{word} → {pig_latin(word)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Exercise 3\n",
    "\n",
    "Python's `random` module includes a function `choice()` which randomly chooses an item from a sequence, e.g. `choice('aehh ')` will produce one of four possible characters, with the letter \"h\" being twice as frequent as the others. Write a generator expression that produces a sequence of 500 randomly chosen letters drawn from the string \"aehh \", and put this expression inside a call to the `''.join()` function, to concatenate them into one long string. You should get a result that looks like uncontrolled sneezing or maniacal laughter: \"he  haha ee  heheeh eha\". Use `split()` and `join()` again to normalize the whitespace in this string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a h hahh hahheehahhaaehehhhhhhhh heahee hhh ehahe eehh a aa hah ehhaehhahaehhhhha eaa hhhhhhahehhhehhhah h hheae ahahhhaeea ehheeahehehe a ahheahhea h hha hehh aaeh h ha hehh h hhhhhhh ehh ahahae eaheeh hh aaahaehh ha hhhh hahhheh heehhahehhhhaeehh aeh aaeah ahahha h aaae eh haahhhehhehhheaa eahhhh ehehhee aahhhhhhhhh a a h ehhhahhea ah hh hheh a hhhah hhe ehahh heaeahhhhehhehehahhheahhhehhe hehehah hheah hhhhhehhhhhhhhehhhheah hahhaheeah hea hhhh hheheeheehhhe h e he aheehh\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "# Generate a sequence of 500 randomly chosen letters from \"aehh \"\n",
    "random_sequence = ''.join(random.choice('aehh ') for _ in range(500))\n",
    "\n",
    "# Normalize the whitespace\n",
    "    #.split() handls the spaces, where it treat them as once space separator and then returns as list of words withoutspaces\n",
    "    #' '.join() joins the words back and ensures that there is only one space between them. \n",
    "normalized_sequence = ' '.join(random_sequence.split())\n",
    "\n",
    "print(normalized_sequence)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4\n",
    "\n",
    "Readability measures are used to score the reading difficulty of a text, for the purposes of selecting texts of appropriate difficulty for language learners. Let us define $\\mu_w$ to be the average number of letters per word, and $\\mu_s$ to be the average number of words per sentence, in a given text. The Automated Readability Index (ARI) of the text is defined to be: $4.71 \\mu_w + 0.5 \\mu_s - 21.43$. Compute the ARI score for the \"lore\" and \"learned\" genre of the Brown Corpus. Make use of the fact that `nltk.corpus.brown.words()` produces a sequence of words, while `nltk.corpus.brown.sents()` produces a sequence of sentences.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| **ARI Score**       | **Grade Level**    | **Reading Level**     | **Ages**         |\n",
    "|---------------------|--------------------|-----------------------|------------------|\n",
    "| 1 -- 5              | Kindergarten       | Extremely Easy        | 5-6 yrs. old     |\n",
    "| 1 -- 5              | First Grade        | Extremely Easy        | 6-7 yrs. old     |\n",
    "| 6 -- 7              | Second Grade       | Very Easy             | 7-8 yrs. old     |\n",
    "| 8 -- 9              | Third Grade        | Very Easy             | 8-9 yrs. old     |\n",
    "| 10 -- 11            | Fourth Grade       | Easy                  | 9-10 yrs. old    |\n",
    "| 12 -- 13            | Fifth Grade        | Fairly Easy           | 10-11 yrs. old   |\n",
    "| 14 -- 15            | Sixth Grade        | Fairly Easy           | 11-12 yrs. old   |\n",
    "| 16 -- 17            | Seventh Grade      | Average               | 12-13 yrs. old   |\n",
    "| 18 -- 19            | Eighth Grade       | Average               | 13-14 yrs. old   |\n",
    "| 20 -- 21            | Ninth Grade        | Slightly Difficult    | 14-15 yrs. old   |\n",
    "| 22 -- 23            | Tenth Grade        | Somewhat Difficult    | 15-16 yrs. old   |\n",
    "| 24 -- 25            | Eleventh grade     | Fairly Difficult      | 16-17 yrs. old   |\n",
    "| 26 -- 27            | Twelfth grade      | Difficult             | 17-18 yrs. old   |\n",
    "| 28 -- and above     | College            | Very Difficult        | 18-22 yrs. old   |\n",
    "\n",
    "Reference:[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def ari_score(genre):\n",
    "\n",
    "# First, we calculate the average number of letters per word\n",
    "    # get all the words in the given genre\n",
    "    words = brown.words(categories=genre)\n",
    "    \n",
    "    # divide the number of all letters for all words by the number of the words\n",
    "    mu_w = sum(len(word) for word in words) / len(words)\n",
    "    \n",
    "# Second, we calculate the average number of words per sentence\n",
    "    # get all the sentences in the given genre\n",
    "    sentences = brown.sents(categories=genre)\n",
    "    \n",
    "    # divide the number of all words for all sentences by the number of the sentences\n",
    "    mu_s = sum(len(sentence) for sentence in sentences) / len(sentences)\n",
    "    \n",
    "# finally, we calculate the Automated Readability Index by the given formula\n",
    "\n",
    "    return 4.71 * mu_w + 0.5 * mu_s -21.43\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the Automated Readability Index for 'lore' genre: 10.25\n",
      "the Automated Readability Index for 'learned' genre: 11.93\n"
     ]
    }
   ],
   "source": [
    "# Compute ARI for the \"lore\" and \"learned\" genres\n",
    "ari_lore = ari_score(\"lore\")\n",
    "ari_learned = ari_score(\"learned\")\n",
    "\n",
    "print(f\"the Automated Readability Index for 'lore' genre: {ari_lore:.2f}\")\n",
    "print(f\"the Automated Readability Index for 'learned' genre: {ari_learned:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5\n",
    "\n",
    "Define a variable `silly` to contain the string: 'newly formed bland ideas are inexpressible in an infuriating way'. Now write code to perform the following tasks:\n",
    "\n",
    "a) Split `silly` into a list of strings, one per word, using Python's `split()` operation, and save this to a variable called `bland`.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['newly',\n",
       " 'formed',\n",
       " 'bland',\n",
       " 'ideas',\n",
       " 'are',\n",
       " 'inexpressible',\n",
       " 'in',\n",
       " 'an',\n",
       " 'infuriating',\n",
       " 'way']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "silly = 'newly formed bland ideas are inexpressible in an infuriating way'\n",
    "bland = silly.split()\n",
    "bland"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "b) Extract the second letter of each word in `silly` and join them into a string, to get 'eoldrnnnna'.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'eoldrnnnna'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "second_letters = ''.join([word[1] for word in bland if len(word) > 1])\n",
    "second_letters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c) Combine the words in `bland` back into a single string, using `join()`. Make sure the words in the resulting string are separated with whitespace.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'newly formed bland ideas are inexpressible in an infuriating way'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined_bland = ' '.join(bland)\n",
    "combined_bland"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d) Print the words of `silly` in alphabetical order, one per line. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "an\n",
      "are\n",
      "bland\n",
      "formed\n",
      "ideas\n",
      "in\n",
      "inexpressible\n",
      "infuriating\n",
      "newly\n",
      "way\n"
     ]
    }
   ],
   "source": [
    "for word in sorted(bland):\n",
    "    print(word)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6\n",
    "\n",
    "Rewrite the following nested loop as a nested list comprehension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aiuio', 'eaiou', 'eouio', 'euoia', 'oauaio', 'uiieioa']"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = ['attribution', 'confabulation', 'tenacious', 'elocution',\n",
    "         'sequoia', 'tenacious', 'unidirectional']\n",
    "vsequences = set()\n",
    "for word in words:\n",
    "    vowels = []\n",
    "    for char in word:\n",
    "        if char in 'aeiou':\n",
    "            vowels.append(char)\n",
    "    vsequences.add(''.join(vowels))\n",
    "sorted(vsequences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['aiuio', 'eaiou', 'eouio', 'euoia', 'oauaio', 'uiieioa']\n"
     ]
    }
   ],
   "source": [
    "words = ['attribution', 'confabulation', 'tenacious', 'elocution',\n",
    "         'sequoia', 'tenacious', 'unidirectional']\n",
    "\n",
    "vsequences = {''.join([char for char in word if char in 'aeiou']) for word in words}\n",
    "\n",
    "output = sorted(vsequences)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "References:\n",
    "[1] https://readabilityformulas.com/the-automated-readability-index-ari/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}