esselunga.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Esselunga online shop scraping\n",
    "\n",
    "Product names, prices per unit and prices per kg/liter, nutritional information.\n",
    "\n",
    "Purpose: help calculating price and convenience of meals and diets, for example knowing which are the cheapest products per gram of protein (€/grams of protein) or per calories (€/cal)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial data scraping\n",
    "\n",
    "The initial scraping has been done manually, just going to every category of the online shop and launching a JS script to get to the bottom of the infinite scroll. This triggers all the xhr requests. Then, the JSON responses for every category have been saved to `*.har` files by right-clicking and selecting _\"Save All As HAR\"_.\n",
    "\n",
    "### JS autoscroll code\n",
    "\n",
    "from https://www.quora.com/Is-there-a-way-to-automatically-load-a-page-with-infinite-scroll-until-it-gets-to-the-bottom-of-the-feed-Is-there-a-software-or-a-setting-that-helps-to-do-this\n",
    "\n",
    "```var lastScrollHeight = 0;\n",
    "function autoScroll() {\n",
    "  var sh = document.documentElement.scrollHeight;\n",
    "  if (sh != lastScrollHeight) {\n",
    "    lastScrollHeight = sh;\n",
    "    document.documentElement.scrollTop = sh;\n",
    "  }\n",
    "}\n",
    "window.setInterval(autoScroll, 100);```\n",
    "\n",
    "## Parse responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re, os, json\n",
    "\n",
    "def label_to_price_per_kg(label):\n",
    "    regex = r'Euro (\\d+,\\d+) \\/ ([a-z]+)'\n",
    "    match = re.match(regex, label)\n",
    "    \n",
    "    if not match:\n",
    "        return None\n",
    "    \n",
    "    groups = list(match.groups())\n",
    "    price_per_kg = float(groups[0].replace(',', '.'))\n",
    "    \n",
    "    # Adjust price per kg if necessary\n",
    "    if groups[1] == 'g' or groups[1] == 'ml':\n",
    "        price_per_kg *= 1000\n",
    "    elif groups[1] == 'hg':\n",
    "        price_per_kg *= 10\n",
    "    elif groups[1] == 'pz':\n",
    "        price_per_kg = None\n",
    "    \n",
    "    return price_per_kg\n",
    "\n",
    "def parse_har(har_filename):\n",
    "    # Parse har as json\n",
    "    with open('har/{}'.format(har_filename)) as f:\n",
    "        js = json.loads(f.read())\n",
    "    \n",
    "    # Iterate over responses and get all the entities\n",
    "    entities = []\n",
    "    for entry in js['log']['entries']:\n",
    "        response = json.loads(entry['response']['content']['text'])\n",
    "        entities.extend(response['entities'])\n",
    "        \n",
    "    for entity in entities:\n",
    "        # Strip description\n",
    "        entity['description'] = entity['description'].strip()\n",
    "        \n",
    "        # Calculate price per kg by parsing the label\n",
    "        entity['price_per_kg'] = label_to_price_per_kg(entity['label'])\n",
    "        \n",
    "        # Add qty\n",
    "        entity['qty'] = ' '.join([entity['unitValue'], entity['unitText']])\n",
    "        \n",
    "        # Add category\n",
    "        entity['category'] = har_filename[:-4]\n",
    "        \n",
    "    # Only keep products with price per kg\n",
    "    entities = list(filter(lambda entity: entity['price_per_kg'], entities))\n",
    "    \n",
    "    # Keep only interesting keys\n",
    "    keys_to_keep = ['description', 'category', 'price', 'qty', 'price_per_kg']\n",
    "    \n",
    "    # Create dictionary with product id as key\n",
    "    entities = { entity['id'] : { k: entity[k] for k in keys_to_keep } for entity in entities }\n",
    "    \n",
    "    return entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build database from HARs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "entities = {}\n",
    "for har_filename in next(os.walk('har'))[2]:\n",
    "    entities.update(parse_har(har_filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scrape nutrition facts from IDs\n",
    "\n",
    "The `nutrition_facts` dictionary has the same keys as `entities`.\n",
    "If the value is `repeat`, it means the request has failed for some reason and needs to be repeated.\n",
    "Otherwise, if the response has status code 200 OK, the value would be the text of the response.\n",
    "Initially, the dictionary is created by copying `entities` and setting all the keys to `repeat`.\n",
    "\n",
    "This is done so that the data can be scraped in multiple passes without losing intermediate results, because after about half of the request I noticed my IP address has been 'banned' and every request ended in timeout.\n",
    "\n",
    "The code below opens and reads the dictionary in the JSON file and tries to make requests to the Esselunga online shop website to scrape nutritional information. To do so, first make a valid request from the browser in order to obtain valid cookies and headers for the request."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "import json, requests\n",
    "\n",
    "nutrition_facts = json.loads(open('nutrition_facts/nutrition_facts.json', 'r').read())\n",
    "\n",
    "print('Scraping. {} out of {} requests to repeat.'.format(sum([1 for value in nutrition_facts.values() if value == 'repeat']), len(nutrition_facts)))\n",
    "\n",
    "def scrape_nutrition_facts(_id):\n",
    "    # === START COOKIES AND HEADERS ===\n",
    "    \n",
    "    cookies = {}\n",
    "\n",
    "    headers = {}\n",
    "    \n",
    "    # === END COOKIES AND HEADERS ===\n",
    "    \n",
    "    url = 'https://www.esselungaacasa.it/ecommerce/resources/auth/displayable/detail/{}'.format(_id)\n",
    "    \n",
    "    try:\n",
    "        response = requests.get(url, headers=headers, cookies=cookies, timeout=5)\n",
    "    except Exception as e:\n",
    "        return 'repeat'\n",
    "    \n",
    "    if response.status_code == 200:\n",
    "        return json.loads(response.text)\n",
    "    else:\n",
    "        return 'repeat'\n",
    "\n",
    "for _id in nutrition_facts.keys():\n",
    "    if nutrition_facts[_id] == 'repeat':\n",
    "        nutrition_facts[_id] = scrape_nutrition_facts(_id)\n",
    "\n",
    "print('Finished. Remaining: {} out of {} requests to repeat.'.format(sum([1 for value in nutrition_facts.values() if value == 'repeat']), len(nutrition_facts)))\n",
    "        \n",
    "with open('nutrition_facts/nutrition_facts.json', 'w') as f:\n",
    "    f.write(json.dumps(nutrition_facts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse nutrition facts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6179 total entities. 1253 are missing nutrition facts.\n"
     ]
    }
   ],
   "source": [
    "def delete_nutrition_facts_by_id(_id):\n",
    "    nutrition_facts = json.loads(open('nutrition_facts/nutrition_facts.json', 'r').read())\n",
    "    \n",
    "    nutrition_facts.pop(str(_id))\n",
    "    \n",
    "    with open('nutrition_facts/nutrition_facts.json', 'w') as f:\n",
    "        f.write(json.dumps(nutrition_facts))\n",
    "\n",
    "def parse_table(_id):\n",
    "    return pd.read_html(nutrition_facts[_id])[0]\n",
    "\n",
    "def parse_nutrition_facts(nutrition_facts):\n",
    "    # Find tables containing nutritional info.\n",
    "    missing = 0\n",
    "    for _id in nutrition_facts.keys():\n",
    "        found = False\n",
    "        for idx, el in enumerate(nutrition_facts[_id]['informations']):\n",
    "            if 'value' in el and 'Informazioni nutrizionali' in el['value']:\n",
    "                nutrition_facts[_id] = el['value']\n",
    "                found = True\n",
    "                break\n",
    "                    \n",
    "        if not found:\n",
    "            nutrition_facts[_id] = None\n",
    "            missing += 1\n",
    "    \n",
    "    # Fill missing keys with None.\n",
    "    for _id in entities.keys():\n",
    "        if str(_id) not in nutrition_facts:\n",
    "            nutrition_facts[_id] = None\n",
    "            missing += 1\n",
    "            \n",
    "    # Now 'entities' and 'nutrition_facts' have the same keys.\n",
    "    print('{} total entities. {} are missing nutrition facts.'.format(len(nutrition_facts), missing))\n",
    "    \n",
    "    # Convert str keys to int keys\n",
    "    nutrition_facts = { int(_id) : fact for _id, fact in nutrition_facts.items() }\n",
    "    \n",
    "    # Parse raw HTML table and extract data\n",
    "    for _id, html in nutrition_facts.items():\n",
    "    \n",
    "        if html is None:\n",
    "            continue\n",
    "        \n",
    "        # Parse table\n",
    "        table = pd.read_html(html)[0]\n",
    "        \n",
    "        # Identify columns in table: labels ('nutrizionali') and values per 100g\n",
    "        cols_nutrizionali = [col for col in table.columns if 'nutrizionali' in col]\n",
    "        cols_100 = [col for col in table.columns if '100' in col]\n",
    "        \n",
    "        # If either column is not found, forget the nutrition facts for this item\n",
    "        if not cols_nutrizionali:\n",
    "            nutrition_facts[_id] = None\n",
    "            continue\n",
    "        \n",
    "        if not cols_100:\n",
    "            nutrition_facts[_id] = None\n",
    "            continue\n",
    "            \n",
    "        col_nutrizionali = cols_nutrizionali[0]\n",
    "            \n",
    "        # Get calories separately because the corresponding label is not always consistent\n",
    "        # col_100 will be the one where kcal are found\n",
    "        regexes = [r'(\\d+),?\\d*? *kcal', r'kcal *(\\d+)']\n",
    "        matches = pd.DataFrame()\n",
    "        \n",
    "        it = iter([(col, regex) for col in cols_100 for regex in regexes])\n",
    "\n",
    "        col_regex = None\n",
    "        while matches.empty:\n",
    "            try:\n",
    "                col_regex = next(it)\n",
    "            except StopIteration:\n",
    "                break\n",
    "            \n",
    "            matches = table[col_regex[0]].astype(str).str.extract(col_regex[1], flags=re.IGNORECASE).dropna()\n",
    "        \n",
    "        col_100 = col_regex[0]\n",
    "        kcal = int(matches[0].tolist()[0])\n",
    "        \n",
    "        d = {}\n",
    "        d['kcal'] = kcal\n",
    "        \n",
    "        keywords = ['carboidrati', 'zuccheri', 'proteine', 'grassi', 'saturi', 'monoinsaturi', 'polinsaturi', 'fibre', 'sale']\n",
    "        translate = {'carboidrati' : 'carbs',\n",
    "                     'zuccheri': 'sugars',\n",
    "                     'proteine': 'protein',\n",
    "                     'grassi': 'fat',\n",
    "                     'saturi': 'saturated',\n",
    "                     'monoinsaturi': 'monounsat.',\n",
    "                     'polinsaturi': 'polyunsat.',\n",
    "                     'fibre': 'fiber',\n",
    "                     'sale': 'salt'}\n",
    "        \n",
    "        # For each keyword, find it in 'col_nutrizionali' and get the corresponding value in 'col_100'\n",
    "        for keyword in keywords:\n",
    "            found = list(table[table[col_nutrizionali].str.contains(keyword, case=False).fillna(False)][col_100])\n",
    "            \n",
    "            if not found:\n",
    "                continue\n",
    "                \n",
    "            found = found[0]\n",
    "            \n",
    "            if isinstance(found, float) or isinstance(found, int):\n",
    "                d[translate[keyword]] = float(found)\n",
    "                continue\n",
    "\n",
    "            if isinstance(found, str):\n",
    "                found = found.replace(',', '.').strip()\n",
    "            \n",
    "            numeric_const_pattern = '[-+]? (?: (?: \\d* \\. \\d+ ) | (?: \\d+ \\.? ) )(?: [Ee] [+-]? \\d+ ) ?'\n",
    "            rx = re.compile(numeric_const_pattern, re.VERBOSE)\n",
    "            parsed = rx.findall(found)\n",
    "                    \n",
    "            if not parsed:\n",
    "                d[translate[keyword]] = float(0)\n",
    "            else:\n",
    "                d[translate[keyword]] = float(parsed[0])\n",
    "                    \n",
    "        nutrition_facts[_id] = d\n",
    "    \n",
    "    return nutrition_facts\n",
    "\n",
    "nutrition_facts = json.loads(open('nutrition_facts/nutrition_facts.json', 'r').read())\n",
    "nutrition_facts = parse_nutrition_facts(nutrition_facts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge with database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First make sure keys are exactly the same\n",
    "assert set(entities.keys()) == set(nutrition_facts.keys())\n",
    "\n",
    "# Merge things\n",
    "for _id in entities.keys():\n",
    "    \n",
    "    d = nutrition_facts[_id]\n",
    "    \n",
    "    if d is not None:\n",
    "        entities[_id].update(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "tols = [5.0, 10.0, 20.0, 50.0, 100.0]\n",
    "count = [0, 0, 0, 0, 0]\n",
    "diffs = []\n",
    "\n",
    "for _id, entity in entities.items():\n",
    "    kcal_actual = entity.get('kcal', 0.0)\n",
    "    kcal_calculated = 4 * entity.get('carboidrati', 0.0) + 4 * entity.get('proteine', 0.0) + 9 * entity.get('grassi', 0.0) + 1.5 * entity.get('fibre', 0.0)\n",
    "    \n",
    "    if not math.isnan(kcal_calculated):\n",
    "        diffs.append(abs(kcal_actual - kcal_calculated))\n",
    "    \n",
    "    for i, tol in enumerate(tols):\n",
    "        if not math.isclose(kcal_actual, kcal_calculated, abs_tol=tol):\n",
    "            count[i] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAELCAYAAADOeWEXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VGX+/vH3J53QA6GHJiAgRSTUIKioiKCAomCDVRERVFzYdXW/7tq26O6CWABFUbGsiCjFhgKyIKFIIihIjYKEmiAtlACB5/fHHPxlWZQEMjnJ5H5d11zMPDmT3OMQb845c57HnHOIiIjkVZjfAUREpHhRcYiISL6oOEREJF9UHCIiki8qDhERyRcVh4iI5IuKQ0RE8kXFISIi+aLiEBGRfInwO0AwVK5c2dWtW9fvGCIixUpqauou51z8mbYLyeKoW7cuKSkpfscQESlWzOzHvGynQ1UiIpIvKg4REckXFYeIiOSLikNERPJFxSEiIvmi4hARkXxRcYiISL6oOHLZd/gYExduJDltF7sOHPE7johIkRSSFwCerXU7snjyo9U/P65cJorzq5Xl/KrlaFytLOdXK0ujqmUpFRXuY0oREX+pOHJpWy+OZf93Oet2ZLF2x37W7chi3c4s/v3Vj2QfOwGAGdSJiw0USrX/Xyh1K5UmPMx8fgUiIsGn4jhFfNlo4stG06lh5Z/Hjp9wbN59iHU79rN2R1agUHZkMXv1Tk64wDbREWE0rFqGxtXKcXXzanRuGE9EuI4EikjoMeec3xkKXGJioiuMuaqyjx1nw84DrNuZ9XOprNy6j72HjlG1XDR9W9fixsQE6lQqHfQsIiLnysxSnXOJZ9xOxVGwjh0/wdw1GUxJSec/6zI44aB9/Tj6tUmge7PqxETq/IiIFE0qjiIwO+6OfdlMTU1nSsoWNu8+RNmYCHpfWJN+bRJoVrO83/FERP6LiqMIFMdJJ044lmz8iSnL0vlk1Q6O5pygafVy9GuTQO8La1I+NtLviCIiKo6iVBy57Tt0jBnfbOXdZel8t20/URFhdG9WjX6JCbSvX4kwfTJLRHyi4iiixZHbqq37mJKSzvTlW9mfnUPtuFhuaF2Lvom1qF6+lN/xRKSEUXEUg+I4KfvYcWat2sG7y9JZ/MNPhBl0aRRPvzYJXNa4KlER+liviASfiqMYFUduP/50kPdStjA1dQs79mdTqXQU110UOKHeoEpZv+OJSAhTcRTT4jjp+AnHgvWZTF62mblrMsg54WhdpyL9EhPo0aI6paN17aaIFCwVRzEvjtwys44wbfkW3l2WzveZBykdFU7PFjXo1zaBVgkVMNMJdRE5dyqOECqOk5xzfL15D5O/Sufjlds5dPQ4DauUoV+bBPq0qkmlMtF+RxSRYkzFEYLFkduBIzl89M023k1JZ/nmvUSGG5c3qUq/Ngl0aRSvvRARyTcVR4gXR27rd2YxZVk6Hyzfyu6DR+nRojqjbmip6U1EJF/yWhz6nGcIaFS1LI/0bMqSh7vy+27n88nK7fSfsITMLC1GJSIFT8URQqIiwhh2aQPG39KatTv203tsMmt37Pc7loiEGBVHCLqqWTWm3N2BY8dP0Hf8Yuaty/A7koiEEBVHiGpRqwIz7k2idlwsd76+jEmLNvkdSURChIojhFUvX4r3hnTgssZVeHTmdzw6YxU5x0/4HUtEijkVR4grHR3BS7clMqhTPSYt/pFBb6SQlX3M71giUowFvTjMLNzMlpvZR95jM7O/mtl6M1tjZvfnGn/OzNLM7FszuyjX9xhoZhu828BgZw414WHGIz2b8rc+zflywy76jl/Mlj2H/I4lIsVUYexxDAfW5Hr8GyABaOycawJM9sa7Aw2922BgPICZxQGPAu2AtsCjZlaxEHKHnJvb1WbS7W3Ztu8wvccm8/XmPX5HEpFiKKjFYWa1gB7AK7mG7wGecM6dAHDOnfzITy/gDRewBKhgZtWBbsBs59xu59weYDZwVTBzh7JODSszbWhHYqMi6D9hCR9+s83vSCJSzAR7j2MM8CCQ+4zseUA/M0sxs0/NrKE3XhNIz7XdFm/sl8blLDWoUpbpw5JoWas8972znOfmbiAUZxAQkeAIWnGYWU8gwzmXesqXooFs77L2l4FXTz7lNN/G/cr4qT9vsFdGKZmZmeeQvGSIKx3FW4Pa0adVTUbPXs+IKd9wJOe437FEpBgI5h5HEnCtmW0icB7jMjN7i8Aew/veNtOAFt79LQTOfZxUC9j2K+P/xTk3wTmX6JxLjI+PL8jXEbKiI8IZfWNLRl7RiGnLt3LrK0vZffCo37FEpIgLWnE45x52ztVyztUF+gNfOOduBaYDl3mbdQHWe/dnAgO8T1e1B/Y557YDnwFXmllF76T4ld6YFAAz476uDXn+plZ8s2Ufvccmk5ZxwO9YIlKE+XEdx1PA9Wa2Evg7MMgb/wT4AUgjcAhrKIBzbjfwJLDMuz3hjUkBuqZlDSYPbs+hozn0GZfMwg27/I4kIkWUplWX/5K++xCDJqWQlnmAJ3s14+Z2tf2OJCKFRNOqy1lJiItl6j0d6NSgMn+ctpK/fLSa4ydC7x8XInL2VBzyP8rGRDJxYCIDO9ThlYUbufvNVA4eyfE7logUESoOOa2I8DAe79WMx6+9gC/W7uSGFxezfd9hv2OJSBGg4pBfNbBjXSb+pg2bdx+i1wvJrNyyz+9IIuIzFYec0aXnV+H9ezoSGR7GDS8tYtaq7X5HEhEfqTgkT86vFpimpHG1cgx562tenP+9pikRKaFUHJJn8WWjmTy4PT1aVOepT9fyh/e/5WiOFoYSKWki/A4gxUtMZDjP929F/cqlef6LNNJ3H2b8rRdRITbK72giUki0xyH5FhZmjLzyfEbf2JLUH/dw3bhFbNx10O9YIlJIVBxy1q67qBZv39WOPYeO0mdcMkt++MnvSCJSCFQcck7a1I1j+rAkKpWO4raJS3kvJf3MTxKRYk3FIeesTqXSfDA0ibb14vj91G95etZaTmiaEpGQpeKQAlG+VCSv396Wm9rWZvx/vmfYv7/m8FEtDCUSilQcUmAiw8P4W59mPNKjCbO+20G/CYvJ2J/tdywRKWAqDilQZsagi+sz4bZE0jIO0HtsMqu37fc7logUIBWHBMUVTavy3pAOnHBww4uLmLtmp9+RRKSAqDgkaC6oUZ4Z9yZRP74Md72RwsSFGzVNiUgIUHFIUFUtF8O7d7fniqZVefKj1TwyfRXHjmuaEpHiTMUhQRcbFcH4W1ozpMt5vL10M3e8vox9h4/5HUtEzpKKQwpFWJjxUPfG/OP6Fiz+/ieuH7+IzT8d8juWiJwFFYcUqhvbJPDmne3IzDpC73HJpGza7XckEcknFYcUug7nVWLa0I6Ui4ng5peXMmPFVr8jiUg+qDjEF/XjyzBtaBKtaldg+OQVPDN7vT5xJVJMqDjENxVLR/Hmne3o27oWz87dwPDJK8g+pmlKRIo6LeQkvoqKCOOffVtQP740/5i1ji17DjFhQCKVy0T7HU1EfoH2OMR3ZsbQSxow/paLWL19P73HJrN+Z5bfsUTkF6g4pMjo3rw67w7uwJGcE1w/bhHz12f6HUlETkPFIUVKy4QKzBiWRK24WO54fRlvLt7kdyQROYWKQ4qcGhVK8d6QDlzSKJ4/zfiOx2Z+x3EtDCVSZKg4pEgqEx3BhAGJ3NmpHq8v2sSgScs4cCTH71gigopDirDwMONPPZvyl97NWLBhF33HL2Lr3sN+xxIp8VQcUuTd2r4Or9/ehq17D9PrhWRWpO/1O5JIiabikGLh4obxfHBPR0pFhdHvpcV8snK735FESiwVhxQbDauWZfrQJJrXLM/Qt79m7Lw0TVMi4gMVhxQrlcpE89agdvS6sAb//Gwdv3vvW47kaJoSkcKkKUek2ImJDGdMvwupX7kMz8xZT/ruQ7x4W2viSkf5HU2kRNAehxRLZsbwyxvy3E2tWLFlL33GJfN95gG/Y4mUCCoOKdaubVmDd+5qz4HsHPqMTWZR2i6/I4mEvKAXh5mFm9lyM/volPHnzexArsfRZvaumaWZ2VIzq5vraw974+vMrFuwM0vx0rpORaYPS6Ja+RgGvPoVk7/a7HckkZBWGHscw4E1uQfMLBGocMp2dwJ7nHMNgGeAp71tmwL9gQuAq4BxZhYe7NBSvCTExTL1no50bFCZhz5Yyd8/WcMJTVMiEhRBLQ4zqwX0AF7JNRYO/BN48JTNewGTvPtTga5mZt74ZOfcEefcRiANaBvM3FI8lYuJ5NWBiQzoUIeXFvzAkLdSOXRU05SIFLRg73GMIVAQJ3KN3QvMdM6degVXTSAdwDmXA+wDKuUe92zxxkT+R0R4GE/0asZj1zRlzpqd3PjSYnbsy/Y7lkhICVpxmFlPIMM5l5prrAZwA/D86Z5ymjH3K+On/rzBZpZiZimZmVrHoaT7TVI9Jg5sw8bMg/Qau5BVW/f5HUkkZARzjyMJuNbMNgGTgcuA74AGQJo3Hmtmad72W4AEADOLAMoDu3OPe2oB2079Yc65Cc65ROdcYnx8fFBekBQvlzauwvtDOxIRFsYNLy7m8+92+B1JJCQErTiccw8752o55+oSOLn9hXOuonOumnOurjd+yDsZDjATGOjd7+tt77zx/t6nruoBDYGvgpVbQkvjauWYNqwjjaqV5e63Upmw4HtNUyJyjorSdRwTgUreHsgI4CEA59x3wBRgNTALGOac0xwTkmdVysbw7uD2XN2sOn/7ZC0Pf7CSY8dPnPmJInJaFor/+kpMTHQpKSl+x5Ai5sQJx+jZ63lhXhodz6vE+FtaUz420u9YIkWGmaU65xLPtF1R2uMQCaqwMON33c5n1A0tWbZpN33GJ7Np10G/Y4kUOyoOKXGub12Ltwe1Z8/Bo/QZl8xXG3f7HUmkWFFxSInUtl4c04YmUbF0FLe8soT3U7f4HUmk2FBxSIlVt3Jppt2TRJu6cYx87xv+9dk6TVMikgcqDinRysdGMumOttzUNoEX5qVx3zvLyT6mD+2J/Bot5CQlXmR4GH/r05z6lcvwt0/XsGXvYV4e0JoqZWP8jiZSJGmPQ4TAwlB3da7PS7e2Zv2OLPqMXcSa7fv9jiVSJKk4RHK58oJqvDekAzknTtB3/CLmrc3wO5JIkaPiEDlFs5rlmTGsE3Url+bOSct4PXmj35FEipQ8FYeZzc3LmEioqFY+hveGdODyJlV57MPV/HnGKnI0TYkIcIbiMLMYM4sDKptZRTOL8251gRqFEVDEL7FREbx4a2vu7lyfNxb/yJ2TUtiffczvWCK+O9Mex91AKtDY+/PkbQYwNrjRRPwXFmY8fHUTnrquOclpu+g7fhHpuw/5HUvEV79aHM65Z51z9YDfOefqO+fqebeWzrkXCimjiO/6t63NG3e0Zce+bPqMSyb1xz1+RxLxTZ7OcTjnnjezjmZ2s5kNOHkLdjiRoqRjg8pMG5ZE6egIbnp5CTO/+Z/1xERKhLyeHH8T+BfQCWjj3c449a5IqDkvvgzThyZxYUIF7n9nOc/O2aCFoaTEyeuV44lAU6ffEBEqlo7izTvb8scPVvHMnPVs3HWAp65vQUxkuN/RRApFXotjFVAN2B7ELCLFRnREOP+6oQX140vzz8/Wkb7nMBNua02lMtF+RxMJurxeAFgZWG1mn5nZzJO3YAYTKerMjGGXNmDszRexaus+eo9LZsPOLL9jiQRdXvc4HgtmCJHirEeL6tSsWIpBk1K4bvwinr6+Bd2bVcPM/I4mEhRac1ykgGzde5ghb6aycus+Lm9Shcd7NaNmhVJ+xxLJswJdc9zMssxsv3fLNrPjZqapQ0VyqVmhFNOGduSRHk1ITvuJK0bPZ+LCjRzX4lASYvJ6HUdZ51w57xYDXA/oAkCRU0SEhzHo4vp8/tvOtKsXx5MfrabPuGRWbd3ndzSRAnNWs+M656YDlxVwFpGQkRAXy6u/acMLN7di295seo1N5q8fr+bQ0Ry/o4mcszydHDez63I9DCNwXYf2v0V+hZnRs0UNLm4Qz1Oz1vLylxv5ZOUO/tK7GZc2ruJ3PJGzltc9jmty3boBWUCvYIUSCSXlYyP5+3XNeW9IB0pFhXP768u4999fk5GV7Xc0kbOiT1WJFKIjOcd5af4PvPBFGjGRYTx8dRP6JSYQFqaP7or/CvpTVbXMbJqZZZjZTjN738xqnXtMkZIlOiKc+7s2ZNYDF9O0Rjke/mAl/SYsJi1DFw5K8ZHXQ1WvATMJLN5UE/jQGxORs1A/vgzv3NWef/RtwYaMA3R/9ktGz15P9rHjfkcTOaO8Fke8c+4151yOd3sdiA9iLpGQZ2bcmJjA3BFd6NmiBs/N3cDVz37J4u9/8juayK/Ka3HsMrNbzSzcu90K6G+3SAGoVCaaZ/pdyJt3tiXnhOOml5fw4NRv2HvoqN/RRE4rr8VxB3AjsIPADLl9gduDFUqkJLq4YTyfPdCZey45j/e/3krXUfOZvnyr1vuQIievxfEkMNA5F++cq0KgSB4LWiqREqpUVDh/uKoxH93XiYS4WB54dwUDXv2KzT9pnXMpOvJaHC2ccz8vsuyc2w20Ck4kEWlSvRzv39ORJ3pdwPLNe7lyzHzG/+d7jh0/4Xc0kTwXR5iZVTz5wMziyPuU7CJyFsLDjAEd6jJnRBe6NIrn6Vlrueb5haxI3+t3NCnh8loco4BFZvakmT0BLAL+EbxYInJStfIxvHRbIi/d1pq9h47RZ1wyj85YRVb2Mb+jSQmV5yvHzawpgYkNDZjrnFsdzGDnQleOS6jKyj7GqM/XM2nxJqqWjeHxXhfQ7YJqfseSEJHXK8c15YhIMbQifS8Pf7CSNdv3c2XTqjze6wKql9eiUXJuCnTKEREpWi5MqMDMe5N4qHtjFmzI5IrRC5i0aJMWjZJCEfTi8C4YXG5mH3mP3zazdWa2ysxeNbNIb9zM7DkzSzOzb83solzfY6CZbfBuA4OdWaQ4iAwPY0iX8/j8gS60ql2BR2d+x/XjF7FmuxbnlOAqjD2O4cCaXI/fBhoDzYFSwCBvvDvQ0LsNBsbDz5/gehRoB7QFHs39CS+Rkq52pVjeuKMtz/a/kPTdh+j5/EKe+nQth49q3isJjqAWhzeDbg/glZNjzrlPnAf4Cjg5y24v4A3vS0uACmZWncD6H7Odc7u9a0lmA1cFM7dIcWNm9LqwJnNHdqHvRbV4cf73XDlmPgvWZ/odTUJQsPc4xgAPAv9z1ZJ3iOo2YJY3VBNIz7XJFm/sl8ZF5BQVYqN4um8LJg9uT2R4GANe/YoHJi9n14EjfkeTEBK04jCznkCGcy71FzYZByxwzn158imn2cb9yvipP2+wmaWYWUpmpv6VJSVb+/qV+HT4xQzv2pCPV27n8tHzmbIsXfNeSYEI5h5HEnCtmW0CJgOXmdlbAGb2KIFp2Ufk2n4LkJDrcS1g26+M/xfn3ATnXKJzLjE+XjO+i0RHhPPbKxrx6fCLaVSlLA++/y03vbyE7zMP+B1NirmgFYdz7mHnXC3nXF2gP/CFc+5WMxtE4LzFTc653IewZgIDvE9XtQf2Oee2A58BV5pZRe+k+JXemIjkQYMqZZk8uD1PXdec1dv2033Mlzw7ZwNHcnTyXM6OH9dxvAhUBRab2Qoz+7M3/gnwA5AGvAwMhZ8nVHwSWObdnvDGRCSPwsKM/m1rM2dkF7o1q8Yzc9bT47mFLNukXyXJP105LlICzVuXwSPTVrF172FuapvAQ1c1oXxspN+xxGe6clxEftGl51dh9ojODO5cnykpW+g6ej4ffrNNJ88lT1QcIiVUbFQEf7y6CTPvTaJGhRjue2c5t7++jPTdWjRKfp2KQ6SEu6BGeaYNTeLPPZvy1cbdXPnMAl5e8AM5WjRKfoGKQ0QIDzPu6FSP2SO6kNSgEn/9ZA29xibz7RYtGiX/S8UhIj+rWaEULw9IZPwtF5GZdYTeY5N54sPVHDyS43c0KUJUHCLyX8yM7s2rM2dkF25pV4fXFm3kitHzmbN6p9/RpIhQcYjIaZWLieTJ3s2YOqQjZWMiGfRGCkPfTiVjf7bf0cRnKg4R+VWt61Tkw/s68ftu5zNnTQZdR83nzSU/ckKLRpVYKg4ROaOoiDCGXdqAzx/oTIuE8vxp+ipueGkx63Zk+R1NfKDiEJE8q1u5NG/d2Y7RN7bkh8wD9HjuS/752Vqyj2neq5JExSEi+WJmXHdRLeaOvIReF9Zk7LzvuWrMAhal7fI7mhQSFYeInJW40lGMurElbw9qB8DNryxlxJQV7D541OdkEmwqDhE5J0kNKjPrgc7ce2kDZq7YRtdR/+H91C2a9yqEqThE5JzFRIbzu27n88nwi6kfX4aR733DrROXsmnXQb+jSRCoOESkwDSqWpb37u7AX3o349v0fXQbs4Cx89I4mqN5r0KJikNEClRYmHFr+zrMGdmFrk2q8M/P1nHN8wtJ/VGLRoUKFYeIBEXVcjGMu6U1EwcmkpV9jL4vLuaR6SvZn33M72hyjlQcIhJUXZtUZfaILtyRVI9/L93M5aPm8+nK7Tp5XoypOEQk6EpHR/Cnnk2ZMawT8WWjueftr7nrjRS27T3sdzQ5CyoOESk0zWuVZ8awJP7v6iYkp/3E5aPnM3HhRo5r3qtiRcUhIoUqIjyMuzrX5/PfdqZdvTie/Gg1fcYls2rrPr+jSR6pOETEFwlxsbz6mza8cHMrtu3NptfYZP768WoOHdWiUUWdikNEfGNm9GxRg7kjunBjYgIvf7mRK0YvYN66DL+jya9QcYiI78rHRvL365rz3pAOlIoK5/bXlnHvv78mI0uLRhVFKg4RKTLa1I3j4/s7MeKKRnz+3U4uHzWfd77arEWjihgVh4gUKdER4dzftSGzHriYpjXK8fAHK+k3YTFpGVo0qqhQcYhIkVQ/vgzv3NWef/RtwYaMA3R/9ktGz16vRaOKABWHiBRZZsaNiQnMGdGFHs2r89zcDVz97Jcs/v4nv6OVaCoOESnyKpeJZkz/VrxxR1tyTjhuenkJD079hr2HtGiUH1QcIlJsdG4Uz2cPdOaeS87j/a+30nXUfGas2Kp5rwqZikNEipVSUeH84arGfHRfJxLiYhk+eQUDXv2KzT8d8jtaiaHiEJFiqUn1crx/T0cev/YClm/ey5Vj5jP+P99z7LgWjQo2FYeIFFvhYcbAjnWZM6ILXRrF8/SstVzz/EJWpO/1O1pIU3GISLFXrXwML92WyEu3tWbvoWP0GZfMYzO/I0uLRgWFikNEQka3C6oxe0RnBnaoy6TFm7hi9AI++26H37FCjopDREJK2ZhIHrv2Aj64pyMVYiO5+81UBr+RwvZ9WjSqoKg4RCQktapdkQ/v68RD3RuzYEMmV4xewKRFm7RoVAFQcYhIyIoMD2NIl/P4/IEutKpdgUdnfsf14xexZvt+v6MVa0EvDjMLN7PlZvaR97iemS01sw1m9q6ZRXnj0d7jNO/rdXN9j4e98XVm1i3YmUUktNSuFMsbd7Tl2f4Xkr77ENc8v5CnPl3L4aOa9+psFMYex3BgTa7HTwPPOOcaAnuAO73xO4E9zrkGwDPedphZU6A/cAFwFTDOzMILIbeIhBAzo9eFNZk7sgvXXVSTF+d/T7cxC1iwPtPvaMVOUIvDzGoBPYBXvMcGXAZM9TaZBPT27vfyHuN9vau3fS9gsnPuiHNuI5AGtA1mbhEJXRVio/hH35ZMHtyeiHBjwKtf8cDk5ew6cMTvaMVGsPc4xgAPAicv5awE7HXOnVxUeAtQ07tfE0gH8L6+z9v+5/HTPEdE5Ky0r1+JT4dfzPCuDfl45XYuHz2fKSnpmvcqD4JWHGbWE8hwzqXmHj7Npu4MX/u15+T+eYPNLMXMUjIztespImcWHRHOb69oxKfDL6ZRlbI8OPVbbnp5CT9kHvA7WpEWzD2OJOBaM9sETCZwiGoMUMHMIrxtagHbvPtbgAQA7+vlgd25x0/znJ855yY45xKdc4nx8fEF/2pEJGQ1qFKWyYPb8/frmrN6236uGvMlz87ZwJEcnTw/naAVh3PuYedcLedcXQInt79wzt0CzAP6epsNBGZ492d6j/G+/oUL7DPOBPp7n7qqBzQEvgpWbhEpmcLCjJva1mbOyC50a1aNZ+asp8dzC1m2abff0YocP67j+AMwwszSCJzDmOiNTwQqeeMjgIcAnHPfAVOA1cAsYJhzTv8MEJGgqFI2hudvasVrt7fh8NHj3PDiYh7+YCX7Dmneq5MsFE8EJSYmupSUFL9jiEgxd+hoDmPmbGDiwo1UjI3i0Wua0rNFdQIf+Aw9ZpbqnEs803a6clxE5BfERkXwx6ubMGNYEjUqxHDfO8u54/VlpO8u2YtGqThERM6gWc3yTBuaxJ97NmXpxt1c+cwCXl7wAzkldNEoFYeISB6Ehxl3dKrH7BFdSGpQib9+soZeY5NZuWWf39EKnYpDRCQfalYoxcsDEhl/y0VkZh2h19iFPPHhag4eyTnzk0OEikNEJJ/MjO7NqzNnZBdubleb1xZt5MpnFjB3zU6/oxUKFYeIyFkqFxPJX3o3Z+qQjpSJjuDOSSkMfTuVjP3ZfkcLKhWHiMg5al0nsGjU77udz5w1GXQdNZ+3lvzIiRBdNErFISJSAKIiwhh2aQM+f6AzLRLK88j0Vdzw0mLW78zyO1qBU3GIiBSgupVL89ad7Rh1Q0t+yDxAj+e+5F+frSP7WOhMeKHiEBEpYGbG9a1rMXfkJVzbsiYvzEvjqjELWJS2y+9oBULFISISJHGloxh1Y0veHtQOgJtfWcrIKd+w++BRn5OdGxWHiEiQJTWozKwHOnPvpQ2YsWIrXUf9hw++3lJsF41ScYiIFIKYyHB+1+18Pr7/YurHl2HElG+4deJSNu066He0fFNxiIgUovOrleW9uzvwl97N+DZ9H93GLGDsvDSO5hSfea9UHCIihSwszLi1fR3mjOxC1yZV+Odn67jm+YWk/rjH72h5ouIQEfFJ1XIxjLulNa8MSCQr+xh9X1zEI9NXsj+7aC8apeIQEfHZ5U2rMntEF27vWI9/L93M5aPm8+n/liEXAAAHA0lEQVTK7UX25LmKQ0SkCCgdHcGfr2nKjGGdiC8bzT1vf81db6Swbe9hv6P9DxWHiEgR0rxWeWYMS+L/rm5CctpPXD56Pq8u3MjxIjTvlYpDRKSIiQgP467O9fn8t51pWy+OJz5aTZ9xyazaWjQWjVJxiIgUUQlxsbz2mzY8f1Mrtu3NptfYZP72yRoOHfV30SgVh4hIEWZmXNOyBnNHdOHGxAQmLPiBK0YvYN66DN8yqThERIqB8rGR/P265rw3pAOlosK5/bVl3PfOcjKzjhR6FhWHiEgx0qZuHB/f34kRVzTis1U76DrqP0z+anOhLhql4hARKWaiI8K5v2tDPn3gYprWKMdDH6yk/4QlpGUUzqJRKg4RkWLqvPgyvHNXe/7RtwXrM7Lo/uyXjJ2XFvSfGxH0nyAiIkFjZtyYmMBljavwl49WF8rV5ioOEZEQULlMNGP6tyqU4tChKhGREGJmQf8ZKg4REckXFYeIiOSLikNERPJFxSEiIvmi4hARkXxRcYiISL6oOEREJF+sqK5pey7MLBP40e8cPqoM7PI7hI/0+vX69frPTh3nXPyZNgrJ4ijpzCzFOZfodw6/6PXr9ev1B/f161CViIjki4pDRETyRcURmib4HcBnev0lm15/kOkch4iI5Iv2OEREJF9UHMWYmSWY2TwzW2Nm35nZcG88zsxmm9kG78+KfmcNJjMLN7PlZvaR97iemS31Xv+7Zhbld8ZgMbMKZjbVzNZ6fw86lKT338x+6/3dX2Vm75hZTKi//2b2qpllmNmqXGOnfc8t4DkzSzOzb83sooLIoOIo3nKAkc65JkB7YJiZNQUeAuY65xoCc73HoWw4sCbX46eBZ7zXvwe405dUheNZYJZzrjHQksB/hxLx/ptZTeB+INE51wwIB/oT+u//68BVp4z90nveHWjo3QYD4wsigIqjGHPObXfOfe3dzyLwP42aQC9gkrfZJKC3PwmDz8xqAT2AV7zHBlwGTPU2CdnXb2blgM7ARADn3FHn3F5K0PtPYBXTUmYWAcQC2wnx9985twDYfcrwL73nvYA3XMASoIKZVT/XDCqOEGFmdYFWwFKgqnNuOwTKBajiX7KgGwM8CJzwHlcC9jrncrzHWwiUaSiqD2QCr3mH6l4xs9KUkPffObcV+BewmUBh7ANSKTnvf26/9J7XBNJzbVcg/z1UHCHAzMoA7wMPOOf2+52nsJhZTyDDOZeae/g0m4bqRwcjgIuA8c65VsBBQvSw1Ol4x/F7AfWAGkBpAodmThWq739eBOX3QcVRzJlZJIHSeNs594E3vPPk7qj3Z4Zf+YIsCbjWzDYBkwkcohhDYHc8wtumFrDNn3hBtwXY4pxb6j2eSqBISsr7fzmw0TmX6Zw7BnwAdKTkvP+5/dJ7vgVIyLVdgfz3UHEUY97x/InAGufc6FxfmgkM9O4PBGYUdrbC4Jx72DlXyzlXl8BJ0S+cc7cA84C+3mah/Pp3AOlmdr431BVYTQl5/wkcompvZrHe78LJ118i3v9T/NJ7PhMY4H26qj2w7+QhrXOhCwCLMTPrBHwJrOT/H+P/I4HzHFOA2gR+uW5wzp16Mi2kmNklwO+ccz3NrD6BPZA4YDlwq3PuiJ/5gsXMLiTwwYAo4AfgdgL/ICwR77+ZPQ70I/AJw+XAIALH8EP2/Tezd4BLCMyCuxN4FJjOad5zr1BfIPAprEPA7c65lHPOoOIQEZH80KEqERHJFxWHiIjki4pDRETyRcUhIiL5ouIQEZF8UXGIiEi+qDhECoCZHTjH519iZh3zsN1jZva7c/lZIudKxSFSNFxCYLoMkSJPxSGST2Y23cxSvQWEBucaH2VmX5vZXDOL98buN7PV3iI6k3/h+9UFhgC/NbMVZnaxmdXxvs+33p+1C+O1ieSFikMk/+5wzrUGEoH7zawSgZlZv3bOXQTMJzANBARmq23lnGtBoBz+h3NuE/AigcWHLnTOfUlgmog3vOe9DTwXzBckkh8qDpH8u9/MvgGWEJh5tCGBucLe9b7+FtDJu/8t8LaZ3UpgPqW86gD827v/Zq7vJ+I7FYdIPniTKV4OdHDOtSQwiV7MaTY9OQlcD2As0BpIzTXdd35pUjkpMlQcIvlTHtjjnDtkZo0JrPUOgd+lk1N53wwsNLMwIME5N4/AKoUVgDK/8H2zgLK5Hi8iMFU8wC3AwoJ7CSLn5mz/9SNSUs0ChpjZt8A6AoerILD63gVmlkpgCdN+QDjwlpmVJ7AS2zPemuCn8yEw1cx6AfcB9wOvmtnvCSwPe3uwXpBIfmladRERyRcdqhIRkXzRoSqRQmRmtwPDTxlOds4N8yOPyNnQoSoREckXHaoSEZF8UXGIiEi+qDhERCRfVBwiIpIvKg4REcmX/wd7ww/LzwNN7QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.plot(tols, count)\n",
    "plt.xlabel('abs_tol')\n",
    "plt.ylabel('count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most products are within 20 calories of the calculated ones.\n",
    "\n",
    "Most of the products which are **not** are _weird candy with polyols_.\n",
    "\n",
    "Some of them are blatant errors though. Let's remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAG+ZJREFUeJzt3XtwXOWd5vHvo5stX2VsAbZssJk4mUBCCGiMcyuYMAMmy+DU1qYCm0k8Iax3ZsluksnUBJKqYYcUtdlMKtmlNmHKu6iAGgLLLmRiUs4SD4GQVIWLINyMAStAQJKNZGzJ2Gpbt9/+0aehI0uy7qe7z/Op6tLpX7/dek9h+tF5z3vOq4jAzMyypyrtDpiZWTocAGZmGeUAMDPLKAeAmVlGOQDMzDLKAWBmllEOADOzjHIAmJlllAPAzCyjatLuwHhWrFgRa9euTbsbZmZl5YknntgfEY0nalfSAbB27VpaW1vT7oaZWVmR9LuJtPMQkJlZRjkAzMwyygFgZpZRDgAzs4xyAJiZZZQDwMwsoxwAZmYZ5QAwMysx9zzRzp2PvTbrv8cBYGZWYu55sp17nmif9d/jADAzKzE9fQM0LKid9d/jADAzKzG9uQGW1DsAzMwypzc3QEN93az/HgeAmVkJGRga5vCxQQ8BmZllTW9uAIClHgIyM8uWQgD4CMDMLGN6+nwEYGaWSb25fsABYGaWOe8MAXkWkJlZphSGgBpK4QhA0hpJD0raLWmXpC8l9X+Q9IKkZyT9SFJD0Xuuk9Qm6UVJlxTVNyW1NknXzs4umZmVr0IAlMqFYIPAVyPivcBG4BpJZwI7gfdFxNnAS8B1AMlrVwBnAZuAH0iqllQNfB+4FDgTuDJpa2Zmid7cAIvn11BdpVn/XScMgIjYGxFPJttvAbuBpoj4WUQMJs0eAVYn25uBuyLiWES8ArQBG5JHW0S8HBH9wF1JWzMzS/Tm5uY+QDDJcwCS1gIfBB4d8dJVwE+T7Sbg9aLX2pPaWPWRv2OrpFZJrd3d3ZPpnplZ2evp65+TGUAwiQCQtAi4B/hyRBwqqn+D/DDRHYXSKG+Pceq/X4jYFhHNEdHc2Ng40e6ZmVWEuboPEEDNRBpJqiX/5X9HRNxbVN8CXAZcFBGFL/N2YE3R21cDncn2WHUzMwN6cgOsbKifk981kVlAAm4BdkfEd4vqm4CvAZdHRF/RW7YDV0iaJ2kdsB54DHgcWC9pnaQ68ieKt8/crpiZlb/evoE5GwKayBHAR4DPAs9KeiqpfR24CZgH7MxnBI9ExF9GxC5JdwPPkx8auiYihgAkfRG4H6gGWiJi14zujZlZGYuIZAioRAIgIn7F6OP3O8Z5z43AjaPUd4z3PjOzLDvSP8TgcJTmLCAzM5s9PX1zdx8gcACYmZWMd9YCmJtZQA4AM7MS0ds3d2sBgAPAzKxk9MzhamDgADAzKxlzuRoYOADMzErGO7eC9jkAM7NM6cn1U1ddxfzauflqdgCYmZWIQ7kBli6oJbm4dtY5AMzMSkRP39xdBQwOADOzktEzh/cBAgeAmVnJ6JnDxWDAAWBmVjIO5Qbm7CpgcACYmZWMuVwNDBwAZmYlYWBomCP9Qx4CMjPLmrm+ChgcAGZmJaFwFbCHgMzMMqY3N7drAYADwMysJLwzBFRCs4AkrZH0oKTdknZJ+lJSP0nSTkl7kp/Lkrok3SSpTdIzks4t+qwtSfs9krbM3m6ZmZWXUh0CGgS+GhHvBTYC10g6E7gWeCAi1gMPJM8BLgXWJ4+twM2QDwzgeuB8YANwfSE0zMyy7p07gZZQAETE3oh4Mtl+C9gNNAGbgduSZrcBn0y2NwO3R94jQIOklcAlwM6IOBARB4GdwKYZ3RszszJVGAJaUkoBUEzSWuCDwKPAKRGxF/IhAZycNGsCXi96W3tSG6s+8ndsldQqqbW7u3sy3TMzK1u9uQEWz6+humpu7gQKkwgASYuAe4AvR8Sh8ZqOUotx6r9fiNgWEc0R0dzY2DjR7pmZlbWevv45vQYAJhgAkmrJf/nfERH3JuU3kqEdkp9dSb0dWFP09tVA5zh1M7PM680NzNlKYAUTmQUk4BZgd0R8t+il7UBhJs8W4MdF9c8ls4E2Ar3JENH9wMWSliUnfy9OamZmmdeTm9tbQQPUTKDNR4DPAs9KeiqpfR34FnC3pC8ArwGfSl7bAXwCaAP6gM8DRMQBSd8EHk/a3RARB2ZkL8zMylxv3wCrGurn9HeeMAAi4leMPn4PcNEo7QO4ZozPagFaJtNBM7MsyA8BleA5ADMzmz0RkcoQkAPAzCxlh48NMjQcpTkLyMzMZs/b9wEqtVlAZmY2uwq3gZjLq4DBAWBmlro0FoMBB4CZWeocAGZmGZXGraDBAWBmlrqeZDUwnwQ2M8uY3twAdTVVzK+d269kB4CZWcp6+/IXgeVvvTZ3HABmZinr6Zv720CAA8DMLHW9uYE5nwEEDgAzs9SlcR8gcACYmaWut6+fpXM8AwgcAGZmqfMQkJlZBvUPDnOkf8hDQGZmWZPWbSBgYmsCt0jqkvRcUe0cSY9IekpSq6QNSV2SbpLUJukZSecWvWeLpD3JY8tov8vMLGsKAVCqRwC3AptG1L4N/H1EnAP8XfIc4FJgffLYCtwMIOkk4HrgfGADcH2yMLyZWab1JreBKMkAiIiHgZGLtwewJNleCnQm25uB2yPvEaBB0krgEmBnRByIiIPATo4PFTOzzCncCK5hwdzPAjrhovBj+DJwv6TvkA+RDyf1JuD1onbtSW2suplZpr2zGlgJHgGM4a+Ar0TEGuArwC1JfbQbWcQ49eNI2pqcV2jt7u6eYvfMzMpDWreChqkHwBbg3mT7/5Af14f8X/ZritqtJj88NFb9OBGxLSKaI6K5sbFxit0zMysPPbl0loOEqQdAJ3BBsv1xYE+yvR34XDIbaCPQGxF7gfuBiyUtS07+XpzUzMwy7VBugCXza6iumts7gcIEzgFIuhO4EFghqZ38bJ5/B/x3STXAUfIzfgB2AJ8A2oA+4PMAEXFA0jeBx5N2N0TEyBPLZmaZ09PXz9IUrgGACQRARFw5xkvnjdI2gGvG+JwWoGVSvTMzq3A9uYE5XwmswFcCm5mlKK37AIEDwMwsVb19A6mcAAYHgJlZqvJDQA4AM7NMiQgPAZmZZdHhY4MMDUcqF4GBA8DMLDVv3wfIs4DMzLKlN8WrgMEBYGaWmjQXgwEHgJlZat65FbQDwMwsU3pSXAwGHABmZql5Zy0AnwQ2M8uU3r4B6mqqmF+bzlexA8DMLCU9fQMsra9FmvtbQYMDwMwsNb0p3gYCHABmZqnpyfWnNgMIHABmZqkpDAGlxQFgZpaS3twAS1OaAQQTWBHMzMxmVkTwvX/Zw97eo/zhqYtT68cJjwAktUjqkvTciPp/lPSipF2Svl1Uv05SW/LaJUX1TUmtTdK1M7sbZmblISL47s6XuOmBPXzqvNVc9dF1qfVlIkcAtwL/A7i9UJD0x8Bm4OyIOCbp5KR+JnAFcBawCvgXSe9O3vZ94E+BduBxSdsj4vmZ2hEzs1IXEXznZy/y/Qd/y6eb1/Bf/vX7qapKZwooTGxR+IclrR1R/ivgWxFxLGnTldQ3A3cl9VcktQEbktfaIuJlAEl3JW0dAGaWCRHBt+9/kZsf+i1XbljDjZ9M98sfpn4S+N3AxyQ9KukXkv4oqTcBrxe1a09qY9XNzCpeRPCtn77AzQ/9ls+cf1pJfPnD1E8C1wDLgI3AHwF3SzoDGG2PgtGDJkb7YElbga0Ap5122hS7Z2aWrv7BYd44dJR9h45y39Od3P7r3/HZjadzw+azUrvyd6SpBkA7cG9EBPCYpGFgRVJfU9RuNdCZbI9V/z0RsQ3YBtDc3DxqSJiZlZqOnhzfvO952nv62Nd7lP2H+3/v9S0fOp3/fHnpfPnD1APgn4GPAw8lJ3nrgP3AduCHkr5L/iTweuAx8kcG6yWtAzrInyj+t9Psu5lZyfj57jf4f7v28bH1K3h/01JOXVLPqUvncerSepoa6nnXyYvS7uJxThgAku4ELgRWSGoHrgdagJZkamg/sCU5Gtgl6W7yJ3cHgWsiYij5nC8C9wPVQEtE7JqF/TEzS0V7T4666ipu+/yGkhjfn4iJzAK6coyX/nyM9jcCN45S3wHsmFTvzMzKRMfBHCsb5pfNlz/4VhBmZjOioydHU0N92t2YFAeAmdkM6HQAmJllT//gMF1vHaNpmQPAzCxT9vbmiMBHAGZmWdNxMAc4AMzMMqe9JwkADwGZmWVLx8EcEqxc6gAwM8uUjp4cJy+eR11NeX2llldvzcxKUMfB8psCCg4AM7Np6+jJ0bRsQdrdmDQHgJnZNAwPB3t7fQRgZpY53YePMTAUZTcDCBwAZmbT0v72NQDzU+7J5DkAzMymoaNwDUCDzwGYmWXK21cBewjIzCxbOnr6WFpfy6J5U11gMT0OADOzaSjXawDAAWBmNi35awAqNAAktUjqStb/Hfna30gKSSuS55J0k6Q2Sc9IOreo7RZJe5LHlpndDTOzuRcRFX8EcCuwaWRR0hrgT4HXisqXAuuTx1bg5qTtSeQXkz8f2ABcL2nZdDpuZpa2Q7lBjvQPsbpSjwAi4mHgwCgvfQ/4WyCKapuB2yPvEaBB0krgEmBnRByIiIPATkYJFTOzctLe0weU3zoABVM6ByDpcqAjIp4e8VIT8HrR8/akNlZ9tM/eKqlVUmt3d/dUumdmNicKU0BXZSUAJC0AvgH83Wgvj1KLcerHFyO2RURzRDQ3NjZOtntmZnOmo0wXgimYyhHAHwDrgKclvQqsBp6UdCr5v+zXFLVdDXSOUzczK1sdB3PMr61i+cK6tLsyJZMOgIh4NiJOjoi1EbGW/Jf7uRGxD9gOfC6ZDbQR6I2IvcD9wMWSliUnfy9OamZmZaujJ8eqhnqk0QY5St9EpoHeCfwaeI+kdklfGKf5DuBloA34n8B/AIiIA8A3gceTxw1JzcysbHX0lO8UUIATXrscEVee4PW1RdsBXDNGuxagZZL9MzMrWR0Hc5y1akna3ZgyXwlsZjYFuf4h3jzSX9ZHAA4AM7Mp6Owt7xlA4AAwM5uSt68BWOoAMDPLlHK/BgAcAGZmU9JxMEd1lTh1SfktBVngADAzm4KOnhynLplPTXX5fo2Wb8/NzFJUzreBLnAAmJlNQTkvBFPgADAzm6TBoWH2HTrqIwAzs6zZd+goQ8PhIwAzs6zp7DkKlO9CMAUOADOzSepIVgIr14VgChwAZmaTVLgK2EcAZmYZ09GTY/nCOurrqtPuyrQ4AMzMJqn9YPlPAQUHgJnZpJX7QjAFDgAzs0mICDodAGZm2fPmkX6ODgxnYwhIUoukLknPFdX+QdILkp6R9CNJDUWvXSepTdKLki4pqm9Kam2Srp35XTEzm32VMgMIJnYEcCuwaURtJ/C+iDgbeAm4DkDSmcAVwFnJe34gqVpSNfB94FLgTODKpK2ZWVnpTNYBKPdrAGACARARDwMHRtR+FhGDydNHgNXJ9mbgrog4FhGvAG3AhuTRFhEvR0Q/cFfS1sysrBQWglmdhSGgCbgK+Gmy3QS8XvRae1Ibq34cSVsltUpq7e7unoHumZnNnLauwyyeV8PS+tq0uzJt0woASd8ABoE7CqVRmsU49eOLEdsiojkimhsbG6fTPTOzGTUwNMz9u/ZxwXsakUb7WisvNVN9o6QtwGXARRFR+DJvB9YUNVsNdCbbY9XNzMrCr9r2c7BvgMs/sCrtrsyIKR0BSNoEfA24PCL6il7aDlwhaZ6kdcB64DHgcWC9pHWS6sifKN4+va6bmc2t+57uZPH8Gi54T2WMTpzwCEDSncCFwApJ7cD15Gf9zAN2JodBj0TEX0bELkl3A8+THxq6JiKGks/5InA/UA20RMSuWdgfM7NZcXRgiJ/teoNL33cq82rK+x5ABScMgIi4cpTyLeO0vxG4cZT6DmDHpHpnZlYiHnyhi8PHBrn8nMoY/gFfCWxmNiH3PdPJikV1fOiM5Wl3ZcY4AMzMTuCtowM8sLuLT7x/JTXVlfO1WTl7YmY2S3Y+/wbHBocrZvZPgQPAzOwEtj/dSVNDPeeetiztrswoB4CZ2TgOHOnnV3v2c9kHVlJVVf4XfxVzAJiZjeOnz+1lcDj4s7Mra/gHHABmZuPa/lQnZzQu5KxVS9LuyoxzAJiZjWFf71Eee/UAl39gVUXc+2ckB4CZ2Rh+8kwnEfBnFTb7p8ABYGY2hvue7uSsVUv4g8ZFaXdlVjgAzMxG8er+Izzd3ltxc/+LOQDMzEbxk2fyd6y/zAFgZpYtO57dx3mnL6uIxd/H4gAwMxvhjUNHeX7vIf7kvaek3ZVZ5QAwMxvhFy/m1yO/sEIWfhmLA8DMbISHXurilCXz+MNTF6fdlVnlADAzKzI4NMwv9+zngndXxsLv4zlhAEhqkdQl6bmi2kmSdkrak/xcltQl6SZJbZKekXRu0Xu2JO33JAvKm5mVnCdf6+Gto4Nc+J6T0+7KrJvIEcCtwKYRtWuBByJiPfBA8hzgUvILwa8HtgI3Qz4wyK8lfD6wAbi+EBpmZqXkoRe7qK4SH3nXirS7MutOGAAR8TBwYER5M3Bbsn0b8Mmi+u2R9wjQIGklcAmwMyIORMRBYCfHh4qZWeoeerGb805bxtL62rS7Muumeg7glIjYC5D8LBwrNQGvF7VrT2pj1c3MSkZXMv3zggqf/VMw0yeBRztjEuPUj/8AaaukVkmt3d3dM9o5M7PxPPRSNqZ/Fkw1AN5IhnZIfnYl9XZgTVG71UDnOPXjRMS2iGiOiObGxmz8RzCz0vCLF7s5efE8zlxZeff+H81UA2A7UJjJswX4cVH9c8lsoI1AbzJEdD9wsaRlycnfi5OamVlJyE//7M7E9M+CmhM1kHQncCGwQlI7+dk83wLulvQF4DXgU0nzHcAngDagD/g8QEQckPRN4PGk3Q0RMfLEsplZan7zeg+HMjL9s+CEARARV47x0kWjtA3gmjE+pwVomVTvzMzmSGH650fXV/70zwJfCWxmRn7657mnNWRi+meBA8DMMq/rraPs6jyUqeEfcACYmb19988L3p2tmYcOADPLvIde6qZx8TzOWpWN6Z8FDgAzy7TBoWF++VK2pn8WOADMLNOeenv6Z7aGf8ABYGYZ94uXuqkSfOxd2QuAE14HYNlzdGCI/YeP0dM3wLHBYQaHhhkYCgaGhxkYHGZweNTbOI1raDgYGBpOHvntwaGY0meZzaT7nu7k3NOWsXRBdqZ/FjgAMmBgaJi2rsN09uTozQ1wKDdAb26QQ0cH6M0N0NM3wJtHjvHm4X7ePHyMI/1DaXfZbE5d9dF1aXchFQ6ACnN0YIgX9r3Fcx297OrsZVfnIV7Y9xb9g8PHtV1YV83S+lqW1NfSuHgep5+0gOWL5rF8UR3LF9bRsKCOeTVV1FbnHzXVoi75qVFv8Dq2KpH/nJoqaqtFbVV+u1oiY+fdrATNr61OuwupcABUkOc7D/G5lsfYf/gYAEvra3lf0xL+4sNrOWvVEtYuX/j2F/6S+TXUVPsUkFmWOQAqxDPtPXz2lsdYWFfNDz5zLmevXkpTQ33mprWZ2cQ5ACrAE787yF+0PEbDwlp+ePVG1py0IO0umVkZcACUuUdefpOrbn2cU5bM546rz2dVQ33aXTKzMuEAKGO/2rOfq29/nNXLFvDDq8/n5CXz0+6SmZURB0CZevCFLv79Pz3BGSsW8k9Xn8+KRfPS7pKZlRkHQInqHxwm1z9ET66fl/cf4ZXuI7y8/zCv7D/Cy91H2Nt7lPc3LeX2qzawbGFd2t01szJUkQHQ09fPp/7x12l3Y1IGh4O+/kH6+ofI9Q+NeoXskvk1nNG4iA+dsZz1pyzmMxtPY8n87F29aGYzY1oBIOkrwNVAAM+SXwN4JXAXcBLwJPDZiOiXNA+4HTgPeBP4dES8Op3fP5aqKrH+lEWz8dGzprqqigW11dTXVbMgedTX1bB4fg3rVizkjBULOWlhnad1mtmMmXIASGoC/hNwZkTkJN0NXEF+UfjvRcRdkv4R+AJwc/LzYES8S9IVwH8FPj3tPRjFkvm1/OAz583GR5uZVYzpXgpaA9RLqgEWAHuBjwP/N3n9NuCTyfbm5DnJ6xfJf86amaVmygEQER3Ad4DXyH/x9wJPAD0RMZg0aweaku0m4PXkvYNJ++VT/f1mZjY9Uw4AScvI/1W/DlgFLAQuHaVp4WzmaH/tH3emU9JWSa2SWru7u6faPTMzO4HpDAH9CfBKRHRHxABwL/BhoCEZEgJYDXQm2+3AGoDk9aXAgZEfGhHbIqI5IpobG7O3QIOZ2VyZTgC8BmyUtCAZy78IeB54EPg3SZstwI+T7e3Jc5LXfx4RXg3EzCwl0zkH8Cj5k7lPkp8CWgVsA74G/LWkNvJj/Lckb7kFWJ7U/xq4dhr9NjOzaVIp/xHe3Nwcra2taXfDzKysSHoiIppP1M4rgpiZZVRJHwFI6gZ+N42PWAHsn6HulBPvd7Z4v7NlIvt9ekSccBZNSQfAdElqnchhUKXxfmeL9ztbZnK/PQRkZpZRDgAzs4yq9ADYlnYHUuL9zhbvd7bM2H5X9DkAMzMbW6UfAZiZ2RgqMgAkbZL0oqQ2SRV9xbGkFkldkp4rqp0kaaekPcnPZWn2caZJWiPpQUm7Je2S9KWkXun7PV/SY5KeTvb775P6OkmPJvv9vyVV5Bqhkqol/UbST5LnWdnvVyU9K+kpSa1JbUb+rVdcAEiqBr5P/s6kZwJXSjoz3V7NqluBTSNq1wIPRMR64AEq77Ybg8BXI+K9wEbgmuS/caXv9zHg4xHxAeAcYJOkjeQXV/pest8HyS++VIm+BOwuep6V/Qb444g4p2j654z8W6+4AAA2AG0R8XJE9JNfnnJzyn2aNRHxMMffVbV48Z3iRXkqQkTsjYgnk+23yH8pNFH5+x0RcTh5Wps8grEXYaoYklYD/wr4X8lzkYH9HseM/FuvxAB4e+GZRPGiNFlxSkTshfyXJXByyv2ZNZLWAh8EHiUD+50MgzwFdAE7gd8y9iJMleS/AX8LDCfPl5ON/YZ8yP9M0hOStia1Gfm3Pq1F4UvUhBaesfInaRFwD/DliDiUhRVGI2IIOEdSA/Aj4L2jNZvbXs0uSZcBXRHxhKQLC+VRmlbUfhf5SER0SjoZ2CnphZn64Eo8Anh74ZlE8aI0WfGGpJUAyc+ulPsz4yTVkv/yvyMi7k3KFb/fBRHRAzxE/hzIWIswVYqPAJdLepX8kO7HyR8RVPp+AxARncnPLvKhv4EZ+rdeiQHwOLA+mSFQB1xBfjGaLClefKd4UZ6KkIz/3gLsjojvFr1U6fvdmPzlj6R68qvy7WbsRZgqQkRcFxGrI2It+f+ffx4Rn6HC9xtA0kJJiwvbwMXAc8zQv/WKvBBM0ifI/4VQDbRExI0pd2nWSLoTuJD8HQLfAK4H/hm4GziN/Mptn4qI45bfLFeSPgr8kvxCRIUx4a+TPw9Qyft9NvkTftXk/3i7OyJukHQG+b+MTwJ+A/x5RBxLr6ezJxkC+puIuCwL+53s44+SpzXADyPiRknLmYF/6xUZAGZmdmKVOARkZmYT4AAwM8soB4CZWUY5AMzMMsoBYGaWUQ4AM7OMcgCYmWWUA8DMLKP+PxZFVCcAEFTDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "diffs.sort()\n",
    "plt.plot(diffs[-50:])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_remove = [5416230, 5416233, 5399643, 5427633, 5758337, 5417410, 5759062, 5417420, 5410182, 5724116, 5717056, 5409164, 5410135, 5405325, 5381793]\n",
    "\n",
    "for _id in to_remove:\n",
    "    entities.pop(_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add useful data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "for entity in entities.values():\n",
    "    entity.update( { '€/kcal' : (entity['price_per_kg'] / 10.0) / entity['kcal'] if 'kcal' in entity and entity['kcal'] != 0.0 else 0.0,\n",
    "                     '€/g carb' : (entity['price_per_kg'] / 10.0) / entity['carbs'] if 'carbs' in entity and entity['carbs'] != 0.0 else 0.0,\n",
    "                     '€/g protein' : (entity['price_per_kg'] / 10.0) / entity['protein'] if 'protein' in entity and entity['protein'] != 0.0 else 0.0,\n",
    "                     '€/g fat' : (entity['price_per_kg'] / 10.0) / entity['fat'] if 'fat' in entity and entity['fat'] != 0.0 else 0.0} )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save to csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(entities)\n",
    "df = df.transpose()\n",
    "df.to_csv('csv/esselunga.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}