notebooks/Link_Extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracting hyperlinks from webpages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import org.archive.webservices.archivespark._\n",
    "import org.archive.webservices.archivespark.functions._\n",
    "import org.archive.webservices.archivespark.specific.warc._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the dataset\n",
    "\n",
    "In this example, the web archive dataset will be loaded from local WARC / CDX files (created in [this recipe](Downloading_WARC_from_Wayback.ipynb)). However, any other [Data Specification (DataSpec)](https://github.com/helgeho/ArchiveSpark/blob/master/docs/DataSpecs.md) could be used here too, in order to load your records of different types and from different local or remote sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "val warcPath = \"/data/helgeholzmann-de.warc.gz\"\n",
    "val cdxPath = warcPath + \"/*.cdx.gz\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "val records = ArchiveSpark.load(WarcSpec.fromFiles(cdxPath, warcPath))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering irrelevant records\n",
    "\n",
    "For link extraction, we are not interested in videos, images, stylesheets and any other files except for webpages ([mime type](https://en.wikipedia.org/wiki/Media_type) *text/html*), neither are we interested in webpages that were unavailable when they were crawled ([status code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) == 200). Hence, we will filter out those records.\n",
    "\n",
    "*It is important to note that this filtering is done only based on metadata, so up to this point ArchiveSpark does not even touch the actual web archive records, which is the core efficiency feature of ArchiveSpark.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "val pages = records.filter(r => r.mime == \"text/html\" && r.status == 200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the first record in our remaining dataset, we can see that this indeed is of type *text/html* and was *online* (status 200) at the time of crawl:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"record\" : {\n",
       "        \"redirectUrl\" : \"-\",\n",
       "        \"timestamp\" : \"20190528152652\",\n",
       "        \"digest\" : \"sha1:HCHVDRUSN7WDGNZFJES2Y4KZADQ6KINN\",\n",
       "        \"originalUrl\" : \"https://www.helgeholzmann.de/\",\n",
       "        \"surtUrl\" : \"de,helgeholzmann)/\",\n",
       "        \"mime\" : \"text/html\",\n",
       "        \"compressedSize\" : 2087,\n",
       "        \"meta\" : \"-\",\n",
       "        \"status\" : 200\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pages.peekJson"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enriching metadata\n",
    "\n",
    "This is the point when ArchiveSpark actually accesses the full records in order to enrich our metadata records with the desired information. To do so, we define the required [Enrichment Functions](https://github.com/helgeho/ArchiveSpark/blob/master/docs/EnrichFuncs.md) (`Links`, `LinkUrls`, `LinkTexts`) based on existing ones (`Html`, `SURT`, `HtmlAttribute`, `HtmlText`).\n",
    "\n",
    "`Html.all` extracts all hyperlinks / anchors (tag `a`) from the pages. This results in a list of multiple values, one for each link. From these we want to extract the link target (attribute `href`) of each link. This can be done by changing the dependency of the `HtmlAttribute` Enrichment Function using the `ofEach` operation ([see the docs for more details](https://github.com/helgeho/ArchiveSpark/blob/master/docs/Operations.md)) Although this again will result in multiple values, it is only one for each link, so we use the single dependency operation `of` to apply `SURT` on these and convert the URLs into the SURT format. Similarly, we apply `HtmlText` on each link to get the anchor text of the link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "val Links = Html.all(\"a\")\n",
    "val LinkUrls = SURT.of(HtmlAttribute(\"href\").ofEach(Links))\n",
    "val LinkTexts = HtmlText.ofEach(Links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To enrich the filtered records in our dataset with the link information, we call `enrich` for every Enrichment Function that we explicitely want to have in the dataset. As we are not interested in the raw `a` tags, we do not enrich it with `Links` here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "val pagesWithLinks = pages.enrich(LinkUrls).enrich(LinkTexts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A look at the first record shows what we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"record\" : {\n",
       "        \"redirectUrl\" : \"-\",\n",
       "        \"timestamp\" : \"20190528152652\",\n",
       "        \"digest\" : \"sha1:HCHVDRUSN7WDGNZFJES2Y4KZADQ6KINN\",\n",
       "        \"originalUrl\" : \"https://www.helgeholzmann.de/\",\n",
       "        \"surtUrl\" : \"de,helgeholzmann)/\",\n",
       "        \"mime\" : \"text/html\",\n",
       "        \"compressedSize\" : 2087,\n",
       "        \"meta\" : \"-\",\n",
       "        \"status\" : 200\n",
       "    },\n",
       "    \"payload\" : {\n",
       "        \"string\" : {\n",
       "            \"html\" : {\n",
       "                \"a\" : [\n",
       "                    {\n",
       "                        \"attributes\" : {\n",
       "                            \"href\" : {\n",
       "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
       "                            }\n",
       "                        },\n",
       "                        \"text\" : \"\"\n",
       "                    },\n",
       "                    {\n",
       "                        \"attrib..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pagesWithLinks.peekJson"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving the derived corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to save our derived corpus with the link information in this JSON format as shown above, we could simply call `.saveAsJson`. This preserves metadata of each records as well as the the lineage of each value by nexting derived values under their parents. JSON is a very universal format and can be read by many third-party tools to post-process this datset.\n",
    "\n",
    "*By adding a .gz extension to the output path, the data will be automatically compressed with GZip.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "pagesWithLinks.saveAsJson(\"/data/pages-with-links.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving plain links (src, timestamp, dst, text)\n",
    "\n",
    "Instead of this output, we can also transform the dataset into a temporal edgelist to keep the hyperlink information only, with source URL, timestamp of the capture, destination URL of each link as well as the anchor text if available.\n",
    "\n",
    "There are two preferred ways to achieve this with ArchiveSpark:\n",
    "\n",
    "1. We create a single value using the Enrichment Function `Values` that combines destination URL and text for each link. Then, in a `map` operation, can access these values and create our very own output format by adding additional information, like source and timestmap.\n",
    "2. We create a single value using the Enrichment Function `Values` for each link like before, but this time we include the source and timestamp in this value, so that we only need to [flat map the values (`flatMapValues`)](https://github.com/helgeho/ArchiveSpark/blob/master/docs/Operations.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Custom `map`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "val LinkRepresentation = Values(\"link-dst-text\", LinkUrls, LinkTexts).onEach(Links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*`println` is used in the following to show the complete record and don't have Jupyter cut if off*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"record\" : {\n",
      "        \"redirectUrl\" : \"-\",\n",
      "        \"timestamp\" : \"20190528152652\",\n",
      "        \"digest\" : \"sha1:HCHVDRUSN7WDGNZFJES2Y4KZADQ6KINN\",\n",
      "        \"originalUrl\" : \"https://www.helgeholzmann.de/\",\n",
      "        \"surtUrl\" : \"de,helgeholzmann)/\",\n",
      "        \"mime\" : \"text/html\",\n",
      "        \"compressedSize\" : 2087,\n",
      "        \"meta\" : \"-\",\n",
      "        \"status\" : 200\n",
      "    },\n",
      "    \"payload\" : {\n",
      "        \"string\" : {\n",
      "            \"html\" : {\n",
      "                \"a\" : [\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Home\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Home\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Research\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Research\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/publications\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Publications\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/publications\",\n",
      "                            \"Publications\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Private\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Private\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Projects\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Projects\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/contact\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Contact\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/contact\",\n",
      "                            \"Contact\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,l3s)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"L3S Research Center\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,l3s)/\",\n",
      "                            \"L3S Research Center\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,google)/maps/place/hannover\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Hannover, Germany\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,google)/maps/place/hannover\",\n",
      "                            \"Hannover, Germany\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"com,twitter)/search?q=@helgeho\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"@helgeho on Twitter\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"com,twitter)/search?q=@helgeho\",\n",
      "                            \"@helgeho on Twitter\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"com,github)/helgeho\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"helgeho on GitHub\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"com,github)/helgeho\",\n",
      "                            \"helgeho on GitHub\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"org,arxiv)/a/holzmann_h_1\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Helge on arXiv\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"org,arxiv)/a/holzmann_h_1\",\n",
      "                            \"Helge on arXiv\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Email me!\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Email me!\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/publications\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"My publications\",\n",
      "                        \"link-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/publications\",\n",
      "                            \"My publications\"\n",
      "                        ]\n",
      "                    }\n",
      "                ]\n",
      "            }\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "println(pagesWithLinks.enrich(LinkRepresentation).peekJson)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since `LinkRepresentation` is originally a single value function that's been applied to all links, we need to convert it to a multi-value pointer by calling `.multi` on it. Finally, we concatenate the link properties delimited by a tab (`\\t`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "val links = pagesWithLinks.enrich(LinkRepresentation).flatMap { record =>\n",
    "    record.valueOrElse(LinkRepresentation.multi, Seq.empty).map { case Seq(dst, text) =>\n",
    "        Seq(record.surtUrl, record.timestamp, dst, text).mkString(\"\\t\")\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the first 10 lines of this dataset to see what we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\t\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\tHome\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\tResearch\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/publications\tPublications\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\tPrivate\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\tProjects\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/contact\tContact\n",
      "de,helgeholzmann)/\t20190528152652\tde,l3s)/\tL3S Research Center\n",
      "de,helgeholzmann)/\t20190528152652\tde,google)/maps/place/hannover\tHannover, Germany\n",
      "de,helgeholzmann)/\t20190528152652\tcom,twitter)/search?q=@helgeho\t@helgeho on Twitter\n"
     ]
    }
   ],
   "source": [
    "links.take(10).foreach(println)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save as text file (GZip compressed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "links.saveAsTextFile(\"/data/links.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. `flatMapValues` ([ArchiveSpark Operations](https://github.com/helgeho/ArchiveSpark/blob/master/docs/Operations.md))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An enrichment function to derive values from the CDX metadata, we map from the pointer of `root`. This requires two additional modules to be imported, i.e., `pointers` and `cdx`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import org.archive.webservices.archivespark.model.pointers._\n",
    "import org.archive.webservices.sparkling.cdx._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "val SurtURL = FieldPointer.root[WarcRecord, CdxRecord].map(\"surtUrl\") { cdx: CdxRecord => cdx.surtUrl}\n",
    "val Timestamp = FieldPointer.root[WarcRecord, CdxRecord].map(\"timestamp\") { cdx: CdxRecord => cdx.timestamp}\n",
    "val LinkRepresentation = Values(\"src-timestamp-dst-text\", SurtURL, Timestamp, LinkUrls, LinkTexts).onEach(Links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*`println` is used in the following to show the complete record and don't have Jupyter cut if off*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"record\" : {\n",
      "        \"redirectUrl\" : \"-\",\n",
      "        \"timestamp\" : \"20190528152652\",\n",
      "        \"digest\" : \"sha1:HCHVDRUSN7WDGNZFJES2Y4KZADQ6KINN\",\n",
      "        \"originalUrl\" : \"https://www.helgeholzmann.de/\",\n",
      "        \"surtUrl\" : \"de,helgeholzmann)/\",\n",
      "        \"mime\" : \"text/html\",\n",
      "        \"compressedSize\" : 2087,\n",
      "        \"meta\" : \"-\",\n",
      "        \"status\" : 200\n",
      "    },\n",
      "    \"surtUrl\" : \"de,helgeholzmann)/\",\n",
      "    \"timestamp\" : \"20190528152652\",\n",
      "    \"payload\" : {\n",
      "        \"string\" : {\n",
      "            \"html\" : {\n",
      "                \"a\" : [\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Home\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Home\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Research\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Research\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/publications\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Publications\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/publications\",\n",
      "                            \"Publications\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Private\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Private\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Projects\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Projects\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/contact\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Contact\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/contact\",\n",
      "                            \"Contact\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,l3s)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"L3S Research Center\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,l3s)/\",\n",
      "                            \"L3S Research Center\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,google)/maps/place/hannover\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Hannover, Germany\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,google)/maps/place/hannover\",\n",
      "                            \"Hannover, Germany\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"com,twitter)/search?q=@helgeho\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"@helgeho on Twitter\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"com,twitter)/search?q=@helgeho\",\n",
      "                            \"@helgeho on Twitter\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"com,github)/helgeho\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"helgeho on GitHub\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"com,github)/helgeho\",\n",
      "                            \"helgeho on GitHub\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"org,arxiv)/a/holzmann_h_1\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Helge on arXiv\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"org,arxiv)/a/holzmann_h_1\",\n",
      "                            \"Helge on arXiv\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"Email me!\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"Email me!\"\n",
      "                        ]\n",
      "                    },\n",
      "                    {\n",
      "                        \"attributes\" : {\n",
      "                            \"href\" : {\n",
      "                                \"SURT\" : \"de,helgeholzmann)/publications\"\n",
      "                            }\n",
      "                        },\n",
      "                        \"text\" : \"My publications\",\n",
      "                        \"src-timestamp-dst-text\" : [\n",
      "                            \"de,helgeholzmann)/\",\n",
      "                            \"20190528152652\",\n",
      "                            \"de,helgeholzmann)/publications\",\n",
      "                            \"My publications\"\n",
      "                        ]\n",
      "                    }\n",
      "                ]\n",
      "            }\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "println(pagesWithLinks.enrich(SurtURL).enrich(Timestamp).enrich(LinkRepresentation).peekJson)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We concatenate the link properties delimited by a tab (`\\t`) values before saving them as text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "val links = pagesWithLinks.enrich(SurtURL).enrich(Timestamp).flatMapValues(LinkRepresentation).map(_.mkString(\"\\t\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\t\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\tHome\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\tResearch\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/publications\tPublications\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\tPrivate\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/\tProjects\n",
      "de,helgeholzmann)/\t20190528152652\tde,helgeholzmann)/contact\tContact\n",
      "de,helgeholzmann)/\t20190528152652\tde,l3s)/\tL3S Research Center\n",
      "de,helgeholzmann)/\t20190528152652\tde,google)/maps/place/hannover\tHannover, Germany\n",
      "de,helgeholzmann)/\t20190528152652\tcom,twitter)/search?q=@helgeho\t@helgeho on Twitter\n"
     ]
    }
   ],
   "source": [
    "links.take(10).foreach(println)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "links.saveAsTextFile(\"/data/links1.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graph Analysis\n",
    "\n",
    "In order to analyze the dataset as a graph, links are converted into edges of the form src -> dst:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "val srcDst = pages.enrich(LinkUrls).flatMap(r => r.valueOrElse(LinkUrls.multi, Seq.empty).map(dst => (r.surtUrl, dst)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For [Spark's GraphX](https://spark.apache.org/graphx/) the nodes (URLs) need to be assigned IDs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "val urlIdMap = srcDst.flatMap{case (src, dst) => Iterator(src, dst)}.distinct.zipWithUniqueId.collectAsMap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "val ids = sc.broadcast(urlIdMap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "val edges = srcDst.map{case (src, dst) => (ids.value(src), ids.value(dst))}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import org.apache.spark.graphx._\n",
    "val graph = Graph.fromEdgeTuples(edges, true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graph.numVertices"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ArchiveSpark",
   "language": "",
   "name": "archivespark"
  },
  "language_info": {
   "name": "scala",
   "version": "2.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}