2_Complex_Queries_Basic_Visualization_py3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Complex Queries and Basic Visualization\n",
    "\n",
    "This notebook will cover how to make more complex queries, using both basic HTTP requests and using sharepa - the SHARE parsing and analysis library.\n",
    "\n",
    "We'll also go over aggregations, or queries that will return summary statistics about the whole dataset. We'll use those aggregations to make some simple data visualizations using pandas and matplotlib."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Here we'll define a helper function and specify the SHARE API url that we'll use for querying.\n",
    "\n",
    "We'll also define another helper function to nicely print out our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Json library parses JSON from strings or files. The library parses JSON into a Python dictionary or list. \n",
    "    # It can also convert Python dictionaries or lists into JSON strings. \n",
    "    # https://docs.python.org/2.7/library/json.html\n",
    "import json\n",
    "\n",
    "# Requests library allows you to send organic, grass-fed HTTP/1.1 requests, no need to manually add query strings \n",
    "    # to your URLs, or to form-encode your POST data. Docs: http://docs.python-requests.org/en/master/\n",
    "import requests\n",
    "\n",
    "# This takes the URL and puts it into a variable (so we only need to ever reference this variable, \n",
    "    # and so we don't have to repeat adding this URL when we want to work with the data)\n",
    "SHARE_API = 'https://staging-share.osf.io/api/search/abstractcreativework/_search'\n",
    "\n",
    "# A helper function that will use the requests library, pass along the correct headers, and make the query we want\n",
    "def query_share(url, query):\n",
    "    headers = {'Content-Type': 'application/json'}\n",
    "    data = json.dumps(query)\n",
    "    return requests.post(url, headers=headers, data=data, verify=False).json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A function that prints out the results in a numbered list\n",
    "def print_numbered_results(results):\n",
    "    print(\n",
    "        'There are {} total results and {} results on this page'.format(\n",
    "            results['hits']['total'],\n",
    "            len(results['hits']['hits'])\n",
    "        )\n",
    "    )\n",
    "    print('---------------')    \n",
    "    for result in enumerate(results['hits']['hits']):\n",
    "        print('{}. {}'.format(result[0] + 1, result[1]['_source']['title']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complex Queries\n",
    "\n",
    "### Pagination\n",
    "\n",
    "One request to the SHARE API will return just 10 results by default. To get more results, you can use the \"size\" parameter in your request, or paginate through the results you get back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 429 total results and 10 results on this page\n",
      "---------------\n",
      "1. Frogs mount speedy defence against pesticide threat\n",
      "2. Behavioural ecology: Fungus makes tree frogs sing\n",
      "3. Abundance of Green Tree Frogs and Insects in Artificial  Canopy Gaps in a Bottomland Hardwood Forest.\n",
      "4. Representative two-color flow cytometry analysis of splenocytes from uninfected control frogs (A), and frogs infected for 6 days (B)\n",
      "5. <p>Total annual biomass (g) of recaptured breeding wood frogs at 11 vernal pools in east-central Maine, USA.</p>\n",
      "6. Boiling Frogs and Burning Out\n",
      "7. Birds and Frogs: Selected Papers, 1990–2014\n",
      "8. Zoology: Venomous frogs headbutt foe\n",
      "9. Evolution: Wooing frogs are bat bait\n",
      "10. Marsupial frogs: Gastrotheca & allied genera\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\n",
      "  InsecureRequestWarning)\n"
     ]
    }
   ],
   "source": [
    "# We are setting up a query for items in the SHARE dataset that have the keyword \"frogs\"\n",
    "basic_query = {\n",
    "    \"query\": {\n",
    "        \"query_string\": {\n",
    "            \"query\": \"frogs\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "#this puts the results of querying SHARE_API with what we outlined in basic_query (frogs)\n",
    "query_results = query_share(SHARE_API, basic_query)\n",
    "\n",
    "#print out the numbered list of the results\n",
    "print_numbered_results(query_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get more results either by changing the number of results returned, or by paginating through the results.\n",
    "\n",
    "First, we'll return 20 results by specifying the size in our query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 429 total results and 20 results on this page\n",
      "---------------\n",
      "1. Frogs mount speedy defence against pesticide threat\n",
      "2. Behavioural ecology: Fungus makes tree frogs sing\n",
      "3. Abundance of Green Tree Frogs and Insects in Artificial  Canopy Gaps in a Bottomland Hardwood Forest.\n",
      "4. Representative two-color flow cytometry analysis of splenocytes from uninfected control frogs (A), and frogs infected for 6 days (B)\n",
      "5. <p>Total annual biomass (g) of recaptured breeding wood frogs at 11 vernal pools in east-central Maine, USA.</p>\n",
      "6. Boiling Frogs and Burning Out\n",
      "7. Birds and Frogs: Selected Papers, 1990–2014\n",
      "8. Zoology: Venomous frogs headbutt foe\n",
      "9. Evolution: Wooing frogs are bat bait\n",
      "10. Marsupial frogs: Gastrotheca & allied genera\n",
      "11. Representative two-color flow cytometry analysis of splenocytes from uninfected control frogs (A), and frogs infected for 6 days (B)\n",
      "12. Residues of Polybrominated Diphenyl Ethers in Frogs (<i>Rana limnocharis</i>) from a Contaminated Site, South China: Tissue Distribution, Biomagnification, and Maternal Transfer\n",
      "13. RAY CATARACT IN THE LENS OF THE EYE CAUSED BY WOUNDING AFTER X-IRRADIATION\n",
      "14. An Endogenous Bile Acid\n",
      "and Dietary Sucrose from Skin Secretions of Alkaloid-Sequestering\n",
      "Poison Frogs\n",
      "15. Polychlorinated biphenyls and chlorinated pesticides in southern Ontario, Canada, green frogs\n",
      "16. <p>Optional Endoreplication and Selective Elimination of Parental Genomes during Oogenesis in Diploid and Triploid Hybrid European Water Frogs</p>\n",
      "17. Abundance of green tree frogs and insects in artificial canopy gaps in a bottomland hardwood forest.\n",
      "18. <i>Frankixalus</i>, a New Rhacophorid Genus of Tree Hole Breeding Frogs with Oophagous Tadpoles\n",
      "19. Natural levels of abnormalities in the trilling frog (neobactrachus centralis) at the Olympic dam mine\n",
      "20. Extremely Abundant Antimicrobial Peptides Existed in the Skins of Nine Kinds of Chinese Odorous Frogs\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\n",
      "  InsecureRequestWarning)\n"
     ]
    }
   ],
   "source": [
    "# We are setting up a query for items in the SHARE dataset that have the keyword \"frogs\" but limiting the \n",
    "    # results to 20 items\n",
    "basic_query = {\n",
    "    \"query\": {\n",
    "        \"query_string\": {\n",
    "            \"query\": \"frogs\"\n",
    "        }\n",
    "    },\n",
    "    \"size\": 20\n",
    "}\n",
    "\n",
    "# this puts the results of querying SHARE_API with what we outlined in basic_query (frogs)\n",
    "query_results = query_share(SHARE_API, basic_query)\n",
    "\n",
    "# print out the numbered list of the results\n",
    "print_numbered_results(query_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also paginate through results by specifying the place to start in all of the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 429 total results and 10 results on this page\n",
      "---------------\n",
      "1. Frogs mount speedy defence against pesticide threat\n",
      "2. Behavioural ecology: Fungus makes tree frogs sing\n",
      "3. Abundance of Green Tree Frogs and Insects in Artificial  Canopy Gaps in a Bottomland Hardwood Forest.\n",
      "4. Representative two-color flow cytometry analysis of splenocytes from uninfected control frogs (A), and frogs infected for 6 days (B)\n",
      "5. <p>Total annual biomass (g) of recaptured breeding wood frogs at 11 vernal pools in east-central Maine, USA.</p>\n",
      "6. Boiling Frogs and Burning Out\n",
      "7. Birds and Frogs: Selected Papers, 1990–2014\n",
      "8. Zoology: Venomous frogs headbutt foe\n",
      "9. Evolution: Wooing frogs are bat bait\n",
      "10. Marsupial frogs: Gastrotheca & allied genera\n",
      "------------------------------------------\n",
      "*** Making another query for the next page ***\n",
      "*** These next titles will be different! ***\n",
      "------------------------------------------\n",
      "There are 429 total results and 10 results on this page\n",
      "---------------\n",
      "1. Representative two-color flow cytometry analysis of splenocytes from uninfected control frogs (A), and frogs infected for 6 days (B)\n",
      "2. Residues of Polybrominated Diphenyl Ethers in Frogs (<i>Rana limnocharis</i>) from a Contaminated Site, South China: Tissue Distribution, Biomagnification, and Maternal Transfer\n",
      "3. RAY CATARACT IN THE LENS OF THE EYE CAUSED BY WOUNDING AFTER X-IRRADIATION\n",
      "4. An Endogenous Bile Acid\n",
      "and Dietary Sucrose from Skin Secretions of Alkaloid-Sequestering\n",
      "Poison Frogs\n",
      "5. Polychlorinated biphenyls and chlorinated pesticides in southern Ontario, Canada, green frogs\n",
      "6. <p>Optional Endoreplication and Selective Elimination of Parental Genomes during Oogenesis in Diploid and Triploid Hybrid European Water Frogs</p>\n",
      "7. Abundance of green tree frogs and insects in artificial canopy gaps in a bottomland hardwood forest.\n",
      "8. <i>Frankixalus</i>, a New Rhacophorid Genus of Tree Hole Breeding Frogs with Oophagous Tadpoles\n",
      "9. Natural levels of abnormalities in the trilling frog (neobactrachus centralis) at the Olympic dam mine\n",
      "10. Extremely Abundant Antimicrobial Peptides Existed in the Skins of Nine Kinds of Chinese Odorous Frogs\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\n",
      "  InsecureRequestWarning)\n",
      "/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\n",
      "  InsecureRequestWarning)\n"
     ]
    }
   ],
   "source": [
    "# We are setting up a query for items in the SHARE dataset that have the keyword \"frogs\" \n",
    "basic_query = {\n",
    "    \"query\": {\n",
    "        \"query_string\": {\n",
    "            \"query\": \"frogs\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "# creates a list of the first 10 results from the search query \"frogs\"\n",
    "query_results = query_share(SHARE_API, basic_query)\n",
    "\n",
    "# print the results of the search we made in a numbered list\n",
    "print_numbered_results(query_results)\n",
    "\n",
    "# make it visually pretty and readable for us humans\n",
    "print('------------------------------------------')\n",
    "print('*** Making another query for the next page ***')\n",
    "print('*** These next titles will be different! ***')\n",
    "print('------------------------------------------')\n",
    "\n",
    "basic_query['from'] = 10  # Add the 'from' parameter to the query to pick up at the next page of results\n",
    "\n",
    "# creates a list of the next 10 results \n",
    "query_results = query_share(SHARE_API, basic_query)\n",
    "\n",
    "# print the results of the search we made in a numbered list\n",
    "print_numbered_results(query_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pagination with sharepa\n",
    "\n",
    "You can also use sharepa to paginate through all of the results in your query, and to access slices of your query at any time. The ShareSearch object returns a generator that you can use to access all results, using slices.\n",
    "\n",
    "First, we'll redefine our helper function for nicer printing with data returned from sharepa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# this is a function that helps us print the lists with pagination between results (10, next 10, etc.)\n",
    "def print_numbered_sharepa_results(search_obj):\n",
    "    results = search_obj.execute()\n",
    "    print(\n",
    "        'There are {} total results and {} results on this page'.format(\n",
    "            search_obj.count(),\n",
    "            len(results.hits)\n",
    "        )\n",
    "    )\n",
    "    print('---------------')    \n",
    "    for result in enumerate(results.hits):\n",
    "        print('{}. {}'.format(result[0] + 1, result[1]['title']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Sharepa is a python client for  browsing and analyzing SHARE data specifically using elasticsearch querying.\n",
    "    # We can use this to aggregate, graph, and analyze the data. \n",
    "    # Helpful Links:\n",
    "        # https://github.com/CenterForOpenScience/sharepa\n",
    "        # https://pypi.python.org/pypi/sharepa\n",
    "    # here, we import the specific function from Sharepa called ShareSearch and pretty_print\n",
    "from sharepa import ShareSearch\n",
    "from sharepa.helpers import pretty_print\n",
    "\n",
    "# we are creating a new local search!\n",
    "frogs_search = ShareSearch()\n",
    "\n",
    "# this sets up what we will actually search for -- keyword \"frogs\"\n",
    "frogs_search = frogs_search.query(\n",
    "    'query_string',\n",
    "    query='frogs'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1303 total results and 10 results on this page\n",
      "---------------\n",
      "1. Herpetology Collection - Royal Ontario Museum\n",
      "2. Herpetology Collection - Royal Ontario Museum\n",
      "3. On a collection of reptiles and frogs chiefly from Singapore.\n",
      "4. Effects of nonlethal predation and competition on life-history characteristics and behavior of larval Oregon spotted frog (Rana pretiosa) and larval red-legged frog (Rana aurora)\n",
      "5. Differences in foraging habits during the day and night time in the green frog, Rana clamitans.\n",
      "6. Calling activity of Eleutherodactylus frogs of Puerto Rico and habitat distribution of E. richmondi\n",
      "7. Evaluating group housing strategies for the ex-situ conservation of harlequin frogs (Atelopus spp.) using behavioral and physiological indicators\n",
      "8. Calling activity of Eleutherodactylus frogs of Puerto Rico and habitat distribution of E. richmondi\n",
      "9. <p>Total annual biomass (g) of recaptured breeding wood frogs at 11 vernal pools in east-central Maine, USA.</p>\n",
      "10. Correction: A Novel Reproductive Mode in Frogs: A New Species of Fanged Frog with Internal Fertilization and Birth of Tadpoles\n"
     ]
    }
   ],
   "source": [
    "# print the results of the search we made for \"frogs\" keyword in a numbered list\n",
    "print_numbered_sharepa_results(frogs_search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1303 total results and 10 results on this page\n",
      "---------------\n",
      "1. Fatigue and work capacity of muscles from frogs treated with male sex hormone\n",
      "2. The frog book; North American toads and frogs, with a study of the habits and life histories of those of the northeastern states. \n",
      "3. Condition-dependent reproductive effort in frogs infected by a widespread pathogen\n",
      "4. Physiologische studien über die Hemmungsmechanismen für die reflexthätigkeit des Rückenmarks im gehirne des Frosches / \n",
      "5. Overwintering of Columbia spotted frogs in northeastern Oregon\n",
      "6. Mycobacterium liflandii Outbreak in a Research Colony of Xenopus (Silurana) tropicalis Frogs\n",
      "7. <i>Frankixalus</i>, a New Rhacophorid Genus of Tree Hole Breeding Frogs with Oophagous Tadpoles\n",
      "8. Residues of Polybrominated Diphenyl Ethers in Frogs (<i>Rana limnocharis</i>) from a Contaminated Site, South China: Tissue Distribution, Biomagnification, and Maternal Transfer\n",
      "9. Cryoprotectants and Extreme Freeze Tolerance in a Subarctic Population of the Wood Frog\n",
      "10. THE EFFECT OF A LOW TEMPERATURE ON THE DEVELOPMENT OF RADIATION INJURY IN FROGS\n"
     ]
    }
   ],
   "source": [
    "# print the 10th - 20th results of the search we made for \"frogs\" keyword in a numbered list\n",
    "print_numbered_sharepa_results(frogs_search[10:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregations\n",
    "\n",
    "While searching for individual results is useful, sharepa also lets you make aggregation queries that give you results across the entirety of the SHARE dataset at once. This is useful if you're curious about the completeness of data sets.\n",
    "\n",
    "For example, we can find the number of documents per source that are missing tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this aggregates the number of documents that have no tags, per source, using query boolean (not tags) while also \n",
    "    # grabbing all sources in the aggregations: sources below\n",
    "missing_tags_aggregation = {\n",
    "    \"query\": {\n",
    "        \"bool\": {\n",
    "            \"must_not\": {\n",
    "                \"exists\": {\n",
    "                    \"field\": \"tags\"\n",
    "                  }\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    \"aggregations\": {\n",
    "        \"sources\": {\n",
    "            \"terms\": {\n",
    "                \"field\": \"sources\", # A field where the SHARE source is stored                \n",
    "                \"min_doc_count\": 0, \n",
    "                \"size\": 0  # Will return all sources, regardless if there are results\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# puts all the items without tags into a list\n",
    "results_without_tags = query_share(SHARE_API, missing_tags_aggregation)\n",
    "\n",
    "# counts the number of items without tags\n",
    "missing_tags_counts = results_without_tags['aggregations']['sources']['buckets']\n",
    "\n",
    "# this prints out the number of documents with missing tags, separated by sources\n",
    "for source in missing_tags_counts:\n",
    "    print('{} has {} documents without tags'.format(source['key'], source['doc_count'], ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This information isn't terribly useful if we don't also know how many documents are in each source.\n",
    "\n",
    "Let's get that information as well, along stats for what percentage of documents from each source are missing titles. \n",
    "\n",
    "We'll do this with an elasticsearch \"sigificant terms\" aggregation. We're only interested in results that have 1 document or more, meaning all documents from the other sources have titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# this does the same as above, but adds the percentage of documents missing tags from the total number of documents\n",
    "    # per source\n",
    "no_tags_query = {\n",
    "    \"query\": {\n",
    "        \"bool\": {\n",
    "            \"must_not\": {\n",
    "                \"exists\": {\n",
    "                    \"field\": \"tags\"\n",
    "                  }\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    \"aggs\": {\n",
    "        \"sources\":{\n",
    "            \"significant_terms\":{\n",
    "                \"field\": \"sources\", # A field where the SHARE source is stored                \n",
    "                \"min_doc_count\": 0, \n",
    "                \"size\": 0,  # Will return all sources, regardless if there are results\n",
    "                \"percentage\": {} # This will make the \"score\" parameter a percentage\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# creates a list of the documents with no tags\n",
    "docs_with_no_tags_results = query_share(SHARE_API, no_tags_query)\n",
    "\n",
    "#creates a dictionary that shows the results of the search along with the aggregations we outlined above\n",
    "docs_with_no_tags = docs_with_no_tags_results['aggregations']['sources']['buckets']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this prints out the number of documents with missing tags, separated by sources, with the percentage that makes up\n",
    "    # the total number of documents for each source\n",
    "for source in docs_with_no_tags:\n",
    "    print(\n",
    "        '{}% (or {}/{}) of documents from {} have no tags'.format(\n",
    "            format(source['score']*100, '.2f'),\n",
    "            source['doc_count'],\n",
    "            source['bg_count'],\n",
    "            source['key']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Aggregations with sharepa\n",
    "\n",
    "You can also use sharepa to do aggregations.\n",
    "\n",
    "Let's make a sharepa search object that will give us the number of documents per sourcethat don't have language specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# yay! creating another new search! \n",
    "no_language_search = ShareSearch()\n",
    "\n",
    "# this sets up our search query: all documents with no language field\n",
    "no_language_search = no_language_search.query(\n",
    "    'bool',\n",
    "    must_not={\"exists\": {\"field\": \"language\"}}\n",
    ")\n",
    "\n",
    "no_language_search.aggs.bucket(\n",
    "    'sources',  # Every aggregation needs a name\n",
    "    'significant_terms',  # There are many kinds of aggregations\n",
    "    field='sources',  # We store the source of a document in its type, so this will aggregate by source\n",
    "    min_doc_count=1,\n",
    "    percentage={},\n",
    "    size=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see which query is actually going to be sent to elasticsearch by printing out the query. This is very similar to the queries we were defining by hand up above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# print (prettily!) the items grabbed the search that have no language field\n",
    "pretty_print(no_language_search.to_dict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# here we grab the results of items that have no language field + their sources and significant terms\n",
    "aggregated_results = no_language_search.execute()\n",
    "\n",
    "# this prints out the percentage of items that don't have a language field\n",
    "for source in aggregated_results.aggregations['sources']['buckets']:\n",
    "    print(\n",
    "        '{}% of documents from {} do not have language'.format(\n",
    "            format(source['score']*100, '.2f'),\n",
    "            source['key'] \n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top tags \n",
    "\n",
    "Let's do an elasticsearch query to find out what the most used tags are used in the dataset across all sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    {\n",
      "        \"key\": \"journal\",\n",
      "        \"doc_count\": 143877\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"science\",\n",
      "        \"doc_count\": 95645\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"all\",\n",
      "        \"doc_count\": 91333\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"j\",\n",
      "        \"doc_count\": 82511\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"medicine\",\n",
      "        \"doc_count\": 68313\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"research\",\n",
      "        \"doc_count\": 53404\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"engineering\",\n",
      "        \"doc_count\": 49775\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"the\",\n",
      "        \"doc_count\": 47455\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"chemistry\",\n",
      "        \"doc_count\": 45017\n",
      "    },\n",
      "    {\n",
      "        \"key\": \"health\",\n",
      "        \"doc_count\": 44403\n",
      "    }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "# creating another new search!\n",
    "top_tag_search = ShareSearch()\n",
    "\n",
    "# this sets up our search query with the aggregations\n",
    "top_tag_search.aggs.bucket(\n",
    "    'tagsTermFilter',  # Every aggregation needs a name\n",
    "    'terms',  # There are many kinds of aggregations\n",
    "    field='tags',  # We store the source of a document in its type, so this will aggregate by source\n",
    "    min_doc_count=1,\n",
    "    exclude= \"of|and|or\",\n",
    "    size=10\n",
    ")\n",
    "\n",
    "# pretty_print(top_tag_search.to_dict())\n",
    "\n",
    "# this executes the search as we've outlined it above\n",
    "top_tag_results_executed = top_tag_search.execute()\n",
    "\n",
    "# this places the results of the search into this dictionary\n",
    "top_tag_results = top_tag_results_executed.aggregations.tagsTermFilter.to_dict()['buckets']\n",
    "\n",
    "# this prints out our search results (prettily)\n",
    "pretty_print(top_tag_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Plotting\n",
    "\n",
    "Sharepa has some basic functions to get you started making plots using matplotlib and pandas.\n",
    "\n",
    "### Creating a dataframe from sharepa data\n",
    "\n",
    "Raw sharepa data is in the same format as elasticsearch results, represented as a nested structure. To convert the data into a format that pandas can recognize, we have to convert it into a dataframe.\n",
    "\n",
    "Let's take our top tags aggregation, make it into a pandas data frame, and plot a bar graph. Then, we'll plot the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_count</th>\n",
       "      <th>key</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>143877</td>\n",
       "      <td>journal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>95645</td>\n",
       "      <td>science</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>91333</td>\n",
       "      <td>all</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>82511</td>\n",
       "      <td>j</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68313</td>\n",
       "      <td>medicine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>53404</td>\n",
       "      <td>research</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>49775</td>\n",
       "      <td>engineering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>47455</td>\n",
       "      <td>the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>45017</td>\n",
       "      <td>chemistry</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>44403</td>\n",
       "      <td>health</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   doc_count          key\n",
       "0     143877      journal\n",
       "1      95645      science\n",
       "2      91333          all\n",
       "3      82511            j\n",
       "4      68313     medicine\n",
       "5      53404     research\n",
       "6      49775  engineering\n",
       "7      47455          the\n",
       "8      45017    chemistry\n",
       "9      44403       health"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pandas is a python library that is used for data manipulation and analysis -- good for numbers + time series.\n",
    "    # Pandas gives us some extra data structures (arrays are data structures, for example) which is nice\n",
    "    # We are calling Pandas pd by using the \"as\" -- locally, we know Pandas as pd\n",
    "    # Helpful Links:\n",
    "        # https://en.wikipedia.org/wiki/Pandas_(software)\n",
    "        # http://pandas.pydata.org/ \n",
    "import pandas as pd\n",
    "\n",
    "# this transforms our results from the cell above into a dataframe\n",
    "top_tags_dataframe = pd.DataFrame(top_tag_results)\n",
    "\n",
    "#this prints out our dataframe -- looks like a nice table!\n",
    "top_tags_dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAFBCAYAAACcr2PXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu4VdV57/HvT5AYDRDQCgYFb+AlMSpR1Can2Q0V1Bg1\nSSUkaURD03Oijbn0pBV7GiA2Rq1pqemjbROCaLSoxEaiRtDobmvUqPECEYJYw1UgDSDRRA3ge/6Y\nY8Fiz33fa8012fv3eZ717LnGmpd37b32euccY8wxFBGYmZlV26vRAZiZWfk4OZiZWY6Tg5mZ5Tg5\nmJlZjpODmZnlODmYmVlOh8lB0mxJGyUtblH+OUnLJC2RdFVV+TRJK9JrE6rKx0paLOl5SbOqygdI\nmpe2eVTSyKrXpqT1l0u6oOdv18zMOqMzVw5zgInVBZKagA8Bx0XEccC1qfwYYBJwDHAmcL0kpc1u\nAKZGxBhgjKTKPqcCmyNiNDALuCbtawjwFeBk4BRguqTB3XyfZmbWBR0mh4h4GNjSovizwFURsT2t\n86tUfi4wLyK2R8RKYAUwTtJwYGBEPJHWuwk4r2qbuWl5PvCBtDwRWBQRWyPiZWARcEYX35+ZmXVD\nd9scxgB/IOkxSQ9Jek8qHwGsqVpvXSobAaytKl+bynbbJiJ2AFslDW1nX2ZmVmf9e7DdkIg4VdLJ\nwB3A4TWKSR2v0mIDyWOAmJl1Q0S0+p3b3SuHNcCdacdPADsk7U92dj+yar2DU9k64JBWyql+TVI/\nYFBEbG5nX62KiB49pk+f3uN91OJRhjjKEENZ4ihDDGWJowwxlCWOMsRQizja09nkIHY/o/8+qW1A\n0hhgQERsAhYAH0s9kA4DjgQej4gNZNVF41ID9QXAXWlfC4Apafl84MG0vBA4XdLg1Dh9eiozM7M6\n67BaSdKtQBOwv6TVwHTgO8AcSUuAN8i+7ImIpZJuB5YC24CLY1d6ugS4EdgHuDci7kvls4GbJa0A\nNgGT0762SLoCeBIIYGZkDdNmZlZnHSaHiPhEGy99qo31vw58vZXynwLHtVL+Bln319b2dSNZQqm7\npqamIg7ToTLEUYYYoBxxlCEGKEccZYgByhFHGWKA+sahjuqd9gSSoje8DzOzIkki2miQ7m5vJTOz\nnQ499FBWrVrV6DCsDaNGjWLlypVd2sZXDmbWY+kMtNFhWBva+vu0d+XggffMzCzHycHMzHKcHMzM\nLMfJwczMcpwczMwsx8nBzOpi+PBDkVS3x/Dhh3Y7tosuuoivfOUrtXuzvZCTg5nVxcaNq8hGvqnP\nI9u/zZw5kwsuqP1EmU4OZmaW4+RgZr3e008/zXve8x4GDx7M5MmTef3113e+9q1vfYvRo0dzwAEH\ncN5557F+/fqdrz333HNMmDCB/fffn4MOOoirrrqq3eO8+eabXHnllRx55JEMHjyYk08+mXXrspkG\nHnnkEcaNG8eQIUM45ZRTePTRR3dud9hhh/Hggw/ufD5z5kw+9als+LpVq1ax1157cdNNNzFq1CgO\nPPBArrzySgAWLlzIlVdeyW233cbAgQM58cQTe/7LSpwczKxX27ZtGx/+8IeZMmUKmzdv5vzzz+d7\n3/seAA899BCXX3458+fPZ/369YwcOZLJkycD8Oqrr3L66adz1llnsX79el544QXGjx/f7rG+8Y1v\ncNttt3HfffexdetWvvOd77DvvvuyZcsWzj77bL7whS+wadMmvvjFL/LBD36QLVtazsC8Sza7wS4/\n/vGPWbFiBQ888ABf/epXWb58ORMnTuTyyy/nYx/7GK+88gpPP/10D39buzg5mFmv9thjj7F9+3Yu\nvfRS+vXrx0c/+lFOPvlkAG655RamTp3K8ccfz957783Xv/51HnvsMVavXs3dd9/NQQcdxBe+8AUG\nDBjAfvvtt3O7tsyePZuvfe1rHHnkkQAcd9xxDBkyhHvuuYcxY8bwiU98gr322ovJkydz9NFH84Mf\n/KBT70ESM2bMYMCAAbz73e/m+OOP59lnn+3ZL6YDTg5m1qu99NJLjBix+/Tzo0aNIiJ46aWXGDVq\n1M7y/fbbj6FDh7Ju3TrWrFnDEUcc0aVjrVmzhsMPz8+Y3PI4lRgqVU6dMWzYsJ3L++67L6+++mqX\nYusqJwcz69UOOuig3Jfw6tWrkcSIESN2G630N7/5DZs2bWLEiBEccsgh/Pd//3eXjjVy5MhWt3nH\nO96RGxV19erVO5PWfvvtx29/+9udr23YsKHTx2xZ/VQrTg5mVhfDho1i1wzDtX9k++/YaaedRv/+\n/fnmN7/J9u3bufPOO3n88ccBmDx5MjfeeCOLFy/mjTfe4PLLL+fUU09l5MiRnH322WzYsIHrrruO\n3/3ud7z66qs7t2vL1KlT+Zu/+RteeOEFAJYsWcKWLVs466yzWLFiBfPmzWPHjh3cdtttLFu2jLPP\nPhuAE044gXnz5rF9+3aefPJJ5s+fv9t+2xvxdtiwYaxcubLmo+I6OZhZXWzYsLLbE9935rFhw8pO\nxbH33ntz5513MmfOHPbff3/uuOMOPvrRjwIwfvx4rrjiCj7ykY8wYsQIfvGLXzBv3jwA3va2t3H/\n/fezYMEChg8fzpgxY2hubm73WF/60peYNGkSEyZMYPDgwfzpn/4pr732GkOHDuXuu+/m2muv5YAD\nDuDaa6/lnnvuYejQoQBcccUVvPDCCwwdOpSZM2fyyU9+crf9trw6qH5+/vnnExHsv//+nHTSSZ36\nnXRGn5nPYfjwQ3t808ywYaM6/YE060s8n0O51WU+B0mzJW2UtLiV1/5C0puShlaVTZO0QtIySROq\nysdKWizpeUmzqsoHSJqXtnlU0siq16ak9ZdL6tEtgLW4W9N3ZJpZX9GZaqU5wMSWhZIOBk4HVlWV\nHQNMAo4BzgSu167rnxuAqRExBhgjqbLPqcDmiBgNzAKuSfsaAnwFOBk4BZguaXCX36GZWQ2dddZZ\nDBw4kEGDBjFo0KCdyx3dILen6XAO6Yh4WFJrLT//AHwZWFBVdi4wLyK2AyslrQDGSVoFDIyIJ9J6\nNwHnAQvTNtNT+Xzgm2l5IrAoIrYCSFoEnAHc1oX3Z2ZWU/fee2+jQyhEtxqkJZ0DrImIJS1eGgGs\nqXq+LpWNANZWla9NZbttExE7gK2pmqqtfZmZWZ11eOXQkqS3ApeTVSnVQ7c67c6YMWPnclNTE01N\nTTUKx8ysd2hubu6wx1VFl5MDcARwKPBsak84GHhK0jiys/uRVesenMrWAYe0Uk7Vay9J6gcMiojN\nktYBTS22eaitoKqTg5kVa9SoUXW7Gct6rnJ3dssT55kzZ7a5TWeTQ+XOEyLiZ8DwnS9IvwDGRsQW\nSQuAWyT9PVkV0JHA4xERkramBPIEcAFwXdrFAmAK8BPgfKAyNOFC4GupEXovsiuVyzoZr5kVqOXd\nv7bn6zA5SLqV7Ax+f0mrgekRMadqlWBX4lgq6XZgKbANuLjqBoRLgBuBfYB7I+K+VD4buDk1Xm8C\nJqd9bZF0BfBkOsbMiHi5B+/VzMw6qc/cBJdd8vb0vfpGHzPrPXp0E5yZmfU9Tg5mZpbj5GBmZjlO\nDmZmluPkYGZmOU4OZmaW4+RgZmY5Tg5mZpbj5GBmZjlODmZmluPkYGZmOU4OZmaW4+RgZmY5Tg5m\nZpbj5GBmZjlODmZmluPkYGZmOU4OZmaW4+RgZmY5HSYHSbMlbZS0uKrsGknLJD0j6XuSBlW9Nk3S\nivT6hKrysZIWS3pe0qyq8gGS5qVtHpU0suq1KWn95ZIuqM1bNjOzjnTmymEOMLFF2SLgnRFxArAC\nmAYg6VhgEnAMcCZwvaTK5NU3AFMjYgwwRlJln1OBzRExGpgFXJP2NQT4CnAycAowXdLgbr1LMzPr\nkg6TQ0Q8DGxpUfZARLyZnj4GHJyWzwHmRcT2iFhJljjGSRoODIyIJ9J6NwHnpeVzgblpeT7wgbQ8\nEVgUEVsj4mWyhHRGF9+fmZl1Qy3aHD4N3JuWRwBrql5bl8pGAGurytemst22iYgdwFZJQ9vZl5mZ\n1Vn/nmws6a+BbRHxbzWKB0Adr5I3Y8aMnctNTU00NTXVKBwzs96hubmZ5ubmTq3b7eQg6ULgLHZV\nA0F2dn9I1fODU1lb5dXbvCSpHzAoIjZLWgc0tdjmobbiqU4OZmaW1/LEeebMmW2u29lqJVF1Ri/p\nDODLwDkR8UbVeguAyakH0mHAkcDjEbGBrLpoXGqgvgC4q2qbKWn5fODBtLwQOF3S4NQ4fXoqMzOz\nOuvwykHSrWRn8PtLWg1MBy4HBgD3p85Ij0XExRGxVNLtwFJgG3BxRETa1SXAjcA+wL0RcV8qnw3c\nLGkFsAmYDBARWyRdATwJBDAzNUybmVmdadd3955LUnT0PrIk1tP3KnrD78vMDLLvxYhotZ3Xd0ib\nmVmOk4OZmeU4OZiZWY6Tg5mZ5Tg5mJlZjpODmZnlODmYmVmOk4OZmeU4OZiZWY6Tg5mZ5Tg5mJlZ\njpODmZnlODmYmVmOk4OZmeU4OZiZWY6Tg5mZ5Tg5mJlZjpODmZnlODmYmVlOh8lB0mxJGyUtriob\nImmRpOWSFkoaXPXaNEkrJC2TNKGqfKykxZKelzSrqnyApHlpm0cljax6bUpaf7mkC2rzls3MrCOd\nuXKYA0xsUXYZ8EBEHAU8CEwDkHQsMAk4BjgTuF5SZfLqG4CpETEGGCOpss+pwOaIGA3MAq5J+xoC\nfAU4GTgFmF6dhMzMrH46TA4R8TCwpUXxucDctDwXOC8tnwPMi4jtEbESWAGMkzQcGBgRT6T1bqra\npnpf84EPpOWJwKKI2BoRLwOLgDO68N7MzKybutvmcGBEbASIiA3Agal8BLCmar11qWwEsLaqfG0q\n222biNgBbJU0tJ19mZlZnfWv0X6iRvsBUMer5M2YMWPnclNTE01NTTUKx8ysd2hubqa5ublT63Y3\nOWyUNCwiNqYqo1+m8nXAIVXrHZzK2iqv3uYlSf2AQRGxWdI6oKnFNg+1FVB1cjAzs7yWJ84zZ85s\nc93OViuJ3c/oFwAXpuUpwF1V5ZNTD6TDgCOBx1PV01ZJ41ID9QUttpmSls8na+AGWAicLmlwapw+\nPZWZmVmddXjlIOlWsjP4/SWtBqYDVwF3SPo0sIqshxIRsVTS7cBSYBtwcURUqpwuAW4E9gHujYj7\nUvls4GZJK4BNwOS0ry2SrgCeJKu2mpkaps3MrM6067t7zyUpOnof2QVLT9+r6A2/LzMzyL4XI6LV\ndl7fIW1mZjlODgUaPvxQJPXoMXz4oY1+G2bWB7haqWtH6lG1UhliMDOrcLWSmZl1iZODmZnlODmY\nmVmOk4OZmeU4OfRB7jVlZh1xb6WuHalX9FYqSxxm1ljurWRmZl3i5GBmZjlODmZmluPkYGZmOU4O\nZmaW4+RgZmY5Tg5mZpbj5GBmZjlODmZmluPkYGZmOT1KDpK+KOlnkhZLukXSAElDJC2StFzSQkmD\nq9afJmmFpGWSJlSVj037eF7SrKryAZLmpW0elTSyJ/GamVnndDs5SHoH8DlgbES8G+gPfBy4DHgg\nIo4CHgSmpfWPBSYBxwBnAtcrG+QH4AZgakSMAcZImpjKpwKbI2I0MAu4prvxWrl48D+zcutptVI/\nYD9J/YG3AuuAc4G56fW5wHlp+RxgXkRsj4iVwApgnKThwMCIeCKtd1PVNtX7mg+M72G8VhIbN64i\nG/yv+49sH2ZWD91ODhHxEvANYDVZUtgaEQ8AwyJiY1pnA3Bg2mQEsKZqF+tS2QhgbVX52lS22zYR\nsQN4WdLQ7sZsZmad07+7G0p6O9mZ/ShgK3CHpE+SHwu6luM6tzq0LMCMGTN2Ljc1NdHU1FTDw5qZ\n7fmam5tpbm7u1LrdTg7AHwEvRsRmAEn/Dvw+sFHSsIjYmKqMfpnWXwccUrX9wamsrfLqbV6S1A8Y\nVDleS9XJwczM8lqeOM+cObPNdXvS5rAaOFXSPqlheTywFFgAXJjWmQLclZYXAJNTD6TDgCOBx1PV\n01ZJ49J+LmixzZS0fD5ZA7eZmdVZt68cIuJxSfOBp4Ft6ee/AgOB2yV9GlhF1kOJiFgq6XayBLIN\nuLhq+rZLgBuBfYB7I+K+VD4buFnSCmATMLm78ZqZWed5mtCuHcnThNYojjLEYNbXeZpQMzPrEicH\nMzPLcXIwM7McJwczM8txcjAzsxwnBzMzy3FyMDOzHCcHMzPLcXIwM7McJwczM8txcjAzsxwnB+vT\nejpdqacqtd7KA+917Ui9YrC5MsRRhhhqE4cH/7M9lwfeMzOzLnFyMDOzHCcHMzPLcXIwM7McJwcz\nM8txcjAzs5weJQdJgyXdIWmZpOcknSJpiKRFkpZLWihpcNX60yStSOtPqCofK2mxpOclzaoqHyBp\nXtrmUUkjexKvmZl1Tk+vHP4RuDcijgGOB34OXAY8EBFHAQ8C0wAkHQtMAo4BzgSuV9bJHOAGYGpE\njAHGSJqYyqcCmyNiNDALuKaH8ZqZWSd0OzlIGgT8r4iYAxAR2yNiK3AuMDetNhc4Ly2fA8xL660E\nVgDjJA0HBkbEE2m9m6q2qd7XfGB8d+M1M7PO68mVw2HAryTNkfSUpH+VtC8wLCI2AkTEBuDAtP4I\nYE3V9utS2QhgbVX52lS22zYRsQN4WdLQHsRsZmad0L+H244FLomIJyX9A1mVUsuxBGo5tkCrt3kD\nzJgxY+dyU1MTTU1NNTysmdmer7m5mebm5k6t2+2xlSQNAx6NiMPT8/eRJYcjgKaI2JiqjB6KiGMk\nXQZERFyd1r8PmA6sqqyTyicD74+Iz1bWiYifSOoHrI+IA1uJxWMr7WFxlCGG2sThsZVsz1WXsZVS\n1dEaSWNS0XjgOWABcGEqmwLclZYXAJNTD6TDgCOBx1PV01ZJ41ID9QUttpmSls8na+A2M7M660m1\nEsClwC2S9gZeBC4C+gG3S/o02VXBJICIWCrpdmApsA24uOp0/xLgRmAfst5P96Xy2cDNklYAm4DJ\nPYzXzMw6wUN2d+1IrkqpURxliKE2cbhayfZcHrLbzMy6xMnBzMxynBzMzCzHycHMzHKcHMwabPjw\nQ5HUo8fw4Yc2+m1YL+PeSl07knvo1CiOMsRQmzjKEENt4rC+x72VzMysS5wczMwsx8nBzMxynBzM\nDOh5w7gbxXsXN0h37UhuhK1RHGWIoTZxlCGGssThRvE9jRukzcysS5wczKw0fM9HebhaqWtHclVK\njeIoQwy1iaMMMZQljjLEUJs4+gpXK5mZWZc4OZiZWY6Tg5lZC+7W6+RgZpazceMqsraP7j2y7Xum\n0Y3zbpDu2pHcCFujOMoQQ23iKEMMZYmjDDGUJY4yxNBxHHVtkJa0l6SnJC1Iz4dIWiRpuaSFkgZX\nrTtN0gpJyyRNqCofK2mxpOclzaoqHyBpXtrmUUkjexqvmZl1rBbVSp8HllY9vwx4ICKOAh4EpgFI\nOhaYBBwDnAlcryw1AtwATI2IMcAYSRNT+VRgc0SMBmYB19QgXjMz60CPkoOkg4GzgG9XFZ8LzE3L\nc4Hz0vI5wLyI2B4RK4EVwDhJw4GBEfFEWu+mqm2q9zUfGN+TeM3MrHN6euXwD8CX2b1ibFhEbASI\niA3Agal8BLCmar11qWwEsLaqfG0q222biNgBvCxpaA9jNjOzDvTv7oaSPghsjIhnJDW1s2otW7xb\nbTgBmDFjxs7lpqYmmpqaanhYM7M9X3NzM83NzZ1at9u9lSRdCfwJsB14KzAQ+HfgJKApIjamKqOH\nIuIYSZcBERFXp+3vA6YDqyrrpPLJwPsj4rOVdSLiJ5L6Aesj4sAWobi30h4YRxliqE0cZYihLHGU\nIYayxFGGGDqOoy69lSLi8ogYGRGHA5OBByPiU8APgAvTalOAu9LyAmBy6oF0GHAk8HiqetoqaVxq\noL6gxTZT0vL5ZA3cZmZWZ92uVmrHVcDtkj5NdlUwCSAilkq6naxn0zbg4qrT/UuAG4F9gHsj4r5U\nPhu4WdIKYBNZEjIzszrzTXBdO5KrUmoURxliqE0cZYihLHGUIYayxFGGGDqOw6OymplZlzg5mJlZ\njpODmZnlODmYmVmOk4OZmeU4OZiZWY6Tg5mZ5Tg5mJlZjpODmZnlODmYmVmOk4OZmeU4OZiZWY6T\ng5mZ5Tg5mJlZjpODmZnlODmYmVmOk4OZmeU4OZiZWU63k4OkgyU9KOk5SUskXZrKh0haJGm5pIWS\nBldtM03SCknLJE2oKh8rabGk5yXNqiofIGle2uZRSSO7G6+ZmXVeT64ctgNfioh3AqcBl0g6GrgM\neCAijgIeBKYBSDoWmAQcA5wJXK9sklSAG4CpETEGGCNpYiqfCmyOiNHALOCaHsRrZmad1O3kEBEb\nIuKZtPwqsAw4GDgXmJtWmwucl5bPAeZFxPaIWAmsAMZJGg4MjIgn0no3VW1Tva/5wPjuxmtmZp1X\nkzYHSYcCJwCPAcMiYiNkCQQ4MK02AlhTtdm6VDYCWFtVvjaV7bZNROwAXpY0tBYxm5lZ23qcHCS9\njeys/vPpCiJarNLyeY8OV8N9mZlZG/r3ZGNJ/ckSw80RcVcq3ihpWERsTFVGv0zl64BDqjY/OJW1\nVV69zUuS+gGDImJza7HMmDFj53JTUxNNTU09eGdmZr1Pc3Mzzc3NnVpXEd0/sZd0E/CriPhSVdnV\nZI3IV0v6K2BIRFyWGqRvAU4hqy66HxgdESHpMeBS4AngHuC6iLhP0sXAuyLiYkmTgfMiYnIrcURH\n7yNr++7pRYzo4e+r4TGUJY4yxFCbOMoQQ1niKEMMZYmjDDF0HIckIqLVGpluJwdJ7wX+E1hC9g4C\nuBx4HLid7Ix/FTApIl5O20wj64G0jawaalEqfw9wI7APcG9EfD6VvwW4GTgR2ARMTo3ZLWNxctjD\n4ihDDLWJowwxlCWOMsRQljjKEEPHcdQlOZSJk8OeF0cZYqhNHGWIoSxxlCGGssRRhhg6jqO95OA7\npM3MLMfJwczMcpwczMwsx8nBzMxynBzMzCzHycHMzHKcHMzMLMfJwczMcpwczMwsx8nBzMxynBzM\nzCzHycHMzHKcHMzMLMfJwczMcpwczMwsx8nBzMxynBzMzCzHycHMzHKcHMzMLGePSA6SzpD0c0nP\nS/qrRsdjZtbblT45SNoL+CdgIvBO4OOSjq79kZprv8tuaW50AJQjBihHHM2NDiBpbnQAlCMGKEcc\nzY0OIGmu255LnxyAccCKiFgVEduAecC5tT9Mc+132S3NjQ6AcsQA5YijudEBJM2NDoByxADliKO5\n0QEkzXXb856QHEYAa6qer01lZmZWJ3tCcjAzs4IpIhodQ7sknQrMiIgz0vPLgIiIq6vWKfebMDMr\nqYhQa+V7QnLoBywHxgPrgceBj0fEsoYGZmbWi/VvdAAdiYgdkv4cWERWDTbbicHMrL5Kf+VgZmbF\nc4O0mZnlODlYw0l6OP18RdKvWzy2SvqFpIsbHadZX9LnqpUkLQFae9Mi6wX17oJDQtJbgZERsbzo\nY+8JJO0PPBIRRxV4zFHA6Ih4IP19+kfEK0Ud38pH0nERsaTRcRSlLyaHUe29HhGriooFQNKHgGuB\nARFxmKQTgK9GxDkFHf8V2k+Wg4qIoyOSDoqI9QUd6zPAnwFDI+IISaOBf46I8UUcP8XwXmAGMIqs\n40jl73F4UTGkOMa2UrwVWBUR2wuKYRhwJfCOiDhT0rHAaRExu4jjV8XxX8BbgBuBWyJia5HHTzEU\n9rnoc8mhbCT9FPgA0BwRJ6ayJRFxXGMj67skPUM2bMtPGvU3kfRz4IvAT4EdlfKI2FRUDCmOx4Cx\nwGKyL6J3Ac8Bg4HPRsSiAmL4ITAH+OuIOF5Sf+DpRvyPpBOFTwPnk3WrnxMR9xd4/MI+F322zUHS\nqZKekPSqpN9J2iHp1w0IZVsrZyCFZWxJQ9t7FBVHybwREb+rPElfRkWfRW2NiB9GxC8jYlPlUXAM\nAC8BJ0bESRHxHuBE4EXgdOCagmI4ICJuB94ESFcsO9rfpD4iYgXw/4C/At4PXJdGjP5IQSEU9rko\n/X0OdfRPwGTgDuAk4AJgTAPieE7SJ4B+6azkUuCRAo//U7Ivvuq7JCvPAyi0GqMk/kPS5cBbJZ0O\nXAz8oIgDV1XjPCTp74A7gTcqr0fEU0XEUWVMRDxXdfylko6OiBelVm+srYffpHangJ2jJjSiSufd\nwEXAB4H7gQ9FxFOS3gE8Sva3qtexC/9c9NlqJUlPRsRJkhZXGqElPV2pRigwjn2BvwYmpKKFwN9G\nxOtFxpFiGQqMBvaplEXEfxQdR6OlYeKnkv1NRPY3+XYU8M8i6aF2Xo6I+EC9Y6gm6TZgM9loyAAf\nAw4APgU8HBEnFxDDWOCbZFVaPwN+D/jjiFhc72O3iOM/gG8D8yPitRavfSoibq7jsQv/XPTl5PCf\nwB+R/bE3kA3NcWFEHN/QwBpE0p8CnwcOBp4BTiXrIVRYI6yVT+qpdTHwvlT0Y+B64HVg34h4taA4\n+gNHkSXr5Wn4/sKkYXxujohPFHncVuI4PCJe7KisFvpsmwPZmc9ewJ8DvwEOAT5adBCS7pf09qrn\nQyQtLDoOssRwMlkvlD8kq1su/NK9DCS9N/1dnpf0YrrPoub/fB3EcGUrn4u/LTIGgIh4LSK+EREf\nTo9rI+K3EfFmUYkhGQccT9Y4/nFJFxR4bCJiB3CIpAFFHrcV81spu6MeB+qzbQ5VXVZfB2Y2MJQD\nIuLlypOI2CLpwAbE8XpEvC4JSW+JiJ9LKuy+gpKZTSs9Qgp2ZkRcXnmSPhdnkTWGFqaVrpOVeApr\ni5J0M3AE2RVt5e8RwE1FxZD8AvixpAVkJ5RZIBF/X+8Dp9kv3wkMbtH4PYiqauBa6rPJoQwf+uRN\nSSMjYnWKaxTF94wBWJvOVL8P3C9pC1DoPR8lsjUiftjgGPqlJP0G7KzeeUsD4ihDojwJOLaINp8O\n/Hd67AUMTGVFxXQUcDbwduBDVeWvAJ+pxwH7bHKgHB96yBqjH06NXQL+F9kNWIWKiA+nxRmp8Wsw\ncF/RcZQg/BHoAAAM8klEQVREGXoK3QL8SNKc9PwiYG6Bx68oQ6L8GTCcrF2wkZZGxG5VOJLOL+LA\nEXEXcJek0yLi0SKO2ZcbpH8SEac0Og4ASQeQNQADPBYRv2pkPH1dGz1DGtFT6AyyThMA90dE4W1R\nkq4C+tGARCnpB2Rn5gOBE8huOquOoZBRBKrieSoixnZUVqdjf5N2rlIi4tJaH7MvXzmU4eyw4i1k\n3QX7A8dKIiL+swFxGJAa5Bsm9Yx5IMXR6Ku3ygnUSVVlQXZXf71dS3Y1fTVwXlV5pawQks4EzgJG\nSLqu6qVBQCFDiABPFnScnfpycmjkh34nSVeT9R1/jnQHaIrDyaFgkv4kIr4r6UutvV5Ew2M6zg5J\nb0oa3Ijxe1rE0rBEWbnHRtLeLe+3SW0wRXmJ7Mv5HLJq6IpXyKqm6y4iCq9S7JPJId3kdEO6Jb/R\nzgOOqjQ8WkPtl34ObHetYrwKLJF0P7v3jKl59UFrypAoJX2W7B6LwyVV3/A2kOx+i0JExLPAs5Ju\nrdxfIWkIcEhEbCkqjnTc3yMbuuNYdr9ZteYntX0yOUTEm5L+EihDcngR2Juqqi1rjIj4l/SzkV2b\nK+6kjsMxdEIZEuWtwA+BrwOXVZW/EhGbGxDP/ZLOIfve/CnwS0mPREQhVw/JLcBtZEN4/B9gCvA/\n9ThQX26Qvgr4FdkvuvrMrNAPnaTvkd3c8yN2b/so5AzR8iTNBT5fuf8knSV+IyI+3djIipXaPi6N\niH9odCxlUBleJ40mcEhETK8efqegGH4aEe9pMezPE/UYxqRPXjkkH0s/L6kqa8RAcwvSw8rj3a3c\nmFj0mFujyc6YW1YfFPb5TG0fHwecHDL9JR0ETCLrgt4IlWFD1kv6IFl7SF1GT+6zySEiDmt0DJA1\nNMkzwZXNXpKGVOqT04CERf+vzAGmk30x/yHZfQ6NGO7mx5L+ifwVdiN69TXaV8kGYXw4Ip6QdDiw\nouAY/lbSYOAvyAYjHESdGsX7crVSq2OzRESht+SrwTPBWV76bFxONmaNgD8GvlbPUTdbiaFSfbBz\nkqFKWVExpGOW4p4PK16fvXIgG2SuYh9gPPAUxY/XMoNsULFmgIh4Jp2RWINExE2SnmRXt+aPRMTS\ngsN4I/WqWyHpz4F1wNsKjqHh93yUgaS/jIhr2roRrcj2QUljgBuAYRHxLmVzTJwTETUflLHPJoeI\n+Fz18zSu0Lw2Vq+nbRGxVbtPnPJmWytb/UgaFBG/TtVIG8h6y1ReG1pwZ4XPA/uSTf50BVnV0pQC\njw+UZ/7mBluWfhZ+I1orvgV8Gaj0rFss6VbAyaGOfgM0oh2i0TPB2S63kg1uVpkdr6LwWfEi4gkA\nSW9GxEVFHbcVN5Lmb07Pnydrf+gzySEifpB+NmJsq5b2jYjHW5xM1uUu7T6bHKrGbYFs7JhjaMx9\nD58j+8d7A/g3sgavKxoQR58XEWennw3vrCDpNLIv4LcBIyUdD/zviLi44FAOiIjbJU2DbP5mSY0c\nqLJhJJ1E9r/aciTnwrqyAr+SdAS7pkz9Y+o0IGGfTQ5kjcAV28kmuVlbdBAR8VuyD1yjusZZol3z\n9Laq4B46s4CJpG7OEfGspD8o8PgVpZi/uSRuIavSWULjqn4vAf4VOFrSOrI5Jj5ZjwP12eQQEf+R\n6lMrDdOFdkmTNCsivtDiCqY6PvdWKt430s99yMbcepasSundZPXNpxUZTESsaVF90Igz9i+RJagj\nJP2YNH9zA+Iog/+JiEbfk7SOrJrvIbL7G35N1hb11VofqM8mB0mTgL8j6yUk4JuSvhwRrU3DVw+V\nbpHXtruWFabSM0fSncDYiFiSnr+LrFdZkdZI+n0gJO1N1kC9rINtai4inpL0fho4f3OJTJf0bfKj\nGRQ5zMldwMtkPStfqueB+vJ9Ds8Cp0fEL9Pz3yMbJvn4guPYD3gtIt5Mz/sBb0nVTdYAkp6LiHd2\nVFbnGA4A/pFsPgcBi8iG9NhUVAwpjn3Jrh5GRcRnUqeJoyLi7iLjKANJ3wWOpsUIykUOqyLpZxHx\nriKO1WevHIC9Kokh2URj7kD9EdkXQGWy9reSfRH8fgNisczidIb43fT8k8DidtavuTThU13qkrto\nDlnvrUqV2jqymwP7XHIATo6IRs+r/oik4ypXtfXUiC/DsrhP0kJJF0q6ELiXbATIou0TEZXEQFre\ntwFx2C4XkZ0dfj49lqaywki6RtIgSXtL+pGk/5H0J0XGkBwREdeQxvRJV7Rqf5Ne65F0n0fhJC1J\nw5a/D3hK0nJJi6vKa67PXjlExJclfQR4byr654j4fgNC+Y2ksZWeMKm73GsNiMOSiHhd0j8D9zZw\nvKsJEfGXkj4MrAQ+QjYB1Hfb3ar2fpfG/qr0VjqCvju8/KnAM5J+QfY7EFm1UhFdWc8u4Bi76XPJ\nQdLDEfE+Sa+QfeArZ0F/JulNsuk6/y4iri8opM8Dd0iqNC4dxK4RY60B0pj9fwcMABo13lXlf/OD\nwB2t3EVflOlkU5UeIukWspOpCxsRSAmc0agDR8Sqoo/Z55JDRLwv/Wx1EpPUp/sRoKjkcBhwIjCS\n7OzwFNqZSNwKMZ38eFdF3xh3t6Sfk11FfjZ1mHi94BiIiPslPUV21iyyRvFfFR1HGUTEKknvA0ZH\nxJz0Nyl8vKui9OU2h1al3iBNBR7ybyLi18DbycbPuZ5sYC1rnG2Rn7u50IQdEZeRdUo4KXUd/S1w\nbpExVNkH2ELWp/7YBt2M13CSppNN0TktFe1N8dV8helzVw6dERF1uR29DZUbmz4IfCsi7pFU80G0\nrEsaPt5V6kJ6MdkV5Z8B7yC716DQXkKSriar5tyt+yZZ+0df82Gyq/ynACLiJUllmG+8LpwcGm+d\npH8BTgeulvQWfEXXaNXjXd1KY8a7qnQhrXRpblQX0vPI7mvoq43Q1X4XESGp0ji/X0cb7Mn8JdR4\nk8i+fCamqSmHko3fYo1zbHr0J6tSORd4ouAYytKF9EWy6hOD29OJ3NslfQZ4gGwI7V7JVw4Nlv7p\n76x6vp46jbJonXYL8H+Bn9G4AdbK0oX0t2TdN1sOGVHYBDdlERHXSjqdrO3lKOArEXF/g8Oqmz47\nfIZZWyrdnRt4fAGfAqaSXcEsInUhjYjmgmNpdYKhksxtYHXk5GDWgqTxwMdp4ABrkpaQ9ZqrdCF9\nrK92IS2LdNPs1cCBZH+Tyk1wgxoaWJ24Wsks7yKyAdb2ZvceOkWOvvkUcHhE3FPgMXeSdHtETEpJ\nqrUh5Yuc4KYsrgE+FBGFj47bCL5yMGtB0vJGD7CWboA7ElhFNoVtkUM1IOmgiFgvaVRrrzfijt1G\nk/TjiHhvx2v2Dk4OZi1ImkM2hMrSBsbgL+WSSNVJAO8HhgPfp3HzORTGycGsBUnLgCPIpmAseoC1\nUqkag6zaVrKZ8f4iIl4sPqpipZMF2H0stopC53MoktsczPIaNsBaCc0C1pLdDChgMlnifAr4DsUO\nNdMQEXERgKS5ZGNLvZyeD2HX1LK9jq8czKxNkp5tOTuipGci4oTWXuvNJD0dESd2VNZb+A5pM2vP\nbyVNkrRXekxi1+iwfe3Mcq90tQCApKH04tqXXvvGzKwmPkk2l/X1ZMngMeBP0t3bf97IwBrgG8Cj\nku5Iz88HvtbAeOrK1UpmZp2Upgn9QHr6YCN7tNWbk4OZtSlNaPMZ4FCqahp6aw8d28XVSmbWnruA\n/yIbgXRHB+taL+IrBzNrU6VnUqPjsOK5t5KZteduSWc1Oggrnq8czKxN6Q7pfYHfkU081KtHIrVd\n3OZgZu0ZTNad9bCI+KqkkcBBDY7JCuArBzNrk6QbyIYt/0BEHJNuAlsUESc3ODSrM185mFl7TomI\nsZKeBoiILZIGNDooqz83SJtZe7ZJ6seuuax/j8bNq20FcnIws/ZcB/w7cKCkrwEPA1c2NiQrgtsc\nzKxdko4GxpP1VPpRX5kms69zcjAzsxxXK5mZWY6Tg5mZ5Tg5mJlZjpODWQ9JGiVpSaPjMKslJwez\n2nDPDutVnBzMakjS4ZKeknSypGsk/UTSM5I+k16fK+mcqvW/K+lDjYvYrHVODmY1ImkMMB+YApwA\nvBwRpwDjgD+TNAqYDVyU1h8EnAbc05iIzdrm5GBWGwcC3wc+HhFLgAnABWlMop8AQ4HREfGfwJGS\n9gc+DnwvIjwchZWOB94zq42twGrgD4DlZHcTfy4i7m9l3ZuATwGTgQuLCtCsK5wczGrjDeDDwCJJ\nrwILgYslPRQR2yWNBtZGxGvAXOBxYH1E/LxxIZu1zcnBrEYi4jVJZwOLgCuA54CnJAn4JXBeWu+X\nkpaRDWhnVkoeW8msYJL2BZ4FxkbEK42Ox6w1bpA2K5Ck8cBS4DonBiszXzmYmVmOrxzMzCzHycHM\nzHKcHMzMLMfJwczMcpwczMws5/8D1cGFhx9fFOQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45a76c7160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Matplot lib is a a python 2D plotting library which produces publication quality figures in a variety of hardcopy\n",
    "    # formats and interactive environments across platforms. \n",
    "    # Read more about Matplotlib here: http://matplotlib.org/\n",
    "from matplotlib import pyplot\n",
    "\n",
    "# this is used specifically with iPython notebooks to display the matplotlib chart or graph in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "#this takes our results dictionary from the cell above and plots them into a bar chart\n",
    "top_tags_dataframe.plot(kind='bar', x='key', y='doc_count')\n",
    "\n",
    "# this prints out the bar chart we just made!\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complex Queries and Dataframes\n",
    "\n",
    "Let's plot the number of document that each source has. We'll limit it to the top 30 sources to make sure that the graph is readable. Here we'll use the sharepa helper function bucket_to_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/ipykernel/__main__.py:30: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....)\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "Empty 'DataFrame': no numeric data to plot",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-7030efbea6a6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     31\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m \u001b[0;31m# this creates a bar chart\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 33\u001b[0;31m \u001b[0mall_results_frame_sorted\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m30\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkind\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'bar'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/pandas/tools/plotting.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, x, y, kind, ax, subplots, sharex, sharey, layout, figsize, use_index, title, grid, legend, style, logx, logy, loglog, xticks, yticks, xlim, ylim, rot, fontsize, colormap, table, yerr, xerr, secondary_y, sort_columns, **kwds)\u001b[0m\n\u001b[1;32m   3738\u001b[0m                           \u001b[0mfontsize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfontsize\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolormap\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcolormap\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtable\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3739\u001b[0m                           \u001b[0myerr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0myerr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxerr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mxerr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msecondary_y\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msecondary_y\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3740\u001b[0;31m                           sort_columns=sort_columns, **kwds)\n\u001b[0m\u001b[1;32m   3741\u001b[0m     \u001b[0m__call__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mplot_frame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3742\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/pandas/tools/plotting.py\u001b[0m in \u001b[0;36mplot_frame\u001b[0;34m(data, x, y, kind, ax, subplots, sharex, sharey, layout, figsize, use_index, title, grid, legend, style, logx, logy, loglog, xticks, yticks, xlim, ylim, rot, fontsize, colormap, table, yerr, xerr, secondary_y, sort_columns, **kwds)\u001b[0m\n\u001b[1;32m   2612\u001b[0m                  \u001b[0myerr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0myerr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxerr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mxerr\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2613\u001b[0m                  \u001b[0msecondary_y\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msecondary_y\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msort_columns\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msort_columns\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2614\u001b[0;31m                  **kwds)\n\u001b[0m\u001b[1;32m   2615\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2616\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/pandas/tools/plotting.py\u001b[0m in \u001b[0;36m_plot\u001b[0;34m(data, x, y, subplots, ax, kind, **kwds)\u001b[0m\n\u001b[1;32m   2439\u001b[0m         \u001b[0mplot_obj\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mklass\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msubplots\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msubplots\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0max\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0max\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkind\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkind\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2440\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2441\u001b[0;31m     \u001b[0mplot_obj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgenerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2442\u001b[0m     \u001b[0mplot_obj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2443\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mplot_obj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/pandas/tools/plotting.py\u001b[0m in \u001b[0;36mgenerate\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1024\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mgenerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1025\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_args_adjust\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1026\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_compute_plot_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1027\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_setup_subplots\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1028\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_plot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/vicky/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/pandas/tools/plotting.py\u001b[0m in \u001b[0;36m_compute_plot_data\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1133\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mis_empty\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1134\u001b[0m             raise TypeError('Empty {0!r}: no numeric data to '\n\u001b[0;32m-> 1135\u001b[0;31m                             'plot'.format(numeric_data.__class__.__name__))\n\u001b[0m\u001b[1;32m   1136\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1137\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumeric_data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: Empty 'DataFrame': no numeric data to plot"
     ]
    }
   ],
   "source": [
    "# from the SHAREPA library, we import this function to transform a bucket into a dataframe so we can plot it!\n",
    "from sharepa import bucket_to_dataframe\n",
    "\n",
    "# creating a new search! \n",
    "all_results = ShareSearch()\n",
    "\n",
    "# creating our search query!\n",
    "all_results = all_results.query(\n",
    "    'query_string', # Type of query, will accept a lucene query string\n",
    "    query='*', # This lucene query string will find all documents that don't have tags\n",
    "    analyze_wildcard=True  # This will make elasticsearch pay attention to the asterisk (which matches anything)\n",
    ")\n",
    "\n",
    "# Lucene supports fielded data. When performing a search you can either specify a field, or use the default field. \n",
    "all_results.aggs.bucket(\n",
    "    'sources',  # Every aggregation needs a name\n",
    "    'terms',  # There are many kinds of aggregations, terms is a pretty useful one though\n",
    "    field='sources',  # We store the source of a document in its type, so this will aggregate by source\n",
    "    size=0,  # These are just to make sure we get numbers for all the sources, to make it easier to combine graphs\n",
    "    min_doc_count=0\n",
    ")\n",
    "\n",
    "# this executes our search!\n",
    "all_results = all_results.execute()\n",
    "\n",
    "# this uses that function we imported above to transform our aggregated search into a dataframe so we can plot it!\n",
    "all_results_frame = bucket_to_dataframe('# documents by source', all_results.aggregations.sources.buckets)\n",
    "\n",
    "# this sorts the dataframe by the number of documents by source (descending order)\n",
    "all_results_frame_sorted = all_results_frame.sort(ascending=False,  columns='# documents by source')\n",
    "\n",
    "# this creates a bar chart that displays the first 30 results \n",
    "all_results_frame_sorted[:30].plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can choose different types of plots to generate. Here, we'll make a pie chart of the data sources with the top 10 most results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Creating a pie graph using the first 10 items in the data frame with no legend\n",
    "all_results_frame_sorted[:10].plot(kind='pie', y=\"# documents by source\", legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}