1. Compile Parliament members and their interests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Interests register\n",
    "In this notebook, we compile all companies that politicians have declared interests in. The politicians and their interest lists were obtained with scraper/*parliament.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "SAMPLING = False\n",
    "SAMPLE_SIZE = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "swiss_parliament = pd.read_json(\"data/parliament_members_interests.json\")\n",
    "# Short sample for testing\n",
    "if SAMPLING:\n",
    "    # We don't use pandas sample function because the code breaks\n",
    "    # if it samples only MPs with no interests.\n",
    "    swiss_parliament = swiss_parliament[3000:3459]\n",
    "# Unpack the interests with one row per interest\n",
    "interests_list = swiss_parliament.groupby(level=0).Interests.apply(\n",
    "    lambda x: pd.DataFrame(x.values[0])\n",
    ")\n",
    "\n",
    "# Join the one-row-per-interest table with the info from the original table\n",
    "interests_list.index = interests_list.index.droplevel(1)\n",
    "swiss_parliament = swiss_parliament.drop(\"Interests\", axis=1)\n",
    "\n",
    "# Replace NaNs with \"\" so split can work\n",
    "swiss_parliament_interests = swiss_parliament.join(interests_list, how='inner')[[\"FirstName\", \"LastName\", \"InterestName\",\"FunctionInAgencyText\"]].fillna(\"\")\n",
    "swiss_parliament_interests[\"InterestName\"] = swiss_parliament_interests[\"InterestName\"].apply(lambda x: x.split(\",\")[0])\n",
    "swiss_parliament_interests = swiss_parliament_interests.rename(\n",
    "    columns={\n",
    "        \"FirstName\": \"first_name\",\n",
    "        \"LastName\": \"last_name\",\n",
    "        \"FunctionInAgencyText\": \"position\",\n",
    "        \"InterestName\": \"interest_name\",\n",
    "    }\n",
    ")\n",
    "swiss_parliament_interests[\"data_source\"] = \"federal_parliament\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do the same with the Grand Council of Geneva."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#See the federal parliament import section for detailed explanations.\n",
    "geneva_parliament = pd.read_json(\"data/geneva_parliament_members_interests.json\")\n",
    "\n",
    "if SAMPLING:\n",
    "    geneva_parliament = geneva_parliament[495:499]\n",
    "\n",
    "interests_list = geneva_parliament.groupby(level=0).interests.apply(\n",
    "    lambda x: pd.DataFrame(x.values[0])\n",
    ")\n",
    "\n",
    "geneva_parliament = geneva_parliament.drop(\"interests\", axis=1)\n",
    "interests_list.index = interests_list.index.droplevel(1)\n",
    "geneva_interests = geneva_parliament.join(interests_list, how='inner')[[\"first_name\", \"last_name\", \"company\", \"position\"]].fillna(\"\")\n",
    "geneva_interests[\"data_source\"] = \"geneva_parliament\"\n",
    "geneva_interests = geneva_interests.rename(columns={\"company\": \"interest_name\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool! We now have a list of MPs with their interests, allowing us to lookup an MP from a company name and conversely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "all_politicians_interests = pd.concat([swiss_parliament_interests, geneva_interests])\n",
    "all_politicians_interests[all_politicians_interests.interest_name.str.contains(\"fondation ema\", case=False)]#.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's compile a list of unique interests. We will use it later when we will compile a list of board members of those companies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "all_interests = pd.Series(\n",
    "    all_politicians_interests.reset_index()[\"interest_name\"].unique()\n",
    ")\n",
    "all_interests = pd.DataFrame(\n",
    "    all_politicians_interests.groupby(\"interest_name\").size().rename('count')\n",
    ")\n",
    "all_interests.sort_values(by=\"count\", ascending=False)#.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "all_politicians_interests.reset_index().to_json('data/all_interests.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}