examples/variables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variable frequency comparison\n",
    "\n",
    "Imagine now that you are interested in looking at the relationship several of Seshat's data variables, to see which appear to be correlated in the number of polities that have them recorded as \"Present\".\n",
    "\n",
    "For example, the relationship between a transport infrastructure variable such as \"Road\" and a profession variable like \"Professional soldier\". Your hypothesis could be that the polities with professional soldiers need roads for them to move on, so there should be a strong correlation between the number of polities with soldiers and the number of polities with roads in any given year.\n",
    "\n",
    "We could compare this to some other variables that we believe less likely to be correlated.\n",
    "\n",
    "*In this notebook I decided to contibute back some of the functions for data transformation I started writing here into the package itself, for easy re-use. Other users of the API might not want to do this, but instead maintain their own set of data wrangling/transofrmation functions in a separate repository.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seshat_api import SeshatAPI, get_frequencies, get_variable_classes\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import pearsonr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = SeshatAPI(base_url=\"https://seshatdata.com/api\")\n",
    "# client = SeshatAPI(base_url=\"https://seshat-db.com/api\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the range of years to consider\n",
    "years = range(-1000, 1700)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a look at the available variables\n",
    "get_variable_classes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a list of the variable classes we are interested in\n",
    "class_names = ['Roads',\n",
    "               'ProfessionalSoldiers',\n",
    "               'Judges',\n",
    "               'Courts',\n",
    "               'Elephants']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `get_frequencies` function returns a pandas DataFrame with the frequency of each variable having a particular value across polities over time. In this case, we are dealing with absent/present variables, and we want to consider polities where each variable is counted as present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This could take a few moments to run\n",
    "example_frequency_df = get_frequencies(client, class_names, years, 'present')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's take a look at 5 random years\n",
    "example_frequency_df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting the data\n",
    "\n",
    "Let's first plot all the variables to see the frequency of \"Present\" being recorded across the selected year range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(13, 7))\n",
    "# plot each column from the DataFrame using a for loop\n",
    "for column in example_frequency_df.columns:\n",
    "    plt.plot(example_frequency_df.index, example_frequency_df[column], label=column)\n",
    "\n",
    "plt.xlabel('Year')\n",
    "plt.ylabel('Number of Polities')\n",
    "plt.title('Number of Polities with selected variables \"Present\" over time')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Soldiers and Roads\n",
    "\n",
    "Now let's explore the relationship between 2 specific variables.\n",
    "\n",
    "First, let's see if our hypothesis that the number of polities with roads should be correlated to the number of polities with professional soldiers. \n",
    "\n",
    "We'll create a scatter plot where each point represents a particular year, and the number of polities having each variable (recorded as \"Present\") on the x and y axes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def frequency_scatter(df, x_var, y_var):\n",
    "    corr_coef, _ = pearsonr(df[x_var], df[y_var])\n",
    "    plt.figure(figsize=(13, 7))\n",
    "    scatter = plt.scatter(\n",
    "        df[x_var], \n",
    "        df[y_var], \n",
    "        c=df.index,\n",
    "        cmap='viridis',\n",
    "    )\n",
    "    plt.xlabel(f'Number of polities with \"{x_var}\" present')\n",
    "    plt.ylabel(f'Number of polities with \"{y_var}\" present')\n",
    "    plt.title(f'Polities recorded as having {x_var} vs {y_var}: 1000 BCE - 2024 CE')\n",
    "    plt.colorbar(scatter, label='Year')\n",
    "    plt.text(0.05, 0.95, f'Correlation Coefficient: {corr_coef:.2f}', transform=plt.gca().transAxes, fontsize=12, verticalalignment='top')\n",
    "    plt.show()\n",
    "\n",
    "frequency_scatter(example_frequency_df, 'professional_soldier', 'road')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other variable combos\n",
    "\n",
    "Now let's make the same plot, but comparing two of the variables we pulled from the API that we expect to be strongly correlated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "frequency_scatter(example_frequency_df, 'judge', 'court')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at two variables we expect to not be correlated at all:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "frequency_scatter(example_frequency_df, 'court', 'elephant')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (seshat_api_1)",
   "language": "python",
   "name": "seshat_api_1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}