5.analyze-stability/1.stability-within-z.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Determining Within Z Dimension Stability with SVCCA\n",
    "\n",
    "**Gregory Way 2018**\n",
    "\n",
    "Here, we apply Singular Vector Canonical Correlation Analysis ([Raghu et al. 2017](https://arxiv.org/abs/1706.05806 \"SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability\")) ([github](https://github.com/google/svcca)) to the neuron matrices Z to quantify model stability both within and across algorithms over each bottleneck dimensionality.\n",
    "\n",
    "Briefly, SVCCA uses Singular Value Decomposition (SVD) to extract the components explaining 99% of the variation.\n",
    "This is done to remove potential dimensions described by noise.\n",
    "Next, SVCCA performs a Canonical Correlation Analysis (CCA) on the SVD matrices to identify maximum correlations of linear combinations of both input matrices. The algorithm will identify the canonical correlations of highest magnitude across and within algorithms of the same dimensionality. \n",
    "\n",
    "The output of the SVCCA analysis is the SVCCA mean similarity score. This single number can be interpreted as a measure of similarity, or stability, in the solutions identified from each compression algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from scripts.util import read_in_matrices, get_svcca_across_algorithm_stability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets = ['TARGET', 'TCGA', 'GTEX']\n",
    "algorithms = ['pca', 'ica', 'nmf', 'dae', 'vae']\n",
    "z_dims = [2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 16, 18, 20, 25, 30,\n",
    "          35, 40, 45, 50, 60, 70, 80, 90, 100, 125, 150, 200]\n",
    "shuffled_data = (True, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating... dataset TARGET for shuffled dimension 2\n",
      "Calculating... dataset TARGET for signal dimension 2\n",
      "Calculating... dataset TARGET for shuffled dimension 3\n",
      "Calculating... dataset TARGET for signal dimension 3\n",
      "Calculating... dataset TARGET for shuffled dimension 4\n",
      "Calculating... dataset TARGET for signal dimension 4\n",
      "Calculating... dataset TARGET for shuffled dimension 5\n",
      "Calculating... dataset TARGET for signal dimension 5\n",
      "Calculating... dataset TARGET for shuffled dimension 6\n",
      "Calculating... dataset TARGET for signal dimension 6\n",
      "Calculating... dataset TARGET for shuffled dimension 7\n",
      "Calculating... dataset TARGET for signal dimension 7\n",
      "Calculating... dataset TARGET for shuffled dimension 8\n",
      "Calculating... dataset TARGET for signal dimension 8\n",
      "Calculating... dataset TARGET for shuffled dimension 9\n",
      "Calculating... dataset TARGET for signal dimension 9\n",
      "Calculating... dataset TARGET for shuffled dimension 10\n",
      "Calculating... dataset TARGET for signal dimension 10\n",
      "Calculating... dataset TARGET for shuffled dimension 12\n",
      "Calculating... dataset TARGET for signal dimension 12\n",
      "Calculating... dataset TARGET for shuffled dimension 14\n",
      "Calculating... dataset TARGET for signal dimension 14\n",
      "Calculating... dataset TARGET for shuffled dimension 16\n",
      "Calculating... dataset TARGET for signal dimension 16\n",
      "Calculating... dataset TARGET for shuffled dimension 18\n",
      "Calculating... dataset TARGET for signal dimension 18\n",
      "Calculating... dataset TARGET for shuffled dimension 20\n",
      "Calculating... dataset TARGET for signal dimension 20\n",
      "Calculating... dataset TARGET for shuffled dimension 25\n",
      "Calculating... dataset TARGET for signal dimension 25\n",
      "Calculating... dataset TARGET for shuffled dimension 30\n",
      "Calculating... dataset TARGET for signal dimension 30\n",
      "Calculating... dataset TARGET for shuffled dimension 35\n",
      "Calculating... dataset TARGET for signal dimension 35\n",
      "Calculating... dataset TARGET for shuffled dimension 40\n",
      "Calculating... dataset TARGET for signal dimension 40\n",
      "Calculating... dataset TARGET for shuffled dimension 45\n",
      "Calculating... dataset TARGET for signal dimension 45\n",
      "Calculating... dataset TARGET for shuffled dimension 50\n",
      "Calculating... dataset TARGET for signal dimension 50\n",
      "Calculating... dataset TARGET for shuffled dimension 60\n",
      "Calculating... dataset TARGET for signal dimension 60\n",
      "Calculating... dataset TARGET for shuffled dimension 70\n",
      "Calculating... dataset TARGET for signal dimension 70\n",
      "Calculating... dataset TARGET for shuffled dimension 80\n",
      "Calculating... dataset TARGET for signal dimension 80\n",
      "Calculating... dataset TARGET for shuffled dimension 90\n",
      "Calculating... dataset TARGET for signal dimension 90\n",
      "Calculating... dataset TARGET for shuffled dimension 100\n",
      "Calculating... dataset TARGET for signal dimension 100\n",
      "Calculating... dataset TARGET for shuffled dimension 125\n",
      "Calculating... dataset TARGET for signal dimension 125\n",
      "Calculating... dataset TARGET for shuffled dimension 150\n",
      "Calculating... dataset TARGET for signal dimension 150\n",
      "Calculating... dataset TARGET for shuffled dimension 200\n",
      "Calculating... dataset TARGET for signal dimension 200\n",
      "Calculating... dataset TCGA for shuffled dimension 2\n",
      "Calculating... dataset TCGA for signal dimension 2\n",
      "Calculating... dataset TCGA for shuffled dimension 3\n",
      "Calculating... dataset TCGA for signal dimension 3\n",
      "Calculating... dataset TCGA for shuffled dimension 4\n",
      "Calculating... dataset TCGA for signal dimension 4\n",
      "Calculating... dataset TCGA for shuffled dimension 5\n",
      "Calculating... dataset TCGA for signal dimension 5\n",
      "Calculating... dataset TCGA for shuffled dimension 6\n",
      "Calculating... dataset TCGA for signal dimension 6\n",
      "Calculating... dataset TCGA for shuffled dimension 7\n",
      "Calculating... dataset TCGA for signal dimension 7\n",
      "Calculating... dataset TCGA for shuffled dimension 8\n",
      "Calculating... dataset TCGA for signal dimension 8\n",
      "Calculating... dataset TCGA for shuffled dimension 9\n",
      "Calculating... dataset TCGA for signal dimension 9\n",
      "Calculating... dataset TCGA for shuffled dimension 10\n",
      "Calculating... dataset TCGA for signal dimension 10\n",
      "Calculating... dataset TCGA for shuffled dimension 12\n",
      "Calculating... dataset TCGA for signal dimension 12\n",
      "Calculating... dataset TCGA for shuffled dimension 14\n",
      "Calculating... dataset TCGA for signal dimension 14\n",
      "Calculating... dataset TCGA for shuffled dimension 16\n",
      "Calculating... dataset TCGA for signal dimension 16\n",
      "Calculating... dataset TCGA for shuffled dimension 18\n",
      "Calculating... dataset TCGA for signal dimension 18\n",
      "Calculating... dataset TCGA for shuffled dimension 20\n",
      "Calculating... dataset TCGA for signal dimension 20\n",
      "Calculating... dataset TCGA for shuffled dimension 25\n",
      "Calculating... dataset TCGA for signal dimension 25\n",
      "Calculating... dataset TCGA for shuffled dimension 30\n",
      "Calculating... dataset TCGA for signal dimension 30\n",
      "Calculating... dataset TCGA for shuffled dimension 35\n",
      "Calculating... dataset TCGA for signal dimension 35\n",
      "Calculating... dataset TCGA for shuffled dimension 40\n",
      "Calculating... dataset TCGA for signal dimension 40\n",
      "Calculating... dataset TCGA for shuffled dimension 45\n",
      "Calculating... dataset TCGA for signal dimension 45\n",
      "Calculating... dataset TCGA for shuffled dimension 50\n",
      "Calculating... dataset TCGA for signal dimension 50\n",
      "Calculating... dataset TCGA for shuffled dimension 60\n",
      "Calculating... dataset TCGA for signal dimension 60\n",
      "Calculating... dataset TCGA for shuffled dimension 70\n",
      "Calculating... dataset TCGA for signal dimension 70\n",
      "Calculating... dataset TCGA for shuffled dimension 80\n",
      "Calculating... dataset TCGA for signal dimension 80\n",
      "Calculating... dataset TCGA for shuffled dimension 90\n",
      "Calculating... dataset TCGA for signal dimension 90\n",
      "Calculating... dataset TCGA for shuffled dimension 100\n",
      "Calculating... dataset TCGA for signal dimension 100\n",
      "Calculating... dataset TCGA for shuffled dimension 125\n",
      "Calculating... dataset TCGA for signal dimension 125\n",
      "Calculating... dataset TCGA for shuffled dimension 150\n",
      "Calculating... dataset TCGA for signal dimension 150\n",
      "Calculating... dataset TCGA for shuffled dimension 200\n",
      "Calculating... dataset TCGA for signal dimension 200\n",
      "Calculating... dataset GTEX for shuffled dimension 2\n",
      "Calculating... dataset GTEX for signal dimension 2\n",
      "Calculating... dataset GTEX for shuffled dimension 3\n",
      "Calculating... dataset GTEX for signal dimension 3\n",
      "Calculating... dataset GTEX for shuffled dimension 4\n",
      "Calculating... dataset GTEX for signal dimension 4\n",
      "Calculating... dataset GTEX for shuffled dimension 5\n",
      "Calculating... dataset GTEX for signal dimension 5\n",
      "Calculating... dataset GTEX for shuffled dimension 6\n",
      "Calculating... dataset GTEX for signal dimension 6\n",
      "Calculating... dataset GTEX for shuffled dimension 7\n",
      "Calculating... dataset GTEX for signal dimension 7\n",
      "Calculating... dataset GTEX for shuffled dimension 8\n",
      "Calculating... dataset GTEX for signal dimension 8\n",
      "Calculating... dataset GTEX for shuffled dimension 9\n",
      "Calculating... dataset GTEX for signal dimension 9\n",
      "Calculating... dataset GTEX for shuffled dimension 10\n",
      "Calculating... dataset GTEX for signal dimension 10\n",
      "Calculating... dataset GTEX for shuffled dimension 12\n",
      "Calculating... dataset GTEX for signal dimension 12\n",
      "Calculating... dataset GTEX for shuffled dimension 14\n",
      "Calculating... dataset GTEX for signal dimension 14\n",
      "Calculating... dataset GTEX for shuffled dimension 16\n",
      "Calculating... dataset GTEX for signal dimension 16\n",
      "Calculating... dataset GTEX for shuffled dimension 18\n",
      "Calculating... dataset GTEX for signal dimension 18\n",
      "Calculating... dataset GTEX for shuffled dimension 20\n",
      "Calculating... dataset GTEX for signal dimension 20\n",
      "Calculating... dataset GTEX for shuffled dimension 25\n",
      "Calculating... dataset GTEX for signal dimension 25\n",
      "Calculating... dataset GTEX for shuffled dimension 30\n",
      "Calculating... dataset GTEX for signal dimension 30\n",
      "Calculating... dataset GTEX for shuffled dimension 35\n",
      "Calculating... dataset GTEX for signal dimension 35\n",
      "Calculating... dataset GTEX for shuffled dimension 40\n",
      "Calculating... dataset GTEX for signal dimension 40\n",
      "Calculating... dataset GTEX for shuffled dimension 45\n",
      "Calculating... dataset GTEX for signal dimension 45\n",
      "Calculating... dataset GTEX for shuffled dimension 50\n",
      "Calculating... dataset GTEX for signal dimension 50\n",
      "Calculating... dataset GTEX for shuffled dimension 60\n",
      "Calculating... dataset GTEX for signal dimension 60\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating... dataset GTEX for shuffled dimension 70\n",
      "Calculating... dataset GTEX for signal dimension 70\n",
      "Calculating... dataset GTEX for shuffled dimension 80\n",
      "Calculating... dataset GTEX for signal dimension 80\n",
      "Calculating... dataset GTEX for shuffled dimension 90\n",
      "Calculating... dataset GTEX for signal dimension 90\n",
      "Calculating... dataset GTEX for shuffled dimension 100\n",
      "Calculating... dataset GTEX for signal dimension 100\n",
      "Calculating... dataset GTEX for shuffled dimension 125\n",
      "Calculating... dataset GTEX for signal dimension 125\n",
      "Calculating... dataset GTEX for shuffled dimension 150\n",
      "Calculating... dataset GTEX for signal dimension 150\n",
      "Calculating... dataset GTEX for shuffled dimension 200\n",
      "Calculating... dataset GTEX for signal dimension 200\n"
     ]
    }
   ],
   "source": [
    "large_svcca_results_list = []\n",
    "for dataset in datasets:\n",
    "\n",
    "    for z in z_dims:\n",
    "\n",
    "        for signal in shuffled_data:\n",
    "\n",
    "            if signal:\n",
    "                shuffled_status = 'shuffled'\n",
    "            else:\n",
    "                shuffled_status = 'signal'\n",
    "\n",
    "            print(\"Calculating... dataset {} for {} dimension {}\"\n",
    "                  .format(dataset, shuffled_status, z))\n",
    "\n",
    "            # Read in the z matrix of interest\n",
    "            z_dict = read_in_matrices(\n",
    "                dataset=dataset,\n",
    "                z_dim=z,\n",
    "                algorithm='all',\n",
    "                shuffled_data=signal,\n",
    "                load_weights=True\n",
    "            )\n",
    "\n",
    "            # Perform across algorithm SVCCA\n",
    "            svcca_out_df = (\n",
    "                get_svcca_across_algorithm_stability(z_dict=z_dict,\n",
    "                                                     algorithms=algorithms)\n",
    "            )\n",
    "            \n",
    "            # Append info to the output dataframe\n",
    "            svcca_out_df = svcca_out_df.assign(\n",
    "                dataset=dataset,\n",
    "                z_dim=z,\n",
    "                shuffled=shuffled_status\n",
    "            )\n",
    "\n",
    "            large_svcca_results_list.append(svcca_out_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seed_1</th>\n",
       "      <th>seed_2</th>\n",
       "      <th>algorithm_1</th>\n",
       "      <th>algorithm_2</th>\n",
       "      <th>svcca_mean_similarity</th>\n",
       "      <th>dataset</th>\n",
       "      <th>z_dim</th>\n",
       "      <th>shuffled</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>165158</td>\n",
       "      <td>451283</td>\n",
       "      <td>pca</td>\n",
       "      <td>pca</td>\n",
       "      <td>0.021895</td>\n",
       "      <td>TARGET</td>\n",
       "      <td>2</td>\n",
       "      <td>shuffled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>165158</td>\n",
       "      <td>451283</td>\n",
       "      <td>pca</td>\n",
       "      <td>ica</td>\n",
       "      <td>0.018736</td>\n",
       "      <td>TARGET</td>\n",
       "      <td>2</td>\n",
       "      <td>shuffled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>165158</td>\n",
       "      <td>451283</td>\n",
       "      <td>pca</td>\n",
       "      <td>nmf</td>\n",
       "      <td>0.021642</td>\n",
       "      <td>TARGET</td>\n",
       "      <td>2</td>\n",
       "      <td>shuffled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>165158</td>\n",
       "      <td>451283</td>\n",
       "      <td>pca</td>\n",
       "      <td>dae</td>\n",
       "      <td>0.013379</td>\n",
       "      <td>TARGET</td>\n",
       "      <td>2</td>\n",
       "      <td>shuffled</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>165158</td>\n",
       "      <td>451283</td>\n",
       "      <td>pca</td>\n",
       "      <td>vae</td>\n",
       "      <td>0.013280</td>\n",
       "      <td>TARGET</td>\n",
       "      <td>2</td>\n",
       "      <td>shuffled</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   seed_1  seed_2 algorithm_1 algorithm_2  svcca_mean_similarity dataset  \\\n",
       "0  165158  451283         pca         pca               0.021895  TARGET   \n",
       "1  165158  451283         pca         ica               0.018736  TARGET   \n",
       "2  165158  451283         pca         nmf               0.021642  TARGET   \n",
       "3  165158  451283         pca         dae               0.013379  TARGET   \n",
       "4  165158  451283         pca         vae               0.013280  TARGET   \n",
       "\n",
       "   z_dim  shuffled  \n",
       "0      2  shuffled  \n",
       "1      2  shuffled  \n",
       "2      2  shuffled  \n",
       "3      2  shuffled  \n",
       "4      2  shuffled  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svcca_results_df = pd.concat(large_svcca_results_list)\n",
    "svcca_results_df.columns = ['seed_1', 'seed_2', 'algorithm_1', 'algorithm_2',\n",
    "                            'svcca_mean_similarity', 'dataset', 'z_dim', 'shuffled']\n",
    "svcca_results_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_file = os.path.join('results', 'svcca_within_mean_correlation_weights.tsv.gz')\n",
    "svcca_results_df.to_csv(out_file, sep='\\t', index=False, compression='gzip')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:interpret-compression]",
   "language": "python",
   "name": "conda-env-interpret-compression-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}