10 - ML Trial 4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3ec066bd",
   "metadata": {},
   "source": [
    "# BA prediction ML Trial 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caef988e",
   "metadata": {},
   "source": [
    "## Summary up to this point:\n",
    "### Previous:\n",
    "* __\"1D Feature Experiments:\"__\n",
    "  * The _\"1D Features\"_ refers to the original experiments based on quantitative (and typically, whole-molecule) calculations in RDKit for each drug... e.g., MolWt, LogP, NumHDonors, NumHAcceptors, etc. \n",
    "  * Of the many RDKit calculations available as features, we tested performance of linear regression model according to various feature sets -- the feature sets represented different subsets of the RDKit calculations\n",
    "    * feature set 1 (1a,1b) had all RdKit calculations, only differed in whether VSA was represented as absolute value of VSA _versus_ percent-VSA of total VSA.\n",
    "    * feature set 4 had the fewest number of calculations, focusing only on those reported in literature as being strongly tied to bioavailability; these are the metrics commonly used as filters in screening\n",
    "    * feature set 2 was a copy of feature set 1, but narrowed by running set 1 thru a cross-correlation matrix and removing variables with correlation > 0.95\n",
    "    * there were also ANOVA variants of the feature sets, which used an ANOVA test against a 3-category label (low BA, mid BA, high BA), and deleted features which did not meet statistical significance criteria (i.e., features for which no statistically significant difference existed in that variable for at least 1 of the categories)\n",
    "  * Regarding feature selection, __Feature set 1b__ was the best performing RDKit feature subset, followed most closely by 1b-ANOVA.\n",
    "  * Regarding labeling scheme, only _Continuous_ (regression) and _3-category_ (classification) were tested.\n",
    "    * for regression, the best r2 value (i.e. feature set 1b) was typically between 0.14-0.20 in Linear Regression models.\n",
    "    * for classification (3-category), accuracy was somewhere around 0.5 \n",
    "<br><br>\n",
    "* __\"Fragment experiments:\"__\n",
    "  * Hypothesis: \n",
    "    1. it is hypothesized that the model does not predict well __because it covers many drug types / classes, which are not significantly differentiated by the features__. \n",
    "    2. it is further hypothesized that the missing information about drug classes _could be_ tied to its 2D structure - or more specifically, tied to the presence/absence of key structural motifs / scaffolds in the chemical structure of various drug classes. _(For a simple example of a link, a drug with acidic/basic groups allow its preparation as a pharmaceutical salt, boosting bioavailability.)_\n",
    "    <br>\n",
    "  * Thus, it was decided to attempt a representation of drug classes and chemical structure using information about their 2D chemical structure - _specifically_, using an array of chemical fragments. \n",
    "    * Each drug was subjected to fragmentation, creating a fragment library.\n",
    "    * Each drug's Murcko Scaffold, and fragmentation products thereof, were added to the fragment library.\n",
    "    * Drugs were linked to the fragments, scaffolds, and scaffold fragments by explicit lookup between drug & fragment library, _but were also_ linked indirectly by searching for Substructure Matches across all drugs/fragments. <br><br>\n",
    "  * __Latest finding:__ \n",
    "    * The fragment set having most correlation with bioavailability was __`frags_all`__, which is the comprehensive set based on both fragment production _and_ matches of substructure search.  \n",
    "    * The NLP technique best representing the generated fragments was: (1) using __CountVectorizer__ on fragment array... (2) __TF-IDF__... (3) using __n-gram (n=2) + CountVectorizer__.... followed by (4) word2vec\n",
    "\n",
    "### Agenda for this NB:\n",
    "\n",
    "* Determine the best Regression Model for BA prediction based on fragments\n",
    "* Determine the best Classification Model for BA prediction based on fragments\n",
    "* Eventually:\n",
    "  * compare performance of fragments when used in conjunction with original 1D feature sets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8562eb4d",
   "metadata": {},
   "source": [
    "## UPDATED Agenda for this NB:\n",
    "\n",
    "* Initially, an attempt was made to combine the fragment NLP features w/ the RDKit \"1D\" features. The resulting r2 value was terrible (negative); this is because the numerical RDKIT features become rather useless once they've been joined into the large / high-dimensionality vector high of the NLP features.\n",
    "* __To resolve this problem,__ \n",
    "  * We will use NLP features and the Numerical RDKit calculations separately, instead of combining.\n",
    "  * Essentially, the new idea will be to: \n",
    "    - Run a Clasification Model using the NLP features to classify the BA of each drug (use cat labels)\n",
    "    - Run a Regression Model using numerical features _and predicted class_ to predict final bioavailability \n",
    "\n",
    "### <font color='red'> Actual Steps: </font>\n",
    "* <font color='green'>__Step 1.__</font> verify this approach is feasible using RFC w/ NLP features (__\"Model 1\"__) followed by RFR w/ Numerical features (__\"Model 2\"__) <br><br>\n",
    "* <font color='green'>__Step 2(a).__</font> if feasible (as per above), then determine the best model type for _Model 1_ (classification step) \n",
    "* <font color='green'>__Step 2(b).__</font> does the choice of categorical label affect which model appears best model for classification _Model 1_ ?  <br> _(e.g., label_cat0, label_cat1, label_cat2, label_cat3)_ <br><br>\n",
    "\n",
    "* <font color='green'>__Step 3.__</font> Run trials comparing the combination of Model 1 and Model 2. <br> __Variables to study include:__\n",
    "  * Classification as 1, Regression as 2 <br> _vs._ Regression as 1, Classification as 2 <br> _vs._ Regression as 1 and 2 <br> _vs._ Classification as 1 and 2 \n",
    "  * number of classes for Classification _(e.g., label_cat0, label_cat1, label_cat2, label_cat3)_\n",
    "  * Labels for Regression Model 2: predict absolute BA value..? or predict ___residual BA___ from model 1?\n",
    "  * Manual combination of Model 1/Model 2 predictions, <br> _versus_ direct use of Model 1 prediction in Model 2 features?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "effd5687",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22/09/08 16:35:20 WARN Utils: Your hostname, mitch-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)\n",
      "22/09/08 16:35:20 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22/09/08 16:35:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n"
     ]
    }
   ],
   "source": [
    "# MSM VM config prep\n",
    "import findspark\n",
    "findspark.init('/home/mitch/spark-3.3.0-bin-hadoop2')\n",
    "import pyspark\n",
    " \n",
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.appName('BApredsV4').getOrCreate()\n",
    "\n",
    "# --- suppress future spark warnings/error/etc output ---\n",
    "spark.sparkContext.setLogLevel(\"OFF\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "df2b1589",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_pickle(\"data_final_NEW_2022-09-05_aspandas.pkl\")\n",
    "pd.set_option('display.max_columns',None)\n",
    "data = data.rename(columns={'Name':'name','_c0':'index'})\n",
    "data = data.drop(columns='drug_name')\n",
    "data = spark.createDataFrame(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "42ae4ca7",
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- index: long (nullable = true)\n",
      " |-- name: string (nullable = true)\n",
      " |-- label_q0: double (nullable = true)\n",
      " |-- label_cat0: double (nullable = true)\n",
      " |-- label_cat1: double (nullable = true)\n",
      " |-- label_cat2: double (nullable = true)\n",
      " |-- label_cat3: double (nullable = true)\n",
      " |-- label_cat4: double (nullable = true)\n",
      " |-- smile: string (nullable = true)\n",
      " |-- MolWt: double (nullable = true)\n",
      " |-- ExactMolWt: double (nullable = true)\n",
      " |-- qed: double (nullable = true)\n",
      " |-- MolLogP: double (nullable = true)\n",
      " |-- MolMR: double (nullable = true)\n",
      " |-- VSA_total: double (nullable = true)\n",
      " |-- LabuteASA: double (nullable = true)\n",
      " |-- TPSA: double (nullable = true)\n",
      " |-- MaxPartialCharge: double (nullable = true)\n",
      " |-- MinPartialCharge: double (nullable = true)\n",
      " |-- MaxAbsPartialCharge: double (nullable = true)\n",
      " |-- MinAbsPartialCharge: double (nullable = true)\n",
      " |-- NumHAcceptors: double (nullable = true)\n",
      " |-- NumHDonors: double (nullable = true)\n",
      " |-- HeavyAtomCount: double (nullable = true)\n",
      " |-- NumHeteroatoms: double (nullable = true)\n",
      " |-- NumRotatableBonds: double (nullable = true)\n",
      " |-- NHOHCount: double (nullable = true)\n",
      " |-- NOCount: double (nullable = true)\n",
      " |-- FractionCSP3: double (nullable = true)\n",
      " |-- RingCount: double (nullable = true)\n",
      " |-- NumAliphaticRings: double (nullable = true)\n",
      " |-- NumAromaticRings: double (nullable = true)\n",
      " |-- NumAliphaticHeterocycles: double (nullable = true)\n",
      " |-- NumAromaticHeterocycles: double (nullable = true)\n",
      " |-- NumSaturatedHeterocycles: double (nullable = true)\n",
      " |-- NumSaturatedRings: double (nullable = true)\n",
      " |-- BalabanJ: double (nullable = true)\n",
      " |-- BertzCT: double (nullable = true)\n",
      " |-- HallKierAlpha: double (nullable = true)\n",
      " |-- fracVSA_PEOE01: double (nullable = true)\n",
      " |-- fracVSA_PEOE02: double (nullable = true)\n",
      " |-- fracVSA_PEOE03: double (nullable = true)\n",
      " |-- fracVSA_PEOE04: double (nullable = true)\n",
      " |-- fracVSA_PEOE05: double (nullable = true)\n",
      " |-- fracVSA_PEOE06: double (nullable = true)\n",
      " |-- fracVSA_PEOE07: double (nullable = true)\n",
      " |-- fracVSA_PEOE08: double (nullable = true)\n",
      " |-- fracVSA_PEOE09: double (nullable = true)\n",
      " |-- fracVSA_PEOE10: double (nullable = true)\n",
      " |-- fracVSA_PEOE11: double (nullable = true)\n",
      " |-- fracVSA_PEOE12: double (nullable = true)\n",
      " |-- fracVSA_PEOE13: double (nullable = true)\n",
      " |-- fracVSA_PEOE14: double (nullable = true)\n",
      " |-- fracVSA_SMR01: double (nullable = true)\n",
      " |-- fracVSA_SMR02: double (nullable = true)\n",
      " |-- fracVSA_SMR03: double (nullable = true)\n",
      " |-- fracVSA_SMR04: double (nullable = true)\n",
      " |-- fracVSA_SMR05: double (nullable = true)\n",
      " |-- fracVSA_SMR06: double (nullable = true)\n",
      " |-- fracVSA_SMR07: double (nullable = true)\n",
      " |-- fracVSA_SMR08: double (nullable = true)\n",
      " |-- fracVSA_SMR09: double (nullable = true)\n",
      " |-- fracVSA_SMR10: double (nullable = true)\n",
      " |-- fracVSA_SlogP01: double (nullable = true)\n",
      " |-- fracVSA_SlogP02: double (nullable = true)\n",
      " |-- fracVSA_SlogP03: double (nullable = true)\n",
      " |-- fracVSA_SlogP04: double (nullable = true)\n",
      " |-- fracVSA_SlogP05: double (nullable = true)\n",
      " |-- fracVSA_SlogP06: double (nullable = true)\n",
      " |-- fracVSA_SlogP07: double (nullable = true)\n",
      " |-- fracVSA_SlogP08: double (nullable = true)\n",
      " |-- fracVSA_SlogP09: double (nullable = true)\n",
      " |-- fracVSA_SlogP10: double (nullable = true)\n",
      " |-- fracVSA_SlogP11: double (nullable = true)\n",
      " |-- fracVSA_SlogP12: double (nullable = true)\n",
      " |-- FEAT_rdkit_1a: vector (nullable = true)\n",
      " |-- FEAT_rdkit_1b: vector (nullable = true)\n",
      " |-- FEAT_rdkit_2a: vector (nullable = true)\n",
      " |-- FEAT_rdkit_2b: vector (nullable = true)\n",
      " |-- FEAT_rdkit_3: vector (nullable = true)\n",
      " |-- FEAT_rdkit_4a: vector (nullable = true)\n",
      " |-- FEAT_rdkit_4b: vector (nullable = true)\n",
      " |-- FEAT_rdkit_1bANOVA: vector (nullable = true)\n",
      " |-- FEAT_rdkit_2bANOVA: vector (nullable = true)\n",
      " |-- frags_all: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- frags_better: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- frags_best: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- frags_efgs: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- frags_brics: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- frags_cv: vector (nullable = true)\n",
      " |-- frags_cv_idf: vector (nullable = true)\n",
      " |-- frags_tf: vector (nullable = true)\n",
      " |-- frags_tf_idf: vector (nullable = true)\n",
      " |-- frags_w2v: vector (nullable = true)\n",
      " |-- frags_n2g: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- frags_n2g_cv: vector (nullable = true)\n",
      " |-- frags_n2g_cv_idf: vector (nullable = true)\n",
      " |-- FEAT_frags_cv: vector (nullable = true)\n",
      " |-- FEAT_frags_cv_idf: vector (nullable = true)\n",
      " |-- FEAT_frags_tf: vector (nullable = true)\n",
      " |-- FEAT_frags_tf_idf: vector (nullable = true)\n",
      " |-- FEAT_frags_w2v: vector (nullable = true)\n",
      " |-- FEAT_frags_n2g_cv: vector (nullable = true)\n",
      " |-- FEAT_frags_n2g_cv_idf: vector (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "147e01c4",
   "metadata": {},
   "source": [
    "## List of column titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "181fda87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# all column titles in data\n",
    "columns_all = [\n",
    "    'index','name',\n",
    "    'label_q0','label_cat0','label_cat1','label_cat2','label_cat3','label_cat4',\n",
    "    'smile','MolWt','ExactMolWt','qed','MolLogP','MolMR','VSA_total','LabuteASA',\n",
    "    'TPSA','MaxPartialCharge','MinPartialCharge','MaxAbsPartialCharge','MinAbsPartialCharge','NumHAcceptors',\n",
    "    'NumHDonors','HeavyAtomCount','NumHeteroatoms','NumRotatableBonds','NHOHCount','NOCount','FractionCSP3',\n",
    "    'RingCount','NumAliphaticRings','NumAromaticRings','NumAliphaticHeterocycles','NumAromaticHeterocycles',\n",
    "    'NumSaturatedHeterocycles','NumSaturatedRings','BalabanJ','BertzCT','HallKierAlpha','fracVSA_PEOE01',\n",
    "    'fracVSA_PEOE02','fracVSA_PEOE03','fracVSA_PEOE04','fracVSA_PEOE05','fracVSA_PEOE06','fracVSA_PEOE07',\n",
    "    'fracVSA_PEOE08','fracVSA_PEOE09','fracVSA_PEOE10','fracVSA_PEOE11','fracVSA_PEOE12','fracVSA_PEOE13',\n",
    "    'fracVSA_PEOE14','fracVSA_SMR01','fracVSA_SMR02','fracVSA_SMR03','fracVSA_SMR04','fracVSA_SMR05',\n",
    "    'fracVSA_SMR06','fracVSA_SMR07','fracVSA_SMR08','fracVSA_SMR09','fracVSA_SMR10','fracVSA_SlogP01',\n",
    "    'fracVSA_SlogP02','fracVSA_SlogP03','fracVSA_SlogP04','fracVSA_SlogP05','fracVSA_SlogP06','fracVSA_SlogP07',\n",
    "    'fracVSA_SlogP08','fracVSA_SlogP09','fracVSA_SlogP10','fracVSA_SlogP11','fracVSA_SlogP12',\n",
    "    'FEAT_rdkit_1a','FEAT_rdkit_1b','FEAT_rdkit_2a','FEAT_rdkit_2b','FEAT_rdkit_3','FEAT_rdkit_4a',\n",
    "    'FEAT_rdkit_4b','FEAT_rdkit_1bANOVA','FEAT_rdkit_2bANOVA',\n",
    "    'frags_all','frags_better','frags_best','frags_efgs','frags_brics',\n",
    "    'frags_cv','frags_cv_idf','frags_tf','frags_tf_idf','frags_w2v','frags_n2g','frags_n2g_cv','frags_n2g_cv_idf',\n",
    "    'FEAT_frags_cv','FEAT_frags_cv_idf','FEAT_frags_tf','FEAT_frags_tf_idf',\n",
    "    'FEAT_frags_w2v','FEAT_frags_n2g_cv','FEAT_frags_n2g_cv_idf']\n",
    "\n",
    "# colun titles - all labels and feature vectors\n",
    "columns_labelsAndFeatures_all = [\n",
    "    'index','name',\n",
    "    'label_q0','label_cat0','label_cat1','label_cat2','label_cat3','label_cat4',\n",
    "    \n",
    "    'FEAT_rdkit_1a','FEAT_rdkit_1b','FEAT_rdkit_2a','FEAT_rdkit_2b','FEAT_rdkit_3','FEAT_rdkit_4a',\n",
    "    'FEAT_rdkit_4b','FEAT_rdkit_1bANOVA','FEAT_rdkit_2bANOVA',\n",
    "    \n",
    "    'FEAT_frags_cv','FEAT_frags_cv_idf','FEAT_frags_tf','FEAT_frags_tf_idf',\n",
    "    'FEAT_frags_w2v','FEAT_frags_n2g_cv','FEAT_frags_n2g_cv_idf']\n",
    "\n",
    "# column titles - labels + feature vectors (in order of performance)\n",
    "columns_labelsAndFeatures_subset = [\n",
    "    'index','name',\n",
    "    'label_q0','label_cat0','label_cat1','label_cat2','label_cat3','label_cat4',\n",
    "    \n",
    "    # rdkit features in order of efficacy\n",
    "    'FEAT_rdkit_1b','FEAT_rdkit_1bANOVA','FEAT_rdkit_2b','FEAT_rdkit_3',\n",
    "    \n",
    "    # fragment features in order of efficacy\n",
    "    'FEAT_frags_cv','FEAT_frags_cv_idf', \n",
    "    'FEAT_frags_tf','FEAT_frags_tf_idf',\n",
    "    'FEAT_frags_n2g_cv','FEAT_frags_n2g_cv_idf',\n",
    "    'FEAT_frags_w2v']\n",
    "\n",
    "columns_labelsAndFeatures = [\n",
    "    'index','name',\n",
    "    'label_q0','label_cat0','label_cat1','label_cat2','label_cat3','label_cat4',\n",
    "    'FEAT_rdkit_1b',\n",
    "    'FEAT_frags_cv']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe343da7",
   "metadata": {},
   "source": [
    "### Select data subset for use in training and testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "7dbd5d5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_final = data.select(columns_labelsAndFeatures_subset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "997c4e54",
   "metadata": {},
   "outputs": [],
   "source": [
    "(training,testing) = data_final.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aa04d0d",
   "metadata": {},
   "source": [
    "# Notebook Agenda, step 1:\n",
    "* <font color='green'>__Step 1.__</font> verify this approach is feasible using RFC w/ NLP features (__\"Model 1\"__) followed by RFR w/ Numerical features (__\"Model 2\"__) <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44e99801",
   "metadata": {},
   "source": [
    "* Note: labels are\n",
    "  *\t'label_q0' (Regression)\n",
    "  *\t'label_cat0' (5-class Classification)\n",
    "  *\t'label_cat1' (3-class Classification)\n",
    "  *\t'label_cat2' (4-class Classification)\n",
    "  *\t'label_cat3' (5-class Classification) \n",
    "  *\t'label_cat4' (5-class Classification)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "7762ff2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "(training,testing) = data_final.randomSplit([0.5,0.5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "964545f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RFC accuracy = 0.4889705882352941\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.classification import RandomForestClassifier\n",
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
    "\n",
    "featuresName = 'FEAT_frags_cv_idf'\n",
    "labelName = 'label_cat1' #'label_cat4'\n",
    "\n",
    "rfc = RandomForestClassifier(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction_1')\n",
    "mcEvaluator = MulticlassClassificationEvaluator(labelCol=labelName, predictionCol=\"prediction_1\", metricName=\"accuracy\")\n",
    "\n",
    "mymodel_rfc = rfc.fit(training)\n",
    "myresults_rfc = mymodel_rfc.transform(testing)\n",
    "\n",
    "model_eval_rfc = mcEvaluator.evaluate(myresults_rfc)\n",
    "\n",
    "print(f\"RFC accuracy = {model_eval_rfc}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "71254309",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RFR_1 r2 = 0.20825864238974512\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.regression import RandomForestRegressor\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "\n",
    "featuresName = 'FEAT_rdkit_1b'\n",
    "labelName = 'label_q0' \n",
    "\n",
    "rfr = RandomForestRegressor(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction_2')\n",
    "regEvaluator = RegressionEvaluator(labelCol=labelName,predictionCol='prediction_2',metricName='r2')\n",
    "\n",
    "mymodel_rfr1 = rfr.fit(training)\n",
    "myresults_rfr1 = mymodel_rfr1.transform(myresults_rfc)\n",
    "\n",
    "model_eval_rfr1 = regEvaluator.evaluate(myresults_rfr1)\n",
    "\n",
    "print(f\"RFR_1 r2 = {model_eval_rfr1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeb2edbe",
   "metadata": {},
   "source": [
    "* let's try training an alternative regression model to RFR1, that incorporates the rfc prediction into its features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "a3f3ce5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "\n",
    "assembler = VectorAssembler( inputCols=['prediction_1','FEAT_rdkit_1b'],outputCol='features_RFR2')\n",
    "\n",
    "combined_model_data = assembler.transform(myresults_rfc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "d7057ebc",
   "metadata": {},
   "outputs": [],
   "source": [
    "(combined_training,combined_testing) = combined_model_data.randomSplit([0.5,0.5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "351757c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RFR_2 r2 = 0.28112328420151345\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.regression import RandomForestRegressor\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "\n",
    "featuresName = 'features_RFR2'\n",
    "labelName = 'label_q0' \n",
    "\n",
    "rfr2 = RandomForestRegressor(featuresCol=featuresName,labelCol=labelName,predictionCol='combinedPrediction_1')\n",
    "regEvaluator2 = RegressionEvaluator(labelCol=labelName,predictionCol='combinedPrediction_1',metricName='r2')\n",
    "\n",
    "mymodel_rfr2 = rfr2.fit(combined_training)\n",
    "myresults_rfr2 = mymodel_rfr2.transform(combined_testing)\n",
    "\n",
    "model_eval_rfr2 = regEvaluator2.evaluate(myresults_rfr2)\n",
    "\n",
    "print(f\"RFR_2 r2 = {model_eval_rfr2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fc4d883",
   "metadata": {},
   "source": [
    "* let's try just training a separate model to analyze the first two independent predictions (rfc and rfr1) only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "3e8e7aa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "\n",
    "assembler = VectorAssembler( inputCols=['prediction_1','prediction_2'],outputCol='features_RFR3')\n",
    "combined_model_data = assembler.transform(myresults_rfr1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "744e5c45",
   "metadata": {},
   "outputs": [],
   "source": [
    "(combined_training,combined_testing) = combined_model_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "63a68d78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RFR_3 r2 = 0.20084863301003586\n"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.regression import RandomForestRegressor\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "\n",
    "featuresName = 'features_RFR3'\n",
    "labelName = 'label_q0' \n",
    "\n",
    "rfr3 = RandomForestRegressor(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction_3')\n",
    "regEvaluator3 = RegressionEvaluator(labelCol=labelName,predictionCol='prediction_3',metricName='r2')\n",
    "\n",
    "mymodel_rfr3 = rfr3.fit(combined_training)\n",
    "myresults_rfr3 = mymodel_rfr3.transform(combined_testing)\n",
    "\n",
    "model_eval_rfr3 = regEvaluator3.evaluate(myresults_rfr3)\n",
    "\n",
    "print(f\"RFR_3 r2 = {model_eval_rfr3}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc0f99a3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0557733c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "05344e38",
   "metadata": {},
   "source": [
    "# <font color='red'> Prior Scratch Work </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a88f9bd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "''' # create a history log for evaluation trials\n",
    "'''\n",
    "evaluation_history = {}\n",
    "\n",
    "import pickle\n",
    "with open('evaluation_history_temp.pickle', 'wb') as handle:\n",
    "    pickle.dump(evaluation_history, handle, protocol=pickle.HIGHEST_PROTOCOL)\n",
    "\n",
    "\n",
    "# Note: to load eval history, use:\n",
    "# with open('evaluation_history.pickle', 'rb') as handle:\n",
    "#    evaluation_history = pickle.load(handle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "81500e6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "(training,testing) = data_final.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "a3de6fc6",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "trial = 0\n",
    "trial_description = 'regression performance test: best NLP & RDkit features combined (frags_all CV with RDKit features_1b)'\n",
    "featuresName = 'features' \n",
    "labelName = 'label_q0'  \n",
    "\n",
    "'''# Import Regression Models\n",
    "'''\n",
    "from pyspark.ml.regression import (LinearRegression,\n",
    "                                   DecisionTreeRegressor,RandomForestRegressor,GBTRegressor,\n",
    "                                   GeneralizedLinearRegression,IsotonicRegression)\n",
    "\n",
    "lr = LinearRegression(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction')\n",
    "dtr = DecisionTreeRegressor(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction')\n",
    "rfr = RandomForestRegressor(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction')\n",
    "gbtr = GBTRegressor(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction')\n",
    "glr = GeneralizedLinearRegression(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction')\n",
    "ir = IsotonicRegression(featuresCol=featuresName,labelCol=labelName,predictionCol='prediction')\n",
    "\n",
    "models_regression = {'lr':lr,'dtr':dtr,'rfr':rfr,'gbtr':gbtr,'glr':glr,'ir':ir}\n",
    "\n",
    "\n",
    "''' # Create DataFrame for displaying eval metrics of current trial\n",
    "'''\n",
    "eval_df = pd.DataFrame() # already exists from prior run\n",
    "\n",
    "\n",
    "''' # Specify experiment type\n",
    "'''\n",
    "modeltype = lr  # --- SPECIFY!!! (lr,dtr,rfr,gbtr,glr,ir)\n",
    "modelname = 'lr'\n",
    "modelname_short = 'lr' # SPECIFY\n",
    "iteration = 0\n",
    "\n",
    "\n",
    "''' # Load eval History file for logging eval metrics of all trials\n",
    "'''\n",
    "import pickle\n",
    "with open('evaluation_history.pickle', 'rb') as handle:\n",
    "    evaluation_history = pickle.load(handle)\n",
    "\n",
    "trialName = f\"trial_{trial}\"\n",
    "iterationName = f\"iteration_{iteration}\"\n",
    "evaluation_history[trialName] = {}\n",
    "evaluation_history[trialName]['features_set'] = featuresName\n",
    "evaluation_history[trialName]['description'] = trial_description\n",
    "evaluation_history[trialName][modelname_short] = {}\n",
    "evaluation_history[trialName][modelname_short][iteration] = {}\n",
    "evaluation_history[trialName][modelname_short][iteration]['label'] = labelName\n",
    "\n",
    "\n",
    "''' # BEGIN TRIAL\n",
    "'''\n",
    "\n",
    "\n",
    "evaluation_history[trialName][modelname_short][iteration] = {}\n",
    "\n",
    "mymodel = modeltype.fit(training)\n",
    "myresults = mymodel.transform(testing)\n",
    "\n",
    "# CALCULATE KEY EVALS\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "regEvaluator = RegressionEvaluator(labelCol=labelName,predictionCol='prediction')\n",
    "\n",
    "evaluator = regEvaluator\n",
    "evalMetrics = {regEvaluator:['rmse','mse','mae','r2','var']}\n",
    "\n",
    "evaluation = []\n",
    "\n",
    "for each_metric in evalMetrics[evaluator]:        \n",
    "    metric = each_metric\n",
    "\n",
    "    result = evaluator.evaluate(myresults, {evaluator.metricName: metric})\n",
    "\n",
    "    evaluation.append((metric,result))\n",
    "\n",
    "    evaluation_history[trialName][modelname_short][iteration][metric] = result\n",
    "\n",
    "#r2_adj = mymodel.summary.r2adj\n",
    "#evaluation.append(('r2_adj(Training)',r2_adj))\n",
    "column0 = [x for x,y in evaluation]\n",
    "column1 = [y for x,y in evaluation]\n",
    "eval_df['metric'] = column0\n",
    "eval_df[modelname] = column1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "799d69d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>lr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rmse</td>\n",
       "      <td>62.343237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mse</td>\n",
       "      <td>3886.679154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mae</td>\n",
       "      <td>44.574759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>r2</td>\n",
       "      <td>-2.518276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>var</td>\n",
       "      <td>3521.569333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  metric           lr\n",
       "0   rmse    62.343237\n",
       "1    mse  3886.679154\n",
       "2    mae    44.574759\n",
       "3     r2    -2.518276\n",
       "4    var  3521.569333"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "15bfab22",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>lr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rmse</td>\n",
       "      <td>61.310566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mse</td>\n",
       "      <td>3758.985443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mae</td>\n",
       "      <td>46.397048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>r2</td>\n",
       "      <td>-2.205402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>var</td>\n",
       "      <td>3413.480928</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  metric           lr\n",
       "0   rmse    61.310566\n",
       "1    mse  3758.985443\n",
       "2    mae    46.397048\n",
       "3     r2    -2.205402\n",
       "4    var  3413.480928"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29be4b49",
   "metadata": {},
   "outputs": [],
   "source": [
    "''' # BACKUP EVALUATION HISTORY\n",
    "'''\n",
    "import pickle\n",
    "with open('evaluation_history.pickle', 'wb') as handle:\n",
    "    pickle.dump(evaluation_history, handle, protocol=pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "761a3557",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>rfr5_FEAT_frags_cv</th>\n",
       "      <th>rfr5_FEAT_frags_cv_idf</th>\n",
       "      <th>rfr5_FEAT_frags_tf</th>\n",
       "      <th>rfr5_FEAT_frags_tf_idf</th>\n",
       "      <th>rfr5_FEAT_frags_w2v</th>\n",
       "      <th>rfr5_FEAT_frags_n2g_cv</th>\n",
       "      <th>rfr5_FEAT_frags_n2g_cv_idf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rmse</td>\n",
       "      <td>31.357003</td>\n",
       "      <td>31.357003</td>\n",
       "      <td>31.452699</td>\n",
       "      <td>31.452699</td>\n",
       "      <td>32.463218</td>\n",
       "      <td>32.245083</td>\n",
       "      <td>32.245083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mse</td>\n",
       "      <td>983.261616</td>\n",
       "      <td>983.261616</td>\n",
       "      <td>989.272279</td>\n",
       "      <td>989.272279</td>\n",
       "      <td>1053.860549</td>\n",
       "      <td>1039.745390</td>\n",
       "      <td>1039.745390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mae</td>\n",
       "      <td>26.999519</td>\n",
       "      <td>26.999519</td>\n",
       "      <td>27.024890</td>\n",
       "      <td>27.024890</td>\n",
       "      <td>27.826697</td>\n",
       "      <td>27.886761</td>\n",
       "      <td>27.886761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.142043</td>\n",
       "      <td>0.142043</td>\n",
       "      <td>0.136798</td>\n",
       "      <td>0.136798</td>\n",
       "      <td>0.080441</td>\n",
       "      <td>0.092757</td>\n",
       "      <td>0.092757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>var</td>\n",
       "      <td>113.862026</td>\n",
       "      <td>113.862026</td>\n",
       "      <td>109.195099</td>\n",
       "      <td>109.195099</td>\n",
       "      <td>93.289556</td>\n",
       "      <td>71.799789</td>\n",
       "      <td>71.799789</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  metric  rfr5_FEAT_frags_cv  rfr5_FEAT_frags_cv_idf  rfr5_FEAT_frags_tf  \\\n",
       "0   rmse           31.357003               31.357003           31.452699   \n",
       "1    mse          983.261616              983.261616          989.272279   \n",
       "2    mae           26.999519               26.999519           27.024890   \n",
       "3     r2            0.142043                0.142043            0.136798   \n",
       "4    var          113.862026              113.862026          109.195099   \n",
       "\n",
       "   rfr5_FEAT_frags_tf_idf  rfr5_FEAT_frags_w2v  rfr5_FEAT_frags_n2g_cv  \\\n",
       "0               31.452699            32.463218               32.245083   \n",
       "1              989.272279          1053.860549             1039.745390   \n",
       "2               27.024890            27.826697               27.886761   \n",
       "3                0.136798             0.080441                0.092757   \n",
       "4              109.195099            93.289556               71.799789   \n",
       "\n",
       "   rfr5_FEAT_frags_n2g_cv_idf  \n",
       "0                   32.245083  \n",
       "1                 1039.745390  \n",
       "2                   27.886761  \n",
       "3                    0.092757  \n",
       "4                   71.799789  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# iteration 5\n",
    "pd.set_option('display.max_columns',None)\n",
    "eval_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5d5692fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>rfr4_all_cv</th>\n",
       "      <th>rfr4_all_cv_idf</th>\n",
       "      <th>rfr4_all_w2v</th>\n",
       "      <th>rfr4_all_n2g_cv</th>\n",
       "      <th>rfr4_all_n2g_cv_idf</th>\n",
       "      <th>rfr4_better_cv</th>\n",
       "      <th>rfr4_better_cv_idf</th>\n",
       "      <th>rfr4_better_w2v</th>\n",
       "      <th>rfr4_better_n2g_cv</th>\n",
       "      <th>rfr4_better_n2g_cv_idf</th>\n",
       "      <th>rfr4_best_cv</th>\n",
       "      <th>rfr4_best_cv_idf</th>\n",
       "      <th>rfr4_best_w2v</th>\n",
       "      <th>rfr4_best_n2g_cv</th>\n",
       "      <th>rfr4_best_n2g_cv_idf</th>\n",
       "      <th>rfr4_efgs_cv</th>\n",
       "      <th>rfr4_efgs_cv_idf</th>\n",
       "      <th>rfr4_efgs_w2v</th>\n",
       "      <th>rfr4_efgs_n2g_cv</th>\n",
       "      <th>rfr4_efgs_n2g_cv_idf</th>\n",
       "      <th>rfr4_brics_cv</th>\n",
       "      <th>rfr4_brics_cv_idf</th>\n",
       "      <th>rfr4_brics_w2v</th>\n",
       "      <th>rfr4_brics_n2g_cv</th>\n",
       "      <th>rfr4_brics_n2g_cv_idf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rmse</td>\n",
       "      <td>30.175980</td>\n",
       "      <td>30.175980</td>\n",
       "      <td>31.409608</td>\n",
       "      <td>30.949282</td>\n",
       "      <td>30.949282</td>\n",
       "      <td>30.280202</td>\n",
       "      <td>30.280202</td>\n",
       "      <td>31.080018</td>\n",
       "      <td>30.751594</td>\n",
       "      <td>30.751594</td>\n",
       "      <td>32.942816</td>\n",
       "      <td>32.942816</td>\n",
       "      <td>32.942816</td>\n",
       "      <td>32.942816</td>\n",
       "      <td>32.942816</td>\n",
       "      <td>30.937496</td>\n",
       "      <td>30.937496</td>\n",
       "      <td>31.273240</td>\n",
       "      <td>31.722631</td>\n",
       "      <td>31.722631</td>\n",
       "      <td>31.940796</td>\n",
       "      <td>31.940796</td>\n",
       "      <td>32.455095</td>\n",
       "      <td>32.442060</td>\n",
       "      <td>32.442060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mse</td>\n",
       "      <td>910.589795</td>\n",
       "      <td>910.589795</td>\n",
       "      <td>986.563479</td>\n",
       "      <td>957.858080</td>\n",
       "      <td>957.858080</td>\n",
       "      <td>916.890642</td>\n",
       "      <td>916.890642</td>\n",
       "      <td>965.967548</td>\n",
       "      <td>945.660530</td>\n",
       "      <td>945.660530</td>\n",
       "      <td>1085.229105</td>\n",
       "      <td>1085.229105</td>\n",
       "      <td>1085.229105</td>\n",
       "      <td>1085.229105</td>\n",
       "      <td>1085.229105</td>\n",
       "      <td>957.128681</td>\n",
       "      <td>957.128681</td>\n",
       "      <td>978.015518</td>\n",
       "      <td>1006.325306</td>\n",
       "      <td>1006.325306</td>\n",
       "      <td>1020.214464</td>\n",
       "      <td>1020.214464</td>\n",
       "      <td>1053.333220</td>\n",
       "      <td>1052.487287</td>\n",
       "      <td>1052.487287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mae</td>\n",
       "      <td>25.837154</td>\n",
       "      <td>25.837154</td>\n",
       "      <td>26.815872</td>\n",
       "      <td>26.594224</td>\n",
       "      <td>26.594224</td>\n",
       "      <td>26.007038</td>\n",
       "      <td>26.007038</td>\n",
       "      <td>26.338364</td>\n",
       "      <td>26.332094</td>\n",
       "      <td>26.332094</td>\n",
       "      <td>28.537745</td>\n",
       "      <td>28.537745</td>\n",
       "      <td>28.537745</td>\n",
       "      <td>28.537745</td>\n",
       "      <td>28.537745</td>\n",
       "      <td>26.648317</td>\n",
       "      <td>26.648317</td>\n",
       "      <td>26.654979</td>\n",
       "      <td>27.422521</td>\n",
       "      <td>27.422521</td>\n",
       "      <td>27.685907</td>\n",
       "      <td>27.685907</td>\n",
       "      <td>27.903745</td>\n",
       "      <td>28.052790</td>\n",
       "      <td>28.052790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.160775</td>\n",
       "      <td>0.160775</td>\n",
       "      <td>0.090755</td>\n",
       "      <td>0.117211</td>\n",
       "      <td>0.117211</td>\n",
       "      <td>0.154968</td>\n",
       "      <td>0.154968</td>\n",
       "      <td>0.109737</td>\n",
       "      <td>0.128453</td>\n",
       "      <td>0.128453</td>\n",
       "      <td>-0.000178</td>\n",
       "      <td>-0.000178</td>\n",
       "      <td>-0.000178</td>\n",
       "      <td>-0.000178</td>\n",
       "      <td>-0.000178</td>\n",
       "      <td>0.117883</td>\n",
       "      <td>0.117883</td>\n",
       "      <td>0.098633</td>\n",
       "      <td>0.072542</td>\n",
       "      <td>0.072542</td>\n",
       "      <td>0.059742</td>\n",
       "      <td>0.059742</td>\n",
       "      <td>0.029218</td>\n",
       "      <td>0.029998</td>\n",
       "      <td>0.029998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>var</td>\n",
       "      <td>115.263109</td>\n",
       "      <td>115.263109</td>\n",
       "      <td>89.208954</td>\n",
       "      <td>69.161578</td>\n",
       "      <td>69.161578</td>\n",
       "      <td>116.994071</td>\n",
       "      <td>116.994071</td>\n",
       "      <td>119.705079</td>\n",
       "      <td>90.484681</td>\n",
       "      <td>90.484681</td>\n",
       "      <td>0.192768</td>\n",
       "      <td>0.192768</td>\n",
       "      <td>0.192768</td>\n",
       "      <td>0.192768</td>\n",
       "      <td>0.192768</td>\n",
       "      <td>63.837409</td>\n",
       "      <td>63.837409</td>\n",
       "      <td>131.507831</td>\n",
       "      <td>39.141144</td>\n",
       "      <td>39.141144</td>\n",
       "      <td>47.292935</td>\n",
       "      <td>47.292935</td>\n",
       "      <td>85.452032</td>\n",
       "      <td>19.895657</td>\n",
       "      <td>19.895657</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  metric  rfr4_all_cv  rfr4_all_cv_idf  rfr4_all_w2v  rfr4_all_n2g_cv  \\\n",
       "0   rmse    30.175980        30.175980     31.409608        30.949282   \n",
       "1    mse   910.589795       910.589795    986.563479       957.858080   \n",
       "2    mae    25.837154        25.837154     26.815872        26.594224   \n",
       "3     r2     0.160775         0.160775      0.090755         0.117211   \n",
       "4    var   115.263109       115.263109     89.208954        69.161578   \n",
       "\n",
       "   rfr4_all_n2g_cv_idf  rfr4_better_cv  rfr4_better_cv_idf  rfr4_better_w2v  \\\n",
       "0            30.949282       30.280202           30.280202        31.080018   \n",
       "1           957.858080      916.890642          916.890642       965.967548   \n",
       "2            26.594224       26.007038           26.007038        26.338364   \n",
       "3             0.117211        0.154968            0.154968         0.109737   \n",
       "4            69.161578      116.994071          116.994071       119.705079   \n",
       "\n",
       "   rfr4_better_n2g_cv  rfr4_better_n2g_cv_idf  rfr4_best_cv  rfr4_best_cv_idf  \\\n",
       "0           30.751594               30.751594     32.942816         32.942816   \n",
       "1          945.660530              945.660530   1085.229105       1085.229105   \n",
       "2           26.332094               26.332094     28.537745         28.537745   \n",
       "3            0.128453                0.128453     -0.000178         -0.000178   \n",
       "4           90.484681               90.484681      0.192768          0.192768   \n",
       "\n",
       "   rfr4_best_w2v  rfr4_best_n2g_cv  rfr4_best_n2g_cv_idf  rfr4_efgs_cv  \\\n",
       "0      32.942816         32.942816             32.942816     30.937496   \n",
       "1    1085.229105       1085.229105           1085.229105    957.128681   \n",
       "2      28.537745         28.537745             28.537745     26.648317   \n",
       "3      -0.000178         -0.000178             -0.000178      0.117883   \n",
       "4       0.192768          0.192768              0.192768     63.837409   \n",
       "\n",
       "   rfr4_efgs_cv_idf  rfr4_efgs_w2v  rfr4_efgs_n2g_cv  rfr4_efgs_n2g_cv_idf  \\\n",
       "0         30.937496      31.273240         31.722631             31.722631   \n",
       "1        957.128681     978.015518       1006.325306           1006.325306   \n",
       "2         26.648317      26.654979         27.422521             27.422521   \n",
       "3          0.117883       0.098633          0.072542              0.072542   \n",
       "4         63.837409     131.507831         39.141144             39.141144   \n",
       "\n",
       "   rfr4_brics_cv  rfr4_brics_cv_idf  rfr4_brics_w2v  rfr4_brics_n2g_cv  \\\n",
       "0      31.940796          31.940796       32.455095          32.442060   \n",
       "1    1020.214464        1020.214464     1053.333220        1052.487287   \n",
       "2      27.685907          27.685907       27.903745          28.052790   \n",
       "3       0.059742           0.059742        0.029218           0.029998   \n",
       "4      47.292935          47.292935       85.452032          19.895657   \n",
       "\n",
       "   rfr4_brics_n2g_cv_idf  \n",
       "0              32.442060  \n",
       "1            1052.487287  \n",
       "2              28.052790  \n",
       "3               0.029998  \n",
       "4              19.895657  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# iteration 4\n",
    "pd.set_option('display.max_columns',None)\n",
    "eval_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "877cf3ee",
   "metadata": {},
   "source": [
    "### Results:\n",
    "**Findings:**\n",
    "1. the most effective fragment representation is `frags_all`\n",
    "2. the most effective nlp representation is **CountVectorizer** or CV-IDF > bigram CV > word2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce489ff7",
   "metadata": {},
   "source": [
    "## NEW: Train & Evaluate `frags_all` (CV, bigram CV, word2vec) using different regression models\n",
    "**Goal:** \n",
    "1. Determine the most useful type of regression model for this data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07b82e7b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a7487679",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>rfr3_w2vec_def</th>\n",
       "      <th>rfr3_w2vec_cust</th>\n",
       "      <th>rfr3_cv_idf</th>\n",
       "      <th>rfr3_tf_idf</th>\n",
       "      <th>rfr3_n2grams_cv_idf</th>\n",
       "      <th>rfr3_n3grams_cv_idf</th>\n",
       "      <th>rfr3_n4grams_cv_idf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rmse</td>\n",
       "      <td>31.168074</td>\n",
       "      <td>31.133882</td>\n",
       "      <td>30.123685</td>\n",
       "      <td>30.159424</td>\n",
       "      <td>30.834382</td>\n",
       "      <td>31.182895</td>\n",
       "      <td>31.495046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mse</td>\n",
       "      <td>971.448852</td>\n",
       "      <td>969.318623</td>\n",
       "      <td>907.436409</td>\n",
       "      <td>909.590845</td>\n",
       "      <td>950.759135</td>\n",
       "      <td>972.372924</td>\n",
       "      <td>991.937937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mae</td>\n",
       "      <td>26.523286</td>\n",
       "      <td>26.843417</td>\n",
       "      <td>25.845951</td>\n",
       "      <td>25.864042</td>\n",
       "      <td>26.494599</td>\n",
       "      <td>26.798564</td>\n",
       "      <td>27.159820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.102098</td>\n",
       "      <td>0.104067</td>\n",
       "      <td>0.161264</td>\n",
       "      <td>0.159273</td>\n",
       "      <td>0.121221</td>\n",
       "      <td>0.101244</td>\n",
       "      <td>0.083160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>var</td>\n",
       "      <td>111.697618</td>\n",
       "      <td>81.278117</td>\n",
       "      <td>114.494916</td>\n",
       "      <td>117.116534</td>\n",
       "      <td>69.047863</td>\n",
       "      <td>69.245370</td>\n",
       "      <td>60.039057</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  metric  rfr3_w2vec_def  rfr3_w2vec_cust  rfr3_cv_idf  rfr3_tf_idf  \\\n",
       "0   rmse       31.168074        31.133882    30.123685    30.159424   \n",
       "1    mse      971.448852       969.318623   907.436409   909.590845   \n",
       "2    mae       26.523286        26.843417    25.845951    25.864042   \n",
       "3     r2        0.102098         0.104067     0.161264     0.159273   \n",
       "4    var      111.697618        81.278117   114.494916   117.116534   \n",
       "\n",
       "   rfr3_n2grams_cv_idf  rfr3_n3grams_cv_idf  rfr3_n4grams_cv_idf  \n",
       "0            30.834382            31.182895            31.495046  \n",
       "1           950.759135           972.372924           991.937937  \n",
       "2            26.494599            26.798564            27.159820  \n",
       "3             0.121221             0.101244             0.083160  \n",
       "4            69.047863            69.245370            60.039057  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# iteration 3\n",
    "eval_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4f444ec",
   "metadata": {},
   "source": [
    "* Observations:\n",
    "  * at least when sourced from \"frags_all\", the most effective vector features for the fragments are:\n",
    "    * cv-idf > tf-idf > 2-gram-cv-idf > word2vec\n",
    "    * sometimes, word2vec has been better than bigram cv-idf\n",
    "    * default settings word2vec usually perform best\n",
    "  * it might be interesting to **check how well bigrams perform in a word2vec model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5b50024d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' \\n\\n'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "''' # BACKUP EVALUATION HISTORY\n",
    "# --- NOTE: to load eval history, use:\n",
    "# with open('evaluation_history.pickle', 'rb') as handle:\n",
    "#    evaluation_history = pickle.load(handle)\n",
    "'''\n",
    "import pickle\n",
    "\n",
    "with open('evaluation_history.pickle', 'wb') as handle:\n",
    "    pickle.dump(evaluation_history, handle, protocol=pickle.HIGHEST_PROTOCOL)\n",
    "\n",
    "\n",
    "''' \n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "fd4ffa7c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>rfr_features_w2vec_def</th>\n",
       "      <th>rfr_features_w2vec_cust</th>\n",
       "      <th>rfr_features_cv_idf</th>\n",
       "      <th>rfr_features_tf_idf</th>\n",
       "      <th>rfr_features_ngrams_cv_idf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rmse</td>\n",
       "      <td>30.965411</td>\n",
       "      <td>31.486140</td>\n",
       "      <td>29.843273</td>\n",
       "      <td>30.128548</td>\n",
       "      <td>30.991749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mse</td>\n",
       "      <td>958.856685</td>\n",
       "      <td>991.377036</td>\n",
       "      <td>890.620956</td>\n",
       "      <td>907.729404</td>\n",
       "      <td>960.488518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mae</td>\n",
       "      <td>26.766015</td>\n",
       "      <td>27.368773</td>\n",
       "      <td>25.910357</td>\n",
       "      <td>26.088403</td>\n",
       "      <td>26.862186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>r2</td>\n",
       "      <td>0.115671</td>\n",
       "      <td>0.085678</td>\n",
       "      <td>0.178603</td>\n",
       "      <td>0.162824</td>\n",
       "      <td>0.114166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>var</td>\n",
       "      <td>88.740958</td>\n",
       "      <td>70.540491</td>\n",
       "      <td>96.338847</td>\n",
       "      <td>105.874621</td>\n",
       "      <td>64.765121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  metric  rfr_features_w2vec_def  rfr_features_w2vec_cust  \\\n",
       "0   rmse               30.965411                31.486140   \n",
       "1    mse              958.856685               991.377036   \n",
       "2    mae               26.766015                27.368773   \n",
       "3     r2                0.115671                 0.085678   \n",
       "4    var               88.740958                70.540491   \n",
       "\n",
       "   rfr_features_cv_idf  rfr_features_tf_idf  rfr_features_ngrams_cv_idf  \n",
       "0            29.843273            30.128548                   30.991749  \n",
       "1           890.620956           907.729404                  960.488518  \n",
       "2            25.910357            26.088403                   26.862186  \n",
       "3             0.178603             0.162824                    0.114166  \n",
       "4            96.338847           105.874621                   64.765121  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "0c4a3224",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>metric</th>\n",
       "      <th>rmse</th>\n",
       "      <th>mse</th>\n",
       "      <th>mae</th>\n",
       "      <th>r2</th>\n",
       "      <th>var</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>rfr_features_w2vec_def</th>\n",
       "      <td>32.081343</td>\n",
       "      <td>1029.212583</td>\n",
       "      <td>27.718737</td>\n",
       "      <td>0.091856</td>\n",
       "      <td>100.840483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rfr_features_w2vec_cust</th>\n",
       "      <td>32.149451</td>\n",
       "      <td>1033.587185</td>\n",
       "      <td>27.879429</td>\n",
       "      <td>0.087996</td>\n",
       "      <td>85.211317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rfr_features_cv_idf</th>\n",
       "      <td>31.647149</td>\n",
       "      <td>1001.542048</td>\n",
       "      <td>27.534192</td>\n",
       "      <td>0.116271</td>\n",
       "      <td>114.702764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rfr_features_tf_idf</th>\n",
       "      <td>31.820964</td>\n",
       "      <td>1012.573776</td>\n",
       "      <td>27.683315</td>\n",
       "      <td>0.106537</td>\n",
       "      <td>94.232878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rfr_features_ngrams_cv_idf</th>\n",
       "      <td>32.336071</td>\n",
       "      <td>1045.621459</td>\n",
       "      <td>28.126130</td>\n",
       "      <td>0.077377</td>\n",
       "      <td>55.617227</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "metric                           rmse          mse        mae        r2  \\\n",
       "rfr_features_w2vec_def      32.081343  1029.212583  27.718737  0.091856   \n",
       "rfr_features_w2vec_cust     32.149451  1033.587185  27.879429  0.087996   \n",
       "rfr_features_cv_idf         31.647149  1001.542048  27.534192  0.116271   \n",
       "rfr_features_tf_idf         31.820964  1012.573776  27.683315  0.106537   \n",
       "rfr_features_ngrams_cv_idf  32.336071  1045.621459  28.126130  0.077377   \n",
       "\n",
       "metric                             var  \n",
       "rfr_features_w2vec_def      100.840483  \n",
       "rfr_features_w2vec_cust      85.211317  \n",
       "rfr_features_cv_idf         114.702764  \n",
       "rfr_features_tf_idf          94.232878  \n",
       "rfr_features_ngrams_cv_idf   55.617227  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testx = eval_df.set_index('metric')\n",
    "testx = testx.transpose()\n",
    "testx.head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "07cdb256",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>idx</th>\n",
       "      <th>name</th>\n",
       "      <th>Importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>582</th>\n",
       "      <td>582</td>\n",
       "      <td>frags_tf_idf_582</td>\n",
       "      <td>0.082723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1602</th>\n",
       "      <td>1602</td>\n",
       "      <td>frags_tf_idf_1602</td>\n",
       "      <td>0.074515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2375</th>\n",
       "      <td>2375</td>\n",
       "      <td>frags_tf_idf_2375</td>\n",
       "      <td>0.047800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>866</th>\n",
       "      <td>866</td>\n",
       "      <td>frags_tf_idf_866</td>\n",
       "      <td>0.039371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2280</th>\n",
       "      <td>2280</td>\n",
       "      <td>frags_tf_idf_2280</td>\n",
       "      <td>0.034088</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       idx               name  Importance\n",
       "582    582   frags_tf_idf_582    0.082723\n",
       "1602  1602  frags_tf_idf_1602    0.074515\n",
       "2375  2375  frags_tf_idf_2375    0.047800\n",
       "866    866   frags_tf_idf_866    0.039371\n",
       "2280  2280  frags_tf_idf_2280    0.034088"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "''' # Check feature importances\n",
    "'''\n",
    "import pandas as pd\n",
    "for k, v in test_subset_final.schema[\"features_tf_idf\"].metadata[\"ml_attr\"][\"attrs\"].items():\n",
    "    features_df = pd.DataFrame(v)\n",
    "    # format into a pandas dataframe and display\n",
    "    rf_output = rfmodel.featureImportances\n",
    "    features_df['Importance'] = features_df['idx'].apply(lambda x: rf_output[x] if x in rf_output.indices else 0)\n",
    "\n",
    "features_df.sort_values(\"Importance\", ascending=False, inplace=True)\n",
    "features_df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1563d658",
   "metadata": {},
   "source": [
    "* NOTE: to get vocab back, you can do it from the CountVectorizer indices\n",
    "  * fit CountVectorizer separately, store the vocab like this:\n",
    "    * `vectorizer = CountVectorizer(inputCol=\"tokens\", outputCol=\"features\").fit(df)`\n",
    "    * `vectorizer.vocabulary`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebfcdd46",
   "metadata": {},
   "source": [
    "## Test: Check feature importances of fragments as features "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "534bfd40",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4cf913d3",
   "metadata": {},
   "source": [
    "# <font color='red'> Proceed with other feature vectors </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7b3cfbf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "''' \n",
    "# FEATURE SELECTION:\n",
    "'''\n",
    "# to load the Features Information, use the command:\n",
    "featuresDF = pd.read_parquet('featuresCatalogDF_2022-08-16.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa98b8fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#index_pos = featuresDF[featuresDF['name']=='F1a'].index[0]\n",
    "feature_set1a = featuresDF.loc[0,'features']\n",
    "feature_set1b = featuresDF.loc[1,'features']\n",
    "feature_set2a = featuresDF.loc[2,'features']\n",
    "feature_set2b = featuresDF.loc[3,'features']\n",
    "feature_set3  = featuresDF.loc[4,'features']\n",
    "feature_set4a = featuresDF.loc[5,'features']\n",
    "feature_set4b = featuresDF.loc[6,'features']\n",
    "\n",
    "\n",
    "# VECTOR ASSEMBLY - feature sets 1a,1b,2a,2b,3,4a,4b\n",
    "from pyspark.ml.linalg import Vector\n",
    "from pyspark.ml.feature import (VectorAssembler,VectorIndexer)\n",
    "\n",
    "vec_assembler1a = VectorAssembler(inputCols = feature_set1a, outputCol='features1a')\n",
    "vec_assembler1b = VectorAssembler(inputCols = feature_set1b, outputCol='features1b')\n",
    "vec_assembler2a = VectorAssembler(inputCols = feature_set2a, outputCol='features2a')\n",
    "vec_assembler2b = VectorAssembler(inputCols = feature_set2b, outputCol='features2b')\n",
    "vec_assembler3 = VectorAssembler(inputCols = feature_set3, outputCol='features3')\n",
    "vec_assembler4a = VectorAssembler(inputCols = feature_set4a, outputCol='features4a')\n",
    "vec_assembler4b = VectorAssembler(inputCols = feature_set4b, outputCol='features4b')\n",
    "\n",
    "from pyspark.ml import Pipeline\n",
    "feature_pipeline = Pipeline(stages=[vec_assembler1a,\n",
    "                                    vec_assembler1b,\n",
    "                                    vec_assembler2a,\n",
    "                                    vec_assembler2b,\n",
    "                                    vec_assembler3,\n",
    "                                    vec_assembler4a,\n",
    "                                    vec_assembler4b])\n",
    "data_features = feature_pipeline.fit(data).transform(data)\n",
    "\n",
    "\n",
    "''' \n",
    "# LABELS \n",
    "# Data has 5 labels; 1 continuous BA percentage, 4 discretized groups.\n",
    "#  - ranges of discretized groups were selected by examining histogram distributions, mean & stdev of BA %\n",
    "#  - we'll add one more discretized label alternative using Spark's QuantileDiscretizer, into 5 groups\n",
    "\n",
    "# -- Add QuantileDiscretizer labels\n",
    "from pyspark.ml.feature import QuantileDiscretizer\n",
    "import pandas as pd\n",
    "qd5 = QuantileDiscretizer(numBuckets=5,inputCol='BA_pct',outputCol='label_QD5')\n",
    "\n",
    "data_features = qd5.fit(data_features).transform(data_features)\n",
    "\n",
    "# -- INDEX / ENCODE LABELS\n",
    "from pyspark.ml.feature import (StringIndexer,OneHotEncoder)\n",
    "\n",
    "label_quant0 = 'BA_pct'\n",
    "label_cat0_vector = OneHotEncoder(inputCol='label_QD5',outputCol='label_cat0_vector')\n",
    "\n",
    "label_cat1_index = StringIndexer(inputCol='label1',outputCol='label_cat1_index')\n",
    "label_cat1_vector = OneHotEncoder(inputCol='label_cat1_index',outputCol='label_cat1_vector')\n",
    "\n",
    "label_cat2_index = StringIndexer(inputCol='label2',outputCol='label_cat2_index')\n",
    "label_cat2_vector = OneHotEncoder(inputCol='label_cat2_index',outputCol='label_cat2_vector')\n",
    "\n",
    "label_cat3_index = StringIndexer(inputCol='label3a',outputCol='label_cat3_index')\n",
    "label_cat3_vector = OneHotEncoder(inputCol='label_cat3_index',outputCol='label_cat3_vector')\n",
    "\n",
    "label_cat4_index = StringIndexer(inputCol='label3b',outputCol='label_cat4_index')\n",
    "label_cat4_vector = OneHotEncoder(inputCol='label_cat4_index',outputCol='label_cat4_vector')\n",
    "\n",
    "from pyspark.ml import Pipeline\n",
    "label_pipeline = Pipeline(stages=[label_cat0_vector,\n",
    "                                 label_cat1_index,label_cat1_vector,\n",
    "                                 label_cat2_index,label_cat2_vector,\n",
    "                                 label_cat3_index,label_cat3_vector,\n",
    "                                 label_cat4_index,label_cat4_vector])\n",
    "'''\n",
    "data_features = data_features.select(['Name','BA_pct','label_QD5','label1','label2','label3a','label3b','features1a','features1b','features2a','features2b','features3','features4a','features4b'])\n",
    "\n",
    "#data_prefinal = label_pipeline.fit(data_features).transform(data_features)\n",
    "\n",
    "data_prefinal2 = data_prefinal.withColumnRenamed('BA_pct','label_q0')\n",
    "data_prefinal2 = data_prefinal2.withColumnRenamed('label_QD5','label_cat0')\n",
    "data_prefinal2 = data_prefinal2.withColumnRenamed('label_cat1_index','label_cat1')\n",
    "data_prefinal2 = data_prefinal2.withColumnRenamed('label_cat2_index','label_cat2')\n",
    "data_prefinal2 = data_prefinal2.withColumnRenamed('label3a','label3')\n",
    "data_prefinal2 = data_prefinal2.withColumnRenamed('label3b','label4')\n",
    "data_prefinal2 = data_prefinal2.withColumnRenamed('label_cat3_index','label_cat3')\n",
    "data_prefinal2 = data_prefinal2.withColumnRenamed('label_cat4_index','label_cat4')\n",
    "\n",
    "data_final = data_prefinal2.select(['Name','label_q0',\n",
    "                                    'label_cat0','label_cat1','label_cat2','label_cat3','label_cat4',\n",
    "                                    'features1a','features1b','features2a','features2b','features3','features4a','features4b'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}