notebooks/Churn-Prediction-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2a74eccf",
   "metadata": {},
   "source": [
    "# Предсказание оттока пользователей телеком компании"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77d42a47",
   "metadata": {},
   "source": [
    "В данном исследовании моделируется отток пользователей телеком компании с помощью различных моделей машинного обучения с целью определения лучшей модели. В качестве метрики качества используется ROC AUC.\n",
    "\n",
    "Данные для исследования взяты с платформы [Kaggle](https://www.kaggle.com/c/advanced-dls-spring-2021/).\n",
    "\n",
    "\n",
    "**Структура данных:**\n",
    "\n",
    "Информация о пользователях телеком компании находится в папке `data`:\n",
    "\n",
    "`data/train.csv` - обучающая выборка:\n",
    "\n",
    "* `ClientPeriod` – количество полных месяцев с момента регистрации пользователя\n",
    "\n",
    "\n",
    "* `MonthlySpending` – средние расходы пользователя в месяц\n",
    "\n",
    "\n",
    "* `TotalSpent` – общее количество денег, потраченых клиентом\n",
    "\n",
    "\n",
    "* `Sex` – пол пользователя\n",
    "\n",
    "\n",
    "* `IsSeniorCitizen` – является ли клиент пенсионером\n",
    "\n",
    "\n",
    "* `HasPartner` – наличие у клиента супруга(-и)\n",
    "\n",
    "\n",
    "* `HasChild` – наличие у клиента детей\n",
    "\n",
    "\n",
    "* `HasPhoneService` – подключение телефонного сервиса\n",
    "\n",
    "\n",
    "* `HasMultiplePhoneNumbers` – подключения нескольких телефонных номеров\n",
    "\n",
    "\n",
    "* `HasInternetService` – подключение интернет сервиса\n",
    "\n",
    "\n",
    "* `HasOnlineSecurityService` – подключение сервиса по онлайн безопасности\n",
    "\n",
    "\n",
    "* `HasOnlineBackup` – наличие облачной резервировной копии данных\n",
    "\n",
    "\n",
    "* `HasDeviceProtection` – наличие защиты устройства\n",
    "\n",
    "\n",
    "* `HasTechSupportAccess` – подключение услуги технической поддержки\n",
    "\n",
    "\n",
    "* `HasOnlineTV` – подключение онлайн телевидения\n",
    "\n",
    "\n",
    "* `HasMovieSubscription` – наличие подписки на онлайн-кинотеатр\n",
    "\n",
    "\n",
    "* `HasContractPhone` – тип контракта для телефонного сервиса\n",
    "\n",
    "\n",
    "* `IsBillingPaperless` – факт безналичной оплаты\n",
    "\n",
    "\n",
    "* `PaymentMethod` – метод оплаты\n",
    "\n",
    "\n",
    "* `Churn` – факт ухода пользователя\n",
    "\n",
    "\n",
    "`data/test.csv` - тестовая выборка, её схема аналогична обучающей выборке.\n",
    "\n",
    "\n",
    "**План:**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9583ecf",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "518b4152",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.047930Z",
     "start_time": "2022-10-17T22:33:30.832001Z"
    }
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "import numpy as np\n",
    "import optuna\n",
    "import pandas as pd\n",
    "import plotly.express as px\n",
    "import plotly.graph_objects as go\n",
    "import plotly.io as pio\n",
    "import sklearn\n",
    "from catboost import CatBoostClassifier\n",
    "from ipywidgets import fixed, interact\n",
    "from lightgbm import LGBMClassifier\n",
    "from matplotlib_inline.backend_inline import set_matplotlib_formats\n",
    "from plotly.subplots import make_subplots\n",
    "from scipy import stats\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "from sklearn.metrics import (\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    roc_auc_score,\n",
    ")\n",
    "from sklearn.model_selection import (\n",
    "    GridSearchCV,\n",
    "    RandomizedSearchCV,\n",
    "    StratifiedShuffleSplit,\n",
    "    cross_validate,\n",
    "    train_test_split,\n",
    ")\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import (\n",
    "    LabelEncoder,\n",
    "    OneHotEncoder,\n",
    "    OrdinalEncoder,\n",
    "    StandardScaler,\n",
    ")\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bffd8d2e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.286124Z",
     "start_time": "2022-10-17T22:33:32.049588Z"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "pd.set_option(\"display.float_format\", \"{:,.3f}\".format)\n",
    "set_matplotlib_formats(\"svg\")\n",
    "pio.templates.default = \"plotly_white\"\n",
    "pio.templates[\"plotly_white\"][\"layout\"][\"font\"] = {\"size\": 14, \"color\": \"black\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84f68f49",
   "metadata": {},
   "source": [
    "# Исследование данных"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87228eac",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.308723Z",
     "start_time": "2022-10-17T22:33:32.287415Z"
    }
   },
   "outputs": [],
   "source": [
    "train_data = pd.read_csv(\"data/train.csv\")\n",
    "test_data = pd.read_csv(\"data/test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abdea5ab",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.341221Z",
     "start_time": "2022-10-17T22:33:32.310883Z"
    }
   },
   "outputs": [],
   "source": [
    "interact(lambda data: display(globals()[data + \"_data\"]), data=[\"train\", \"test\"]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de1b4057",
   "metadata": {},
   "source": [
    "Приведем названия столбцов к snake_case стилю."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0635865f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.344585Z",
     "start_time": "2022-10-17T22:33:32.342533Z"
    }
   },
   "outputs": [],
   "source": [
    "def reformat_column_names(column_name):\n",
    "    return \"_\".join(re.findall(\"[A-Z][^A-Z]*\", column_name)).lower().replace(\"t_v\", \"tv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d00ff13d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.349315Z",
     "start_time": "2022-10-17T22:33:32.346370Z"
    }
   },
   "outputs": [],
   "source": [
    "train_data.columns = map(reformat_column_names, train_data.columns)\n",
    "test_data.columns = map(reformat_column_names, test_data.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31eeb33c",
   "metadata": {},
   "source": [
    "Посмотрим на общую информацию о данных."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8640a5f9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.453585Z",
     "start_time": "2022-10-17T22:33:32.350750Z"
    }
   },
   "outputs": [],
   "source": [
    "interact(lambda df: globals()[df].info(memory_usage=\"deep\"), df=[\"train_data\", \"test_data\"]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e98b08b6",
   "metadata": {},
   "source": [
    "В данных нет явных пропусков, однако столбец `total_spent` почему-то имеет тип данных `object`, хотя в нем записаны числовые значения. Проверим, нет ли в нем неявные пропущенные значения."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38e68eb4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.468178Z",
     "start_time": "2022-10-17T22:33:32.455088Z"
    }
   },
   "outputs": [],
   "source": [
    "train_data.query(\"total_spent in ['', ' ', 'None', 'nan']\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b7f9ee8",
   "metadata": {},
   "source": [
    "Действительно, в данных есть неявные пропуски, обозначенные как `' '`. Заменим их и преобразуем столбец к типу `float`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27c4ec1d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.476402Z",
     "start_time": "2022-10-17T22:33:32.469557Z"
    }
   },
   "outputs": [],
   "source": [
    "train_data.replace(\" \", np.nan, inplace=True)\n",
    "test_data.replace(\" \", np.nan, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e49815b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.485096Z",
     "start_time": "2022-10-17T22:33:32.479634Z"
    }
   },
   "outputs": [],
   "source": [
    "train_data[\"total_spent\"] = pd.to_numeric(train_data[\"total_spent\"], downcast=\"float\")\n",
    "test_data[\"total_spent\"] = pd.to_numeric(test_data[\"total_spent\"], downcast=\"float\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cee4c22e",
   "metadata": {},
   "source": [
    "Данные конвертировались в тип `float` без ошибок, значит, других нечисленных значений нет. Посмотрим на количество пропусков."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1a9752c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.511865Z",
     "start_time": "2022-10-17T22:33:32.486168Z"
    }
   },
   "outputs": [],
   "source": [
    "interact(lambda data: globals()[data + \"_data\"].isna().sum(), data=[\"train\", \"test\"]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d2affd",
   "metadata": {},
   "source": [
    "Всего 9 и 2 пропущенных значения в обучающей и тестовой выборке соотвественно. При построении моделей заполним их средними значениями среди соотвествующих столбцов."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f5424d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-01T20:55:20.712584Z",
     "start_time": "2022-08-01T20:55:20.703119Z"
    }
   },
   "source": [
    "Разделим данные на категориальные и количественные."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cc9dc04",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.522226Z",
     "start_time": "2022-10-17T22:33:32.513356Z"
    }
   },
   "outputs": [],
   "source": [
    "categorical_columns = [column for column in train_data.columns if train_data[column].nunique() < 20]\n",
    "numeric_columns = train_data.columns.difference(categorical_columns).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb45e959",
   "metadata": {},
   "source": [
    "## Категориальные данные"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef62da10",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.526818Z",
     "start_time": "2022-10-17T22:33:32.523488Z"
    }
   },
   "outputs": [],
   "source": [
    "len(categorical_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "525d1754",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.533742Z",
     "start_time": "2022-10-17T22:33:32.528441Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_categorical_data_distribution_pieplot(data, rows, cols, rotations=None, height=None):\n",
    "    fig = make_subplots(\n",
    "        rows,\n",
    "        cols,\n",
    "        specs=[[{\"type\": \"domain\"}] * cols] * rows,\n",
    "        vertical_spacing=0.08,\n",
    "        horizontal_spacing=0.1,\n",
    "        subplot_titles=data.columns,\n",
    "    )\n",
    "\n",
    "    rotations = rotations if rotations else [0] * data.shape[1]\n",
    "\n",
    "    for i, (column, rotation) in enumerate(zip(data.columns, rotations)):\n",
    "        value_counts = data[column].value_counts().reset_index()\n",
    "        value_counts.columns = [column, \"percent\"]\n",
    "\n",
    "        feature_fig = go.Pie(\n",
    "            labels=value_counts[column],\n",
    "            values=value_counts[\"percent\"],\n",
    "            rotation=rotation,\n",
    "            marker_colors=pio.templates[\"plotly_white\"][\"layout\"][\"colorway\"],\n",
    "            name=column,\n",
    "        )\n",
    "        feature_fig.update(textinfo=\"label+percent\", insidetextfont_color=\"white\", insidetextorientation=\"horizontal\")\n",
    "\n",
    "        fig.add_trace(feature_fig, row=i // cols + 1, col=i % cols + 1)\n",
    "\n",
    "    fig.update_layout(showlegend=False, font_size=10, margin=dict(l=0, r=0, b=0, t=50), height=height)\n",
    "\n",
    "    for annotation in fig.layout.annotations:\n",
    "        annotation.update(y=annotation.y + 0.01, font_size=10, text=\"<b>\" + annotation.text + \"</b>\")\n",
    "\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52c786f9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.538095Z",
     "start_time": "2022-10-17T22:33:32.535082Z"
    }
   },
   "outputs": [],
   "source": [
    "def categorical_distribution_by_dataset(data):\n",
    "    rotations = [0] * 4 + [-30] + [0] * 8 + [45] + [0] + [-110] + [0]\n",
    "    if data == \"train\":\n",
    "        return get_categorical_data_distribution_pieplot(train_data[categorical_columns], 5, 4, rotations, height=1000)\n",
    "    if data == \"test\":\n",
    "        return get_categorical_data_distribution_pieplot(\n",
    "            train_data[categorical_columns[:-1]], 4, 4, rotations, height=880\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e380e77",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.823658Z",
     "start_time": "2022-10-17T22:33:32.539270Z"
    }
   },
   "outputs": [],
   "source": [
    "interact(categorical_distribution_by_dataset, data=[\"train\", \"test\"]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbb1e549",
   "metadata": {},
   "source": [
    "Распределения обучающей и тестовой выборки одинаковые, а в целевой переменной присутствует значительный дисбаланс классов."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eb46c7c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-01T19:10:50.184047Z",
     "start_time": "2022-08-01T19:10:50.180786Z"
    }
   },
   "source": [
    "## Числовые данные"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "939201e7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.828653Z",
     "start_time": "2022-10-17T22:33:32.825750Z"
    }
   },
   "outputs": [],
   "source": [
    "len(numeric_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ece0e72d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.834142Z",
     "start_time": "2022-10-17T22:33:32.829838Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_numeric_data_distribution_histogram():\n",
    "    fig = make_subplots(2, 3, subplot_titles=numeric_columns, row_titles=[\"train_data\", \"test_data\"])\n",
    "\n",
    "    for i, (column, color) in enumerate(\n",
    "        zip(numeric_columns, pio.templates[\"plotly_white\"][\"layout\"][\"colorway\"]), start=1\n",
    "    ):\n",
    "        train_feature_fig = go.Histogram(x=train_data[column], marker_color=color)\n",
    "        test_feature_fig = go.Histogram(x=test_data[column], marker_color=color)\n",
    "        fig.add_trace(train_feature_fig, row=1, col=i)\n",
    "        fig.add_trace(test_feature_fig, row=2, col=i)\n",
    "\n",
    "    fig.update_layout(showlegend=False, margin=dict(r=0, b=0, t=50), height=300)\n",
    "\n",
    "    for annotation in fig.layout.annotations:\n",
    "        if annotation.text in numeric_columns:\n",
    "            annotation.update(y=annotation.y + 0.01)\n",
    "        if annotation.text in [\"train_data\", \"test_data\"]:\n",
    "            annotation.update(x=-0.09, textangle=-90)\n",
    "\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8e47df2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.910785Z",
     "start_time": "2022-10-17T22:33:32.835262Z"
    }
   },
   "outputs": [],
   "source": [
    "get_numeric_data_distribution_histogram()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e250005",
   "metadata": {},
   "source": [
    "Количественные распределения выборок тоже практически одинаковые."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19329c84",
   "metadata": {},
   "source": [
    "# Предсказание оттока"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e9aa4df",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-02T00:46:56.378362Z",
     "start_time": "2022-08-02T00:46:56.369527Z"
    }
   },
   "source": [
    "Мы попробуем много различных моделей и посмотрим, какой результат они дадут."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a4f48ba",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.915735Z",
     "start_time": "2022-10-17T22:33:32.912032Z"
    }
   },
   "outputs": [],
   "source": [
    "X = train_data.drop(\"churn\", axis=1)\n",
    "y = train_data[\"churn\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b759f6a",
   "metadata": {},
   "source": [
    "Разобьем `train_data` на обучающую и валидационную выборки. Так как в данных присутствует дисбаланс классов, выборки сделаем стратифицированными."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a58dfd7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.922821Z",
     "start_time": "2022-10-17T22:33:32.916872Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(X, y, train_size=0.8, stratify=y, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c7331e8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.926825Z",
     "start_time": "2022-10-17T22:33:32.924422Z"
    }
   },
   "outputs": [],
   "source": [
    "# удаление столбца churn\n",
    "categorical_columns.pop();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b2ec530",
   "metadata": {},
   "source": [
    "Прежде, чем передавать данные в модель, сначала избавимся от пропусков в них, нормализируем числовые значения и закодируем категориальные."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c1b8e6d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.931952Z",
     "start_time": "2022-10-17T22:33:32.927982Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_model(model: type, encoder: sklearn.preprocessing._encoders = None, *args, **kwargs):\n",
    "\n",
    "    encoder = encoder if encoder else OneHotEncoder(handle_unknown=\"ignore\")\n",
    "\n",
    "    numeric_preprocessor = Pipeline(\n",
    "        [\n",
    "            (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
    "            (\"standard_scaler\", StandardScaler()),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    categorical_preprocessor = Pipeline(\n",
    "        [\n",
    "            (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n",
    "            (\"one_hot_encoder\", encoder),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    preprocessor = ColumnTransformer(\n",
    "        [\n",
    "            (\"numeric_preprocessor\", numeric_preprocessor, numeric_columns),\n",
    "            (\"categorical_preprocessor\", categorical_preprocessor, categorical_columns),\n",
    "        ],\n",
    "        n_jobs=-1,\n",
    "    )\n",
    "\n",
    "    model = Pipeline([(\"preprocessor\", preprocessor), (\"classifier\", model(*args, **kwargs))])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d9edaa",
   "metadata": {},
   "source": [
    "## Метод k-ближайших соседей"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb05ac45",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-02T00:53:45.954433Z",
     "start_time": "2022-08-02T00:53:45.947584Z"
    }
   },
   "source": [
    "Сначала будем тестировать одну из самых простых моделей - метод k-ближайших соседей."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20a812cb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:32.971123Z",
     "start_time": "2022-10-17T22:33:32.933338Z"
    }
   },
   "outputs": [],
   "source": [
    "knn_model = create_model(KNeighborsClassifier, n_jobs=-1)\n",
    "knn_model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e195f298",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-02T00:54:01.033129Z",
     "start_time": "2022-08-02T00:54:01.027548Z"
    }
   },
   "source": [
    "Посмотрим на базовое качество модели."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cdc9c2a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:35.546026Z",
     "start_time": "2022-10-17T22:33:32.972666Z"
    }
   },
   "outputs": [],
   "source": [
    "knn_model.fit(X_train, y_train)\n",
    "y_pred = knn_model.predict(X_val)\n",
    "print(classification_report(y_val, y_pred, digits=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e04f0a0",
   "metadata": {},
   "source": [
    "Невысокая точность и низкие precision и recall – последние два нам интересны даже больше, потому что у нас дисбаланс классов. Посмотрим на кросс-валидацию."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2eebea6f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:37.003458Z",
     "start_time": "2022-10-17T22:33:35.548888Z"
    }
   },
   "outputs": [],
   "source": [
    "cv = StratifiedShuffleSplit(n_splits=10, train_size=0.8)\n",
    "\n",
    "cv_results_base = pd.DataFrame(cross_validate(knn_model, X, y, cv=cv, n_jobs=-1, scoring=(\"accuracy\", \"f1\", \"roc_auc\")))\n",
    "cv_results_base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91de53d8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:37.015372Z",
     "start_time": "2022-10-17T22:33:37.007354Z"
    }
   },
   "outputs": [],
   "source": [
    "cv_results_base[[\"test_f1\", \"test_roc_auc\"]].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6b85b7d",
   "metadata": {},
   "source": [
    "Качество модели стабильно невысокое, F1 на валидационной выборке не превышает 0.6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fea7bb7c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:37.035598Z",
     "start_time": "2022-10-17T22:33:37.027526Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.DataFrame(confusion_matrix(y_val, y_pred), index=[\"y_true_0\", \"y_true_1\"], columns=[\"y_pred_0\", \"y_pred_1\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2639b09",
   "metadata": {},
   "source": [
    "Попытаемся улучшить качество модели, изменив гиперпараметры. Для этого будем использовать RandomSearch. В отличие от GridSearch он позволяет сэкономить время и работает над большим полем гиперпараметров."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9426226",
   "metadata": {},
   "source": [
    "### Случайный поиск"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b283bab",
   "metadata": {},
   "source": [
    "Для некоторых параметров нужны целочисленные значения, так что создадим функцию для преобразования распределений модуля `scipy.stats` в целочисленные."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4b4b810",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:37.041063Z",
     "start_time": "2022-10-17T22:33:37.037459Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_int_distribution(distribution):\n",
    "    class int_distribution:\n",
    "        def __init__(self, *args, **kwargs):\n",
    "            self._distribution = distribution(*args, **kwargs)\n",
    "\n",
    "        def rvs(self, *args, **kwargs):\n",
    "            return self._distribution.rvs(*args, **kwargs).astype(int)\n",
    "\n",
    "    return int_distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da99adeb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:37.045959Z",
     "start_time": "2022-10-17T22:33:37.042948Z"
    }
   },
   "outputs": [],
   "source": [
    "uniform_int = create_int_distribution(stats.uniform)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91b5a46d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:37.053446Z",
     "start_time": "2022-10-17T22:33:37.047581Z"
    }
   },
   "outputs": [],
   "source": [
    "knn_param_distributions = dict(\n",
    "    n_neighbors=uniform_int(5, 64),\n",
    "    weights=[\"uniform\", \"distance\"],\n",
    "    algorithm=[\"auto\", \"ball_tree\", \"kd_tree\", \"brute\"],\n",
    "    leaf_size=uniform_int(5, 64),\n",
    "    p=[1, 2],\n",
    ")\n",
    "\n",
    "knn_param_names = list(knn_param_distributions.keys())\n",
    "\n",
    "knn_param_distributions = {\"classifier__\" + key: value for key, value in knn_param_distributions.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a5458a0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:33:37.058092Z",
     "start_time": "2022-10-17T22:33:37.055278Z"
    }
   },
   "outputs": [],
   "source": [
    "knn_model_random_search = RandomizedSearchCV(\n",
    "    knn_model, knn_param_distributions, n_iter=100, scoring=(\"f1\", \"roc_auc\"), refit=\"f1\", n_jobs=-1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "782ace94",
   "metadata": {},
   "source": [
    "Протестируем на кросс-валидации."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04d83582",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:36:48.326940Z",
     "start_time": "2022-10-17T22:33:37.060039Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "cv = StratifiedShuffleSplit(n_splits=5, train_size=0.8)\n",
    "cv_results_random_search = pd.DataFrame(\n",
    "    cross_validate(knn_model_random_search, X, y, cv=cv, scoring=(\"f1\", \"roc_auc\"), n_jobs=-1)\n",
    ")\n",
    "cv_results_random_search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10474614",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:36:48.344890Z",
     "start_time": "2022-10-17T22:36:48.330262Z"
    }
   },
   "outputs": [],
   "source": [
    "cv_results = pd.DataFrame(\n",
    "    [\n",
    "        cv_results_base[[\"test_f1\", \"test_roc_auc\"]].mean(),\n",
    "        cv_results_random_search[[\"test_f1\", \"test_roc_auc\"]].mean(),\n",
    "    ],\n",
    "    index=[\"base\", \"random_search\"],\n",
    ").T\n",
    "\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a43879de",
   "metadata": {},
   "source": [
    "Метрики F1 и ROC AUC увеличились на ~0.05. Попробуем искать больше различных вариантов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "736c8f6f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:40:39.198096Z",
     "start_time": "2022-10-17T22:40:39.191602Z"
    }
   },
   "outputs": [],
   "source": [
    "knn_model_random_search = RandomizedSearchCV(\n",
    "    knn_model, knn_param_distributions, n_iter=100, scoring=(\"f1\", \"roc_auc\"), refit=\"f1\", n_jobs=-1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b8f171a",
   "metadata": {},
   "source": [
    "Делать вложенную кросс-валидацию для RandomSearch с 500 итерациями слишком времязатратно, так что обойдемся без нее."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57f92f8a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:41:34.310432Z",
     "start_time": "2022-10-17T22:40:39.997149Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "knn_model_random_search.fit(X_train, y_train)\n",
    "y_pred = knn_model_random_search.predict(X_val)\n",
    "print(classification_report(y_val, y_pred, digits=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45a3487a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.655630Z",
     "start_time": "2022-10-17T22:38:13.211860Z"
    }
   },
   "outputs": [],
   "source": [
    "y_pred_proba = knn_model_random_search.predict_proba(X_val)\n",
    "roc_auc_score(y_val, y_pred_proba[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41d319fa",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-02T01:17:35.517202Z",
     "start_time": "2022-08-02T01:17:35.509408Z"
    }
   },
   "source": [
    "Смогли еще немного увеличить качество модели. В итоге лучшее качество для модели k-ближайших соседей:\n",
    "\n",
    "**F1:** 0.618\n",
    "<br>\n",
    "**ROC AUC:** 0.847\n",
    "\n",
    "Теперь посмотрим, как выбор различных гиперпараметров влияет на качество модели."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38112724",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.682558Z",
     "start_time": "2022-10-17T22:38:13.682548Z"
    }
   },
   "outputs": [],
   "source": [
    "knn_random_search_cv_results = pd.DataFrame(knn_model_random_search.cv_results_)\n",
    "knn_random_search_cv_results.columns = knn_random_search_cv_results.columns.str.replace(\"param_classifier__\", \"\")\n",
    "knn_random_search_cv_results = knn_random_search_cv_results[knn_param_names + [\"mean_test_f1\", \"mean_test_roc_auc\"]]\n",
    "knn_random_search_cv_results = knn_random_search_cv_results.apply(pd.to_numeric, errors=\"ignore\")\n",
    "knn_random_search_cv_results[\"p\"] = knn_random_search_cv_results[\"p\"].astype(object)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "625a5812",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.683295Z",
     "start_time": "2022-10-17T22:38:13.683285Z"
    }
   },
   "outputs": [],
   "source": [
    "knn_random_search_cv_results.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "885a34d4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.684423Z",
     "start_time": "2022-10-17T22:38:13.684413Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_features_parcoord(cv_results, score_name, only_top=False):\n",
    "    dimensions = []\n",
    "    label_encoder = LabelEncoder()\n",
    "\n",
    "    for (column_name, column) in cv_results.iteritems():\n",
    "        if only_top and column_name == score_name:\n",
    "            constraintrange = [column.max() * 0.98, column.max()]\n",
    "        else:\n",
    "            constraintrange = None\n",
    "\n",
    "        if column.dtype == object:\n",
    "            dimension = dict(\n",
    "                range=[-1, column.nunique()],\n",
    "                label=column_name,\n",
    "                values=label_encoder.fit_transform(column),\n",
    "                tickvals=np.arange(0, column.nunique()),\n",
    "                ticktext=label_encoder.classes_,\n",
    "            )\n",
    "        else:\n",
    "            dimension = dict(\n",
    "                range=[column.min(), column.max()],\n",
    "                constraintrange=constraintrange,\n",
    "                label=column_name,\n",
    "                values=column,\n",
    "            )\n",
    "        dimensions.append(dimension)\n",
    "\n",
    "    fig = go.Figure(\n",
    "        go.Parcoords(\n",
    "            line=dict(\n",
    "                color=cv_results[score_name],\n",
    "                colorscale=\"Plasma\",\n",
    "                showscale=True,\n",
    "                cmin=cv_results[score_name].min() * 1.1,\n",
    "                cmax=cv_results[score_name].max(),\n",
    "                colorbar=dict(title=dict(text=score_name, font_size=16, side=\"right\"), tickfont_size=12),\n",
    "            ),\n",
    "            dimensions=dimensions,\n",
    "        )\n",
    "    )\n",
    "\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e85ee600",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.685499Z",
     "start_time": "2022-10-17T22:38:13.685490Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_features_parcoord_by_metric(random_search_cv_results, metric, only_top=False):\n",
    "    if metric == \"mean_test_f1\":\n",
    "        return create_features_parcoord(\n",
    "            random_search_cv_results.drop(\"mean_test_roc_auc\", axis=1), \"mean_test_f1\", only_top\n",
    "        )\n",
    "    if metric == \"mean_test_roc_auc\":\n",
    "        return create_features_parcoord(\n",
    "            random_search_cv_results.drop(\"mean_test_f1\", axis=1), \"mean_test_roc_auc\", only_top\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "213aec08",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.686642Z",
     "start_time": "2022-10-17T22:38:13.686632Z"
    }
   },
   "outputs": [],
   "source": [
    "interact(\n",
    "    create_features_parcoord_by_metric,\n",
    "    random_search_cv_results=fixed(knn_random_search_cv_results),\n",
    "    metric=[\"mean_test_f1\", \"mean_test_roc_auc\"],\n",
    "    only_top=[False, True],\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7130dad5-fdb0-424e-a7b1-1d5f19717f1e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.687389Z",
     "start_time": "2022-10-17T22:38:13.687378Z"
    }
   },
   "outputs": [],
   "source": [
    "fig = create_features_parcoord(\n",
    "    cv_results=knn_random_search_cv_results.drop(\"mean_test_roc_auc\", axis=1), score_name=\"mean_test_f1\"\n",
    ")\n",
    "fig.write_html(\"fig.html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f5fd0ea",
   "metadata": {},
   "source": [
    "У топ 2% лучших моделей используется равномерное усреднение соседей, а не взвешенное. Остальные параметры не сильно влияют на качество модели."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9493b869",
   "metadata": {},
   "source": [
    "## Логистическая регрессия"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9084dded",
   "metadata": {},
   "source": [
    "Посмотрим, как с задачей справится линейная модель."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a63f746f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:42:46.979741Z",
     "start_time": "2022-10-17T22:42:46.929370Z"
    }
   },
   "outputs": [],
   "source": [
    "lr_model = create_model(LogisticRegressionCV, scoring=\"f1\", max_iter=1000, n_jobs=-1)\n",
    "lr_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b2ea5d6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:42:48.897100Z",
     "start_time": "2022-10-17T22:42:47.535860Z"
    }
   },
   "outputs": [],
   "source": [
    "lr_model.fit(X_train, y_train)\n",
    "y_pred = lr_model.predict(X_val)\n",
    "print(classification_report(y_val, y_pred, digits=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6bd12bb",
   "metadata": {},
   "source": [
    "Логистическая регрессия кажется немного лучше метода k-ближайших соседей. Посмотрим на кросс-валидацию."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "079612b3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:42:52.118210Z",
     "start_time": "2022-10-17T22:42:49.412189Z"
    }
   },
   "outputs": [],
   "source": [
    "cv = StratifiedShuffleSplit(n_splits=10, train_size=0.8)\n",
    "\n",
    "cv_results_base = pd.DataFrame(cross_validate(lr_model, X, y, cv=cv, n_jobs=-1, scoring=(\"f1\", \"roc_auc\")))\n",
    "cv_results_base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17d86612",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:42:52.638771Z",
     "start_time": "2022-10-17T22:42:52.634293Z"
    }
   },
   "outputs": [],
   "source": [
    "cv_results_base[[\"test_f1\", \"test_roc_auc\"]].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba812515",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-02T14:17:35.023036Z",
     "start_time": "2022-08-02T14:17:35.014972Z"
    }
   },
   "source": [
    "Значения метрик у базовой логистической регрессии чуть хуже, чем значения у оптимизированной модели k-ближайших соседей. Попробуем отыскать оптимальные параметры модели. У логистической регрессии параметров не так много, так что можно использовать GridSearch, чтобы перебрать все комбинации."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64a72085",
   "metadata": {},
   "source": [
    "### Поиск по сетке"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cd1daf3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:42:53.137752Z",
     "start_time": "2022-10-17T22:42:53.134779Z"
    }
   },
   "outputs": [],
   "source": [
    "lr_param_grid = dict(\n",
    "    penalty=[\"l1\", \"l2\", \"elasticnet\"],\n",
    "    solver=[\"newton-cg\", \"lbfgs\", \"liblinear\", \"sag\", \"saga\"],\n",
    "    class_weight=[None, \"balanced\"],\n",
    ")\n",
    "\n",
    "lr_param_names = list(lr_param_grid.keys())\n",
    "\n",
    "lr_param_grid = {\"classifier__\" + key: value for key, value in lr_param_grid.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e1a523b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:42:53.641161Z",
     "start_time": "2022-10-17T22:42:53.638415Z"
    }
   },
   "outputs": [],
   "source": [
    "lr_model_grid_search = GridSearchCV(lr_model, lr_param_grid, scoring=(\"f1\", \"roc_auc\"), refit=\"f1\", n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4d3b31d",
   "metadata": {},
   "source": [
    "Из-за наличия несовместимых параметров, GridSearch будет выкидывать ошибку обучения соотвествующей модели. Я не смог найти, как отключить эти предупреждения, и судя по всему это баг, связанный с мультипроцессингом и параметром `n_jobs=-1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21064a6e",
   "metadata": {
    "ExecuteTime": {
     "start_time": "2022-10-17T22:42:54.658Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "# from sklearn.utils._testing import ignore_warnings\n",
    "# from sklearn.exceptions import FitFailedWarning, ConvergenceWarning\n",
    "# with ignore_warnings(category=[ConvergenceWarning, FitFailedWarning]):\n",
    "cv = StratifiedShuffleSplit(n_splits=5, train_size=0.8)\n",
    "\n",
    "cv_results_grid_search = pd.DataFrame(\n",
    "    cross_validate(lr_model_grid_search, X, y, cv=cv, scoring=(\"f1\", \"roc_auc\"), n_jobs=-1)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "649f4d43",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.698334Z",
     "start_time": "2022-10-17T22:38:13.698325Z"
    }
   },
   "outputs": [],
   "source": [
    "%mprun cv_results_grid_search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cb0195d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.699190Z",
     "start_time": "2022-10-17T22:38:13.699181Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cv_results_grid_search[[\"test_f1\", \"test_roc_auc\"]].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3decae70",
   "metadata": {},
   "source": [
    "Оптимизированная логистическая регрессия показывает примерно те же результаты, что и метод k-ближайших соседей.\n",
    "<br>\n",
    "\n",
    "Лучшее качество для логистической регрессии:\n",
    "\n",
    "**F1:** 0.622\n",
    "<br>\n",
    "**ROC AUC:** 0.847"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "850cdac1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-02T10:53:09.478497Z",
     "start_time": "2022-08-02T10:53:09.459034Z"
    }
   },
   "source": [
    "## Решающее дерево"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edfe0633",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.699991Z",
     "start_time": "2022-10-17T22:38:13.699983Z"
    }
   },
   "outputs": [],
   "source": [
    "ordinal_encoder = OrdinalEncoder(handle_unknown=\"use_encoded_value\", unknown_value=-1)\n",
    "dt_model = create_model(DecisionTreeClassifier, encoder=ordinal_encoder)\n",
    "dt_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6539694d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.700995Z",
     "start_time": "2022-10-17T22:38:13.700986Z"
    }
   },
   "outputs": [],
   "source": [
    "dt_model.fit(X_train, y_train)\n",
    "y_pred = dt_model.predict(X_val)\n",
    "print(classification_report(y_val, y_pred, digits=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b7e720e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.701807Z",
     "start_time": "2022-10-17T22:38:13.701795Z"
    }
   },
   "outputs": [],
   "source": [
    "y_pred_proba = dt_model.predict_proba(X_val)\n",
    "roc_auc_score(y_val, y_pred_proba[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc55640d",
   "metadata": {},
   "source": [
    "Модель решающего дерева с параметрами по умолчанию хуже предыдущих двух. Будем оптимизировать. В этот раз будем использовать библиотеку **Optuna**, реализующую различные методы оптимизации, такие как баесовская оптимизация, алгоритм имитации отжига и другие. Прежде, чем оптимизировать, с помощью RandomSearch посмотрим на то, как разные параметры влияют на качество модели."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3727370",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-08-02T20:52:59.739168Z",
     "start_time": "2022-08-02T20:52:59.733471Z"
    }
   },
   "source": [
    "### Случайный поиск"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c71829d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.702852Z",
     "start_time": "2022-10-17T22:38:13.702844Z"
    }
   },
   "outputs": [],
   "source": [
    "dt_param_distributions = dict(\n",
    "    criterion=[\"gini\", \"entropy\", \"log_loss\"],\n",
    "    max_depth=uniform_int(8, 128),\n",
    "    min_samples_split=uniform_int(2, 128),\n",
    "    min_samples_leaf=uniform_int(1, 128),\n",
    "    max_features=[None, \"sqrt\", \"log2\"],\n",
    "    class_weight=[None, \"balanced\"],\n",
    ")\n",
    "\n",
    "dt_param_names = list(dt_param_distributions.keys())\n",
    "\n",
    "dt_param_distributions = {\"classifier__\" + key: value for key, value in dt_param_distributions.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d60e25d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.703944Z",
     "start_time": "2022-10-17T22:38:13.703937Z"
    }
   },
   "outputs": [],
   "source": [
    "dt_model_random_search = RandomizedSearchCV(\n",
    "    dt_model,\n",
    "    dt_param_distributions,\n",
    "    n_iter=5000,\n",
    "    scoring=(\"f1\", \"roc_auc\"),\n",
    "    refit=\"f1\",\n",
    "    return_train_score=True,\n",
    "    n_jobs=-1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05f24473",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.705077Z",
     "start_time": "2022-10-17T22:38:13.705070Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "dt_model_random_search.fit(X_train, y_train)\n",
    "y_pred = dt_model_random_search.predict(X_val)\n",
    "print(classification_report(y_val, y_pred, digits=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d1ae332",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.706031Z",
     "start_time": "2022-10-17T22:38:13.706022Z"
    }
   },
   "outputs": [],
   "source": [
    "y_pred_proba = dt_model_random_search.predict_proba(X_val)\n",
    "roc_auc_score(y_val, y_pred_proba[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f6f83a3",
   "metadata": {},
   "source": [
    "У модели значительно увеличился recall, но практически не вырос precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b9a10c7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.706963Z",
     "start_time": "2022-10-17T22:38:13.706955Z"
    }
   },
   "outputs": [],
   "source": [
    "dt_random_search_cv_results = pd.DataFrame(dt_model_random_search.cv_results_)\n",
    "dt_random_search_cv_results.columns = dt_random_search_cv_results.columns.str.replace(\"param_classifier__\", \"\")\n",
    "dt_random_search_cv_results = dt_random_search_cv_results[dt_param_names + [\"mean_test_f1\", \"mean_test_roc_auc\"]]\n",
    "dt_random_search_cv_results = dt_random_search_cv_results.apply(pd.to_numeric, errors=\"ignore\")\n",
    "dt_random_search_cv_results.fillna(\"None\", inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb5551d1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.707874Z",
     "start_time": "2022-10-17T22:38:13.707862Z"
    }
   },
   "outputs": [],
   "source": [
    "dt_random_search_cv_results.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24d81968",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.708635Z",
     "start_time": "2022-10-17T22:38:13.708627Z"
    }
   },
   "outputs": [],
   "source": [
    "interact(\n",
    "    create_features_parcoord_by_metric,\n",
    "    random_search_cv_results=fixed(dt_random_search_cv_results),\n",
    "    metric=[\"mean_test_f1\", \"mean_test_roc_auc\"],\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "642050ce",
   "metadata": {},
   "source": [
    "Все лучшие модели имеют параметр `class_weight` равный `balanced`. В остальном выбор параметров не сильно влияет на качество модели. Попробуем улучшить результат с помощью баесовской оптимизации."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6cdd7ff",
   "metadata": {},
   "source": [
    "### Баесовская оптимизация"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfbe0a09",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.710128Z",
     "start_time": "2022-10-17T22:38:13.710117Z"
    }
   },
   "outputs": [],
   "source": [
    "from ray.tune.suggest.optuna import OptunaSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7831f10e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.711463Z",
     "start_time": "2022-10-17T22:38:13.711451Z"
    }
   },
   "outputs": [],
   "source": [
    "def objective(trial):\n",
    "\n",
    "    params = dict(\n",
    "        class_weight=trial.suggest_categorical(\"class_weight\", [None, \"balanced\"]),\n",
    "        max_features=trial.suggest_categorical(\"max_features\", [None, \"log2\", \"sqrt\"]),\n",
    "        min_samples_leaf=trial.suggest_int(\"min_samples_leaf\", 20, 100),\n",
    "        min_samples_split=trial.suggest_int(\"min_samples_split\", 2, 120),\n",
    "        max_depth=trial.suggest_int(\"max_depth\", 8, 120),\n",
    "        criterion=trial.suggest_categorical(\"criterion\", [\"entropy\", \"gini\", \"log_loss\"]),\n",
    "    )\n",
    "\n",
    "    dt_model.set_params(**{\"classifier__\" + key: value for key, value in params.items()})\n",
    "\n",
    "    cv = StratifiedShuffleSplit(n_splits=10, train_size=0.8)\n",
    "    try:\n",
    "        cv_results = pd.DataFrame(\n",
    "            cross_validate(dt_model, X_train, y_train, cv=cv, n_jobs=-1, scoring=(\"f1\", \"roc_auc\"))\n",
    "        )\n",
    "        f1_score = cv_results[\"test_f1\"].mean()\n",
    "        return f1_score\n",
    "    except Exception:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66954983",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.712178Z",
     "start_time": "2022-10-17T22:38:13.712169Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ordinal_encoder = OrdinalEncoder(handle_unknown=\"use_encoded_value\", unknown_value=-1)\n",
    "dt_model = create_model(DecisionTreeClassifier, encoder=ordinal_encoder, class_weight=\"balanced\")\n",
    "study = optuna.create_study(study_name=\"decision_tree_optimization\", direction=\"maximize\")\n",
    "study.optimize(objective, n_trials=2000, show_progress_bar=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f162a4ac",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.713150Z",
     "start_time": "2022-10-17T22:38:13.713142Z"
    }
   },
   "outputs": [],
   "source": [
    "study.best_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88c94961",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.714002Z",
     "start_time": "2022-10-17T22:38:13.713991Z"
    }
   },
   "outputs": [],
   "source": [
    "trials = study.trials_dataframe()\n",
    "trials.columns = trials.columns.str.replace(\"params_\", \"\").str.replace(\"value\", \"mean_test_f1\")\n",
    "trials = trials[dt_param_names[:-1] + [\"mean_test_f1\"]].fillna(\"None\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfb41b01",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.715270Z",
     "start_time": "2022-10-17T22:38:13.715260Z"
    }
   },
   "outputs": [],
   "source": [
    "trials.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc967c80",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.715955Z",
     "start_time": "2022-10-17T22:38:13.715947Z"
    }
   },
   "outputs": [],
   "source": [
    "create_features_parcoord(trials, \"mean_test_f1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7172a5a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.716757Z",
     "start_time": "2022-10-17T22:38:13.716749Z"
    }
   },
   "outputs": [],
   "source": [
    "dt_model.set_params(**{\"classifier__\" + key: value for key, value in study.best_params.items()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32a58eff",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.717618Z",
     "start_time": "2022-10-17T22:38:13.717609Z"
    }
   },
   "outputs": [],
   "source": [
    "dt_model.fit(X_train, y_train)\n",
    "y_pred = dt_model.predict(X_val)\n",
    "print(classification_report(y_val, y_pred, digits=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24b9bf84",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T22:38:13.718853Z",
     "start_time": "2022-10-17T22:38:13.718844Z"
    }
   },
   "outputs": [],
   "source": [
    "y_pred_proba = dt_model.predict_proba(X_val)\n",
    "roc_auc_score(y_val, y_pred_proba[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47f6c4ff",
   "metadata": {},
   "source": [
    "## Случайный лес"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74cd107",
   "metadata": {},
   "source": [
    "## Градиентный бустинг\n"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "notify_time": "10",
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "288px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": "40",
    "lenType": 16,
    "lenVar": "16"
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "oldHeight": 320,
   "position": {
    "height": "342px",
    "left": "1070px",
    "right": "20px",
    "top": "119px",
    "width": "350px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "varInspector_section_display": "block",
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}