逻辑回归/05. 多元分类问题的求解.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在上一节中，我们根据梯度下降法得到参数 w 和 b 的更新公式：\n",
    "\n",
    "$$\n",
    "w := w - \\eta \\frac{\\partial loss}{\\partial w} = w + \\eta \\sum_{i=1}^n (y_i - \\pi(x_i))x_i\n",
    "$$\n",
    "\n",
    "$$\n",
    "b := b - \\eta \\frac{\\partial loss}{\\partial b} = b + \\eta \\sum_{i=1}^n (y_i - \\pi(x_i))\n",
    "$$\n",
    "\n",
    "这个公式可以用于一元分类的情况，也就是只有一个变量，我们可以进一步将其扩展到二元甚至多元的情况。多元逻辑函数可以写成矩阵的形式：\n",
    "\n",
    "$$\n",
    "y = \\frac{1}{1 + e^{-\\theta^TX}}\n",
    "$$\n",
    "\n",
    "比较容易得出，它对应的梯度更新公式为：\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j + \\eta \\sum_{i=1}^n (y^{(i)} - \\pi(x^{(i)}))x_j^{(i)}\n",
    "$$\n",
    "\n",
    "其中 $x_j^{(i)}$ 表示的是样本 $x^{(i)}$ 的第 $j$ 个分量，对应权重 $\\theta_j$。写成矩阵形式就是：\n",
    "\n",
    "$$\n",
    "\\Theta := \\Theta + \\eta X^T (Y - \\pi(X))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实例一、二元线性分类\n",
    "\n",
    "Peter Harrington 的 [《机器学习实战》第 5 章](../其他资料/书籍/04.%20机器学习实战.Peter%20Harrington.ipynb#第5章%E3%80%80Logistic回归) 介绍了一个例子，使用梯度上升法（和梯度下降法是一样的）求解二元线性分类问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实例二、预测肿瘤的良性和恶性\n",
    "\n",
    "这是一个多元分类的例子，这个例子来自 [这里](https://www.cnblogs.com/chenice/p/7202045.html)，使用了 UCI 的一份肿瘤数据集，并通过 sklearn 提供的逻辑回归和梯度下降方法来预测肿瘤的良性和恶性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(683, 11)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "column_names = [\n",
    "    'Sample code number',\n",
    "    'Clump Thickness',\n",
    "    'Uniformity of Cell Size',\n",
    "    'Uniformity of Cell Shape',\n",
    "    'Marginal Adhesion',\n",
    "    'Single Epithelial Cell Size',\n",
    "    'Bare Nuclei',\n",
    "    'Bland Chromatin',\n",
    "    'Normal Nucleoli',\n",
    "    'Mitoses',\n",
    "    'Class']\n",
    "\n",
    "data = pd.read_csv(\n",
    "    'https://archive.ics.uci.edu' + \n",
    "    '/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data',\n",
    "    names=column_names)\n",
    "data = data.replace(to_replace='?',value = np.nan)\n",
    "data = data.dropna(how='any')\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sample code number</th>\n",
       "      <th>Clump Thickness</th>\n",
       "      <th>Uniformity of Cell Size</th>\n",
       "      <th>Uniformity of Cell Shape</th>\n",
       "      <th>Marginal Adhesion</th>\n",
       "      <th>Single Epithelial Cell Size</th>\n",
       "      <th>Bare Nuclei</th>\n",
       "      <th>Bland Chromatin</th>\n",
       "      <th>Normal Nucleoli</th>\n",
       "      <th>Mitoses</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1000025</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002945</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1015425</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1016277</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1017023</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1017122</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1018099</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1018561</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1033078</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1033078</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sample code number  Clump Thickness  Uniformity of Cell Size  \\\n",
       "0             1000025                5                        1   \n",
       "1             1002945                5                        4   \n",
       "2             1015425                3                        1   \n",
       "3             1016277                6                        8   \n",
       "4             1017023                4                        1   \n",
       "5             1017122                8                       10   \n",
       "6             1018099                1                        1   \n",
       "7             1018561                2                        1   \n",
       "8             1033078                2                        1   \n",
       "9             1033078                4                        2   \n",
       "\n",
       "   Uniformity of Cell Shape  Marginal Adhesion  Single Epithelial Cell Size  \\\n",
       "0                         1                  1                            2   \n",
       "1                         4                  5                            7   \n",
       "2                         1                  1                            2   \n",
       "3                         8                  1                            3   \n",
       "4                         1                  3                            2   \n",
       "5                        10                  8                            7   \n",
       "6                         1                  1                            2   \n",
       "7                         2                  1                            2   \n",
       "8                         1                  1                            2   \n",
       "9                         1                  1                            2   \n",
       "\n",
       "  Bare Nuclei  Bland Chromatin  Normal Nucleoli  Mitoses  Class  \n",
       "0           1                3                1        1      2  \n",
       "1          10                3                2        1      2  \n",
       "2           2                3                1        1      2  \n",
       "3           4                3                7        1      2  \n",
       "4           1                3                1        1      2  \n",
       "5          10                9                7        1      4  \n",
       "6          10                3                1        1      2  \n",
       "7           1                3                1        1      2  \n",
       "8           1                1                1        5      2  \n",
       "9           1                2                1        1      2  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对数据进行分割，25%作为测试集，其余作为训练集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    100\n",
       "4     71\n",
       "Name: Class, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train,X_test,y_train,y_test = train_test_split(\n",
    "    data[column_names[1:10]],\n",
    "    data[column_names[10]],\n",
    "    test_size = 0.25,\n",
    "    random_state = 33)\n",
    "y_train.value_counts()\n",
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of LR Classifier: 0.9883040935672515\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "      Benign       0.99      0.99      0.99       100\n",
      "   Malignant       0.99      0.99      0.99        71\n",
      "\n",
      "   micro avg       0.99      0.99      0.99       171\n",
      "   macro avg       0.99      0.99      0.99       171\n",
      "weighted avg       0.99      0.99      0.99       171\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/aneasystone/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "/home/aneasystone/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/stochastic_gradient.py:166: FutureWarning: max_iter and tol parameters have been added in SGDClassifier in 0.19. If both are left unset, they default to max_iter=5 and tol=None. If tol is not None, max_iter defaults to max_iter=1000. From 0.21, default max_iter will be 1000, and default tol will be 1e-3.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "ss = StandardScaler()\n",
    "X_train = ss.fit_transform(X_train)\n",
    "X_test = ss.transform(X_test)\n",
    "\n",
    "# 逻辑回归\n",
    "lr = LogisticRegression()\n",
    "lr.fit(X_train,y_train)\n",
    "lr_y_predict = lr.predict(X_test)\n",
    "\n",
    "# 梯度下降\n",
    "sgdc = SGDClassifier()\n",
    "sgdc.fit(X_train,y_train)\n",
    "sgdc_y_predict = sgdc.predict(X_test)\n",
    "\n",
    "\n",
    "print('Accuracy of LR Classifier:',lr.score(X_test,y_test))\n",
    "print(classification_report(y_test,lr_y_predict,target_names=['Benign','Malignant']))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}