6-2,训练模型的3种方法.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6edd5547",
   "metadata": {},
   "source": [
    "# 6-2,训练模型的3种方法\n",
    "\n",
    "Pytorch通常需要用户编写自定义训练循环，训练循环的代码风格因人而异。\n",
    "\n",
    "有3类典型的训练循环代码风格：脚本形式训练循环，函数形式训练循环，类形式训练循环。\n",
    "\n",
    "下面以minist数据集的多分类模型的训练为例，演示这3种训练模型的风格。\n",
    "\n",
    "其中类形式训练循环我们同时演示torchkeras.KerasModel和torchkeras.LightModel两种示范。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02dff65a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24b101ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "import torchkeras\n",
    "print(\"torch.__version__ = \", torch.__version__)\n",
    "print(\"torchkeras.__version__ = \", torchkeras.__version__) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2586a627",
   "metadata": {},
   "source": [
    "```\n",
    "torch.__version__ =  1.10.0\n",
    "torchkeras.__version__ =  3.2.3\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b143f46a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "52642b45",
   "metadata": {},
   "source": [
    "### 〇，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04d2d65b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn \n",
    "\n",
    "import torchvision \n",
    "from torchvision import transforms\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fa1c7d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "transform = transforms.Compose([transforms.ToTensor()])\n",
    "\n",
    "ds_train = torchvision.datasets.MNIST(root=\"./data/minist/\",train=True,download=True,transform=transform)\n",
    "ds_val = torchvision.datasets.MNIST(root=\"./data/minist/\",train=False,download=True,transform=transform)\n",
    "\n",
    "dl_train =  torch.utils.data.DataLoader(ds_train, batch_size=128, shuffle=True, num_workers=4)\n",
    "dl_val =  torch.utils.data.DataLoader(ds_val, batch_size=128, shuffle=False, num_workers=4)\n",
    "\n",
    "print(len(ds_train))\n",
    "print(len(ds_val))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb15a7e4",
   "metadata": {},
   "source": [
    "```\n",
    "60000\n",
    "10000\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7130d4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'svg'\n",
    "\n",
    "#查看部分样本\n",
    "from matplotlib import pyplot as plt \n",
    "\n",
    "plt.figure(figsize=(8,8)) \n",
    "for i in range(9):\n",
    "    img,label = ds_train[i]\n",
    "    img = torch.squeeze(img)\n",
    "    ax=plt.subplot(3,3,i+1)\n",
    "    ax.imshow(img.numpy())\n",
    "    ax.set_title(\"label = %d\"%label)\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([]) \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55d3e3f5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6af1b177",
   "metadata": {},
   "source": [
    "![](./data/6-2-minist.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b42ca4a2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a3238e3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "99c0512e",
   "metadata": {},
   "source": [
    "### 一，脚本风格"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d305aa4",
   "metadata": {},
   "source": [
    "脚本风格的训练循环最为常见。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dad7971",
   "metadata": {},
   "outputs": [],
   "source": [
    "net = nn.Sequential()\n",
    "net.add_module(\"conv1\",nn.Conv2d(in_channels=1,out_channels=32,kernel_size = 3))\n",
    "net.add_module(\"pool1\",nn.MaxPool2d(kernel_size = 2,stride = 2))\n",
    "net.add_module(\"conv2\",nn.Conv2d(in_channels=32,out_channels=64,kernel_size = 5))\n",
    "net.add_module(\"pool2\",nn.MaxPool2d(kernel_size = 2,stride = 2))\n",
    "net.add_module(\"dropout\",nn.Dropout2d(p = 0.1))\n",
    "net.add_module(\"adaptive_pool\",nn.AdaptiveMaxPool2d((1,1)))\n",
    "net.add_module(\"flatten\",nn.Flatten())\n",
    "net.add_module(\"linear1\",nn.Linear(64,32))\n",
    "net.add_module(\"relu\",nn.ReLU())\n",
    "net.add_module(\"linear2\",nn.Linear(32,10))\n",
    "\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f99414",
   "metadata": {},
   "source": [
    "```\n",
    "Sequential(\n",
    "  (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))\n",
    "  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
    "  (conv2): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1))\n",
    "  (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
    "  (dropout): Dropout2d(p=0.1, inplace=False)\n",
    "  (adaptive_pool): AdaptiveMaxPool2d(output_size=(1, 1))\n",
    "  (flatten): Flatten()\n",
    "  (linear1): Linear(in_features=64, out_features=32, bias=True)\n",
    "  (relu): ReLU()\n",
    "  (linear2): Linear(in_features=32, out_features=10, bias=True)\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68444322",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os,sys,time\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datetime \n",
    "from tqdm import tqdm \n",
    "\n",
    "import torch\n",
    "from torch import nn \n",
    "from copy import deepcopy\n",
    "from torchmetrics import Accuracy\n",
    "#注：多分类使用torchmetrics中的评估指标，二分类使用torchkeras.metrics中的评估指标\n",
    "\n",
    "def printlog(info):\n",
    "    nowtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')\n",
    "    print(\"\\n\"+\"==========\"*8 + \"%s\"%nowtime)\n",
    "    print(str(info)+\"\\n\")\n",
    "    \n",
    "\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer= torch.optim.Adam(net.parameters(),lr = 0.01)   \n",
    "metrics_dict = {\"acc\":Accuracy()}\n",
    "\n",
    "epochs = 20 \n",
    "ckpt_path='checkpoint.pt'\n",
    "\n",
    "#early_stopping相关设置\n",
    "monitor=\"val_acc\"\n",
    "patience=5\n",
    "mode=\"max\"\n",
    "\n",
    "history = {}\n",
    "\n",
    "for epoch in range(1, epochs+1):\n",
    "    printlog(\"Epoch {0} / {1}\".format(epoch, epochs))\n",
    "\n",
    "    # 1，train -------------------------------------------------  \n",
    "    net.train()\n",
    "    \n",
    "    total_loss,step = 0,0\n",
    "    \n",
    "    loop = tqdm(enumerate(dl_train), total =len(dl_train))\n",
    "    train_metrics_dict = deepcopy(metrics_dict) \n",
    "    \n",
    "    for i, batch in loop: \n",
    "        \n",
    "        features,labels = batch\n",
    "        #forward\n",
    "        preds = net(features)\n",
    "        loss = loss_fn(preds,labels)\n",
    "        \n",
    "        #backward\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "            \n",
    "        #metrics\n",
    "        step_metrics = {\"train_\"+name:metric_fn(preds, labels).item() \n",
    "                        for name,metric_fn in train_metrics_dict.items()}\n",
    "        \n",
    "        step_log = dict({\"train_loss\":loss.item()},**step_metrics)\n",
    "\n",
    "        total_loss += loss.item()\n",
    "        \n",
    "        step+=1\n",
    "        if i!=len(dl_train)-1:\n",
    "            loop.set_postfix(**step_log)\n",
    "        else:\n",
    "            epoch_loss = total_loss/step\n",
    "            epoch_metrics = {\"train_\"+name:metric_fn.compute().item() \n",
    "                             for name,metric_fn in train_metrics_dict.items()}\n",
    "            epoch_log = dict({\"train_loss\":epoch_loss},**epoch_metrics)\n",
    "            loop.set_postfix(**epoch_log)\n",
    "\n",
    "            for name,metric_fn in train_metrics_dict.items():\n",
    "                metric_fn.reset()\n",
    "                \n",
    "    for name, metric in epoch_log.items():\n",
    "        history[name] = history.get(name, []) + [metric]\n",
    "        \n",
    "\n",
    "    # 2，validate -------------------------------------------------\n",
    "    net.eval()\n",
    "    \n",
    "    total_loss,step = 0,0\n",
    "    loop = tqdm(enumerate(dl_val), total =len(dl_val))\n",
    "    \n",
    "    val_metrics_dict = deepcopy(metrics_dict) \n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for i, batch in loop: \n",
    "\n",
    "            features,labels = batch\n",
    "            \n",
    "            #forward\n",
    "            preds = net(features)\n",
    "            loss = loss_fn(preds,labels)\n",
    "\n",
    "            #metrics\n",
    "            step_metrics = {\"val_\"+name:metric_fn(preds, labels).item() \n",
    "                            for name,metric_fn in val_metrics_dict.items()}\n",
    "\n",
    "            step_log = dict({\"val_loss\":loss.item()},**step_metrics)\n",
    "\n",
    "            total_loss += loss.item()\n",
    "            step+=1\n",
    "            if i!=len(dl_val)-1:\n",
    "                loop.set_postfix(**step_log)\n",
    "            else:\n",
    "                epoch_loss = (total_loss/step)\n",
    "                epoch_metrics = {\"val_\"+name:metric_fn.compute().item() \n",
    "                                 for name,metric_fn in val_metrics_dict.items()}\n",
    "                epoch_log = dict({\"val_loss\":epoch_loss},**epoch_metrics)\n",
    "                loop.set_postfix(**epoch_log)\n",
    "\n",
    "                for name,metric_fn in val_metrics_dict.items():\n",
    "                    metric_fn.reset()\n",
    "                    \n",
    "    epoch_log[\"epoch\"] = epoch           \n",
    "    for name, metric in epoch_log.items():\n",
    "        history[name] = history.get(name, []) + [metric]\n",
    "\n",
    "    # 3，early-stopping -------------------------------------------------\n",
    "    arr_scores = history[monitor]\n",
    "    best_score_idx = np.argmax(arr_scores) if mode==\"max\" else np.argmin(arr_scores)\n",
    "    if best_score_idx==len(arr_scores)-1:\n",
    "        torch.save(net.state_dict(),ckpt_path)\n",
    "        print(\"<<<<<< reach best {0} : {1} >>>>>>\".format(monitor,\n",
    "             arr_scores[best_score_idx]),file=sys.stderr)\n",
    "    if len(arr_scores)-best_score_idx>patience:\n",
    "        print(\"<<<<<< {} without improvement in {} epoch, early stopping >>>>>>\".format(\n",
    "            monitor,patience),file=sys.stderr)\n",
    "        break \n",
    "    net.load_state_dict(torch.load(ckpt_path))\n",
    "    \n",
    "dfhistory = pd.DataFrame(history)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "018920b1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2f7bc866",
   "metadata": {},
   "source": [
    "```\n",
    "================================================================================2022-07-17 19:21:50\n",
    "Epoch 17 / 20\n",
    "\n",
    "\n",
    "100%|██████████| 469/469 [01:01<00:00,  7.67it/s, train_acc=0.985, train_loss=0.06]   \n",
    "100%|██████████| 79/79 [00:07<00:00, 10.45it/s, val_acc=0.985, val_loss=0.0659] \n",
    "\n",
    "================================================================================2022-07-17 19:22:59\n",
    "Epoch 18 / 20\n",
    "\n",
    "\n",
    "<<<<<< reach best val_acc : 0.9851999878883362 >>>>>>\n",
    "100%|██████████| 469/469 [00:57<00:00,  8.14it/s, train_acc=0.983, train_loss=0.0692]\n",
    "100%|██████████| 79/79 [00:07<00:00, 10.89it/s, val_acc=0.984, val_loss=0.0849]\n",
    "\n",
    "================================================================================2022-07-17 19:24:04\n",
    "Epoch 19 / 20\n",
    "\n",
    "\n",
    "100%|██████████| 469/469 [01:08<00:00,  6.82it/s, train_acc=0.984, train_loss=0.0679] \n",
    "100%|██████████| 79/79 [00:06<00:00, 11.39it/s, val_acc=0.983, val_loss=0.087]  \n",
    "\n",
    "================================================================================2022-07-17 19:25:20\n",
    "Epoch 20 / 20\n",
    "\n",
    "\n",
    "100%|██████████| 469/469 [00:57<00:00,  8.14it/s, train_acc=0.986, train_loss=0.0553] \n",
    "100%|██████████| 79/79 [00:06<00:00, 11.35it/s, val_acc=0.983, val_loss=0.0813]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c726168a",
   "metadata": {},
   "source": [
    "### 二，函数风格"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48817db4",
   "metadata": {},
   "source": [
    "该风格在脚本形式上做了进一步的函数封装。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a1a41d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.layers = nn.ModuleList([\n",
    "            nn.Conv2d(in_channels=1,out_channels=32,kernel_size = 3),\n",
    "            nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "            nn.Conv2d(in_channels=32,out_channels=64,kernel_size = 5),\n",
    "            nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "            nn.Dropout2d(p = 0.1),\n",
    "            nn.AdaptiveMaxPool2d((1,1)),\n",
    "            nn.Flatten(),\n",
    "            nn.Linear(64,32),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(32,10)]\n",
    "        )\n",
    "    def forward(self,x):\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n",
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b66cb36e",
   "metadata": {},
   "source": [
    "```\n",
    "Net(\n",
    "  (layers): ModuleList(\n",
    "    (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))\n",
    "    (1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
    "    (2): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1))\n",
    "    (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
    "    (4): Dropout2d(p=0.1, inplace=False)\n",
    "    (5): AdaptiveMaxPool2d(output_size=(1, 1))\n",
    "    (6): Flatten()\n",
    "    (7): Linear(in_features=64, out_features=32, bias=True)\n",
    "    (8): ReLU()\n",
    "    (9): Linear(in_features=32, out_features=10, bias=True)\n",
    "  )\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "840dec1d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c97a606",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os,sys,time\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datetime \n",
    "from tqdm import tqdm \n",
    "\n",
    "import torch\n",
    "from torch import nn \n",
    "from copy import deepcopy\n",
    "\n",
    "def printlog(info):\n",
    "    nowtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')\n",
    "    print(\"\\n\"+\"==========\"*8 + \"%s\"%nowtime)\n",
    "    print(str(info)+\"\\n\")\n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn,\n",
    "                 stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer = optimizer\n",
    "            \n",
    "    def step(self, features, labels):\n",
    "        #loss\n",
    "        preds = self.net(features)\n",
    "        loss = self.loss_fn(preds,labels)\n",
    "        \n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\": \n",
    "            loss.backward()\n",
    "            self.optimizer.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        #metrics\n",
    "        step_metrics = {self.stage+\"_\"+name:metric_fn(preds, labels).item() \n",
    "                        for name,metric_fn in self.metrics_dict.items()}\n",
    "        return loss.item(),step_metrics\n",
    "    \n",
    "    def train_step(self,features,labels):\n",
    "        self.net.train() #训练模式, dropout层发生作用\n",
    "        return self.step(features,labels)\n",
    "    \n",
    "    @torch.no_grad()\n",
    "    def eval_step(self,features,labels):\n",
    "        self.net.eval() #预测模式, dropout层不发生作用\n",
    "        return self.step(features,labels)\n",
    "    \n",
    "    def __call__(self,features,labels):\n",
    "        if self.stage==\"train\":\n",
    "            return self.train_step(features,labels) \n",
    "        else:\n",
    "            return self.eval_step(features,labels)\n",
    "        \n",
    "class EpochRunner:\n",
    "    def __init__(self,steprunner):\n",
    "        self.steprunner = steprunner\n",
    "        self.stage = steprunner.stage\n",
    "        \n",
    "    def __call__(self,dataloader):\n",
    "        total_loss,step = 0,0\n",
    "        loop = tqdm(enumerate(dataloader), total =len(dataloader))\n",
    "        for i, batch in loop: \n",
    "            loss, step_metrics = self.steprunner(*batch)\n",
    "            step_log = dict({self.stage+\"_loss\":loss},**step_metrics)\n",
    "            total_loss += loss\n",
    "            step+=1\n",
    "            if i!=len(dataloader)-1:\n",
    "                loop.set_postfix(**step_log)\n",
    "            else:\n",
    "                epoch_loss = total_loss/step\n",
    "                epoch_metrics = {self.stage+\"_\"+name:metric_fn.compute().item() \n",
    "                                 for name,metric_fn in self.steprunner.metrics_dict.items()}\n",
    "                epoch_log = dict({self.stage+\"_loss\":epoch_loss},**epoch_metrics)\n",
    "                loop.set_postfix(**epoch_log)\n",
    "\n",
    "                for name,metric_fn in self.steprunner.metrics_dict.items():\n",
    "                    metric_fn.reset()\n",
    "        return epoch_log\n",
    "\n",
    "\n",
    "def train_model(net, optimizer, loss_fn, metrics_dict, \n",
    "                train_data, val_data=None, \n",
    "                epochs=10, ckpt_path='checkpoint.pt',\n",
    "                patience=5, monitor=\"val_loss\", mode=\"min\"):\n",
    "    \n",
    "    history = {}\n",
    "\n",
    "    for epoch in range(1, epochs+1):\n",
    "        printlog(\"Epoch {0} / {1}\".format(epoch, epochs))\n",
    "\n",
    "        # 1，train -------------------------------------------------  \n",
    "        train_step_runner = StepRunner(net = net,stage=\"train\",\n",
    "                loss_fn = loss_fn,metrics_dict=deepcopy(metrics_dict),\n",
    "                optimizer = optimizer)\n",
    "        train_epoch_runner = EpochRunner(train_step_runner)\n",
    "        train_metrics = train_epoch_runner(train_data)\n",
    "\n",
    "        for name, metric in train_metrics.items():\n",
    "            history[name] = history.get(name, []) + [metric]\n",
    "\n",
    "        # 2，validate -------------------------------------------------\n",
    "        if val_data:\n",
    "            val_step_runner = StepRunner(net = net,stage=\"val\",\n",
    "                loss_fn = loss_fn,metrics_dict=deepcopy(metrics_dict))\n",
    "            val_epoch_runner = EpochRunner(val_step_runner)\n",
    "            with torch.no_grad():\n",
    "                val_metrics = val_epoch_runner(val_data)\n",
    "            val_metrics[\"epoch\"] = epoch\n",
    "            for name, metric in val_metrics.items():\n",
    "                history[name] = history.get(name, []) + [metric]\n",
    "\n",
    "        # 3，early-stopping -------------------------------------------------\n",
    "        arr_scores = history[monitor]\n",
    "        best_score_idx = np.argmax(arr_scores) if mode==\"max\" else np.argmin(arr_scores)\n",
    "        if best_score_idx==len(arr_scores)-1:\n",
    "            torch.save(net.state_dict(),ckpt_path)\n",
    "            print(\"<<<<<< reach best {0} : {1} >>>>>>\".format(monitor,\n",
    "                 arr_scores[best_score_idx]),file=sys.stderr)\n",
    "        if len(arr_scores)-best_score_idx>patience:\n",
    "            print(\"<<<<<< {} without improvement in {} epoch, early stopping >>>>>>\".format(\n",
    "                monitor,patience),file=sys.stderr)\n",
    "            break \n",
    "        net.load_state_dict(torch.load(ckpt_path))\n",
    "\n",
    "    return pd.DataFrame(history)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "feb456ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchmetrics import Accuracy\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer= torch.optim.Adam(net.parameters(),lr = 0.01)   \n",
    "metrics_dict = {\"acc\":Accuracy()}\n",
    "\n",
    "dfhistory = train_model(net,\n",
    "    optimizer,\n",
    "    loss_fn,\n",
    "    metrics_dict,\n",
    "    train_data = dl_train,\n",
    "    val_data= dl_val,\n",
    "    epochs=10,\n",
    "    patience=3,\n",
    "    monitor=\"val_acc\", \n",
    "    mode=\"max\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1ddb3cb",
   "metadata": {},
   "source": [
    "```\n",
    "================================================================================2022-07-17 19:27:50\n",
    "Epoch 1 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:00<00:00,  7.71it/s, train_acc=0.905, train_loss=0.296] \n",
    "100%|██████████| 79/79 [00:07<00:00, 11.18it/s, val_acc=0.96, val_loss=0.129]  \n",
    "\n",
    "================================================================================2022-07-17 19:28:58\n",
    "Epoch 2 / 10\n",
    "\n",
    "\n",
    "<<<<<< reach best val_acc : 0.9603000283241272 >>>>>>\n",
    "100%|██████████| 469/469 [00:57<00:00,  8.19it/s, train_acc=0.966, train_loss=0.115] \n",
    "100%|██████████| 79/79 [00:06<00:00, 11.55it/s, val_acc=0.982, val_loss=0.0639]\n",
    "\n",
    "================================================================================2022-07-17 19:30:02\n",
    "Epoch 3 / 10\n",
    "\n",
    "\n",
    "<<<<<< reach best val_acc : 0.9815999865531921 >>>>>>\n",
    "100%|██████████| 469/469 [01:00<00:00,  7.71it/s, train_acc=0.971, train_loss=0.0982]\n",
    "100%|██████████| 79/79 [00:07<00:00, 10.21it/s, val_acc=0.976, val_loss=0.0831] \n",
    "\n",
    "================================================================================2022-07-17 19:31:11\n",
    "Epoch 4 / 10\n",
    "\n",
    "\n",
    "100%|██████████| 469/469 [00:58<00:00,  8.06it/s, train_acc=0.972, train_loss=0.0932]\n",
    "100%|██████████| 79/79 [00:06<00:00, 11.32it/s, val_acc=0.978, val_loss=0.0786]\n",
    "\n",
    "================================================================================2022-07-17 19:32:17\n",
    "Epoch 5 / 10\n",
    "\n",
    "\n",
    "100%|██████████| 469/469 [00:57<00:00,  8.21it/s, train_acc=0.971, train_loss=0.0992]\n",
    "100%|██████████| 79/79 [00:07<00:00, 11.04it/s, val_acc=0.978, val_loss=0.0774]\n",
    "<<<<<< val_acc without improvement in 3 epoch, early stopping >>>>>>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bcd9c07",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d085b560",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0b68de55",
   "metadata": {},
   "source": [
    "### 三，类风格 torchkeras.KerasModel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40c2c331",
   "metadata": {},
   "source": [
    "此处使用torchkeras.KerasModel高层次API接口中的fit方法训练模型。\n",
    "\n",
    "使用该形式训练模型非常简洁明了。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eab1d853",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.layers = nn.ModuleList([\n",
    "            nn.Conv2d(in_channels=1,out_channels=32,kernel_size = 3),\n",
    "            nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "            nn.Conv2d(in_channels=32,out_channels=64,kernel_size = 5),\n",
    "            nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "            nn.Dropout2d(p = 0.1),\n",
    "            nn.AdaptiveMaxPool2d((1,1)),\n",
    "            nn.Flatten(),\n",
    "            nn.Linear(64,32),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(32,10)]\n",
    "        )\n",
    "    def forward(self,x):\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n",
    "    \n",
    "net = Net() \n",
    "\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "794e7ab9",
   "metadata": {},
   "source": [
    "```\n",
    "Net(\n",
    "  (layers): ModuleList(\n",
    "    (0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))\n",
    "    (1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
    "    (2): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1))\n",
    "    (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
    "    (4): Dropout2d(p=0.1, inplace=False)\n",
    "    (5): AdaptiveMaxPool2d(output_size=(1, 1))\n",
    "    (6): Flatten(start_dim=1, end_dim=-1)\n",
    "    (7): Linear(in_features=64, out_features=32, bias=True)\n",
    "    (8): ReLU()\n",
    "    (9): Linear(in_features=32, out_features=10, bias=True)\n",
    "  )\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf6ae428",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchmetrics import Accuracy\n",
    "\n",
    "model = KerasModel(net,\n",
    "                   loss_fn=nn.CrossEntropyLoss(),\n",
    "                   metrics_dict = {\"acc\":Accuracy()},\n",
    "                   optimizer = torch.optim.Adam(net.parameters(),lr = 0.01)  )\n",
    "\n",
    "model.fit(\n",
    "    train_data = dl_train,\n",
    "    val_data= dl_val,\n",
    "    epochs=10,\n",
    "    patience=3,\n",
    "    monitor=\"val_acc\", \n",
    "    mode=\"max\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d960e9ad",
   "metadata": {},
   "source": [
    "```\n",
    "================================================================================2022-07-17 21:01:18\n",
    "Epoch 1 / 10\n",
    "\n",
    "100%|██████████| 469/469 [00:58<00:00,  7.98it/s, train_acc=0.906, train_loss=0.291] \n",
    "100%|██████████| 79/79 [00:07<00:00, 10.60it/s, val_acc=0.974, val_loss=0.0773]\n",
    "<<<<<< reach best val_acc : 0.9739999771118164 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:02:24\n",
    "Epoch 2 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:00<00:00,  7.78it/s, train_acc=0.968, train_loss=0.105] \n",
    "100%|██████████| 79/79 [00:07<00:00,  9.92it/s, val_acc=0.971, val_loss=0.0882]\n",
    "\n",
    "================================================================================2022-07-17 21:03:32\n",
    "Epoch 3 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:11<00:00,  6.60it/s, train_acc=0.973, train_loss=0.0868]\n",
    "100%|██████████| 79/79 [00:08<00:00,  9.25it/s, val_acc=0.978, val_loss=0.0769]\n",
    "<<<<<< reach best val_acc : 0.9779999852180481 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:04:52\n",
    "Epoch 4 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:13<00:00,  6.34it/s, train_acc=0.973, train_loss=0.0888]\n",
    "100%|██████████| 79/79 [00:12<00:00,  6.47it/s, val_acc=0.979, val_loss=0.0789] \n",
    "<<<<<< reach best val_acc : 0.9793999791145325 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:06:18\n",
    "Epoch 5 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:04<00:00,  7.30it/s, train_acc=0.977, train_loss=0.08]  \n",
    "100%|██████████| 79/79 [00:12<00:00,  6.19it/s, val_acc=0.975, val_loss=0.0828]\n",
    "\n",
    "================================================================================2022-07-17 21:07:35\n",
    "Epoch 6 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:03<00:00,  7.44it/s, train_acc=0.979, train_loss=0.0719]\n",
    "100%|██████████| 79/79 [00:08<00:00,  9.51it/s, val_acc=0.981, val_loss=0.0664] \n",
    "<<<<<< reach best val_acc : 0.9805999994277954 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:08:47\n",
    "Epoch 7 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:01<00:00,  7.57it/s, train_acc=0.979, train_loss=0.0738]\n",
    "100%|██████████| 79/79 [00:07<00:00, 10.12it/s, val_acc=0.982, val_loss=0.0707] \n",
    "<<<<<< reach best val_acc : 0.9817000031471252 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:09:56\n",
    "Epoch 8 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:02<00:00,  7.50it/s, train_acc=0.979, train_loss=0.0747]\n",
    "100%|██████████| 79/79 [00:07<00:00,  9.91it/s, val_acc=0.983, val_loss=0.0667] \n",
    "<<<<<< reach best val_acc : 0.9833999872207642 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:11:07\n",
    "Epoch 9 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:03<00:00,  7.44it/s, train_acc=0.98, train_loss=0.0748]  \n",
    "100%|██████████| 79/79 [00:07<00:00, 10.20it/s, val_acc=0.985, val_loss=0.0658]\n",
    "<<<<<< reach best val_acc : 0.9850000143051147 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:12:18\n",
    "Epoch 10 / 10\n",
    "\n",
    "100%|██████████| 469/469 [01:02<00:00,  7.51it/s, train_acc=0.979, train_loss=0.0742]\n",
    "100%|██████████| 79/79 [00:08<00:00,  9.58it/s, val_acc=0.982, val_loss=0.0751]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24d8c747",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5a56a18",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a13efa44",
   "metadata": {},
   "source": [
    "### 四，类风格 torchkeras.LightModel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfae718b",
   "metadata": {},
   "source": [
    "除了torchkeras.KerasModel,torchkeras还提供了torchkeras.LightModel 来支持更多的功能。\n",
    "\n",
    "\n",
    "torchkeras.KerasModel 更加简单, 推荐给新手用户。\n",
    "\n",
    "而LightModel借鉴了 pytorch_lightning 库中的许多功能，并演示了对pytorch_lightning的一种最佳实践。\n",
    "\n",
    "\n",
    "尽管存在着一些差异, torchkeras.KerasModel 和 torchkeras.LightModel 的使用方式和特性是非常相似的。\n",
    "\n",
    "详情参考：https://github.com/lyhue1991/torchkeras \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "|features| torchkeras.KerasModel     |  torchkeras.LightModel   | \n",
    "|----:|:-------------------------:|:-----------:|\n",
    "|progress bar | ✅    |✅    |\n",
    "|early stopping | ✅    |✅    |\n",
    "|metrics from torchmetrics | ✅    |✅    |\n",
    "|gpu training | ✅    |✅    |\n",
    "|multi-gpus training |   ❌  |✅    |\n",
    "|tensorboard callback |   ❌  |✅    |\n",
    "|simple source code|   ✅   |❌  |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c628b28a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import LightModel \n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.layers = nn.ModuleList([\n",
    "            nn.Conv2d(in_channels=1,out_channels=32,kernel_size = 3),\n",
    "            nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "            nn.Conv2d(in_channels=32,out_channels=64,kernel_size = 5),\n",
    "            nn.MaxPool2d(kernel_size = 2,stride = 2),\n",
    "            nn.Dropout2d(p = 0.1),\n",
    "            nn.AdaptiveMaxPool2d((1,1)),\n",
    "            nn.Flatten(),\n",
    "            nn.Linear(64,32),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(32,10)]\n",
    "        )\n",
    "    def forward(self,x):\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        return x\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99073760",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchmetrics import Accuracy \n",
    "import pytorch_lightning as pl   \n",
    "\n",
    "net = Net()\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "metric_dict = {\"acc\":Accuracy()}\n",
    "\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=0.01)\n",
    "\n",
    "model = torchkeras.LightModel(net,\n",
    "                   loss_fn = loss_fn,\n",
    "                   metrics_dict= metric_dict,\n",
    "                   optimizer = optimizer\n",
    "                  )       \n",
    "\n",
    "\n",
    "  \n",
    "#1，设置回调函数\n",
    "model_ckpt = pl.callbacks.ModelCheckpoint(\n",
    "    monitor='val_acc',\n",
    "    save_top_k=1,\n",
    "    mode='max'\n",
    ")\n",
    "\n",
    "early_stopping = pl.callbacks.EarlyStopping(monitor = 'val_acc',\n",
    "                           patience=3,\n",
    "                           mode = 'max'\n",
    "                          )\n",
    "\n",
    "#2，设置训练参数\n",
    "\n",
    "trainer = pl.Trainer(logger=True,\n",
    "                     min_epochs=10,max_epochs=20,\n",
    "                     gpus=0,\n",
    "                     callbacks = [model_ckpt,early_stopping],\n",
    "                     enable_progress_bar = True) \n",
    "\n",
    "\n",
    "##3，启动训练循环\n",
    "trainer.fit(model,dl_train,dl_val)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcf4a5be",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b0cff3ac",
   "metadata": {},
   "source": [
    "```\n",
    " | Name          | Type             | Params\n",
    "---------------------------------------------------\n",
    "0 | net           | Net              | 54.0 K\n",
    "1 | train_metrics | ModuleDict       | 0     \n",
    "2 | val_metrics   | ModuleDict       | 0     \n",
    "3 | test_metrics  | ModuleDict       | 0     \n",
    "4 | loss_fn       | CrossEntropyLoss | 0     \n",
    "---------------------------------------------------\n",
    "54.0 K    Trainable params\n",
    "0         Non-trainable params\n",
    "54.0 K    Total params\n",
    "0.216     Total estimated model params size (MB)\n",
    "\n",
    "================================================================================2022-07-17 21:30:14\n",
    "{'epoch': 0, 'val_loss': 2.330533266067505, 'val_acc': 0.1015625}\n",
    "<<<<<< reach best val_acc : 0.1015625 >>>>>>\n",
    "Epoch 9: 100%\n",
    "548/548 [01:08<00:00, 8.04it/s, loss=0.0785, v_num=2, acc=0.948]\n",
    "\n",
    "================================================================================2022-07-17 21:31:23\n",
    "{'epoch': 0, 'val_loss': 0.12257852405309677, 'val_acc': 0.961899995803833}\n",
    "{'epoch': 0, 'train_loss': 0.3137461841106415, 'train_acc': 0.8978333473205566}\n",
    "<<<<<< reach best val_acc : 0.961899995803833 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:32:37\n",
    "{'epoch': 1, 'val_loss': 0.08038929104804993, 'val_acc': 0.9764000177383423}\n",
    "{'epoch': 1, 'train_loss': 0.1108873263001442, 'train_acc': 0.9664333462715149}\n",
    "<<<<<< reach best val_acc : 0.9764000177383423 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:33:46\n",
    "{'epoch': 2, 'val_loss': 0.07084609568119049, 'val_acc': 0.9800000190734863}\n",
    "{'epoch': 2, 'train_loss': 0.09520334005355835, 'train_acc': 0.9722499847412109}\n",
    "<<<<<< reach best val_acc : 0.9800000190734863 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:34:55\n",
    "{'epoch': 3, 'val_loss': 0.0732991024851799, 'val_acc': 0.9796000123023987}\n",
    "{'epoch': 3, 'train_loss': 0.0875784158706665, 'train_acc': 0.973550021648407}\n",
    "\n",
    "================================================================================2022-07-17 21:36:04\n",
    "{'epoch': 4, 'val_loss': 0.07758694887161255, 'val_acc': 0.9775999784469604}\n",
    "{'epoch': 4, 'train_loss': 0.0757126733660698, 'train_acc': 0.9774666428565979}\n",
    "\n",
    "================================================================================2022-07-17 21:37:13\n",
    "{'epoch': 5, 'val_loss': 0.05971676856279373, 'val_acc': 0.9832000136375427}\n",
    "{'epoch': 5, 'train_loss': 0.07704824209213257, 'train_acc': 0.9775000214576721}\n",
    "<<<<<< reach best val_acc : 0.9832000136375427 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:38:22\n",
    "{'epoch': 6, 'val_loss': 0.054445140063762665, 'val_acc': 0.9847000241279602}\n",
    "{'epoch': 6, 'train_loss': 0.07280954718589783, 'train_acc': 0.9792166948318481}\n",
    "<<<<<< reach best val_acc : 0.9847000241279602 >>>>>>\n",
    "\n",
    "================================================================================2022-07-17 21:39:29\n",
    "{'epoch': 7, 'val_loss': 0.08798510581254959, 'val_acc': 0.9764999747276306}\n",
    "{'epoch': 7, 'train_loss': 0.07288103550672531, 'train_acc': 0.9790499806404114}\n",
    "\n",
    "================================================================================2022-07-17 21:40:32\n",
    "{'epoch': 8, 'val_loss': 0.08194874972105026, 'val_acc': 0.9761000275611877}\n",
    "{'epoch': 8, 'train_loss': 0.06978205591440201, 'train_acc': 0.9799833297729492}\n",
    "\n",
    "================================================================================2022-07-17 21:41:40\n",
    "{'epoch': 9, 'val_loss': 0.07893478125333786, 'val_acc': 0.9810000061988831}\n",
    "{'epoch': 9, 'train_loss': 0.06853801012039185, 'train_acc': 0.9806166887283325}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "654f7576",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "597ea3df",
   "metadata": {},
   "source": [
    "**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号\"算法美食屋\"下留言。作者时间和精力有限，会酌情予以回复。\n",
    "\n",
    "也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。\n",
    "\n",
    "![算法美食屋logo.png](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdc05012",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9270eea",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}