Part3_GAN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\newcommand{\\mat}[1]{\\boldsymbol {#1}}\n",
    "\\newcommand{\\mattr}[1]{\\boldsymbol {#1}^\\top}\n",
    "\\newcommand{\\matinv}[1]{\\boldsymbol {#1}^{-1}}\n",
    "\\newcommand{\\vec}[1]{\\boldsymbol {#1}}\n",
    "\\newcommand{\\vectr}[1]{\\boldsymbol {#1}^\\top}\n",
    "\\newcommand{\\rvar}[1]{\\mathrm {#1}}\n",
    "\\newcommand{\\rvec}[1]{\\boldsymbol{\\mathrm{#1}}}\n",
    "\\newcommand{\\diag}{\\mathop{\\mathrm {diag}}}\n",
    "\\newcommand{\\set}[1]{\\mathbb {#1}}\n",
    "\\newcommand{\\norm}[1]{\\left\\lVert#1\\right\\rVert}\n",
    "\\newcommand{\\pderiv}[2]{\\frac{\\partial #1}{\\partial #2}}\n",
    "\\newcommand{\\bm}[1]{{\\bf #1}}\n",
    "\\newcommand{\\bb}[1]{\\bm{\\mathrm{#1}}}\n",
    "$$\n",
    "\n",
    "# Part 3: Generative Adversarial Networks\n",
    "<a id=part3></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part we will implement and train a generative adversarial network and apply it to the task of image generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unittest\n",
    "import os\n",
    "import sys\n",
    "import pathlib\n",
    "import urllib\n",
    "import shutil\n",
    "import re\n",
    "import zipfile\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "test = unittest.TestCase()\n",
    "plt.rcParams.update({'font.size': 12})\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print('Using device:', device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obtaining the dataset\n",
    "<a id=part3_1></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the same data as in Part 2.\n",
    "\n",
    "But again, to use a custom dataset, edit the `PART3_CUSTOM_DATA_URL` variable in `hw3/answers.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cs236605.plot as plot\n",
    "import cs236605.download\n",
    "from hw3.answers import PART3_CUSTOM_DATA_URL as CUSTOM_DATA_URL\n",
    "\n",
    "DATA_DIR = pathlib.Path.home().joinpath('.pytorch-datasets')\n",
    "if CUSTOM_DATA_URL is None:\n",
    "    DATA_URL = 'http://vis-www.cs.umass.edu/lfw/lfw-bush.zip'\n",
    "else:\n",
    "    DATA_URL = CUSTOM_DATA_URL\n",
    "\n",
    "_, dataset_dir = cs236605.download.download_data(out_path=DATA_DIR, url=DATA_URL, extract=True, force=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `Dataset` object that will load the extraced images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision.transforms as T\n",
    "from torchvision.datasets import ImageFolder\n",
    "\n",
    "im_size = 64\n",
    "tf = T.Compose([\n",
    "    # Resize to constant spatial dimensions\n",
    "    T.Resize((im_size, im_size)),\n",
    "    # PIL.Image -> torch.Tensor\n",
    "    T.ToTensor(),\n",
    "    # Dynamic range [0,1] -> [-1, 1]\n",
    "    T.Normalize(mean=(.5,.5,.5), std=(.5,.5,.5)),\n",
    "])\n",
    "\n",
    "ds_gwb = ImageFolder(os.path.dirname(dataset_dir), tf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, let's see what we got. You can run the following block multiple times to display a random subset of images from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = plot.dataset_first_n(ds_gwb, 50, figsize=(10,5), nrows=5)\n",
    "print(f'Found {len(ds_gwb)} images in dataset folder.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x0, y0 = ds_gwb[0]\n",
    "x0 = x0.unsqueeze(0).to(device)\n",
    "print(x0.shape)\n",
    "\n",
    "test.assertSequenceEqual(x0.shape, (1, 3, im_size, im_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generative Adversarial Nets (GANs)\n",
    "<a id=part3_2></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GANs, first proposed in a [paper](https://arxiv.org/pdf/1406.2661.pdf) by Ian Goodfellow in 2014 are today arguably the most popular type of generative model. GANs are currently producing state of the art\n",
    "results in generative tasks over many different domains.\n",
    "\n",
    "In a GAN model, two different neural networks compete against each other: A **generator** and a **discriminator**.\n",
    "\n",
    "- The Generator, which we'll denote as $\\Psi _{\\bb{\\gamma}} : \\mathcal{U} \\rightarrow \\mathcal{X}$, maps a latent-space variable\n",
    "$\\bb{u}\\sim\\mathcal{N}(\\bb{0},\\bb{I})$ to an instance-space variable $\\bb{x}$ (e.g. an image).\n",
    "Thus a parametric evidence distribution $p_{\\bb{\\gamma}}(\\bb{X})$ is generated,\n",
    "which we typically would like to be as close as possible to the real evidence distribution, $p(\\bb{X})$.\n",
    "\n",
    "- The Discriminator, $\\Delta _{\\bb{\\delta}} : \\mathcal{X} \\rightarrow [0,1]$, is a network which,\n",
    "given an instance-space variable $\\bb{x}$, returns the  probability that $\\bb{x}$ is real, i.e. that $\\bb{x}$\n",
    "was sampled from $p(\\bb{X})$ and not $p_{\\bb{\\gamma}}(\\bb{X})$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"imgs/gan.jpg\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training GANs\n",
    "<a id=part3_3></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generator is trained to generate \"fake\" instances which will maximally fool the\n",
    "discriminator into returning that they're real. Mathematically, the generator's parameters\n",
    "$\\bb{\\gamma}$ should be chosen such as to **maximize** the expression\n",
    "$$\n",
    "\\mathbb{E} _{\\bb{z} \\sim p(\\bb{Z}) } \\log (\\Delta _{\\bb{\\delta}}(\\Psi _{\\bb{\\gamma}} (\\bb{z}) )).\n",
    "$$\n",
    "\n",
    "The discriminator is trained to classify between real images, coming from the training set,\n",
    "and fake images generated by the generator.\n",
    "Mathematically, the discriminator's parameters $\\bb{\\delta}$ should be chosen such as to\n",
    "**maximize** the expression\n",
    "$$\n",
    "\\mathbb{E} _{\\bb{x} \\sim p(\\bb{X}) } \\log \\Delta _{\\bb{\\delta}}(\\bb{x})  \\, + \\,\n",
    "\\mathbb{E} _{\\bb{z} \\sim p(\\bb{Z}) } \\log (1-\\Delta _{\\bb{\\delta}}(\\Psi _{\\bb{\\gamma}} (\\bb{z}) )).\n",
    "$$\n",
    "\n",
    "These two competing objectives can thus be expressed as the following min-max optimization:\n",
    "$$\n",
    "\\min _{\\bb{\\gamma}} \\max _{\\bb{\\delta}} \\,\n",
    "\\mathbb{E} _{\\bb{x} \\sim p(\\bb{X}) } \\log \\Delta _{\\bb{\\delta}}(\\bb{x})  \\, + \\,\n",
    "\\mathbb{E} _{\\bb{z} \\sim p(\\bb{Z}) } \\log (1-\\Delta _{\\bb{\\delta}}(\\Psi _{\\bb{\\gamma}} (\\bb{z}) )).\n",
    "$$\n",
    "\n",
    "A key insight into GANs is that we can interpret the above maximum as the loss with respect to\n",
    "$\\bb{\\gamma}$:\n",
    "\n",
    "$$\n",
    "L({\\bb{\\gamma}}) =\n",
    "\\max _{\\bb{\\delta}} \\, \\mathbb{E} _{\\bb{x} \\sim p(\\bb{X}) } \\log \\Delta _{\\bb{\\delta}}(\\bb{x})  \\, + \\,\n",
    "  \\mathbb{E} _{\\bb{z} \\sim p(\\bb{Z}) } \\log (1-\\Delta _{\\bb{\\delta}}(\\Psi _{\\bb{\\gamma}} (\\bb{z}) )).\n",
    "$$\n",
    "\n",
    "This means that the generator's loss function trains together with the generator\n",
    "itself in an adversarial manner. In contrast, when training our VAE we used a fixed L2 norm\n",
    "as a data loss term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Implementation\n",
    "<a id=part3_4></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now implement a Deep Convolutional GAN (DCGAN) model.\n",
    "See the DCGAN [paper](https://arxiv.org/pdf/1511.06434.pdf) for architecture ideas and tips for training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Implement the `Discriminator` class in the `hw3/gan.py` module.\n",
    "If you wish you can reuse the `EncoderCNN` class from the VAE model as the first part of the Discriminator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hw3.gan as gan\n",
    "\n",
    "dsc = gan.Discriminator(in_size=x0[0].shape).to(device)\n",
    "print(dsc)\n",
    "\n",
    "d0 = dsc(x0)\n",
    "print(d0.shape)\n",
    "\n",
    "test.assertSequenceEqual(d0.shape, (1,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Implement the `Generator` class in the `hw3/gan.py` module.\n",
    "If you wish you can reuse the `DecoderCNN` class from the VAE model as the last part of the Generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z_dim = 128\n",
    "gen = gan.Generator(z_dim, 4).to(device)\n",
    "print(gen)\n",
    "\n",
    "z = torch.randn(1, z_dim).to(device)\n",
    "xr = gen(z)\n",
    "print(xr.shape)\n",
    "\n",
    "test.assertSequenceEqual(x0.shape, xr.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Implementation\n",
    "<a id=part3_5></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin with the discriminator's loss function.\n",
    "Based on the above we can flip the sign and say we want to update the \n",
    "Discriminator's parameters $\\bb{\\delta}$ so that they **minimize** the expression\n",
    "$$\n",
    "- \\mathbb{E} _{\\bb{x} \\sim p(\\bb{X}) } \\log \\Delta _{\\bb{\\delta}}(\\bb{x})  \\, - \\,\n",
    "\\mathbb{E} _{\\bb{z} \\sim p(\\bb{Z}) } \\log (1-\\Delta _{\\bb{\\delta}}(\\Psi _{\\bb{\\gamma}} (\\bb{z}) )).\n",
    "$$\n",
    "\n",
    "We're using the Discriminator twice in this expression;\n",
    "once to classify data from the real data distribution and\n",
    "once again to classify generated data.\n",
    "Therefore our loss should be computed based on these two terms.\n",
    "Notice that since the discriminator returns a probability, we can formulate the above as two cross-entropy losses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GANs are notoriously diffucult to train.\n",
    "One common trick for improving GAN stability during training is to make the classification labels noisy for the discriminator. This can be seen as a form of regularization, to help prevent the discriminator from overfitting.\n",
    "\n",
    "We'll incorporate this idea into our loss function. Instead of labels being equal to 0 or 1, we'll make them\n",
    "\"fuzzy\", i.e. random numbers in the ranges $[0\\pm\\epsilon]$ and $[1\\pm\\epsilon]$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Implement the `discriminator_loss_fn()` function in the `hw3/gan.py` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hw3.gan import discriminator_loss_fn\n",
    "torch.manual_seed(42)\n",
    "\n",
    "y_data = torch.rand(10) * 10\n",
    "y_generated = torch.rand(10) * 10\n",
    "\n",
    "loss = discriminator_loss_fn(y_data, y_generated, data_label=1, label_noise=0.3)\n",
    "print(loss)\n",
    "\n",
    "test.assertAlmostEqual(loss.item(), 6.4808731, delta=1e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, the generator's parameters $\\bb{\\gamma}$ should **minimize** the expression\n",
    "$$\n",
    "-\\mathbb{E} _{\\bb{z} \\sim p(\\bb{Z}) } \\log (\\Delta _{\\bb{\\delta}}(\\Psi _{\\bb{\\gamma}} (\\bb{z}) ))\n",
    "$$\n",
    "\n",
    "which can also be seen as a cross-entropy term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Implement the `generator_loss_fn()` function in the `hw3/gan.py` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hw3.gan import generator_loss_fn\n",
    "torch.manual_seed(42)\n",
    "\n",
    "y_generated = torch.rand(20) * 10\n",
    "\n",
    "loss = generator_loss_fn(y_generated, data_label=1)\n",
    "print(loss)\n",
    "\n",
    "test.assertAlmostEqual(loss.item(), 0.0222969, delta=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sampling\n",
    "<a id=part3_6></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sampling from a GAN is straightforward, since it learns to generate data from an isotropic Gaussian latent space distribution.\n",
    "\n",
    "There is an important nuance however. Sampling is required during the process of training the GAN, since\n",
    "we generate fake images to show the discriminator.\n",
    "As you'll seen in the next section,  in some cases we'll need our samples to have gradients (i.e., to be part of\n",
    "the Generator's computation graph)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Implement the `sample()` method in the `Generator` class within the `hw3/gan.py` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = gen.sample(5, with_grad=False)\n",
    "test.assertSequenceEqual(samples.shape, (5, *x0.shape[1:]))\n",
    "test.assertIsNone(samples.grad_fn)\n",
    "_ = plot.tensors_as_images(samples.cpu())\n",
    "\n",
    "samples = gen.sample(5, with_grad=True)\n",
    "test.assertSequenceEqual(samples.shape, (5, *x0.shape[1:]))\n",
    "test.assertIsNotNone(samples.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "<a id=part3_7></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training GANs is a bit different since we need to train two models simultaneously, each with it's own separate loss function and optimizer. We'll implement the training logic as a function that handles one batch of data\n",
    "and updates both the discriminator and the generator based on it.\n",
    "\n",
    "As mentioned above, GANs are considered hard to train. To get some ideas and tips you can see this [paper](https://arxiv.org/pdf/1606.03498.pdf), this list of [\"GAN hacks\"](https://github.com/soumith/ganhacks) or just do it the hard way :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**:\n",
    "1. Implement the `train_batch` function in the `hw3/gan.py` module.\n",
    "2. Tweak the hyperparameters in the `part3_gan_hyperparam()` function within the `hw3/answers.py` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader\n",
    "from hw3.answers import part3_gan_hyperparams\n",
    "\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# Hyperparams\n",
    "hp = part3_gan_hyperparams()\n",
    "batch_size = hp['batch_size']\n",
    "z_dim = hp['z_dim']\n",
    "\n",
    "# Data\n",
    "dl_train = DataLoader(ds_gwb, batch_size, shuffle=True)\n",
    "im_size = ds_gwb[0][0].shape\n",
    "\n",
    "# Model\n",
    "dsc = gan.Discriminator(im_size).to(device)\n",
    "gen = gan.Generator(z_dim, featuremap_size=4).to(device)\n",
    "\n",
    "# Optimizer\n",
    "def create_optimizer(model_params, opt_params):\n",
    "    opt_params = opt_params.copy()\n",
    "    optimizer_type = opt_params['type']\n",
    "    opt_params.pop('type')\n",
    "    return optim.__dict__[optimizer_type](model_params, **opt_params)\n",
    "dsc_optimizer = create_optimizer(dsc.parameters(), hp['discriminator_optimizer'])\n",
    "gen_optimizer = create_optimizer(gen.parameters(), hp['generator_optimizer'])\n",
    "\n",
    "# Loss\n",
    "def dsc_loss_fn(y_data, y_generated):\n",
    "    return gan.discriminator_loss_fn(y_data, y_generated, hp['data_label'], hp['label_noise'])\n",
    "\n",
    "def gen_loss_fn(y_generated):\n",
    "    return gan.generator_loss_fn(y_generated, hp['data_label'])\n",
    "\n",
    "# Training\n",
    "checkpoint_file = 'checkpoints/gan'\n",
    "checkpoint_file_final = f'{checkpoint_file}_final'\n",
    "if os.path.isfile(f'{checkpoint_file}.pt'):\n",
    "    os.remove(f'{checkpoint_file}.pt')\n",
    "\n",
    "# Show hypers\n",
    "print(hp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**:\n",
    "1. Run the following block to train. It will sample some images from your model every few epochs so you can see the progress.\n",
    "2. When you're satisfied with your results, rename the checkpoints file by adding `_final`. When you run the `main.py` script to generate your submission, the final checkpoints file will be loaded instead of running training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import IPython.display\n",
    "import tqdm\n",
    "from hw3.gan import train_batch\n",
    "\n",
    "num_epochs = 100\n",
    "\n",
    "if os.path.isfile(f'{checkpoint_file_final}.pt'):\n",
    "    print(f'*** Loading final checkpoint file {checkpoint_file_final} instead of training')\n",
    "    num_epochs = 0\n",
    "    gen = torch.load(f'{checkpoint_file_final}.pt', map_location=device)\n",
    "    checkpoint_file = checkpoint_file_final\n",
    "\n",
    "for epoch_idx in range(num_epochs):\n",
    "    # We'll accumulate batch losses and show an average once per epoch.\n",
    "    dsc_losses = []\n",
    "    gen_losses = []\n",
    "    print(f'--- EPOCH {epoch_idx+1}/{num_epochs} ---')\n",
    "    \n",
    "    with tqdm.tqdm(total=len(dl_train.batch_sampler), file=sys.stdout) as pbar:\n",
    "        for batch_idx, (x_data, _) in enumerate(dl_train):\n",
    "            x_data = x_data.to(device)\n",
    "            dsc_loss, gen_loss = train_batch(\n",
    "                dsc, gen,\n",
    "                dsc_loss_fn, gen_loss_fn,\n",
    "                dsc_optimizer, gen_optimizer,\n",
    "                x_data)\n",
    "            dsc_losses.append(dsc_loss)\n",
    "            gen_losses.append(gen_loss)\n",
    "            pbar.update()\n",
    "\n",
    "    dsc_avg_loss, gen_avg_loss = np.mean(dsc_losses), np.mean(gen_losses)\n",
    "    print(f'Discriminator loss: {dsc_avg_loss}')\n",
    "    print(f'Generator loss:     {gen_avg_loss}')\n",
    "        \n",
    "    samples = gen.sample(5, with_grad=False)\n",
    "    fig, _ = plot.tensors_as_images(samples.cpu(), figsize=(6,2))\n",
    "    IPython.display.display(fig)\n",
    "    plt.close(fig)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot images from best or last model\n",
    "if os.path.isfile(f'{checkpoint_file}.pt'):\n",
    "    gen = torch.load(f'{checkpoint_file}.pt', map_location=device)\n",
    "print('*** Images Generated from best model:')\n",
    "samples = gen.sample(n=15, with_grad=False).cpu()\n",
    "fig, _ = plot.tensors_as_images(samples, nrows=3, figsize=(6,6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions\n",
    "<a id=part3_8></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO** Answer the following questions. Write your answers in the appropriate variables in the module `hw3/answers.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cs236605.answers import display_answer\n",
    "import hw3.answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "\n",
    "Explain in detail why during training we sometimes need to maintain gradients when sampling from the GAN,\n",
    "and other times we don't. When are they maintained and why? When are they discarded and why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_answer(hw3.answers.part3_q1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "1. When training a GAN to generate images, should we decide to stop training solely based on the fact that  the Generator loss is below some threshold?\n",
    "Why or why not?\n",
    "\n",
    "2. What does it mean if the discriminator loss remains at a constant value while the generator loss decreases?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_answer(hw3.answers.part3_q2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "Compare the results you got when generating images with the VAE to the GAN results.\n",
    "What's the main difference and what's causing it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_answer(hw3.answers.part3_q3)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}