regression_xor_twolayers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ff5888c",
   "metadata": {},
   "source": [
    "Sascha Spors,\n",
    "Professorship Signal Theory and Digital Signal Processing,\n",
    "Institute of Communications Engineering (INT),\n",
    "Faculty of Computer Science and Electrical Engineering (IEF),\n",
    "University of Rostock,\n",
    "Germany\n",
    "\n",
    "# Data Driven Audio Signal Processing - A Tutorial with Computational Examples\n",
    "\n",
    "Winter Semester 2023/24 (Master Course #24512)\n",
    "\n",
    "- lecture: https://github.com/spatialaudio/data-driven-audio-signal-processing-lecture\n",
    "- tutorial: https://github.com/spatialaudio/data-driven-audio-signal-processing-exercise\n",
    "\n",
    "Feel free to contact lecturer frank.schultz@uni-rostock.de"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a199b3df",
   "metadata": {},
   "source": [
    "# XOR with Two-Layer Non-Linear  Model\n",
    "\n",
    "- we use TensorFlow & Keras API\n",
    "- we follow the discussion in the highly recommended textbook of I. Goodfellow, Y. Bengio, A. Courville, \"Deep Learning\". MIT Press, 2016, ch 6.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80a46046",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "print(\n",
    "    \"TF version\",\n",
    "    tf.__version__,\n",
    ")\n",
    "\n",
    "tf.keras.backend.set_floatx(\"float64\")  # we could use double precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9d77047c",
   "metadata": {},
   "outputs": [],
   "source": [
    "verbose = 1  # plot training status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9c96247",
   "metadata": {},
   "outputs": [],
   "source": [
    "# data set consists of the 4 conditions for the XOR logical table\n",
    "X = np.array([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])\n",
    "print(\"X\\n\", X)\n",
    "y = np.array([[0.0], [1.0], [1.0], [0.0]])\n",
    "print(\"y\\n\", y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ad27d96",
   "metadata": {},
   "outputs": [],
   "source": [
    "# a simple XOR non-linear model with two layers is known from the textbook\n",
    "# I. Goodfellow, Y. Bengio, A. Courville, Deep Learning. MIT Press, 2016, ch 6.1\n",
    "# the model parameters are given in the book and it is stated that these\n",
    "# belong to the global minimum for the mean squared error loss function\n",
    "\n",
    "# layer 1 with relu activation and the weights/bias:\n",
    "wl1 = np.array([[1, 1], [1, 1]])\n",
    "bl1 = np.array([[0], [-1]])\n",
    "# layer 2 with linear activation an dthe weights/bias:\n",
    "wl2 = np.array([[1], [-2]])\n",
    "bl2 = np.array([[0]])\n",
    "\n",
    "# we could calc model predictions on the data in X\n",
    "# layer 1 with two perceptrons: apply weights / bias\n",
    "z1 = wl1.T @ X.T + bl1  # transpose input to be TF compatible\n",
    "# layer 1 with two perceptrons: apply relu activation\n",
    "z1[z1 < 0] = 0\n",
    "# layer 2 with one perceptron: apply weights / bias\n",
    "z2 = wl2.T @ z1 + bl2\n",
    "# layer 2 with one perceptron: apply linear activation\n",
    "y_pred = z2.T  # transpose output to be TF compatible\n",
    "print(y_pred)\n",
    "print(y == y_pred)  # check true and predicted data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8294af6",
   "metadata": {},
   "source": [
    "## Tensor Flow Model\n",
    "\n",
    "The model is actually not easy to train to the global minimum, as it is unusual to train a binary classification problem with MSE loss and linear activation (which is rather typical for regression tasks).\n",
    "\n",
    "So, we actually expect two numbers, 0 and 1, as model output. However, the linear activation yields real numbers as model output, which in the optimum case are 0 and 1, but for not optimum trained models they might be very close to 0 and 1 or even completely 'wrong'. So, the model needs to be trained exactly to the above given weights, to have the intended binary classification characteristics.\n",
    "\n",
    "This is a nice toy example to see what model training can (not) do on a rather simple problem. We should spend some time to really understand, how the model output is calculated, i.e. how the model prediction works. If we got this, we are ready to work with larger models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1c792c68",
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 2**8\n",
    "batch_size = X.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "842503d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam()\n",
    "loss = keras.losses.MeanSquaredError()\n",
    "metrics = [keras.metrics.MeanSquaredError()]\n",
    "model = keras.Sequential()\n",
    "model.add(keras.Input(shape=(2,)))\n",
    "model.add(keras.layers.Dense(2, activation=\"relu\"))\n",
    "model.add(keras.layers.Dense(1, activation=\"linear\"))\n",
    "model.compile(optimizer=optimizer, loss=loss, metrics=metrics)\n",
    "tw = 0\n",
    "for w in model.trainable_weights:\n",
    "    print(w)\n",
    "    tw += np.prod(w.shape)\n",
    "print(\"\\ntrainable_weights:\", tw, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc525703",
   "metadata": {},
   "source": [
    "We could init the model parameters close (e.g. `offset=1e-2`) or even exact (`offset=0`) to the optimum parameters that are known above. To use this set `if True:`.\n",
    "\n",
    "With a robust gradient descent method, such as Adam, training should get close to or remain at the optimum parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0198cd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "wl1 = np.array([[1, 1], [1, 1]])\n",
    "bl1 = np.array([0, -1])\n",
    "wl2 = np.array([[1], [-2]])\n",
    "bl2 = np.array([0])\n",
    "if True:\n",
    "    offset = 1e-5\n",
    "    model.set_weights([wl1 + offset, bl1 + offset, wl2 + offset, bl2])\n",
    "model.get_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f4e4f44",
   "metadata": {},
   "source": [
    "### Train / Fit the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43d50b98",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(X, y, epochs=epochs, batch_size=batch_size, verbose=verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd901f4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.summary())\n",
    "print(\"model weights\\n\", model.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24158b77",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = model.evaluate(X, y, batch_size=X.shape[0], verbose=False)\n",
    "y_pred = model.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b17e2fdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.loss(y, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b7edb9f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_class(y):\n",
    "    y[y < 0.5], y[y >= 0.5] = 0, 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a59cc9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"real numbered model ouput\\n\", y_pred)\n",
    "predict_class(y_pred)  # real numbered ouput -> classification (0,1) output\n",
    "print(\"classification ouput\\n\", y_pred)\n",
    "print(\"check true vs. predicted:\\n\", y == y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "651b1eff",
   "metadata": {},
   "source": [
    "## Copyright\n",
    "\n",
    "- the notebooks are provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources)\n",
    "- feel free to use the notebooks for your own purposes\n",
    "- the text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/)\n",
    "- the code of the IPython examples is licensed under the [MIT license](https://opensource.org/licenses/MIT)\n",
    "- please attribute the work as follows: *Frank Schultz, Data Driven Audio Signal Processing - A Tutorial Featuring Computational Examples, University of Rostock* ideally with relevant file(s), github URL https://github.com/spatialaudio/data-driven-audio-signal-processing-exercise, commit number and/or version tag, year."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myddasp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}