C1_W5_Lab_2_custom-callbacks (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1OykC-6lI4gv"
   },
   "source": [
    "# Ungraded Lab: Keras custom callbacks\n",
    "A custom callback is a powerful tool to customize the behavior of a Keras model during training, evaluation, or inference. Towards the end of this guide, there will be demos of creating a couple of simple callback applications to get you started on your custom callback."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version:  2.1.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "import datetime\n",
    "import io\n",
    "\n",
    "from PIL import Image\n",
    "from IPython.display import Image as IPyImage\n",
    "import imageio\n",
    "\n",
    "print(\"Version: \", tf.__version__)\n",
    "tf.get_logger().setLevel('INFO')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ct0VCSI2dt3a"
   },
   "outputs": [],
   "source": [
    "# Define the Keras model to add callbacks to\n",
    "def get_model():\n",
    "    model = tf.keras.Sequential()\n",
    "    model.add(tf.keras.layers.Dense(1, activation = 'linear', input_dim = 784))\n",
    "    model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.1), loss='mean_squared_error', metrics=['mae'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ySzdG1IqNgah"
   },
   "source": [
    "Then, load the MNIST data for training and testing from Keras datasets API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fwo9LMKGNPWr"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "# Load example MNIST data and pre-process it\n",
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "x_train = x_train.reshape(60000, 784).astype('float32') / 255\n",
    "x_test = x_test.reshape(10000, 784).astype('float32') / 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kHVK7kceNqH2"
   },
   "source": [
    "Now, define a simple custom callback to track the start and end of every batch of data. During those calls, it prints the index of the current batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-dfuGTMINKRR"
   },
   "outputs": [],
   "source": [
    "class MyCustomCallback(tf.keras.callbacks.Callback):\n",
    "\n",
    "    def on_train_batch_begin(self, batch, logs=None):\n",
    "        print('Training: batch {} begins at {}'.format(batch, datetime.datetime.now().time()))\n",
    "\n",
    "    def on_train_batch_end(self, batch, logs=None):\n",
    "        print('Training: batch {} ends at {}'.format(batch, datetime.datetime.now().time()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z4FTUUIBN3WG"
   },
   "source": [
    "Providing a callback to model methods such as `tf.keras.Model.fit()` ensures the methods are called at those stages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NJV6Tj3sNGzg"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training: batch 0 begins at 06:37:04.668564\n",
      "Training: batch 0 ends at 06:37:04.858114\n",
      "Training: batch 1 begins at 06:37:04.858441\n",
      "Training: batch 1 ends at 06:37:04.859475\n",
      "Training: batch 2 begins at 06:37:04.859709\n",
      "Training: batch 2 ends at 06:37:04.860672\n",
      "Training: batch 3 begins at 06:37:04.860956\n",
      "Training: batch 3 ends at 06:37:04.861797\n"
     ]
    }
   ],
   "source": [
    "model = get_model()\n",
    "_ = model.fit(x_train, y_train,\n",
    "          batch_size=64,\n",
    "          epochs=1,\n",
    "          steps_per_epoch=4,\n",
    "          verbose=0,\n",
    "          callbacks=[MyCustomCallback()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "13n44LVkYQsV"
   },
   "source": [
    "## An overview of callback methods\n",
    "\n",
    "\n",
    "### Common methods for training/testing/predicting\n",
    "For training, testing, and predicting, following methods are provided to be overridden.\n",
    "#### `on_(train|test|predict)_begin(self, logs=None)`\n",
    "Called at the beginning of `fit`/`evaluate`/`predict`.\n",
    "#### `on_(train|test|predict)_end(self, logs=None)`\n",
    "Called at the end of `fit`/`evaluate`/`predict`.\n",
    "#### `on_(train|test|predict)_batch_begin(self, batch, logs=None)`\n",
    "Called right before processing a batch during training/testing/predicting. Within this method, `logs` is a dict with `batch` and `size` available keys, representing the current batch number and the size of the batch.\n",
    "#### `on_(train|test|predict)_batch_end(self, batch, logs=None)`\n",
    "Called at the end of training/testing/predicting a batch. Within this method, `logs` is a dict containing the stateful metrics result.\n",
    "\n",
    "### Training specific methods\n",
    "In addition, for training, following are provided.\n",
    "#### `on_epoch_begin(self, epoch, logs=None)`\n",
    "Called at the beginning of an epoch during training.\n",
    "#### `on_epoch_end(self, epoch, logs=None)`\n",
    "Called at the end of an epoch during training.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SWf3mXYoceCz"
   },
   "source": [
    "### Usage of `logs` dict\n",
    "The `logs` dict contains the loss value, and all the metrics at the end of a batch or epoch. Example includes the loss and mean absolute error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FA_IdseELLKI"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0, Val/Train loss ratio: 1.21\n",
      "Epoch: 1, Val/Train loss ratio: 1.18\n",
      "Epoch: 2, Val/Train loss ratio: 0.79\n"
     ]
    }
   ],
   "source": [
    "callback = tf.keras.callbacks.LambdaCallback(\n",
    "    on_epoch_end=lambda epoch,logs: \n",
    "    print(\"Epoch: {}, Val/Train loss ratio: {:.2f}\".format(epoch, logs[\"val_loss\"] / logs[\"loss\"]))\n",
    ")\n",
    "\n",
    "model = get_model()\n",
    "_ = model.fit(x_train, y_train,\n",
    "          validation_data=(x_test, y_test),\n",
    "          batch_size=64,\n",
    "          epochs=3,\n",
    "          verbose=0, # was 0\n",
    "          callbacks=[callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "u4wIdcF9BjJH"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0, Val/Train loss ratio: 0.56\n",
      "Epoch: 1, Val/Train loss ratio: 0.81\n",
      "Epoch: 2, Val/Train loss ratio: 1.18\n",
      "Stopping training...\n"
     ]
    }
   ],
   "source": [
    "class DetectOverfittingCallback(tf.keras.callbacks.Callback):\n",
    "    def __init__(self, threshold=0.9):\n",
    "        super(DetectOverfittingCallback, self).__init__()\n",
    "        self.threshold = threshold\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        ratio = logs[\"val_loss\"] / logs[\"loss\"]\n",
    "        print(\"Epoch: {}, Val/Train loss ratio: {:.2f}\".format(epoch, ratio))\n",
    "\n",
    "        if ratio > self.threshold:\n",
    "            print(\"Stopping training...\")\n",
    "            self.model.stop_training = True\n",
    "\n",
    "model = get_model()\n",
    "_ = model.fit(x_train, y_train,\n",
    "              validation_data=(x_test, y_test),\n",
    "              batch_size=256,\n",
    "              epochs=3,\n",
    "              verbose=0,\n",
    "              callbacks=[DetectOverfittingCallback()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LbXqvC8FHqeu"
   },
   "source": [
    "Similarly, one can provide callbacks in `evaluate()` calls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mFeFFmDAH7v9"
   },
   "source": [
    "## Custom callback to Visualize predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PbCPq3ewYaln"
   },
   "outputs": [],
   "source": [
    "# Load example MNIST data and pre-process it\n",
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "x_train = x_train.reshape(60000, 784).astype('float32') / 255\n",
    "x_test = x_test.reshape(10000, 784).astype('float32') / 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "3e3MrLVxQSnU"
   },
   "outputs": [],
   "source": [
    "# Visualization utilities\n",
    "plt.rc('font', size=20)\n",
    "plt.rc('figure', figsize=(15, 3))\n",
    "\n",
    "def display_digits(inputs, outputs, ground_truth, epoch, n=10):\n",
    "    plt.clf()\n",
    "\n",
    "    plt.yticks([])\n",
    "    plt.grid(None)\n",
    "    inputs = np.reshape(inputs, [n, 28, 28])\n",
    "    inputs = np.swapaxes(inputs, 0, 1)\n",
    "    inputs = np.reshape(inputs, [28, 28*n])\n",
    "    plt.imshow(inputs)\n",
    "    plt.xticks([28*x+14 for x in range(n)], outputs)\n",
    "    for i,t in enumerate(plt.gca().xaxis.get_ticklabels()):\n",
    "        if outputs[i] == ground_truth[i]: \n",
    "            t.set_color('green') \n",
    "        else: \n",
    "            t.set_color('red')\n",
    "    plt.grid(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "lZIMbL_bW33F"
   },
   "outputs": [],
   "source": [
    "GIF_PATH = './animation.gif'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3J2u7yZzINig"
   },
   "outputs": [],
   "source": [
    "class VisCallback(tf.keras.callbacks.Callback):\n",
    "    def __init__(self, inputs, ground_truth, display_freq=10, n_samples=10):\n",
    "        self.inputs = inputs\n",
    "        self.ground_truth = ground_truth\n",
    "        self.images = []\n",
    "        self.display_freq = display_freq\n",
    "        self.n_samples = n_samples\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        # Randomly sample data\n",
    "        indexes = np.random.choice(len(self.inputs), size=self.n_samples)\n",
    "        X_test, y_test = self.inputs[indexes], self.ground_truth[indexes]\n",
    "        predictions = np.argmax(self.model.predict(X_test), axis=1)\n",
    "\n",
    "        # Plot the digits\n",
    "        display_digits(X_test, predictions, y_test, epoch, n=self.n_samples)\n",
    "\n",
    "        # Save the figure\n",
    "        buf = io.BytesIO()\n",
    "        plt.savefig(buf, format='png')\n",
    "        buf.seek(0)\n",
    "        image = Image.open(buf)\n",
    "        self.images.append(np.array(image))\n",
    "\n",
    "        # Display the digits every 'display_freq' number of epochs\n",
    "        if epoch % self.display_freq == 0:\n",
    "            plt.show()\n",
    "\n",
    "    def on_train_end(self, logs=None):\n",
    "        imageio.mimsave(GIF_PATH, self.images, fps=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AKLRfx9gOPw2"
   },
   "outputs": [],
   "source": [
    "def get_model():\n",
    "    model = tf.keras.Sequential()\n",
    "    model.add(tf.keras.layers.Dense(32, activation='linear', input_dim=784))\n",
    "    model.add(tf.keras.layers.Dense(10, activation='softmax'))\n",
    "    model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=1e-4), loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MeGfW4t8Yttu"
   },
   "outputs": [],
   "source": [
    "model = get_model()\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=64,\n",
    "          epochs=20,\n",
    "          verbose=0,\n",
    "          callbacks=[VisCallback(x_test, y_test)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "xV0BpUuDoa76"
   },
   "outputs": [],
   "source": [
    "SCALE = 60\n",
    "\n",
    "# FYI, the format is set to PNG here to bypass checks for acceptable embeddings\n",
    "IPyImage(GIF_PATH, format='png', width=15 * SCALE, height=3 * SCALE) "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Custom callbacks",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}