C2_W2_Lab_2_training-categorical.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/https-deeplearning-ai/tensorflow-3-public/blob/main/Course%202%20-%20Custom%20Training%20loops%2C%20Gradients%20and%20Distributed%20Training/Week%202%20-%20Simple%20Custom%20Training/C2_W2_Lab_2_training-categorical.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XNFVMtUhMt7l"
   },
   "source": [
    "# Fashion MNIST using Custom Training Loop\n",
    "In this ungraded lab, you will build a custom training loop including a validation loop so as to train a model on the [Fashion MNIST](https://research.zalando.com/welcome/mission/research-projects/fashion-mnist/) dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NenrAcsiM7Zl"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JkMXve8XuN5X"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "  # %tensorflow_version only exists in Colab.\n",
    "  %tensorflow_version 2.x\n",
    "except Exception:\n",
    "  pass\n",
    "  \n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Dense, Input\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import itertools\n",
    "from tqdm import tqdm\n",
    "import tensorflow_datasets as tfds\n",
    "import matplotlib.ticker as mticker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wtcG5Of7M-IV"
   },
   "source": [
    "## Load and Preprocess Data\n",
    "You will load the [Fashion MNIST](https://research.zalando.com/welcome/mission/research-projects/fashion-mnist/) dataset using Tensorflow Datasets. This dataset has 28 x 28 grayscale images of articles of clothing belonging to 10 clases.\n",
    "\n",
    "Here you are going to use the training and testing splits of the data. Testing split will be used for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K1qm4y2FmvWJ"
   },
   "outputs": [],
   "source": [
    "train_data, info = tfds.load(\"fashion_mnist\", split = \"train\", with_info = True, data_dir='./data/', download=False)\n",
    "test_data = tfds.load(\"fashion_mnist\", split = \"test\", data_dir='./data/', download=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DbliOEMHNiug"
   },
   "outputs": [],
   "source": [
    "class_names = [\"T-shirt/top\", \"Trouser/pants\", \"Pullover shirt\", \"Dress\", \"Coat\", \"Sandal\", \"Shirt\", \"Sneaker\", \"Bag\", \"Ankle boot\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sn718Y0LOIaY"
   },
   "source": [
    "Next, you normalize the images by dividing them by 255.0 so as to make the pixels fall in the range (0, 1). You also reshape the data so as to flatten the 28 x 28 pixel array into a flattened 784 pixel array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oxwzgw3BmkoD"
   },
   "outputs": [],
   "source": [
    "def format_image(data):        \n",
    "    image = data[\"image\"]\n",
    "    image = tf.reshape(image, [-1])\n",
    "    image = tf.cast(image, 'float32')\n",
    "    image = image / 255.0\n",
    "    return image, data[\"label\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c26dmIL5nmNU"
   },
   "outputs": [],
   "source": [
    "train_data = train_data.map(format_image)\n",
    "test_data = test_data.map(format_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ws3N-uOgOnMf"
   },
   "source": [
    "Now you shuffle and batch your training and test datasets before feeding them to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "h9qdsNPen5-F"
   },
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "train = train_data.shuffle(buffer_size=1024).batch(batch_size)\n",
    "\n",
    "test =  test_data.batch(batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fuCf0s7eOxKQ"
   },
   "source": [
    "## Define the Model\n",
    "You are using a simple model in this example. You use Keras Functional API to connect two dense layers. The final layer is a softmax that outputs one of the 10 classes since this is a multi class classification problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HU3qcM9WBcMh"
   },
   "outputs": [],
   "source": [
    "def base_model():\n",
    "  inputs = tf.keras.Input(shape=(784,), name='digits')\n",
    "  x = tf.keras.layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
    "  x = tf.keras.layers.Dense(64, activation='relu', name='dense_2')(x)\n",
    "  outputs = tf.keras.layers.Dense(10, activation='softmax', name='predictions')(x)\n",
    "  model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "  return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mxaHy1NYPGSb"
   },
   "source": [
    "## Define Optimizer and Loss Function\n",
    "\n",
    "You have chosen `adam` optimizer and sparse categorical crossentropy loss for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "v5B3vh6fs84i"
   },
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam()\n",
    "loss_object = tf.keras.losses.SparseCategoricalCrossentropy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "w1fJsdYIPTb8"
   },
   "source": [
    "## Define Metrics\n",
    "\n",
    "You will also define metrics so that your training loop can update and display them. Here you are using `SparseCategoricalAccuracy`defined in `tf.keras.metrics` since the problem at hand is a multi class classification problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7Pa_x-5-CH_V"
   },
   "outputs": [],
   "source": [
    "train_acc_metric = tf.keras.metrics.SparseCategoricalAccuracy()\n",
    "val_acc_metric = tf.keras.metrics.SparseCategoricalAccuracy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HVFI54MpQUDp"
   },
   "source": [
    "## Building Training Loop\n",
    "In this section you build your training loop consisting of training and validation sequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9n90NqQmQfJv"
   },
   "source": [
    "The core of training is using the model to calculate the logits on specific set of inputs and compute loss (in this case **sparse categorical crossentropy**) by comparing the predicted outputs to the true outputs. You then update the trainable weights using the optimizer algorithm chosen. Optimizer algorithm requires your computed loss and partial derivatives of loss with respect to each of the trainable weights to make updates to the same.\n",
    "\n",
    "You use gradient tape to calculate the gradients and then update the model trainable weights using the optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MMPe25Dstn0v"
   },
   "outputs": [],
   "source": [
    "def apply_gradient(optimizer, model, x, y):\n",
    "  with tf.GradientTape() as tape:\n",
    "    logits = model(x)\n",
    "    loss_value = loss_object(y_true=y, y_pred=logits)\n",
    "  \n",
    "  gradients = tape.gradient(loss_value, model.trainable_weights)\n",
    "  optimizer.apply_gradients(zip(gradients, model.trainable_weights))\n",
    "  \n",
    "  return logits, loss_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KZnuYWn8QrU9"
   },
   "source": [
    "This function performs training during one epoch. You run through all batches of training data in each epoch to make updates to trainable weights using your previous function. You can see that we also call update_state on your metrics to accumulate the value of your metrics.\n",
    "You are displaying a progress bar to indicate completion of training in each epoch. Here you use tqdm for displaying the progress bar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3fHoh_hgz2PC"
   },
   "outputs": [],
   "source": [
    "def train_data_for_one_epoch():\n",
    "  losses = []\n",
    "  pbar = tqdm(total=len(list(enumerate(train))), position=0, leave=True, bar_format='{l_bar}{bar}| {n_fmt}/{total_fmt} ')\n",
    "  for step, (x_batch_train, y_batch_train) in enumerate(train):\n",
    "      logits, loss_value = apply_gradient(optimizer, model, x_batch_train, y_batch_train)\n",
    "      \n",
    "      losses.append(loss_value)\n",
    "      \n",
    "      train_acc_metric(y_batch_train, logits)\n",
    "      pbar.set_description(\"Training loss for step %s: %.4f\" % (int(step), float(loss_value)))\n",
    "      pbar.update()\n",
    "  return losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kBZyXnuUQxVn"
   },
   "source": [
    "At the end of each epoch you have to validate the model on the test dataset. The following function calculates the loss on test dataset and updates the states of the validation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5gLJyAJE0YRc"
   },
   "outputs": [],
   "source": [
    "def perform_validation():\n",
    "  losses = []\n",
    "  for x_val, y_val in test:\n",
    "      val_logits = model(x_val)\n",
    "      val_loss = loss_object(y_true=y_val, y_pred=val_logits)\n",
    "      losses.append(val_loss)\n",
    "      val_acc_metric(y_val, val_logits)\n",
    "  return losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rh74YX2nQ2i1"
   },
   "source": [
    "Next you define the training loop that runs through the training samples repeatedly over a fixed number of epochs. Here you combine the functions you built earlier to establish the following flow:\n",
    "1. Perform training over all batches of training data.\n",
    "2. Get values of metrics.\n",
    "3. Perform validation to calculate loss and update validation metrics on test data.\n",
    "4. Reset the metrics at the end of epoch.\n",
    "5. Display statistics at the end of each epoch.\n",
    "\n",
    "**Note** : You also calculate the training and validation losses for the whole epoch at the end of the epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OOO1x3VyuPUV"
   },
   "outputs": [],
   "source": [
    "\n",
    "model = base_model()\n",
    "\n",
    "# Iterate over epochs.\n",
    "epochs = 10\n",
    "epochs_val_losses, epochs_train_losses = [], []\n",
    "for epoch in range(epochs):\n",
    "  print('Start of epoch %d' % (epoch,))\n",
    "  \n",
    "  losses_train = train_data_for_one_epoch()\n",
    "  train_acc = train_acc_metric.result()\n",
    "\n",
    "  losses_val = perform_validation()\n",
    "  val_acc = val_acc_metric.result()\n",
    "\n",
    "  losses_train_mean = np.mean(losses_train)\n",
    "  losses_val_mean = np.mean(losses_val)\n",
    "  epochs_val_losses.append(losses_val_mean)\n",
    "  epochs_train_losses.append(losses_train_mean)\n",
    "\n",
    "  print('\\n Epoch %s: Train loss: %.4f  Validation Loss: %.4f, Train Accuracy: %.4f, Validation Accuracy %.4f' % (epoch, float(losses_train_mean), float(losses_val_mean), float(train_acc), float(val_acc)))\n",
    "  \n",
    "  train_acc_metric.reset_states()\n",
    "  val_acc_metric.reset_states()\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ltKpkpzKK_Up"
   },
   "source": [
    "## Evaluate Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tfGc-gMPLCDn"
   },
   "source": [
    "### Plots for Evaluation\n",
    "You plot the progress of loss as training proceeds over number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NjzIlGipJwC_"
   },
   "outputs": [],
   "source": [
    "def plot_metrics(train_metric, val_metric, metric_name, title, ylim=5):\n",
    "  plt.title(title)\n",
    "  plt.ylim(0,ylim)\n",
    "  plt.gca().xaxis.set_major_locator(mticker.MultipleLocator(1))\n",
    "  plt.plot(train_metric,color='blue',label=metric_name)\n",
    "  plt.plot(val_metric,color='green',label='val_' + metric_name)\n",
    "\n",
    "plot_metrics(epochs_train_losses, epochs_val_losses, \"Loss\", \"Loss\", ylim=1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "adpLKxFfZzTD"
   },
   "source": [
    "This function displays a row of images with their predictions and true labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_3PJnCRIO8bM"
   },
   "outputs": [],
   "source": [
    "# utility to display a row of images with their predictions and true labels\n",
    "def display_images(image, predictions, labels, title, n):\n",
    "\n",
    "  display_strings = [str(i) + \"\\n\\n\" + str(j) for i, j in zip(predictions, labels)] \n",
    "\n",
    "  plt.figure(figsize=(17,3))\n",
    "  plt.title(title)\n",
    "  plt.yticks([])\n",
    "  plt.xticks([28*x+14 for x in range(n)], display_strings)\n",
    "  plt.grid(None)\n",
    "  image = np.reshape(image, [n, 28, 28])\n",
    "  image = np.swapaxes(image, 0, 1)\n",
    "  image = np.reshape(image, [28, 28*n])\n",
    "  plt.imshow(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "505DveJuaCNO"
   },
   "source": [
    "You make predictions on the test dataset and plot the images with their true and predicted values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5ybveIIcPgVr"
   },
   "outputs": [],
   "source": [
    "test_inputs = test_data.batch(batch_size=1000001)\n",
    "x_batches, y_pred_batches, y_true_batches = [], [], []\n",
    "\n",
    "for x, y in test_inputs:\n",
    "  y_pred = model(x)\n",
    "  y_pred_batches = y_pred.numpy()\n",
    "  y_true_batches = y.numpy()\n",
    "  x_batches = x.numpy()\n",
    "\n",
    "indexes = np.random.choice(len(y_pred_batches), size=10)\n",
    "images_to_plot = x_batches[indexes]\n",
    "y_pred_to_plot = y_pred_batches[indexes]\n",
    "y_true_to_plot = y_true_batches[indexes]\n",
    "\n",
    "y_pred_labels = [class_names[np.argmax(sel_y_pred)] for sel_y_pred in y_pred_to_plot]\n",
    "y_true_labels = [class_names[sel_y_true] for sel_y_true in y_true_to_plot]\n",
    "display_images(images_to_plot, y_pred_labels, y_true_labels, \"Predicted and True Values\", 10)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "Training Categorical.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}