C3_W4_Lab_2_CatsDogs_CAM.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "C3_W4_Lab_2_CatsDogs-CAM.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vsnupoudel/AdvancedCvCourses/blob/main/C3_W4_Lab_2_CatsDogs_CAM.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oAuRT75GdLFw"
      },
      "source": [
        "# Ungraded Lab: Cats vs. Dogs Class Activation Maps\n",
        "\n",
        "You will again practice with CAMs in this lab and this time there will only be two classes: Cats and Dogs. You will be revisiting this exercise in this week's programming assignment so it's best if you become familiar with the steps discussed here, particularly in preprocessing the image and building the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9rK94t33nwDC"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zSyMHuCVys-O"
      },
      "source": [
        "import tensorflow_datasets as tfds\n",
        "import tensorflow as tf\n",
        "\n",
        "import keras\n",
        "from keras.models import Sequential,Model\n",
        "from keras.layers import Dense,Conv2D,Flatten,MaxPooling2D,GlobalAveragePooling2D\n",
        "from  keras.utils.vis_utils import  plot_model\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import scipy as sp\n",
        "import cv2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TKmwbLquDRMx"
      },
      "source": [
        "# dir(utils.vis_utils)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Un5nFWgnyem"
      },
      "source": [
        "## Download and Prepare the Dataset\n",
        "\n",
        "We will use the [Cats vs Dogs](https://www.tensorflow.org/datasets/catalog/cats_vs_dogs) dataset and we can load it via Tensorflow Datasets. The images are labeled 0 for cats and 1 for dogs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "01974419yy5W"
      },
      "source": [
        "train_data = tfds.load('cats_vs_dogs', split='train[:80%]', as_supervised=True)\n",
        "validation_data = tfds.load('cats_vs_dogs', split='train[80%:90%]', as_supervised=True)\n",
        "test_data = tfds.load('cats_vs_dogs', split='train[-10%:]', as_supervised=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fIFfQ3I3oc1g"
      },
      "source": [
        "The cell below will preprocess the images and create batches before feeding it to our model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ViiwPuL8aP_A"
      },
      "source": [
        "def augment_images(image, label):\n",
        "  \n",
        "  # cast to float\n",
        "  image = tf.cast(image, tf.float32)\n",
        "  # normalize the pixel values\n",
        "  image = (image/255)\n",
        "  # resize to 300 x 300\n",
        "  image = tf.image.resize(image,(300,300))\n",
        "\n",
        "  return image, label\n",
        "\n",
        "# use the utility function above to preprocess the images\n",
        "augmented_training_data = train_data.map(augment_images)\n",
        "\n",
        "# shuffle and create batches before training\n",
        "train_batches = augmented_training_data.shuffle(1024).batch(32)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xEkePAHippbY"
      },
      "source": [
        "## Build the classifier\n",
        "\n",
        "This will look familiar to you because it is almost identical to the previous model we built. The key difference is the output is just one unit that is sigmoid activated. This is because we're only dealing with two classes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QyCoMd93zpc_"
      },
      "source": [
        "model = Sequential()\n",
        "model.add(Conv2D(16,input_shape=(300,300,3),kernel_size=(3,3),activation='relu',padding='same'))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "\n",
        "model.add(Conv2D(32,kernel_size=(3,3),activation='relu',padding='same'))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "\n",
        "model.add(Conv2D(64,kernel_size=(3,3),activation='relu',padding='same'))\n",
        "model.add(MaxPooling2D(pool_size=(2,2)))\n",
        "\n",
        "model.add(Conv2D(128,kernel_size=(3,3),activation='relu',padding='same'))\n",
        "model.add(GlobalAveragePooling2D())\n",
        "model.add(Dense(1,activation='sigmoid'))\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-k5CLZEOqev3"
      },
      "source": [
        "The loss can be adjusted from last time to deal with just two classes. For that, we pick `binary_crossentropy`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MEqbqodQqeBV"
      },
      "source": [
        "# Training will take around 30 minutes to complete using a GPU. Time for a break!\n",
        "\n",
        "model.compile(loss='binary_crossentropy',metrics=['accuracy'],optimizer=tf.keras.optimizers.RMSprop(lr=0.001))\n",
        "model.fit(train_batches,epochs=25)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_7n-Z8SMrNPc"
      },
      "source": [
        "## Building the CAM model\n",
        "\n",
        "You will follow the same steps as before in generating the class activation maps."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sAcgt746qt19"
      },
      "source": [
        "#  model.layers[-1].get_weights()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aJGvrHIu0Vnt"
      },
      "source": [
        "gap_weights = model.layers[-1].get_weights()[0]\n",
        "gap_weights.shape\n",
        "\n",
        "bias = model.layers[-1].get_weights()[1]\n",
        "\n",
        "cam_model  = Model(inputs=model.input, outputs=(model.layers[-3].output,model.layers[-1].output))\n",
        "cam_model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZcyzeiaO0pQs"
      },
      "source": [
        "def show_cam(image_value, features, results):\n",
        "  '''\n",
        "  Displays the class activation map of an image\n",
        "\n",
        "  Args:\n",
        "    image_value (tensor) -- preprocessed input image with size 300 x 300\n",
        "    features (array) -- features of the image, shape (1, 37, 37, 128)\n",
        "    results (array) -- output of the sigmoid layer\n",
        "  '''\n",
        "\n",
        "  # there is only one image in the batch so we index at `0`\n",
        "  features_for_img = features[0]\n",
        "  prediction = results[0]\n",
        "\n",
        "  # there is only one unit in the output so we get the weights connected to it\n",
        "  class_activation_weights = gap_weights[:,0]\n",
        "\n",
        "  # upsample to the image size\n",
        "  class_activation_features = sp.ndimage.zoom(features_for_img, (300/37, 300/37, 1), order=2)\n",
        "  \n",
        "  # compute the intensity of each feature in the CAM\n",
        "  cam_output  = np.dot(class_activation_features, class_activation_weights) + np.array(bias)\n",
        "\n",
        "  # visualize the results\n",
        "  print(f'sigmoid output: {results}')\n",
        "  print(f\"prediction: {'dog' if round(results[0][0]) else 'cat'}\")\n",
        "  plt.figure(figsize=(8,8))\n",
        "  plt.imshow(cam_output, cmap='jet', alpha=0.75)\n",
        "  plt.imshow(tf.squeeze(image_value), alpha=0.5)\n",
        "  plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fHr2c5qEsKNi"
      },
      "source": [
        "## Testing the Model\n",
        "\n",
        "Let's download a few images and see how the class activation maps look like."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3dlTIG-wh9wV"
      },
      "source": [
        "!wget -O cat1.jpg https://storage.googleapis.com/laurencemoroney-blog.appspot.com/MLColabImages/cat1.jpg\n",
        "!wget -O cat2.jpg https://storage.googleapis.com/laurencemoroney-blog.appspot.com/MLColabImages/cat2.jpg\n",
        "!wget -O catanddog.jpg https://storage.googleapis.com/laurencemoroney-blog.appspot.com/MLColabImages/catanddog.jpg\n",
        "!wget -O dog1.jpg https://storage.googleapis.com/laurencemoroney-blog.appspot.com/MLColabImages/dog1.jpg\n",
        "!wget -O dog2.jpg https://storage.googleapis.com/laurencemoroney-blog.appspot.com/MLColabImages/dog2.jpg"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vcbx69oaiYWZ"
      },
      "source": [
        "# utility function to preprocess an image and show the CAM\n",
        "def convert_and_classify(image):\n",
        "\n",
        "  # load the image\n",
        "  img = cv2.imread(image)\n",
        "\n",
        "  # preprocess the image before feeding it to the model\n",
        "  img = cv2.resize(img, (300,300)) / 255.0\n",
        "\n",
        "  # add a batch dimension because the model expects it\n",
        "  tensor_image = np.expand_dims(img, axis=0)\n",
        "\n",
        "  # get the features and prediction\n",
        "  features,results = cam_model.predict(tensor_image)\n",
        "  \n",
        "  # generate the CAM\n",
        "  show_cam(tensor_image, features, results)\n",
        "\n",
        "convert_and_classify('cat1.jpg')\n",
        "convert_and_classify('cat2.jpg')\n",
        "convert_and_classify('catanddog.jpg')\n",
        "convert_and_classify('dog1.jpg')\n",
        "convert_and_classify('dog2.jpg')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ngA8BiZssCPi"
      },
      "source": [
        "**1st cat classification is a bit random. No eyes or ears**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5OwTrzHa6F9P"
      },
      "source": [
        "Let's also try it with some of the test images before we make some observations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YByJ8J1008Ms"
      },
      "source": [
        "# preprocess the test images\n",
        "augmented_test_data = test_data.map(augment_images)\n",
        "test_batches = augmented_test_data.batch(1)\n",
        "\n",
        "\n",
        "for img, lbl in test_batches.take(5):\n",
        "  print(f\"ground truth: {'dog' if lbl else 'cat'}\")\n",
        "  features,results = cam_model.predict(img)\n",
        "  show_cam(img, features, results)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KCkFD_Us2LvL"
      },
      "source": [
        "If your training reached 80% accuracy, you may notice from the images above that the presence of eyes and nose play a big part in determining a dog, while whiskers and a colar mostly point to a cat. Some can be misclassified based on the presence or absence of these features. This tells us that the model is not yet performing optimally and we need to tweak our process (e.g. add more data, train longer, use a different model, etc)."
      ]
    }
  ]
}