ModelingPT1_Base.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f0e70ac4",
   "metadata": {},
   "source": [
    "# Insert Title Here\n",
    "**DATA102 S11 Group 3*\n",
    "- Banzon, Beatrice Elaine B.\n",
    "- Buitre, Cameron\n",
    "- Marcelo, Andrea Jean C.\n",
    "- Navarro, Alyssa Riantha R.\n",
    "- Vicente, Francheska Josefa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ee8a36f",
   "metadata": {},
   "source": [
    "## Requirements and Imports\n",
    "\n",
    "Before starting, the relevant libraries and files in building and training the model should be loaded into the notebook first."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02257198",
   "metadata": {},
   "source": [
    "### Imports\n",
    "Several libraries are required to perform a thorough analysis of the dataset. Each of these libraries will be imported and described below:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87d6f11",
   "metadata": {},
   "source": [
    "**Basic Libraries**\n",
    "\n",
    "Import `numpy`, `pandas`, and `datasets`.\n",
    "\n",
    "* `numpy` contains a large collection of mathematical functions\n",
    "* `pandas` contains functions that are designed for data manipulation and data analysis\n",
    "* `datasets` contains functions that allow easier pre-processing for datasets and smart caching for easier loading of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "83b421a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83ab390a",
   "metadata": {},
   "source": [
    "**Machine Learning Libraries**\n",
    "\n",
    "The `train_test_split` is a function that allows the dataset to be split into two randomly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "99272409",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86704d1a",
   "metadata": {},
   "source": [
    "Meanwhile, the following imports are used to create the dataset :\n",
    "* `torch` library is an open source ML library for deep neural network creation\n",
    "* `Dataset` and `DataLoader` are two data primitives that makes loading and using dataset easier\n",
    "* `RandomSampler` and `SequentialSampler` are samplers that is used by the `DataLoader`\n",
    "* `ProgressBarBase` and `RichProgressBar` are components that shows the progress bar of training the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4873b00e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler\n",
    "from pytorch_lightning.callbacks import ProgressBarBase, RichProgressBar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d649634",
   "metadata": {},
   "source": [
    "The next imports are from `transformers`, which contains pre-trained models and tokenizers that can be fine-tuned.\n",
    "* `AutoTokenizer` automatically creates the tokenizer based on the architecture passed\n",
    "* `AutoModelForSequenceClassification` automatically instantiates a sequence classification model based on the type of model passed\n",
    "* `TrainerCallback` is an object that determines how the training loop will behave\n",
    "* `TrainingArguments` is a dataclass that allows the customization of the arguments in training\n",
    "* `Trainer` is a class that has a complete training and validation loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2c7a2e84",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import (AutoTokenizer, AutoModelForSequenceClassification, \n",
    "                          TrainerCallback, TrainingArguments, Trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832cde81",
   "metadata": {},
   "source": [
    "On the other hand, these classes computes and visualizes the different scores about how well a model works.\n",
    "* `f1_score` computes the balanced F-score by comparing the actual classes and the predicted classes\n",
    "* `hamming_loss` computes the fraction of labels that were incorrectly labeled by the model\n",
    "* `accuracy_score` computes the accuracy by determining how many classes were correctly predicted\n",
    "* `EvalPrediction` is an object in transformers that holds the prediction of the model and the target output\n",
    "* `evaluate` is a libray that is used to evaluate and compare metrics\n",
    "* `load_metric` is a function in the datasets library that allows different metrics to be loaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9eee55fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score, hamming_loss, accuracy_score\n",
    "from transformers import EvalPrediction\n",
    "import evaluate\n",
    "from datasets import load_metric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fc9bab1",
   "metadata": {},
   "source": [
    "Next, `optuna` is used to tune the hyperparameters of machine learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c42ff241",
   "metadata": {},
   "outputs": [],
   "source": [
    "import optuna"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96a118d6",
   "metadata": {},
   "source": [
    "Last, `pickle` is a module that can serialize and deserialize objects. In this notebook, it is used to save and load models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e848d025",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbb1e919",
   "metadata": {},
   "source": [
    "### Datasets and Files\n",
    "To train the BERT and RoBERTa model, let us load the cleaned dataset using the [`read_csv`](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e7300021",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Ayon sa TheWrap.com, naghain ng kaso si Krupa,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Kilala rin ang singer sa pagkumpas ng kanyang ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>BLANTYRE, Malawi (AP) -- Bumiyahe patungong Ma...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>Kasama sa programa ang pananalangin, bulaklak ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Linisin ang Friendship Department dahil dadala...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23130</th>\n",
       "      <td>0</td>\n",
       "      <td>The winner of the special election in Cavite t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23131</th>\n",
       "      <td>0</td>\n",
       "      <td>The remains of four people inside the Cessna p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23132</th>\n",
       "      <td>0</td>\n",
       "      <td>A Kabataan Party-list representative visited t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23133</th>\n",
       "      <td>0</td>\n",
       "      <td>The Philippine Coast Guard is expected to have...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23134</th>\n",
       "      <td>0</td>\n",
       "      <td>National Bureau of Investigation-National Capi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23135 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       label                                               text\n",
       "0          0  Ayon sa TheWrap.com, naghain ng kaso si Krupa,...\n",
       "1          0  Kilala rin ang singer sa pagkumpas ng kanyang ...\n",
       "2          0  BLANTYRE, Malawi (AP) -- Bumiyahe patungong Ma...\n",
       "3          0  Kasama sa programa ang pananalangin, bulaklak ...\n",
       "4          0  Linisin ang Friendship Department dahil dadala...\n",
       "...      ...                                                ...\n",
       "23130      0  The winner of the special election in Cavite t...\n",
       "23131      0  The remains of four people inside the Cessna p...\n",
       "23132      0  A Kabataan Party-list representative visited t...\n",
       "23133      0  The Philippine Coast Guard is expected to have...\n",
       "23134      0  National Bureau of Investigation-National Capi...\n",
       "\n",
       "[23135 rows x 2 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv ('cleaned_data.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cca6183",
   "metadata": {},
   "source": [
    "Before we start directly dealing with the data, we will set the **device** on where the model will run. If there is an existence of a CUDA-enabled device, it will automatically pick CUDA as its device. Otherwise, it will run on the CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "060c830c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74bc7c89",
   "metadata": {},
   "source": [
    "## Preparing data for Feature Engineering\n",
    "\n",
    "Before creating the features that the BERT and RoBERTa models will use for training, there are two steps that we must first do: (1) splitting the dataset into the train, val, and test sets, and (2) transforming our [`DataFrame`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) into a [`Dataset`](https://pypi.org/project/datasets/). This would allow us to utilize the data for the training more easily."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bea6faa1",
   "metadata": {},
   "source": [
    "### Splitting the Dataset into Train, Val, and Test Split\n",
    "Let us first define the **X** (input) and **y** (target/output) of our model. This is done to allow the stratifying of the data when it is split into the train, val and test.\n",
    "\n",
    "The **X** (input) can be retrieved by getting the `text` column in the original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ef2d8078",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        Ayon sa TheWrap.com, naghain ng kaso si Krupa,...\n",
       "1        Kilala rin ang singer sa pagkumpas ng kanyang ...\n",
       "2        BLANTYRE, Malawi (AP) -- Bumiyahe patungong Ma...\n",
       "3        Kasama sa programa ang pananalangin, bulaklak ...\n",
       "4        Linisin ang Friendship Department dahil dadala...\n",
       "                               ...                        \n",
       "23130    The winner of the special election in Cavite t...\n",
       "23131    The remains of four people inside the Cessna p...\n",
       "23132    A Kabataan Party-list representative visited t...\n",
       "23133    The Philippine Coast Guard is expected to have...\n",
       "23134    National Bureau of Investigation-National Capi...\n",
       "Name: text, Length: 23135, dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df ['text']\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25cfa14a",
   "metadata": {},
   "source": [
    "Meanwhile, the **y** value (i.e., the value that we would be \"feeding\" our models) is the `class` column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "83e11442",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        0\n",
       "1        0\n",
       "2        0\n",
       "3        0\n",
       "4        0\n",
       "        ..\n",
       "23130    0\n",
       "23131    0\n",
       "23132    0\n",
       "23133    0\n",
       "23134    0\n",
       "Name: label, Length: 23135, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = df ['label']\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c02546fd",
   "metadata": {},
   "source": [
    "Now that we have declared the input and the target output of our models, we can use the [`train_test_split`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function to divide the dataset into two splits. Some things to note are: (1) the split is stratified based on the **y values**, (2) the value of the random state was set to 42 for reproducibility, and (3) the dataset is shuffled.\n",
    "\n",
    "First, let us create the train and test set. The test set is made up of 20% of the original dataset, which infers that the second split is 80% of the original. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "314aab70",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2,\n",
    "                                                    stratify = y,\n",
    "                                                    random_state = 42, \n",
    "                                                    shuffle = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2076432c",
   "metadata": {},
   "source": [
    "Second, we will be splitting the remaining 80% of the original dataset into two: the train and val sets. The train set will be 90% of the second split, while the val set will be 10% of it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9aeb7acd",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(X_train, \n",
    "                                                  y_train, \n",
    "                                                  test_size = 0.1,\n",
    "                                                  stratify = y_train,\n",
    "                                                  random_state = 42, \n",
    "                                                  shuffle = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e037a8",
   "metadata": {},
   "source": [
    "To check if the shapes of the input and output are the same, we will be looking at the shapes of the resulting DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e2b8deec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train\n",
      "Input  shape:  (16657,)\n",
      "Output shape:  (16657,) \n",
      "\n",
      "Val\n",
      "Input  shape:  (1851,)\n",
      "Output shape:  (1851,) \n",
      "\n",
      "Test\n",
      "Input  shape:  (4627,)\n",
      "Output shape:  (4627,) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('Train')\n",
    "print('Input  shape: ', X_train.shape)\n",
    "print('Output shape: ', y_train.shape, '\\n')\n",
    "\n",
    "print('Val')\n",
    "print('Input  shape: ', X_val.shape)\n",
    "print('Output shape: ', y_val.shape, '\\n')\n",
    "\n",
    "print('Test')\n",
    "print('Input  shape: ', X_test.shape)\n",
    "print('Output shape: ', y_test.shape, '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "869e7e29",
   "metadata": {},
   "source": [
    "As we have already split the data into three (i.e., train, val, test) sets, we can now combine the **X** and **y** values per set through the use of [`concat`](https://pandas.pydata.org/docs/reference/api/pandas.concat.html). This is done for easier tokenizing of the dataset when using BERT and RoBERTa. In addition, using the [`reset_index`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) function, we would also be resetting the index to make it sequential starting from 0. \n",
    "\n",
    "First, we would concatenate the **X** and **y** values of the train set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b8cd9f3e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Kahit nagluwag na noong Pebrero sa COVID-19 re...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Former Interior Undersecretary Jonathan Malaya...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Saker Message: No current Saker messages. Russ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Japan’s Lost Black Hole Satellite Took This LA...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Home › SCIENCE &amp; TECHNOLOGY › MOBILE PASSES DE...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16652</th>\n",
       "      <td>TRUTH: No Apartheid in Israel, Says Black Sout...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16653</th>\n",
       "      <td>Mark Warner, Virginia Ron Wyden, Oregon   In b...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16654</th>\n",
       "      <td>WASHINGTON -- President Barack Obama joined ne...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16655</th>\n",
       "      <td>Ayon sa TheWrap.com, naghain ng kaso si Krupa,...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16656</th>\n",
       "      <td>Lawmakers on Wednesday confirmed the appointme...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>16657 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text  label\n",
       "0      Kahit nagluwag na noong Pebrero sa COVID-19 re...      0\n",
       "1      Former Interior Undersecretary Jonathan Malaya...      0\n",
       "2      Saker Message: No current Saker messages. Russ...      1\n",
       "3      Japan’s Lost Black Hole Satellite Took This LA...      1\n",
       "4      Home › SCIENCE & TECHNOLOGY › MOBILE PASSES DE...      1\n",
       "...                                                  ...    ...\n",
       "16652  TRUTH: No Apartheid in Israel, Says Black Sout...      1\n",
       "16653  Mark Warner, Virginia Ron Wyden, Oregon   In b...      1\n",
       "16654  WASHINGTON -- President Barack Obama joined ne...      0\n",
       "16655  Ayon sa TheWrap.com, naghain ng kaso si Krupa,...      0\n",
       "16656  Lawmakers on Wednesday confirmed the appointme...      0\n",
       "\n",
       "[16657 rows x 2 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = pd.concat([X_train, y_train], axis = 1).reset_index(drop = True)\n",
    "train_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "290fee6f",
   "metadata": {},
   "source": [
    "Next, let us combine for the val (i.e., validation) set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "437f17b9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LABUAN BAJO — President Ferdinand \"Bongbong\" M...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CorbettReport.com November 8, 2016   In Dougla...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>WASHINGTON - The United States scrambled F-16 ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Miami (CNN) There were few softballs Wednesday...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Trump and Sanders get all the attention for th...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1846</th>\n",
       "      <td>The owner of a convenience store in Malvar, Ba...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1847</th>\n",
       "      <td>While the full field of Republican presidentia...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1848</th>\n",
       "      <td>WASHINGTON - US President Joe Biden called on ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1849</th>\n",
       "      <td>Baltimore leaders say the first night of the c...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1850</th>\n",
       "      <td>How Hillary Clinton Locked Up The Democratic N...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1851 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   text  label\n",
       "0     LABUAN BAJO — President Ferdinand \"Bongbong\" M...      0\n",
       "1     CorbettReport.com November 8, 2016   In Dougla...      1\n",
       "2     WASHINGTON - The United States scrambled F-16 ...      0\n",
       "3     Miami (CNN) There were few softballs Wednesday...      0\n",
       "4     Trump and Sanders get all the attention for th...      0\n",
       "...                                                 ...    ...\n",
       "1846  The owner of a convenience store in Malvar, Ba...      0\n",
       "1847  While the full field of Republican presidentia...      0\n",
       "1848  WASHINGTON - US President Joe Biden called on ...      0\n",
       "1849  Baltimore leaders say the first night of the c...      0\n",
       "1850  How Hillary Clinton Locked Up The Democratic N...      0\n",
       "\n",
       "[1851 rows x 2 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_df = pd.concat([X_val, y_val], axis = 1).reset_index(drop = True)\n",
    "val_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bada419",
   "metadata": {},
   "source": [
    "Last, we would also be doing these same steps to the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5cbabe1b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OTTAWA, Canada - Canada's government struck a ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Editor’s note: This press release is sponsored...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(CNN) Donald Trump and Bernie Sanders are conf...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The Alliance of Concerned Teachers (ACT) calle...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Manila's City Engineering Office has received ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4622</th>\n",
       "      <td>The House Committee on Human Rights has approv...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4623</th>\n",
       "      <td>The Metro Rail Transit Line 3 (MRT-3) has impl...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4624</th>\n",
       "      <td>October 28, 2016 at 9:00 PM   Why would Putin ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4625</th>\n",
       "      <td>PHNOM PENH — An eleven-year-old girl in Cambod...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4626</th>\n",
       "      <td>Justice Secretary Jesus Crispin Remulla on Thu...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4627 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   text  label\n",
       "0     OTTAWA, Canada - Canada's government struck a ...      0\n",
       "1     Editor’s note: This press release is sponsored...      0\n",
       "2     (CNN) Donald Trump and Bernie Sanders are conf...      0\n",
       "3     The Alliance of Concerned Teachers (ACT) calle...      0\n",
       "4     Manila's City Engineering Office has received ...      0\n",
       "...                                                 ...    ...\n",
       "4622  The House Committee on Human Rights has approv...      0\n",
       "4623  The Metro Rail Transit Line 3 (MRT-3) has impl...      0\n",
       "4624  October 28, 2016 at 9:00 PM   Why would Putin ...      1\n",
       "4625  PHNOM PENH — An eleven-year-old girl in Cambod...      0\n",
       "4626  Justice Secretary Jesus Crispin Remulla on Thu...      0\n",
       "\n",
       "[4627 rows x 2 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = pd.concat([X_test, y_test], axis = 1).reset_index(drop = True)\n",
    "test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c3fbdf",
   "metadata": {},
   "source": [
    "Next, to ensure that there would be no **NaN** data when we train our models, we would be dropping the rows that has at least one **na** value using the function [`dropna`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.dropna.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "421ec33b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Kahit nagluwag na noong Pebrero sa COVID-19 re...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Former Interior Undersecretary Jonathan Malaya...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Saker Message: No current Saker messages. Russ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Japan’s Lost Black Hole Satellite Took This LA...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Home › SCIENCE &amp; TECHNOLOGY › MOBILE PASSES DE...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16652</th>\n",
       "      <td>TRUTH: No Apartheid in Israel, Says Black Sout...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16653</th>\n",
       "      <td>Mark Warner, Virginia Ron Wyden, Oregon   In b...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16654</th>\n",
       "      <td>WASHINGTON -- President Barack Obama joined ne...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16655</th>\n",
       "      <td>Ayon sa TheWrap.com, naghain ng kaso si Krupa,...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16656</th>\n",
       "      <td>Lawmakers on Wednesday confirmed the appointme...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>16657 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text  label\n",
       "0      Kahit nagluwag na noong Pebrero sa COVID-19 re...      0\n",
       "1      Former Interior Undersecretary Jonathan Malaya...      0\n",
       "2      Saker Message: No current Saker messages. Russ...      1\n",
       "3      Japan’s Lost Black Hole Satellite Took This LA...      1\n",
       "4      Home › SCIENCE & TECHNOLOGY › MOBILE PASSES DE...      1\n",
       "...                                                  ...    ...\n",
       "16652  TRUTH: No Apartheid in Israel, Says Black Sout...      1\n",
       "16653  Mark Warner, Virginia Ron Wyden, Oregon   In b...      1\n",
       "16654  WASHINGTON -- President Barack Obama joined ne...      0\n",
       "16655  Ayon sa TheWrap.com, naghain ng kaso si Krupa,...      0\n",
       "16656  Lawmakers on Wednesday confirmed the appointme...      0\n",
       "\n",
       "[16657 rows x 2 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = train_df.dropna (how = 'any')\n",
    "train_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e78c6d37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LABUAN BAJO — President Ferdinand \"Bongbong\" M...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CorbettReport.com November 8, 2016   In Dougla...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>WASHINGTON - The United States scrambled F-16 ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Miami (CNN) There were few softballs Wednesday...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Trump and Sanders get all the attention for th...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1845</th>\n",
       "      <td>The owner of a convenience store in Malvar, Ba...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1846</th>\n",
       "      <td>While the full field of Republican presidentia...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1847</th>\n",
       "      <td>WASHINGTON - US President Joe Biden called on ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1848</th>\n",
       "      <td>Baltimore leaders say the first night of the c...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1849</th>\n",
       "      <td>How Hillary Clinton Locked Up The Democratic N...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1850 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   text  label\n",
       "0     LABUAN BAJO — President Ferdinand \"Bongbong\" M...      0\n",
       "1     CorbettReport.com November 8, 2016   In Dougla...      1\n",
       "2     WASHINGTON - The United States scrambled F-16 ...      0\n",
       "3     Miami (CNN) There were few softballs Wednesday...      0\n",
       "4     Trump and Sanders get all the attention for th...      0\n",
       "...                                                 ...    ...\n",
       "1845  The owner of a convenience store in Malvar, Ba...      0\n",
       "1846  While the full field of Republican presidentia...      0\n",
       "1847  WASHINGTON - US President Joe Biden called on ...      0\n",
       "1848  Baltimore leaders say the first night of the c...      0\n",
       "1849  How Hillary Clinton Locked Up The Democratic N...      0\n",
       "\n",
       "[1850 rows x 2 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_df = val_df.dropna (how = 'any').reset_index (drop = True)\n",
    "val_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "131fc3f5",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OTTAWA, Canada - Canada's government struck a ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Editor’s note: This press release is sponsored...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(CNN) Donald Trump and Bernie Sanders are conf...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The Alliance of Concerned Teachers (ACT) calle...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Manila's City Engineering Office has received ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4622</th>\n",
       "      <td>The House Committee on Human Rights has approv...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4623</th>\n",
       "      <td>The Metro Rail Transit Line 3 (MRT-3) has impl...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4624</th>\n",
       "      <td>October 28, 2016 at 9:00 PM   Why would Putin ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4625</th>\n",
       "      <td>PHNOM PENH — An eleven-year-old girl in Cambod...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4626</th>\n",
       "      <td>Justice Secretary Jesus Crispin Remulla on Thu...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4627 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   text  label\n",
       "0     OTTAWA, Canada - Canada's government struck a ...      0\n",
       "1     Editor’s note: This press release is sponsored...      0\n",
       "2     (CNN) Donald Trump and Bernie Sanders are conf...      0\n",
       "3     The Alliance of Concerned Teachers (ACT) calle...      0\n",
       "4     Manila's City Engineering Office has received ...      0\n",
       "...                                                 ...    ...\n",
       "4622  The House Committee on Human Rights has approv...      0\n",
       "4623  The Metro Rail Transit Line 3 (MRT-3) has impl...      0\n",
       "4624  October 28, 2016 at 9:00 PM   Why would Putin ...      1\n",
       "4625  PHNOM PENH — An eleven-year-old girl in Cambod...      0\n",
       "4626  Justice Secretary Jesus Crispin Remulla on Thu...      0\n",
       "\n",
       "[4627 rows x 2 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = test_df.dropna (how = 'any').reset_index (drop = True)\n",
    "test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6c4afb0",
   "metadata": {},
   "source": [
    "Next, using [`value_counts`](https://pandas.pydata.org/docs/reference/api/pandas.Series.value_counts.html), let's check if the dataset is balanced (i.e., has equal instances of positive and negative texts)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6f377c2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    13359\n",
       "1     3298\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df ['label'].value_counts ()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3625603f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1484\n",
       "1     366\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_df ['label'].value_counts ()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d8e8b465",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    3711\n",
       "1     916\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df ['label'].value_counts ()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a02148f",
   "metadata": {},
   "source": [
    "From this, we can see that the instances of news and fake news are not balanced in our training, validation, and testing data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd61a052",
   "metadata": {},
   "source": [
    "### Creation of Dataset\n",
    "Since we have already created three different sets, we can now transform our DataFrames into one single Dataset. To do this, we first have to transform each set into a single dataset before combining them into one dataset.\n",
    "\n",
    "First, we would be converting out train DataFrame into a dataset. In this, it can be seen that there are **16,657** rows in our train dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6c555f45",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text', 'label'],\n",
       "    num_rows: 16657\n",
       "})"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset = datasets.Dataset.from_pandas(train_df)\n",
    "train_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "898b7a41",
   "metadata": {},
   "source": [
    "This is followed by transforming the val DataFrame also. This would result in a dataset with **1,850** rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3092821e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text', 'label'],\n",
       "    num_rows: 1850\n",
       "})"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_dataset = datasets.Dataset.from_pandas(val_df)\n",
    "val_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9294b67d",
   "metadata": {},
   "source": [
    "Last is the test DataFrame, which would become a dataset with **4,627** rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1ac0956b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['text', 'label'],\n",
       "    num_rows: 4627\n",
       "})"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataset = datasets.Dataset.from_pandas(test_df)\n",
    "test_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "607e3fb1",
   "metadata": {},
   "source": [
    "As we now have a dataset form for all of our sets, we can now merge them together into one dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "673bbef5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 16657\n",
       "    })\n",
       "    val: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 1850\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 4627\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = datasets.DatasetDict({\n",
    "    \"train\" : train_dataset, \n",
    "    \"val\" : val_dataset, \n",
    "    \"test\" : test_dataset\n",
    "})\n",
    "\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03c95903",
   "metadata": {},
   "source": [
    "## Feature Engineering\n",
    "\n",
    "Because we are done preparing our data, we can now start with transforming it into a form that the machine learning algorithms can understand through feature engineering. For this notebook, we will be utilizing tokenization, specifically through the use of BERT and RoBERTa tokenizers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f80bccf6",
   "metadata": {},
   "source": [
    "### Defining of Functions and Values\n",
    "Before starting with the tokenizing itself, we will first have to define the needed functions and values. \n",
    "\n",
    "One of these values is the **MAX_LENGTH**, which determines the maximum length that will be allowed by the model. This means that it will be used by the tokenizer in two ways: (1) inputs that are longer than this length will be truncated to this value, and (2) inputs that are shorter than this length will be padded so that it will reach this length. For this notebook, **512** is set as the maximum length. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "2d9d4028",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LENGTH = 512"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66e4cd1",
   "metadata": {},
   "source": [
    "In addition, the preprocessing function for an instance is created. In this function, a text is tokenized by the tokenizer (i.e., padded and truncated to the maximum length) and its corresponding label is transformed into a tensor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "fca978aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_function(examples, tokenizer):\n",
    "    encoding = tokenizer(examples[\"text\"], add_special_tokens = True,\n",
    "                         padding = \"max_length\", truncation = True, max_length = MAX_LENGTH)\n",
    "    \n",
    "    encoding[\"labels\"] = torch.tensor(examples ['label'])\n",
    "    return encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "434646b7",
   "metadata": {},
   "source": [
    "Last, the function that would call the preprocessing function on the dataset is defined. In this function, the dataset is also set into a **torch** format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "82755943",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_encoded_dataset (tokenizer):\n",
    "    encoded_dataset = dataset.map(preprocess_function, \n",
    "                                  batched=True, \n",
    "                                  remove_columns=dataset['train'].column_names, \n",
    "                                  fn_kwargs = {\"tokenizer\": tokenizer})\n",
    "    \n",
    "    encoded_dataset.set_format(\"torch\")\n",
    "    \n",
    "    return encoded_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "900fffaf",
   "metadata": {},
   "source": [
    "### Tokenizing with BERT\n",
    "As our functions and values are ready, the tokenizer can be instantiated. Since we would be utilizing a BERT model, specifically the **bert-base-cased** model, we would be creating a tokenizer that can prepare the text data into the input accepted by the model. \n",
    "\n",
    "This can be done through the [`AutoTokenizer`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoTokenizer) class and the `from_pretrained` function, since the model and the tokenizer that we want to use has already been pretrained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "48eeead9",
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_tokenizer = AutoTokenizer.from_pretrained('bert-base-cased', use_fast = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd05ac1f",
   "metadata": {},
   "source": [
    "With this tokenizer, we will be encoding the dataset into the correct form that is needed by the BERT model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "0ad56be8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f43bf9351574467ab02e0b04f036258",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/17 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "698dfc5f08d54675a884869efb809f29",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8528fc6b3a9f4a4b94df30388342a377",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bert_encoded_dataset = create_encoded_dataset (bert_tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "787036ae",
   "metadata": {},
   "source": [
    "### Tokenizing with RoBERTa\n",
    "Next, as we also want to use a pretrained RoBERTa model (i.e., **roberta-base**), we also have to do the same steps.\n",
    "\n",
    "To start with, we need to create an instance of the specific RoBERTa model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "1c7e5f7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "roberta_tokenizer = AutoTokenizer.from_pretrained('roberta-base')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eccb13a8",
   "metadata": {},
   "source": [
    "Since we already have an instance of the tokenizer, we can now use this tokenizer and the pre-processing function we defined previously to transform the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ab5b42a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "885653a3a2bc46cf84adc3c1ec191114",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/17 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dba205543f2a469b86eeb05ed6696681",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92640c39df964350bb18badc3d07528e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "roberta_encoded_dataset = create_encoded_dataset (roberta_tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b779da9d",
   "metadata": {},
   "source": [
    "## Modeling and Evaluation\n",
    "\n",
    "As we have already created the features that we would be using for our models, we can now proceed with the modeling proper. For this project, we would be fine-tuning two pre-trained models: **BERT** and **RoBERTa**. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "756f4719",
   "metadata": {},
   "source": [
    "### Defining of Functions and Values\n",
    "\n",
    "Before we start with the training proper, we would need to define the functions that will be used for training and evaluating. \n",
    "\n",
    "First, we would be creating the function that would be used to compute the scores of the model. In this, we would be using four metrics to evaluate our models: (1) **F1 Macro Score**, (2) **Accuracy**, (3) **Precision**, and (4) **Recall**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c6a9b8d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(p: EvalPrediction):\n",
    "    logits, labels = p\n",
    "    predictions = np.argmax(logits, axis=-1)\n",
    "    \n",
    "    precision_metric = load_metric(\"precision\")\n",
    "    recall_metric = load_metric(\"recall\")\n",
    "    accuracy_metric = load_metric(\"accuracy\")\n",
    "    f1_metric = load_metric(\"f1\")\n",
    "    \n",
    "    f1_macro_score = f1_metric.compute(predictions=predictions, references=labels, average=\"macro\")\n",
    "    accuracy_score = accuracy_metric.compute(predictions=predictions, references=labels)\n",
    "    precision_score = precision_metric.compute(predictions=predictions, references=labels)\n",
    "    recall_score = recall_metric.compute(predictions=predictions, references=labels)\n",
    "    \n",
    "    results = {\n",
    "        'Accuracy' : accuracy_score ['accuracy'],\n",
    "        'F1 Macro Score' : f1_macro_score ['f1'], \n",
    "        'Precision' : precision_score[\"precision\"],\n",
    "        'Recall' : recall_score[\"recall\"]\n",
    "    }\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a535459",
   "metadata": {},
   "source": [
    "Second, we would be specifying the hyperparameter space that would determine the possible hyperparameter vaues to be tuned. In this, only three hyperparameters would be considered for tuning: (1) the **learning rate**, (2) the **train batch size**, and (3) the **number of training epochs**.\n",
    "\n",
    "Note that the combination of values would be randomized from the sets of values, and there would only be three combinations that would be used for the tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "50a5cdeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def optuna_hp_space(trial):\n",
    "    return {\n",
    "        \"learning_rate\": trial.suggest_categorical(\"learning_rate\", [0.1, 0.01, 0.001]),\n",
    "        \"per_device_train_batch_size\": trial.suggest_categorical(\"per_device_train_batch_size\", [8, 16]),\n",
    "        \"num_train_epochs\": trial.suggest_categorical(\"num_train_epochs\", [2, 3, 4])\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88bb8827",
   "metadata": {},
   "source": [
    "### BERT Model\n",
    "Now, we are ready to move on to training the BERT model. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4841056d",
   "metadata": {},
   "source": [
    "#### Model Training \n",
    "\n",
    "To start with, let us define the pre-trained model that we would be using. For the BERT, [**bert-base-cased**](https://huggingface.co/bert-base-cased)—a model that was pre-trained on a case-sensitive English corpus for masked language modeling (MLM)—would be utilized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "49f8a9fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_checkpoint = 'bert-base-cased'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e3ff527",
   "metadata": {},
   "source": [
    "Let us create an instance of a BERT model using this pretrained model. \n",
    "\n",
    "As we would be fine-tuning this model to classify text (i.e., if it is a fake news or not), an instance of [`AutoModelForSequenceClassification`](https://huggingface.co/transformers/v3.0.2/model_doc/auto.html#automodelforsequenceclassification) would be created specifically. It is also important to note that the input that it would accept is based on the **MAX_LENGTH** variable that we have previously declare, which has the value of **512**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7dcd65db",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "bert_model = AutoModelForSequenceClassification.from_pretrained(\n",
    "    model_checkpoint,\n",
    "    num_labels = 2, \n",
    "    max_length = MAX_LENGTH\n",
    ").to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb8dff9d",
   "metadata": {},
   "source": [
    "Next, we would be defining the [`TrainingArguments`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) that the [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) would be using. The parameters for the [`TrainingArguments`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) that are used for the training loop are as follows:\n",
    "* `output_dir` indicates that the model predictions and checkpoints will be saved in the **bert_trainer** folder\n",
    "* `save_steps` means that the checkpoint will be saved every **20,000** steps\n",
    "* `save_strategy` specifies that the saving of checkpoint will be based on the number of steps that the model has done \n",
    "* `fp16` stipulates that the **16-bit floating point precision** will be used (since its value is True) to save memory\n",
    "* `evaluation_strategy` designates that the **evaluation** should be done **every after epochs**\n",
    "* `resume_from_checkpoint` indicates that the training could be **restarted from a previous checkpoint**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "0b5940d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args = TrainingArguments(output_dir = \"bert_trainer\", \n",
    "                                  save_steps = 20000,\n",
    "                                  save_strategy = 'steps',\n",
    "                                  fp16 = True,\n",
    "                                  evaluation_strategy = \"epoch\", \n",
    "                                  resume_from_checkpoint = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0853dbab",
   "metadata": {},
   "source": [
    "As we have now declared the pre-trained model and the training arguments that we would be using, we can now instantiate a [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) that can do training and evaluation using the following parameters:\n",
    "* `model` is the BERT model that we would be using for sequence classification\n",
    "* `args` holds the training arguments that we have previously defined\n",
    "* `train_dataset` is the tokenized dataset that we would be using for training\n",
    "* `eval_dataset` is the tokenized dataset that we would be using for evaluating (i.e., the val set)\n",
    "* `tokenizer` is the tokenizer that we used to prepare our data for the BERT model\n",
    "* `compute_metrics` is the function that the evaluation loop would use to score the model\n",
    "* `callbacks` holds the **ProgressBar**, which would allow us to see the progress of our model in training and evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "d6ba97d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using cuda_amp half precision backend\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(\n",
    "    model = bert_model,\n",
    "    args = training_args,\n",
    "    train_dataset = bert_encoded_dataset ['train'],\n",
    "    eval_dataset = bert_encoded_dataset ['val'],\n",
    "    tokenizer = bert_tokenizer,\n",
    "    compute_metrics = compute_metrics,\n",
    "    callbacks = [TrainerCallback()]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828c02cb",
   "metadata": {},
   "source": [
    "Using the instance of [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) that we have created, we can now fine-tune the pre-trained BERT model through the use of the [`train`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.train) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "43cb5bc6",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\admin\\anaconda3\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 16657\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 8\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 8\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 6249\n",
      "  Number of trainable parameters = 108311810\n",
      "Automatic Weights & Biases logging enabled, to disable set os.environ[\"WANDB_DISABLED\"] = \"true\"\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mfrancheska_vicente\u001b[0m (\u001b[33mtonely\u001b[0m). Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.15.5 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.10"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>C:\\Users\\admin\\Desktop\\Uni\\DATA102\\github-repositories\\data102-fake-news\\wandb\\run-20230722_203924-b2mojuyl</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/tonely/huggingface/runs/b2mojuyl' target=\"_blank\">dazzling-oath-125</a></strong> to <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">https://wandb.ai/tonely/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/tonely/huggingface/runs/b2mojuyl' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/b2mojuyl</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='6249' max='6249' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [6249/6249 32:25, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 macro score</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.137200</td>\n",
       "      <td>0.093312</td>\n",
       "      <td>0.976216</td>\n",
       "      <td>0.962762</td>\n",
       "      <td>0.932796</td>\n",
       "      <td>0.948087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.076900</td>\n",
       "      <td>0.150093</td>\n",
       "      <td>0.970811</td>\n",
       "      <td>0.954847</td>\n",
       "      <td>0.906250</td>\n",
       "      <td>0.950820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.032500</td>\n",
       "      <td>0.108422</td>\n",
       "      <td>0.982162</td>\n",
       "      <td>0.971871</td>\n",
       "      <td>0.956164</td>\n",
       "      <td>0.953552</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Local\\Temp\\ipykernel_15240\\730230934.py:5: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate\n",
      "  precision_metric = load_metric(\"precision\")\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=6249, training_loss=0.096801379486587, metrics={'train_runtime': 1962.4095, 'train_samples_per_second': 25.464, 'train_steps_per_second': 3.184, 'total_flos': 1.314792254739456e+16, 'train_loss': 0.096801379486587, 'epoch': 3.0})"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee220550",
   "metadata": {},
   "source": [
    "From the result above, we can see that the model received the highest evaluation score on the validation set on the third epoch. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72e5de1c",
   "metadata": {},
   "source": [
    "#### Saving BERT base model\n",
    "To use this model outside the notebook, we would be saving the model. First, let us define the folder where we would be saving the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "c84dd6f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "path_for_models ='./saved_models/BERTv1'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71290ce5",
   "metadata": {},
   "source": [
    "Now, let us save the [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) (i.e., with the weights, the configurations, and the model) and the [`BertTokenizer`](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer) in the specified folder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "331f536c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./saved_models/BERTv1\n",
      "Configuration saved in ./saved_models/BERTv1\\config.json\n",
      "Model weights saved in ./saved_models/BERTv1\\pytorch_model.bin\n",
      "tokenizer config file saved in ./saved_models/BERTv1\\tokenizer_config.json\n",
      "Special tokens file saved in ./saved_models/BERTv1\\special_tokens_map.json\n",
      "tokenizer config file saved in ./saved_models/BERTv1\\tokenizer_config.json\n",
      "Special tokens file saved in ./saved_models/BERTv1\\special_tokens_map.json\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('./saved_models/BERTv1\\\\tokenizer_config.json',\n",
       " './saved_models/BERTv1\\\\special_tokens_map.json',\n",
       " './saved_models/BERTv1\\\\vocab.txt',\n",
       " './saved_models/BERTv1\\\\added_tokens.json')"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.save_model(path_for_models)\n",
    "bert_tokenizer.save_pretrained(path_for_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93672b7d",
   "metadata": {},
   "source": [
    "Using the [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) we have trained, we can now [`evaluate`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.evaluate) the model using the test set to determine its test score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "63aa0e2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 4627\n",
      "  Batch size = 8\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.07904903590679169,\n",
       " 'eval_Accuracy': 0.9846552842014265,\n",
       " 'eval_F1 Macro Score': 0.9756279882453602,\n",
       " 'eval_Precision': 0.9720670391061452,\n",
       " 'eval_Recall': 0.9497816593886463,\n",
       " 'eval_runtime': 56.1269,\n",
       " 'eval_samples_per_second': 82.438,\n",
       " 'eval_steps_per_second': 10.316,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate(eval_dataset=bert_encoded_dataset['test'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86b6ef9e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "98aeef73",
   "metadata": {},
   "source": [
    "From the result above, it can be seen that the model was able to be correctly trained. It achieved the following scores: 98.46% for Accuracy, 97.56% F1 Macro Score, 97.20% for Precision, and 94.98% for Recall."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94eb64f4",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "Now, let us try to tune the hyperparameters (i.e., the learning rate, the number of training epochs and the training batch size) of the model, which means that we would try to find the value that would give us the highest score. In this, we would be trying three combinations of these hyperparameters, and we would compare the scores received by the three combinations to the score of the base model. \n",
    "\n",
    "To do this, we will first create a function that would return a base model of a BERT [`AutoModelForSequenceClassification`](https://huggingface.co/transformers/v3.0.2/model_doc/auto.html#automodelforsequenceclassification) for initializaiton. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "cbb450f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_init():\n",
    "    return AutoModelForSequenceClassification.from_pretrained(model_checkpoint,\n",
    "                                                              num_labels = 2, \n",
    "                                                              max_length = MAX_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28b0ef38",
   "metadata": {},
   "source": [
    "Like in training the base model, we would be creating the [`TrainingArguments`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) that we would be using for training. We would be using the same parameters for the [`TrainingArguments`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) as before, except for the **fp16**. \n",
    "\n",
    "In the tuning, **bf16** (bfloat16) will be used. This was done because using **fp16** resulted in 0.0 scores due to the loss of floating points in fp16."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "7530d21c",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args_tuning = TrainingArguments(output_dir = \"bert_trainer\", \n",
    "                                         save_steps = 20000, \n",
    "                                         bf16 = True,\n",
    "                                         save_strategy = 'steps',\n",
    "                                         evaluation_strategy = \"epoch\", \n",
    "                                         resume_from_checkpoint = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21250177",
   "metadata": {},
   "source": [
    "Next, we can create an instance of [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) class. Since we would be using the [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) for tuning, we passed an **initialization of the model** instead of a model. This initial model is used as the base (i.e., the model is reinitialized every run of new hyperparameter values). This means that all of the models use the values of the base model and only the values of the hyperparameter passed are changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "b3e177b2",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--bert-base-cased\\snapshots\\5532cc56f74641d4bb33641f5c76a55d11f846e0\\config.json\n",
      "Model config BertConfig {\n",
      "  \"_name_or_path\": \"bert-base-cased\",\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 28996\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--bert-base-cased\\snapshots\\5532cc56f74641d4bb33641f5c76a55d11f846e0\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Using cuda_amp half precision backend\n"
     ]
    }
   ],
   "source": [
    "trainer_tuning = Trainer(\n",
    "    model_init = model_init,\n",
    "    args = training_args_tuning,\n",
    "    train_dataset = bert_encoded_dataset ['train'],\n",
    "    eval_dataset = bert_encoded_dataset ['val'],\n",
    "    tokenizer = bert_tokenizer,\n",
    "    compute_metrics = compute_metrics,\n",
    "    callbacks = [TrainerCallback()]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46eafa14",
   "metadata": {},
   "source": [
    "Using the [`hyperparameter_search`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.hyperparameter_search) function, we can now start finding the best values of the hyperparameters to use. Note that this function will return the information about the best run (i.e., the model that received the best score)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "3358c16f",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2023-07-22 21:22:36,884]\u001b[0m A new study created in memory with name: no-name-1603b342-70a4-4d1b-8959-69a98b43ca31\u001b[0m\n",
      "Trial: {'learning_rate': 0.01, 'per_device_train_batch_size': 16, 'num_train_epochs': 3}\n",
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--bert-base-cased\\snapshots\\5532cc56f74641d4bb33641f5c76a55d11f846e0\\config.json\n",
      "Model config BertConfig {\n",
      "  \"_name_or_path\": \"bert-base-cased\",\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 28996\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--bert-base-cased\\snapshots\\5532cc56f74641d4bb33641f5c76a55d11f846e0\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "C:\\Users\\admin\\anaconda3\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 16657\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 16\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 16\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 3126\n",
      "  Number of trainable parameters = 108311810\n",
      "Automatic Weights & Biases logging enabled, to disable set os.environ[\"WANDB_DISABLED\"] = \"true\"\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mfrancheska_vicente\u001b[0m (\u001b[33mtonely\u001b[0m). Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.15.5 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.10"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>C:\\Users\\admin\\Desktop\\Uni\\DATA102\\github-repositories\\data102-fake-news\\wandb\\run-20230722_212240-imjbi71r</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/tonely/huggingface/runs/imjbi71r' target=\"_blank\">hardy-darkness-127</a></strong> to <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">https://wandb.ai/tonely/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/tonely/huggingface/runs/imjbi71r' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/imjbi71r</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='3126' max='3126' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [3126/3126 28:26, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 macro score</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.804900</td>\n",
       "      <td>0.758189</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.697500</td>\n",
       "      <td>0.525683</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.558700</td>\n",
       "      <td>0.497785</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Local\\Temp\\ipykernel_19380\\730230934.py:5: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate\n",
      "  precision_metric = load_metric(\"precision\")\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "\u001b[32m[I 2023-07-22 21:51:21,700]\u001b[0m Trial 0 finished with value: 1.2472731399666013 and parameters: {'learning_rate': 0.01, 'per_device_train_batch_size': 16, 'num_train_epochs': 3}. Best is trial 0 with value: 1.2472731399666013.\u001b[0m\n",
      "Trial: {'learning_rate': 0.001, 'per_device_train_batch_size': 16, 'num_train_epochs': 3}\n",
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--bert-base-cased\\snapshots\\5532cc56f74641d4bb33641f5c76a55d11f846e0\\config.json\n",
      "Model config BertConfig {\n",
      "  \"_name_or_path\": \"bert-base-cased\",\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 28996\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--bert-base-cased\\snapshots\\5532cc56f74641d4bb33641f5c76a55d11f846e0\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "C:\\Users\\admin\\anaconda3\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 16657\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 16\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 16\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 3126\n",
      "  Number of trainable parameters = 108311810\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6fb0ae39a8c4a7c83ca43d514989e27",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.001 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/Accuracy</td><td>▁▁▁</td></tr><tr><td>eval/F1 Macro Score</td><td>▁▁▁</td></tr><tr><td>eval/Precision</td><td>▁▁▁</td></tr><tr><td>eval/Recall</td><td>▁▁▁</td></tr><tr><td>eval/loss</td><td>█▂▁</td></tr><tr><td>eval/runtime</td><td>█▁▃</td></tr><tr><td>eval/samples_per_second</td><td>▁█▆</td></tr><tr><td>eval/steps_per_second</td><td>▁█▆</td></tr><tr><td>train/epoch</td><td>▁▂▂▄▅▅▆███</td></tr><tr><td>train/global_step</td><td>▁▂▂▄▅▅▆███</td></tr><tr><td>train/learning_rate</td><td>█▇▅▄▂▁</td></tr><tr><td>train/loss</td><td>█▅▆▃▂▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/Accuracy</td><td>0.80216</td></tr><tr><td>eval/F1 Macro Score</td><td>0.44511</td></tr><tr><td>eval/Precision</td><td>0.0</td></tr><tr><td>eval/Recall</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.49779</td></tr><tr><td>eval/runtime</td><td>25.7737</td></tr><tr><td>eval/samples_per_second</td><td>71.779</td></tr><tr><td>eval/steps_per_second</td><td>9.001</td></tr><tr><td>train/epoch</td><td>3.0</td></tr><tr><td>train/global_step</td><td>3126</td></tr><tr><td>train/learning_rate</td><td>0.0004</td></tr><tr><td>train/loss</td><td>0.5587</td></tr><tr><td>train/total_flos</td><td>1.314792254739456e+16</td></tr><tr><td>train/train_loss</td><td>0.74371</td></tr><tr><td>train/train_runtime</td><td>1722.9108</td></tr><tr><td>train/train_samples_per_second</td><td>29.004</td></tr><tr><td>train/train_steps_per_second</td><td>1.814</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">hardy-darkness-127</strong> at: <a href='https://wandb.ai/tonely/huggingface/runs/imjbi71r' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/imjbi71r</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230722_212240-imjbi71r\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatic Weights & Biases logging enabled, to disable set os.environ[\"WANDB_DISABLED\"] = \"true\"\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "690ca37aa81e40f9911404c0184a6e9b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='Waiting for wandb.init()...\\r'), FloatProgress(value=0.016933333333266395, max=1.0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.15.5 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.10"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>C:\\Users\\admin\\Desktop\\Uni\\DATA102\\github-repositories\\data102-fake-news\\wandb\\run-20230722_215128-g2czdg0d</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/tonely/huggingface/runs/g2czdg0d' target=\"_blank\">kind-wave-128</a></strong> to <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">https://wandb.ai/tonely/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/tonely/huggingface/runs/g2czdg0d' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/g2czdg0d</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='3126' max='3126' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [3126/3126 28:05, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 macro score</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.534200</td>\n",
       "      <td>0.504901</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.518700</td>\n",
       "      <td>0.501915</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.505900</td>\n",
       "      <td>0.497616</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "\u001b[32m[I 2023-07-22 22:19:48,145]\u001b[0m Trial 1 finished with value: 1.2472731399666013 and parameters: {'learning_rate': 0.001, 'per_device_train_batch_size': 16, 'num_train_epochs': 3}. Best is trial 0 with value: 1.2472731399666013.\u001b[0m\n",
      "Trial: {'learning_rate': 0.001, 'per_device_train_batch_size': 8, 'num_train_epochs': 3}\n",
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--bert-base-cased\\snapshots\\5532cc56f74641d4bb33641f5c76a55d11f846e0\\config.json\n",
      "Model config BertConfig {\n",
      "  \"_name_or_path\": \"bert-base-cased\",\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 28996\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--bert-base-cased\\snapshots\\5532cc56f74641d4bb33641f5c76a55d11f846e0\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "C:\\Users\\admin\\anaconda3\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 16657\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 8\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 8\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 6249\n",
      "  Number of trainable parameters = 108311810\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/Accuracy</td><td>▁▁▁</td></tr><tr><td>eval/F1 Macro Score</td><td>▁▁▁</td></tr><tr><td>eval/Precision</td><td>▁▁▁</td></tr><tr><td>eval/Recall</td><td>▁▁▁</td></tr><tr><td>eval/loss</td><td>█▅▁</td></tr><tr><td>eval/runtime</td><td>▁▇█</td></tr><tr><td>eval/samples_per_second</td><td>█▂▁</td></tr><tr><td>eval/steps_per_second</td><td>█▂▁</td></tr><tr><td>train/epoch</td><td>▁▂▂▄▅▅▆███</td></tr><tr><td>train/global_step</td><td>▁▂▂▄▅▅▆███</td></tr><tr><td>train/learning_rate</td><td>█▇▅▄▂▁</td></tr><tr><td>train/loss</td><td>█▇▄▄▂▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/Accuracy</td><td>0.80216</td></tr><tr><td>eval/F1 Macro Score</td><td>0.44511</td></tr><tr><td>eval/Precision</td><td>0.0</td></tr><tr><td>eval/Recall</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.49762</td></tr><tr><td>eval/runtime</td><td>25.8073</td></tr><tr><td>eval/samples_per_second</td><td>71.685</td></tr><tr><td>eval/steps_per_second</td><td>8.99</td></tr><tr><td>train/epoch</td><td>3.0</td></tr><tr><td>train/global_step</td><td>3126</td></tr><tr><td>train/learning_rate</td><td>4e-05</td></tr><tr><td>train/loss</td><td>0.5059</td></tr><tr><td>train/total_flos</td><td>1.314792254739456e+16</td></tr><tr><td>train/train_loss</td><td>0.52188</td></tr><tr><td>train/train_runtime</td><td>1704.5817</td></tr><tr><td>train/train_samples_per_second</td><td>29.316</td></tr><tr><td>train/train_steps_per_second</td><td>1.834</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">kind-wave-128</strong> at: <a href='https://wandb.ai/tonely/huggingface/runs/g2czdg0d' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/g2czdg0d</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230722_215128-g2czdg0d\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatic Weights & Biases logging enabled, to disable set os.environ[\"WANDB_DISABLED\"] = \"true\"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.15.5 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.10"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>C:\\Users\\admin\\Desktop\\Uni\\DATA102\\github-repositories\\data102-fake-news\\wandb\\run-20230722_221955-5e0vzg4n</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/tonely/huggingface/runs/5e0vzg4n' target=\"_blank\">super-darkness-129</a></strong> to <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">https://wandb.ai/tonely/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/tonely/huggingface/runs/5e0vzg4n' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/5e0vzg4n</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='6249' max='6249' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [6249/6249 32:20, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 macro score</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.552400</td>\n",
       "      <td>0.505642</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.524100</td>\n",
       "      <td>0.498315</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.501400</td>\n",
       "      <td>0.498992</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "\u001b[32m[I 2023-07-22 22:52:30,000]\u001b[0m Trial 2 finished with value: 1.2472731399666013 and parameters: {'learning_rate': 0.001, 'per_device_train_batch_size': 8, 'num_train_epochs': 3}. Best is trial 0 with value: 1.2472731399666013.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "best_trial = trainer_tuning.hyperparameter_search(\n",
    "    direction = \"maximize\",\n",
    "    backend = \"optuna\",\n",
    "    hp_space = optuna_hp_space,\n",
    "    n_trials = 3\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "fbd70eaf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BestRun(run_id='0', objective=1.2472731399666013, hyperparameters={'learning_rate': 0.01, 'per_device_train_batch_size': 16, 'num_train_epochs': 3})"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_trial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "319a2c94",
   "metadata": {},
   "source": [
    "In this, it can be seen that there were three BERT models that were created in tuning, with the following hyperparameters:\n",
    "* **Learning Rate** = 0.001, **Train Batch Size** = 8, **Number of Train Epochs** = 3\n",
    "* **Learning Rate** = 0.001, **Train Batch Size** = 16, **Number of Train Epochs** = 3\n",
    "* **Learning Rate** = 0.01, **Train Batch Size** = 16, **Number of Train Epochs** = 3\n",
    "\n",
    "These values were randomly generated based on the hyperparameter space that we have declared."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bda5e1e3",
   "metadata": {},
   "source": [
    "##### Saving BERT tuned model\n",
    "\n",
    "Like in the base model, we will also save the files of the best trial of the tuned model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "c7ab72ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./saved_models/BERTv1_tuned\n",
      "Configuration saved in ./saved_models/BERTv1_tuned\\config.json\n",
      "Model weights saved in ./saved_models/BERTv1_tuned\\pytorch_model.bin\n",
      "tokenizer config file saved in ./saved_models/BERTv1_tuned\\tokenizer_config.json\n",
      "Special tokens file saved in ./saved_models/BERTv1_tuned\\special_tokens_map.json\n"
     ]
    }
   ],
   "source": [
    "path_for_models ='./saved_models/BERTv1_tuned'\n",
    "trainer_tuning.save_model(path_for_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afc78564",
   "metadata": {},
   "source": [
    "#### Evaluation\n",
    "\n",
    "To test how the best trial of the BERT tuning fared in the test dataset, we will be using the [`evaluate`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.evaluate) function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "1379f892",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 4627\n",
      "  Batch size = 8\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='579' max='579' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [579/579 00:49]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.49919524788856506,\n",
       " 'eval_Accuracy': 0.802031553922628,\n",
       " 'eval_F1 Macro Score': 0.4450707603741904,\n",
       " 'eval_Precision': 0.0,\n",
       " 'eval_Recall': 0.0,\n",
       " 'eval_runtime': 53.8581,\n",
       " 'eval_samples_per_second': 85.911,\n",
       " 'eval_steps_per_second': 10.75,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer_tuning.evaluate(eval_dataset=bert_encoded_dataset['test'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "328e73cc",
   "metadata": {},
   "source": [
    "In this result, it can be seen that the BERT model (with the learning rate of 0.001 and a train batch size of 16) was only accurate on 80.20% of the test samples. However, it is important to remember that the samples are heavily imbalanced, which is why, the model could receive this accuracy even if they just label everything as 0 (i.e., not a fake news)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77630873",
   "metadata": {},
   "source": [
    "Comparing the scores of these two models from tuning to the base model in the validation, the scores received by the base model was still better. especially the Precision and Recall. Thus, for the BERT model, we will consider the base model as our best model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fe26554",
   "metadata": {},
   "source": [
    "### RoBERTa Model\n",
    "Now, we can move on to training the RoBERTa model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f5a119a",
   "metadata": {},
   "source": [
    "#### Model Training \n",
    "Like in the BERT model, we would need to define the pre-trained model that we would be fine-tuning. For this, we would be using [**roberta-base**](https://huggingface.co/roberta-base). This model, which is case-sensitive, was also pre-trained for the purpose of masked language modeling (MLM) on an English corpus, however, it uses the RoBERTa architecture, instead of the BERT architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "55dfa1cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_checkpoint_roberta = 'roberta-base'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92839b72",
   "metadata": {},
   "source": [
    "Using this pre-trained model, we can instantiate a [`AutoModelForSequenceClassification`](https://huggingface.co/transformers/v3.0.2/model_doc/auto.html#automodelforsequenceclassification) object, which will create a RoBERTa model. In addition, we would also be defining the **MAX_LENGTH** of the model to be the same as the previously defined **MAX_LENGTH** (i.e., 512)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "cd78e4c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\config.json\n",
      "Model config RobertaConfig {\n",
      "  \"_name_or_path\": \"roberta-base\",\n",
      "  \"architectures\": [\n",
      "    \"RobertaForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"bos_token_id\": 0,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-05,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 514,\n",
      "  \"model_type\": \"roberta\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 1,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 1,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 50265\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at roberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.layer_norm.weight', 'roberta.pooler.dense.bias', 'roberta.pooler.dense.weight', 'lm_head.dense.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.layer_norm.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "roberta_model = AutoModelForSequenceClassification.from_pretrained(\n",
    "    model_checkpoint_roberta,\n",
    "    num_labels = 2, \n",
    "    max_length = MAX_LENGTH\n",
    ").to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54634df6",
   "metadata": {},
   "source": [
    "We would also need to create an instance of [`TrainingArguments`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments). This would have the same values as the previous [`TrainingArguments`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) of the BERT model, except for the `output_dir`, as we wnat to save the checkpoints in another folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "2d7d267e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PyTorch: setting up devices\n",
      "The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).\n"
     ]
    }
   ],
   "source": [
    "training_args = TrainingArguments(output_dir = \"roberta_trainer\", \n",
    "                                  save_steps = 20000,\n",
    "                                  save_strategy = 'steps',\n",
    "                                  fp16 = True,\n",
    "                                  evaluation_strategy = \"epoch\", \n",
    "                                  resume_from_checkpoint = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "819754d8",
   "metadata": {},
   "source": [
    "Using this RoBERTa model and the previously created [`TrainingArguments`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) object, we can now create a  [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer). Its parameters are also the same with the [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) for BERT, but the `model`, `train_dataset`, and `eval_dataset` are changed to the RoBERTa counterparts.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "71ce2c5c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using cuda_amp half precision backend\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(\n",
    "    model = roberta_model,\n",
    "    args = training_args,\n",
    "    train_dataset = roberta_encoded_dataset ['train'],\n",
    "    eval_dataset = roberta_encoded_dataset ['val'],\n",
    "    compute_metrics = compute_metrics,\n",
    "    callbacks = [TrainerCallback()]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a261cf79",
   "metadata": {},
   "source": [
    "Now, we can train the RoBERTa model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "0364213f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\admin\\anaconda3\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 16657\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 8\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 8\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 6249\n",
      "  Number of trainable parameters = 124647170\n",
      "Automatic Weights & Biases logging enabled, to disable set os.environ[\"WANDB_DISABLED\"] = \"true\"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='6249' max='6249' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [6249/6249 32:45, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 macro score</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.274600</td>\n",
       "      <td>0.303108</td>\n",
       "      <td>0.918919</td>\n",
       "      <td>0.860363</td>\n",
       "      <td>0.877622</td>\n",
       "      <td>0.685792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.243800</td>\n",
       "      <td>0.171242</td>\n",
       "      <td>0.938378</td>\n",
       "      <td>0.906341</td>\n",
       "      <td>0.813433</td>\n",
       "      <td>0.893443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.090200</td>\n",
       "      <td>0.059516</td>\n",
       "      <td>0.983784</td>\n",
       "      <td>0.974348</td>\n",
       "      <td>0.964088</td>\n",
       "      <td>0.953552</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=6249, training_loss=0.20192054520227143, metrics={'train_runtime': 1965.7021, 'train_samples_per_second': 25.421, 'train_steps_per_second': 3.179, 'total_flos': 1.314792254739456e+16, 'train_loss': 0.20192054520227143, 'epoch': 3.0})"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b46f322b",
   "metadata": {},
   "source": [
    "From this, it can be seen that, in the third epoch, the RoBERTa base model was able to achieve an **Accuracy** of **98.38%**, a **F1 Macro Score** of **97.43%**, a **Precision** of **96.41%**, and a **Recall** of **95.36%**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82f3224c",
   "metadata": {},
   "source": [
    "#### Saving RoBERTa base model\n",
    "Since we are done training the model, we would be saving the RoBERTa model, and its configuration and tokenizer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "8ea34aa3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./saved_models/RoBERTav1\n",
      "Configuration saved in ./saved_models/RoBERTav1\\config.json\n",
      "Model weights saved in ./saved_models/RoBERTav1\\pytorch_model.bin\n",
      "tokenizer config file saved in ./saved_models/RoBERTav1\\tokenizer_config.json\n",
      "Special tokens file saved in ./saved_models/RoBERTav1\\special_tokens_map.json\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('./saved_models/RoBERTav1\\\\tokenizer_config.json',\n",
       " './saved_models/RoBERTav1\\\\special_tokens_map.json',\n",
       " './saved_models/RoBERTav1\\\\vocab.json',\n",
       " './saved_models/RoBERTav1\\\\merges.txt',\n",
       " './saved_models/RoBERTav1\\\\added_tokens.json',\n",
       " './saved_models/RoBERTav1\\\\tokenizer.json')"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_for_models ='./saved_models/RoBERTav1'\n",
    "trainer.save_model(path_for_models)\n",
    "roberta_tokenizer.save_pretrained(path_for_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd0b2394",
   "metadata": {},
   "source": [
    "We can now [`evaluate`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.evaluate) this RoBERTa model on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "a3c5b0f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 4627\n",
      "  Batch size = 8\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='579' max='579' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [579/579 00:49]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.08169636875391006,\n",
       " 'eval_Accuracy': 0.9807650745623514,\n",
       " 'eval_F1 Macro Score': 0.9690856993273599,\n",
       " 'eval_Precision': 0.9769319492502884,\n",
       " 'eval_Recall': 0.9246724890829694,\n",
       " 'eval_runtime': 53.9361,\n",
       " 'eval_samples_per_second': 85.787,\n",
       " 'eval_steps_per_second': 10.735,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate(eval_dataset=roberta_encoded_dataset['test'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca56b2e1",
   "metadata": {},
   "source": [
    "Comparing the scores received by the RoBERTa base model and the best BERT model, it is apparent that the **BERT model received higher scores in every metric except for Precision**. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e2d71b6",
   "metadata": {},
   "source": [
    "#### Hyperparameter Tuning\n",
    "To further see if we can improve the current RoBERTa model, we can tune the model's hyperparameters. \n",
    "\n",
    "Like in the BERT model, we would first need to create a function that would return the initial state of the model that would be tuned. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "5adedfa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_init_roberta ():\n",
    "    return AutoModelForSequenceClassification.from_pretrained(model_checkpoint_roberta,\n",
    "                                                              num_labels = 2, \n",
    "                                                              max_length = MAX_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c8fe75f",
   "metadata": {},
   "source": [
    "Next, we would have to create the [`TrainingArguments`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments) that we would be using for the training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "45512936",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args_tuning = TrainingArguments(output_dir = \"roberta_trainer\", \n",
    "                                         save_steps = 20000, \n",
    "                                         bf16 = True,\n",
    "                                         save_strategy = 'steps',\n",
    "                                         evaluation_strategy = \"epoch\", \n",
    "                                         resume_from_checkpoint = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42f64286",
   "metadata": {},
   "source": [
    "With this, we can now proceed with creating an instance of the [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "cb52b1ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\config.json\n",
      "Model config RobertaConfig {\n",
      "  \"_name_or_path\": \"roberta-base\",\n",
      "  \"architectures\": [\n",
      "    \"RobertaForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"bos_token_id\": 0,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-05,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 514,\n",
      "  \"model_type\": \"roberta\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 1,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 1,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 50265\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at roberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight', 'roberta.pooler.dense.weight']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.out_proj.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Using cuda_amp half precision backend\n"
     ]
    }
   ],
   "source": [
    "trainer_tuning = Trainer(\n",
    "    model_init = model_init_roberta,\n",
    "    args = training_args_tuning,\n",
    "    train_dataset = roberta_encoded_dataset ['train'],\n",
    "    eval_dataset = roberta_encoded_dataset ['val'],\n",
    "    compute_metrics = compute_metrics,\n",
    "    callbacks = [TrainerCallback()]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48f4b2b7",
   "metadata": {},
   "source": [
    "We can now proceed with utilizing the [`hyperparameter_search`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.hyperparameter_search) function to: (1) randomize values for the three hyperparameters that we want to tune based on the search space, (2) train three models using the values, and (3) pick the best model from the three trained models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "62f231c0",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2023-07-22 23:33:08,759]\u001b[0m A new study created in memory with name: no-name-af6798e5-2b0b-4bb6-8aca-b4461caa04c9\u001b[0m\n",
      "Trial: {'learning_rate': 0.1, 'per_device_train_batch_size': 8, 'num_train_epochs': 2}\n",
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\config.json\n",
      "Model config RobertaConfig {\n",
      "  \"_name_or_path\": \"roberta-base\",\n",
      "  \"architectures\": [\n",
      "    \"RobertaForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"bos_token_id\": 0,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-05,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 514,\n",
      "  \"model_type\": \"roberta\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 1,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 1,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 50265\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at roberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight', 'roberta.pooler.dense.weight']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.out_proj.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "C:\\Users\\admin\\anaconda3\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 16657\n",
      "  Num Epochs = 2\n",
      "  Instantaneous batch size per device = 8\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 8\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 4166\n",
      "  Number of trainable parameters = 124647170\n",
      "Automatic Weights & Biases logging enabled, to disable set os.environ[\"WANDB_DISABLED\"] = \"true\"\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mfrancheska_vicente\u001b[0m (\u001b[33mtonely\u001b[0m). Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.15.5 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.10"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>C:\\Users\\admin\\Desktop\\Uni\\DATA102\\github-repositories\\data102-fake-news\\wandb\\run-20230722_233312-j8tpb6ts</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/tonely/huggingface/runs/j8tpb6ts' target=\"_blank\">glorious-violet-131</a></strong> to <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">https://wandb.ai/tonely/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/tonely/huggingface/runs/j8tpb6ts' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/j8tpb6ts</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='4166' max='4166' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [4166/4166 21:46, Epoch 2/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 macro score</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>6.516400</td>\n",
       "      <td>0.500437</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.140700</td>\n",
       "      <td>0.817817</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Local\\Temp\\ipykernel_20084\\730230934.py:5: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate\n",
      "  precision_metric = load_metric(\"precision\")\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "\u001b[32m[I 2023-07-22 23:55:13,114]\u001b[0m Trial 0 finished with value: 1.2472731399666013 and parameters: {'learning_rate': 0.1, 'per_device_train_batch_size': 8, 'num_train_epochs': 2}. Best is trial 0 with value: 1.2472731399666013.\u001b[0m\n",
      "Trial: {'learning_rate': 0.1, 'per_device_train_batch_size': 16, 'num_train_epochs': 3}\n",
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\config.json\n",
      "Model config RobertaConfig {\n",
      "  \"_name_or_path\": \"roberta-base\",\n",
      "  \"architectures\": [\n",
      "    \"RobertaForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"bos_token_id\": 0,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-05,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 514,\n",
      "  \"model_type\": \"roberta\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 1,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 1,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 50265\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at roberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight', 'roberta.pooler.dense.weight']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.out_proj.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "C:\\Users\\admin\\anaconda3\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 16657\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 16\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 16\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 3126\n",
      "  Number of trainable parameters = 124647170\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69ff62c1544647d6967f54942ec9c6d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.001 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/Accuracy</td><td>▁▁</td></tr><tr><td>eval/F1 Macro Score</td><td>▁▁</td></tr><tr><td>eval/Precision</td><td>▁▁</td></tr><tr><td>eval/Recall</td><td>▁▁</td></tr><tr><td>eval/loss</td><td>▁█</td></tr><tr><td>eval/runtime</td><td>▁█</td></tr><tr><td>eval/samples_per_second</td><td>█▁</td></tr><tr><td>eval/steps_per_second</td><td>█▁</td></tr><tr><td>train/epoch</td><td>▁▂▃▄▄▅▆▇███</td></tr><tr><td>train/global_step</td><td>▁▂▃▄▄▅▆▇███</td></tr><tr><td>train/learning_rate</td><td>█▇▆▅▄▃▂▁</td></tr><tr><td>train/loss</td><td>▇█▆▆▅▄▂▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/Accuracy</td><td>0.80216</td></tr><tr><td>eval/F1 Macro Score</td><td>0.44511</td></tr><tr><td>eval/Precision</td><td>0.0</td></tr><tr><td>eval/Recall</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.81782</td></tr><tr><td>eval/runtime</td><td>25.6617</td></tr><tr><td>eval/samples_per_second</td><td>72.092</td></tr><tr><td>eval/steps_per_second</td><td>9.041</td></tr><tr><td>train/epoch</td><td>2.0</td></tr><tr><td>train/global_step</td><td>4166</td></tr><tr><td>train/learning_rate</td><td>0.00398</td></tr><tr><td>train/loss</td><td>2.1407</td></tr><tr><td>train/total_flos</td><td>8765281698263040.0</td></tr><tr><td>train/train_loss</td><td>5.53209</td></tr><tr><td>train/train_runtime</td><td>1322.4901</td></tr><tr><td>train/train_samples_per_second</td><td>25.19</td></tr><tr><td>train/train_steps_per_second</td><td>3.15</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">glorious-violet-131</strong> at: <a href='https://wandb.ai/tonely/huggingface/runs/j8tpb6ts' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/j8tpb6ts</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230722_233312-j8tpb6ts\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatic Weights & Biases logging enabled, to disable set os.environ[\"WANDB_DISABLED\"] = \"true\"\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "117ca97457734ebead4ecc0439c6748e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='Waiting for wandb.init()...\\r'), FloatProgress(value=0.016666666666666666, max=1.0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.15.5 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.10"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>C:\\Users\\admin\\Desktop\\Uni\\DATA102\\github-repositories\\data102-fake-news\\wandb\\run-20230722_235520-x5e2qe9y</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/tonely/huggingface/runs/x5e2qe9y' target=\"_blank\">spring-sound-132</a></strong> to <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">https://wandb.ai/tonely/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/tonely/huggingface/runs/x5e2qe9y' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/x5e2qe9y</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='3126' max='3126' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [3126/3126 28:30, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 macro score</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>5.664600</td>\n",
       "      <td>1.941329</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.412800</td>\n",
       "      <td>3.758919</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.394800</td>\n",
       "      <td>0.496893</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "\u001b[32m[I 2023-07-23 00:24:04,955]\u001b[0m Trial 1 finished with value: 1.2472731399666013 and parameters: {'learning_rate': 0.1, 'per_device_train_batch_size': 16, 'num_train_epochs': 3}. Best is trial 0 with value: 1.2472731399666013.\u001b[0m\n",
      "Trial: {'learning_rate': 0.01, 'per_device_train_batch_size': 16, 'num_train_epochs': 3}\n",
      "loading configuration file config.json from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\config.json\n",
      "Model config RobertaConfig {\n",
      "  \"_name_or_path\": \"roberta-base\",\n",
      "  \"architectures\": [\n",
      "    \"RobertaForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"bos_token_id\": 0,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-05,\n",
      "  \"max_length\": 512,\n",
      "  \"max_position_embeddings\": 514,\n",
      "  \"model_type\": \"roberta\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"pad_token_id\": 1,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"type_vocab_size\": 1,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 50265\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at C:\\Users\\admin/.cache\\huggingface\\hub\\models--roberta-base\\snapshots\\bc2764f8af2e92b6eb5679868df33e224075ca68\\pytorch_model.bin\n",
      "Some weights of the model checkpoint at roberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight', 'roberta.pooler.dense.weight']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.out_proj.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "C:\\Users\\admin\\anaconda3\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 16657\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 16\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 16\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 3126\n",
      "  Number of trainable parameters = 124647170\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "049034153f3e46959cfda57fa7158fca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='0.001 MB of 0.042 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=0.032111…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/Accuracy</td><td>▁▁▁</td></tr><tr><td>eval/F1 Macro Score</td><td>▁▁▁</td></tr><tr><td>eval/Precision</td><td>▁▁▁</td></tr><tr><td>eval/Recall</td><td>▁▁▁</td></tr><tr><td>eval/loss</td><td>▄█▁</td></tr><tr><td>eval/runtime</td><td>▄█▁</td></tr><tr><td>eval/samples_per_second</td><td>▅▁█</td></tr><tr><td>eval/steps_per_second</td><td>▅▁█</td></tr><tr><td>train/epoch</td><td>▁▂▂▄▅▅▆███</td></tr><tr><td>train/global_step</td><td>▁▂▂▄▅▅▆███</td></tr><tr><td>train/learning_rate</td><td>█▇▅▄▂▁</td></tr><tr><td>train/loss</td><td>█▅▄▃▃▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/Accuracy</td><td>0.80216</td></tr><tr><td>eval/F1 Macro Score</td><td>0.44511</td></tr><tr><td>eval/Precision</td><td>0.0</td></tr><tr><td>eval/Recall</td><td>0.0</td></tr><tr><td>eval/loss</td><td>0.49689</td></tr><tr><td>eval/runtime</td><td>25.5622</td></tr><tr><td>eval/samples_per_second</td><td>72.373</td></tr><tr><td>eval/steps_per_second</td><td>9.076</td></tr><tr><td>train/epoch</td><td>3.0</td></tr><tr><td>train/global_step</td><td>3126</td></tr><tr><td>train/learning_rate</td><td>0.00403</td></tr><tr><td>train/loss</td><td>1.3948</td></tr><tr><td>train/total_flos</td><td>1.314792254739456e+16</td></tr><tr><td>train/train_loss</td><td>4.41426</td></tr><tr><td>train/train_runtime</td><td>1729.7859</td></tr><tr><td>train/train_samples_per_second</td><td>28.889</td></tr><tr><td>train/train_steps_per_second</td><td>1.807</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">spring-sound-132</strong> at: <a href='https://wandb.ai/tonely/huggingface/runs/x5e2qe9y' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/x5e2qe9y</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230722_235520-x5e2qe9y\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Automatic Weights & Biases logging enabled, to disable set os.environ[\"WANDB_DISABLED\"] = \"true\"\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b09f2e919cf44a2d975a09a0d2e814bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='Waiting for wandb.init()...\\r'), FloatProgress(value=0.016933333333145128, max=1.0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.15.5 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.10"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>C:\\Users\\admin\\Desktop\\Uni\\DATA102\\github-repositories\\data102-fake-news\\wandb\\run-20230723_002411-82ejuoiu</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/tonely/huggingface/runs/82ejuoiu' target=\"_blank\">good-wave-133</a></strong> to <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/tonely/huggingface' target=\"_blank\">https://wandb.ai/tonely/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/tonely/huggingface/runs/82ejuoiu' target=\"_blank\">https://wandb.ai/tonely/huggingface/runs/82ejuoiu</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='3126' max='3126' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [3126/3126 28:30, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1 macro score</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.837900</td>\n",
       "      <td>2.268301</td>\n",
       "      <td>0.197838</td>\n",
       "      <td>0.165162</td>\n",
       "      <td>0.197838</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.676300</td>\n",
       "      <td>0.557782</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.579200</td>\n",
       "      <td>0.501915</td>\n",
       "      <td>0.802162</td>\n",
       "      <td>0.445111</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 1850\n",
      "  Batch size = 8\n",
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "\u001b[32m[I 2023-07-23 00:52:55,520]\u001b[0m Trial 2 finished with value: 1.2472731399666013 and parameters: {'learning_rate': 0.01, 'per_device_train_batch_size': 16, 'num_train_epochs': 3}. Best is trial 0 with value: 1.2472731399666013.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "best_trial_roberta = trainer_tuning.hyperparameter_search(\n",
    "    direction = \"maximize\",\n",
    "    backend = \"optuna\",\n",
    "    hp_space = optuna_hp_space,\n",
    "    n_trials = 3\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "2b925b7b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BestRun(run_id='0', objective=1.2472731399666013, hyperparameters={'learning_rate': 0.1, 'per_device_train_batch_size': 8, 'num_train_epochs': 2})"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_trial_roberta"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58d1ee67",
   "metadata": {},
   "source": [
    "In the tuning, three RoBERTa models were created and compared with the following hyperparameter values:\n",
    "* **Learning Rate** = 0.1, **Train Batch Size** = 8, **Number of Train Epochs** = 2\n",
    "* **Learning Rate** = 0.001, **Train Batch Size** = 16, **Number of Train Epochs** = 3\n",
    "* **Learning Rate** = 0.01, **Train Batch Size** = 16, **Number of Train Epochs** = 3\n",
    "\n",
    "Out of these three, the best run for the RoBERTa model was the first model. However, based on the performance on the validation set, we can see that the BERT base still performed better."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae7d6063",
   "metadata": {},
   "source": [
    "##### Saving RoBERTa tuned model\n",
    "\n",
    "To use this model outside of this notebook, we will save the RoBERTa [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer) object and the [`RoBERTa Tokenizer`](https://huggingface.co/docs/transformers/model_doc/roberta#transformers.RobertaTokenizer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "def36697",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./saved_models/RoBERTav1_tuned\n",
      "Configuration saved in ./saved_models/RoBERTav1_tuned\\config.json\n",
      "Model weights saved in ./saved_models/RoBERTav1_tuned\\pytorch_model.bin\n",
      "tokenizer config file saved in ./saved_models/RoBERTav1_tuned\\tokenizer_config.json\n",
      "Special tokens file saved in ./saved_models/RoBERTav1_tuned\\special_tokens_map.json\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('./saved_models/RoBERTav1_tuned\\\\tokenizer_config.json',\n",
       " './saved_models/RoBERTav1_tuned\\\\special_tokens_map.json',\n",
       " './saved_models/RoBERTav1_tuned\\\\vocab.json',\n",
       " './saved_models/RoBERTav1_tuned\\\\merges.txt',\n",
       " './saved_models/RoBERTav1_tuned\\\\added_tokens.json',\n",
       " './saved_models/RoBERTav1_tuned\\\\tokenizer.json')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_for_models ='./saved_models/RoBERTav1_tuned'\n",
    "trainer_tuning.save_model(path_for_models)\n",
    "roberta_tokenizer.save_pretrained(path_for_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec933140",
   "metadata": {},
   "source": [
    "#### Evaluation\n",
    "\n",
    "Last, let us see how the best model from the RoBERTa tuning fared in the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a54749fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 4627\n",
      "  Batch size = 8\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='579' max='579' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [579/579 00:50]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\admin\\AppData\\Roaming\\Python\\Python39\\site-packages\\sklearn\\metrics\\_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.5021297931671143,\n",
       " 'eval_Accuracy': 0.802031553922628,\n",
       " 'eval_F1 Macro Score': 0.4450707603741904,\n",
       " 'eval_Precision': 0.0,\n",
       " 'eval_Recall': 0.0,\n",
       " 'eval_runtime': 54.5782,\n",
       " 'eval_samples_per_second': 84.778,\n",
       " 'eval_steps_per_second': 10.609,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer_tuning.evaluate(eval_dataset = roberta_encoded_dataset['test'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f279a843",
   "metadata": {},
   "source": [
    "From this, it is evident that the BERT base performed better even in the test set compared to the model returned in the tuning.\n",
    "\n",
    "In conclusion, comparing the final models of the BERT and RoBERTa (i.e., which made use of the default values for their hyperparameters and the MAX_LENGTH of 512), the BERT received a higher score for all of the metrics except for Precision. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}