Data_Processing_Regulations.Rmd

---
title: "Data_Processing_Regulations"
author: "Susanne_Rehorst"
date: "23-5-2022"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

Load or install packages
```{r warning=FALSE}
library(readtext)
library(readr)
library(sylcount)
library(stringr)
library(udpipe)
library(textclean)
library(dplyr)
```

Set working directory
```{r}
setwd("~/Thesis")
```

# Data Processing Regulations
This R Markdown contains the steps to transform the raw text data with ESG reporting standards into cleaned data, so that it is ready for text analysis. 

## Loading in Data
For every ESG regulation, the text is loaded in from a .txt file. Subsequently, every regulation is stored as a string value in one cell. Thereafter, these cells with text strings are merged into one character string. Also, a document ID is created, which can be used for merging different data sets later on.
```{r warning=FALSE}
setwd("~/Thesis/Raw_Data_Text")
Raw_Data <- readtext::readtext("ID*.txt", encoding = "UTF-8")
Vector_Data <- Raw_Data$text
DOC_ID <- c(seq(1, length(Vector_Data), by=1))
rm(Raw_Data)
unloadNamespace("readtext")
```

## Data Cleaning
First, data is cleaned. This includes removing line break expressions, special characters, several punctuation marks, numbers, web addresses, double white spaces, and consecutive periods. Also, contractions are replaced. Punctuation marks that are kept are periods (.). This is because periods are needed to identify sentences.

Moreover, for the readability analysis an extra step is added by removing apostrophes. They need to be removed so that the readability analysis can work optimal. For later analysis, apostrophes are kept, because they are needed to correctly identify lemma's. 
```{r}
## ----- General cleaning that is used for all analysis -----

# Remove line break expressions: 
Vector_Data <- gsub("\r?\n|\r"," ", Vector_Data)
# Remove double quotation marks, comma's, parentheses, special characters and bullet points
Vector_Data <- gsub('[“”"(),•%$€@/#]', '', Vector_Data)
Vector_Data <- gsub("\\[|\\]","",Vector_Data)
# Replace : or ; with period(.)
Vector_Data <- gsub('[:;]', '.', Vector_Data)
# Remove period(.) if it's in between 2 numbers
Vector_Data <- gsub("[0-9].[0-9]","",Vector_Data)
# Remove mixed ordinal numbers (e.g., 1st, 2nd)
Vector_Data <- replace_ordinal(Vector_Data, num.paste = FALSE, remove = TRUE)

# Remove numbers
Vector_Data <- gsub('[0-9]+', '', Vector_Data)

# Remove web addresses
Vector_Data <- gsub('http|www\\S+\\s*',"", Vector_Data)
Vector_Data <- gsub(".//","",Vector_Data)

# Remove double white spaces
Vector_Data <- gsub("\\s+", " ", Vector_Data)
Vector_Data <- str_squish(Vector_Data)
Vector_Data <- gsub("\\s\\.", "\\.", Vector_Data)

# Replace consecutive periods by 1 single period
Vector_Data <- gsub("\\.+",".",Vector_Data)

# Replace contractions. For example "he's" becomes "he is"
Vector_Data <- replace_contraction(Vector_Data, contraction.key = lexicon::key_contractions)

# Final cleaning to make sure all punctuation except periods are removed
Vector_Data <- strip(Vector_Data, char.keep= c("."), digit.remove = TRUE, apostrophe.remove = TRUE,
lower.case = FALSE)

## ----- Special cleaning that is only used for the readability formulas -----

# Remove single quotation marks because it reduces quality of readability analysis.
Vector_Data_Q <- gsub('[\'’]', '', Vector_Data)

```

## Create quantity and readability measures
The Vector_Data_Q can now be used to calculate quantity and readability measures for each document in the sample. This includes calculating the number of words, sentences, characters and syllables for every document. Also, several readability measures are calculated.   
```{r}
Quantity_Data <- readability(Vector_Data_Q) 
Quantity_Data$DOC_ID <- 1:nrow(Quantity_Data) 
rm(Vector_Data_Q)
unloadNamespace("sylcount")

Quantity_Data<- Quantity_Data[!is.na(Quantity_Data$ari), ] #Remove missing values
head(Quantity_Data)
```

## Make text lowercase
After readability measures are calculated, the texts are set to lowercase and remaining punctuations are removed.
```{r}
library(tm)
Vector_Data <- removePunctuation(Vector_Data,
                              preserve_intra_word_contractions = FALSE,
                              preserve_intra_word_dashes = FALSE,
                              ucp = FALSE)
Vector_Data <- tolower(Vector_Data)
unloadNamespace("tm")
```

# Measure restrictiveness of texts
The content of the texts is examined by searching for specific set of words, which are categorized in different categories. Main focus is on restrictive words and advise words. Also, mentions of frameworks are counted. 
```{r}
# ---------------- Create list of words that will be used to count restrictiveness  ------------------------ #

# Restrictive words
requirement <- c("requirement","require","required","requires","requiring")
must <- c("must")
need_to <- c("need to","needs to","needing")
have_to <- c("have to","has to","having to")
shall <- c("shall")
prohibtion <- c("prohibit","prohibited","prohibition","prohibiting")
can_not <- c("can not","cannot","do not")
necessary <- c("necessary")
will_not <- c("will not","will")

# Advice words
should <- c("should","should not")
may <- c("may")
ought <- c("ought")
can <- c("can")
optional <- c("advice","advised","advises","advices","advising")

# Action words
keep <- c("keep","keeps","kept","keeping")
indicate <- c("indicate","indicates","indicated","indicating")
do <- c("do","doing","done")
show <- c("show","shows","showing","showed")
use <- c("use","uses","using","used")
report <- c("report","reports","reporting","reported","disclose","discloses","disclosed","disclosing")

# Strongly negative or strongly positive words
negative_strong <- c("never","none","not","nor","nothing","neither")
positive_strong <- c("always","entire","entirely","everywhere","all","everything")

# Exemptions
exemption <- c("except","exemption","exempted","exempt","exempting","excluded","excluding")

# Examples
example <- c("example","for instance")

# Frameworks
cdp <- c("cdp")
cdsb <- c("cdsb")
gri <- c("gri")
iirc <- c("iirc")
sasb <- c("sasb")
tcfd <- c("tcfd")
sdg <- c("sdg","sdgs")

# ----------- Count the occurrence of words and store the results in a data frame ------------------------- #

# set word boundaries so that only a unique match is counted
requirement<- paste0("\\b",requirement,"\\b")
must<- paste0("\\b",must,"\\b")
need_to<- paste0("\\b",need_to,"\\b")
have_to<- paste0("\\b",have_to,"\\b")
shall<- paste0("\\b",shall,"\\b")
prohibtion<- paste0("\\b",prohibtion,"\\b")
can_not<- paste0("\\b",can_not,"\\b")
necessary<- paste0("\\b",necessary,"\\b")
will_not<- paste0("\\b",will_not,"\\b")
should<- paste0("\\b",should,"\\b")
may<- paste0("\\b",may,"\\b")
ought<- paste0("\\b",ought,"\\b")
can<- paste0("\\b",can,"\\b")
optional<- paste0("\\b",optional,"\\b")
keep<- paste0("\\b",keep,"\\b")
indicate<- paste0("\\b",indicate,"\\b")
do<- paste0("\\b",do,"\\b")
show<- paste0("\\b",show,"\\b")
use<- paste0("\\b",use,"\\b")
report<- paste0("\\b",report,"\\b")
negative_strong<- paste0("\\b",negative_strong,"\\b")
positive_strong<- paste0("\\b",positive_strong,"\\b")
exemption<- paste0("\\b",exemption,"\\b")
example<- paste0("\\b",example,"\\b")
cdp <- paste0("\\b",cdp,"\\b")
cdsb <- paste0("\\b",cdsb,"\\b")
gri <- paste0("\\b",gri,"\\b")
iirc <- paste0("\\b",iirc,"\\b")
sasb <- paste0("\\b",sasb,"\\b")
tcfd <- paste0("\\b",tcfd,"\\b")
sdg <- paste0("\\b",sdg,"\\b")

# Create a data frame to store the word counts. First, create a DOC_ID column
Restriction_Data <- data.frame(DOC_ID)

# Count words
Restriction_Data$requirement<-str_count(Vector_Data,paste(requirement, collapse="|"))
Restriction_Data$must<-str_count(Vector_Data,paste(must, collapse="|"))
Restriction_Data$need_to<-str_count(Vector_Data,paste(need_to, collapse="|"))
Restriction_Data$have_to<-str_count(Vector_Data,paste(have_to, collapse="|"))
Restriction_Data$shall<-str_count(Vector_Data,paste(shall, collapse="|"))
Restriction_Data$prohibtion<-str_count(Vector_Data,paste(prohibtion, collapse="|"))
Restriction_Data$can_not<-str_count(Vector_Data,paste(can_not, collapse="|"))
Restriction_Data$necessary<-str_count(Vector_Data,paste(necessary, collapse="|"))
Restriction_Data$will_not<-str_count(Vector_Data,paste(will_not, collapse="|"))
Restriction_Data$should<-str_count(Vector_Data,paste(should, collapse="|"))
Restriction_Data$may<-str_count(Vector_Data,paste(may, collapse="|"))
Restriction_Data$ought<-str_count(Vector_Data,paste(ought, collapse="|"))
Restriction_Data$can<-str_count(Vector_Data,paste(can, collapse="|"))
Restriction_Data$optional<-str_count(Vector_Data,paste(optional, collapse="|"))
Restriction_Data$keep<-str_count(Vector_Data,paste(keep, collapse="|"))
Restriction_Data$indicate<-str_count(Vector_Data,paste(indicate, collapse="|"))
Restriction_Data$do<-str_count(Vector_Data,paste(do, collapse="|"))
Restriction_Data$show<-str_count(Vector_Data,paste(show, collapse="|"))
Restriction_Data$use<-str_count(Vector_Data,paste(use, collapse="|"))
Restriction_Data$report<-str_count(Vector_Data,paste(report, collapse="|"))
Restriction_Data$negative_strong<-str_count(Vector_Data,paste(negative_strong, collapse="|"))
Restriction_Data$positive_strong<-str_count(Vector_Data,paste(positive_strong, collapse="|"))
Restriction_Data$exemption<-str_count(Vector_Data,paste(exemption, collapse="|"))
Restriction_Data$example<-str_count(Vector_Data,paste(example, collapse="|"))
Restriction_Data$cdp<-str_count(Vector_Data,paste(cdp, collapse="|"))
Restriction_Data$cdsb<-str_count(Vector_Data,paste(cdsb, collapse="|"))
Restriction_Data$gri<-str_count(Vector_Data,paste(gri, collapse="|"))
Restriction_Data$iirc<-str_count(Vector_Data,paste(iirc, collapse="|"))
Restriction_Data$sasb<-str_count(Vector_Data,paste(sasb, collapse="|"))
Restriction_Data$tcfd<-str_count(Vector_Data,paste(tcfd, collapse="|"))
Restriction_Data$sdg<-str_count(Vector_Data,paste(sdg, collapse="|"))

# Sum up word counts within specific categories      
Restriction_Data$restriction <- rowSums(Restriction_Data[,c(2,3,4,5,6,7,8,9,10)])
Restriction_Data$advice <- rowSums(Restriction_Data[,c(11,12,13,14,15)])
Restriction_Data$action <- rowSums(Restriction_Data[,c(16,17,18,19,20,21)])
Restriction_Data$framework <- rowSums(Restriction_Data[,c(26,27,28,29,30,31)])

# Remove DOC_IDs 3, 4 and 5 (missing values).
Restriction_Data <- Restriction_Data[Restriction_Data$DOC_ID != 3 & Restriction_Data$DOC_ID != 4 & Restriction_Data$DOC_ID !=5, ] 

```

## Merge descriptive data
As a next step, the Quantity data and the Restrictive data are merged. Furthermore, they are also merged with the meta data of the regulations documents. By merging these 3 documents, a descriptive analysis can be performed later on. 
```{r}
# Load the meta data
Meta_Data_Regulations <- read_delim("Meta_Data_Regulations.csv", 
    delim = ";", escape_double = FALSE, trim_ws = TRUE)

# Merge the quantity and the restriction data by DOC_ID
Descriptive_Data <- merge(Quantity_Data,Restriction_Data,by.x ="DOC_ID", by.y = "DOC_ID",all.x = TRUE)
rm(Quantity_Data)
rm(Restriction_Data)

# Merge with the meta data
Descriptive_Data <- merge(Descriptive_Data,Meta_Data_Regulations, by.x ="DOC_ID", by.y = "DOC_ID",all.x = TRUE)
rm(Meta_Data_Regulations)

head(Descriptive_Data)
```

## Save the descriptive data
The data set is saved so that it can be accessed directly when performing analysis in the Data_Analysis.Rmd. Also, it can now be removed from the current environment, which frees up space for the remaining part of the data processing in this document. 
```{r}
save(Descriptive_Data, file = "Descriptive_Data.RData")
rm(Descriptive_Data)
```

## Tokenizing and lemmatization
The udpipe package is used to tokenize and lemmatize the text. Also, POS tagging is performed. 

The UD_English-GUM treebank is utilized. Two other potential candidates were the ParTuT treebank and the EWT treebank. After testing all 3 treebanks, the GUM treebank gave the best results in terms of assigning the right POS and lemmatization. For example, the ParTuT treebank and the EWT treebank could not tag 1304 and 750 words respectively, while GUM only had 469 words that could not be tagged. Also, inspecting a sample of the tagged word showed that the quality of tagging was better for the GUM treebank. The GUM treebank consists of 162319 tokens, and built on different text sources (academic, blogs, fiction, government, news, non fiction, social, spoken, web, wiki). 

Reference Udpipe: (Straka et al. 2016) > Straka Milan, Hajič Jan, Straková Jana. UDPipe: Trainable Pipeline for Processing CoNLL-U Files Performing Tokenization, Morphological Analysis, POS Tagging and Parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, May 2016. 
```{r}
# Download GUM model
udpipe_download_model(
  language = "english-gum",
  model_dir = ("~"),
  udpipe_model_repo = "jwijffels/udpipe.models.ud.2.5",
  overwrite = FALSE)
# Load the model
udmodel_english_gum <- udpipe_load_model(file = "~/english-gum-ud-2.5-191206.udpipe")
```

After the model is loaded, the data is tokenized, lemmatized and every token is tagged with a POS tag. Tokenizing the text results in 462249 tokens. 
```{r}
Processed_Data <- udpipe_annotate(udmodel_english_gum, Vector_Data, doc_id = DOC_ID, tokenizer = "tokenizer", tagger = "default", parser = "none")
Processed_Data <- as.data.frame(Processed_Data)
Processed_Data <- Processed_Data %>% select(doc_id, token_id, lemma, upos)
```

Remove vector data after it has been tokenized. 
```{r}
rm(Vector_Data)
```

## Remove stopwords
Remove stop words that do not add meaning to the text. First, a stop word list is retrieved. Subsequently, words that equal the stop word are removed from the corpus. 
```{r}
part1 <- c("a", "able", "about", "above", "according", "accordingly", "across", "actually", "after", "afterwards", "again", "against", "ain't", "allows", "almost", "alone", "along", "already", "also", "although", "am", "among", "amongst", "an", "and", "another", "any", "anybody", "anyhow", "anyone", "anything", "anyway", "anyways", "anywhere", "apart", "appear", "appreciate", "appropriate", "are", "aren't", "around", "as", "a's", "aside", "ask", "asking", "associated", "at", "available", "away", "awfully", "b", "be", "became", "because", "become", "becomes", "becoming", "been", "before", "beforehand", "behind", "being", "believe", "below", "beside", "besides", "best", "better", "between", "beyond", "both", "brief", "but", "by", "c", "came", "cause", "causes", "certain", "certainly", "changes", "clearly", "c'mon", "co", "com", "come", "comes", "concerning", "consequently", "consider", "considering", "contain", "containing", "contains", "corresponding", "could", "couldn't", "course", "c's", "currently", "d", "definitely", "described", "despite", "did", "didn't", "different", "does", "doesn't", "done", "don't", "down", "downwards", "during", "e", "each", "edu", "eg", "eight", "either", "else", "elsewhere", "enough", "especially", "et", "etc", "even", "ever", "every", "everybody", "everyone", "ex", "exactly", "f", "far", "few", "fifth", "first", "five", "followed", "following", "follows", "for", "former", "formerly", "forth", "four", "from", "further", "furthermore", "g", "get", "gets", "getting", "given", "gives", "go", "goes", "going", "gone", "got", "gotten", "greetings", "h", "had", "hadn't", "happens", "hardly", "has", "hasn't", "have", "haven't", "having", "he", "hello", "help", "hence", "her", "here", "hereafter", "hereby", "herein", "here's", "hereupon", "hers", "herself", "he's", "hi", "him", "himself", "his", "hither", "hopefully", "how", "howbeit", "however", "i", "i'd", "ie", "if", "ignored", "i'll", "i'm", "immediate", "in", "inasmuch", "inc", "indeed", "indicated", "indicates", "inner", "insofar", "instead", "into", "inward", "is", "isn't", "it", "it'd", "it'll", "its", "it's", "itself", "i've", "j", "just", "k", "keeps", "kept", "know", "known", "knows", "l", "last", "lately", "later", "latter", "latterly", "least", "less", "lest", "let", "let's", "like", "liked", "likely", "little", "look", "looking", "looks", "ltd", "m", "mainly", "many", "maybe", "me", "mean", "meanwhile", "merely", "might", "more", "moreover", "most", "mostly", "much", "my", "myself", "n", "name", "namely", "nd", "near", "nearly", "needs", "nevertheless", "new", "next", "nine", "nobody", "non", "noone", "normally", "novel", "now", "nowhere", "o", "obviously", "of", "off", "often", "oh", "ok", "okay", "old", "on", "once", "one", "ones", "only", "onto", "or", "other", "others", "otherwise", "our", "ours", "ourselves", "out", "outside", "over", "overall", "own", "p", "particular", "particularly", "per", "perhaps", "placed", "please", "plus", "possible", "presumably", "probably", "provides", "q", "que", "quite", "qv", "r", "rather", "rd", "re", "really", "reasonably", "regarding", "regardless", "regards", "relatively", "respectively", "right", "s", "said", "same", "saw", "say", "saying", "says", "second", "secondly", "see", "seeing", "seem", "seemed", "seeming", "seems", "seen", "self", "selves", "sensible", "sent", "serious", "seriously", "seven", "several", "she", "since", "six", "so", "some", "somebody", "somehow", "someone", "something", "sometime", "sometimes", "somewhat", "somewhere", "soon", "sorry", "specified", "specify", "specifying", "still", "sub", "such", "sup", "sure", "t", "take", "taken", "tell", "tends", "th", "than", "thank", "thanks", "thanx", "that", "thats", "that's", "the", "their", "theirs", "them", "themselves", "then", "thence", "there", "thereafter", "thereby", "therefore", "therein", "theres", "there's", "thereupon", "these", "they", "they'd", "they'll", "they're", "they've", "think", "third", "this", "thorough", "thoroughly", "those", "though", "three", "through", "throughout", "thru", "thus")

part2 <- c("to", "together", "too", "took", "toward", "towards", "tried", "tries", "truly", "try", "trying", "t's", "twice", "two", "u", "un", "under", "unfortunately", "unless", "unlikely", "until", "unto", "up", "upon", "us", "used", "useful", "uses", "usually", "uucp", "v", "value", "various", "very", "via", "viz", "vs", "w", "want", "wants", "was", "wasn't", "way", "we", "we'd", "welcome", "well", "we'll", "went", "were", "we're", "weren't", "we've", "what", "whatever", "what's", "when", "whence", "whenever", "where", "whereafter", "whereas", "whereby", "wherein", "where's", "whereupon", "wherever", "whether", "which", "while", "whither", "who", "whoever", "whole", "whom", "who's", "whose", "why", "willing", "wish", "with", "within", "without", "wonder", "would", "would", "wouldn't", "x", "y", "yes", "yet", "you", "you'd", "you'll", "your", "you're", "yours", "yourself", "yourselves", "you've", "z", "zero")

stopwords <- c(part1, part2)
rm(part1)
rm(part2)
```

After removing stopwords, there are 266994 tokens remaining. 
```{r}
Processed_Data <- Processed_Data[ ! Processed_Data$lemma %in% stopwords, ]
```

## Filter out tokens based on POS
Distribution of POS tags is examined.
```{r}
Processed_Data %>% count(upos, sort=FALSE)
```

Only "open class" words are maintained. These are verbs, adverbs, nouns, proper nouns, adjectives, and auxiliary verbs. Also, words that couldn't be assigned are maintained ("X"). After filtering on POS tags, there are 255248 tokens remaining. 
```{r}
POS <- c("ADJ","ADV","AUX","NOUN","PROPN","VERB","X")
Processed_Data <- Processed_Data[Processed_Data$upos %in% POS, ]   
```

Remove DOC_ID 3, 4, and 5. They do not contain any text.
```{r}
Processed_Data <- Processed_Data[Processed_Data$doc_id != "3" & Processed_Data$doc_id != "4" & Processed_Data$doc_id !="5", ] 
```

## Save the processed text data
Data is saved, so that it can be analyzed in the Data_Analysis.Rmd markdown file.  
```{r}
# Tidy text format table
save(Processed_Data, file = "Processed_Data.RData")
```