-
Notifications
You must be signed in to change notification settings - Fork 22
/
Copy pathR06_5p_GET_parse_HTML.Rmd
120 lines (81 loc) · 2.36 KB
/
R06_5p_GET_parse_HTML.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
---
title: "Parsing HTML"
author: "Jilung Hsieh"
date: "`r Sys.Date()`"
output: html_document
---
````
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
````
```{r}
library(httr)
library(rvest) # for traversing html structure
library(tidyverse)
```
# Getting county
## No pipeline
```{r}
# Assigning the page URL to var url
# Using browseURL() to examine the url
# STEP 1. Getting url, parsing it as html,
# using read_html() to get back url and assign to a var doc
# read_html() will get back the url and parse it and convert it to a specific datatype, named html_document.
# Checking the class od the doc (xml_doc)
# STEP 2. with doc, Using CSS Selector or XPath to get the data nodes
# html_node() for getting only the first eligible node
# html_nodes() for getting all eligible nodes
# Checking the length of nodes you parsed, using length()
# STEP 3. Converting the selected node into text data
# html_text() to get the content between a pair of openning and closing tags
# html_attr() to get the attribute of a specific element
# print out results
```
# Getting PTT
```{r}
# Assigning url
url <- "https://www.ptt.cc/bbs/BabyMother/index7351.html"
browseURL(url)
# Using read_html() to get back and convert to xml_document
# Checking the clas()
doc <- read_html(url)
class(doc)
# Using html_node() or html_nodes() to get the nodes you want
# Checking the length
nodes <- html_nodes(doc, ".r-ent .title a")
# Using html_text() or html_attr() to convert the node to data
# Getting titles using html_text()
titles <- html_text(nodes)
# Getting links using html_attr()
links <- html_attr(nodes, "href")
links
# setting prefix of url
pre <- "https://www.ptt.cc"
# Combines titles and links to a data.frame and adding prefix url to links
df <- data.frame(titles, links) %>%
mutate(links = str_c(pre, links))
# Examining data
browseURL(df$links[1])
links[1]
browseURL(links[1])
```
# Pipeline
## None Pipeline
```{r}
url <- "https://www.ibon.com.tw/retail_inquiry.aspx#gsc.tab=0"
# Get and parse html -> XML document
doc <- read_html(url)
# Select nodes by CSS selector
nodes <- html_nodes(doc, "#Class1 > option")
# Retrieve text of nodes
counties <- html_text(nodes)
```
## Pipeline
```{r}
counties <- "https://www.ibon.com.tw/retail_inquiry.aspx#gsc.tab=0" %>%
read_html() %>%
html_nodes("#Class1 option") %>%
html_text()
counties
```