YouTube
Authenticating with the YouTube API
To collect YouTube data, it is necessary to first create a Google app
with access to the YouTube Data API via the Google APIs
console and generate an associated API key. The following shows the
creation of a YouTube access token by passing a Google developer API key
to the Autnenticate() function:
The YouTube access token can optionally be saved to disk for use in a later session:
The following loads into the current session a previously-created authentication object:
Collecting Data from YouTube
The YouTube video IDs (the part after “=” in the YouTube URL) are required in order to collect YouTube comment data. These IDs can either be manually provided or automatically extracted from the URLs:
The character vector containing the YouTube video IDs or URLs is
passed as a parameter to the Collect() function (the
following code also shows the YouTube access token being piped to
Collect()). In the following example, we are collecting
comments from a YouTube video titled “Australia bushfires - a national
catastrophe | DW News”, which was uploaded by the German Deutsche Welle
news service on 5th January 2020. The comment data were collected on
10th January 2020: the total number of comments at that time was over
1100, but we are using the maxComments parameter to collect
a maximum of 500 top-level comments (and all the reply comments to these
top-level comments).
videoID <- "https://www.youtube.com/watch?v=pJ_NyEYRkLQ"
youtubeData <- youtubeAuth |>
Collect(videoID, maxComments = 500, writeToFile = TRUE)The Collect() function takes the following arguments
(when used for collecting YouTube data): credential (object
generated from Authenticate() with class name “youtube”
(above we pass this via the pipe), videoIDs (character
vector specifying one or more youtube video IDs),
maxComments (numeric integer specifying how many top-level
comments to collect from each video), writeToFile (whether
to write the returned dataframe to disk as an .rds file;
default is FALSE), and verbose (whether to output
information about the data collection; default is FALSE).
Collect() returns an R dataframe with the following
structure (data have been modified to preserve anonymity):
> str(youtubeData)
Classes ‘dataource’, ‘youtube’ and 'data.frame': 603 obs. of 12 variables:
$ Comment : chr "xxxxx"
$ AuthorDisplayName : chr "xx" "xx" "xx" "xx"
$ AuthorProfileImageUrl: chr "https://xx" "https://xx" "https://xx"
$ AuthorChannelUrl : chr "http://xx" "http://xx" "http://xx" "http://xx"
$ AuthorChannelID : chr "xx" "xx" "xx" "xx"
$ ReplyCount : chr "0" "0" "0" "0"
$ LikeCount : chr "0" "0" "0" "0"
$ PublishedAt : chr "2020-01-10T02:23:43" "2020-01-09T20:56:23"
"2020-01-09T20:44:00" "2020-01-09T19:31:32"
$ UpdatedAt : chr "2020-01-10T02:23:43" "2020-01-09T20:56:23"
"2020-01-09T20:44:00" "2020-01-09T19:31:32"
$ CommentID : chr "xx" "xx" "xx" "xx"
$ ParentID : chr NA NA NA NA
$ VideoID : chr "pJ_NyLQ" "pJ_NyLQ" "pJ_NyLQ" "pJ_NyLQ"Creating YouTube Networks
It is currently possible to create two types of networks using YouTube data: (1) actor network and (2) activity network.
Actor Network
In the YouTube actor network the nodes are users who have commented on videos (and the videos themselves are included in the network as special nodes) and the edges are the interactions between users in the comments. We can distinguish a top-level comment, which is a comment that is directed to a video from a reply comment, which is a comment directed to a top-level comment. The YouTube user interface does not allow a user to direct a reply to another reply. However, users can achieve the “reply to a reply” functionality by starting their comment with the username of the person that they are replying to (and they often prepend the username with “@”). So there will be an edge from user i to user j if i replied to a top-level comment authored by j or else i prepended their comment with j’s username.
actorNetwork <- youtubeData |> Create("actor") |> AddText(youtubeData)
actorGraph <- actorNetwork |> Graph(writeToFile = TRUE)Create("actor") returns a named list containing two
dataframes named “nodes” and “edges” (the following has been modified to
preserve anonymity):
> actorNetwork
$nodes
# A tibble: 522 x 3
id screen_name node_type
<chr> <chr> <chr>
1 xxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxx actor
2 xxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxx actor
[snip]
# … with 512 more rows
$edges
# A tibble: 604 x 6
from to video_id comment_id edge_type vosonTxt_comment
<chr> <chr> <chr> <chr> <chr> <chr>
1 xxxxxxxx… VIDEOID… pJ_NyEY… xxxxxxxxxxx… comment "xxxxx"
2 xxxxxxxx… VIDEOID… pJ_NyEY… xxxxxxxxxxx… comment "xxxxx"
[snip]
# … with 594 more rows
attr(,"class")
[1] "list" "network" "actor" "youtube" "voson_text"Note that in the above, AddText() was used to add the
comment text data to the network dataframe, stored as an edge attribute.
Also, note that there is an optional parameter
replies_from_text that can be passed to
AddText() when used with YouTube network creation, to
extract the “reply to reply” comments.
This list is then passed to Graph(), which returns an
igraph graph object. Remember that it is possible to
achieve the above using a single line of code:
The following is an an annonymised summary of the igraph
graph object.
> actorGraph
IGRAPH 79e5456 DN-- 522 604 --
+ attr: type (g/c), name (v/c), screen_name (v/c), node_type (v/c),
| label (v/c), video_id (e/c), comment_id (e/c), edge_type (e/c),
| vosonTxt_comment (e/c)
+ edges from 79e5456 (vertex names):
[1] xxxx->VIDEOID:pJ_NyEYRkLQ
[2] xxxx->VIDEOID:pJ_NyEYRkLQ
[snip]
+ ... omitted several edgesThe YouTube actor network node contains a graph attribute
type (set to “youtube”). The node attributes are:
name (Channel ID, which is YouTube’s unique user ID),
screen_name (the users displayed name),
node_type (‘actor’ or ‘video’) and label (a
concatenation of the ID and screen name). The edge attributes are:
video_id (the ID of the video for which the data have been
collected), comment_id (the ID of the comment),
edge_type (whether the edge is a ‘comment’ i.e. top-level
comment, ‘reply-comment’ i.e. reply to top-level comment or reply to
reply or ‘self-loop’, which is a special edge connecting the video to
itself, as a means of including text posted with the video). In the
above example, because of our earlier use of AddText(),
there is also an edge attribute vosonTxt_comment which is
the text associated with the comment, reply or video.
The example YouTube actor network contains 522 nodes and 604 edges.
The following indicates that there were 500 top-level comments (we
constrained the collection to this number), 103 replies to top-level
comments (note: we did not use AddText() to collect replies
embedded within the text), and there is the single self-loop from the
video to itself.
We can visualize this network, using red to identify the video nodes.
# change color of nodes with type video to red and others grey
V(actorGraph)$color <- ifelse(
V(actorGraph)$node_type == "video", "red", "grey"
)
# open and write plot to a png file
png("youtube_actor.png", width = 600, height = 600)
plot(actorGraph, vertex.label = "", vertex.size = 4, edge.arrow.size = 0.5)
dev.off()
The following creates a sub-network containing only the replies to top-level comments. In removing the other edges (top-level comments and the self-loop) we create a number isolate nodes (nodes with no connections) that we also remove. We have also used red to indicate the people who have written comments containing particular terms that have been present in the online commentary about the bushfires.
# removed edges that are not of type reply-comment
g2 <- delete.edges(
actorGraph, which(E(actorGraph)$edge_type != "reply-comment")
)
# check number of isolates
> length(which(degree(g2) == 0))
[1] 417
# remove isolates
g2 <- delete.vertices(g2, which(degree(g2) == 0))
# get node indexes for the tails of edges that have comments containing
# words of interest change the indexed node colors to red and others grey
V(g2)$color <- "grey"
ind <- tail_of(
actorGraph,
grep("arson|backburn|climate change", tolower(E(g2)$vosonTxt_comment))
)
V(g2)$color[ind] <- "red"
# open and write plot to a png file
png("youtube_actor_reply.png", width = 600, height = 600)
plot(g2, vertex.label = "", vertex.size = 4, edge.arrow.size = 0.5)
dev.off()
Finally, the AddVideoData() function supplements the
network data with additional video information.
AddVideoData() returns a named list containing three
dataframes named “nodes” (identical to the dataframe contained in the
list actorNetwork in the example able), “edges” (this has
three additional columns: “video_title”, “video_description”,
“video_published_at”) and a new dataframe “videos” (the following has
been modified to preserve anonymity):
> actorNetwork_withVideoInfo
$nodes
# A tibble: 522 x 3
id screen_name node_type
<chr> <chr> <chr>
1 xxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxx actor
2 xxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxx actor
[snip]
# … with 512 more rows
$edges
# A tibble: 604 x 9
from to video_id comment_id edge_type vosonTxt_comment video_title
<chr> <chr> <chr> <chr> <chr> <chr> <chr>
1 xxxx… xxxx… pJ_NyEY… xxxxxxxxx… comment xxxxxxxxxxxx … Australia …
2 xxxx… xxxx… pJ_NyEY… xxxxxxxxx… comment "xxxx" Australia …
[snip]
# … with 594 more rows, and 2 more variables: video_description <chr>,
# video_published_at <chr>
$videos
# A tibble: 1 x 6
VideoID VideoTitle VideoDescription VideoPublishedAt ChannelID ChannelTitle
<chr> <chr> <chr> <chr> <chr> <chr>
1 pJ_NyEY… Australia … "As Australia ba… 2020-01-05T12:3… UCknLrEd… DW News
attr(,"class")
[1] "list" "network" "actor" "youtube"
[5] "voson_text" "voson_video_data"It should also be noted that AddVideoData() can
optionally substitute references to the video ID in the “nodes” and
“edges” dataframes with the video publishers channel ID (this is done by
setting the parameter actorSubOnly to TRUE.
Activity Network
In the YouTube activity network, nodes are either comments or videos (videos represent a starting comment).
activityNetwork <- youtubeData |> Create("activity") |> AddText(youtubeData)
activityGraph <- activityNetwork |> Graph()Create("activity") returns a named list containing two
dataframes named “nodes” and “edges” (the following has been modified to
preserve anonymity).
> activityNetwork
$edges
# A tibble: 603 x 3
from to edge_type
<chr> <chr> <chr>
1 xxxxxxxxxxxxxxxxxxxxxxxxxx VIDEOID:pJ_NyEYRkLQ comment
2 xxxxxxxxxxxxxxxxxxxxxxxxxx VIDEOID:pJ_NyEYRkLQ comment
[snip]
# … with 593 more rows
$nodes
# A tibble: 604 x 8
id video_id published_at updated_at author_id screen_name node_type
<chr> <chr> <chr> <chr> <chr> <chr> <chr>
1 xxxx… pJ_NyEY… 2020-01-10T… 2020-01-1… xxxxxxxx… xxxxxxxxxx… comment
2 xxxx… pJ_NyEY… 2020-01-09T… 2020-01-0… xxxxxxxx… xxxxxxxxxx… comment
[snip]
# … with 594 more rows, and 1 more variable: vosonTxt_comment <chr>
attr(,"class")
[1] "list" "network" "activity" "youtube" "voson_text"Note that in the above, AddText() was used to add the
comment text data to the network dataframe, stored as a node attribute.
This list is then passed to Graph(), which returns an
igraph graph object (this has been anonymised).
IGRAPH 02664d1 DN-- 604 603 --
+ attr: type (g/c), name (v/c), video_id (v/c), published_at (v/c),
| updated_at (v/c), author_id (v/c), screen_name (v/c), node_type
| (v/c), vosonTxt_comment (v/c), label (v/c), edge_type (e/c)
+ edges from 02664d1 (vertex names):
[1] xxxx->VIDEOID:pJ_NyEYRkLQ
[2] xxxx->VIDEOID:pJ_NyEYRkLQ
[3] xxxx->VIDEOID:pJ_NyEYRkLQ
[4] xxxx->VIDEOID:pJ_NyEYRkLQ
[5] xxxx->VIDEOID:pJ_NyEYRkLQ
[6] xxxx->VIDEOID:pJ_NyEYRkLQ
+ ... omitted several edgesThe YouTube activity network contains a graph attribute
type (set to “youtube”). The node attributes are:
name (character string ID number for the comment or video),
video_id (character string ID of the video for which the
comments collected - in this example, “pJ_NyEYRkLQ”),
published_at (timestamp of when the comment was published,
this is NA for the video itself), updated_at
(timestamp of when a comment was updated), author_id
(user’s Channel ID), screen_name (user’s display name),
node_type (whether the node is a ‘comment’ i.e. top-level
comment, ‘reply-comment’ i.e. reply to top-level comment or reply to
reply or ‘video’), vosonText_comment (the comment text,
NA for the video), label (concatenation of
name and screen_name). The edge attributes
edge_type which is ‘comment’ for all edges connecting a
top-level comment to the video, and ‘reply-comment’ for all other
edges.
The example YouTube activity network contains 604 nodes and 603
edges. The following is an igraph visualization of the
network, where the video is indicated by a red node, and blue indicates
comments that include one of the following terms: “arson”, “bakcburn”,
“climate change”.
# set all video node colors to red and others to grey
V(activityGraph)$color <- "grey"
V(activityGraph)$color[which(V(activityGraph)$node_type == "video")] <- "red"
# get node indexes of comments that contain terms of interest
# set their node colors to blue
ind <- grep(
"arson|backburn|climate change", tolower(V(activityGraph)$vosonTxt_comment)
)
V(activityGraph)$color[ind] <- "blue"
# open and write plot to a png file
png("youtube_activity.png", width = 600, height = 600)
plot(activityGraph, vertex.label = "", vertex.size = 4, edge.arrow.size = 0.5)
dev.off()
