Web Scraping

library(SwimmeR)
library(rvest)
library(stringr)
library(dplyr)
library(purrr)
library(ggplot2)

ISL releases their results as .pdf files, and I’ve collected all of those files on github.

The process here is about the same as in my previous web-scraping post.

1. Get web address for where the stuff of interest is
2. Identify CSS for the stuff
3. Make list of links to the stuff with rvest functions html_nodes and html_attr
4. Clean up list of links
5. Use SwimmeR functions to grab contents of list

We’ll start with season 2, since it’s the most recent one. Season 1 will be handled later with the same methods.

The web address is easy, just copy it out of your browser window.

# Part 1 - web address
web_url_season_2 <- "https://github.com/gpilgrim2670/Pilgrim_Data/tree/master/ISL/Season_2_2020"

CSS can be a bit more complicated, but some clicking around with the selector gadget got me what we need.

# Part 2 - CSS
selector <- ".js-navigation-open"

Now for rvest. We’ll use read_html to get the contents of web_url_season_2 and then html_attr and html_nodes to get those contents which are links.

# Part 3 - rvest fun
page_contents <- read_html(web_url_season_2)
links_season_2 <- html_attr(html_nodes(page_contents, selector), "href")
web_url_season_2 <- "https://github.com/gpilgrim2670/Pilgrim_Data/tree/master/ISL/Season_2_2020"
head(links_season_2, 10)

##  [1] ""                                                                                            
##  [2] ""                                                                                            
##  [3] ""                                                                                            
##  [4] ""                                                                                            
##  [5] "/gpilgrim2670/Pilgrim_Data/tree/master/ISL"                                                  
##  [6] "/gpilgrim2670/Pilgrim_Data/blob/master/ISL/Season_2_2020/ISL%202020%20Season%202%20Notes.txt"
##  [7] "/gpilgrim2670/Pilgrim_Data/blob/master/ISL/Season_2_2020/ISL_01112020_Budapest_Match_6.pdf"  
##  [8] "/gpilgrim2670/Pilgrim_Data/blob/master/ISL/Season_2_2020/ISL_05112020_Budapest_Match_7.pdf"  
##  [9] "/gpilgrim2670/Pilgrim_Data/blob/master/ISL/Season_2_2020/ISL_05112020_Budapest_Match_8.pdf"  
## [10] "/gpilgrim2670/Pilgrim_Data/blob/master/ISL/Season_2_2020/ISL_09112020_Budapest_Match_10.pdf"

The first 5 elements (R indexes start at 1) I don’t want. Maybe if I was better at CSS selectors I could have avoided them outright, but I didn’t. I’ll just get rid of them here. The links are also partials, missing their beginnings, so we’ll add “http://github.com” to the beginning of each with paste0. We’ll also need to change “blob” to “raw” so that we just get the .pdfs, rather than the .pdfs and a github landing page. Little github trick there.

# Part 4 - cleaning
links_season_2 <- links_season_2[7:19] # only want links 7-19
links_season_2 <- paste0("https://github.com", links_season_2) # add beginning to link
links_season_2 <- str_replace(links_season_2, "blob", "raw") # replace blob with raw

Now it’s just up to SwimmeR, with a bit of help from purrr. We’ll use read_results inside of map to read in all of the elements of links_season_2. Then we’ll map again, to apply swim_parse_ISL, inside of safely to all the results we read in. We’ll also set splits = TRUE and relay_swimmers = TRUE to capture both splits and relay swimmers. After that we’ll name each of our results based on their source link and then stick them all together with bind_rows.

# Part 5 - deploy SwimmeR
season_2 <- map(links_season_2, read_results)
season_2_parse <- map(season_2, safely(swim_parse_ISL, otherwise = NA), splits = TRUE, relay_swimmers = TRUE)
names(season_2_parse) <- str_split_fixed(links_season_2, "/", n = 10)[, 10]
season_2_parse <- SwimmeR:::discard_errors(season_2_parse)
season_2_df <- bind_rows(season_2_parse, .id = "ID")

A Season 2 Demo

Just to show you what we’ve got in this data set here’s a little exercise to get the major contributors to each team’s point total. Since the ISL is focusing on points and contributing to one’s team rather than times we’ll do the same.

season_2_df <- season_2_df %>% 
  mutate(ID = str_remove(ID, "\\.pdf\\.result")) %>% 
  mutate(Start_Date = as.Date(str_split_fixed(ID, "_", 4)[, 2], format = "%d%m%Y")) %>% 
  mutate(Match = str_split_fixed(ID, "_", 4)[, 4]) %>% 
  mutate(Season = 2) %>% 
  arrange(Start_Date) %>% 
  select(-ID)

ISL teams have 28 swimmers eligible for individual events. If they all contributed equally to the team score then each would account for about 3.5% of the total. That doesn’t happen though, most teams have a few athletes who score most of the points, and the rest are what Shaquille O’Neal refers to as “others” - role players who contribute in different ways. We’re going to name all the athletes who scored more than 7% of their team’s total as the major contributors - those athletes are worth twice their share of points.

What we’ll do here is collect all athletes scoring less than 7% of their team’s total points as “Other” and look at the breakdown for each team. This is also useful because trying to differentiate 28 different colors or point shapes or whatever on a plot, one for each athlete, is real pain.

First we’ll sum up each team’s total score. Mind you some teams competed in the post season and some did not. Teams that participated in the Semis and Championships had the most opportunity to earn points. By dividing each athlete’s total by their team’s total we’re removing this effect. It’s still worth noting though that an athlete scoring 15% of the league leading California Condors’ points has scored a lot more points than another athlete who scored 15% of the league trailing DC Trident.

team_scores <- season_2_df %>% 
  filter(is.na(Name) == FALSE) %>% 
  group_by(Team) %>% 
  summarise(Team_Score = sum(Points, na.rm = TRUE)) # score for each team

Now we need the total scores for each swimmer as a percentage of their team’s total score.

individual_scores <- season_2_df %>%
  filter(is.na(Name) == FALSE) %>%
  group_by(Name) %>%
  summarise(
    Score = sum(Points, na.rm = TRUE), # score for each athlete
    Team = unique(Team)
  ) %>%
  group_by(Team) %>%
  left_join(team_scores) %>% # get team scores
  mutate(Score_Perc = 100 * round(Score/Team_Score, 5)) %>% # calculate percentage of team score for each athlete
  mutate(Name = case_when(Score_Perc < 7 ~ "Other", # rename athletes with less than 7% as other
                          TRUE ~ Name)) %>% 
  group_by(Name, Team) %>% 
  summarise(Score = sum(Score, na.rm = TRUE), # collect all "others" together
            Score_Perc = sum(Score_Perc)) %>%
  group_by(Team) %>% 
  mutate(Score_Rank = rank(desc(Score_Perc))) # get numeric rank for each athlete and other in terms of their percent of the total team score

Here’s the Condors. Four swimmers scored almost 50% of their points, including our favorite, the undefeated queen of ISL 100 breaststroke, Lilly King!

individual_scores %>%
  filter(Team == "CAC") %>% # just want cali condors
  arrange(Score_Perc) %>%
  mutate(Name = factor(Name, unique(Name))) %>% # order athlete names by sore_perc
  ggplot(aes(
    x = "",
    y = Score_Perc,
    fill = Name,
    label = paste0(round(Score_Perc, 2), "%") # put score_perc values in plot
  )) +
  geom_bar(width = 1, stat = "identity") +
  geom_text(size = 4, position = position_stack(vjust = 0.5)) +
  theme_bw() +
  theme(axis.text.x = element_blank(),
        axis.title.x = element_blank()) +
  labs(title = "California Condors",
       y = "Percentage of Total Team Score")

Pie charts aren’t terribly well loved in the data-vis community. They’re fairly panned as being hard to get values off of, and the colors can make them difficult to read. Pie charts are pretty though - they look like flowers. Just look at this, ten pretty, pretty data flowers, one for each team.

individual_scores %>% 
  ggplot(aes(x = "", y = Score_Perc, fill = as.factor(Score_Rank))) +
  geom_bar(width = 1, stat = "identity") +
  coord_polar("y", start = 0) + # converts bar chart into pie chart
  scale_fill_discrete(name = "Point Source", labels = c("Other", "#1 Swimmer", "#2 Swimmer", "#3 Swimmer", "#4 Swimmer", "#5 Swimmer")) + # labels for colors
  facet_wrap(. ~ Team) + # one plot per team
  theme_void() +
  theme(axis.text.x = element_blank()) +
        # legend.position = "none") +
  labs(title = "ISL Season 2")

We can convey a similar point with a chart like this - the percent of each team’s points scored by their top (non-other) athletes.

individual_scores %>% 
  filter(Name != "Other") %>%
  group_by(Team) %>% 
  summarise(Score_Perc = sum(Score_Perc)) %>% 
  arrange(Score_Perc) %>% 
  mutate(Team = factor(Team, unique(Team))) %>% 
  ggplot(aes(x = reorder(Team, Score_Perc), y = Score_Perc)) +
  geom_point() +
  scale_y_continuous(breaks = seq(5, 50, 5)) +
  theme_bw() +
  labs(title = "Percentage Of Total Points Scored by Top Swimmers",
       x = "Team",
       y = "Score Percentage")

Some teams, like the London Roar are more balanced than others, like the Condors are more top heavy. There’s not an obvious lesson to be drawn here though, because LON, CAC, ENS and LAC where the best teams in the league in Season 2 and they’re all over this plot. Also there’s nothing to be surmised about next season, because we don’t know anything about next season. We’re just doing this for fun.

Season 1 Data Set

The ISL made a lot of changes between season 1 and season 2, so they’re not directly comparable. It’s still possible to build a data set from season 1 though. The methods are exactly the same as for season 2 above.

web_url_season_1 <- "https://github.com/gpilgrim2670/Pilgrim_Data/tree/master/ISL/Season_1_2019"
selector <- ".js-navigation-open"

page_contents <- read_html(web_url_season_1)
links_season_1 <- html_attr(html_nodes(page_contents, selector), "href")
links_season_1 <- links_season_1[5:18]
links_season_1 <- paste0("https://github.com", links_season_1)
links_season_1 <- str_replace(links_season_1, "blob", "raw")
links_season_1 <- links_season_1[2:14]

season_1 <- map(links_season_1, read_results)
season_1_parse <- map(season_1, safely(swim_parse_ISL, otherwise = NA), splits = TRUE, relay_swimmers = TRUE)
names(season_1_parse) <- str_split_fixed(links_season_1, "/", n = 10)[, 10]
season_1_parse <- SwimmeR:::discard_errors(season_1_parse)
season_1_df <- bind_rows(season_1_parse, .id = "ID") %>% 
  mutate(ID = str_remove(ID, "\\.pdf\\.result")) %>% 
  mutate(Start_Date = as.Date(str_split_fixed(ID, "_", 4)[, 2], format = "%d%m%Y")) %>% 
  mutate(Match = str_split_fixed(ID, "_", 5)[, 3]) %>% 
  mutate(Season = 1) %>% 
  arrange(Start_Date) %>% 
  select(-ID)

In closing

Thank you for once again joining us here at Swimming + Data Science. We hope to see you back next time, just like we hope to see the ISL back next season!