Data Wrangling II

Pipelines and Grouped Operations

Last Time

1. git tracks changes to files.
2. A repository (repo) is a directory tracked by git.
3. The staging area holds changes for a snapshot.
4. A commit is a snapshot of staged changes with a message.
5. A remote is a web-hosted repo (GitHub) you push to and clone from.

Key Ideas for Today

1. Two more verbs: distinct() and count().
2. Chain operations using the pipe operator (|>).
3. Debug a pipeline by breaking the pipe and inspecting each stage.
4. group_by() puts rows into groups for downstream verbs.

Concept Acquisition

1. Nesting, step-by-step objects, and pipelines are three ways to express the same multi-step computation; each trades off readability, intermediate clutter, and repetition of the data frame name.
2. The native pipe |> follows magrittr’s %>% and works with any function, not just dplyr verbs.
3. Grouping changes the scope of the functions inside a verb: summarize() returns one row per group, filter() and mutate() evaluate within groups, and arrange() ignores grouping entirely.
4. A grouped data frame stays grouped until it is reduced to one row per group or explicitly ungrouped.

Tool Acquisition

1. |>.
2. group_by(), ungroup(), and the .by argument.
3. Grouped forms of summarize(), filter(), mutate(), and arrange().

Concept Application

1. Rewrite a nested or step-by-step computation as a pipeline.
2. Predict the dimensions of the data frame at each stage of a pipe (the A/B/C/D question).
3. Compute a group-wise summary and a group-wise z-score, and explain why the second requires ungroup() or .by downstream.

Before class, sketch out the effect that the different dplyr verbs from last class have on the shape of a data frame.

Two more verbs

library(dplyr)
star_wars <- data.frame(
    name = c("Anakin", "Padme", "Luke", "JarJar"),
    homeworld = c("Tatooine", "Naboo", "Tatooine", "Naboo"),
    height = c(1.8, 1.6, 1.7, 1.9),
    weight = c(84, 45, 77, 90)
)

Two questions we can’t yet answer with slice(), select(), filter(), mutate(), arrange(), and summarize():

1. Which homeworlds show up in this data set?
2. How many characters come from each one?

distinct()

Returns the unique rows of a data frame.

star_wars

    name homeworld height weight
1 Anakin  Tatooine    1.8     84
2  Padme     Naboo    1.6     45
3   Luke  Tatooine    1.7     77
4 JarJar     Naboo    1.9     90

distinct(star_wars, homeworld)

  homeworld
1  Tatooine
2     Naboo

distinct()

Returns the unique rows of a data frame.

star_wars

    name homeworld height weight
1 Anakin  Tatooine    1.8     84
2  Padme     Naboo    1.6     45
3   Luke  Tatooine    1.7     77
4 JarJar     Naboo    1.9     90

distinct(star_wars, homeworld,
         heavy = weight > 50)

  homeworld heavy
1  Tatooine  TRUE
2     Naboo FALSE
3     Naboo  TRUE

With multiple variables, it returns the unique combinations.

distinct()

Returns the unique rows of a data frame.

star_wars

    name homeworld height weight
1 Anakin  Tatooine    1.8     84
2  Padme     Naboo    1.6     45
3   Luke  Tatooine    1.7     77
4 JarJar     Naboo    1.9     90

distinct(star_wars)

    name homeworld height weight
1 Anakin  Tatooine    1.8     84
2  Padme     Naboo    1.6     45
3   Luke  Tatooine    1.7     77
4 JarJar     Naboo    1.9     90

With no variables named, it works on the whole data frame — a quick check for duplicate rows.

count()

Counts the number of rows for each value of a variable.

star_wars

    name homeworld height weight
1 Anakin  Tatooine    1.8     84
2  Padme     Naboo    1.6     45
3   Luke  Tatooine    1.7     77
4 JarJar     Naboo    1.9     90

count(star_wars, homeworld)

  homeworld n
1     Naboo 2
2  Tatooine 2

Same rows as distinct(), plus a column n of counts.

count()

Counts the number of rows for each value of a variable.

star_wars

    name homeworld height weight
1 Anakin  Tatooine    1.8     84
2  Padme     Naboo    1.6     45
3   Luke  Tatooine    1.7     77
4 JarJar     Naboo    1.9     90

count(star_wars, homeworld,
      heavy = weight > 50)

  homeworld heavy n
1     Naboo FALSE 1
2     Naboo  TRUE 1
3  Tatooine  TRUE 2

Like distinct(), it counts unique combinations of several variables.

Data pipelines

library(dplyr)
star_wars <- data.frame(
    name = c("Anakin", "Padme", "Luke", "JarJar"),
    homeworld = c("Tatooine", "Naboo", "Tatooine", "Naboo"),
    height = c(1.8, 1.6, 1.7, 1.9),
    weight = c(84, 45, 77, 90)
)

Using base R data frame subsetting, create from star_wars…

4. The average height of characters from Tatooine

Let’s look at three ways to solve this.

Nesting

summarize(filter(star_wars, homeworld == "Tatooine"), mean(height))

  mean(height)
1         1.75

• Must be read from the inside out 👎
• Hard to keep track of arguments 👎

• All in one line of code 👍
• Only refer to one data frame 👍

Step-by-step

star_wars2 <- filter(star_wars, homeworld == "Tatooine")
summarize(star_wars2, mean(height))

  mean(height)
1         1.75

• Have to repeat data frame names 👎
• Creates unnecessary objects 👎

• Stores intermediate objects 👍
• Can be read from top to bottom 👍

Data Pipelines in R

2016: magrittr introduces %>%

|>

2021:

R now provides a simple native forward pipe syntax |>. The simple form of the forward pipe inserts the left-hand side as the first argument in the right-hand side call.

Using the Pipe Operator

star_wars |>

Using the Pipe Operator

star_wars |>
    filter(homeworld == "Tatooine") |>

Using the Pipe Operator

star_wars |>
    filter(homeworld == "Tatooine") |>
    summarize(mean(height))

  mean(height)
1         1.75

• 🤷‍♂️

• Can be read like an english paragraph 👍
• Only type the data once 👍
• No leftover objects 👍

Understanding your pipeline

It’s good practice to understand the output of each line by breaking the pipe.

star_wars |>
    select(homeworld) |>
    filter(mean(height))

Error in `filter()`:
ℹ In argument: `mean(height)`.
Caused by error:
! object 'height' not found

star_wars |>
    select(homeworld)

  homeworld
1  Tatooine
2     Naboo
3  Tatooine
4     Naboo

star_wars |> # A #<<
    filter(homeworld == "Naboo",
           name %in% c("Padme", "JarJar")) |> # B #<<
    select(height, weight) |> # C #<<
    summarize(mean(height), sd(height),
              mean(weight), sd(weight)) # D #<<

Question: What are the dimensions of the data frame at each stage of the pipe: A, B, C, and D?

02:00

Show answer in drop by breaking the pipe

|> works everywhere

R now provides a simple native forward pipe syntax |>. The simple form of the forward pipe inserts the left-hand side as the first argument in the right-hand side call.

factor(c("cat", "cat", "dog")) |> summary()

cat dog 
  2   1 

Grouped Operations

library(dplyr)
star_wars <- data.frame(
    name = c("Anakin", "Padme", "Luke", "JarJar"),
    homeworld = c("Tatooine", "Naboo", "Tatooine", "Naboo"),
    height = c(1.8, 1.6, 1.7, 1.9),
    weight = c(84, 45, 77, 90)
)

Using base R data frame subsetting, create from star_wars…

4. The average height of characters from Tatooine across each planet

You could set up two pipelines with different filters. But there’s a better way.

group_by()

Flags the rows of a data frame as belonging to a group defined by a variable, for use in downstream operations.

star_wars |>
    group_by(homeworld)

# A tibble: 4 × 4
# Groups:   homeworld [2]
  name   homeworld height weight
  <chr>  <chr>      <dbl>  <dbl>
1 Anakin Tatooine     1.8     84
2 Padme  Naboo        1.6     45
3 Luke   Tatooine     1.7     77
4 JarJar Naboo        1.9     90

group_by()

Flags the rows of a data frame as belonging to a group defined by a variable, for use in downstream operations.

star_wars |>
    group_by(homeworld) |>
    summarize(mean(height))

# A tibble: 2 × 2
  homeworld `mean(height)`
  <chr>              <dbl>
1 Naboo               1.75
2 Tatooine            1.75

Draw diagram

group_by() with summarize()

star_wars |>
    summarize(mean(height))

  mean(height)
1         1.75

group_by() with summarize()

star_wars |>
    group_by(homeworld) |>
    summarize(mean(height))

# A tibble: 2 × 2
  homeworld `mean(height)`
  <chr>              <dbl>
1 Naboo               1.75
2 Tatooine            1.75

Makes a summary row for each group.

group_by() + summarize() + n()

star_wars |>
    group_by(homeworld) |>
    summarize(n = n())

# A tibble: 2 × 2
  homeworld     n
  <chr>     <int>
1 Naboo         2
2 Tatooine      2

Same result as count(star_wars, homeworld) — count() is a shortcut for this exact pattern.

group_by() with two variables

star_wars |>
    group_by(homeworld, heavy = weight > 50) |>
    summarize(n = n())

# A tibble: 3 × 3
# Groups:   homeworld [2]
  homeworld heavy     n
  <chr>     <lgl> <int>
1 Naboo     FALSE     1
2 Naboo     TRUE      1
3 Tatooine  TRUE      2

Groups are defined by every combination of the variables — same idea as count(star_wars, homeworld, heavy = weight > 50).

group_by() with filter()

star_wars |>
    filter(height == max(height))

    name homeworld height weight
1 JarJar     Naboo    1.9     90

group_by() with filter()

star_wars |>
    group_by(homeworld) |>
    filter(height == max(height))

# A tibble: 2 × 4
# Groups:   homeworld [2]
  name   homeworld height weight
  <chr>  <chr>      <dbl>  <dbl>
1 Anakin Tatooine     1.8     84
2 JarJar Naboo        1.9     90

Changes the scope of functions inside filter() to operate within groups.

Can bring up drop to demonstrate how a condition like height > 1.7 will operate the same with or without the grouping.

group_by() with arrange()

star_wars |>
    arrange(desc(height))

    name homeworld height weight
1 JarJar     Naboo    1.9     90
2 Anakin  Tatooine    1.8     84
3   Luke  Tatooine    1.7     77
4  Padme     Naboo    1.6     45

group_by() with arrange()

star_wars |>
    group_by(homeworld) |>
    arrange(desc(height))

# A tibble: 4 × 4
# Groups:   homeworld [2]
  name   homeworld height weight
  <chr>  <chr>      <dbl>  <dbl>
1 JarJar Naboo        1.9     90
2 Anakin Tatooine     1.8     84
3 Luke   Tatooine     1.7     77
4 Padme  Naboo        1.6     45

Arrange ignores group_by() and is always global

Why???

star_wars |>
    arrange(homeworld, height)

    name homeworld height weight
1  Padme     Naboo    1.6     45
2 JarJar     Naboo    1.9     90
3   Luke  Tatooine    1.7     77
4 Anakin  Tatooine    1.8     84

group_by() with mutate()

star_wars |>
    mutate(height_z = (height - mean(height)) / sd(height))

    name homeworld height weight   height_z
1 Anakin  Tatooine    1.8     84  0.3872983
2  Padme     Naboo    1.6     45 -1.1618950
3   Luke  Tatooine    1.7     77 -0.3872983
4 JarJar     Naboo    1.9     90  1.1618950

group_by() with mutate()

star_wars |>
    group_by(homeworld) |>
    mutate(height_z = (height - mean(height)) / sd(height))

# A tibble: 4 × 5
# Groups:   homeworld [2]
  name   homeworld height weight height_z
  <chr>  <chr>      <dbl>  <dbl>    <dbl>
1 Anakin Tatooine     1.8     84    0.707
2 Padme  Naboo        1.6     45   -0.707
3 Luke   Tatooine     1.7     77   -0.707
4 JarJar Naboo        1.9     90    0.707

Changes the scope of functions inside mutate() to operate within groups.

Statefulness

What will this produce?

star_wars |>
    group_by(homeworld) |>
    mutate(height_z = (height - mean(height)) / sd(height)) |>
    summarize(mean(height_z))

# A tibble: 2 × 2
  homeworld `mean(height_z)`
  <chr>                <dbl>
1 Naboo                    0
2 Tatooine                 0

Once grouped, a data frame stays grouped until reduced to one-row-per group or it is ungrouped.

can break the pipe after mutate to show it is still grouped

Statefulness

What will this produce?

star_wars |>
    group_by(homeworld) |>
    mutate(height_z = (height - mean(height)) / sd(height)) |>
    ungroup() |>
    summarize(mean(height_z))

# A tibble: 1 × 1
  `mean(height_z)`
             <dbl>
1                0

A shortcut

.by

An argument available in most dplyr functions that applies the function to the groups of another variable.

• Not stateful
• Good for single line operations

star_wars |>
    summarize(mean(height), .by = homeworld)

  homeworld mean(height)
1  Tatooine         1.75
2     Naboo         1.75

Guess the Metaphor

Group-by are like tents: they put soldiers into groups so that you can do operations tent-by-tent instead of across all soldiers.

soldiers <- data.frame(
    name = c("Ali", "Boone", "Cruz", "Diaz", "Eze", "Fox"),
    tent = c("A", "A", "B", "B", "C", "C"),
    rations = c(3, 5, 2, 4, 6, 2))

soldiers |>
    group_by(tent) |>
    summarize(soldiers = n(), 
    rations = sum(rations))

# A tibble: 3 × 3
  tent  soldiers rations
  <chr>    <int>   <dbl>
1 A            2       8
2 B            2       6
3 C            2       8

Key Ideas for Today

1. Two more verbs: distinct() and count().
2. Chain operations using the pipe operator (|>).
3. Debug a pipeline by breaking the pipe and inspecting each stage.
4. group_by() puts rows into groups for downstream verbs.

For Next Time

1. Project 2 is underway — read the project description and worksheet.
2. Start PS 3 (due 10/03 at 11:59 pm).