Exploratory and Explanatory Data Analysis

Follow your Curiosity, Focus your Story

Last Time

1. Plain text files store characters; binary files need special software.
2. YAML, CSV, and JSON add structure to plain text.

Key Ideas for Today

1. EDA uncovers structure and generates new questions.
2. Explanatory plots tell one story to one audience.
3. Labels and scales clarify and focus comparisons.
4. Themes and set aesthetics polish the look.

Concept Acquisition

1. EDA is for the analyst: its goal is to uncover structure and generate new questions, guided by a set of common exploratory questions.
2. Exploratory and explanatory visualizations differ in audience, goal, and level of polish.
3. Axis limits, non-linear scales, and the ordering of categories decide which comparisons a plot makes easy.
4. A theme controls the non-data elements of a plot; theme() tweaks the current theme, so add it after a complete theme.

Tool Acquisition

1. dim(), sapply(df, class), and sum() / mean() of is.na() for a first look.
2. geom_bar(), geom_col(), and count().
3. labs() for titles, axis labels, and legends.
4. lims(), scale_x_log10(), scale_y_sqrt(), fct_infreq(), and fct_reorder().
5. theme_gray() / theme_bw() / theme_minimal(), theme(axis.text.x = element_text(...)), and setting color / fill.

Concept Application

1. Your Turn: sketch the data frame behind the EV bar chart, describe its mappings and geometry, and state its main story.
2. Work through the common exploratory questions on the EV registrations data.
3. Build two explanatory versions of the EV plot, each for a different story.

EDA

Draw a diagram showing the “computational machine” fron day one with the question originating outside the box and the first phase within it being Data Cleaning.

Then draw an arrow from Data Cleaning to a second phase, EDA, that can generate it’s own questions. Draw those questions outside the box but with arrows to and fro. Then draw arrow an arrow from EDA back to Data Cleaning.

Exploratory Data Analysis (EDA)

• consumer: the analyst
• goal: uncover structure for further analysis, generate new questions
• visualization often follows templates
• prioritize quick and accurate interpretation

Common Exploratory Questions

• What is the unit of observation?
• What are the variables, their types, and levels?
• How is missingness encoded? How prevalent is it?
• What is the distribution of the variables?
• What is the relationship between the variables?
• Are there any interesting outliers?

The Dashboard is a classic exploratory tool Use View() in Positron

Guess the Metaphor

EDA is like an archaeological dig: you never know what you might find and the more you discover, the more you dig.

Your Turn

Questions:

1. Sketch the data frame that was used to create this plot.
2. Describe the aesthetic mappings and geometry used.
3. What is the main story?

Write these on the board.

02:30

As you debrief can also ask an explanatory question: “what is effective about this plot? How could it be improved?””

Exploratory Data Analysis

The Data

library(readr)
ev <- read_csv("data/ev-registrations.csv")
ev

# A tibble: 51 × 2
   State                  Count
   <chr>                  <dbl>
 1 Alabama                13047
 2 Alaska                  2697
 3 Arizona                89798
 4 Arkansas                7108
 5 California           1256646
 6 Colorado               90083
 7 Connecticut            31557
 8 Delaware                8435
 9 District of Columbia    8066
10 Florida               254878
# ℹ 41 more rows

Q1: What are the cols and rows?

dim(ev)

[1] 51  2

sapply(ev, class)

      State       Count 
"character"   "numeric" 

Q1: What are the cols and rows?

ev

# A tibble: 51 × 2
   State                  Count
   <chr>                  <dbl>
 1 Alabama                13047
 2 Alaska                  2697
 3 Arizona                89798
 4 Arkansas                7108
 5 California           1256646
 6 Colorado               90083
 7 Connecticut            31557
 8 Delaware                8435
 9 District of Columbia    8066
10 Florida               254878
# ℹ 41 more rows

• Unit of observation: a state (or DC)
• Variables: name (character) and registration count (double)

Q2: Is there any missing data?

Option 1: Total missing counts

library(dplyr)
ev |>
  summarize(n_missing = sum(is.na(Count)))

# A tibble: 1 × 1
  n_missing
      <int>
1         0

Option 2: Proportion missing counts

ev |>
  summarize(p_missing = mean(is.na(Count)))

# A tibble: 1 × 1
  p_missing
      <dbl>
1         0

Q3: What is the distribution of count?

library(ggplot2)
ev |>
  ggplot(aes(x = Count)) +
  geom_bar()

Q3: What is the distribution of count?

ev |>
  ggplot(aes(x = State)) +
  geom_bar()

Q3: What is the distribution of count?

ev |>
  ggplot(aes(y = State)) +
  geom_bar()

geom_bar()

Counts up the number of observations in each level of a single variable, then draws bars up to that height.

library(palmerpenguins)
penguins

# A tibble: 344 × 8
   species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
   <fct>   <fct>              <dbl>         <dbl>             <int>       <int>
 1 Adelie  Torgersen           39.1          18.7               181        3750
 2 Adelie  Torgersen           39.5          17.4               186        3800
 3 Adelie  Torgersen           40.3          18                 195        3250
 4 Adelie  Torgersen           NA            NA                  NA          NA
 5 Adelie  Torgersen           36.7          19.3               193        3450
 6 Adelie  Torgersen           39.3          20.6               190        3650
 7 Adelie  Torgersen           38.9          17.8               181        3625
 8 Adelie  Torgersen           39.2          19.6               195        4675
 9 Adelie  Torgersen           34.1          18.1               193        3475
10 Adelie  Torgersen           42            20.2               190        4250
# ℹ 334 more rows
# ℹ 2 more variables: sex <fct>, year <int>

geom_bar()

Counts up the number of observations in each level of a single variable, then draws bars up to that height.

library(palmerpenguins)
penguins |>
  ggplot(aes(y = island)) +
  geom_bar()

Summarizing with counts

penguins |>
  group_by(island) |>
  summarize(count = n())

# A tibble: 3 × 2
  island    count
  <fct>     <int>
1 Biscoe      168
2 Dream       124
3 Torgersen    52

Summarizing with counts

penguins |>
  count(island)

# A tibble: 3 × 2
  island        n
  <fct>     <int>
1 Biscoe      168
2 Dream       124
3 Torgersen    52

geom_col()

Takes one column of categories and draws a bar for each up to the height of a second column of counts.

penguins |>
  count(island) |>
  ggplot(aes(y = island, x = n)) +
  geom_col()

Q3: What is the distribution of count?

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col()

Q3: What is the distribution of count?

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5))

Exploratory Data Analysis (EDA)

Features

• consumer: the analyst
• goal: uncover structure for further analysis
• follow templates
• prioritize quick and accurate interpretation

Common Exploratory Questions

• What is the unit of observation?
• What are the variables, their types, and levels?
• How is missingness encoded? How prevalent is it?
• What is the dist. of the variables?
• What is the relationship between the variables?

• Are there any interesting outliers?

Explanatory Data Analysis

Explanatory Data Analysis

Features

• consumer: stakeholder
• goal: tell a specific story
• customized
• polished aesthetic

Common Explanatory Questions

• Who is my audience?
• What is the main story?
• How can I facilitate comparisons?
• Can my plot be clearly read?
• Is it polished and customized for audience?

Gameplan

1. Clarify with Labels
2. Facilitate Comparisons
3. Polish Your Look
   1. Themes
   2. Setting aesthetics
4. Electric Vehicles Revisited

Clarify with Labels

Annotate plot showing the following labels: x, y, color, main, subtitle.

Labels with labs()

The labels on the plot can refer to many different places where text is added to describe what is plotted.

• title
• subtitle
• x-axis
• y-axis
• legends

In ggplot2, these are arguments to a labs() layer.

• title
• subtitle
• color
• fill
• size
• shape

penguins |>
  ggplot(aes(x = bill_length_mm,
             y = flipper_length_mm,
             color = species)) +
  geom_point()

penguins |>
  ggplot(aes(x = bill_length_mm,
             y = flipper_length_mm,
             color = species)) +
  geom_point() +
  labs(title = "Penguin size differs by island",
       x = "Bill Length (mm)",
       y = "Bill Depth (mm)",
       color = "Species")

Facilitate Comparisons

Facilitating Comparison

1. How does this compare to that?
2. How does this compare to what’s typical?
3. How big is this variability relative to that variability?

Compares one species to another species.

p2 <- penguins |>
  ggplot(aes(x = bill_length_mm,
             y = flipper_length_mm,
             color = species)) +
  geom_point() +
  labs(title = "Penguin size differs by island",
       x = "Bill Length (mm)",
       y = "Bill Depth (mm)",
       color = "Species") +
  lims(x = c(0, 60),
       y = c(0, 233))

p2

Compares each species to a baseline of zero.

p1

#  scale_x_log10()

Pay attention to where the x = 40 and x = 60 lines are…

p1 +
  scale_x_log10()

Limits and Scales

Emphasize certain comparisons (and occlude others) by altering the limits of the axes to include a relevant baseline and by using non-linear scales.

• lims()
• scale_x_log10()
• scale_y_log10()
• General scale_ family of layers

Facilitating Comparison

1. How does this compare to that?
2. How does this compare to what’s typical?
3. How big is this variability relative to that variability?

How could I make it easiest to understand the relative commonality of the three species?

penguins |>
  ggplot(aes(x = species)) +
  geom_bar()

forcats

library(forcats) # useful functions for factors
penguins |> 
  mutate(species = fct_infreq(species)) |>
  ggplot(aes(x = species)) +
  geom_bar()

Polish Your Look

Return to the previous slide and note the aspects of the plot that aren’t data and aren’t labels: the grid lines, the axis tick marks, the font, the font size, the background. These are all elements of the theme.

Theme

A set of visual design choices for the plot on things like the background, the gridlines, the axes, the tick marks, and the fonts.

Some ggplot2 themes (each one can be added as layer)

• theme_gray(): the default
• theme_bw()
• theme_minimal()

p1 <- penguins |>
  ggplot(aes(x = bill_length_mm,
             y = flipper_length_mm,
             color = species)) +
  geom_point() +
  labs(title = "Penguin size differs by island",
       x = "Bill Length (mm)",
       y = "Bill Depth (mm)",
       color = "Species")
p1

p1 +
  theme_bw()

p1 +
  theme_minimal()

EV: Explanatory Plot A

The Plan

Audience: Residents and decisions makers in all fifty states.

My story: There is considerable variability from state to state in EV registrations.

My strategy: Display the registration counts in a manner that makes it easy to see the counts of each individual state equally well.

Where we left off

ev <- read_csv("data/ev-registrations.csv")
ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5))

Where we left off

Labels

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count")

Labels

Labels

Labels

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count")

Labels

What next?

Facilitating Comparisons

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt()

Facilitating Comparisons

Facilitating Comparisons

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt(labels = scales::comma)

Facilitating Comparisons

What next?

Themes

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") + 
  scale_y_sqrt(labels = scales::comma) +
  theme_minimal()

Themes

Themes

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme_minimal(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt(labels = scales::comma)

Error in `theme_minimal()`:
! unused arguments (axis.text.x = element_text(angle = 270, hjust = 0, vjust = 0.5), plot.title = element_text(hjust = 0.5))

Themes

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col() +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt(labels = scales::comma)

Themes

Themes

Themes describe a set of visual design choices made for the plot on things like the background, the gridlines, the axes, the tick marks, and the fonts.

Some ggplot2 themes (each one can be added as layer)

• theme_gray(): the default
• theme_bw()
• theme_minimal()

theme() will change defaults of the exiting theme, so add it last.

What next?

Setting Aesthetics

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col(color = "blue") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt(labels = scales::comma)

Setting Aesthetics

Setting Aesthetics

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col(fill = "blue") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt(labels = scales::comma)

Setting Aesthetics

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col(fill = "steelblue") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt(labels = scales::comma)

EV: Explanatory Plot B

The Plan

Audience: Residents and decisions makers in all fifty states.

My story: Just a few states account for most of the electrical vehicle registrations

My strategy: Display the registration counts in a manner that draws attention to the discrepancy between the highets and lowest states.

ev |>
  ggplot(aes(x = State, y = Count)) +
  geom_col(fill = "steelblue") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt(labels = scales::comma)

ev |>
  mutate(State = fct_reorder(State, Count, .desc = TRUE)) |>
  ggplot(aes(x = State, y = Count)) +
  geom_col(fill = "steelblue") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count") +
  scale_y_sqrt(labels = scales::comma)

ev |>
  mutate(State = fct_reorder(State, Count, .desc = TRUE)) |>
  ggplot(aes(x = State, y = Count)) +
  geom_col(fill = "steelblue") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 270, 
                                   hjust = 0, 
                                   vjust = 0.5),
        plot.title = element_text(hjust = 0.5)) +
  labs(title = "Electric Vehicle Registrations by State",
       x = "",
       y = "Registration Count")

Key Ideas for Today

1. EDA uncovers structure and generates new questions.
2. Explanatory plots tell one story to one audience.
3. Labels and scales clarify and focus comparisons.
4. Themes and set aesthetics polish the look.

For Next Time

1. Quiz 3 is Mon/Tue (10/5-10/6) on PrairieTest.
2. Start PS 4, due Saturday 10/17 at 11:59 pm.
3. Keep collecting your personal data exports for Project 3.