02-lineups.Rmd

# Perception through lineups {#lineups}

## Introduction

To lay a foundation for future exploration of the use of log scales, we begin with the most fundamental ability: to identify differences in charts. Identifying differences does not require that participants understand exponential growth, identify log scales, or have any mathematical training. 
Instead, we are simply testing the change in \emph{perceptual sensitivity} resulting from visualization choices.
The study in this chapter is conducted through visual inference and the use of statistical lineups [@buja_statistical_2009] to differentiate between exponentially increasing curves with differing levels of curvature, using linear and log scales.

## Visual Inference

In [Section 1.4](#testing-statistical-graphics), we explained how a data plot can be evaluated and treated as a visual statistic, a numerical function which summarizes the data.
To evaluate a graph, the statistic (data plot) must be run through a visual evaluation - a person.
We can conclude that two visualization methods are significantly different if the visual evaluation is different.
Recent graphical experiments have utilized statistical lineups to quantify the perception of graphical design choices [@hofmann_graphical_2012; @loy_model_2017; @loy_variations_2016; @vanderplas_clusters_2017]. 
Statistical lineups provide an elegant way of combining perception and statistical hypothesis testing using graphical experiments [@majumder_validation_2013; @vanderplas_testing_2020; @wickham2010graphical].
'Lineups' are named after the 'police lineup' of criminal investigations where witnesses are asked to identify the criminal from a set of individuals. 
Similarly, a statistical lineup is a plot consisting of smaller panels where the viewer is asked to identify the panel containing the real data from among a set of decoy null plots. 
Null plots display data under the assumption there is no relationship and can be generated by permutation or simulation.
A statistical lineup typically consists of 20 panels - one target panel and 19 null panels.
If the viewer can identify the target panel randomly embedded within the set of null panels, this suggests that the real data is visually distinct from data generated under the null model. 
\cref{fig:lineup-example} provides examples of statistical lineups. 
The lineup plot on the left displays increasing exponential data displayed on a linear scale with panel 13 as the target; the lineup plot on the right displays increasing exponential data on the log base ten scale with panel 4 as the target.

```{r lineup-example, fig.height = 2.75, fig.width = 5.75, fig.scap = "Lineup examples", fig.cap = "The lineup plot on the left displays increasing exponential data on a linear scale with panel (2 x 5) + 3 as the target. The lineup plot on the right displays increasing exponential data on the log scale with panel 2 x 2 as the target."}
lineupData_linear <- read.csv(file = "data/01-lineups/lineup-example-data-linear.csv")
linearPlot <- ggplot(lineupData_linear, aes(x=x, y=y)) +
  facet_wrap(~.sample, ncol=5) +
  geom_point(size = .05) +
  theme(aspect.ratio = 1) +
  theme_bw(base_size = 14) +
  theme(axis.title.y = element_blank(),
        axis.title.x = element_blank(),
        axis.text.y  = element_blank(),
        axis.text.x  = element_blank(),
        strip.text = element_text(size = 5, margin = margin(0.05,0,0.05,0, "cm")),
        strip.background = element_rect(size = 0.5)
  )

lineupData_log <- read.csv(file = "data/01-lineups/lineup-example-data-log.csv")
logPlot <- ggplot(lineupData_log, aes(x=x, y=y)) +
  facet_wrap(~.sample, ncol=5) +
  geom_point(size = .05) +
  theme(aspect.ratio = 1) +
  theme_bw(base_size = 14) +
  theme(axis.title.y = element_blank(),
        axis.title.x = element_blank(),
        axis.text.y  = element_blank(),
        axis.text.x  = element_blank(),
        strip.text = element_text(size = 5, margin = margin(0.05,0,0.05,0, "cm")),
        strip.background = element_rect(size = 0.5)
  ) +
  scale_y_continuous(trans = "log10")

grid.arrange(linearPlot, logPlot, ncol = 2)
```

While explicit graphical tests direct the participant to a specific feature of a plot to answer a specific question, implicit graphical tests require the user to identify both the purpose and function of the plot in order to evaluate the plots shown [@vanderplas_testing_2020]. 
Implicit graphical tests, such as lineups, have the advantage of simultaneously visually testing for multiple visual features including outliers, clusters, linear and nonlinear relationships. Responses from multiple viewers are collected through convenience sampling (in informal situations) or crowd sourcing websites such as Prolific, Amazon Mechanical Turk, and Reddit (in more formal situations).

## Data Generation

In this study, both the target and null data sets were generated by simulating data from an exponential model; the models differ in the parameters selected for the null and target panels. 
In order to guarantee the simulated data spans the same domain and range of values, we began with a domain constraint of $x\in [0,20]$ and a range constraint of $y\in [10,100]$ with $N = 50$ points randomly assigned throughout the domain and mapped to the $y$-axis using the exponential model with the selected parameters. 
These constraints provide some assurance that participants who select the target plot are doing so because of their visual perception differentiating between curvature or growth rate rather than different starting or ending values.

Data were simulated based on a three-parameter exponential model with multiplicative errors: 
\begin{align}
y_i & = \alpha\cdot e^{\beta\cdot x_i + \epsilon_i} + \theta \\
\text{with } \epsilon_i & \sim N(0, \sigma^2). \nonumber
\end{align} 
The parameters $\alpha$ and $\theta$ were adjusted based on $\beta$ and $\sigma^2$ to guarantee the range and domain constraints are met. 
The model generated $N = 50$ points $(x_i, y_i), i = 1,...,N$ where $x$ and $y$ have an increasing exponential relationship. 
The heuristic data generation procedure is described in \cref{alg:lineup-parameter-estimation-algorithm} and \cref{alg:lineup-exponential-data-simulation-algorithm}.

\begin{algorithm}
  \caption{Lineup Parameter Estimation}\label{alg:lineup-parameter-estimation-algorithm}
  \begin{algorithmic}[1]
    \Statex \textbullet~\textbf{Input Parameters:} domain $x\in[0,20]$, range $y\in[10,100]$, midpoint $x_{mid}$.
    \Statex \textbullet~\textbf{Output Parameters:} estimated model parameters $\hat\alpha, \hat\beta, \hat\theta$.
    \State Determine the $y=-x$ line scaled to fit the assigned domain and range.
    \State Map the values $x_{mid} - 0.1$ and $x_{mid} + 0.1$ to the $y=-x$ line for two additional points.
    \State From the set of points $(x_k, y_k)$ for $k = 1,2,3,4$, calculate the coefficients from the linear regression model $\ln(y_k) = b_0 +b_1x_k$ to obtain starting values - $\alpha_0 = e^{b_0}, \beta_0 =  b_1, \theta_0 = 0.5\cdot \min(y)$
    \State Using the \texttt{nls} function from the base \texttt{stats} package in Rstudio and the starting parameter values - $\alpha_0, \beta_0, \theta_0$ - fit the nonlinear model, $y_k = \alpha\cdot e^{\beta\cdot x_k}+\theta$ to get estimated parameter values - $\hat\alpha, \hat\beta, \hat\theta.$
  \end{algorithmic}
\end{algorithm}

\begin{algorithm}
  \caption{Lineup Exponential Data Simulation}\label{alg:lineup-exponential-data-simulation-algorithm}
  \begin{algorithmic}[1]
    \Statex \textbullet~\textbf{Input Parameters:} sample size $N = 50$, estimated parameters $\hat\alpha$, $\hat\beta$, and $\hat\theta$, from \cref{alg:lineup-parameter-estimation-algorithm}, and standard deviation $\sigma$ from the exponential curve.
    \Statex \textbullet~\textbf{Output Parameters:} $N$ points, in the form of vectors $\mathbf{x}$ and $\mathbf{y}$.
    \State Generate $\tilde x_j, j = 1,..., \frac{3}{4}N$ as a sequence of evenly spaced points in $[0,20]$. This ensures the full domain of $x$ is used, fulfilling the constraints of spanning the same domain and range for each parameter combination.
    \State Obtain $\tilde x_i, i = 1,...N$ by sampling $N = 50$ values from the set of $\tilde x_j$ values. This guarantees some variability and potential clustering in the exponential growth curve disrupting the perception due to continuity of points.
    \State Obtain the final $x_i$ values by jittering $\tilde x_i$.
    \State Calculate $\tilde\alpha = \frac{\hat\alpha}{e^{\sigma^2/2}}.$ This ensures that the range of simulated values for different standard deviation parameters has an equal expected value for a given rate of change due to the non-constant variance across the domain.
    \State Generate $y_i = \tilde\alpha\cdot e^{\hat\beta x_i + e_i}+\hat\theta$ where $e_i\sim N(0,\sigma^2).$
  \end{algorithmic}
\end{algorithm}

## Parameter Selection {#lineups-parameter-selection}

We followed a 'Goldilocks' inspired procedure to choose three levels of trend curvature (low curvature, medium curvature, and high curvature). For each curvature level, we simulated 1,000 data sets of $(x_{ij}, y_{ij})$ points for $i = 1,...,50$ increments of $x$-values and replicate $j = 1,...,10$ corresponding $y$-values per $x$-value.
Each generated $x_i$ point from \cref{alg:lineup-exponential-data-simulation-algorithm} was replicated ten times. 
On each of the individual data sets, we fit a linear regression model and computed the lack of fit statistic (LOF) which measures the deviation of the data from the linear regression model.
The density curves of the LOF statistics for each level of curvature are plotted \pcref{fig:lof-density-curves} to provide a metric for differentiating between the curvature levels and thus detecting the target plot.
While the LOF statistic provides a numerical value for discriminating between the difficulty levels, it cannot be directly related to the perceptual discriminability; it serves primarily as an approximation to ensure that we are testing parameters at several distinct curvature levels.
Final parameters used for data simulation are shown in \cref{tab:parameter-data}.

```{r lof-density-curves, fig.height = 2.5, fig.width = 5, fig.scap = "Lineup parameter selection", fig.cap = "Density plot of the lack of fit statistic showing separation of difficulty levels: obvious curvature, noticable curvature, and almost linear.", out.width = "100%"}
lofData <- read.csv(file = "data/01-lineups/lineup-lof-data.csv")
lofPlot_curvature <- lofData %>%
  mutate(Curvature = factor(Curvature, levels = c("Obvious Curvature", "Noticeable Curvature", "Almost Linear"), labels = c("High Curvature", "Medium Curvature", "Low Curvature"))) %>%
  mutate(Variability = factor(Variability, levels = c("Low"))) %>%
  ggplot(aes(x = statistic, fill = Curvature, color = Curvature)) +
  geom_density(alpha = 0.7) +
  scale_fill_manual("Difficulty", values = c("#004400", "#116611", "#55aa55")) +
  scale_color_manual("Difficulty", values = c("#004400", "#116611", "#55aa55")) +
  theme_bw(base_size = 14) +
  theme(legend.position = "bottom",
        axis.text    = element_text(size = 10),
        axis.title   = element_text(size = 10),
        legend.title = element_text(size = 10),
        legend.text  = element_text(size = 10),
        legend.key.size = unit(0.5, "line")
        ) +
  scale_x_continuous("Lack of Fit Statistic") +
  scale_y_continuous("Density")
lofPlot_curvature
```

```{r parameter-data}
parameter_data <- read.csv(file = "data/01-lineups/lineup-parameter-data.csv")
parameter_data %>%
  mutate(difficulty = ifelse(difficulty == "Obvious Curvature", "High Curvature",
                             ifelse(difficulty == "Noticable Curvature", "Medium Curvature", "Low Curvature"))
         ) %>%
  select(difficulty, xMid, alphahat, alphatilde, betahat, thetahat, sigma_vals) %>%
  knitr::kable("latex", digits = 2, escape = F, booktabs = T, linesep = "", align = "c", label = "parameter-data",
        col.names = c("",   "$x_{mid}$", "$\\hat\\alpha$", "$\\tilde\\alpha$", "$\\hat\\beta$", "$\\hat\\theta$", "$\\hat\\sigma$"),
        caption = "Lineup data simulation final parameters")
```

## Lineup Setup 

Lineup plots were generated by mapping one simulated data set corresponding to curvature level A to a scatter plot to be identified as the target panel while multiple simulated data sets corresponding to curvature level B were individually mapped to scatter plots for the null panels.
The `nullabor` package in R [@nullabor] was used to randomly assign the target plot to one of the panels surrounded by panels containing null plots.
For example, a target plot with simulated data following an increasing exponential curve with high curvature is randomly embedded within null plots with simulated data following an increasing exponential trend with low curvature. 
By the implemented constraints, the target panel and null panels spanned a similar domain and range. 
There were a total of six lineup curvature combinations; \cref{fig:curvature-combination-example} illustrates the six lineup curvature combinations (top: linear scale; bottom: log scale) where the green line indicates the curvature level designated to the target plot while the black line indicates the curvature level assigned to the null plots.
Two sets of each lineup curvature combination were simulated (total of twelve test data sets) and plotted on both the linear scale and the log scale (total of 24 test lineup plots).
In addition, there were three curvature combinations which generated homogeneous "Rorschach" lineups, where all panels were from the same distribution. 
Each participant evaluated one of these lineups, but for simplicity, these evaluations are not described in this chapter and their analysis is left to a later date.

```{r curvature-combination-example, fig.height = 6, fig.width = 9, fig.scap = "Lineup curvature combinations", fig.cap = "Thumbnail plots illustrating the six curvature combinations displayed on both scales (linear and log). The green line indicates the curvature level to be identified as the target plot from amongst a set of null plots with the curvature level indicated by the black line.", out.width="100%"}
simData <- read.csv("data/01-lineups/difficulty-comparison-data.csv")

# simData  %>%
#   mutate(curvature = factor(curvature, levels = c("E", "M", "H"), labels = c("High Curvature", "Medium Curvature", "Low Curvature"))) %>%
#   ggplot(aes(x = x, y = y, color = curvature)) +
#   geom_line(size = 1) +
#   theme_bw() +
#   theme(aspect.ratio = 1) +
#   theme(axis.text = element_blank(),
#         axis.title = element_blank(),
#         legend.position = "bottom") +
#   scale_color_manual("Difficulty", values = c("#004400", "#116611", "#55aa55"))

tE_nM <- simData  %>%
  mutate(Target = "High",
         Null = "Medium") %>%
  filter(curvature %in% c("E", "M")) %>%
  mutate(curvature = factor(curvature, levels = c("E", "M", "H"))) %>%
  ggplot(aes(x = x, y = y, color = curvature)) +
  geom_line(size = 1) +
  theme_test() +
  theme(aspect.ratio = 1) +
  theme(axis.text = element_blank(),
        axis.title = element_blank(),
        legend.position = "none") +
  scale_x_continuous(limits = c(0,20)) +
  scale_color_manual(values = c("green3", "black"))

tE_nH <- simData  %>%
  mutate(Target = "High",
         Null = "Low") %>%
  filter(curvature %in% c("E", "H")) %>%
  mutate(curvature = factor(curvature, levels = c("E", "M", "H"))) %>%
  ggplot(aes(x = x, y = y, color = curvature)) +
  geom_line(size = 1) +
  theme_test() +
  theme(aspect.ratio = 1) +
  theme(axis.text = element_blank(),
        axis.title = element_blank(),
        legend.position = "none") +
  scale_x_continuous(limits = c(0,20)) +
  scale_color_manual(values = c("green3", "black"))

tM_nE <- simData  %>%
  mutate(Target = "Medium",
         Null = "High") %>%
  filter(curvature %in% c("E", "M")) %>%
  mutate(curvature = factor(curvature, levels = c("E", "M", "H"))) %>%
  ggplot(aes(x = x, y = y, color = curvature)) +
  geom_line(size = 1) +
  theme_test() +
  theme(aspect.ratio = 1) +
  theme(axis.text = element_blank(),
        axis.title = element_blank(),
        legend.position = "none") +
  scale_x_continuous(limits = c(0,20)) +
  scale_color_manual(values = c("black", "green3"))

tM_nH <- simData  %>%
  mutate(Target = "Medium",
         Null = "Low") %>%
  filter(curvature %in% c("H", "M")) %>%
  mutate(curvature = factor(curvature, levels = c("E", "M", "H"))) %>%
  ggplot(aes(x = x, y = y, color = curvature)) +
  geom_line(size = 1) +
  theme_test() +
  theme(aspect.ratio = 1) +
  theme(axis.text = element_blank(),
        axis.title = element_blank(),
        legend.position = "none") +
  scale_x_continuous(limits = c(0,20)) +
  scale_color_manual(values = c("green3", "black"))

tH_nE <- simData  %>%
  mutate(Target = "Low",
         Null = "High") %>%
  filter(curvature %in% c("E", "H")) %>%
  mutate(curvature = factor(curvature, levels = c("E", "M", "H"))) %>%
  ggplot(aes(x = x, y = y, color = curvature)) +
  geom_line(size = 1) +
  theme_test() +
  theme(aspect.ratio = 1) +
  theme(axis.text = element_blank(),
        axis.title = element_blank(),
        legend.position = "none") +
  scale_x_continuous(limits = c(0,20)) +
  scale_color_manual(values = c("black", "green3"))

tH_nM <- simData  %>%
  mutate(Target = "Low",
         Null = "Medium") %>%
  filter(curvature %in% c("H", "M")) %>%
  mutate(curvature = factor(curvature, levels = c("E", "M", "H"))) %>%
  ggplot(aes(x = x, y = y, color = curvature)) +
  geom_line(size = 1) +
  theme_test() +
  theme(aspect.ratio = 1) +
  theme(axis.text = element_blank(),
        axis.title = element_blank(),
        legend.position = "none") +
  scale_x_continuous(limits = c(0,20)) +
  scale_color_manual(values = c("black", "green3"))

# linear scale

tM_nH_linear <- tM_nH +
  scale_y_continuous(limits = c(10,100))

tE_nH_linear <- tE_nH +
  scale_y_continuous(limits = c(10,100))

tH_nM_linear <- tH_nM +
  scale_y_continuous(limits = c(10,100))

tE_nM_linear <- tE_nM +
  scale_y_continuous(limits = c(10,100))

tH_nE_linear <- tH_nE +
  scale_y_continuous(limits = c(10,100))

tM_nE_linear <- tM_nE +
  scale_y_continuous(limits = c(10,100))

# log scale

library(scales)

tM_nH_log <- tM_nH +
  scale_y_continuous(limits = c(10,100), trans = "log10")

tE_nH_log <- tE_nH +
  scale_y_continuous(limits = c(10,100), trans = "log10")

tH_nM_log <- tH_nM +
  scale_y_continuous(limits = c(10,100), trans = "log10")

tE_nM_log <- tE_nM +
  scale_y_continuous(limits = c(10,100), trans = "log10")

tH_nE_log <- tH_nE +
  scale_y_continuous(limits = c(10,100), trans = "log10")

tM_nE_log <- tM_nE +
  scale_y_continuous(limits = c(10,100), trans = "log10")


# output plots with patchwork
tM_nH_linear +
  ggtitle("Linear Scale") +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tE_nH_linear +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tH_nM_linear +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tE_nM_linear +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tH_nE_linear +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tM_nE_linear +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tM_nH_log +
  ggtitle("Log10 Scale") +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tE_nH_log +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tH_nM_log +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tE_nM_log +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tH_nE_log +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
tM_nE_log +
  facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
plot_layout(ncol = 6)

# tM_nH_linear +
#   ggtitle("Linear Scale") +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tM_nH_log +
#   ggtitle("Log10 Scale") +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tE_nH_linear +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tE_nH_log +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tH_nM_linear +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tH_nM_log +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tE_nM_linear +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tE_nM_log +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tH_nE_linear +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tH_nE_log +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tM_nE_linear +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# tM_nE_log +
#   facet_grid(~ Target + Null, labeller = labeller(Target = label_both, Null = label_both)) +
# plot_layout(ncol = 2)

# save thumbnails for plotting later
# tM_nH_thumbnail <- tM_nH_linear + tM_nH_log
# ggsave(tM_nH_thumbnail, file = "images/01-lineups/tM_nH_thumbnail.png", width = 6, height = 3)
# 
# tE_nH_thumbnail <- tE_nH_linear + tE_nH_log
# ggsave(tE_nH_thumbnail, file = "images/01-lineups/tE_nH_thumbnail.png", width = 6, height = 3)
# 
# tH_nM_thumbnail <- tH_nM_linear + tH_nM_log
# ggsave(tH_nM_thumbnail, file = "images/01-lineups/tH_nM_thumbnail.png", width = 6, height = 3)
# 
# tE_nM_thumbnail <- tE_nM_linear + tE_nM_log
# ggsave(tE_nM_thumbnail, file = "images/01-lineups/tE_nM_thumbnail.png", width = 6, height = 3)
# 
# tH_nE_thumbnail <- tH_nE_linear + tH_nE_log
# ggsave(tH_nE_thumbnail, file = "images/01-lineups/tH_nE_thumbnail.png", width = 6, height = 3)
# 
# tM_nE_thumbnail <- tM_nE_linear + tM_nE_log
# ggsave(tM_nE_thumbnail, file = "images/01-lineups/tM_nE_thumbnail.png", width = 6, height = 3)
```

## Study Design

Each participant was shown a total of thirteen lineup plots (twelve test lineup plots and one Rorschach lineup plot). 
Participants were randomly assigned one of the two replicate data sets for each of the six unique lineup curvature combinations. 
For each assigned test data set, the participant was shown the lineup plot corresponding to both the linear scale and the log scale. 
For the additional Rorschach lineup plot, participants were randomly assigned one data set shown on either the linear or the log scale. 
The order of the thirteen lineup plots shown was randomized for each participant. 

Participants above the age of majority in their region were recruited from Prolific, a survey site that connects researchers to study participants.
Participants were compensated for their time and participated in all three related graphical studies consecutively. 
Previous literature suggests that prior mathematical knowledge or experience with exponential data is not associated with the outcome of graphical experiments involving lineups[@vanderplas2015spatial]. 
The lineup study in this chapter was completed first in the series of graphical studies.
<!-- Participants completed the series of graphical tests using a R Shiny application found [here](https://shiny.srvanderplas.com/perception-of-statistical-graphics/). -->

Participants were shown a series of lineup plots and asked to identify the plot that was most different from the others. 
On each plot, participants were asked to justify their choice and provide their level of confidence in their choice.
The goal of this graphical task was to test an individual's ability to perceptually differentiate exponentially increasing trends with differing levels of curvature on both the linear and log scale. 

## Results

Participant recruitment and study deployment were conducted via Prolific, a crowd sourcing website, on Wednesday, March 23, 2022 during which 325 individuals completed 4,492 unique test lineup evaluations. 
<!-- Previous studies have found that results do not differ on lineup-related tasks between Reddit and other crowd sourcing websites such as Amazon Mechanical Turk [@vanderplas_clusters_2017]. -->
Only participants who completed the lineup study were included in the final data set which included a total of 311 participants and 3,958 lineup evaluations.
Each plot was evaluated between 141 and 203 times (Mean: 164.92, SD: 14.9).
Participants correctly identified the target panel in 47\% of the 1,981 lineup evaluations made on the linear scale and 65.3\% of the 1,977 lineup evaluations made on the log scale.

Each lineup plot evaluated was assigned a binary value based on the participant response (correct target plot identification = 1, not correct target plot identification = 0).
We defined $Y_{ijkl}$ to be the event that participant $l = 1,...,N_\text{participant}$ correctly identified the target plot for data set $k = 1,2$ with curvature combination $j = 1,2,3,4,5,6$ plotted on scale $i = 1,2$.
The binary response was analyzed using a generalized linear mixed model (GLMM) following a binomial distribution with a logit link function with a row-column blocking design accounting for the variation due to participant and data set respectively as 
\begin{equation}
\text{logit }P(Y_{ijk}) = \eta + \delta_i + \gamma_j + \delta \gamma_{ij} + s_l + d_k
\end{equation}
\noindent where
\begin{itemize}
\item $\eta$ is the baseline average probability of selecting the target plot
\item $\delta_i$ is the effect of scale $i = 1,2$
\item $\gamma_j$ is the effect of curvature combination $j = 1,2,3,4,5,6$
\item $\delta\gamma_{ij}$ is the two-way interaction between the $i^{th}$ scale and $j^{th}$ curvature combination
\item $s_l \sim N(0,\sigma^2_\text{participant})$ is the random effect for participant characteristics
\item $d_k \sim N(0,\sigma^2_{\text{data}})$ is the random effect for data specific characteristics. 
\end{itemize}
\noindent We assumed that random effects for data set and participant are independent.
Target plot identification was analyzed using a GLMM implemented in `glmer` from the `lme4` R package [@lme4]. 
Estimates and odds ratio comparisons between the log and linear scales were calculated using the `emmeans` R package [@emmeans].

Results indicated a strong interaction between the curvature combination and scale ($\chi^2_5 = 294.443$; $\text{p} <0.0001$). Variance due to participant and data set were estimated to be $\hat\sigma^2_{\text{participant}} = 1.19$ (s.e. = 1.09) and $\hat\sigma^2_{\text{data}} = 0.433$ (s.e. = 0.66), respectively.

```{r anova, include = F}
read.csv("data/01-lineups/lineups-anova.csv") %>%
  knitr::kable("latex", escape = F, booktabs = T, linesep = "", align = "c", 
        label = "lineup-anova-table",
        caption = "Lineup ANOVA table for fixed effects.")
```

On both the log and linear scales, the highest accuracy occurred in lineup plots where the target model and null model had a large curvature difference and the target plot had more curvature than the null plots (high curvature target plot embedded in low curvature null plots).
There is a decrease in accuracy on the linear scale when comparing a target plot with less curvature to null plots with more curvature (medium curvature target plot embedded in high curvature null plots; low curvature target plot embedded in medium curvature null plots; low curvature target plot embedded in high curvature null plots). 
@best_perception_2007 found that accuracy of identifying the correct curve type was higher when nonlinear trends were presented indicating that it is hard to say something is linear (something has less curvature), but easy to say that it is not linear; our results concur with this observation.
\cref{fig:odds-ratio-plot} displays the estimated (log) odds ratio of successfully identifying the target panel on the log scale compared to the linear scale. The thumbnail figures to the right of the plot illustrate the curvature combination on both the linear (left thumbnail) and log base ten (right thumbnail) scales associated with the $y$-axis label.
The choice of scale had no impact if curvature differences are large and the target plot had more curvature than the null plots (high curvature target plot embedded in low curvature null plots).
However, presenting data on the log scale makes us more sensitive to slight changes in curvature (low or high curvature target plot embedded in medium curvature null plots; medium curvature target plot embedded in high curvature null plots) and large differences in curvature when the target plot had less curvature than the null plots (low curvature target plot embedded in high curvature null plots).
An exception occured when identifying a plot with curvature embedded in null plots close to a linear trend (medium curvature target panel embedded in low curvature null panels).
The results indicate that participants were more accurate at detecting the target panel on the linear scale than the log scale.
When examining this curvature combination, the same perceptual effect occurred as what we previously saw, but in a different context of scales.
On the linear scale, participants were perceptually identifying a curved trend from close to a linear trend whereas after the logarithmic transformation, participants were perceptually identifying a trend close to linear from a curved trend.
This again supports the claim that it is easy to identify a curve in a bunch of lines but harder to identify a line in a bunch of curves [@best_perception_2007].

```{r odds-ratio-plot, echo = F, eval = T, fig.width = 9, fig.height = 6, fig.align='center', fig.scap = "Lineups log(odds) results", fig.cap = "Estimated (log) odds ratio of successfully identifying the target panel on the log scale compared to the linear scale. The y-axis indicates the the model parameters used to simulate the null plots with the target plot model parameter selection designated by shape and shade of green. The thumbnail figures on the right display the curvature combination as shown in \\cref{fig:curvature-combination-example} on both scales (linear - left, log - right).", message = F, warning = F}
lineups_or <- read.csv("data/01-lineups/lineups-odds-ratios.csv")

dodge <- position_dodge(width=0.9)
lineups_or_plot <- lineups_or %>%
  separate(curvature, into = c(NA, "target", NA, "null")) %>%
  mutate(null = factor(null, levels = c("E", "M", "H"), labels = c("High Curvature", "Medium Curvature", "Low Curvature")),
         target = factor(target, levels = c("E", "M", "H"), labels = c("High Curvature", "Medium Curvature", "Low Curvature"))) %>%
  ggplot(aes(x = odds.ratio, y = null, color = target, shape = target)) + 
  geom_point(position = dodge, size = 3) + 
  geom_errorbar(aes(xmin = asymp.LCL, xmax = asymp.UCL), position = dodge, width = .1) +
  geom_vline(xintercept = 1) +
  theme_bw()  +
  theme(axis.title = element_text(size = 12),
        axis.text = element_text(size = 12),
        legend.title = element_text(size = 12),
        legend.text  = element_text(size = 12),
        legend.key.size = unit(1, "line"),
        legend.position = "bottom"
  ) +
  scale_y_discrete("Null Panel Difficulty", position = "left") +
  scale_x_continuous("Odds ratio (on log scale) \n (Log vs Linear)", trans = "log10", labels = scales::comma) + 
  scale_color_manual("Target Panel Difficulty", values = c("#004400", "#116611", "#55aa55")) + 
  scale_shape_discrete("Target Panel Difficulty")

picsList <- c("images/01-lineups/tM_nH_thumbnail.png", 
              "images/01-lineups/tE_nH_thumbnail.png", 
              "images/01-lineups/tH_nM_thumbnail.png", 
              "images/01-lineups/tE_nM_thumbnail.png", 
              "images/01-lineups/tH_nE_thumbnail.png", 
              "images/01-lineups/tM_nE_thumbnail.png"
)

pimage <- axis_canvas(lineups_or_plot, axis = 'y') + 
  draw_image(picsList[1], y = 2.75, scale = 0.75) +
  draw_image(picsList[2], y = 2.25, scale = 0.75) +
  draw_image(picsList[3], y = 1.75, scale = 0.75) +
  draw_image(picsList[4], y = 1.25, scale = 0.75) +
  draw_image(picsList[5], y = 0.75, scale = 0.75) +
  draw_image(picsList[6], y = 0.25, scale = 0.75)

# insert the image strip into the plot
ggdraw(insert_yaxis_grob(lineups_or_plot, pimage, position = "right", clip = "on"))
```

## Discussion and Conclusion

The overall goal of this chapter is to provide basic research to support the principles used to guide design decisions in scientific visualizations of exponential data. 
In this study, we explored the use of linear and log scales to determine whether our ability to notice differences in exponentially increasing trends is impacted by the choice of scale. 
The results indicated that when there was a large difference in curvature between the target plot and null plots and the target plot had more curvature than the null plots, the choice of scale had no impact and participants accurately differentiated between the two curves on both the linear and log scale. 
However, displaying exponentially increasing data on a log scale improved the accuracy of differentiating between models with slight curvature differences or large curvature differences when the target plot had less curvature than the null plots.
An exception occurred when identifying a plot with curvature embedded in surrounding plots closely relating to a linear trend, indicating that it is easy to identify a curve in a group of lines but much harder to identify a line in a group of curves.
The use of visual inference to identify these guidelines suggests that there are \emph{perceptual} advantages to log scales when differences are subtle. 
What remains to be seen is whether there are cognitive disadvantages to log scales: do log scales make it harder to make use of graphical information?