index.Rmd

---
title: "Bach's C-major prelude: a computational study"
author: "Noah Jaffe"
output: 
  flexdashboard::flex_dashboard:
    storyboard: true
    theme: sandstone
    self_contained: false
    
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(tidyverse)
library(remotes)
library(spotifyr)
library(ggplot2)
library(GGally)
library(grid)
library(gridExtra)
library(plotly)
library(compmus)
spotifyr::get_spotify_access_token()

circshift <- function(v, n) {
  if (n == 0) v else c(tail(v, n), head(v, -n))
}

bach_prelude_playlist_features = get_playlist_audio_features("", "4yNYY3xmNhPTDrfFc0qG9b")
bach_prelude_playlist_features = bach_prelude_playlist_features %>%
  mutate(track.duration_min = track.duration_ms/1000/60)
# Unwrap Tempo
tempo_threshold = 95;
time_threshold = 1.5;
bach_prelude_playlist_features = bach_prelude_playlist_features %>% 
  mutate(corrected_tempo = case_when(tempo>tempo_threshold & track.duration_min > time_threshold ~ tempo * 0.5, TRUE ~ tempo))

# Label section
tempo_duration_labels = c('', '', '', '', '', 
                         'Harp', '', '', 'Axel Gillison', '',
                         '', '', '', '',
                          '', 'Jazz Band', '', 'Schiff w/ Fugue', '',
                          '', '', '', '', '',
                         '', '', '', '')

bach_harpsichord_labels = c('', '', '', '', '', 
                         '', '', '', '', 'Harpsichord - Pinnock',
                         '', '', '', '',
                          '', '', '', '', '',
                          '', '', '', '', '',
                         'Harpsichord - Beauséjour', 'Massage - Vocals', '', '')
bach_harpsichord_factor = c('Piano', 'Piano', 'Piano', 'Piano', 'Piano', 
                         'Harp', 'Piano', 'Piano', 'Piano', 'Harpsichord',
                         'Piano', 'Piano', 'Piano', 'Piano',
                          'Piano', 'Piano + Jazz Band', 'Piano', 'Piano', 'Piano',
                          'Piano', 'Piano', 'Piano', 'Piano', 'Piano',
                         'Harpsichord', 'Piano + Vocals', 'Piano', 'Piano')


get_track_artist = function(track_index) {     bach_prelude_playlist_features[['track.album.artists']][[track_index]][['name']][2]
}

```

### **About the corpus** <br>  A look at Bach's C-major Prelude (BWV 846)

This playlist contains tracks of the same piece of music: The C-major prelude (BWV 846) from Bach's [Well-Tempered Clavier](https://en.wikipedia.org/wiki/The_Well-Tempered_Clavier), Book 1. 

The Well-Tempered Clavier is one of Bach's most famous keyboard works consisting of two books, each containing 24 matching preludes and fugues. [The first prelude (BWV 846)](https://en.wikipedia.org/wiki/Prelude_and_Fugue_in_C_major,_BWV_846) is in C-major and is the work studied here. Following the prelude, is a C-major fugue, followed by a C-minor prelude and fugue; the book contains one prelude and fugue for each key signature. 

<iframe style="border-radius:12px" src="https://open.spotify.com/embed/playlist/4yNYY3xmNhPTDrfFc0qG9b?utm_source=generator" width="100%" height="380" frameBorder="0" allowfullscreen="" allow="autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture"></iframe>

***

This playlist contains 29 tracks. 26 of the tracks are of musicians playing exactly the same piece of sheet music (ie: they are trying to play exactly the same notes).  

The **exceptions** include:  
1. The [Bach Eternal recording-labeled "Harp"](https://open.spotify.com/track/7sEU5ZN38FiKZ34zW1FPXt?si=37d325f811c14e95), in which the artist repeated the entire piece, but up an octave.  
2. The [recording by Sir Andras Schiff](https://open.spotify.com/track/2XxC430QMotGdympDP1aBo?si=2f6d288cf1e14acb) which includes the fugue performed after the prelude  
3. The [Jazz Arrangement by Jacques Loussier](https://open.spotify.com/track/2RQVtPukyUHJp1TFE1R66J?si=9824387780614219), which is labeled "Jazz Arrangement"  

This turned out to be a wonderful introduction to data science, R, the tidyverse, the Spotify API, and data science in general. I'll walk you through my amusing findings.

Musicological and research questions:  
1. What are the noticeable differences between famous recordings and unpopular recordings? What about between each other?  
2. Are there features in the music or in the performances that are more recognizable as MIR features than to human ears?  
3. When looking at the top 30 recordings of Bach's Well-Tempered Clavier, where are these recordings from? Are some more popular than others?  What makes a popular recording popular?  


### **Track-level summary**<br>Tempo and Instrument Analysis
``` {r track_summary, echo=FALSE}
wtc = bach_prelude_playlist_features %>% 
  mutate(instrument = bach_harpsichord_factor)
  # Unwrap Tempo
  tempo_threshold = 95;
  time_threshold = 1.5;
  wtc = wtc %>% 
  mutate(tempo = corrected_tempo) %>%
    slice(1:30) %>%
    add_audio_analysis()

  corpus_plot = wtc %>%
  mutate(
    sections =
      map(
        sections,                                    # sections or segments
        summarise_at,
        vars(tempo, loudness, duration),             # features of interest
        list(section_mean = mean, section_sd = sd)   # aggregation functions
      )
  ) %>%
  unnest(sections) %>%
  ggplot(
    aes(
      x = tempo,
      y = tempo_section_sd,
      colour = instrument,
      alpha = loudness
    )
  ) +
  geom_point(aes(size = duration / 60)) +
  geom_rug() +
  theme_minimal() +
  ylim(0, 5) +
  labs(
    x = "Mean Tempo (bpm)",
    y = "SD Tempo",
    colour = "Instrument",
    size = "Duration (min)",
    alpha = "Volume (dBFS)"
  )
corpus_plot
# ggplotly(corpus_plot)  # won't work because of missing data :-(
```

***

This plot shows a multi-dimensional analysis of many track-level features of this playlist. It's a wonderful way to get acquainted to the corpus. It shows that most of the works are approximately the same tempo, duration, loudness, and instrument.  

The purple point is suspicious! This is the recording that has the "vocals" added on top. The vocals sound a bit fake; the standard deviation of the tempo is so low that it seems like the entire track might be synthesized! The jazz band has the largest tempo variation throughout the track.

### **Tempo Overview**<br>Tempi picked by recording artists

```{r tempo, echo=FALSE, fig.show="hold", out.width="50%"}

# Duration Histogram
hist1 = ggplot(bach_prelude_playlist_features, aes(
  x=track.duration_min, 
)) + geom_histogram(bins=12) + xlab("Track Duration (min)") + scale_y_continuous(breaks=c(2,4,6,8)) + ggtitle("Distribution: Track Duration")
hist1

original_tempi = bach_prelude_playlist_features %>% 
  transmute(tempo=tempo, label="Original", duration=track.duration_min)
corrected_tempi = bach_prelude_playlist_features %>% 
  transmute(tempo=corrected_tempo, label="Corrected", duration=track.duration_min)
combined_tempi = rbind(original_tempi, corrected_tempi)

hist2 = ggplot(combined_tempi, aes(
  x=tempo, fill=label
)) + geom_histogram(bins=12, alpha=0.7) + xlab("Tempo") + 
  scale_fill_manual(values = c("Original"="black", "Corrected"="#94C54B")) + 
  theme(legend.title = element_blank()) + 
  scale_x_continuous(breaks= c(40, 60, 80, 100, 120, 140, 160)) + 
  scale_y_continuous(breaks=c(2,4,6,8,10,12,14,16))  + 
  ggtitle("Distribution: Track Tempo") 

hist2
```
<br>

```{r tempo_time, echo=FALSE}

tempo_v_duration = ggplot(corrected_tempi, aes(
  x=duration, y=tempo,
)) + geom_point() + xlab("Track Duration (min)") + ylab("Corrected Tempo (bpm)") +
  geom_text(
    label=tempo_duration_labels,
    nudge_x=.15, nudge_y=1.7
  ) + ggtitle("Corrected Tempo vs Duration") +
    xlim(1, 6.5) + ylim(50, 110)
ggplotly(tempo_v_duration, height=600)

```

***

Comparing raw track feature extractions from Spotify yielded the following histograms for duration and tempo. This data includes the entire corpus. 

Given that most of the musicians are playing the exact same piece of music, there are several expectations. First, the track duration should vary inversely and linearly with tempo. There are a fixed number of notes, and when they are played through more quickly, then the track duration should be shorter. But the distributions (using the same number of bins) look different for Track Duration and Tempo. Duration appears gaussian, with three explainable outliers. Tempo however, is a bimodal mess.

A quick survey of the corpus shows that the harp and one other recording (less than 90 seconds duration) are clearly the fastest performances. Using this information, I presume that any tempo faster than the harp (bpm > 95) is actually the eighth-note tempo, if its duration is longer than 90 seconds. So I divide by two to "unwrap" the tempo:

Re-evaluation shows a much more convincing distribution for tempo! 

Now that the tempo for the tracks is more-or-less calculated correctly and looking proper, the recording labeled Axel Gillison is quite interesting! Listen yields a few interesting observations. Firstly, it is note-per-note a performance of the original piece, except for the fact that it is extraordinarily slow! Also, it's in a lower key!

It appears that the creator of this Axel Gillison recording have done a few tricks! First of all, the Axel Gillison "wrapped" tempo is still twice the true tempo! The correct tempo is likely 32bpm. This would put our recording in line with our Tempo vs Duration curve. If we assume that this recording was produced using a sampling rate change (think of slowing down a record player) to change the tempo, we could correct the tempo back to C-major. The frequency ratio between C and A is 19:16—[a minor third](https://en.wikipedia.org/wiki/Minor_third). If we change the sampling rate by this ratio, we would end up with a tempo of 38bpm. This is still roughly half of our median tempo of 68bpm. So, it would appear that some other kind of fancy resampling has been done to yield such a slow tempo.

### **Tempo Consistency**<br>How steady is a performer?
``` {r tempograms, out.width="100%", fig.height=1.6, echo=FALSE}
fake_bach = get_tidy_audio_analysis("1TJSjNBt6oIHV8QdgdgrBT")
fake_bach_plot = fake_bach %>% tempogram(window_size = 8, hop_size = 1, cyclic = FALSE) %>%
  ggplot(aes(x = time, y = bpm, fill = power)) +
  geom_raster() +
  scale_fill_viridis_c(guide = "none") +
  labs(x = "Time (s)", y = "Tempo (BPM)", title = "Sounds for Massage") +
  theme_classic()

gould = get_tidy_audio_analysis("0eKd2VfI5EzrlgVH202x1i")
gould_plot = gould %>% tempogram(window_size = 8, hop_size = 1, cyclic = FALSE) %>%
  ggplot(aes(x = time, y = bpm, fill = power)) +
  geom_raster() +
  scale_fill_viridis_c(guide = "none") +
  labs(x = "Time (s)", y = "Tempo (BPM)", title = "Glenn Gould") +
  theme_classic()

jazz_bach = get_tidy_audio_analysis("2RQVtPukyUHJp1TFE1R66J")
jazz_plot = jazz_bach %>% tempogram(window_size = 8, hop_size = 1, cyclic = FALSE) %>%
  ggplot(aes(x = time, y = bpm, fill = power)) +
  geom_raster() +
  scale_fill_viridis_c(guide = "none") +
  labs(x = "Time (s)", y = "Tempo (BPM)", title = "Jacques Loussier") +
  theme_classic()
#grid.arrange(fake_bach_plot, gould_plot, jazz_plot, ncol = 1, nrow = 3)
fake_bach_plot
gould_plot
jazz_plot

```

***

These three "tempograms" show the frequency of novel events events in three recordings from the corpus. These tempogram are clearly showing the sixteenth-note tempo. Since the C-major prelude consists of only sixteenth notes, this makes sense that we see a very clean tempogram for the first two. 
First, the sounds for massage seems to confirm suspicions that this is computer generated. The tempo is completely consistent across the entire song. Glenn Gould, despite being one of the most consistent Bach performers of all time, has a noticeable tempo dip around 80 seconds.  
Finally, the Jazz recording by Jacques Loussier is most interesting. In this recording, we see a clear and abrupt tempo shift around the two-minute mark when the jazz gets unleashed. Towards the end when the band returns to the original Bach music, we can see that the tempo is not the original one, even though it feels that way to us as lsiteners. Amusingly, this is a tempo-analogue to sonata form where the original theme returns, but in a different key, here, it returns, in a different tempo. 

###  **Harmonic novelty**<br>Pitch and Timbre Self-Similarity
``` {r self_similarity, echo=FALSE}
gould =   
  get_tidy_audio_analysis("0eKd2VfI5EzrlgVH202x1i") %>% 
  compmus_align(sections, segments) %>%                     # Change `bars`
  select(sections) %>%                                      #   in all three
  unnest(sections) %>%                                      #   of these lines.
  mutate(
    pitches =
      map(segments,
          compmus_summarise, pitches,
          method = "rms", norm = "chebyshev"              # Change summary & norm.
      )
  ) %>%
  mutate(
    timbre =
      map(segments,
          compmus_summarise, timbre,
          method = "rms", norm = "chebyshev"              # Change summary & norm.
      )
  )

gould_pitch_ssm = gould %>%
  compmus_self_similarity(pitches, "cosine") %>% 
  ggplot(
    aes(
      x = xstart + xduration / 2,
      width = xduration,
      y = ystart + yduration / 2,
      height = yduration,
      fill = d
    )
  ) +
  geom_tile() +
  coord_fixed() +
  scale_fill_viridis_c(guide = "none") +
  theme_classic() +
  labs(x = "", y = "", title="Pitch")

gould_timbre_ssm = gould %>%
  compmus_self_similarity(timbre, "cosine") %>% 
  ggplot(
    aes(
      x = xstart + xduration / 2,
      width = xduration,
      y = ystart + yduration / 2,
      height = yduration,
      fill = d
    )
  ) +
  geom_tile() +
  coord_fixed() +
  scale_fill_viridis_c(guide = "none") +
  theme_classic() +
  labs(x = "", y = "", title="Timbre")

gould =   
  get_tidy_audio_analysis("0eKd2VfI5EzrlgVH202x1i") %>% 
  compmus_align(bars, segments) %>%                     # Change `bars`
  select(bars) %>%                                      #   in all three
  unnest(bars) %>%                                      #   of these lines.
  mutate(
    pitches =
      map(segments,
          compmus_summarise, pitches,
          method = "rms", norm = "chebyshev"              # Change summary & norm.
      )
  ) %>%
  mutate(
    timbre =
      map(segments,
          compmus_summarise, timbre,
          method = "rms", norm = "chebyshev"              # Change summary & norm.
      )
  )

gould_pitch_ssm_fine = gould %>%
  compmus_self_similarity(pitches, "cosine") %>% 
  ggplot(
    aes(
      x = xstart + xduration / 2,
      width = xduration,
      y = ystart + yduration / 2,
      height = yduration,
      fill = d
    )
  ) +
  geom_tile() +
  coord_fixed() +
  scale_fill_viridis_c(guide = "none") +
  theme_classic() +
  labs(x = "", y = "", title="Pitch")

gould_timbre_ssm_fine = gould %>%
  compmus_self_similarity(timbre, "cosine") %>% 
  ggplot(
    aes(
      x = xstart + xduration / 2,
      width = xduration,
      y = ystart + yduration / 2,
      height = yduration,
      fill = d
    )
  ) +
  geom_tile() +
  coord_fixed() +
  scale_fill_viridis_c(guide = "none") +
  theme_classic() +
  labs(x = "", y = "", title="Timbre")

grid.arrange(gould_pitch_ssm, gould_timbre_ssm,
             gould_pitch_ssm_fine, gould_timbre_ssm_fine,
             ncol = 2, nrow = 2,
              top=textGrob('Glenn Gould Self Similarity Matricies'))

```

***

Here are four self-similarity matricies which compare a the characteristics of a moment in a musical piece with another moment in the same piece. A dark segment indicates that the two moments are very similar while bright colors indicate large contrast. The difference between the two matricies of the same type is the window size. 

A bright line runs approximately halfway through the piece. This indicates that the pitch content of this spot is quite different from that of the rest of the piece. 

And indeed, Spotify thinks that this moment is the most interesting and defining part of the piece. It's what is played when the track is selected in a preview.

Meanwhile, the timbre self-similarity matrix seems to be most sensitive to notes below A3. This highlights a large difference when the bass line descends and becomes well pronounced pronounced. It's worth noting that while these pieces only contain solo piano, there are large timbre differences apparent, which mostly manifests itself in the range of the piano being played.  

A look at the pitch self-similarity matricies don't reveal much information about musical form, other than there being a harmonically distant section in the middle. 


###  **Timbre and musical form** <br>Cepstrogram: A surprising outlier

``` {r jazz_timbre, echo=FALSE}
jazzy = 
  get_tidy_audio_analysis("2RQVtPukyUHJp1TFE1R66J") %>% 
  compmus_align(sections, segments) %>%                     # Change `bars`
  select(sections) %>%                                      #   in all three
  unnest(sections) %>%                                      #   of these lines.
  mutate(
    pitches =
      map(segments,
          compmus_summarise, pitches,
          method = "rms", norm = "chebyshev"              # Change summary & norm.
      )
  ) %>%
  mutate(
    timbre =
      map(segments,
          compmus_summarise, timbre,
          method = "rms", norm = "chebyshev"              # Change summary & norm.
      )
  )

jazzy %>%
  compmus_gather_timbre() %>%
  ggplot(
    aes(
      x = start + duration / 2,
      width = duration,
      y = basis,
      fill = value
    )
  ) +
  geom_tile() +
  labs(x = "Time (s)", y = NULL, fill = "Magnitude", title = "Cepstrogram") +
  scale_fill_viridis_c() +                              
  theme_classic()

```

*** 

Here is a cepstrogram showing different timbre-elements of [this recording by Jacques Loussier](https://open.spotify.com/track/2RQVtPukyUHJp1TFE1R66J). A cepstrogram shows the energy content contained in various timbre characteristics; a bright line indicates the presence of that timbre element.
There is an abrupt timbre change around 120s, from what is mostly just c02, to a much more complicated timbre sound, only to have it return to the original sound at the very end. A listening reveals that a drumset enters at that point! It exits just before the end, allowing the traditional ending.


### **Comparing Performances**<br>Chromagrams across performances
``` {r chromas, include=FALSE}
tracks = bach_prelude_playlist_features[['track.id']]

make_chromagram = function(track_id, title) {
    track_analysis <-
    get_tidy_audio_analysis(track_id) %>%
    select(segments) %>%
    unnest(segments) %>%
    select(start, duration, pitches)
  
  computed_chroma = track_analysis %>%
    mutate(pitches = map(pitches, compmus_normalise, "euclidean")) %>%
    compmus_gather_chroma()
    
  ggplot(computed_chroma,
      aes(
        x = start + duration / 2,
        width = duration,
        y = pitch_class,
        fill = value
      )
    ) +
    geom_tile() +
    labs(x = "Time (s)", y = NULL, fill = "Magnitude", title = title) +
    theme_minimal() + theme(legend.position = "none") +
    scale_fill_viridis_c()
}

```

```{r pw, out.width="100%", fig.height=2, echo = F}
make_chromagram(tracks[1], get_track_artist(1))
make_chromagram(tracks[2], get_track_artist(2))
make_chromagram(tracks[3], get_track_artist(3))
make_chromagram(tracks[4], get_track_artist(4))
```

***

Here are four chromagrams of different recordings of this piece. The chromagram is indicating the presence of various pitches across time. Visually, they have the same signature despite different tempi and x axes. By looking closely, dark space is in between the notes on Gould's stoccato playing when compared to Gulda. Pre-track silence is visible on Kempf's recording. 


### **Comparing Perfomrances**<br>Dynamic Time Warping:<br>Gould vs Gould vs Gulda

```{r echo = F}
  gould <-
    get_tidy_audio_analysis(tracks[2]) %>%
    select(segments) %>%
    unnest(segments) %>%
    select(start, duration, pitches)

    gulda <-
    get_tidy_audio_analysis(tracks[3]) %>%
    select(segments) %>%
    unnest(segments) %>%
    select(start, duration, pitches)
  
distance = compmus_long_distance(
  gould %>% mutate(pitches = map(pitches, compmus_normalise, "chebyshev")),
  gulda %>% mutate(pitches = map(pitches, compmus_normalise, "chebyshev")),
  feature = pitches,
  method = "euclidean"
) 

distance2 = compmus_long_distance(
  gould %>% mutate(pitches = map(pitches, compmus_normalise, "chebyshev")),
  gould %>% mutate(pitches = map(pitches, compmus_normalise, "chebyshev")),
  feature = pitches,
  method = "euclidean"
) 
  gg1 = ggplot(distance,
    aes(
      x = xstart + xduration / 2,
      width = xduration,
      y = ystart + yduration / 2,
      height = yduration,
      fill = d
    )
  ) +
  geom_tile() +
  coord_equal() +
  labs(x = "Glenn Gould", y = "Friedrich Gulda") +
  theme_minimal() +
  scale_fill_viridis_c(guide = NULL)

  gg2 = ggplot(distance2,
    aes(
      x = xstart + xduration / 2,
      width = xduration,
      y = ystart + yduration / 2,
      height = yduration,
      fill = d
    )
  ) +
  geom_tile() +
  coord_equal() +
  labs(x = "Glenn Gould", y = "Glenn Gould") +
  theme_minimal() +
  scale_fill_viridis_c(guide = NULL)

transpose_pitch = function(pitch_list, number_of_semitones) { # n = 1: C -> C#
  names = names(pitch_list)
  new_list = setNames(circshift(unname(pitch_list), number_of_semitones), names)
  new_list
}

transpose_pitches = function(df, n) {  # n = 1 is C -> C#
  df %>% dplyr::mutate(pitches = purrr::map2(pitches, n, transpose_pitch))  
}

luc = 
  get_tidy_audio_analysis("13zyfbzt2PiYo2RjjJMCsb") %>%
  select(segments) %>%
  unnest(segments) %>%
  select(start, duration, pitches)

luc_t = transpose_pitches(luc, 1)

gould_luc_dist = compmus_long_distance(
  gould %>% mutate(pitches = map(pitches, compmus_normalise, "chebyshev")),
  luc %>% mutate(pitches = map(pitches, compmus_normalise, "chebyshev")),
  feature = pitches,
  method = "euclidean"
) 

gould_luc_t_dist = compmus_long_distance(
  gould %>% mutate(pitches = map(pitches, compmus_normalise, "chebyshev")),
  luc_t %>% mutate(pitches = map(pitches, compmus_normalise, "chebyshev")),
  feature = pitches,
  method = "euclidean"
) 

gg3 = ggplot(gould_luc_dist,
    aes(
      x = xstart + xduration / 2,
      width = xduration,
      y = ystart + yduration / 2,
      height = yduration,
      fill = d
    )
  ) +
  geom_tile() +
  coord_equal() +
  labs(x = "Glenn Gould", y = "Luc Beausejour (original)") +
  theme_minimal() +
  scale_fill_viridis_c(guide = NULL)
  
  gg4 = ggplot(gould_luc_t_dist,
    aes(
      x = xstart + xduration / 2,
      width = xduration,
      y = ystart + yduration / 2,
      height = yduration,
      fill = d
    )
  ) +
  geom_tile() +
  coord_equal() +
  labs(x = "Glenn Gould", y = "Luc Beausejour (transposed)") +
  theme_minimal() +
  scale_fill_viridis_c(guide = NULL)
  
    grid.arrange(gg1, gg2, gg3, gg4,
               ncol = 2, nrow = 2, 
               top=textGrob('Pitch Distance Matricies'))
```

***

Here are pitch distance matricies. A dark spot indicates that the pitch content of the two pieces under comparison is very similar, while brighter color means that the spots are more contrasting. 

The left plot compares the harmonic content between Glenn Gould and Friedrich Gulda.  
The right plot is a self-similarity matrix comparing Glenn Gould's performance to his own.

Despite two different tempi for the performance, the two plots are nearly identical. Gulda has a long time after the final note and both performers keep a very steady tempo and do not deviate, as seen by a perfectly straight diagonal line from the origin. 

I wished to compare Glenn Gould's performance against Luc Beausejour's harpsichord playing, which resulted in the bottom-left pitch distance vector. Not helpful, even though they're playing the same piece! By forcing a transposition of all pitches up by one semitone (eg: B becomes C), the bottom right dynamic time warping (DTW) matrix is produced. A sharp diagonal line is now observed, showing a 1:1 relationship between both recordings. 

###  **A tale of two harpsichords**<br>Tuning and chromagrams 
``` {r chroma, echo=FALSE}
gould = 
  get_tidy_audio_analysis("0eKd2VfI5EzrlgVH202x1i") %>%
  select(segments) %>%
  unnest(segments) %>%
  select(start, duration, pitches)
  
pinnock =
  get_tidy_audio_analysis("3pbZ0GbDAl8ehNiW58wOLO") %>%
  select(segments) %>%
  unnest(segments) %>%
  select(start, duration, pitches)

luc = 
  get_tidy_audio_analysis("13zyfbzt2PiYo2RjjJMCsb") %>%
  select(segments) %>%
  unnest(segments) %>%
  select(start, duration, pitches)

gould_chroma = gould %>%
  mutate(pitches = map(pitches, compmus_normalise, "euclidean")) %>%
  compmus_gather_chroma() %>% 
  ggplot(
    aes(
      x = start + duration / 2,
      width = duration,
      y = pitch_class,
      fill = value
    )
  ) +
  geom_tile() +
  labs(x = "Time (s)", y = NULL, fill = "Magnitude") +
  theme_minimal() +
  scale_fill_viridis_c() + ggtitle("Glenn Gould") + theme(legend.position = "none")

pinnock_chroma = pinnock %>%
  mutate(pitches = map(pitches, compmus_normalise, "euclidean")) %>%
  compmus_gather_chroma() %>% 
  ggplot(
    aes(
      x = start + duration / 2,
      width = duration,
      y = pitch_class,
      fill = value
    )
  ) +
  geom_tile() +
  labs(x = "Time (s)", y = NULL, fill = "Magnitude") +
  theme_minimal() +
  scale_fill_viridis_c() + ggtitle("Trevor Pinnock") + theme(legend.position = "none")

 luc_chroma = luc %>%
  mutate(pitches = map(pitches, compmus_normalise, "euclidean")) %>%
  compmus_gather_chroma() %>% 
  ggplot(
    aes(
      x = start + duration / 2,
      width = duration,
      y = pitch_class,
      fill = value
    )
  ) +
  geom_tile() +
  labs(x = "Time (s)", y = NULL, fill = "Magnitude") +
  theme_minimal() +
  scale_fill_viridis_c() + ggtitle("Luc Beausejour") + theme(legend.position = "none")
grid.arrange(gould_chroma, pinnock_chroma, luc_chroma,ncol = 3,
             nrow = 1, top=textGrob('Chromagrams'))
 
```


***


Here are three chromagrams showing pitch content of three recordings in broad strokes. 
Each instrument/recording has a different tuning. 

1. Glenn Gould: Piano reference, Glenn Gould is playing a Steinway piano tuned with A ~ 440hz.

2. Trevor Pinnock: Harpsichord. This luxurious 2020 recording [was created using a non quite equal-temperement tuning](http://www.biberfan.org/reviews/2020/4/25/the-well-tempered-clavier-book-1-trevor-pinnock) with a pitch for A that is clearly lower than 440Hz. Here in the chromagram, Spotify's pitch detection is unhappy with it. You can see that the pitch class of F seems to contain the bins of Gould's E and F combined. 

3. Luc Beausejour: Harpsichord. This 2007 recording uses a tuning that seems approximately one semi-tone flat (A ~ 415Hz). This is common for baroque recordings. Here, the Spotify chromagram looks quite simlar to Glenn Gould's, except that each row is shifted, indicating that Gould's C is Beausejour's B.


###  **A tale of two harpsichords**<br>Timbre
``` {r timbre, out.width="100%", fig.height=2, echo=FALSE}

 

  # Pinnock 3pbZ0GbDAl8ehNiW58wOLO

make_ceptrogram = function(track_id, plot_title) {
  track_analysis = 
    get_tidy_audio_analysis(track_id) %>% 
    compmus_align(sections, segments) %>%                     # Change `bars`
    select(sections) %>%                                      #   in all three
    unnest(sections) %>%                                      #   of these lines.
    mutate(
      pitches =
        map(segments,
            compmus_summarise, pitches,
            method = "rms", norm = "chebyshev"              # Change summary & norm.
        )
    ) %>%
    mutate(
      timbre =
        map(segments,
            compmus_summarise, timbre,
            method = "rms", norm = "chebyshev"              # Change summary & norm.
        )
    )
  
  return_cept = track_analysis %>%
    compmus_gather_timbre() %>%
    ggplot(
      aes(
        x = start + duration / 2,
        width = duration,
        y = basis,
        fill = value
      )
    ) +
    geom_tile() +
    labs(x = "Time (s)", y = NULL, fill = "Magnitude", title = plot_title) +
    scale_fill_viridis_c() +                              
    theme_classic()
  return_cept
}

gould_cept = make_ceptrogram("0eKd2VfI5EzrlgVH202x1i", "Glenn Gould - Piano")
schiff_cept = make_ceptrogram("2XxC430QMotGdympDP1aBo", "Andras Schiff - Piano")
pinnock_cept = make_ceptrogram("3pbZ0GbDAl8ehNiW58wOLO", "Trevor Pinnock - Harpsichord")
luc_cept = make_ceptrogram("13zyfbzt2PiYo2RjjJMCsb", "Luc Beausejour - Harpsichord")
gould_cept
schiff_cept
pinnock_cept
luc_cept
```


***

Here are four cepstrograms comparing timbre components comparing a piano and harpsichord recordings. The findings are quite surprising. Firstly, it seems Bach played on piano has a signature of being mostly component c02. The harpsichords look completely different, and are even significantly different from each other! The pianoforte is a dynamic instrument whose hammers compress when played at different loudnesses. This literal compression generates a different timbre and would be interesting to see how this compares to that of a

###  **A tale of two harpsichords** <br>  A look at Spotify's feature judgements of *acousticness* and *instrumentalness* for harpischord and piano

```{r download_features, include=FALSE}
harpsichord_labeled_playlist = bach_prelude_playlist_features %>%
  mutate(Label = bach_harpsichord_labels,
         instrument = factor(bach_harpsichord_factor))
```

```{r acousticness, echo=FALSE}
color_values = c("Piano"="black", "Harpsichord"="#D55E00",
                 "Piano + Vocals"="brown", "Piano + Jazz Band"="pink")
acoustic_v_duration = ggplot(harpsichord_labeled_playlist, aes(
  x=track.duration_min, y=acousticness, color=instrument
)) + geom_point() + xlab("Track Duration (min)") + ylab("Acousticness") +
  geom_text(
    label=bach_harpsichord_labels, nudge_x=0, nudge_y=-0.02
  ) + ggtitle("Acousticness vs Track Duration") + ylim(0, 1) + scale_color_manual(values=color_values)
#ggplotly(acoustic_v_duration) Disable Legacy Plot
```

```{r instrumentalness, echo=FALSE}
instrumental_v_duration = ggplot(harpsichord_labeled_playlist, aes(
  x=track.duration_min, y=instrumentalness, color=instrument
)) + geom_point() + xlab("Track Duration (min)") + ylab("Instrumentalness") +
  geom_text(
    label=bach_harpsichord_labels, nudge_x=0, nudge_y=-0.02
  ) + ggtitle("Instrumentalness vs Track Duration") + ylim(0, 1) + 
  scale_color_manual(values=color_values)
#ggplotly(instrumental_v_duration) Disable Legacy Plot
```

```{r parallel_coords, echo=FALSE, fig.height=2}
para_coord_data = harpsichord_labeled_playlist %>% 
  select(instrument, instrumentalness, acousticness, Label)
para_coord_plot = ggparcoord(para_coord_data, 
  columns=2:3, groupColumn=1, scale="globalminmax",
    showPoints = TRUE,
  title = "Spotify Instrumentalness and Acousticness"
)  + xlab("")
ggplotly(para_coord_plot)
```

***

Two of the tracks are recordings of a harpsichord; they are by [Trevor Pinnock](https://open.spotify.com/track/3pbZ0GbDAl8ehNiW58wOLO?si=a10bdc71980b402b) and [Luc Beauséjour](https://open.spotify.com/track/13zyfbzt2PiYo2RjjJMCsb?si=882dc21b84d049be).

To my ears, these performances are quite similar. The tunings are slightly different, as are the performance styles. But in general, they are quite similar. The microphone and audio mixing techniques of the Pinnock recording seem more luxurious to my ears, but I cannot think of any other dissimilarities.

A track on the album [Sounds for Massages](https://open.spotify.com/track/1TJSjNBt6oIHV8QdgdgrBT?si=1b43d60d1f024c00) which contains the piece performed on piano, with a synthesized vocal track of "aah" placed on top. 

The [Spotify API definition](https://developer.spotify.com/documentation/web-api/reference/#/operations/get-audio-features) of *acousticness*:
A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.

*Instrumentalness* predicts whether a track contains no vocals. "Ooh" and "aah" sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly "vocal". The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.

<br>
**Questions**  
Why does the SpotifyAPI think that Trevor Pinnock's recording has vocals?! A score of 0.1 means that the API is quite certain that the there is vocal content. It's clear that Harpsichords have a rich harmonic content. Is it because of fancy microphone techniques that the low strings of his harpsichord resonate in a way different than that of Beauséjour's? Moreover, one would think that deliberately inserting an "aah" sound on top of the recording would tank the score, if that is what the algorithm is looking for! But it's clear that the massage recording is within the distribution of piano-only recordings.

The acousticness metric is quite more vague in the Spotify API, but it seems to be more related to spectral content. Distortion of an electric guitar is caused my compression. This distrotion is what gives gives electric guitar its signature bright sound. It makes sense that harpsichord sounds more similar to electric guitar than the felt tips of a piano. 



### **Key Areas**<br>Jazz vs Baroque Jazz
``` {r keygram, out.width="100%", fig.height=3, echo=FALSE}

major_key <-
  c(6.35, 2.23, 3.48, 2.33, 4.38, 4.09, 2.52, 5.19, 2.39, 3.66, 2.29, 2.88)
minor_key <-
  c(6.33, 2.68, 3.52, 5.38, 2.60, 3.53, 2.54, 4.75, 3.98, 2.69, 3.34, 3.17)

key_templates <-
  tribble(
    ~name, ~template,
    "Gb:maj", circshift(major_key, 6),
    "Bb:min", circshift(minor_key, 10),
    "Db:maj", circshift(major_key, 1),
    "F:min", circshift(minor_key, 5),
    "Ab:maj", circshift(major_key, 8),
    "C:min", circshift(minor_key, 0),
    "Eb:maj", circshift(major_key, 3),
    "G:min", circshift(minor_key, 7),
    "Bb:maj", circshift(major_key, 10),
    "D:min", circshift(minor_key, 2),
    "F:maj", circshift(major_key, 5),
    "A:min", circshift(minor_key, 9),
    "C:maj", circshift(major_key, 0),
    "E:min", circshift(minor_key, 4),
    "G:maj", circshift(major_key, 7),
    "B:min", circshift(minor_key, 11),
    "D:maj", circshift(major_key, 2),
    "F#:min", circshift(minor_key, 6),
    "A:maj", circshift(major_key, 9),
    "C#:min", circshift(minor_key, 1),
    "E:maj", circshift(major_key, 4),
    "G#:min", circshift(minor_key, 8),
    "B:maj", circshift(major_key, 11),
    "D#:min", circshift(minor_key, 3)
  )

jacques <-
  get_tidy_audio_analysis("2RQVtPukyUHJp1TFE1R66J") %>%
  compmus_align(sections, segments) %>%
  select(sections) %>%
  unnest(sections) %>%
  mutate(
    pitches =
      map(segments,
          compmus_summarise, pitches,
          method = "mean", norm = "manhattan"
      )
  )

distance_data = jacques %>% 
  compmus_match_pitch_template(
    key_templates,         # Change to chord_templates if descired
    method = "euclidean",  # Try different distance metrics
    norm = "manhattan"     # Try different norms
  ) %>% mutate(d = log(1/d, 10))  # Inverse log to make bright a good match

  jacques_plot = ggplot(distance_data,
    aes(x = start + duration / 2, width = duration, y = name, fill = d)
  ) +
  geom_tile() +
  scale_fill_viridis_c(guide = "none") +
  theme_minimal() +
  labs(x = "Time (s)", y = "", title = "Jacques Loussier")
  
  gould <-
    get_tidy_audio_analysis("0eKd2VfI5EzrlgVH202x1i") %>%
    compmus_align(sections, segments) %>%
    select(sections) %>%
    unnest(sections) %>%
    mutate(
      pitches =
        map(segments,
            compmus_summarise, pitches,
            method = "mean", norm = "manhattan"
        )
    )
  
  distance_data1 = gould %>% 
    compmus_match_pitch_template(
      key_templates,         # Change to chord_templates if descired
      method = "euclidean",  # Try different distance metrics
      norm = "manhattan"     # Try different norms
    ) %>% mutate(d = log(1/d, 10))  # Inverse log to make bright a good match
  
  gould_key = ggplot(distance_data1,
              aes(x = start + duration / 2, width = duration, y = name, fill = d)
  ) +
    geom_tile() +
    scale_fill_viridis_c(guide = "none") +
    theme_minimal() +
    labs(x = "Time (s)", y = "", title = "Glenn Gould")
  jacques_plot
  gould_key
#  grid.arrange(jacques_plot, gould_key, ncol = 1, nrow = 2, 
#               top=textGrob('Keygrams: Jazz vs Baroque'))
```

***

Here, pitch-distance vectors are masked over sections of music to estimate key areas of pieces. Glenn Gould's recording (bottom) is representative of the original piece of music, while the top is Jacque Loussier's jazz rendition. Bright spots indicate a good match for the indicated key on the left.

The two recordings share the same key signature (pun intended) for the first bit of the piece, where they clearly modulate from C major to d minor, and back. But then the jazz recording gets more exciting. It seems that Jacques Loussier has leaned into this C-D major modulation and uses it again in the Jazz section. Much of the jazz piece is only loosely tied to the C-major key area. 


### **Popularity**<br>What makes a performance popular?
```{r popularity, echo=FALSE}
library(dendextend)

get_best_artist = function(artists){
  if (length(artists[['name']]) > 1) {
      artists[['name']][2] 
  }
    else {
     artists[['name']][1]
    }
}

get_yr = function(release_string){
  str_split(release_string, '-')[[1]][1]
}

meta_df = transmute(bach_prelude_playlist_features,
                    artist = map(track.artists, get_best_artist),
                    release_year = as.numeric(map(track.album.release_date, get_yr)),
                    corrected_tempo = corrected_tempo,
                    track.track_number = track.track_number, 
                    track.album.album_type = track.album.album_type)

extracted_features <- bach_prelude_playlist_features %>% 
  transmute(
    track.popularity = as.numeric(track.popularity, stringsAsFactors = FALSE)
         )
scaled_features = scale(extracted_features)
distances = dist(scaled_features)
hc <- hclust(distances, method = 'complete')
dend = as.dendrogram(hc)
labels(dend) = meta_df[['artist']]
cols <- c("#1b5e20", "#c8e6c9", "#81c784")
dend <- color_branches(dend, k = 3, col = cols)
plot(rev(dend), main="Popularity Distance") 

```


***

The question that everyone wants to know is "what makes a song popular!?", and the question that data scientists want to answer is "can we predict which songs will become popular?!"  
To this query, I suggest that there is no way to know which songs will become popular, but there are several ways to make sure your track will not become popular!

By doing a simple clustering by popularity, I've labeled tracks into three categories: Unpopular, Somewhat Popular, and More Popular. **Light colors represent unpopular, while the darkest are most popular.**

Doing other kinds of cluster analysis with all of the available track-level data yielded nothing of significance. These features included, tempo, liveness, instrumentalness, etc. It's probably obvious, but given that nearly every one of these recordings is of solo piano, it would be surprising if one of these metrics yielded a better performance.

### **Popularity Contest**<br>Timing
```{r tempo_release_year, echo=FALSE}

final_df = extracted_features %>% mutate(cluster = factor(cutree(hc, k=3), labels=c('Somewhat Popular','Unpopular', 'More Popular')), 
                                         artist = c(meta_df[['artist']]),
                                         album_year = c(meta_df[['release_year']]),
                                         tempo = c(meta_df[['corrected_tempo']]),
                                         track.track_number = c(meta_df[['track.track_number']]),
                                          album_type = c(meta_df[['track.album.album_type']]))
                                         
green_popularity_palette = c("Unpopular" = "#c8e6c9",
                             "Somewhat Popular" = "#81c784",
                             "More Popular" = "#1b5e20")

year_plot = ggplot(data=final_df, aes(x=album_year, y=tempo, color=cluster)) + 
  geom_point() + geom_text(label=final_df[['artist']], nudge_y=1) + 
  scale_color_manual(values = green_popularity_palette) +
  xlab('Year album released') + ylab('Tempo (bpm)')

ggplotly(year_plot)
```

***

In this cloud, the clusters from the previous tab are reused to show the relationships of tempo, album release year, and popularity. It seems that Gould and Schiff are popular despite being from the 70's and 80's, but most albums are newer and a bit faster. Surprising findings from this plot are that Lang Lang and Angela Hewitt are unpopular!


### **How to be unpopular**<br>Try being on a compilation album
```{r album_type, echo=FALSE}
fig <- final_df %>%
  plot_ly(
    x = ~album_type,
    y = ~track.popularity,
    split = ~album_type,
    type = 'violin',
    box = list(
      visible = T
    ),
    meanline = list(
      visible = T
    )
  ) 
fig <- fig %>%
  layout(
    xaxis = list(
      title = "Album Type"
    ),
    yaxis = list(
      title = "Popularity (higher is better)"
    )
  )
fig

```

***

This finding is surprising to me! In a world of "Bach for your baby", "classical piano study music" and the like, most classical musicians would sardonically expect that a random recording from a compilation would be more popular than Sir Andras Schiff or Glenn Gould. It turns out, that the opposite is true. Not a single one of the compilation recordings is popular. Perhaps this explains the Lang Lang  unpopularity! It is from a compilation album.


### **Popularity Contest**<br>C-Major Prelude vs the rest of the WTC Book 1
``` {r popularity_plots, echo=FALSE}
album_tracks = bach_prelude_playlist_features %>% 
  filter(track.album.album_type == "album")

get_best_artist = function(artists){
  if (length(artists[['name']]) > 1) {
    artists[['name']][[2]]
  }
  else {
    artists[['name']][[1]]
  }
}

# Adds all tracks from the associated album to the dataframe
add_nested_tracks_from_album = function(playlist_df) {
  get_adjacent_tracks = function(album_id){
    track_query_brief = get_album_tracks(album_id, 48)  
    # ^^ This doesn't return all the info we need, so use the IDs to query again
    # Requery, using IDs:
    album_df = paste(track_query_brief[['id']], collapse=",") %>% 
      get_tracks() %>% select(id, disc_number, track_number, popularity)
    
    afs = get_track_audio_features(paste(track_query_brief[['id']], collapse=','))
    #album_df = album_df %>% mutate(track_audio_features = map(id, get_track_audio_features))
    album_df = merge(album_df, afs, x.on="id", y.on="id")
    # Almost every album is 2 CDs with 24 tracks each, so create an index
    # This doesn't work with Schiff.
    album_df %>% mutate(track_index = (disc_number-1)*24 + track_number) 
  }
  playlist_df %>% mutate(album_tracks = map(track.album.id, get_adjacent_tracks))
}

# Down select artists
artists_to_compare = c( 'Glenn Gould', 'Keith Jarrett', 'Daniel Barenboim',
                        'Luc Beauséjour', 'Trevor Pinnock', 
                        'Wilhelm Kempff', 'Julia Cload')


playlist_with_artists = bach_prelude_playlist_features %>% 
  mutate(album_artist_label = as.character(map(track.artists, get_best_artist)))

filtered_playlist = playlist_with_artists %>% filter(album_artist_label %in% artists_to_compare )

filtered_nested = add_nested_tracks_from_album(filtered_playlist) 
big_df_simple = filtered_nested %>% select(album_tracks, album_artist_label)

unnested_f = unnest(big_df_simple, album_tracks) %>% 
  mutate(popularity = as.numeric(popularity), 
         artist_factor = as.factor(album_artist_label))

stacked_pop = ggplot(unnested_f, aes(x=track_index, y=popularity, color=album_artist_label)) +
  geom_line() + xlab('Track Index') + ylab('Popularity (0-100)') + labs(color="Artist")
ggplotly(stacked_pop)


```


***

In this analysis, the popularity of the track is compared to the tracks that follow it. In each of these cases, this is the first prelude, and is usually the first track on the album, with the highest popularity. Most artists have a C-major prelude that is much more popular than their respective album means. 
In general, as one would expect, the C-major prelude is the most popular track for almost every recording of the Well-Tempered Clavier.  

*Note*: Here, track index is calculated as: (disc_number - 1) * 24 + track_number. 

### **Loudness/Dynamics**<br>C-Major Prelude vs the rest of the WTC Book 1
``` {r loudness_plot, echo=FALSE}
stacked_loudness = ggplot(unnested_f, aes(x=track_index, y=loudness, color=album_artist_label)) +
  geom_line() + xlab('Track Index') + ylab('Loudness (dB)') + labs(color="Artist")
ggplotly(stacked_loudness)

```


***

Here, there are the laoudness for 9 of the given artists, stacked on top of each other. Toggle them off and on to investigate them against each other, or alone. 

Interesting findings:  
1. Glenn Gould seems to really like his major fugues! He plays them loud! Then the following minor prelude seems to always be quiet, creating this sawtooth loudness signature throughout his albums.
2. Trevor Pinnock seems to have just as big of a dynamic range as many pianists, despite being on harpsichord. This proves a key musicological point that the harpsichord can play louder and softer, just not in the same way as a pianoforte! Also, his major figue peaks line up with Glenn Gould's.  
3.  Julia Cload has the largest loudness dynamic range!  
4.  By looking at aligned peaks, some clear trends are visible that many artists interpret specific preludes or fugues similarly. Interestingly, the C-major prelude (track one!) seems to be very contested. Many artists start loud, playing other tracks softer, and vice versa. This data appears highly correlated, but not quite, with 'energy'.

### **Conclusions**<br>Food for thought
In this investigation of Bach's Well-Tempered Clavier through the Spotify Api, there are the following findings:  
1. In Spotify's world, a unique recording is not a unique recording. At first, I had several recording artist's recordings from compilation albums. This threw off popularity metrics, as these tracks are tallied separately from their original sources. As a result, these compilation albums have been removed, and have been replaced by their original album sources.  
2. Compilations (eg: classical boost!) are far less popular than original albums of a single musical work. This was a big shock. When searching for a specific track on Spotify, (eg: "Bwv 846"), Spotify will return a mishmash of recordings that are from compilation albums and from original artist albums. Many of these tracks will be the same exact audio recording.
3. Spotify chroma-based features (eg: key detection) work poorly for tunings where A is not 440hz. For tunings where A=415, features recover. It would be wonderful if Spotify's chroma detection algorithm applied tuning-detection to catch cases where pitches don't cleanly fall into an A=440hz.  
4. Spotify's feature extractions for liveness, acousticness, and instrumentalness do not classify harpsichords correctly all of the time.  
5. Timbre elements can be indicative of huge musical surprises! I am now in love with Jacques Loussier after having discovered his surprise jazz band.  
6. Synthesized music can be detected by looking at the tempo features. A surprisingly low standard-deviation, or lack of curviness to the tempogram indicates that it's fake. This music is unpopular and generally uninteresting when compared to real performances of the same work.  
7. Spotify's tempo calculations heavily bias towards 110-150bpm when there are repeated notes. This is unsurprising as tempo detection is very difficult.  
8. For this piece of music, the most popular recordings are near the mean of tempo, or slower. Faster tempos are not as popular. This could be evidence supporting for the claim that successful classical music that is not as technically demanding (in a fast way) favors a slower tempo than normal.  
9. Simple "loudness" measured in dB can be a surprisingly effective metric for considering interpretation decisions on a large scale of an album. Music nerds love to comb over Glenn Gould's recordings for hypermeter information and some of it is revealed here. This could be immensely helpful for guiding a performer's interpretation of this work.
10. This analysis has reopened a fondness in my heart for the harpsichord. While the instrument lacks the expressive capabilities of a pianoforte in a 19th and 20th century context, this work was written with it in mind. 
   
Points 1 and 2 could be evidence against the claim that "everyone only listens to classical music as background/study music".