Project_May4th

repository for Piano 2.0 MVP v1

wishlist features as of 29.04.2024

• dynamic volume
• looping of compose mode
• bar cursor
• metronome before any play

latest features as of 28.04.2024

• harmony colorization
• improved harmony checking
• improved metronome
• updated music sheets

as of 27.04.2024 synchronization issues addressed

• condition 2 implemented
• dynamic harmony implemented

April Updates

• Implement 4B-4B-4P-4P protocol
• Test Mode for Lessons 1-3
• Compose Mode for Lessons 4-5
• Expand sheets add harmony Fclef
• update music info linked

All lessons received from Sandi. Newer tasks as of 12.01.2024

• Change guidances toggles to checkboxes
• Add guidance features: metronome and rhythm
• music sheets work!
• Music sheets as display and condition 2

All features working as of 30.11.2023 Just pending Lesson 3-5 from Sandi ALREADY ADDED THEM!

Post Study Release Targets

• Lesson 1: Play Swing Through Modes
• Lesson 2: Learn Sequences Through Repetition
• Lesson 3: Learn Motifs Through Phrasing
• Lesson 4: Learn Variations
• Lesson 5: Ques-Ans

open issues and targets

• MAJOR cleanup
• new layout
• deploy webserver for receiving inputs
• capture logs into MIDI-able format
• handled rests
• add Show Piano Roll and Play Audio feature toggles OR - [ ] integrate DAW for music synthesis live OR (all below) ~- [ ] show staff for Play audio feature ~ - [ ] use ParseMIDI to convert logs into .mid files
• drop .mid files with same ID for server to send to synthesizer
• sync timing with recorded elements
• sync timing of highlighting
• process sibelius add tempo
• syncing of midi and aiff
• time x pixelmap
• multiple loading of several audio assets

for each lessons, the following modes will be designed and implemented

Modes and Features

• Watch and Listen Mode
• Try Mode
• Test Mode
• Toggles for Content (C) only, Content and Rhythm (CR) only, Content and Rhythm and Harmony (CHR) only and Content and Rhythm and Harmony and Metronome (CHRM) only

New Assets Needed

• recordings from iRealPro
• recordings of the different C, R, H, M

Newer classes will be implemented to apply these changes

• ModeMgr - manages the different Modes namely WaL, Try and Test
• LessonMgr - manages the content of the lesson based on the toggled C,R, H, M
• VizMgr - will be revised to reflect the new version of toggles in the prototype.

reBuild requirements as of 11.04.2023

design elements

Controllers

• RollScript - responsible for spawning and rolling piano roll
• TimeMgr - for time control, timebar generation, tempo management, time signature etc
• ImprovMgr - manages all highlighted keys that produces viz
• InputMgr - manages scoring, timing in relation to user press and chord information
• VizMgr - contains all toggle-able checkboxes that produces "adaptive" dynamic options for the user

Models

• ChordMgr - contains all chord-key piano information
• CollectionMgr - contains all chord-improv-pairing information.

Views are integrated into the design of the classes. In principle we can see Builder, Factory, Wrapper and Iterator elements among our design.

More information about dependencies and relationships will be revealed soon as we finalise the design.

forms of visualisation

• falling piano roll with recommended improv licks
• press-detected improvisation (and valid while still on press)
• highlighting of sample improvisation licks

new targets

• create classes
• integrate them with the piano canvas environment
• migrate existing functions from now-decommissioned RollScript.cs
• implement TimeMgr
• implement ImprovMgr
• implement ChordMgr
• implement InputMgr
• cleanup RollScript to its new intended purpose
• [?] find a way to implement MapLines() - discussed with bosses.
• rework RollScript.cs specifically SpawnKeys(), RollKeys() and MapLines() functions
• implement CollectionMgr
• implement passing of information based on targetted dependencies
• implement VizMgr and toggle
• demo to bosses
• Study 1 with Improv Teachers
• Develop Post-Study Version - ONGOING
• Study 2 with Improv Students - per lesson
• Study 3 on Spatiotemporal Data collection
• Graduate

MVP version ready by 22 Aug I DONT EVEN KNOW ANYMORE AY WE HERE BRUH

v1 requirements

hardware details ===

• MIDI prototype model: Kurzweil K2000 v3 reference, manual.

software details ===

latest updates as of 12/07/2022 07.02.2023

target completion of MVP 12/08/2022 13/11/2021

#### latest updates as of 16/11/2021

target completion of MVP 13/11/2021 30/11/2021

targets for an minimum viable product (MVP) - ALL DONE

hardware targets

• setup MIDI input with laptop
• see how input data looks like
• (Optional) set output in a target file destination
• allow user to press key
• play sound via garageband
• identify which key is pressed (return keybinding)
• projector setup
• laser cutting of projection board
• applying white paint/tape on black keys to assist projection visuals
• applying subtle key labels for new users
• solve issues with keys C6 to G6 having a delayed startup (significant delay)
• setup back stand of white screen for stability

software targets

• setup unity environment for the piano roll engine
• setup keybindings in unity
• setup new input system in unity
• parse midi into chord information and load them in unity environment (at least one midi file) 14.07.2022
• convert chord information unity applicable assets and information (yscale, location, etc)
• setup virtual piano environment (keys, finger colors, aspect ratio)
• calibrate virtual piano environment with the existing setup
• overlay piano roll sequence onto virtual piano environment
• synthesize sound - used garageband
• tracking of keypress along with piano roll
• black layout
• piano roll melody
• improv suggestive based on bars
• error checking - melody
• error checking - licks
• logging of spatiotemporal data - user presses
• logging of spatiotemporal data - correct presses
• scoring
• export spatio temporal data with musicxml into csv
• export everything as functions
• toggle feature feature (yes this is not a typo)

single chord

• spawn a key from this chord information
• move piano roll key
• destroy upon collision with green line
• test for timing and accuracy
• integrate with piano midi input
• destroy object upon click

multiple chords

• add timer -
• generate co-routines to time multiple spawns -
• generate co-routines for spawn movement (transform position) -
• generate data structures (list) to store piano key coordinates
• test for timing and accuracy
• integrate with piano midi input
• destroy object upon click

some other tasks due to restructuring

• setup music xml with visual interface
• connect music xml to unity engine
• configure musicxml to generate viz
• stream songs from musicxml to unity engine
• followed the steps from https://answers.unity.com/questions/1434686/how-to-integrate-drywetmidi-net-46-library-in-unit.html
• check consistency and correctness of samplesongs
• design scalable SOLID facade for adaptation engine
• file browser integration for song selection
• skeletal menu from home to practice song
• logging of keypresss
• acquiring temporal information from the midi
• synthesize sounds for C2 to B3 and C4 to C7. include sharp/flat keys
• MIDIFileSequencer (to convert .mid into List which can be used by LoadMIDIFile.cs)
• NoteDurationObject for use of SpawnNotesDropDown
• integrate DryWetMidi
• port to v 2017.4.8.f1

integration targets

• setup external speakers to play synthesized sound
• run pilot user study
• measure projection board hind legs
• [z] acquire materials for projection board hind legs

revised schedule ===

• MVP ready for initial spatiotemporal data collection 13/11/2021 30/11/2021
• spatiotemporal data collection until 13/12/2021 23/12/2021
• data cleaning and analysis
• regression modeling
• UIST paper 13/01/2021 23/01/2021

@jrdndj