Skip to content

Latest commit

 

History

History

project

README.md

Project

The main project for the 2018-2019 QMIND Music Team! This year, we created a neural network with the ability to generate a song given an image. There are many facets to this project, if you want to skip to how to run it locally click here. Otherwise keep reading!

Background

The Network

At the core of the project is an LSTM neural network. This network took much inspiration from an article in Towards Data Science, however many modifications were made to make it fit our use case.

The network is trained with data collected from Spotify's Web API and stored in CSV format. As it stands currently, our dataset consists of 100 songs from the following playlist. From there, the cover artwork is downloaded in JPG format and the 30 second previews are downloaded in MP3 format for each song. The cover artwork is converted into an array using the KAZE Algorithm and the songs are converted into MIDI format using the Melodia algorithm. To see how this was implemented in Python, check out the Library folder.

The Web App

To provide an interface for using our neural network, we made a web app. The web app allows users to upload photos from their phone or computer and receive a generated song from the neural network as output. All the song and image pairs are also displayed in a gallery for visitors to enjoy.

The back end of the application is a Flask server. The server accepts requests through A REST API and is also connected to a Google Cloud Storage Bucket for storing image and audio files. The front end of the application is a ReactJS client.

Usage

Requirements

Before running our project locally, make sure your device has the following requirements:

To train the network, run

  $ python3 train.py

Note: all bash commands should be run from the project folder and not the root of this repo

To run the web app, you will first need to create your own Google Cloud Storage Bucket called qmusicbucket. Once you have done this, obtain a service account key for your project. Create a JSON file at the root of this folder called google-config.json and paste the contents of the service account key in this file.

To run the app in development mode, run

  $ make dev

This will run the client with the Webpack Development Server which allows for hot reloading and easy debugging.

To run the app in production mode, run

  $ python3 server.py

The default port will be 4000, however this can be configured with the PORT environment variable.

Contributing

If you wish to contribute to the project (and are not a member of the QMIND Music Team), follow the following steps:

  1. Fork this repo 🍴
  2. Clone it 👽
  3. Execute the following commands and make a PR 🚀
  $ git checkout -b feature/add-new-stuff
  $ # Make some changes now
  $ # .
  $ # .
  $ # .
  $ git commit -am "Some changes"
  $ git push -u origin feature/add-new-stuff