About This Project

Motivation

Consumer AI music tools (Suno, Udio, Google's MusicLM) generate audio directly from text prompts. This project works at a different level of abstraction: generating symbolic MIDI sequences that musicians can edit, transpose, and arrange in a DAW. The distinction matters because symbolic representations preserve musical structure (individual notes, timing, velocity) in a format that's both human-readable and machine-processable.

This approach builds on a line of research into symbolic music generation with neural networks. The Music Transformer (Huang et al., ICLR 2019) demonstrated that self-attention could capture long-range structure in piano performances. The Pop Music Transformer (Huang & Yang, ACM MM 2020) introduced the REMI tokenisation scheme used in this project, showing that encoding beat-relative timing explicitly improves rhythmic coherence. Fradet et al. (EMNLP 2023) then showed that applying Byte Pair Encoding to symbolic music tokens reduces sequence length while improving generation quality, an approach implemented in the MidiTok library that this project uses. The V8 model also trains on the MAESTRO dataset (Hawthorne et al., ICLR 2019), a standard benchmark of 1,276 competition piano performances with aligned MIDI.

Most of this research exists as static papers with cherry-picked audio samples. The goal here was to bridge the gap between research and engineering: build a live, interactive system where each technique can be tested, compared, and tuned in real time. Rather than a paper that claims results, this is a working implementation that demonstrates them. The full pipeline (data processing, tokenisation, model training, inference API, and interactive frontend) ships as a single deployable stack, progressing from basic LSTMs with raw pitch encoding through to LLaMA-style Transformers with BPE tokenisation across seven model versions.

Architecture

The system is a three-tier stack: a Next.js frontend with an in-browser MIDI player and Web Audio synthesiser, a FastAPI backend serving PyTorch models for real-time inference, and PostgreSQL for generation history and metadata. The frontend communicates with the API to request melody generation with configurable parameters (model selection, temperature, sequence length), then renders and plays the resulting MIDI in-browser.

Model Evolution

Seven model versions track the progression from naive baselines to production-quality generation:

Observations

• Same architecture, different data, different vocabularies. V2, V3, and V4 share identical LSTM configs ([256, 512, 256]) but vocabulary ranged from 59 to 1,279 tokens depending on genre complexity. Jazz chord voicings alone drove a 21x vocab increase over R&B.
• Tokenisation changed more than widening the network. V5 widened to [512, 512, 512] but kept raw pitch encoding (629 tokens). V6 kept the same layer width, switched to REMI tokenisation, and the vocabulary dropped to 362 while encoding timing, velocity, and dynamics that raw pitch encoding discards.
• V7 and V8 share the same Transformer config (8 layers, 8 heads, 512 dims, RoPE, SwiGLU, RMSNorm, weight-tied embeddings) but differ in two ways: V8 doubled the BPE vocabulary from 512 to 1,024 and trained on 6x more data (1,551 files vs 275). V7 early-stopped at epoch 45 with val loss 0.095; V8 ran all 100 epochs with val loss 0.920. The losses aren't directly comparable due to different vocabularies and data distributions.

Stack