What if a single note isn’t the right unit? Replace every jazz melody note with a voicing of the chord actually sounding underneath it, and train on that. The loss falls to a third of what it was. And none of that improvement is understanding.
The representation, and what it generates
Chord-melody: a corpus sample and a generation
Not the prettiest music — but the model clearly learned the representation. The point is what the low loss does, and doesn't, mean.
Held-out entropy: note-level 2.33 nats/token vs chord-level 0.85. Accompaniment is only 0.29 nats per note given the melody — nearly free. The melody itself got no easier: 2.31 vs 2.44.
The loss fell by two-thirds, and the melody — the hard part — got no easier at all. The drop is redundancy, not comprehension.
The verdict
Per-token loss is not a scoreboard for musical understanding. When you fold the accompaniment into the representation, most of each token becomes trivially predictable given the melody, so the average plummets while nothing got smarter. This is the lesson that made us distrust the metal model’s very low loss, and it’s the same trap the drum reconstruction nearly fell into.
Researcher notes
- Voicing. Each melody note becomes root + third + seventh of the sounding chord, voiced below the melody — a polyphonic stream the tokenizer handles directly.
- Decomposition. Measuring melody-only entropy vs full-representation entropy separates “the model predicts the melody” (unchanged) from “the model predicts the accompaniment given the melody” (near-free).