Everything so far lived in tonal music. To find out whether the reconstruction result depends on functional harmony at all, the study needed a corpus with none — so it pivoted to metal: 23,000 MIDIs, distorted guitars merged per song, a from-scratch model. And immediately, an honest lesson about its loss.
A model composing metal from one token
g_metal_s1, generating from a single start token
No prompt — one 'begin' token, then the model writes a riff in the low register, chromatic and chugging. The first clip is a real training example for comparison.
g_metal_s1: d512 / 8 layers, ctx 1024, 25.5M params, 10k steps, bf16 on a rented RTX 4090 — 18.7 minutes, about $0.25, 440,000 tokens a second.
The loss is low, and that’s a warning
The model reaches a validation loss of 0.23 nats — much lower than the classical or jazz models. That is not because it’s smarter. Metal guitar in this representation is extremely redundant: velocities are near-constant, durations quantized, tremolo and chug endlessly repeated. The 327 MB training stream gzips to 28 MB — a 12:1 compression that tells you most of the tokens are trivially predictable.
The verdict
A working metal model, built for under a quarter — and the chord-melody lesson applied before anyone got excited: judge it by its output and by per-feature measures, never by the aggregate loss. With the model in hand, the real question was open: does concept deletion still fail on non-functional, power-chord harmony? It does.
Researcher notes
- Corpus. A public Kaggle metal-MIDI dump (PDDL). Per song, merge the distorted/overdrive guitar tracks into one part, dedupe doubled unisons, keep tempo. 90/5/5 split.
- Caveats. The REMI tokenizer ignores time signatures, so odd meters land on a 4/4 grid; near-duplicate transcriptions add some train/test leakage (which doesn’t affect the matched control/edited pairs the real experiments use).