Every “loss” on this site is really a measurement of information. A model’s cross-entropy is the conditional entropy of the next note given its context, in nats — how many bits of surprise are left once the model has done its best. That reframing turns the loss into a ruler for the music itself.
Ranking composers by their unpredictability
Score held-out music by conditional entropy and you get a legible ranking: the more a composer’s next note is determined by what came before, the lower the number. The most chromatic, least tonal writing sits highest; strict tonal writing sits lowest.
How much does a concept actually add?
The same lens measures a concept’s information content: how many nats it adds that aren’t already implied by its surroundings. For the blue note, the answer is almost none — which is exactly why deleting it changes so little.
The verdict
Reconstructibility is 1 − conditional_entropy. A concept reconstructs precisely to the degree that it’s redundant — carried by the grammar around it. Measuring that redundancy directly is the quantitative core of the whole project: the blue note is 90% implied by its context, so a model that never saw one still expects it.
Researcher notes
- Units. Cross-entropy loss = H(note | context) in nats/token. A pitch-shuffled floor gives the marginal entropy; the gap is how much structure the model captured.
- Concept information. The difference between a model’s surprise at a concept’s tokens and the surprise a context-only prediction would carry — small for redundant concepts, large for genuinely novel ones like djent’s register.