Every speller gets the same MIDI notes, without key signatures, and every note is scored by the same three-tier scorer:
- Exact
- spelled as the composer wrote it.
- Flipped
- a coherent transposition: part of a passage spelled consistently on the other enharmonic side (D♭ major read as C♯ major), the same music shifted by a comma. Not an error.
- Wrong
- breaks the spelling of the notes around it (C♯–F–G♯ for C♯–E♯–G♯). The bars below rank by this.
By dataset
| Speller | Exact % | Flipped % | Wrong % | Wrong notes | Note |
|---|
Are the gaps real?
Each of our spellers against the published spellers closest to it. Paired bootstrap over pieces: “better” means the 95% interval on the difference in wrong-note rate (ours minus theirs) lies below zero, “tie” that it includes zero.
| Ours | Against | Wrong notes | Difference | 95% interval | Pieces won / lost / tied | Verdict | Note |
|---|
Across datasets
| Dataset | Two-pass wrong % | Best other speller | its wrong % | Two-pass rank |
|---|
All datasets
| Dataset | Two-pass wrong % | Best other speller | its wrong % | Two-pass rank |
|---|
The held-out sets (ASAP, OpenScore lieder, all DCML corpora pooled, and four fresh sets gathered after our spellers were built) are harder than Meredith: no speller reaches its Meredith accuracy there, so compare the rankings, not the absolute figures with published ones. Each set keeps only the pieces every speller completed: Temperley's program fails on a few pieces per set, and those are removed from every row. PSE was not run on them.
Our four spellers
A ladder: each rung adds one idea to the one before.
- Core: seven letter slots and interval scoring that penalises augmented and diminished intervals. Real time, about a hundred lines.
- Real-time: adds a diatonic frame that holds the enharmonic side of a passage. Zero latency, for live MIDI.
- Look-ahead: also reads ahead to where each note resolves. Near real time.
- Two-pass: a forward and a backward pass over the whole piece, then a merge. Offline.
Read with care
- The other spellers' rows are our own runs of the implementations we could obtain or rebuild (music21, partitura's ps13, PKSpell, MuseScore, and reimplementations of the rest), scored with our scorer. They are not the authors' published figures.
- Meredith's corpus (216 movements, clean and with human-performance timing) is the standard benchmark; some of our design choices were checked on it. The held-out sets (ASAP, OpenScore lieder, DCML, fresh sets) were not used to build the spellers. The lab corpus is our own curated set, so treat it as development data.
- The stitched WTC datasets play each book as one unbroken stream. They test how a speller recovers from key changes, not its accuracy on a single piece; the per-movement datasets are the control.
- The DCML corpora (Beethoven, Mozart, Chopin, Liszt and others, from the Digital and Cognitive Musicology Lab) are expert-encoded scores we never tuned on, pooled into one set. Their spellings come from the encoded score.
- PKSpell was trained on WTC and on most of ASAP (which shares pieces with several DCML corpora), so its rows there are not a fair comparison; on the ASAP pieces it never saw, two-pass makes fewer wrong notes and PKSpell matches the composer's side more often. Four lab pieces are in its training data and are left out of the lab corpus's 50.