Accuracy
Every language is scored at test time against the slots where two independent oracles agree: a Wiktionary extraction and a national or academic lexicon. The engine hits 100% on those. What is left over is the two oracles disagreeing with each other, which we rule on by hand.
What “accurate” means
A form is correct when it is what a fluent native speaker would write for that cell of the paradigm. We can’t ask speakers about every form, so we use a proxy: agreement between two sources built independently of each other. For each language that is a Wiktionary-derived lexicon plus a second source (a national or academic lexicon, a UniMorph dataset, or an Apertium dictionary), and we score the engine only on the forms where the two already agree.
Accuracy is the share of that agreed set the engine reproduces exactly:
accuracy = engine-matched forms ÷ forms the two oracles agree on
Two things stay out of that number. Where the oracles disagree with each other there is no ground truth to score against, so those slots are excluded and instead settled by hand against reference grammars, then published (the Disagreements column). A slot the engine doesn’t produce yet is a coverage gap, tracked separately (the Slots column) instead of counted as wrong. A form the engine invents that neither oracle attests is always a failure, which is exactly what this method is built to catch.
Tiers
- Verified — 100% on the agreement, every paradigm slot covered, and every oracle disagreement resolved.
- Verified·gaps — 100% on the agreement, but with uncovered slots and/or unresolved disagreements still being worked through.
- Beta — 100% against a single oracle (no overlapping independent second oracle yet).
Everything shipped is at least Beta: 100% correct on one oracle.
Across versions
Correctness is measured by the golden harness, added in 0.8.0. Every release since adds a point here, so a regression would show up as a number that drops.
| Release | Languages | Verified forms | Overall agreement |
|---|---|---|---|
| 0.10.0 | 34 | 3,108,362 | 99.998% |
| 0.9.0 | 28 | 3,041,196 | 99.998% |
| 0.8.0 | 22 | 2,914,028 | 99.998% |
By language
| Language | Oracles | Agreed forms | Agreed | Slots | Disagreements | Tier |
|---|---|---|---|---|---|---|
| Armenian | uniparser ∩ unimorph | 50,521 / 50,521 | 100.00% | 63/63 | 179/179 | Verified |
| Bengali | unimorph | 3,864 / 3,864 | 100.00% | 46/46 | — | Beta |
| Catalan | freeling ∩ kaikki | 187,186 / 187,186 | 100.00% | 50/50 | 13/110 | Verified·gaps |
| Czech | morfflex ∩ kaikki | 107,812 / 107,812 | 100.00% | 29/32 | 0/1397 | Verified·gaps |
| Danish | cor-verbs ∩ kaikki | 20,205 / 20,205 | 100.00% | 9/9 | 0/165 | Verified·gaps |
| Dutch | apertium ∩ kaikki | 27,836 / 27,836 | 100.00% | 12/12 | 0/66 | Verified·gaps |
| English | agid ∩ kaikki | 74,600 / 74,600 | 100.00% | 5/5 | 0/163 | Verified·gaps |
| Estonian | vabamorf ∩ kaikki | 24,184 / 24,184 | 100.00% | 41/41 | 0/77 | Verified·gaps |
| Finnish | omorfi ∩ kaikki | 406,209 / 406,209 | 100.00% | 35/35 | 0/8201 | Verified·gaps |
| French | lefff ∩ kaikki | 284,034 / 284,034 | 100.00% | 49/49 | 0/342 | Verified·gaps |
| German | unimorph ∩ kaikki | 193,038 / 193,067 | 99.98% | 30/30 | 0/2141 | Verified·gaps |
| Gujarati | unimorph | 2,880 / 2,880 | 100.00% | 32/32 | — | Beta |
| Hindi | kaikki ∩ unimorph | 48,421 / 48,421 | 100.00% | 205/205 | 3708/3708 | Verified |
| Icelandic | bin ∩ kaikki | 5,749 / 5,749 | 100.00% | 26/26 | 0/42 | Verified·gaps |
| Irish | bunamo ∩ kaikki | 27,404 / 27,404 | 100.00% | 25/25 | 0/488 | Verified·gaps |
| Italian | morphit ∩ kaikki | 280,143 / 280,143 | 100.00% | 49/49 | 0/1185 | Verified·gaps |
| Japanese | ipadic ∩ kaikki | 9,421 / 9,421 | 100.00% | 3/3 | 0/22 | Verified·gaps |
| Kannada | unimorph ∩ kaikki | 411 / 411 | 100.00% | 14/14 | 5/5 | Verified |
| Korean | krdict ∩ kaikki | 3,576 / 3,576 | 100.00% | 4/4 | 0/17 | Verified·gaps |
| Marathi | apertium | 56,551 / 56,551 | 100.00% | 44/46 | — | Beta |
| Persian | kaikki ∩ apertium | 2,683 / 2,683 | 100.00% | 28/28 | 118/118 | Verified |
| Portuguese | morphobr ∩ kaikki | 373,163 / 373,163 | 100.00% | 70/70 | 0/191 | Verified·gaps |
| Romanian | dexonline ∩ kaikki | 225,494 / 225,494 | 100.00% | 35/35 | 0/1977 | Verified·gaps |
| Russian | opencorpora ∩ kaikki | 151,912 / 151,942 | 99.98% | 19/19 | 0/273 | Verified·gaps |
| Slovenian | sloleks ∩ kaikki | 10,757 / 10,757 | 100.00% | 25/25 | 0/226 | Verified·gaps |
| Spanish | freeling ∩ kaikki | 339,200 / 339,200 | 100.00% | 59/59 | 0/723 | Verified·gaps |
| Swahili | swc ∩ kaikki | 1,134 / 1,134 | 100.00% | 26/26 | 36/36 | Verified |
| Swedish | saldo ∩ kaikki | 39,594 / 39,594 | 100.00% | 11/13 | 0/296 | Verified·gaps |
| Tagalog | kaikki ∩ unimorph | 469 / 469 | 100.00% | 7/7 | 0/191 | Verified·gaps |
| Tamil | thamizhi ∩ kaikki | 31,829 / 31,829 | 100.00% | 37/37 | 0/980 | Verified·gaps |
| Telugu | unimorph | 1,159 / 1,159 | 100.00% | 24/24 | — | Beta |
| Turkish | kaikki ∩ unimorph | 44,156 / 44,156 | 100.00% | 93/93 | 0/10763 | Verified·gaps |
| Ukrainian | vesum ∩ kaikki | 71,990 / 71,990 | 100.00% | 14/14 | 0/329 | Verified·gaps |
| Urdu | kaikki ∩ apertium | 777 / 777 | 100.00% | 10/10 | 4/4 | Verified |
Agreed forms — engine matches / slots the two oracles agree on. Slots — paradigm slot types the engine covers / present in the gold. Disagreements — oracle-vs-oracle splits resolved / total.
Measured for ablaut 0.10.0 by the golden harness (regenerated each release). Per-language detail lives in the crate under docs/<lang>/.