Accuracy
Every language is scored at test time against the slots where two independent oracles agree — a Wiktionary extraction crossed with a national or academic lexicon. The engine hits 100% there; the residual is the two oracles disagreeing with each other, which is ruled on case by case.
What “accurate” means
A form is correct when it is the form a competent native speaker writes for that cell of the paradigm. We can’t poll speakers form by form, so we approximate truth with agreement between two independent authorities: for each language we take a Wiktionary-derived lexicon and a second, independently built source — a national or academic lexicon, a UniMorph dataset, or an Apertium dictionary — and score the engine only on the forms where those two already agree.
Accuracy is the share of that agreed set the engine reproduces exactly:
accuracy = engine-matched forms ÷ forms the two oracles agree on
Two things are deliberately kept out of that number. Where the oracles disagree with each other there is no ground truth to score against, so those slots are excluded and instead adjudicated by hand against reference grammars and published (the Disagreements column). And a slot the engine doesn’t yet produce is a coverage gap, tracked separately (the Slots column) rather than counted as a wrong answer. A form the engine invents that neither oracle attests is always a failure — that is the error class this whole method exists to catch.
Tiers
- Verified — 100% on the agreement, every paradigm slot covered, and every oracle disagreement resolved.
- Verified·gaps — 100% on the agreement, but with uncovered slots and/or unresolved disagreements still being worked through.
- Beta — 100% against a single oracle (no overlapping independent second oracle yet).
Everything shipped is at least Beta: 100% correct on one oracle.
Across versions
Correctness is measured by the golden harness, which landed in 0.8.0; every release since adds a point here, so a regression would show up as a number going down.
| Release | Languages | Verified forms | Overall agreement |
|---|---|---|---|
| 0.9.0 | 28 | 3,041,196 | 99.998% |
| 0.8.0 | 22 | 2,914,028 | 99.998% |
By language
| Language | Oracles | Agreed forms | Agreed | Slots | Disagreements | Tier |
|---|---|---|---|---|---|---|
| Armenian | uniparser ∩ unimorph | 50,521 / 50,521 | 100.00% | 63/63 | 179/179 | Verified |
| Catalan | freeling ∩ kaikki | 187,186 / 187,186 | 100.00% | 50/50 | 13/110 | Verified·gaps |
| Czech | morfflex ∩ kaikki | 107,812 / 107,812 | 100.00% | 29/32 | 0/1397 | Verified·gaps |
| Danish | cor-verbs ∩ kaikki | 20,205 / 20,205 | 100.00% | 9/9 | 0/165 | Verified·gaps |
| Dutch | apertium ∩ kaikki | 27,836 / 27,836 | 100.00% | 12/12 | 0/66 | Verified·gaps |
| English | agid ∩ kaikki | 74,600 / 74,600 | 100.00% | 5/5 | 0/163 | Verified·gaps |
| Estonian | vabamorf ∩ kaikki | 24,184 / 24,184 | 100.00% | 41/41 | 0/77 | Verified·gaps |
| Finnish | omorfi ∩ kaikki | 406,209 / 406,209 | 100.00% | 35/35 | 0/8201 | Verified·gaps |
| French | lefff ∩ kaikki | 284,034 / 284,034 | 100.00% | 49/49 | 0/342 | Verified·gaps |
| German | unimorph ∩ kaikki | 193,038 / 193,067 | 99.98% | 30/30 | 0/2141 | Verified·gaps |
| Hindi | kaikki ∩ unimorph | 48,421 / 48,421 | 100.00% | 205/205 | 3708/3708 | Verified |
| Icelandic | bin ∩ kaikki | 5,749 / 5,749 | 100.00% | 26/26 | 0/42 | Verified·gaps |
| Irish | bunamo ∩ kaikki | 27,404 / 27,404 | 100.00% | 25/25 | 0/488 | Verified·gaps |
| Italian | morphit ∩ kaikki | 280,143 / 280,143 | 100.00% | 49/49 | 0/1185 | Verified·gaps |
| Japanese | ipadic ∩ kaikki | 9,421 / 9,421 | 100.00% | 3/3 | 0/22 | Verified·gaps |
| Korean | krdict ∩ kaikki | 3,576 / 3,576 | 100.00% | 4/4 | 0/17 | Verified·gaps |
| Portuguese | morphobr ∩ kaikki | 373,163 / 373,163 | 100.00% | 70/70 | 0/191 | Verified·gaps |
| Romanian | dexonline ∩ kaikki | 225,494 / 225,494 | 100.00% | 35/35 | 0/1977 | Verified·gaps |
| Russian | opencorpora ∩ kaikki | 151,912 / 151,942 | 99.98% | 19/19 | 0/273 | Verified·gaps |
| Slovenian | sloleks ∩ kaikki | 10,757 / 10,757 | 100.00% | 25/25 | 0/226 | Verified·gaps |
| Spanish | freeling ∩ kaikki | 339,200 / 339,200 | 100.00% | 59/59 | 0/723 | Verified·gaps |
| Swahili | swc ∩ kaikki | 1,134 / 1,134 | 100.00% | 26/26 | 36/36 | Verified |
| Swedish | saldo ∩ kaikki | 39,594 / 39,594 | 100.00% | 11/13 | 0/296 | Verified·gaps |
| Tagalog | kaikki ∩ unimorph | 469 / 469 | 100.00% | 7/7 | 0/191 | Verified·gaps |
| Tamil | thamizhi ∩ kaikki | 31,829 / 31,829 | 100.00% | 37/37 | 0/980 | Verified·gaps |
| Telugu | unimorph | 1,159 / 1,159 | 100.00% | 24/24 | — | Beta |
| Turkish | kaikki ∩ unimorph | 44,156 / 44,156 | 100.00% | 93/93 | 0/10763 | Verified·gaps |
| Ukrainian | vesum ∩ kaikki | 71,990 / 71,990 | 100.00% | 14/14 | 0/329 | Verified·gaps |
Agreed forms — engine matches / slots the two oracles agree on. Slots — paradigm slot types the engine covers / present in the gold. Disagreements — oracle-vs-oracle splits resolved / total.
Measured for ablaut 0.9.0 by the golden harness (regenerated each release). Per-language detail lives in the crate under docs/<lang>/.