Finding 02
Finding 2 — Which model size should each device use?
In one sentence: medium is the most accurate model tested, and the table
below shows what you give up on each smaller device.
Why this matters
A bigger speech model is more accurate but costs more to run. The finished system has to run somewhere real, on a budget, so the question is not "which model is best" but "what does each level of compute cost us in accuracy."
Note on phones: in this system the phone only records and uploads — all speech recognition happens on the server. The tiers below are server tiers. Timings come from one laptop GPU and are not handset benchmarks.
WER below means Word Error Rate — the share of words the computer got wrong when listening. Lower is better. 0.20 means roughly 1 word in 5 was misheard.
The comparison
| Model | Realistic device | Words misheard (WER) | English | Filipino | Seconds per recording |
|---|---|---|---|---|---|
tiny | Minimal CPU box | 0.733 | 0.5862 | 0.8917 | 2.4335 |
base | Small CPU box | 0.559 | 0.3783 | 0.7553 | 2.7327 |
small | Modest server / shared CPU | 0.402 | 0.2799 | 0.5336 | 3.0715 |
medium | Entry GPU server | 0.275 | 0.2284 | 0.3255 | 6.2466 |
large-v3-turbo | GPU server — best speed/accuracy trade | 0.421 | 0.3884 | 0.4561 | 6.3971 |
large-v3 | GPU server — highest accuracy | 0.308 | 0.2497 | 0.3716 | 59.8073 |
The English / Filipino gap
This is the finding that matters most for a bilingual Philippine deployment: smaller models are not equally bad in both languages. They tend to fall apart on Filipino while still coping with English.
| Model | How much worse Filipino is |
|---|---|
tiny | +0.305 ⚠️ |
base | +0.377 ⚠️ |
small | +0.254 ⚠️ |
medium | +0.097 |
large-v3-turbo | +0.068 |
large-v3 | +0.122 ⚠️ |
Largest gap: base (+0.377). Smallest gap: large-v3-turbo (+0.068).
A model with a large gap is not safe to ship to Filipino-speaking classrooms, even if its overall average looks acceptable.
How each model fails
Two models can share a WER and be wrong in completely different ways. For reading assessment the kind of error matters more than the amount, because a model that quietly skips words looks like a child who skipped them.
| Model | Words invented | Words misheard | Words skipped | Share of passage returned |
|---|---|---|---|---|
tiny | 6.8 | 45.5 | 29.3 | 0.79 ⚠️ |
base | 9.1 | 43.2 | 12.1 | 0.99 |
small | 4.9 | 25.0 | 16.2 | 0.91 |
medium | 3.9 | 19.2 | 7.4 | 0.97 |
large-v3-turbo | 4.6 | 14.0 | 26.7 | 0.79 ⚠️ |
large-v3 | 6.4 | 14.5 | 13.6 | 0.94 |
Words skipped is the dangerous column. A model that returns much less than the full passage is not reading badly — it is stopping early, and every word it never attempted is counted against the child. Anything below ~0.85 in the last column should not be used for assessment at any speed.
What this means for the app
- Pick the model per device tier, not once for the whole product.
- Each tier needs its own bias correction — a correction fitted for the big model would be wrong for the small one.
- Check the Filipino column before approving any tier for real classroom use.
What this cost to run
Nothing here was trained. Whisper is a pre-trained model, so this is the cost of running it over the corpus — the machine listening to every recording six times over. (The only thing this project trains is the small correction model, which takes milliseconds on a CPU.)
Measured on one laptop with an 8 GB RTX 4060, over 215 recordings per model.
| Model | Listening | Aligning words | Total GPU time | Typical recording | Average |
|---|---|---|---|---|---|
tiny | 0.14 h | 0.11 h | 0.25 h | 2.6 s | 4.2 s |
base | 0.16 h | 0.10 h | 0.27 h | 3.5 s | 4.5 s |
small | 0.18 h | 1.04 h | 1.22 h | 7.4 s | 20.4 s |
medium | 0.37 h | 0.22 h | 0.59 h | 6.4 s | 9.9 s |
large-v3-turbo | 0.38 h | 0.37 h | 0.76 h | 8.8 s | 12.6 s |
large-v3 | 3.57 h | 0.95 h | 4.52 h | 50.9 s | 75.7 s |
| All six | 4.82 h | 2.79 h | 7.61 h |
Use the "typical" column, not the average. On 26 recordings the
model lost its place and produced a runaway transcript, which then took up to
~3 minutes to align instead of ~5 seconds. Those few cases drag the average far
above what a normal recording actually costs — small averages 20 s per
recording but typically takes about 5 s.
This is worth knowing for two reasons: budget from the typical figure, and treat a slow alignment as a free warning sign that the transcript for that recording is probably garbage and should be checked.
large-v3 alone accounts for 3.57 h of the 4.82 h spent listening —
74% of the total, for one of six models. On an 8 GB card it barely
fits in memory, and the shortage slows down the word-alignment step as well.
Practical reading: the whole six-model comparison fits in a single overnight run on one consumer laptop. A school deployment only ever runs one model, so the per-recording column is the number that matters for planning.
Honest limitations
- Timings come from one laptop GPU, not from real phones. Treat the device column as guidance about relative cost, not a measured on-device benchmark.
- WER counts every misheard word equally, including ones a teacher would ignore.
Generated by pipeline/summarize_models.py.