Finding 06
Finding 6 — Do readers fall into natural groups?
In one sentence: No — when the computer groups the readings without being told the answer, the groups it finds are only slightly related, which suggests Phil-IRI's three levels are a useful convention rather than three real, separate kinds of reader.
What this is, and what it cannot tell you
Everywhere else in this study the computer was shown the teacher's scores and asked to match them. Here it is shown nothing — just the measurements it made itself — and asked whether the readings fall into natural groups on their own.
Because there are no right answers involved, this section has no accuracy, precision, or recall. Those need a known correct label to compare against. What it can do is ask whether the structure the data actually has matches the structure Phil-IRI assumes.
Based on 167 recordings and 20 measurements each, from
the medium speech model.
Is three the natural number of groups?
A silhouette score says how cleanly separated groups are. Above 0.5 means real groups; below 0.25 means the data is really one continuous spread with no natural dividing lines.
| Number of groups | Separation score |
|---|---|
| 2 | 0.379 |
| 3 | 0.392 ← best, what Phil-IRI assumes |
| 4 | 0.275 |
| 5 | 0.272 |
| 6 | 0.305 |
Best separation is at 3 groups, scoring 0.392 — weak, could be artificial.
Three does come out best — but only just, and only weakly. It barely edges out two groups (0.379), and the score is well under the 0.50 that would indicate genuinely separate clusters. Read this as a mild tendency, not as confirmation that three reading levels exist in the data.
Reading ability behaves mostly like a continuous scale, not like categories. There is no clean gap where one reading level stops and the next begins. Any cutoff is a decision someone made, not a line the data draws.
Do three forced groups match the Phil-IRI levels?
Forcing exactly three groups and comparing them against the levels teachers assigned gives an Adjusted Rand Index of 0.071 — only slightly related. (0 means no better than random; 1 means identical.)
- Teachers' labels: {'Frustration': 24, 'Instructional': 76, 'Independent': 67}
- Computer's group sizes: [117, 36, 14]
The computer does separate the readings into groups. They are simply not the same groups the reading levels describe.
Note the shapes differ too: the teachers' levels are spread fairly evenly, while the computer puts 70% of readings into one large group and splits off two small ones. It is most likely separating recording quality and fluency extremes — the very slow or badly captured readings — rather than grading reading ability.
This is the interesting part. Three genuinely is the best-fitting number of groups, which sounds like support for Phil-IRI — but the three groups the data forms are almost unrelated to the three Phil-IRI levels. The instrument picked a plausible number and drew the lines somewhere else.
How much is actually being measured?
The system records 20 different numbers per recording, but many move together — a slow reader has long pauses and a low word rate and a long total time. Those are not independent facts.
- 4 underlying dimensions capture 80% of everything measured
- 8 capture 95%
- The two biggest alone explain 63%
So the rich-looking feature set is really only a handful of independent signals. That is a good reason to keep the correction model small — extra columns mostly repeat information that is already there.
Automatically spotting bad recordings
Without being told what "bad" means, the method flagged 9 recordings as unlike the rest. Checking them against how badly the speech recognition did:
| Group | Typical recognition error |
|---|---|
| Flagged as unusual | 0.8232 |
| Everything else | 0.2608 |
The flagged recordings are about 3.2× worse, so the flag is finding something real. This is directly useful in the app: it can mark a result "this one looks unreliable, please check" without any teacher input, and without knowing the correct score.
Why this matters for the rest of the study
Finding 01 showed that every single teacher disagreement about reading level happened within 3 points of a Phil-IRI cutoff. This section reaches the same place from a completely different direction: the data has no natural boundaries there either.
Two unrelated methods agreeing is much stronger than either alone. Together they say: report the percentage, treat the level as a label applied afterwards, and flag borderline cases instead of presenting a category as though it were certain.
Honest limitations
- 167 recordings is small for clustering. Treat this as suggestive.
- Only the measurements the ASR pipeline produces were used. A different feature set could group readings differently.
- Clustering cannot be right or wrong — it has no answer key. Nothing here says the Phil-IRI levels are invalid, only that they are not visible as natural gaps in these measurements.
Generated by pipeline/clustering.py.