Skip to main content
mehen emits every formula’s raw score with provenance rather than averaging. Two formulas on the same text routinely disagree by 2–4 grade levels because they target different comprehension thresholds (SMOG ~100%, FKGL ~75%, Dale-Chall in between). Averaging them is statistically wrong.

​
Formulas

​
Ensemble reporting

  1. Emit every formula with provenance.
  2. Compute an ensemble grade band as [min(FKGL, Fog, ARI, CLI), max(…)] — the interval where those four “running-prose” formulas agree.
  3. Emit FORCAST separately as the preferred single score for non-narrative docs.
  4. Suppress SMOG when sentences < 30.
  5. Report Dale-Chall only with an explicit list: provenance tag (NGSL 1.2 by default — Browne et al., 2013).

​
Syllable counting

Tier 0 default is a vowel-group heuristic (~85% agreement with CMU on open-domain text). Behind --features syllables-cmu, mehen links the CMU Pronouncing Dictionary for exact counts on ~134k words with the heuristic as an OOV fallback.

​
Sentence segmentation

UAX #29 (unicode-segmentation) plus:
  • A bundled ~150-entry English abbreviation list (Mr., e.g., i.e., U.S., v1.2.3).
  • No split when the period is followed by a lowercase letter, a digit, or <space><digit>.
  • Markdown block boundaries (blank line, heading, fence open/close, list item start) are hard terminators regardless of punctuation.

​
Doc-type thresholds

These are conventions synthesized from Google, Microsoft, and 18F style guides. They are tunable profile defaults.

​
References

  • Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology 32(3): 221–233. DOI.
  • Kincaid, J. P., Fishburne, R. P., Rogers, R. L. & Chissom, B. S. (1975). Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Research Branch Report 8-75, Naval Technical Training Command. DTIC PDF.
  • McLaughlin, G. H. (1969). SMOG grading — a new readability formula. Journal of Reading 12(8): 639–646. JSTOR.
  • Gunning, R. (1952). The Technique of Clear Writing. McGraw-Hill.
  • Coleman, M. & Liau, T. L. (1975). A computer readability formula designed for machine scoring. Journal of Applied Psychology 60(2): 283–284. DOI.
  • Senter, R. J. & Smith, E. A. (1967). Automated Readability Index. AMRL-TR-66-220. DTIC PDF.
  • Chall, J. S. & Dale, E. (1995). Readability Revisited: The New Dale-Chall Readability Formula. Brookline Books.
  • Caylor, J. S. & Sticht, T. G. (1973). Development of a Simple Readability Index for Job Reading Material. HumRRO Professional Paper 1-73 (FORCAST). DTIC.
  • Anderson, J. (1983). Lix and Rix: Variations on a little-known readability index. Journal of Reading 26(6): 490–496. JSTOR.

​
See also