This directory ships corpus-frequency band data for Mnemosyne's reading-difficulty analysis. Each file groups headwords (and their attested inflected forms) into three frequency bands corresponding to Laufer & Nation's (1995) Lexical Frequency Profile:
- 1000 — K1 (most-frequent 1000 word families)
- 2000 — K2 (1001–2000)
- 3000 — K3 (2001–3000)
Files#
en-frequency-bands.json#
The New General Service List (NGSL) v1.2 by Browne, Culligan & Phillips (2013), derived from a 273M-token sub-corpus of the 2B-token Cambridge English Corpus. 2,801 headwords organised into three bands with full word-family expansions (inflections, contractions, common derivations).
- Source: https://www.newgeneralservicelist.com/
- Machine-readable redistribution (CC0 1.0): https://github.com/lpmi-13/machine_readable_wordlists
- Citation: Browne, C., Culligan, B. & Phillips, B. (2013). The New General Service List. http://www.newgeneralservicelist.org
es-frequency-bands.json#
Top-3000 Spanish lemmas with their attested surface forms, derived from the doozan/spanish_data frequency CSV (~25k lemmas, OPUS-derived counts). Re-organised into K1/K2/K3 bands by descending lemma frequency.
- Source: https://github.com/doozan/spanish_data (CC-BY-4.0)
- Pre-processing:
countfield → rank → band;usagefield parsed for surface forms.
Format#
Each file is a JSON object whose top-level keys are band labels ("1000",
"2000", "3000"). Each band maps a base headword (lower-cased) to an array of
its inflected/related surface forms (also lower-cased). All forms within a band
together constitute that band's K-set for vocabulary-profile analysis.
{
"1000": {
"be": ["am", "is", "are", "was", "were", "been", "being", ...],
"and": ["ands"],
"of": []
},
"2000": { ... },
"3000": { ... }
}
Adding a language#
- Obtain a permissively licensed lemma-frequency list (CC0/CC-BY/MIT).
- Pre-process into the band-headword-forms shape above and save as
<iso639-1>-frequency-bands.json. - Register the language in
src/frequency-bands.ts(the loader auto-discovers files matching the naming convention but the language → file mapping is declared explicitly there). - Cite the source in this README.
Licensing#
The two shipped files are redistributed under their respective upstream licences (CC0 for NGSL, CC-BY-4.0 for Spanish). See LICENSE notes in each upstream repository.