Eve library

Polyglot Frequency-Band Data

Other

Revision 1

On this page

This directory ships corpus-frequency band data for Mnemosyne's reading-difficulty analysis. Each file groups headwords (and their attested inflected forms) into three frequency bands corresponding to Laufer & Nation's (1995) Lexical Frequency Profile:

  • 1000 — K1 (most-frequent 1000 word families)
  • 2000 — K2 (1001–2000)
  • 3000 — K3 (2001–3000)

Files#

en-frequency-bands.json#

The New General Service List (NGSL) v1.2 by Browne, Culligan & Phillips (2013), derived from a 273M-token sub-corpus of the 2B-token Cambridge English Corpus. 2,801 headwords organised into three bands with full word-family expansions (inflections, contractions, common derivations).

es-frequency-bands.json#

Top-3000 Spanish lemmas with their attested surface forms, derived from the doozan/spanish_data frequency CSV (~25k lemmas, OPUS-derived counts). Re-organised into K1/K2/K3 bands by descending lemma frequency.

Format#

Each file is a JSON object whose top-level keys are band labels ("1000", "2000", "3000"). Each band maps a base headword (lower-cased) to an array of its inflected/related surface forms (also lower-cased). All forms within a band together constitute that band's K-set for vocabulary-profile analysis.

json
{
  "1000": {
    "be":   ["am", "is", "are", "was", "were", "been", "being", ...],
    "and":  ["ands"],
    "of":   []
  },
  "2000": { ... },
  "3000": { ... }
}

Adding a language#

  1. Obtain a permissively licensed lemma-frequency list (CC0/CC-BY/MIT).
  2. Pre-process into the band-headword-forms shape above and save as <iso639-1>-frequency-bands.json.
  3. Register the language in src/frequency-bands.ts (the loader auto-discovers files matching the naming convention but the language → file mapping is declared explicitly there).
  4. Cite the source in this README.

Licensing#

The two shipped files are redistributed under their respective upstream licences (CC0 for NGSL, CC-BY-4.0 for Spanish). See LICENSE notes in each upstream repository.