# Polyglot Frequency-Band Data

This directory ships corpus-frequency band data for Mnemosyne's
reading-difficulty analysis. Each file groups headwords (and their attested
inflected forms) into three frequency bands corresponding to Laufer & Nation's
(1995) Lexical Frequency Profile:

- **1000** — K1 (most-frequent 1000 word families)
- **2000** — K2 (1001–2000)
- **3000** — K3 (2001–3000)

## Files

### `en-frequency-bands.json`

The **New General Service List (NGSL) v1.2** by Browne, Culligan & Phillips
(2013), derived from a 273M-token sub-corpus of the 2B-token Cambridge English
Corpus. 2,801 headwords organised into three bands with full word-family
expansions (inflections, contractions, common derivations).

- Source: <https://www.newgeneralservicelist.com/>
- Machine-readable redistribution (CC0 1.0):
  <https://github.com/lpmi-13/machine_readable_wordlists>
- Citation: Browne, C., Culligan, B. & Phillips, B. (2013). _The New General
  Service List._ <http://www.newgeneralservicelist.org>

### `es-frequency-bands.json`

Top-3000 Spanish lemmas with their attested surface forms, derived from the
**doozan/spanish_data** frequency CSV (~25k lemmas, OPUS-derived counts).
Re-organised into K1/K2/K3 bands by descending lemma frequency.

- Source: <https://github.com/doozan/spanish_data> (CC-BY-4.0)
- Pre-processing: `count` field → rank → band; `usage` field parsed for surface
  forms.

## Format

Each file is a JSON object whose top-level keys are band labels (`"1000"`,
`"2000"`, `"3000"`). Each band maps a base headword (lower-cased) to an array of
its inflected/related surface forms (also lower-cased). All forms within a band
together constitute that band's K-set for vocabulary-profile analysis.

```json
{
  "1000": {
    "be":   ["am", "is", "are", "was", "were", "been", "being", ...],
    "and":  ["ands"],
    "of":   []
  },
  "2000": { ... },
  "3000": { ... }
}
```

## Adding a language

1. Obtain a permissively licensed lemma-frequency list (CC0/CC-BY/MIT).
2. Pre-process into the band-headword-forms shape above and save as
   `<iso639-1>-frequency-bands.json`.
3. Register the language in `src/frequency-bands.ts` (the loader auto-discovers
   files matching the naming convention but the language → file mapping is
   declared explicitly there).
4. Cite the source in this README.

## Licensing

The two shipped files are redistributed under their respective upstream licences
(CC0 for NGSL, CC-BY-4.0 for Spanish). See LICENSE notes in each upstream
repository.
