pauljd/craft-libs
1
0
Fork 0
mirror of https://github.com/storytold/craft-libs.git synced 2026-11-07 12:13:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

CP1.2: English without Language data uses the whole Moby list and every installed English dictionary

Browse Source 
Review of craft-libs#482:
- The built-in fallback is now the full public-domain Moby word list (data/en-words.dic,
  words only, 343,992 bytes, from craft-i18n's en-us.dic by examples/en_words.rs) plus
  WordCraft's own en-extra.txt (MIT OR Apache-2.0). An unknown word is misspelled; the
  near-miss scan (which flagged fiber, theater, organize… and took 14 s on a 2,000-word
  token) is gone. Suggestions are the words one edit away, looked up, for words of up to 32
  letters. spelling_scales_over_one_long_token runs without the data, with a time bound.
- English Hunspell dictionaries ignore compounding (COMPOUNDMIN, COMPOUNDRULE…; their
  ONLYINCOMPOUND stems are left out) and OCONV instead of refusing the file, and a file that
  can't be read is skipped for the next name or folder.
- A load that started before english::reload() isn't kept (reloads counter, as slot.rs);
  Englishes that use the same .dic share one copy. en-GB/en-CA falling back to en_US is
  documented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FypgCX8LsYYE8fsj4byBm2
This commit is contained in:
Martin Ball
2026-11-01 18:37:02 +00:00
co-authored by Claude Opus 5.5
parent 961162ad15
commit c80b6225f1
14 changed files with 738 additions and 11627 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ATTRIBUTION.md
+2 -1
View File
@@ -14,7 +14,8 @@ author, source and license. Library code and original assets are under the repos
| `crates/cms/tests/data/FOGRA39.txt` (unmodified) | Fogra | [Fogra characterisation data](https://fogra.org/en/downloads/work-tools/characterisation-data), `FOGRA39_to_FOGRA60_v2.zip` | Fogra terms: free use and redistribution, unmodified, with Fogra identified as the source; "FOGRA39" identifies the reference data only and implies no certification, approval or endorsement by Fogra (full text in [NOTICE](NOTICE)) |
| `crates/i18n/data/collation.postcard` | Unicode, Inc. (ICU 78.1 collation data, CLDR 48.2.1); generated by `crates/i18n/data/generate-collation.sh` | [ICU](https://github.com/unicode-org/icu/releases/tag/release-78.1rc) | Unicode-3.0 |
| `crates/text/assets/hyphen/en-us.dic` | Grady Ward (Moby Hyphenator II), converted by DesignCraft | [Project Gutenberg eBook #3204](https://www.gutenberg.org/ebooks/3204) | Public domain |
| `crates/proof/data/en-core.txt` | Grady Ward (Moby Hyphenator II word list); selected by craft-proof contributors with `crates/proof/examples/en_core.rs`, ranked by word frequency in 50 public-domain Project Gutenberg books | [Project Gutenberg eBook #3204](https://www.gutenberg.org/ebooks/3204), via craft-i18n `language-data/en-us/en-us.dic` | Public domain |
| `crates/proof/data/en-words.dic` | Grady Ward (Moby Hyphenator II word list); break points dropped by craft-proof contributors with `crates/proof/examples/en_words.rs` | [Project Gutenberg eBook #3204](https://www.gutenberg.org/ebooks/3204), via craft-i18n `language-data/en-us/en-us.dic` | Public domain |
| `crates/proof/data/en-extra.txt` | WordCraft contributors (original list, no other dictionary copied) | craft-i18n `language-data/en-us/en-extra.txt` | MIT OR Apache-2.0 |
| `crates/text/assets/hyphen/en-us.pat` | DesignCraft contributors, trained from the Moby Hyphenator II list | `crates/text/examples/hyphgen.rs` | CC0-1.0 |
| Patterns embedded through `hypher` (optional `craft-text` language features) | hyph-utf8 authors, per language | [tex-hyphen](https://github.com/hyphenation/tex-hyphen), as packaged in [hypher](https://github.com/typst/hypher/tree/main/patterns) | Per file header: LPPL, MIT, BSD-3-Clause, Unlicense or a permit-any-use notice (or a choice including one); Slovak (`hyph-sk.tex`, embedded whenever `hypher/slovak` is on: feature `slovak-gpl`, or another dependency through feature unification) is GPL-2.0-or-later; Spanish (`hyph-es.tex` 5.0, Javier Bezos, CervanTeX) is MIT |
| Word-break dictionaries and LSTM models embedded through `icu_segmenter_data` (optional `craft-text` features `thai`, `lao`, `khmer`, `burmese`) | Unicode, Inc. and ICU4X contributors | [ICU4X](https://github.com/unicode-org/icu4x) | Unicode-3.0 |
NOTICE
+4 -3
View File
@@ -21,9 +21,10 @@ crates/cms/tests/data/FOGRA39.txt: FOGRA39 characterisation data, distributed un
imply certification, approval or endorsement by Fogra."
crates/text/assets/hyphen/en-us.dic: public domain, derived from the Moby Hyphenator II list by
Grady Ward (Project Gutenberg eBook #3204).
crates/proof/data/en-core.txt: public domain, about 11,000 words of the Moby Hyphenator II list by
Grady Ward (Project Gutenberg eBook #3204), ranked by frequency in public-domain Project
Gutenberg books.
crates/proof/data/en-words.dic: public domain, the words of the Moby Hyphenator II list by Grady
Ward (Project Gutenberg eBook #3204), without its break points.
crates/proof/data/en-extra.txt: MIT OR Apache-2.0, WordCraft contributors' own list of modern
English words (a copy of craft-i18n's language-data/en-us/en-extra.txt).
crates/text/assets/hyphen/en-us.pat: CC0-1.0, trained from en-us.dic by DesignCraft's trainer.
Hyphenation patterns for other languages, embedded through the hypher crate only when the
matching craft-text feature is enabled (none are by default). They come from hyph-utf8,
crates/proof/README.md
+20 -15
View File
@@ -13,7 +13,7 @@ craft-proof = { git = "https://github.com/storytold/craft-libs", rev = "<commit>
| Language | Spelling | Grammar | Hyphenation |
|---|---|---|---|
| English (en-US, en-GB and the British-spelling Englishes, en-CA) | Moby word list + `en-extra.txt`, dialect variants (`en-variants.txt`); without them the installed Hunspell English, else the built-in core list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
| English (en-US, en-GB and the British-spelling Englishes, en-CA) | Moby word list + `en-extra.txt`, dialect variants (`en-variants.txt`); without them the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic |
| Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns |
| Croatian, Turkish, Russian, Czech, Indonesian, Spanish (es, es-MX, es-419, es-UY) | the installed Hunspell dictionary (Russian also `ru-extra.txt`) | yes | Turkish and Russian rules; English fallback otherwise |
| German | — | — | TeX patterns |
@@ -55,8 +55,9 @@ The word lists and patterns live in [craft-i18n](https://github.com/storytold/cr
grammar and the Hunspell languages work as before.
- `CRAFT_I18N_REQUIRED=1` (releases) makes a missing directory or file an error.
Tests that pin results of the data are ignored without it; `src/no_data.rs` runs every public
entry point with no data embedded.
Tests that pin results of the data run only when every file is embedded (`craft_proof_data`);
`src/no_data.rs` runs every public entry point when none is (`craft_proof_no_data`). A partial
data folder runs neither set.
### English is always checked
@@ -64,23 +65,27 @@ English spelling uses the first of these that is there; `english_source(lang)` s
1. **Embedded:** the Moby word list from `CRAFT_I18N_DIR` (above). Nothing below is used then.
2. **System Hunspell:** an English `.aff`/`.dic` pair in the dictionaries folder or the system
locations, the run's English first (`en_US`, `en_GB`, `en_CA`…), then another English. None on
the web.
3. **Core list:** `data/en-core.txt`, about 11,000 common words built into the crate (inflections
come from the same rules as the Moby list's). It marks only near misses of common words: two
letters swapped (`teh`, `recieved`), or one letter added, missing or changed in a lower-case
word of four letters or more (`definately`). Other words it doesn't know (rare words, names,
terms) are taken as right.
locations, the run's English first (`en_US`, `en_GB`, `en_CA`…), then another English: en-GB
and en-CA use an `en_US` dictionary when no regional one is installed. A file that can't be
read is skipped for the next name or folder; one file is read once for every English that uses
it. Compounding (`COMPOUNDMIN`, `COMPOUNDRULE`…, used by English dictionaries only for
ordinals) and `OCONV` are ignored for English rather than refused. None on the web.
3. **Built-in list:** `data/en-words.dic`, the whole Moby word list (about 165,000 words, 344 KB
compressed, words only) built into the crate, with `data/en-extra.txt` (WordCraft's supplement
of modern words, a copy of craft-i18n's) and the same inflection rules as the embedded list. A
word it doesn't know is misspelled. Suggestions are the words one edit away (swap, missing,
added or changed letter), looked up in the list, for words of up to 32 letters.
A dictionary that is there but can't be read (encoding, size, parse error, no words) is skipped,
and `dictionary_load_error(lang)` gives the reason; it never panics. Text in other languages is
never checked against English: a language without its dictionary has no spelling marks.
The core list is public domain. Every word is from the Moby Hyphenator II list by Grady Ward
(Project Gutenberg eBook #3204, as in craft-i18n's `language-data/en-us/en-us.dic`); 50
public-domain Project Gutenberg books only rank them by how often they are written in lower case.
No other word list or dictionary was used. Regenerate with
`cargo run -p craft-proof --example en_core -- <en-us.dic> 12000 crates/proof/data/en-core.txt <books…>`.
`data/en-words.dic` is public domain: the Moby Hyphenator II list by Grady Ward (Project Gutenberg
eBook #3204), from craft-i18n's `language-data/en-us/en-us.dic` with its break points dropped.
Regenerate with
`cargo run -p craft-proof --example en_words -- <craft-i18n>/language-data/en-us/en-us.dic crates/proof/data/en-words.dic`.
`data/en-extra.txt` is WordCraft contributors' own list (MIT OR Apache-2.0); keep it in step with
craft-i18n's copy.
## Check
crates/proof/data/en-core.txt
-11372
View File
File diff suppressed because it is too large Load Diff
crates/proof/data/en-extra.txt
+241
View File
@@ -0,0 +1,241 @@
# English spelling supplement: common words the public-domain Moby list (1990s) lacks.
# Written by WordCraft contributors (MIT OR Apache-2.0); see ATTRIBUTION.md.
# One lowercase base form per line, sorted. Plurals, -ed, -ing, -er, -ly… come from the
# inflection rules in crates/proof/src/lib.rs, so list "desktop", not "desktops".
airbag
airfare
artcraft
autocomplete
autocorrect
autofill
autoformat
autosave
backend
bitcoin
bitmap
blockchain
blog
blogger
bluetooth
captcha
caregiver
carpool
cellphone
chatbot
chatroom
checkbox
checklist
childcare
clamour
clickable
colourful
colourisation
colourise
colourless
configurable
copyedit
coronavirus
covid
crowdfunding
crowdsource
crypto
cryptocurrency
customizable
customization
cyber
cybersecurity
cyberspace
database
dataset
daycare
desktop
download
downloadable
downsize
dropdown
earbud
ebook
ecommerce
email
emoji
emoticon
endeavour
esports
facebook
favicon
fintech
firewall
flowchart
foodie
freeware
frontend
fundraiser
gameplay
gamification
geeky
geolocation
gigabit
gigabyte
github
glamour
globalization
google
groupthink
hackathon
hardcode
hashtag
healthcare
homepage
homeschool
honour
honourable
honourably
hotspot
hyperlink
hypertext
inbox
incentivize
infographic
instagram
internet
interoperability
intranet
ipad
iphone
javascript
kickstart
kilobyte
laptop
lifestyle
linkedin
linux
livestream
lockdown
login
logout
lowercase
makeover
malware
manoeuvrable
megabyte
megapixel
meme
metadata
microchip
microsoft
middleware
mindset
multicultural
multiplayer
multitask
nerdy
netbook
netflix
newsfeed
offboard
offline
onboard
online
onscreen
outsource
paediatric
paralegal
passcode
petabyte
phishing
photobomb
playdate
playlist
plugin
podcast
popup
prequel
printout
prioritise
prioritization
prioritize
privatise
ransomware
recyclable
rigour
ringtone
roadmap
rollout
runtime
scalability
screencast
screensaver
screenshot
scrollbar
selfie
signup
sitemap
skillset
smartphone
smartwatch
smiley
snowboard
spellcheck
spellchecker
spreadsheet
spyware
stakeholder
standup
startup
subfolder
sunscreen
superfood
superuser
synthesise
tagline
takeout
techie
telecommute
teleconference
telehealth
telemedicine
terabyte
textbox
timeline
timestamp
toolbar
toolkit
touchpad
touchscreen
trackball
trackpad
travelogue
upcycle
upload
uppercase
upsell
upskill
usability
username
videoconference
videogame
vlog
vlogger
voicemail
walkthrough
webcam
webcast
webinar
weblog
webmaster
webpage
website
whatsapp
whiteboard
wifi
wiki
wikipedia
wireframe
wordcraft
workaround
workflow
workforce
workplace
workspace
workstation
youtube
crates/proof/data/en-words.dic
BIN
View File
Binary file not shown.
crates/proof/examples/en_core.rs
-70
View File
@@ -1,70 +0,0 @@
//! Regenerate `data/en-core.txt`, the English core word list (see the crate README).
//!
//! ```sh
//! cargo run -p craft-proof --example en_core -- <craft-i18n>/language-data/en-us/en-us.dic <count> <out> <book.txt>...
//! ```
//!
//! Every word in the list comes from the public-domain Moby Hyphenator II list (`en-us.dic`).
//! The books (public-domain Project Gutenberg plain-text files) only rank those words: a word is
//! counted where it is written in lower case (so names, which are capitalized, rank low), must
//! appear in at least [`MIN_BOOKS`] of the books, and the `<count>` most frequent are written,
//! sorted. No other word list or dictionary is read.
//!
//! `data/en-core.txt` was made with `<count>` 12000 (11,367 words pass [`MIN_BOOKS`]) from these
//! Project Gutenberg eBooks (`https://www.gutenberg.org/cache/epub/<n>/pg<n>.txt`, fetched
//! 2026-11-01): 11 16 35 36 43 45 55 74 76 84 98 105 113 120 135 141 158 161 174 215 219 244 345
//! 408 514 730 766 768 1023 1080 1184 1228 1260 1342 1400 1497 1661 1952 2542 2554 2600 2701 2814
//! 2852 3600 3825 4300 5230 25344 64317.
use std::collections::{HashMap, HashSet};
/// A word must occur, in lower case, in at least this many books.
const MIN_BOOKS: usize = 8;
fn main() -> Result<(), String> {
let args: Vec<String> = std::env::args().skip(1).collect();
let [dic, count, out, books @ ..] = args.as_slice() else {
return Err("usage: en_core <en-us.dic> <count> <out> <book.txt>...".into());
};
let count: usize = count.parse().map_err(|e| format!("count: {e}"))?;
let bytes = std::fs::read(dic).map_err(|e| format!("{dic}: {e}"))?;
let moby = craft_proof::dict::Dictionary::from_bytes(&bytes)?;
let moby: HashSet<&str> = moby.iter().map(|(w, _)| w).collect();
let mut freq: HashMap<String, (u64, usize)> = HashMap::new();
for book in books {
let text = std::fs::read_to_string(book).map_err(|e| format!("{book}: {e}"))?;
// The Project Gutenberg header and licence are not the book.
let body = text.split_once("*** START OF").map_or(text.as_str(), |(_, b)| b);
let body = body.split_once("*** END OF").map_or(body, |(b, _)| b);
let mut seen: HashSet<&str> = HashSet::new();
for w in body.split(|c: char| !(c.is_ascii_alphabetic() || c == '\'')).map(|w| w.trim_matches('\'')) {
if w.len() < 2 || !w.bytes().all(|b| b.is_ascii_lowercase() || b == b'\'') || !moby.contains(w) {
continue;
}
let e = freq.entry(w.to_string()).or_default();
e.0 += 1;
if seen.insert(w) {
e.1 += 1;
}
}
}
let mut ranked: Vec<(String, u64)> = freq.into_iter().filter(|(_, (_, b))| *b >= MIN_BOOKS).map(|(w, (n, _))| (w, n)).collect();
ranked.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)));
ranked.truncate(count);
let mut words: Vec<String> = ranked.into_iter().map(|(w, _)| w).collect();
words.sort();
let mut o = String::from(
"# English core word list: the built-in fallback when neither the embedded Language data nor a\n\
# system Hunspell English dictionary is available (see crates/proof/README.md).\n\
# Public domain: every word is from the Moby Hyphenator II list by Grady Ward (Project\n\
# Gutenberg eBook #3204), ranked by frequency in public-domain Project Gutenberg books.\n\
# Generated by `cargo run -p craft-proof --example en_core`. Do not edit by hand.\n",
);
for w in &words {
o.push_str(w);
o.push('\n');
}
std::fs::write(out, o).map_err(|e| format!("{out}: {e}"))?;
eprintln!("{} words", words.len());
Ok(())
}
crates/proof/examples/en_words.rs
+28
View File
@@ -0,0 +1,28 @@
//! Regenerate `data/en-words.dic`, the English word list built into the crate (see the crate
//! README): every word of the public-domain Moby Hyphenator II list, without its break points.
//!
//! ```sh
//! cargo run -p craft-proof --example en_words -- <craft-i18n>/language-data/en-us/en-us.dic data/en-words.dic
//! ```
//!
//! The input is craft-i18n's `en-us.dic`: the Moby Hyphenator II list by Grady Ward (Project
//! Gutenberg eBook #3204, `mhyph.txt`), already stripped of the Project Gutenberg header and
//! licence, with its break points. The output keeps the words only (in [`Dictionary`]'s format,
//! every break mask zero), so it is smaller and carries no hyphenation. No other word list or
//! dictionary is read.
use craft_proof::dict::Dictionary;
fn main() -> Result<(), String> {
let args: Vec<String> = std::env::args().skip(1).collect();
let [dic, out] = args.as_slice() else {
return Err("usage: en_words <en-us.dic> <out>".into());
};
let bytes = std::fs::read(dic).map_err(|e| format!("{dic}: {e}"))?;
let moby = Dictionary::from_bytes(&bytes)?;
let words = Dictionary::from_entries(moby.iter().map(|(w, _)| (w.to_string(), 0)).collect());
let data = words.to_bytes();
std::fs::write(out, &data).map_err(|e| format!("{out}: {e}"))?;
eprintln!("{} words, {} bytes", words.len(), data.len());
Ok(())
}
crates/proof/src/affix.rs
+67 -4
View File
@@ -19,7 +19,9 @@
//! read as `fi`) and ignored characters (`IGNORE`), and the suggestion tables `TRY`, `KEY`, `MAP`
//! and `REP`.
//! Morphological data (`AM` aliases, the fields after a stem) is skipped. Rejected: compounding,
//! long or numeric flags, `NEEDAFFIX` and the other directives that change which words are correct.
//! long or numeric flags, `NEEDAFFIX` and the other directives that change which words are correct
//! (an English dictionary's compounding and `OCONV` are ignored instead:
//! [`AffixDict::from_hunspell_english`]).
//!
//! Lookups never panic: stems sit in one string, found through an open-addressing hash table, and
//! a probe allocates only small scratch strings for the candidate roots it tries.
@@ -339,6 +341,8 @@ struct Rules {
substandard: Option<char>,
/// The `KEEPCASE` flag.
keepcase: Option<char>,
/// The `ONLYINCOMPOUND` flag, read for English only ([`AffixDict::from_hunspell_english`]).
only_in_compound: Option<char>,
iconv: Vec<(String, String)>,
ignore: Vec<char>,
/// `AF` flag aliases: alias `n` (1-based) stands for `aliases[n - 1]`.
@@ -422,8 +426,27 @@ fn zero_empty(s: &str) -> String {
/// but it is not a flag of its own: it is dropped wherever flags are read.
const EMOJI_SELECTOR: char = '\u{fe0f}';
/// Read Hunspell `.aff` text (already decoded to UTF-8).
fn parse_aff(aff: &str) -> Result<Rules, String> {
/// Directives an English dictionary may carry that don't change which English words are correct
/// here, ignored by [`AffixDict::from_hunspell_english`]: compounding (English dictionaries use
/// it only for ordinals such as `1st`, which have digits and are skipped by the spelling check)
/// and output conversion (`OCONV`: how suggestions are written, such as `'` as `’`).
const IGNORED_IN_ENGLISH: &[&str] = &[
"COMPOUNDBEGIN",
"COMPOUNDEND",
"COMPOUNDFIRST",
"COMPOUNDFLAG",
"COMPOUNDLAST",
"COMPOUNDMIDDLE",
"COMPOUNDMIN",
"COMPOUNDPERMITFLAG",
"COMPOUNDRULE",
"OCONV",
];
/// Read Hunspell `.aff` text (already decoded to UTF-8). `english`: ignore the directives in
/// [`IGNORED_IN_ENGLISH`], and keep the `ONLYINCOMPOUND` flag so the stems that are only parts of
/// compounds can be left out.
fn parse_aff(aff: &str, english: bool) -> Result<Rules, String> {
let mut r = Rules::default();
let mut cross: HashMap<(bool, char), bool> = HashMap::new();
let mut map_header = false;
@@ -518,6 +541,8 @@ fn parse_aff(aff: &str) -> Result<Rules, String> {
r.prefixes.push(rule);
}
}
"ONLYINCOMPOUND" if english => r.only_in_compound = Some(one_flag(at(1), dir, n)?),
d if english && IGNORED_IN_ENGLISH.contains(&d) => {}
d if UNSUPPORTED.contains(&d) => return Err(format!("line {}: {d} is not supported", n + 1)),
_ => {}
}
@@ -542,11 +567,23 @@ impl AffixDict {
/// Build from Hunspell `.aff` and `.dic` text (decoded to UTF-8; the `.dic` count line and
/// morphological fields are ignored). `comments` (provenance, licence) are kept with the data.
pub fn from_hunspell(aff: &str, dic: &str, comments: &[&str]) -> Result<AffixDict, String> {
AffixDict::from_hunspell_in(aff, dic, comments, false)
}
/// [`AffixDict::from_hunspell`] for an English dictionary: compounding and output conversion
/// (`COMPOUNDMIN`, `COMPOUNDRULE`, `OCONV`…), which other languages need but English spelling
/// doesn't, are ignored rather than refused, and the stems marked `ONLYINCOMPOUND` (parts of
/// compounds only) are left out.
pub fn from_hunspell_english(aff: &str, dic: &str, comments: &[&str]) -> Result<AffixDict, String> {
AffixDict::from_hunspell_in(aff, dic, comments, true)
}
fn from_hunspell_in(aff: &str, dic: &str, comments: &[&str], english: bool) -> Result<AffixDict, String> {
// Some dictionaries open with a UTF-8 BOM; it must not glue itself to the first
// directive or turn the `.dic` count line into a stem.
let aff = aff.strip_prefix('\u{feff}').unwrap_or(aff);
let dic = dic.strip_prefix('\u{feff}').unwrap_or(dic);
let rules = parse_aff(aff)?;
let rules = parse_aff(aff, english)?;
let mut stems: Vec<(String, String)> = Vec::new();
for (n, line) in dic.lines().enumerate() {
// The entry may hold spaces (`ad hoc/B\tpo:adv`); it ends where Hunspell ends it.
@@ -582,6 +619,9 @@ impl AffixDict {
continue;
}
let flags = rules.flags(&flags).to_string();
if rules.only_in_compound.is_some_and(|f| flags.contains(f)) {
continue;
}
stems.push((word, flags));
}
AffixDict::build(rules, stems, comments.iter().map(ToString::to_string).collect())
@@ -1756,6 +1796,29 @@ pub(crate) mod tests {
}
}
/// craft-libs#482 review: an English dictionary with compounding (for ordinals) and output
/// conversion, as the usual en_US and en_GB ones have, loads for English; its `ONLYINCOMPOUND`
/// stems are left out. Other languages still refuse those directives. (A synthetic `.aff`
/// written for this test.)
#[test]
fn english_dictionaries_ignore_compounding_and_output_conversion() {
let aff = "SET UTF-8\nTRY esianrtolcdugmphbyfvkwz'\nICONV 1\nICONV ’ '\nOCONV 1\nOCONV ' ’\nNOSUGGEST !\n\
COMPOUNDMIN 1\nONLYINCOMPOUND c\nCOMPOUNDRULE 2\nCOMPOUNDRULE n*1t\nCOMPOUNDRULE n*mp\nCOMPOUNDFLAG Z\n\
WORDCHARS 0123456789\nSFX S Y 1\nSFX S 0 s [^sxzhy]\nSFX D Y 1\nSFX D 0 ed [^ey]\n";
let dic = "5\nhouse/S\nwalk/SD\n0th/pc\n1st/c\nthe\n";
assert!(AffixDict::from_hunspell(aff, dic, &[]).is_err(), "refused for other languages");
let d = AffixDict::from_hunspell_english(aff, dic, &[]).unwrap();
for w in ["house", "houses", "walked", "walks", "the"] {
assert!(d.check(w), "{w}");
}
assert!(!d.check("0th") && !d.check("1st"), "parts of compounds only");
assert!(!d.check("housed") && !d.check("hous"));
// The directives that change which words are right are still refused.
for dir in ["NEEDAFFIX", "CIRCUMFIX", "PSEUDOROOT", "FULLSTRIP"] {
assert!(AffixDict::from_hunspell_english(&format!("{dir} X\n"), dic, &[]).is_err(), "{dir}");
}
}
#[test]
fn unsupported_and_broken_input_is_an_error() {
assert!(AffixDict::from_hunspell("FLAG long\n", DIC, &[]).is_err());
crates/proof/src/english.rs
+271 -125
View File
@@ -5,27 +5,43 @@
//! 2. **System Hunspell**: an English Hunspell `.aff`/`.dic` pair found by [`crate::hunspell`]
//! (the dictionaries folder first, then the system locations; none on the web). The run's
//! English is tried first (`en_US`, `en_GB`, `en_CA`…), then the other Englishes: a full
//! dictionary in the other spelling marks fewer right words than the core list.
//! 3. **Core list**: [`CORE_LIST`], about 11,000 common words compiled into the crate (with the
//! usual inflections). A word it doesn't know is marked only when it is a near miss of a
//! common word (`teh`, `recieve`, `definately`); any other word it doesn't know (a rare word, a
//! name, a term) is taken as correct, as it may well be right.
//! dictionary in the other spelling marks fewer right words than none. So British and Canadian
//! English use an `en_US` dictionary when no regional one is installed (and American English
//! an `en_GB` one). A file that can't be read is skipped for the next name or folder, and one
//! file is read once for every English that uses it. Directives that only matter for other
//! languages (compounding, output conversion) are ignored ([`crate::hunspell::load_english`]).
//! 3. **Built-in word list**: [`WORDS`], the whole public-domain Moby word list compiled into
//! the crate (without its break points), with WordCraft's own supplement of modern words
//! ([`EXTRA`]) and the usual inflections. A word it doesn't know is misspelled, as with the
//! embedded list.
//!
//! A Hunspell dictionary that is there but can't be read (unsupported encoding, too large, a
//! parse error, no words) is skipped for the core list, and its reason is reported by
//! parse error, no words) is skipped, and its reason is reported by
//! [`crate::dictionary_load_error`]. Non-English text is never checked against any of these.
use std::path::PathBuf;
use std::sync::{Arc, OnceLock, RwLock};
use std::path::{Path, PathBuf};
use std::sync::{Arc, OnceLock, PoisonError, RwLock};
use crate::affix::AffixDict;
use crate::dict::Dictionary;
use crate::{Options, ProofLang};
/// The English core word list (`data/en-core.txt`): the most common words of the public-domain
/// Moby Hyphenator II list by Grady Ward, ranked by their frequency in public-domain Project
/// Gutenberg books (generated by `examples/en_core.rs`). One lowercase word per line, `#`
/// comments.
const CORE_LIST: &str = include_str!("../data/en-core.txt");
/// The English word list built into the crate (`data/en-words.dic`, [`Dictionary`]'s format with
/// no break points): every word of the public-domain Moby Hyphenator II list by Grady Ward
/// (generated by `examples/en_words.rs` from craft-i18n's `en-us.dic`).
const WORDS: &[u8] = include_bytes!("../data/en-words.dic");
/// WordCraft's supplement of common modern words the Moby list lacks (`data/en-extra.txt`, a copy
/// of craft-i18n's `language-data/en-us/en-extra.txt`, MIT OR Apache-2.0): one lowercase base
/// form per line, `#` comments. Used when the Language data doesn't embed its own copy.
pub(crate) const EXTRA: &str = include_str!("../data/en-extra.txt");
/// Longest word (in characters) suggestions are looked for: longer ones get none, which keeps
/// the edits tried per word bounded.
const MAX_SUGGEST_CHARS: usize = 32;
/// Most Hunspell `.aff`/`.dic` pairs tried for one English before giving up on Hunspell.
const MAX_CANDIDATES: usize = 8;
/// Where English spelling comes from now ([`crate::english_source`]).
#[derive(Clone, Debug, PartialEq, Eq)]
@@ -34,19 +50,25 @@ pub enum EnglishSource {
Embedded,
/// An English Hunspell dictionary from the dictionaries folder or the system: its `.dic` file.
Hunspell(PathBuf),
/// The core word list built into the crate: only near misses of common words are marked.
/// The Moby word list built into the crate: a word it doesn't know is misspelled.
CoreList,
}
/// The words of [`CORE_LIST`].
fn core() -> &'static std::collections::HashSet<&'static str> {
static C: OnceLock<std::collections::HashSet<&'static str>> = OnceLock::new();
C.get_or_init(|| CORE_LIST.lines().map(str::trim).filter(|l| !l.is_empty() && !l.starts_with('#')).collect())
/// The words of [`WORDS`] (decoded on first use; empty if they fail to decode, which a test rules
/// out).
fn words() -> &'static Dictionary {
static W: OnceLock<Dictionary> = OnceLock::new();
W.get_or_init(|| {
Dictionary::from_bytes(WORDS).unwrap_or_else(|e| {
log::error!("built-in English word list: {e}");
Dictionary::default()
})
})
}
/// Is `w` (lower case) a word of the core list?
pub(crate) fn core_contains(w: &str) -> bool {
core().contains(w)
/// Is `w` (lower case) a word of the built-in list?
pub(crate) fn word_list_contains(w: &str) -> bool {
words().get(w).is_some()
}
/// Hunspell file names to look for, the run's English first.
@@ -58,7 +80,7 @@ fn names(lang: ProofLang) -> &'static [&'static str] {
}
}
/// The slot of each English in [`HUNSPELL`].
/// The slot of each English in [`State::slots`].
fn slot(lang: ProofLang) -> usize {
match lang {
ProofLang::EnGb => 1,
@@ -70,8 +92,23 @@ fn slot(lang: ProofLang) -> usize {
/// A loaded Hunspell dictionary and its `.dic` file; `None` when none could be loaded.
type Loaded = Option<(Arc<AffixDict>, PathBuf)>;
/// Per English (see [`slot`]): `None` until first use, then what was found.
static HUNSPELL: RwLock<[Option<Loaded>; 3]> = RwLock::new([None, None, None]);
/// The English Hunspell dictionaries, under one lock.
struct State {
/// Per English (see [`slot`]): `None` until first use, then what was found.
slots: [Option<Loaded>; 3],
/// Every `.dic` file read, so Englishes that use the same file share one copy.
files: Vec<(PathBuf, Arc<AffixDict>)>,
/// Counts reloads, so a load that started before one is not kept.
reloads: u64,
}
impl State {
const fn new() -> Self {
State { slots: [None, None, None], files: Vec::new(), reloads: 0 }
}
}
static HUNSPELL: RwLock<State> = RwLock::new(State::new());
/// Why the embedded word list failed to decode, if it did.
static EMBEDDED_ERROR: OnceLock<String> = OnceLock::new();
@@ -79,51 +116,117 @@ static EMBEDDED_ERROR: OnceLock<String> = OnceLock::new();
/// Remember that the embedded word list is there but failed to decode (English then uses the
/// next source).
pub(crate) fn note_embedded_error(e: String) {
log::error!("embedded English word list: {e}; English spelling falls back to Hunspell or the core word list");
log::error!("embedded English word list: {e}; English spelling falls back to Hunspell or the built-in word list");
let _ = EMBEDDED_ERROR.set(e);
}
/// The English Hunspell dictionary for `lang` (read on first use).
fn hunspell(lang: ProofLang) -> Loaded {
let i = slot(lang);
if let Some(Some(found)) = HUNSPELL.read().unwrap_or_else(std::sync::PoisonError::into_inner).get(i) {
return found.clone();
let reloads = {
let s = HUNSPELL.read().unwrap_or_else(PoisonError::into_inner);
if let Some(Some(found)) = s.slots.get(i) {
return found.clone();
}
s.reloads
};
keep(&HUNSPELL, i, reloads, load(lang))
}
/// Keep in `state` what the load of slot `i`, started after `reloads` reloads, found. If the
/// folder was reloaded since, the load may have read the old folder: it serves this one check and
/// the next one loads again.
fn keep(state: &RwLock<State>, i: usize, reloads: u64, loaded: Loaded) -> Loaded {
let mut s = state.write().unwrap_or_else(PoisonError::into_inner);
if s.reloads != reloads {
return match s.slots.get(i) {
Some(Some(found)) => found.clone(),
_ => loaded,
};
}
let loaded = load(lang);
let mut slots = HUNSPELL.write().unwrap_or_else(std::sync::PoisonError::into_inner);
let loaded = loaded.map(|(d, dic)| {
// Another English may have read the same file meanwhile: share its copy.
let d = match s.files.iter().find(|(p, _)| *p == dic) {
Some((_, shared)) => shared.clone(),
None => {
s.files.push((dic.clone(), d.clone()));
d
}
};
(d, dic)
});
// Another thread may have filled it meanwhile: that one wins.
match slots.get_mut(i) {
Some(s) => s.get_or_insert(loaded).clone(),
match s.slots.get_mut(i) {
Some(slot) => slot.get_or_insert(loaded).clone(),
None => loaded,
}
}
/// The dictionary of the `.dic` file `dic` in `state`, if an English already read it.
fn already_read(state: &RwLock<State>, dic: &Path) -> Option<Arc<AffixDict>> {
state.read().unwrap_or_else(PoisonError::into_inner).files.iter().find(|(p, _)| p == dic).map(|(_, d)| d.clone())
}
/// Forget what `state` holds, so the next check looks again.
fn forget(state: &RwLock<State>) {
let mut s = state.write().unwrap_or_else(PoisonError::into_inner);
s.slots = [None, None, None];
s.files.clear();
s.reloads = s.reloads.wrapping_add(1);
}
/// The key load errors of `lang` are kept under ([`crate::hunspell::load_error`]).
fn key(lang: ProofLang) -> &'static str {
names(lang).first().copied().unwrap_or("en_US")
}
/// Read the first English Hunspell pair for `lang` that loads, trying each name in each folder
/// ([`crate::hunspell::candidates`]). The first failure before it (the file `lang` would rather
/// use) is reported for `lang`.
fn load(lang: ProofLang) -> Loaded {
let key = key(lang);
let Some((aff, dic)) = crate::hunspell::find(names(lang)) else {
let candidates = crate::hunspell::candidates(names(lang));
if candidates.is_empty() {
crate::hunspell::clear_load_error(key);
log::info!("English spelling: no Hunspell English dictionary installed; using the core word list");
log::info!("English spelling: no Hunspell English dictionary installed; using the built-in word list");
return None;
};
let loaded = crate::hunspell::load(&aff, &dic).and_then(|d| if d.is_empty() { Err(format!("{}: no words", dic.display())) } else { Ok(d) });
match crate::hunspell::note_load(key, &dic, loaded) {
Ok(d) => Some((Arc::new(d), dic)),
Err(e) => {
log::warn!("English spelling: {e}; using the core word list");
None
}
let mut first_error: Option<(PathBuf, String)> = None;
let mut found = None;
for (aff, dic) in candidates.into_iter().take(MAX_CANDIDATES) {
let loaded = match already_read(&HUNSPELL, &dic) {
Some(d) => Ok(d),
None => crate::hunspell::load_english(&aff, &dic)
.and_then(|d| if d.is_empty() { Err(format!("{}: no words", dic.display())) } else { Ok(d) })
.map(Arc::new),
};
match loaded {
Ok(d) => {
found = Some((d, dic));
break;
}
Err(e) => {
log::warn!("English spelling: {e}; trying the next dictionary");
first_error.get_or_insert((dic, e));
}
}
}
match first_error {
Some((dic, e)) => {
let _ = crate::hunspell::note_load::<()>(key, &dic, Err(e));
}
None => crate::hunspell::clear_load_error(key),
}
if found.is_none() {
log::warn!("English spelling: no English Hunspell dictionary could be read; using the built-in word list");
}
found
}
/// Forget the loaded dictionaries, so the next check looks for them again (the dictionaries
/// folder changed).
pub(crate) fn reload() {
*HUNSPELL.write().unwrap_or_else(std::sync::PoisonError::into_inner) = [None, None, None];
forget(&HUNSPELL);
}
/// Where the spelling of `lang` (an English) comes from now. Reads the Hunspell dictionary once
@@ -149,7 +252,7 @@ pub(crate) fn load_error(lang: ProofLang) -> Option<String> {
}
/// Is `word` spelled right in `lang` (an English) when no word list is embedded: against the
/// Hunspell dictionary, or else the core list.
/// Hunspell dictionary, or else the built-in word list.
pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options) -> bool {
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’').trim_matches(['\'', '’']);
if crate::accepted_by_options(w, opts) || !w.chars().any(|c| c.is_ascii_alphabetic()) {
@@ -158,7 +261,7 @@ pub(crate) fn is_correct(word: &str, lang: ProofLang, opts: &Options) -> bool {
let w = w.replace('’', "'");
match hunspell(lang) {
Some((d, _)) => hunspell_correct(&d, &w),
None => core_correct(&w),
None => list_correct(&w),
}
}
@@ -168,67 +271,75 @@ fn hunspell_correct(d: &AffixDict, w: &str) -> bool {
known(w) || (w.contains('-') && w.split('-').filter(|p| !p.is_empty()).all(known))
}
/// Against the core list: known words and their inflections are right, and so is any other word
/// that isn't a near miss of a common one ([`near_miss`]).
fn core_correct(w: &str) -> bool {
let capitalized = w.chars().next().is_some_and(char::is_uppercase);
let lower = w.to_lowercase();
let ok = |p: &str| crate::known_in_core_list(p) || !near_miss(p, capitalized);
if lower.contains('-') { crate::known_in_core_list(&lower) || lower.split('-').filter(|p| !p.is_empty()).all(ok) } else { ok(&lower) }
}
/// Is `lower` (a word the core list doesn't know) an obvious misspelling of a common word? Two
/// neighbouring letters swapped (`teh`, `Hte`, `recieved`) always is. Otherwise, for lower-case
/// words of four letters or more, one letter added, missing or changed (`definately`,
/// `ocurred`). Shorter words and capitalized ones (names) are left alone: too many right ones are
/// one letter from a common word.
fn near_miss(lower: &str, capitalized: bool) -> bool {
let target: Vec<char> = lower.chars().collect();
let n = target.len();
if !(3..=40).contains(&n) {
return false;
}
if core().iter().any(|c| is_swap(&target, c)) {
/// Against the built-in word list: its words and their inflections, or each part of a
/// hyphenated word. Anything else is misspelled. (Should the list fail to decode, nothing is
/// marked rather than everything.)
fn list_correct(w: &str) -> bool {
if words().is_empty() {
return true;
}
if capitalized || n < 4 {
return false;
}
core().iter().any(|c| {
let m = c.chars().count();
m + 1 >= n && m <= n + 1 && crate::damerau(&target, c) <= 1
})
}
/// Is `b` `a` with two neighbouring letters swapped?
fn is_swap(a: &[char], b: &str) -> bool {
if b.len() < a.len() || b.chars().count() != a.len() {
return false;
}
let diff: Vec<(usize, char)> = b.chars().enumerate().filter(|&(i, c)| a.get(i) != Some(&c)).collect();
match diff.as_slice() {
[(i, x), (j, y)] => *j == i + 1 && a.get(*i) == Some(y) && a.get(*j) == Some(x),
_ => false,
}
let lower = w.to_lowercase();
crate::known_in_word_list(&lower) || (lower.contains('-') && lower.split('-').filter(|p| !p.is_empty()).all(crate::known_in_word_list))
}
/// Spelling suggestions in `lang` (an English) when no word list is embedded: from the Hunspell
/// dictionary, or else the core list.
/// dictionary, or else the built-in word list.
pub(crate) fn suggest(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
match hunspell(lang) {
Some((d, _)) => crate::cs::suggest::suggest_in(&d, word, max),
None => core_suggest(word, max),
None => list_suggest(word, max),
}
}
/// Suggestions from the core list: by edit distance ([`crate::suggest_from`]), two swapped
/// letters first (`teh` → `the` before `tea`).
fn core_suggest(word: &str, max: usize) -> Vec<String> {
let target: Vec<char> = word.to_lowercase().chars().collect();
let mut out = crate::suggest_from(word, max.saturating_add(4), core().iter().copied());
out.sort_by_key(|s| !is_swap(&target, &s.to_lowercase()));
out.truncate(max);
out
/// Letters an edit may add or put in a word's place.
const ALPHABET: &str = "abcdefghijklmnopqrstuvwxyz'";
/// Suggestions from the built-in word list: the words one edit away (two neighbouring letters
/// swapped, one letter missing, added or changed), looked up in the list rather than compared
/// with every word, so the cost is bounded by the word's length (at most
/// [`MAX_SUGGEST_CHARS`]). Swaps first (`hte` → `the` and `het` before `hate`), then words with the same
/// first letter, then the nearest in length.
fn list_suggest(word: &str, max: usize) -> Vec<String> {
let lower = word.to_lowercase().replace('’', "'");
let target: Vec<char> = lower.chars().collect();
let n = target.len();
if max == 0 || n == 0 || n > MAX_SUGGEST_CHARS {
return Vec::new();
}
let mut found: Vec<(bool, bool, usize, String)> = Vec::new();
let mut tried = std::collections::HashSet::new();
let mut consider = |c: Vec<char>, swap: bool| {
let s: String = c.iter().collect();
if s != lower && tried.insert(s.clone()) && crate::known_in_word_list(&s) {
found.push((!swap, c.first() != target.first(), c.len().abs_diff(n), s));
}
};
for i in 1..n {
let mut c = target.clone();
c.swap(i - 1, i);
consider(c, true);
}
for i in 0..n {
let mut c = target.clone();
c.remove(i);
consider(c, false);
}
for i in 0..=n {
for a in ALPHABET.chars() {
if i < n {
let mut c = target.clone();
if let Some(l) = c.get_mut(i) {
*l = a;
}
consider(c, false);
}
let mut c = target.clone();
c.insert(i, a);
consider(c, false);
}
}
found.sort();
found.into_iter().take(max).map(|(_, _, _, w)| crate::match_case(word, w)).collect()
}
#[cfg(test)]
@@ -236,24 +347,20 @@ mod tests {
use super::*;
#[test]
fn core_list_is_small_sorted_lowercase_and_has_the_common_words() {
assert!(CORE_LIST.len() < 100 * 1024, "{} bytes", CORE_LIST.len());
let words: Vec<&str> = CORE_LIST.lines().filter(|l| !l.starts_with('#')).collect();
assert!(words.len() > 5_000, "{}", words.len());
assert!(words.windows(2).all(|w| w[0] < w[1]), "sorted, no repeats");
assert!(words.iter().all(|w| !w.is_empty() && w.chars().all(|c| c.is_ascii_lowercase() || c == '\'')));
for w in ["the", "house", "because", "received", "definitely", "separate", "occurred", "beautiful", "people", "water"] {
assert!(core_contains(w), "{w}");
}
fn built_in_list_is_the_whole_moby_list() {
let d = words();
assert!(d.len() > 150_000, "{}", d.len());
assert!(d.iter().all(|(w, m)| m == 0 && !w.is_empty() && w.chars().all(|c| !c.is_uppercase())));
assert!(WORDS.len() < 400 * 1024, "{} bytes", WORDS.len());
let extra: Vec<&str> = EXTRA.lines().filter(|l| !l.is_empty() && !l.starts_with('#')).collect();
assert!(extra.len() > 100 && extra.windows(2).all(|p| p[0] < p[1]), "the supplement is sorted");
}
/// The core list marks obvious misspellings and leaves right words alone, inflected ones and
/// words it doesn't know included.
/// The built-in list knows right words, the rare and the modern ones included, and marks
/// every word it doesn't know (craft-libs#482 review: the old core list let any unknown word
/// through unless it was a near miss of a common one, and marked right words that were).
#[test]
fn core_list_flags_near_misses_only() {
for w in ["teh", "Teh", "hte", "recieved", "definately", "seperate", "ocurred", "becuase", "peopel", "beautifull", "watre"] {
assert!(!core_correct(w), "{w} is marked");
}
fn built_in_list_marks_unknown_words_only() {
for w in [
"the",
"The",
@@ -264,31 +371,70 @@ mod tests {
"kindness",
"Received",
"well-known",
// Not in the list, but not a near miss of a common word either: left alone.
"photosynthesis",
"zxqv",
"Nguyen",
"Jarndyce",
"cat",
"fiber",
"theater",
"organize",
"budget",
"phone",
"people's",
// WordCraft's supplement.
"login",
"blog",
"email",
"emails",
] {
assert!(core_correct(w), "{w} is not marked");
assert!(list_correct(w), "{w} is not marked");
}
assert!(!core_correct("well-knwon"), "each part of a compound is checked");
for w in ["recieved", "definately", "seperate", "ocurred", "becuase", "peopel", "beautifull", "zxqv", "Jarndycey"] {
assert!(!list_correct(w), "{w} is marked");
}
assert!(!list_correct("well-knwon"), "each part of a compound is checked");
}
#[test]
fn swaps_are_two_neighbouring_letters() {
let a: Vec<char> = "teh".chars().collect();
assert!(is_swap(&a, "the"));
assert!(!is_swap(&a, "teh") && !is_swap(&a, "ten") && !is_swap(&a, "thee") && !is_swap(&a, "het"));
assert!(!is_swap(&"é".chars().collect::<Vec<_>>(), "ab"));
fn built_in_list_suggests_one_edit_away() {
assert_eq!(list_suggest("becuase", 3).first().map(String::as_str), Some("because"));
assert_eq!(list_suggest("Becuase", 3).first().map(String::as_str), Some("Because"));
assert!(list_suggest("hte", 3).contains(&"the".to_string()), "{:?}", list_suggest("hte", 3));
assert!(list_suggest("definately", 5).contains(&"definitely".to_string()));
assert!(list_suggest("recieve", 5).contains(&"receive".to_string()));
assert!(list_suggest("", 5).is_empty() && list_suggest("becuase", 0).is_empty());
assert!(list_suggest(&"a".repeat(MAX_SUGGEST_CHARS + 1), 5).is_empty());
}
/// Suggestions look words up rather than scanning the list: the longest word is still quick.
#[test]
fn core_list_suggests_the_common_word() {
assert_eq!(core_suggest("teh", 3).first().map(String::as_str), Some("the"));
assert_eq!(core_suggest("Teh", 3).first().map(String::as_str), Some("The"));
assert!(core_suggest("definately", 5).contains(&"definitely".to_string()));
assert!(core_suggest("", 5).is_empty() && core_suggest("teh", 0).is_empty());
fn built_in_list_suggestions_are_bounded() {
let _ = words();
let word: String = "qwertyuiopasdfghjklzxcvbnm".chars().cycle().take(MAX_SUGGEST_CHARS).collect();
let start = std::time::Instant::now();
for _ in 0..10 {
let _ = list_suggest(&word, 5);
}
assert!(start.elapsed() < std::time::Duration::from_secs(5), "{:?}", start.elapsed());
}
/// craft-libs#482 review: a load that started before a reload (the dictionaries folder
/// changed) serves its own check but isn't kept, so the next check reads the new folder; a
/// load after it is kept, and shared with the Englishes that read the same file.
#[test]
fn a_load_from_before_a_reload_is_not_kept() {
let state = RwLock::new(State::new());
let dict = || Arc::new(AffixDict::from_hunspell("", "1\nzzyzx\n", &[]).unwrap());
let file = PathBuf::from("/old/folder/en_US.dic");
let before = state.read().unwrap().reloads;
forget(&state); // The folder changes while the load runs.
let got = keep(&state, slot(ProofLang::En), before, Some((dict(), file.clone())));
assert_eq!(got.map(|(_, p)| p), Some(file.clone()), "the stale load still answers its own check");
assert!(state.read().unwrap().slots.iter().all(Option::is_none), "but it isn't kept");
assert!(already_read(&state, &file).is_none());
let now = state.read().unwrap().reloads;
let a = keep(&state, slot(ProofLang::En), now, Some((dict(), file.clone()))).unwrap();
let b = keep(&state, slot(ProofLang::EnGb), now, Some((dict(), file.clone()))).unwrap();
assert!(Arc::ptr_eq(&a.0, &b.0), "one copy of a file for every English that reads it");
assert!(already_read(&state, &file).is_some());
forget(&state);
assert!(already_read(&state, &file).is_none() && state.read().unwrap().slots.iter().all(Option::is_none));
}
}
crates/proof/src/hunspell.rs
+27
View File
@@ -145,6 +145,24 @@ pub fn find(names: &[&str]) -> Option<(PathBuf, PathBuf)> {
search_dirs(lo.as_deref()).into_iter().find_map(|dir| find_in(&dir, names))
}
/// Every `<name>.aff`/`<name>.dic` pair for `names`, in the order [`find`] prefers them: folder
/// by folder ([`search_dirs`]), each folder's in the order of `names`. A caller that can't read
/// one tries the next.
pub fn candidates(names: &[&str]) -> Vec<(PathBuf, PathBuf)> {
let lo = names.first().copied().map(lo_ext_dirs);
let mut out: Vec<(PathBuf, PathBuf)> = Vec::new();
for dir in search_dirs(lo.as_deref()) {
for n in names {
let aff = dir.join(format!("{n}.aff"));
let dic = dir.join(format!("{n}.dic"));
if aff.is_file() && dic.is_file() && !out.iter().any(|(_, d)| *d == dic) {
out.push((aff, dic));
}
}
}
out
}
/// The first `<name>.aff`/`<name>.dic` pair for any of `names` in `dir`.
pub fn find_in(dir: &Path, names: &[&str]) -> Option<(PathBuf, PathBuf)> {
names.iter().find_map(|n| {
@@ -183,6 +201,15 @@ pub fn load(aff: &Path, dic: &Path) -> Result<AffixDict, String> {
AffixDict::from_hunspell(&aff_text, &dic_text, &[source.as_str()])
}
/// [`load`] for an English dictionary: directives that only matter for other languages
/// (compounding, output conversion) are ignored rather than refused
/// ([`AffixDict::from_hunspell_english`]).
pub fn load_english(aff: &Path, dic: &Path) -> Result<AffixDict, String> {
let (aff_text, dic_text) = decode(&read_capped(aff)?, &read_capped(dic)?)?;
let source = format!("Hunspell dictionary {}", dic.display());
AffixDict::from_hunspell_english(&aff_text, &dic_text, &[source.as_str()])
}
/// ISO 8859-1: every byte is the code point of the same value (the original VERO pt_BR uses it).
fn decode_latin1(bytes: &[u8]) -> String {
bytes.iter().map(|&b| char::from(b)).collect()
crates/proof/src/lib.rs
+65 -26
View File
@@ -51,8 +51,8 @@
//! **Language data** (the Moby word list, the pattern files, our word supplements) is embedded at
//! build time from craft-i18n's `language-data/` (`CRAFT_I18N_DIR`, see `build.rs`). English
//! spelling never goes unchecked: without the embedded word list it uses an English Hunspell
//! dictionary installed on the system, and without one the core word list built into the crate
//! (see [`english_source`]). Hyphenation that needs the data is off without it; grammar and the
//! dictionary installed on the system, and without one the whole Moby word list built into the
//! crate (see [`english_source`]). Hyphenation that needs the data is off without it; grammar and the
//! rule-based hyphenation (Bulgarian, Turkish, Russian) are code and still work.
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
@@ -326,8 +326,9 @@ pub fn dictionary_load_error(lang: ProofLang) -> Option<String> {
/// Where the spelling of an English `lang` comes from now (`None` for other languages): the word
/// list embedded at build time, else an English Hunspell dictionary from the dictionaries folder
/// or the system (the run's English first), else the core word list built into the crate, which
/// marks only near misses of common words. Reads the Hunspell dictionary once when there is no
/// or the system (the run's English first), else the Moby word list built into the crate.
/// British and Canadian English use an `en_US` Hunspell dictionary when no regional one is
/// installed. Reads the Hunspell dictionary once when there is no
/// embedded word list (it is cached afterwards).
pub fn english_source(lang: ProofLang) -> Option<EnglishSource> {
lang.is_english().then(|| english::source(lang))
@@ -464,9 +465,11 @@ const COMMON: &str = "a an the and or but if of to in on at by for with from as
/// lowercase base form per line, `#` comments. Inflections come from [`known_core`].
use data::EXTRA_EN;
/// The supplement's words: the Language data's copy, else the crate's own ([`english::EXTRA`]).
fn extra_words() -> &'static HashSet<&'static str> {
static E: OnceLock<HashSet<&str>> = OnceLock::new();
E.get_or_init(|| EXTRA_EN.lines().map(str::trim).filter(|l| !l.is_empty() && !l.starts_with('#')).collect())
let extra = if EXTRA_EN.is_empty() { english::EXTRA } else { EXTRA_EN };
E.get_or_init(|| extra.lines().map(str::trim).filter(|l| !l.is_empty() && !l.starts_with('#')).collect())
}
/// A base form in the dictionary or the supplement.
@@ -479,9 +482,9 @@ fn known_core(w: &str) -> bool {
known_with(w, &|s| base_known(d, s))
}
/// [`known_core`] against the core word list built into the crate (no embedded word list).
pub(crate) fn known_in_core_list(w: &str) -> bool {
known_with(w, &|s| english::core_contains(s) || extra_words().contains(s))
/// [`known_core`] against the word list built into the crate (no embedded word list).
pub(crate) fn known_in_word_list(w: &str) -> bool {
known_with(w, &|s| english::word_list_contains(s) || extra_words().contains(s))
}
/// Is `w` (lower case) a word, by `base_known` (a base form in the word list), the common words,
@@ -608,7 +611,7 @@ pub(crate) fn accepted_by_options(word: &str, opts: &Options) -> bool {
}
/// Is the English word list embedded? Without it English is checked against an installed
/// Hunspell dictionary or the core word list ([`english`]).
/// Hunspell dictionary or the word list built into the crate ([`english`]).
pub(crate) fn english_data() -> bool {
!dict::Dictionary::en_us().is_empty()
}
@@ -1196,7 +1199,7 @@ mod tests {
}
/// Remove the English files of `dir` and set it again: English goes back to what it was
/// (the core list, or a dictionary installed on the system), for the tests that follow.
/// (the built-in list, or a dictionary installed on the system), for the tests that follow.
fn drop_english_files(dir: &std::path::Path, english: &[(&str, &[u8], &[u8])]) {
for (name, _, _) in english {
let _ = std::fs::remove_file(dir.join(format!("{name}.aff")));
@@ -1206,18 +1209,18 @@ mod tests {
}
/// R6.4, English never goes unchecked: with no embedded word list, an English Hunspell
/// dictionary (here in the dictionaries folder) is used before the core word list built into
/// the crate; with one, the embedded list stays in use whatever is installed.
/// dictionary (here in the dictionaries folder) is used before the word list built into the
/// crate; with one, the embedded list stays in use whatever is installed.
#[test]
fn english_hunspell_dictionary_is_used_before_the_core_list() {
fn english_hunspell_dictionary_is_used_before_the_built_in_list() {
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = std::env::temp_dir().join(format!("wc-proof-en-hunspell-{}", std::process::id()));
// A tiny en_US. `teh` is a word in it, so a right `teh` is this dictionary answering (the
// core list marks it); `hous` and `because` aren't.
// A tiny en_US. `hous` and `because` aren't words in it, so a marked `because` is this
// dictionary answering (the built-in list knows it).
let english: &[(&str, &[u8], &[u8])] =
&[("en_US", b"SET UTF-8\nTRY esiarntolcdugmphbyfvkwz\nSFX S Y 1\nSFX S 0 s .\n", b"3\nteh\nthe\nhouse/S\n")];
folder_with_english(&dir, english);
if cfg!(craft_proof_data) {
if english_data() {
assert_eq!(english_source(ProofLang::En), Some(EnglishSource::Embedded));
// (The Moby list has `teh`.)
assert!(!is_correct_in("becuase", ProofLang::En) && is_correct_in("because", ProofLang::En));
@@ -1229,12 +1232,12 @@ mod tests {
assert!(is_correct_in(w, ProofLang::En), "{w}");
}
for w in ["hous", "because", "housez"] {
assert!(!is_correct_in(w, ProofLang::En), "{w}: the dictionary's words, not the core list's");
assert!(!is_correct_in(w, ProofLang::En), "{w}: the dictionary's words, not the built-in list's");
}
assert!(!is_correct("hous") && is_correct("teh"), "the older English-only functions too");
assert_eq!(check_spelling_in("teh hous", ProofLang::En).len(), 1);
assert!(suggest_in("hous", 3, ProofLang::En).contains(&"house".to_string()), "{:?}", suggest_in("hous", 3, ProofLang::En));
// No en_GB or en_CA file: the en_US one beats the core list.
// No en_GB or en_CA file: they use the en_US one (one copy, shared).
assert_eq!(english_source(ProofLang::EnGb), Some(EnglishSource::Hunspell(dic.clone())));
assert_eq!(english_source(ProofLang::EnCa), Some(EnglishSource::Hunspell(dic)));
}
@@ -1243,14 +1246,15 @@ mod tests {
assert!(check_text("hous teh", &[(0..8, ProofLang::De)]).is_empty());
assert_eq!(english_source(ProofLang::Pl), None);
drop_english_files(&dir, english);
if !cfg!(craft_proof_data) {
if !english_data() {
assert_ne!(english_source(ProofLang::En), Some(EnglishSource::Hunspell(dir.join("en_US.dic"))));
}
}
/// R6.4: an English Hunspell dictionary that is there but can't be read (an encoding we don't
/// read, bytes that aren't UTF-8, no words) is an error reported for its English, never a
/// panic, and that English falls back to the core list, which still marks `becuase`.
/// panic, and that English falls back to the next dictionary or the built-in word list, which
/// still marks `becuase`.
#[test]
fn unreadable_english_dictionary_is_reported_and_skipped() {
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
@@ -1259,11 +1263,13 @@ mod tests {
&[("en_US", b"SET UTF-8\n", b"0\n"), ("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"), ("en_CA", b"SET UTF-8\n", b"1\n\xff\xfe\n")];
folder_with_english(&dir, english);
for (lang, reason) in [(ProofLang::En, "no words"), (ProofLang::EnGb, "unsupported encoding"), (ProofLang::EnCa, "not UTF-8")] {
if cfg!(craft_proof_data) {
if english_data() {
assert_eq!(english_source(lang), Some(EnglishSource::Embedded));
assert_eq!(dictionary_load_error(lang), None, "{lang:?}: never read");
} else {
assert_eq!(english_source(lang), Some(EnglishSource::CoreList), "{lang:?}");
// The built-in list, or a dictionary installed on the system (tried next).
let source = english_source(lang);
assert!(!matches!(&source, Some(EnglishSource::Hunspell(p)) if p.starts_with(&dir)), "{lang:?}: {source:?}");
let error = dictionary_load_error(lang).unwrap_or_default();
assert!(error.contains(reason), "{lang:?}: {error}");
}
@@ -1274,6 +1280,36 @@ mod tests {
assert_eq!(dictionary_load_error(ProofLang::EnGb), None, "a new folder forgets the old errors");
}
/// craft-libs#482 review: an English dictionary that can't be read is skipped for the next
/// name or folder, not for the built-in list; and a dictionary with compounding and output
/// conversion (as the usual en_US and en_GB ones have) is read, not refused.
#[test]
fn english_tries_the_next_dictionary_and_reads_real_world_directives() {
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let dir = std::env::temp_dir().join(format!("wc-proof-en-next-{}", std::process::id()));
let english: &[(&str, &[u8], &[u8])] = &[
("en_GB", b"SET SHIFT_JIS\n", b"1\ncolour\n"),
(
"en_US",
b"SET UTF-8\nICONV 1\nICONV \xe2\x80\x99 '\nOCONV 1\nOCONV ' \xe2\x80\x99\nCOMPOUNDMIN 1\nONLYINCOMPOUND c\n\
COMPOUNDRULE 1\nCOMPOUNDRULE n*1t\nSFX S Y 1\nSFX S 0 s .\n",
b"4\nhouse/S\nthe\nqwzzy\n1st/c\n",
),
];
folder_with_english(&dir, english);
if !english_data() {
let us = dir.join("en_US.dic");
for lang in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
assert_eq!(english_source(lang), Some(EnglishSource::Hunspell(us.clone())), "{lang:?}");
assert!(is_correct_in("qwzzy", lang) && is_correct_in("houses", lang) && !is_correct_in("because", lang), "{lang:?}");
}
let error = dictionary_load_error(ProofLang::EnGb).unwrap_or_default();
assert!(error.contains("unsupported encoding"), "the en_GB file it would rather use is reported: {error}");
assert_eq!(dictionary_load_error(ProofLang::En), None);
}
drop_english_files(&dir, english);
}
/// #1063: the dictionaries folder is searched before the system locations; the dictionaries
/// in it are reported and used, and cached checks are dropped. The folder holds the same test
/// dictionaries the other tests install, so tests running meanwhile see the same words; it
@@ -1718,8 +1754,8 @@ mod tests {
assert!(words.iter().all(|w| w.chars().all(|c| c.is_lowercase() || c == '\'' || c == '-')), "not lowercase");
}
/// Without the Language data too: the installed English dictionary or the core list marks
/// both near misses (R6.4).
/// Without the Language data too: the installed English dictionary or the built-in word list
/// marks both misspellings (R6.4).
#[test]
fn spelling_issues_in_text() {
let _guard = DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
@@ -1758,12 +1794,13 @@ mod tests {
assert!(["xyzq", "blorp", "qwirpy", "zxq", "frobq", "zibb", "snorf", "wugwug"].iter().all(|w| bad.contains(w)), "{bad:?}");
}
#[cfg_attr(not(craft_proof_data), ignore = "needs the Language data (CRAFT_I18N_DIR)")]
#[test]
fn spelling_scales_over_one_long_token() {
// #1335: thousands of punctuation-separated words inside a single
// whitespace-free token: one classification for the token, one issue per
// misspelled word, through every public spelling entry point.
// misspelled word, through every public spelling entry point. Without the Language data
// too (craft-libs#482 review: the old core list took 14 s here).
let started = std::time::Instant::now();
let word = |i: usize| {
let mut w = String::from("zxq");
let mut n = i;
@@ -1782,6 +1819,8 @@ mod tests {
let opts = Options { ignore_internet: false, ..Options::default() };
assert_eq!(check_spelling_with(&t, &opts).len(), 2000);
assert_eq!(check_text(&t, &[]).into_iter().filter(|i| i.kind == IssueKind::Spelling).count(), 2000);
// Debug builds on a loaded CI machine: a bound against blowups, not a benchmark.
assert!(started.elapsed() < std::time::Duration::from_secs(10), "{:?}", started.elapsed());
}
#[test]
crates/proof/src/no_data.rs
+12 -10
View File
@@ -1,6 +1,6 @@
//! A build with no Language data embedded (no `CRAFT_I18N_DIR`): every public entry point still
//! works and never panics. English spelling still marks obvious misspellings, against an English
//! Hunspell dictionary installed on the system or else the core word list built into the crate;
//! works and never panics. English spelling still marks misspellings, against an English
//! Hunspell dictionary installed on the system or else the Moby word list built into the crate;
//! the dialect variants and hyphenation that need the data are off, and grammar (code, not data)
//! still works.
@@ -53,23 +53,25 @@ fn nothing_is_embedded() {
}
/// R6.4, English never goes unchecked: without the embedded word list, English is checked against
/// an installed English Hunspell dictionary or, without one, the core word list. Either way an
/// obvious misspelling is marked and common words are right. (Before, everything was right.)
/// an installed English Hunspell dictionary or, without one, the Moby word list built into the
/// crate. Either way a misspelling is marked and common words are right. (Before, everything was right.)
#[test]
fn english_spelling_still_marks_misspellings() {
let _guard = crate::DICTIONARY_TESTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
for l in [ProofLang::En, ProofLang::EnGb, ProofLang::EnCa] {
assert!(matches!(crate::english_source(l), Some(EnglishSource::Hunspell(_) | EnglishSource::CoreList)), "{l:?}");
assert!(!crate::is_correct_in("teh", l), "{l:?}");
assert!(crate::suggest_in("teh", 5, l).contains(&"the".to_string()), "{l:?}");
assert!(!crate::is_correct_in("becuase", l), "{l:?}");
assert!(crate::suggest_in("becuase", 5, l).contains(&"because".to_string()), "{l:?}");
for w in ["the", "The", "house", "houses", "because", "running", "people", "beautiful", "don't"] {
assert!(crate::is_correct_in(w, l), "{w} {l:?}");
}
assert_eq!(crate::check_spelling_in("Teh house of the people", l).len(), 1, "{l:?}");
assert_eq!(crate::check_spelling_in("Becuase the house of the people", l).len(), 1, "{l:?}");
}
assert!(!crate::is_correct("teh") && !crate::is_correct_with("teh", &Options::default()) && !crate::is_correct_for("teh", Some("en-GB")));
assert!(crate::suggest("teh", 5).contains(&"the".to_string()));
assert_eq!(crate::check_spelling("Teh house").len(), 1);
assert!(
!crate::is_correct("becuase") && !crate::is_correct_with("becuase", &Options::default()) && !crate::is_correct_for("becuase", Some("en-GB"))
);
assert!(crate::suggest("becuase", 5).contains(&"because".to_string()));
assert_eq!(crate::check_spelling("Becuase house").len(), 1);
// The dialect variants are Language data: none without it.
for w in ["colour", "color"] {
assert_eq!(en_variants::preferred(w, ProofLang::En), None);
crates/proof/src/vi.rs
+1 -1
View File
@@ -83,7 +83,7 @@ mod checker_tests {
assert!(check_text_with(text, &[], &Options::default(), true, false).is_empty());
}
/// Without the Language data too: the installed English dictionary or the core list marks
/// Without the Language data too: the installed English dictionary or the built-in word list marks
/// `yesterdy` (R6.4).
#[test]
fn english_keeps_its_marks_and_skips_vietnamese_names() {