* i18n: keep fold search, isolate wrapping and locale fallbacks linear
- Folded::matches restarted str::find one character past every occurrence
turned down at a combining mark, so overlapping occurrences of a long
needle cost haystack x needle. It now finds every occurrence, overlapping
ones included, in one Knuth-Morris-Pratt pass and checks the boundaries
as before.
- wrap searched the whole open-control stack for an isolate at every PDI;
it now keeps a count of open isolates.
- candidates made one joined prefix per subtag of a locale tag of any
length; a tag longer than 128 bytes is not a locale and has none.
* address review: length check only on the public candidates entry
I18n::new keeps the embedded text and parses nothing; a locale's rows
are parsed on the first lookup that reaches it (also as a fallback) and
its bad rows logged with log::warn!. new returns only the locale-level
problems. check_all parses every locale and returns all problems.
name() scans for the language.name row instead of parsing the locale.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A value with an embedding, override, isolate or mark (LRE RLE PDF LRO
RLO LRI RLI FSI PDI LRM RLM ALM) makes fmt isolate, even in text with no
right-to-left letter, so an ASCII file name with an RLO can't reorder
the sentence. Embeddings, overrides and isolates a value opens are
closed inside its isolate, and stray PDF and PDI are dropped.
han(zh-US), zh-AU and the other regions CLDR writes in Traditional
characters give Traditional, and pa-PK picks a pa-arab catalog, never
Gurmukhi pa, as pa-Arab-PK does.
Format::parse reads what a Format writes: its digits, its decimal and
grouping separators (the system's included) and the locale's Latin
marks in an Arabic-Indic format, plus every digit set. parse_number is
Format::from_tag(locale).parse, so the decimal marks come from CLDR's
data alone and the hand-made table is gone.
A lone . in a comma locale is the decimal mark when it can't be
grouping (1.5, 0.250, 1234.5, .5); 1.234 and 1.234.567 stay grouped in
German. The same holds for , in a point locale. Percent signs and units
are documented as the caller's to strip.
Plural rules are looked up by language alone: CLDR's parent of sr-Latn
and zh-Hant is the root, whose only category is other, so sr-latn rows
were rejected. The baked data covers every CLDR locale (10 KB): plural
rules for he, ro, nb, ca, sk, th, ms and the rest, and decimal symbols
for nb-NO, sk-SK, bn-BD, ar-MA and English in Europe. Format applies a
Region's separators only where the system set them, and an empty
grouping separator writes none.
trn_plain writes counts in 0-9 without grouping, for documents and file
names. reset_format goes back to the system region. trn_decimal and
Format write infinities and NaN as CLDR does and take the other form.
Tests that don't set a format use en-US, not the machine's region. The
ICU4X crates are pinned to ~2.3, the minor version the data is for.
Collator::new takes sv_SE.UTF-8 and sr_RS@latin. Folding maps every
script's decimal digits to ASCII and, ignoring accents, Persian keheh,
Farsi yeh, alef maksura, teh marbuta and heh with hamza to the Arabic
letters. The collation data grows 4.8 KB (465 bytes gzipped).
Region::decimal and grouping are None unless Windows' or macOS's own
settings set them, so the number locale's CLDR data decides otherwise.
A cleared Windows sThousand is no grouping. LC_ALL=C and LC_NUMERIC=C
(and the other LC_* variables) mean the C locale; only LANG=C gives way
to the preferred language.
fmt and trn add isolates only when the template or an inserted value has
a strong RTL character (bidi class R or AL). Text with none comes out as
before, and a value's own isolate from ltr or isolate is taken off.
Format writes integers, decimals and percentages for a Region on ICU4X's
decimal formatter: the locale's digits (Arabic-Indic in Egypt, Persian in
Iran) or Latin ones on request, its separators and grouping sizes (Indian
12,34,567), CLDR's percent patterns, and the region's own separators where
the system sets them. I18n::format holds the one trn, file_size and
measure use, the system region's until set_format.
Plural rules move from intl_pluralrules to ICU4X, and trn_decimal picks
the form from the digits shown (1.0 files, 1,5 файла).
The CLDR data (decimal symbols, digits, plural rules, percent patterns)
covers the shipped languages and their regional variants; data/generate.sh
rebuilds it.
`Folder` folds text for case- and accent-insensitive search (Turkish and
Azerbaijani i by locale, ß, final sigma, ligatures, full-width forms) and
maps matches back to the original text for highlighting. `Collator`
(feature `collate`) sorts lists in the locale's order with digits compared
by value, on ICU4X with collation data for the shipped locales only.
The Turkish i and sigma handling come from PdfCraft's `fold_case`.
Part of #198.
Co-authored-by: snipeTR <31975916+snipeTR@users.noreply.github.com>
Co-authored-by: nevicia2010-hub <nevicia2010@gmail.com>
I18n::file_size writes a byte count in steps of 1024 (KB or KiB) or 1000
(KB), with decimals per unit, no decimals from a given value up, a
smallest and largest unit, and bytes as B or in words. I18n::measure
writes any value with a unit template such as "{n} px". Unit names are
catalog rows in context `unit`, keyed by the English template, so a
translation can change the symbol and its spacing.
Reads any Unicode decimal digits, the locale's decimal mark plus ٫,
grouping by the locale's other mark, spaces, apostrophes or ٬ (threes
or Indian 2-2-3), and -/−/+ signs, ignoring bidi marks. Input that
fits no single reading is an error, not a guess.
Region holds the number and time locales, metric or US measurement, A4 or
Letter, the decimal separator, and the grouping separator and clock when
the system sets them. Region::system reads Windows' regional format with
the user's changes, Apple's current locale, the POSIX LC_* variables each
on their own, and the preferred language on the web and Android.
Region::from_setting takes an app's saved choice ("auto" or a tag).
Ported from WordCraft's Region (storytold/wordcraft#857, #1494).
Co-authored-by: Dmitrii Khlapov <dkhlapov@pm.me>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
Dylan M. Taylor