vectorcraft-js and Check Script took identifier categories from
unicode-general-category 1.1.0, which carries Unicode 16.0, so letters
and marks assigned in Unicode 17.0 (U+088F at the start of a name,
U+1ACF after its first character) were refused. Both lexers now read
general categories from unicode-properties 0.1.4 (Unicode 17.0), which
Cargo.lock already holds. The version is exact because 0.1.5 adds the
Unicode-3.0 license, which is not on the license allowlist. The
Other_ID_Start, Other_ID_Continue and Pattern_Syntax lists are the same
in PropList 17.0.
The exhaustive identifier test now expects the DerivedCoreProperties
17.0 totals (145,916 ID_Start, 149,240 ID_Continue), and new tests read
U+088F and U+1ACF written out and as escapes in both lexers.
- A new test walks every code point from U+0000 to U+10FFFF, surrogates
left out, and compares the start and part checks with Unicode Standard
Annex #31's general-category definition of ID_Start and ID_Continue,
written out in the test. The test also checks the set sizes against the
totals in Unicode 16.0's DerivedCoreProperties (141,269 and 144,541).
- The Other_ID_Continue test covers every code point of the property,
U+136A to U+1370 and the zero width non-joiner and joiner included.
- The zero width non-joiner and joiner are part of the Other_ID_Continue
list in chars.rs instead of a separate case.
is_id_start now accepts the Other_ID_Start characters (U+1885, U+1886,
U+2118, U+212E, U+309B, U+309C), and is_id_part the Other_ID_Continue
characters (U+00B7, U+0387, U+1369 to U+1371, U+19DA, U+30FB, U+FF65),
as Unicode's ID_Start and ID_Continue properties and the ECMAScript
Names and Keywords clause define them. Letters, marks, digits and
connectors that are Pattern_Syntax are refused, which replaces the
U+2E2F special case. Escaped identifier characters go through the same
check. A supplementary letter right after a number is now the
"identifier starts immediately after numeric literal" error.
Letter categories still come from Unicode 16.0, so characters assigned
in later versions are not identifier characters yet.
Docs: docs/mcp.md, docs/target-app-parity.md.
is_id_start accepts the letter categories (Lu, Ll, Lt, Lm, Lo, Nl) and
is_id_part adds Mn, Mc, Nd and Pc, read with unicode-general-category.
A superscript digit (No) inside an identifier and a mark such as U+09BE
at its start are now SyntaxErrors, and identifiers with marks outside
the old list, such as the Bengali and Tamil viramas, now parse.
Labels in scope are kept in a map, and duplicate labels in a chain and
duplicate parameters of a strict function are found with a hash set, so
long chains, many labeled statements and long parameter lists are
checked in linear time.
New crate vectorcraft-js reads ECMAScript 5.1 source into a syntax tree
with source positions: a UTF-16 lexer (line and column tracking, automatic
semicolon insertion, the regexp and division goals), preprocessor lines
read as directive tokens, a recursive-descent parser to an arena tree, and
the ES5 early errors, strict mode included. Nesting deeper than
Options::max_depth (200 by default) gives a SyntaxError with its line and
column rather than overflowing the stack. Nothing runs scripts yet.
cargo xtask test262 downloads test262 at a pinned commit, checks its
SHA-256, unpacks it into the gitignored corpus/ folder and runs the ES5
syntax tests. A differential test in the engine compares the lexer with
scriptcheck on line counts, directives and unterminated literals.
dotCooCoo