Commit Graph
10 Commits
Author SHA1 Message Date
Dylan M. Taylor 4f3bf746bb Merge pull request #1886 from Hamza0101/fix/sylk-escaped-characters
craft-csv: decode Excel's character escapes in SYLK text
2026-11-16 19:48:42 -05:00
dotCooCoo 52cca84204 fix(csv): count a first record longer than the delimiter sample
sniff_delimiter reads the first 64 KiB of the text and leaves out a
record cut off at that point (#1209). When the first record is longer
than the sample, no record is counted and the function returns comma,
so a semicolon or tab file with a very wide first row opens as one
column.

When no candidate delimiter finds coherent complete records with a
delimiter in each, a candidate that finds no complete record now
counts the delimiters read so far in the first record. Delimiters
inside quoted fields are not counted. A quote inside a field that does
not start with one makes that record incoherent, and the coherent
candidate is picked. When one of those candidates ends the sample
inside a quoted field, each of them counts only up to that field's
opening quote. The per-candidate loop moved into a scan_delimiter
helper, which runs a second scan over the text before that quote.
2026-11-16 11:29:40 -08:00
Hamza 503caef9c4 craft-csv: a SYLK diacritic on a space is the spacing accent
ISO 6937 writes a spacing accent as its diacritic followed by SPACE.
ESC N B SPACE was read as a space and a combining acute; it is now the
acute accent itself, and likewise for the other nine marks glibc's
ISO_6937 map gives a spacing form.
2026-11-16 19:20:31 +05:00
Hamza e096b5eb9b craft-csv: a diacritic before another escape keeps it as written
An escape's diacritic took the next character as its letter even when that was the next escape (ESC N H ESC N H o gave ESC + U+0308 + NHo). A control character is no letter: the first escape is kept as written and the next one decodes on its own (ESC N H ö).
2026-11-16 19:07:27 +05:00
Hamza ab87149855 craft-csv: decode Excel's character escapes in SYLK text
Excel writes a character outside ASCII in a SYLK text value as ESC N and one byte, the character's ISO 6937 code less 0x80; a diacritic's code is followed by the letter it goes on. The reader kept those bytes, so Größe für Köln opened as Gr<ESC>NHo<ESC>N{e f<ESC>NHur K<ESC>NHoln (storytold/gridcraft#4321). Text values now decode them: a diacritic and letter become the precomposed letter (or the letter and its combining mark when Unicode has no single character), a spacing code its character; a code ISO 6937 doesn't define is kept as written.
2026-11-16 18:51:40 +05:00
Dylan M. Taylor 69e457a8eb Add strict UTF-8 text decoder 2026-11-13 21:38:11 +00:00
Dylan M. Taylor 415dc3204c fix(csv): preserve bounded delimiter sampling 2026-11-13 19:25:10 +00:00
Dylan M. Taylor 5d46565f01 test(csv): lock delimiter tie priority 2026-11-13 19:03:26 +00:00
Dylan M. Taylor 8263ccd054 fix(csv): preserve quoted delimiter coherence 2026-11-13 18:28:50 +00:00
Dylan M. Taylor 991ea7dbd6 Add craft-csv: delimited text and SYLK, from GridCraft (#889)
Moved from GridCraft at 5cf52c5f: the encoding detection, delimiter
sniffing, record parser and field quoting of crates/xlsx/src/csv.rs,
and the SYLK reader of crates/engine/src/io/sylk.rs.

Fields stay text and the SYLK reader returns its own Sheet; typed
values, dates and the workbook stay in GridCraft.
2026-11-11 18:07:00 -05:00