Variables: Import Data Sets reads UTF-16 CSV files without a byte-order mark

A CSV file that holds NUL bytes and has no byte-order mark was refused.
It is now read as UTF-16 when at least half of its two-byte units hold
a NUL byte, in the byte order that reads a line break. A file with fewer
NUL bytes (UTF-8 or Windows-1252 text with stray NUL bytes, or UTF-16
text made mostly of characters above U+00FF) is still refused, and so is
a file that reads a line break in both byte orders, has an odd length,
isn't valid UTF-16 in that order or holds a NUL character. Refused files
get the same message as before.
This commit is contained in:
dotCooCoo
2026-11-13 16:34:43 -08:00
parent 4558196e16
commit ead3cd1ebd
5 changed files with 107 additions and 18 deletions
+7 -2
View File
@@ -3115,8 +3115,13 @@ already walks Image Trace. Short agent entry points:
reply's `defined` lists the variables the import defined. While the variables are
locked, an import whose header would define a variable fails with "variables are locked".
A `dataset.import` file that isn't UTF-8 is read as Windows-1252, unless it starts with the UTF-8 byte-order mark.
A file that starts with a UTF-16 byte-order mark is read as UTF-16, and a file without one that holds NUL bytes is
refused. `csv` and `data` are read as the text they hold.
A file that starts with a UTF-16 byte-order mark is read as UTF-16. A file without one that holds NUL bytes is read
as UTF-16 when at least half of its two-byte units hold a NUL byte, as characters from U+0001 to U+00FF do, in the
byte order that reads a line break (`\n` or `\r`). It is refused when fewer of its units hold a NUL byte (UTF-8 or
Windows-1252 text with stray NUL bytes, or UTF-16 text made mostly of other characters, which needs a byte-order
mark), when both byte orders read a line break (the text holds U+0A00 or U+0D00), when it has an odd length, and
when it isn't valid UTF-16 in that order or holds a NUL character. A file with no line break has no data rows and is
refused for that. `csv` and `data` are read as the text they hold.
- Variable libraries: `variable.loadLibrary {path|xml|dataBase64}` (the Variables panel menu's Load Variable
Library…) replaces the document's variables and data sets with a variable library's in one undo step. A load, like a
`dataset.import`, adds no undo step when it leaves the variables and data sets as they were. The library
+1
View File
@@ -779,6 +779,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur
- Variables (data merge): text, visibility and image variables bound to objects (an object may carry several), dataset rows applied in one undo step (`variable.*`, `dataset.*`) from a Variables panel and Window › Variables, with dialogs to define, bind and edit, Make Text/Visibility/Image Dynamic, Capture/Update Data Set, CSV import/export (`dataset.import` / `dataset.export`), and a row in italics when an edit leaves the art not matching it; an image dataset value is a file path and apply keeps the object's bounds; bindings let go of deleted art and locked art is left alone except by visibility values; every command, like the rest, is drivable over MCP and the CLI.
- Export for Screens numbers distinct row files whose names collide, compared without case, as `-2`, `-3`… instead of leaving them out; identical rows still write one file. It puts row folders that differ only in case into one folder (`document.exportForScreens`).
- Variables: Import Data Sets… (`dataset.import`) reads tab-separated files as well as comma-separated ones. Every header cell is a variable, and the rows replace the document's data sets as Data Set 1, Data Set 2 and so on. An import error about a row's values gives the file line the row starts on.
- Variables: Import Data Sets… (`dataset.import`) reads a UTF-16 file without a byte-order mark, instead of refusing it, when at least half of its two-byte units hold a NUL byte; it uses the byte order that reads its line breaks.
- Variables: Import Data Sets… (`dataset.import`) defines the variables a file's header names that the document lacks, unbound. `#Shown` defines a visibility variable, `@Photo` defines an image variable, and a header without a prefix defines a text variable. A header that is a variable's exact name keeps naming it, and any other `%` header (graph data files) is refused.
- Variables: Load Variable Library… in the Variables panel menu (`variable.loadLibrary`) replaces the document's variables and data sets with a variable library's, read from its XML file, in one undo step: every data set is deleted, and every variable is deleted with its bindings and values, except a text variable the library also declares, which stays bound so the library's data sets change its objects (#4136); then the library's other text variables are defined, unbound, and its data sets take the text of each value's `p` elements, one paragraph each. Variables of other kinds are left out with a warning that names their kinds.
- Variables: Save Variable Library… in the Variables panel menu (`variable.saveLibrary`) writes the document's text variables and data sets as a variable library's XML file, which Load Variable Library reads back with the same text variables, data set names and text values. Variables of other kinds are left out with a warning that names their kinds.