Variables: Import Data Sets reads UTF-16 CSV files without a byte-order mark

A CSV file that holds NUL bytes and has no byte-order mark was refused. It is now read as UTF-16 when at least half of its two-byte units hold a NUL byte, in the byte order that reads a line break. A file with fewer NUL bytes (UTF-8 or Windows-1252 text with stray NUL bytes, or UTF-16 text made mostly of characters above U+00FF) is still refused, and so is a file that reads a line break in both byte orders, has an odd length, isn't valid UTF-16 in that order or holds a NUL character. Refused files get the same message as before.
2026-11-15 08:45:25 +00:00 · 2026-11-13 16:34:43 -08:00
parent 4558196e16
commit ead3cd1ebd
5 changed files with 107 additions and 18 deletions
@@ -867,7 +867,7 @@ pub fn specs() -> Vec<CommandSpec> {
            "Import Data Sets…",
            ["Window", "Variables"],
            None,
-            "{path? | data? | dataBase64? | csv?} replace every dataset with the rows of a CSV, comma- or tab-separated, in UTF-16 with a byte-order mark, UTF-8 or else Windows-1252 (header: one variable name per column, whatever its case; one row per dataset, named Data Set 1, 2… in file order). The document's variables and their bindings stay. A header that names no variable defines one, unbound: `#Name` defines a visibility variable, `@Name` defines an image variable, and a header without a prefix defines a text variable. A prefixed header whose name is a variable of another kind, and a `%` header that isn't a variable's exact name (graph data files), are refused, and so are two headers that name one variable and a header that would define a variable while the variables are locked. An import that leaves the variables and data sets as they were adds no undo step → {imported, names, defined, deletedDataSets}",
+            "{path? | data? | dataBase64? | csv?} replace every dataset with the rows of a CSV, comma- or tab-separated, in UTF-16 (with a byte-order mark, or without one when at least half of its two-byte units hold a NUL byte and only one byte order reads a line break; any other file with NUL bytes is refused), UTF-8 or else Windows-1252 (header: one variable name per column, whatever its case; one row per dataset, named Data Set 1, 2… in file order). The document's variables and their bindings stay. A header that names no variable defines one, unbound: `#Name` defines a visibility variable, `@Name` defines an image variable, and a header without a prefix defines a text variable. A prefixed header whose name is a variable of another kind, and a `%` header that isn't a variable's exact name (graph data files), are refused, and so are two headers that name one variable and a header that would define a variable while the variables are locked. An import that leaves the variables and data sets as they were adds no undo step → {imported, names, defined, deletedDataSets}",
            has_doc,
            dataset_import
        ),