The Graph Data window's Import Data read UTF-16 only after a byte-order
mark, so a UTF-16 file without one filled the cells with NUL characters.
It now reads such a file in the byte order Import Data Sets' rule (#7348)
gives it: at least half of the two-byte units hold a NUL byte, and the
byte order is the one that reads the line breaks. The text is then read
as after a byte-order mark: an odd last byte is dropped and an unpaired
surrogate reads as U+FFFD. A file the rule refuses is read as UTF-8, as
before.
The rule now also gives a byte order to a one-line file, which reads a
line break in neither order: big-endian when more of its units start
with a NUL byte than end with one, else little-endian as before. Import
Data Sets still refuses a one-line file, which has no data rows; read in
the new order, such a file can be refused with a different error first.
A CSV file that holds NUL bytes and has no byte-order mark was refused.
It is now read as UTF-16 when at least half of its two-byte units hold
a NUL byte, in the byte order that reads a line break. A file with fewer
NUL bytes (UTF-8 or Windows-1252 text with stray NUL bytes, or UTF-16
text made mostly of characters above U+00FF) is still refused, and so is
a file that reads a line break in both byte orders, has an odd length,
isn't valid UTF-16 in that order or holds a NUL character. Refused files
get the same message as before.
dataset.import and variable.loadLibrary added an undo step, cleared Redo
and marked a saved document modified even when the variables and data
sets came out as they were, such as the same CSV or library loaded
again. Both now build the new variables on a copy and record a step only
when it differs. variable.loadLibrary's params text and docs/mcp.md say
so.
dataset.import read two header cells that named one variable (City
and City, or Shown and #Shown) into the same column, so each row kept
only the last non-blank cell and the other value was dropped without a
word. The import is now refused with the two column numbers, and the
message has a row in every complete-message catalog.
dataset.import replaced the reason for a refused cell with "value for
`X` must match its variable kind", so a visibility value such as
maybe and a graph value over 262,144 bytes gave the same message. The
error now keeps the line and gives the reason, and both messages have
a row in every complete-message catalog.
dataset.export checked only the 1 MiB size, so a document with more than
1000 data sets or 256 variables exported a CSV that dataset.import then
refused. Export now refuses such a document with a message naming the
limit, and the messages have rows in every complete-message catalog.
Found while testing crates/engine/src/cmd/variablecmds/csv.rs:146 (csv_escape) and :158-163 (cell_to_value)
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>