Merge pull request #7554 from dotCooCoo/fix/place-text-utf16-no-bom

Place: text files in UTF-16 without a byte-order mark place as UTF-16

crates/engine/src/cmd.rs

@@ -892,6 +892,10 @@ pub use buildcmds::live_paint_gaps;
 /// Control bar enable it when it changes one).
 pub use brushsym::reset_transformation_changes;
 
+/// The byte order of UTF-16 text without a byte-order mark, by Import Data Sets' rule (the Graph
+/// Data window reads files by it too).
+pub use variablecmds::utf16_order_without_mark;
+
 /// Command ids kept only so older scripts keep working: `(alias, the command it duplicates)`.
 /// They run as before but the command palette leaves them out (it lists the command once).
 pub const ALIASES: &[(&str, &str)] = &[("object.convertDocumentColorMode", "file.documentColorMode")];

crates/engine/src/cmd/place.rs

@@ -75,7 +75,7 @@ pub fn specs() -> Vec<CommandSpec> {
             "Place…",
             ["File"],
             Some("Cmd+Shift+P"),
-            "{path | name | dataBase64 (a file's path, or its name with its bytes), link?: true (a raster image keeps its file's path; a VectorCraft document at path becomes one locked placed document showing artboard `page`, linked to the file and read again when it changes (links.update; saving it in this app updates it at once), drawn as vectors in every output; so does a PDF/.ai page at path, showing its `crop` box, unless password or textAs: \"outlines\" is given; false: an editable copy of its art, as from dataBase64; other files are embedded), text?: {characterSet?: \"unicode\" (UTF-8, or UTF-16 with a byte-order mark; other bytes as the platform's 8-bit set) | \"ansi\" (the platform's 8-bit set), platform?: \"windows\" (Windows-1252) | \"mac\" (Mac Roman), removeLineReturns?: false (each block of lines becomes one paragraph; blank lines end paragraphs), removeParagraphReturns?: false (drop blank lines), replaceSpaces?: n (runs of n ≥ 2 spaces become a tab)} (a .txt file, placed as area type filling rect, the replaced object's bounds, or else the artboard less a 36 pt margin), template?: false (onto a new locked template layer below the current layer), replace?: false (swap the one selected object, keeping its stacking place and transform; no at/rect; refused while an object of a symbol instance's art is selected), at?: [x, y] centre (default: the first artboard's centre), rect?: [x, y, width, height] fit inside; aspect kept (wins over at), page?: 1 (PDF/.ai page, or a native document's artboard), crop?: \"crop\" (clipped to that page or artboard, default; an SVG, only when given: framed and clipped by its viewBox, else by its art's bounds) | \"bounding\" (the bounds of the art that shows, with its strokes and effects, as an export measures them: hidden objects and guides left out) | \"art\" | \"trim\" | \"bleed\" | \"media\" (a PDF page's boxes), password? (an encrypted PDF), dxf?: {…the DXF options of document.open} (a .dxf drawing: fit fills the artboard under at, else the first; fitted or uncentred art lands with its drawing's artboard on that artboard, else it is centred on at)} → {ids, name, format, linked, width, height, warnings, image?: the id of the image of a DCS file placed in a clip group, which the Links commands take}. Raster images come in at 100% of their physical size (the file's ppi, else 72); SVG, DXF, HP-GL (.plt, .hpgl, .hgl, .hpg), PDF/.ai, EMF/WMF and PICT (.pct, .pic, .pict; clipped to the picture's frame), EPS (clipped to its bounding box; as document.open reads it), a .ai saved without PDF compatibility (from its editing data, as document.open reads it: the art of the layers that show on artboard `page`, clipped to that artboard unless crop is \"bounding\") and native documents as one group, with the images, symbols, patterns, swatches and brushes they use. A DCS file (an EPS file with one program per printing plate, DCS 1.0 or 2.0) is placed only from its path with link (else an error, as are document.open and links.embed): one linked image of its plates mixed into CMYK, clipped by the clipping path its composite is drawn through, its spot plates added as spot swatches in the process equivalents the file gives (a swatch of the same name is kept, with a warning); print separations and Separations Preview show each plate on its own ink, and plate files of their own are read from its folder. One undo step; selects what it placed (unless on a template layer); never touches the clipboard. A PNG, JPEG, GIF or WebP image whose pixels can't be read (a file cut short) is an error, and nothing is placed",
+            "{path | name | dataBase64 (a file's path, or its name with its bytes), link?: true (a raster image keeps its file's path; a VectorCraft document at path becomes one locked placed document showing artboard `page`, linked to the file and read again when it changes (links.update; saving it in this app updates it at once), drawn as vectors in every output; so does a PDF/.ai page at path, showing its `crop` box, unless password or textAs: \"outlines\" is given; false: an editable copy of its art, as from dataBase64; other files are embedded), text?: {characterSet?: \"unicode\" (UTF-8, or UTF-16 with a byte-order mark, or without one by dataset.import's rule: an even length, at least half of its two-byte units holding a NUL byte, the byte order that reads a line break, valid UTF-16 with no NUL character; other bytes as the platform's 8-bit set) | \"ansi\" (the platform's 8-bit set), platform?: \"windows\" (Windows-1252) | \"mac\" (Mac Roman), removeLineReturns?: false (each block of lines becomes one paragraph; blank lines end paragraphs), removeParagraphReturns?: false (drop blank lines), replaceSpaces?: n (runs of n ≥ 2 spaces become a tab)} (a .txt file, placed as area type filling rect, the replaced object's bounds, or else the artboard less a 36 pt margin), template?: false (onto a new locked template layer below the current layer), replace?: false (swap the one selected object, keeping its stacking place and transform; no at/rect; refused while an object of a symbol instance's art is selected), at?: [x, y] centre (default: the first artboard's centre), rect?: [x, y, width, height] fit inside; aspect kept (wins over at), page?: 1 (PDF/.ai page, or a native document's artboard), crop?: \"crop\" (clipped to that page or artboard, default; an SVG, only when given: framed and clipped by its viewBox, else by its art's bounds) | \"bounding\" (the bounds of the art that shows, with its strokes and effects, as an export measures them: hidden objects and guides left out) | \"art\" | \"trim\" | \"bleed\" | \"media\" (a PDF page's boxes), password? (an encrypted PDF), dxf?: {…the DXF options of document.open} (a .dxf drawing: fit fills the artboard under at, else the first; fitted or uncentred art lands with its drawing's artboard on that artboard, else it is centred on at)} → {ids, name, format, linked, width, height, warnings, image?: the id of the image of a DCS file placed in a clip group, which the Links commands take}. Raster images come in at 100% of their physical size (the file's ppi, else 72); SVG, DXF, HP-GL (.plt, .hpgl, .hgl, .hpg), PDF/.ai, EMF/WMF and PICT (.pct, .pic, .pict; clipped to the picture's frame), EPS (clipped to its bounding box; as document.open reads it), a .ai saved without PDF compatibility (from its editing data, as document.open reads it: the art of the layers that show on artboard `page`, clipped to that artboard unless crop is \"bounding\") and native documents as one group, with the images, symbols, patterns, swatches and brushes they use. A DCS file (an EPS file with one program per printing plate, DCS 1.0 or 2.0) is placed only from its path with link (else an error, as are document.open and links.embed): one linked image of its plates mixed into CMYK, clipped by the clipping path its composite is drawn through, its spot plates added as spot swatches in the process equivalents the file gives (a swatch of the same name is kept, with a warning); print separations and Separations Preview show each plate on its own ink, and plate files of their own are read from its folder. One undo step; selects what it placed (unless on a template layer); never touches the clipboard. A PNG, JPEG, GIF or WebP image whose pixels can't be read (a file cut short) is an error, and nothing is placed",
             has_doc,
             place
         ),

crates/engine/src/cmd/place/text.rs

@@ -3,6 +3,7 @@
 
 use serde_json::Value;
 
+use super::super::utf16_order_without_mark;
 use super::{Point, Rect, Result, bad, bool_or, fileio, shapes, str_param};
 
 /// Text files as `file.place` reports them (not a format `document.open` reads).
@@ -76,8 +77,10 @@ pub(super) fn import(bytes: &[u8], o: TextOptions, cmd: &str) -> Result<String>
     Ok(clean(&decode(bytes, o), o))
 }
 
-/// `bytes` as text: Unicode (UTF-8, or UTF-16 with a byte-order mark; bytes that aren't UTF-8 are
-/// read as the platform's 8-bit set), or the platform's 8-bit character set.
+/// `bytes` as text: Unicode (UTF-8, or UTF-16 with a byte-order mark, or without one when the
+/// file has an even length, reads as valid UTF-16 with no NUL character in the byte order
+/// [`utf16_order_without_mark`] gives; bytes that aren't UTF-8 are read as the platform's 8-bit
+/// set), or the platform's 8-bit character set.
 fn decode(bytes: &[u8], o: TextOptions) -> String {
     let eight_bit = |b: &[u8]| -> String { b.iter().map(|&c| if o.mac { mac_roman(c) } else { windows_1252(c) }).collect() };
     if o.ansi {
@@ -91,7 +94,17 @@ fn decode(bytes: &[u8], o: TextOptions) -> String {
         [0xEF, 0xBB, 0xBF, rest @ ..] => String::from_utf8_lossy(rest).into_owned(),
         [0xFF, 0xFE, rest @ ..] => utf16(rest, true),
         [0xFE, 0xFF, rest @ ..] => utf16(rest, false),
-        _ => std::str::from_utf8(bytes).map_or_else(|_| eight_bit(bytes), str::to_string),
+        _ => {
+            let without_mark = (bytes.len().is_multiple_of(2) && bytes.contains(&0))
+                .then(|| utf16_order_without_mark(bytes))
+                .flatten()
+                .and_then(|le| {
+                    let units = bytes.as_chunks::<2>().0.iter().map(|&c| if le { u16::from_le_bytes(c) } else { u16::from_be_bytes(c) });
+                    char::decode_utf16(units).collect::<std::result::Result<String, _>>().ok()
+                })
+                .filter(|text| !text.contains('\0'));
+            without_mark.unwrap_or_else(|| std::str::from_utf8(bytes).map_or_else(|_| eight_bit(bytes), str::to_string))
+        }
     }
 }
 

crates/engine/src/cmd/variablecmds.rs

@@ -26,6 +26,7 @@ use super::{CommandSpec, EngineError, Result, Session, bad, bool_or, checked_ids
 mod csv;
 mod library;
 
+pub use csv::utf16_order_without_mark;
 use csv::{MAX_GRAPH_CSV, dataset_export, dataset_import};
 pub(crate) use library::{give_svg_library, svg_library};
 use library::{load_library, save_library};

crates/engine/src/cmd/variablecmds/csv.rs

@@ -59,25 +59,35 @@ fn utf16(bytes: &[u8], le: bool) -> Option<String> {
     char::decode_utf16(units).collect::<std::result::Result<String, _>>().ok().filter(|_| odd.is_empty())
 }
 
-/// `bytes`, which hold NUL bytes and no byte-order mark, as UTF-16, or `None`. At least half of
-/// the two-byte units must hold a NUL byte, as characters from U+0001 to U+00FF do; a file with
-/// fewer, such as UTF-8 or Windows-1252 text with stray NUL bytes, is refused. The byte order is
-/// the one that reads a line break (`\n` or `\r`). The other order reads one only from U+0A00 or
-/// U+0D00, so a file that reads one in both orders is refused. A file that reads none in either
-/// order has no data rows, and is read little-endian so that the import reports that. The text
-/// must be valid UTF-16 in that order and hold no NUL character.
-fn utf16_without_mark(bytes: &[u8]) -> Option<String> {
-    let (units, odd) = bytes.as_chunks::<2>();
-    if !odd.is_empty() || units.iter().filter(|c| c.contains(&0)).count() * 2 < units.len() {
+/// The byte order of `bytes`, which hold NUL bytes and no byte-order mark, read as UTF-16:
+/// `Some(true)` for little-endian, or `None` when they don't read as UTF-16. At least half of the
+/// two-byte units must hold a NUL byte, as characters from U+0001 to U+00FF do; a file with fewer,
+/// such as UTF-8 or Windows-1252 text with stray NUL bytes, is refused. The byte order is the one
+/// that reads a line break (`\n` or `\r`). The other order reads one only from U+0A00 or U+0D00,
+/// so a file that reads one in both orders is refused. A file that reads none in either order (one
+/// line) is big-endian when more of its units start with a NUL byte than end with one, as
+/// characters from U+0001 to U+00FF do in that order, and little-endian otherwise. An odd last
+/// byte is left out of the count.
+pub fn utf16_order_without_mark(bytes: &[u8]) -> Option<bool> {
+    let units = bytes.as_chunks::<2>().0;
+    if units.iter().filter(|c| c.contains(&0)).count() * 2 < units.len() {
         return None;
     }
     let is_line_break = |u: u16| matches!(u, 0x0A | 0x0D);
     let lines_le = units.iter().any(|&c| is_line_break(u16::from_le_bytes(c)));
     let lines_be = units.iter().any(|&c| is_line_break(u16::from_be_bytes(c)));
-    if lines_le && lines_be {
-        return None;
+    match (lines_le, lines_be) {
+        (true, true) => None,
+        (false, false) => Some(units.iter().filter(|c| c[0] == 0).count() <= units.iter().filter(|c| c[1] == 0).count()),
+        (_, lines_be) => Some(!lines_be),
     }
-    utf16(bytes, !lines_be).filter(|text| !text.contains('\0'))
+}
+
+/// `bytes`, which hold NUL bytes and no byte-order mark, as UTF-16 in the byte order
+/// [`utf16_order_without_mark`] gives, or `None`. The file must have an even length, and the text
+/// must be valid UTF-16 in that order and hold no NUL character.
+fn utf16_without_mark(bytes: &[u8]) -> Option<String> {
+    utf16(bytes, utf16_order_without_mark(bytes)?).filter(|text| !text.contains('\0'))
 }
 
 /// A row of a datasets CSV: the file line it starts on, its cells, and for each cell whether it was quoted.

crates/engine/src/tests/textimport.rs

@@ -68,6 +68,49 @@ fn type_placed_on_an_artboard_edge_takes_that_artboards_frame() {
     }
 }
 
+/// `text` as UTF-16 without a byte-order mark, little-endian when `le` is set.
+fn utf16(text: &str, le: bool) -> Vec<u8> {
+    text.encode_utf16().flat_map(|u| if le { u.to_le_bytes() } else { u.to_be_bytes() }).collect()
+}
+
+/// Unicode text in UTF-16 without a byte-order mark is read as UTF-16 when at least half of its
+/// two-byte units hold a NUL byte, in the byte order that reads a line break, or for one line, the
+/// order its NUL bytes give. Other bytes keep today's reading.
+#[test]
+fn utf16_without_a_byte_order_mark_places_by_its_nul_bytes_and_line_breaks() {
+    let mut s = session();
+    for le in [true, false] {
+        let r = place_text(&mut s, &utf16("Hello, 世界 café!\r\nSecond line", le), json!({}));
+        assert_eq!(placed(&s, &r).0, "Hello, 世界 café!\nSecond line", "little-endian: {le}");
+        let r = place_text(&mut s, &utf16("One line, 世界", le), json!({}));
+        assert_eq!(placed(&s, &r).0, "One line, 世界", "little-endian: {le}");
+    }
+    // One line with NUL bytes in both places (U+4E00 ends in a zero byte) takes the order more of
+    // its units give, and a tie is little-endian.
+    let r = place_text(&mut s, &utf16("一AB", false), json!({}));
+    assert_eq!(placed(&s, &r).0, "一AB");
+    let r = place_text(&mut s, &utf16("一A", true), json!({}));
+    assert_eq!(placed(&s, &r).0, "一A");
+    // Not read as UTF-16, so read as UTF-8 or else Windows-1252 with NUL bytes dropped: an odd
+    // length, an unpaired surrogate, a NUL character, and line breaks in both byte orders (U+0A00).
+    let r = place_text(&mut s, b"A\0B", json!({}));
+    assert_eq!(placed(&s, &r).0, "AB");
+    let r = place_text(&mut s, b"\0a\0b\xd8\0", json!({}));
+    assert_eq!(placed(&s, &r).0, "abØ");
+    let r = place_text(&mut s, b"\0\0\xc3\xa9", json!({}));
+    assert_eq!(placed(&s, &r).0, "é");
+    let r = place_text(&mut s, &utf16("A\u{0A00}\nB", true), json!({}));
+    assert_eq!(placed(&s, &r).0, "A\n\nB");
+    // Fewer than half of the two-byte units hold a NUL byte: UTF-8, and Windows-1252.
+    let r = place_text(&mut s, "Fünf\0 Zeilen!".as_bytes(), json!({}));
+    assert_eq!(placed(&s, &r).0, "Fünf Zeilen!");
+    let r = place_text(&mut s, b"caf\xe9\0x", json!({}));
+    assert_eq!(placed(&s, &r).0, "caféx");
+    // ANSI reads each byte as a character of the 8-bit set; NUL and other control bytes are dropped.
+    let r = place_text(&mut s, &utf16("Hi, 世界", true), json!({"text": {"characterSet": "ansi"}}));
+    assert_eq!(placed(&s, &r).0, "Hi, NLu");
+}
+
 #[test]
 fn text_files_report_and_queue_with_their_options() {
     let mut s = session();

crates/engine/src/tests/variables.rs

@@ -2201,6 +2201,16 @@ fn datasets_csv_reads_utf16_files_without_a_byte_order_mark() {
         let e = import(&mut s, &bytes).unwrap_err().to_string();
         assert!(e.contains("CSV has a header but no data rows"), "{e}");
     }
+    // With no line break, the side of each unit that holds the NUL byte gives the byte order, and a
+    // tie is little-endian.
+    let order = crate::cmd::utf16_order_without_mark;
+    assert_eq!(order(&utf16("City", false)), Some(false));
+    assert_eq!(order(&utf16("City", true)), Some(true));
+    assert_eq!(order(b"\0aa\0"), Some(true));
+    assert_eq!(order(&[utf16("City\n", false), vec![0x0A]].concat()), Some(false), "an odd last byte is left out");
+    // Read big-endian, this one-line file ends in an unpaired surrogate.
+    let e = import(&mut s, b"\0a\0b\xd8\0").unwrap_err().to_string();
+    assert!(e.contains("CSV holds NUL bytes"), "{e}");
     import(&mut s, &utf16("City\tName\n一二三丁目\tKai\n", true)).unwrap();
     assert_eq!(value(&s, "Data Set 1"), text_value("一二三丁目"));
     // Most units hold their NUL byte where a big-endian file would (U+4E00); the line breaks

crates/ui-egui/src/dialogs/graph_data.rs

@@ -149,7 +149,10 @@ pub(crate) fn grid_from_text(text: &str) -> Grid {
 }
 
 /// A text file's bytes as text: UTF-16 with its byte-order mark (spreadsheets' "Unicode Text"),
-/// else UTF-8 without one.
+/// else UTF-16 without one when the bytes hold NUL bytes and Import Data Sets' rule gives them a
+/// byte order ([`vectorcraft_engine::cmd::utf16_order_without_mark`]), else UTF-8 with or without
+/// one. UTF-16 is read as it is after a byte-order mark: an odd last byte is left out, and an
+/// unpaired surrogate reads as U+FFFD.
 pub(crate) fn decode(bytes: &[u8]) -> String {
     let utf16 = |rest: &[u8], le: bool| {
         let units: Vec<u16> = rest.as_chunks::<2>().0.iter().map(|p| if le { u16::from_le_bytes(*p) } else { u16::from_be_bytes(*p) }).collect();
@@ -159,7 +162,10 @@ pub(crate) fn decode(bytes: &[u8]) -> String {
         [0xFF, 0xFE, rest @ ..] => utf16(rest, true),
         [0xFE, 0xFF, rest @ ..] => utf16(rest, false),
         [0xEF, 0xBB, 0xBF, rest @ ..] => String::from_utf8_lossy(rest).into_owned(),
-        _ => String::from_utf8_lossy(bytes).into_owned(),
+        _ => match bytes.contains(&0).then(|| vectorcraft_engine::cmd::utf16_order_without_mark(bytes)).flatten() {
+            Some(le) => utf16(bytes, le),
+            None => String::from_utf8_lossy(bytes).into_owned(),
+        },
     }
 }
 
@@ -526,6 +532,28 @@ mod tests {
         assert_eq!(grid_from_text(&wide)[0].len(), MAX_COLS);
     }
 
+    #[test]
+    fn utf16_without_a_byte_order_mark_reads_by_its_line_breaks() {
+        let text = "\tSales é\r\nQ1\t1\n";
+        let le: Vec<u8> = text.encode_utf16().flat_map(u16::to_le_bytes).collect();
+        let be: Vec<u8> = text.encode_utf16().flat_map(u16::to_be_bytes).collect();
+        assert_eq!(decode(&le), text, "little-endian");
+        assert_eq!(decode(&be), text, "big-endian");
+        assert_eq!(grid_from_text(&decode(&le)), g(&[&["", "Sales é"], &["Q1", "1"]]));
+        // One row has no line break: the side of each unit that holds the NUL byte gives the order.
+        for (bytes, order) in [(b"\0Q\x001\0\t\x002".to_vec(), "big-endian"), (b"Q\x001\0\t\x002\0".to_vec(), "little-endian")] {
+            assert_eq!(decode(&bytes), "Q1\t2", "one line, {order}");
+        }
+        // A file cut short by one byte, or with an unpaired surrogate, reads as it would after a
+        // byte-order mark.
+        assert_eq!(decode(&[le.as_slice(), &[0x0A]].concat()), text, "odd length");
+        assert_eq!(decode(&[le.as_slice(), &[0x00, 0xD8]].concat()), format!("{text}\u{FFFD}"), "unpaired surrogate");
+        // A line break in both byte orders leaves the order unknown, and UTF-8 with a stray NUL
+        // byte isn't UTF-16: both read as UTF-8, as before.
+        assert_eq!(decode(&[0x0A, 0, 0, 0x0A]), "\n\0\0\n");
+        assert_eq!(decode(b"a,b\n1,2\0"), "a,b\n1,2\0");
+    }
+
     #[test]
     fn a_sheet_sent_wider_and_taller_than_a_graph_is_cut_to_one() {
         // One long row and one long column: their product would be ten billion cells.

docs/mcp.md

@@ -3273,8 +3273,9 @@ already walks Image Trace. Short agent entry points:
   byte order that reads a line break (`\n` or `\r`). It is refused when fewer of its units hold a NUL byte (UTF-8 or
   Windows-1252 text with stray NUL bytes, or UTF-16 text made mostly of other characters, which needs a byte-order
   mark), when both byte orders read a line break (the text holds U+0A00 or U+0D00), when it has an odd length, and
-  when it isn't valid UTF-16 in that order or holds a NUL character. A file with no line break has no data rows and is
-  refused for that. `csv` and `data` are read as the text they hold.
+  when it isn't valid UTF-16 in that order or holds a NUL character. A file with no line break is read big-endian when
+  more of its two-byte units start with a NUL byte than end with one, and little-endian otherwise; it has no data rows,
+  so the import is refused. `csv` and `data` are read as the text they hold.
 - Variable libraries: `variable.loadLibrary {path|xml|dataBase64}` (the Variables panel menu's Load Variable
   Library…) replaces the document's variables and data sets with a variable library's in one undo step. A load, like a
   `dataset.import`, adds no undo step when it leaves the variables and data sets as they were. The library
@@ -7243,9 +7244,14 @@ the document; so does `object.expand`.
 Placing a text file (`.txt`) sets it as area type: `file.place {path | name+dataBase64, text?: {characterSet?:
 "unicode" | "ansi", platform?: "windows" | "mac", removeLineReturns?, removeParagraphReturns?, replaceSpaces?: n}}`
 (Text Import Options). Unicode reads UTF-8, or UTF-16 with a byte-order mark; ANSI reads Windows-1252 (Mac Roman on
-`mac`). `removeLineReturns` joins the lines of each block into one paragraph, `removeParagraphReturns` drops blank
-lines, `replaceSpaces: 3` turns runs of 3 or more spaces into tabs. The frame fills `rect`, the replaced object's
-bounds, or the artboard less a 36 pt margin.
+`mac`). Unicode reads a file without a byte-order mark as UTF-16 by the rule `dataset.import` uses: at least half of
+its two-byte units hold a NUL byte, in the byte order that reads a line break, or for a file with one line, big-endian
+when more of its units start with a NUL byte than end with one, and little-endian otherwise. Any other file is read as
+UTF-8, else as the platform's 8-bit set: one with fewer NUL bytes, one with an odd length, one where both byte orders
+read a line break, and one that isn't valid UTF-16 in that order or holds a NUL character. `removeLineReturns` joins
+the lines of each block into one paragraph, `removeParagraphReturns` drops blank lines, `replaceSpaces: 3` turns runs
+of 3 or more spaces into tabs. The frame fills `rect`, the replaced object's bounds, or the artboard less a 36 pt
+margin.
 
 ```json
 {"name":"run_command","arguments":{"command":"file.place","params":{"path":"/tmp/notes.txt","text":{"removeLineReturns":true,"removeParagraphReturns":true}}}}

docs/target-app-parity.md

@@ -806,6 +806,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur
   - Export for Screens numbers distinct row files whose names collide, compared without case, as `-2`, `-3`… instead of leaving them out; identical rows still write one file. It puts row folders that differ only in case into one folder (`document.exportForScreens`).
   - Variables: Import Data Sets… (`dataset.import`) reads tab-separated files as well as comma-separated ones. Every header cell is a variable, and the rows replace the document's data sets as Data Set 1, Data Set 2 and so on. An import error about a row's values gives the file line the row starts on.
   - Variables: Import Data Sets… (`dataset.import`) reads a UTF-16 file without a byte-order mark, instead of refusing it, when at least half of its two-byte units hold a NUL byte; it uses the byte order that reads its line breaks.
+  - Graphs: the Graph Data window's Import Data reads a UTF-16 text file without a byte-order mark by the same rule as Import Data Sets…, instead of filling the cells with NUL characters. It reads a file cut short by one byte, or with an unpaired surrogate, as it reads one after a byte-order mark.
   - Variables: Import Data Sets… (`dataset.import`) defines the variables a file's header names that the document lacks, unbound. `#Shown` defines a visibility variable, `@Photo` defines an image variable, and a header without a prefix defines a text variable. A header that is a variable's exact name keeps naming it, and any other `%` header (graph data files) is refused.
   - Variables: Load Variable Library… in the Variables panel menu (`variable.loadLibrary`) replaces the document's variables and data sets with a variable library's, read from its XML file, in one undo step: every data set is deleted, and every variable is deleted with its bindings and values, except a text variable the library also declares, which stays bound so the library's data sets change its objects (#4136); then the library's other text variables are defined, unbound, and its data sets take the text of each value's `p` elements, one paragraph each. Variables of other kinds are left out with a warning that names their kinds.
   - Variables: Save Variable Library… in the Variables panel menu (`variable.saveLibrary`) writes the document's text variables and data sets as a variable library's XML file, which Load Variable Library reads back with the same text variables, data set names and text values. Variables of other kinds are left out with a warning that names their kinds.
@@ -1818,6 +1819,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur
   - A Word paragraph justified with a low, medium or high kashida length places justified with Insert Kashidas set to Short, Medium or Long (`file.place`, `crates/engine/src/tests/textimport.rs`).
   - File › Place reads Word 97-2003 `.doc` documents as area type with the same formatting, read from the main text's pieces, the style sheet's paragraph and character styles and direct formatting. Table rows are paragraphs with their cells separated by tabs and fields place as their results; fields without a result, hidden and deleted text, pictures, headers, footers, notes, comments and text boxes are left out, and `warnings` names each kind left out or changed. Encrypted documents, documents last saved with fast save and Word 6.0 and Word 95 documents are refused (`file.place`, `crates/engine/src/tests/textimport.rs`).
   - Placing a Word 97-2003 `.doc` document works out each style's formatting once and each property run's once, and refuses property pages that are out of order, listed twice or overlap, a style sheet that holds fewer styles than it counts, styles based on each other in a loop, and more than 100,000 runs of character or of paragraph properties, 16 MiB of properties or 100,000 paragraphs, hidden ones included (`file.place`, `crates/engine/src/tests/textimport.rs`).
+  - File › Place of a text file with the Unicode character set reads UTF-16 without a byte-order mark by the rule Import Data Sets… uses (at least half of its two-byte units hold a NUL byte; the byte order that reads a line break; valid UTF-16 with no NUL character), instead of reading it as UTF-8 with other characters in place of those outside U+0001 to U+00FF (`file.place`, `crates/engine/src/tests/textimport.rs`).
   - On the desktop, File › Place reads and decodes files off the UI thread, one at a time, and places each once it is read, with "Placing" in the status bar and the busy spinner until then: the Place dialog's OK, a file dropped on the canvas, a click or drag of the place cursor, and the OK of the PDF, DXF and text import options. The files and thumbnails the place cursor is loaded with are read the same way, and the Place dialog's lines about its files are read on a thread of their own. Files are placed in the order they were started; a file is not placed while a dialog is open or a drag is under way, and a file whose document was closed meanwhile is not placed. `file.place` and `file.place.queue` over MCP and scripts reply as before.
   - A file read for `file.place`, or files read for the place cursor, apart from the session (`Session::place_read`, `Session::place_queue_read`) go only into the document they were read for: while another document is active, or once that document is closed, they are refused with "the document it was prepared for is no longer active" and no document, selection, tool or undo step changes (`crates/engine/src/tests/place.rs`).
   - On the desktop, Paste from a menu or shortcut captures another app's bitmap or reads a copied image file on the UI thread, then decodes the image on the file-reading worker. The status bar shows "Pasting" until the image is pasted. Windows device-independent bitmaps and Linux clipboard images are encoded as PNG on that worker. A Copy or Cut made while the image is read keeps its internal clipboard. `edit.paste*` over MCP and scripts reads the bitmap at once.