diff --git a/crates/engine/src/cmd/place.rs b/crates/engine/src/cmd/place.rs index 20f19ef803..24dba09f37 100644 --- a/crates/engine/src/cmd/place.rs +++ b/crates/engine/src/cmd/place.rs @@ -75,7 +75,7 @@ pub fn specs() -> Vec { "Place…", ["File"], Some("Cmd+Shift+P"), - "{path | name | dataBase64 (a file's path, or its name with its bytes), link?: true (a raster image keeps its file's path; a VectorCraft document at path becomes one locked placed document showing artboard `page`, linked to the file and read again when it changes (links.update; saving it in this app updates it at once), drawn as vectors in every output; so does a PDF/.ai page at path, showing its `crop` box, unless password or textAs: \"outlines\" is given; false: an editable copy of its art, as from dataBase64; other files are embedded), text?: {characterSet?: \"unicode\" (UTF-8, or UTF-16 with a byte-order mark; other bytes as the platform's 8-bit set) | \"ansi\" (the platform's 8-bit set), platform?: \"windows\" (Windows-1252) | \"mac\" (Mac Roman), removeLineReturns?: false (each block of lines becomes one paragraph; blank lines end paragraphs), removeParagraphReturns?: false (drop blank lines), replaceSpaces?: n (runs of n ≥ 2 spaces become a tab)} (a .txt file, placed as area type filling rect, the replaced object's bounds, or else the artboard less a 36 pt margin), template?: false (onto a new locked template layer below the current layer), replace?: false (swap the one selected object, keeping its stacking place and transform; no at/rect; refused while an object of a symbol instance's art is selected), at?: [x, y] centre (default: the first artboard's centre), rect?: [x, y, width, height] fit inside; aspect kept (wins over at), page?: 1 (PDF/.ai page, or a native document's artboard), crop?: \"crop\" (clipped to that page or artboard, default; an SVG, only when given: framed and clipped by its viewBox, else by its art's bounds) | \"bounding\" (the bounds of the art that shows, with its strokes and effects, as an export measures them: hidden objects and guides left out) | \"art\" | \"trim\" | \"bleed\" | \"media\" (a PDF page's boxes), password? (an encrypted PDF), dxf?: {…the DXF options of document.open} (a .dxf drawing: fit fills the artboard under at, else the first; fitted or uncentred art lands with its drawing's artboard on that artboard, else it is centred on at)} → {ids, name, format, linked, width, height, warnings, image?: the id of the image of a DCS file placed in a clip group, which the Links commands take}. Raster images come in at 100% of their physical size (the file's ppi, else 72); SVG, DXF, HP-GL (.plt, .hpgl, .hgl, .hpg), PDF/.ai, EMF/WMF and PICT (.pct, .pic, .pict; clipped to the picture's frame), EPS (clipped to its bounding box; as document.open reads it), a .ai saved without PDF compatibility (from its editing data, as document.open reads it: the art of the layers that show on artboard `page`, clipped to that artboard unless crop is \"bounding\") and native documents as one group, with the images, symbols, patterns, swatches and brushes they use. A DCS file (an EPS file with one program per printing plate, DCS 1.0 or 2.0) is placed only from its path with link (else an error, as are document.open and links.embed): one linked image of its plates mixed into CMYK, clipped by the clipping path its composite is drawn through, its spot plates added as spot swatches in the process equivalents the file gives (a swatch of the same name is kept, with a warning); print separations and Separations Preview show each plate on its own ink, and plate files of their own are read from its folder. One undo step; selects what it placed (unless on a template layer); never touches the clipboard. A PNG, JPEG, GIF or WebP image whose pixels can't be read (a file cut short) is an error, and nothing is placed", + "{path | name | dataBase64 (a file's path, or its name with its bytes), link?: true (a raster image keeps its file's path; a VectorCraft document at path becomes one locked placed document showing artboard `page`, linked to the file and read again when it changes (links.update; saving it in this app updates it at once), drawn as vectors in every output; so does a PDF/.ai page at path, showing its `crop` box, unless password or textAs: \"outlines\" is given; false: an editable copy of its art, as from dataBase64; other files are embedded), text?: {characterSet?: \"unicode\" (UTF-8, or UTF-16 with a byte-order mark, or without one by dataset.import's rule: an even length, at least half of its two-byte units holding a NUL byte, the byte order that reads a line break, valid UTF-16 with no NUL character; other bytes as the platform's 8-bit set) | \"ansi\" (the platform's 8-bit set), platform?: \"windows\" (Windows-1252) | \"mac\" (Mac Roman), removeLineReturns?: false (each block of lines becomes one paragraph; blank lines end paragraphs), removeParagraphReturns?: false (drop blank lines), replaceSpaces?: n (runs of n ≥ 2 spaces become a tab)} (a .txt file, placed as area type filling rect, the replaced object's bounds, or else the artboard less a 36 pt margin), template?: false (onto a new locked template layer below the current layer), replace?: false (swap the one selected object, keeping its stacking place and transform; no at/rect; refused while an object of a symbol instance's art is selected), at?: [x, y] centre (default: the first artboard's centre), rect?: [x, y, width, height] fit inside; aspect kept (wins over at), page?: 1 (PDF/.ai page, or a native document's artboard), crop?: \"crop\" (clipped to that page or artboard, default; an SVG, only when given: framed and clipped by its viewBox, else by its art's bounds) | \"bounding\" (the bounds of the art that shows, with its strokes and effects, as an export measures them: hidden objects and guides left out) | \"art\" | \"trim\" | \"bleed\" | \"media\" (a PDF page's boxes), password? (an encrypted PDF), dxf?: {…the DXF options of document.open} (a .dxf drawing: fit fills the artboard under at, else the first; fitted or uncentred art lands with its drawing's artboard on that artboard, else it is centred on at)} → {ids, name, format, linked, width, height, warnings, image?: the id of the image of a DCS file placed in a clip group, which the Links commands take}. Raster images come in at 100% of their physical size (the file's ppi, else 72); SVG, DXF, HP-GL (.plt, .hpgl, .hgl, .hpg), PDF/.ai, EMF/WMF and PICT (.pct, .pic, .pict; clipped to the picture's frame), EPS (clipped to its bounding box; as document.open reads it), a .ai saved without PDF compatibility (from its editing data, as document.open reads it: the art of the layers that show on artboard `page`, clipped to that artboard unless crop is \"bounding\") and native documents as one group, with the images, symbols, patterns, swatches and brushes they use. A DCS file (an EPS file with one program per printing plate, DCS 1.0 or 2.0) is placed only from its path with link (else an error, as are document.open and links.embed): one linked image of its plates mixed into CMYK, clipped by the clipping path its composite is drawn through, its spot plates added as spot swatches in the process equivalents the file gives (a swatch of the same name is kept, with a warning); print separations and Separations Preview show each plate on its own ink, and plate files of their own are read from its folder. One undo step; selects what it placed (unless on a template layer); never touches the clipboard. A PNG, JPEG, GIF or WebP image whose pixels can't be read (a file cut short) is an error, and nothing is placed", has_doc, place ), diff --git a/crates/engine/src/cmd/place/text.rs b/crates/engine/src/cmd/place/text.rs index 25f5f36dbd..7d8afc92d4 100644 --- a/crates/engine/src/cmd/place/text.rs +++ b/crates/engine/src/cmd/place/text.rs @@ -3,6 +3,7 @@ use serde_json::Value; +use super::super::utf16_order_without_mark; use super::{Point, Rect, Result, bad, bool_or, fileio, shapes, str_param}; /// Text files as `file.place` reports them (not a format `document.open` reads). @@ -76,8 +77,10 @@ pub(super) fn import(bytes: &[u8], o: TextOptions, cmd: &str) -> Result Ok(clean(&decode(bytes, o), o)) } -/// `bytes` as text: Unicode (UTF-8, or UTF-16 with a byte-order mark; bytes that aren't UTF-8 are -/// read as the platform's 8-bit set), or the platform's 8-bit character set. +/// `bytes` as text: Unicode (UTF-8, or UTF-16 with a byte-order mark, or without one when the +/// file has an even length, reads as valid UTF-16 with no NUL character in the byte order +/// [`utf16_order_without_mark`] gives; bytes that aren't UTF-8 are read as the platform's 8-bit +/// set), or the platform's 8-bit character set. fn decode(bytes: &[u8], o: TextOptions) -> String { let eight_bit = |b: &[u8]| -> String { b.iter().map(|&c| if o.mac { mac_roman(c) } else { windows_1252(c) }).collect() }; if o.ansi { @@ -91,7 +94,17 @@ fn decode(bytes: &[u8], o: TextOptions) -> String { [0xEF, 0xBB, 0xBF, rest @ ..] => String::from_utf8_lossy(rest).into_owned(), [0xFF, 0xFE, rest @ ..] => utf16(rest, true), [0xFE, 0xFF, rest @ ..] => utf16(rest, false), - _ => std::str::from_utf8(bytes).map_or_else(|_| eight_bit(bytes), str::to_string), + _ => { + let without_mark = (bytes.len().is_multiple_of(2) && bytes.contains(&0)) + .then(|| utf16_order_without_mark(bytes)) + .flatten() + .and_then(|le| { + let units = bytes.as_chunks::<2>().0.iter().map(|&c| if le { u16::from_le_bytes(c) } else { u16::from_be_bytes(c) }); + char::decode_utf16(units).collect::>().ok() + }) + .filter(|text| !text.contains('\0')); + without_mark.unwrap_or_else(|| std::str::from_utf8(bytes).map_or_else(|_| eight_bit(bytes), str::to_string)) + } } } diff --git a/crates/engine/src/tests/textimport.rs b/crates/engine/src/tests/textimport.rs index 0749964ee6..b369346096 100644 --- a/crates/engine/src/tests/textimport.rs +++ b/crates/engine/src/tests/textimport.rs @@ -68,6 +68,49 @@ fn type_placed_on_an_artboard_edge_takes_that_artboards_frame() { } } +/// `text` as UTF-16 without a byte-order mark, little-endian when `le` is set. +fn utf16(text: &str, le: bool) -> Vec { + text.encode_utf16().flat_map(|u| if le { u.to_le_bytes() } else { u.to_be_bytes() }).collect() +} + +/// Unicode text in UTF-16 without a byte-order mark is read as UTF-16 when at least half of its +/// two-byte units hold a NUL byte, in the byte order that reads a line break, or for one line, the +/// order its NUL bytes give. Other bytes keep today's reading. +#[test] +fn utf16_without_a_byte_order_mark_places_by_its_nul_bytes_and_line_breaks() { + let mut s = session(); + for le in [true, false] { + let r = place_text(&mut s, &utf16("Hello, 世界 café!\r\nSecond line", le), json!({})); + assert_eq!(placed(&s, &r).0, "Hello, 世界 café!\nSecond line", "little-endian: {le}"); + let r = place_text(&mut s, &utf16("One line, 世界", le), json!({})); + assert_eq!(placed(&s, &r).0, "One line, 世界", "little-endian: {le}"); + } + // One line with NUL bytes in both places (U+4E00 ends in a zero byte) takes the order more of + // its units give, and a tie is little-endian. + let r = place_text(&mut s, &utf16("一AB", false), json!({})); + assert_eq!(placed(&s, &r).0, "一AB"); + let r = place_text(&mut s, &utf16("一A", true), json!({})); + assert_eq!(placed(&s, &r).0, "一A"); + // Not read as UTF-16, so read as UTF-8 or else Windows-1252 with NUL bytes dropped: an odd + // length, an unpaired surrogate, a NUL character, and line breaks in both byte orders (U+0A00). + let r = place_text(&mut s, b"A\0B", json!({})); + assert_eq!(placed(&s, &r).0, "AB"); + let r = place_text(&mut s, b"\0a\0b\xd8\0", json!({})); + assert_eq!(placed(&s, &r).0, "abØ"); + let r = place_text(&mut s, b"\0\0\xc3\xa9", json!({})); + assert_eq!(placed(&s, &r).0, "é"); + let r = place_text(&mut s, &utf16("A\u{0A00}\nB", true), json!({})); + assert_eq!(placed(&s, &r).0, "A\n\nB"); + // Fewer than half of the two-byte units hold a NUL byte: UTF-8, and Windows-1252. + let r = place_text(&mut s, "Fünf\0 Zeilen!".as_bytes(), json!({})); + assert_eq!(placed(&s, &r).0, "Fünf Zeilen!"); + let r = place_text(&mut s, b"caf\xe9\0x", json!({})); + assert_eq!(placed(&s, &r).0, "caféx"); + // ANSI reads each byte as a character of the 8-bit set; NUL and other control bytes are dropped. + let r = place_text(&mut s, &utf16("Hi, 世界", true), json!({"text": {"characterSet": "ansi"}})); + assert_eq!(placed(&s, &r).0, "Hi, NLu"); +} + #[test] fn text_files_report_and_queue_with_their_options() { let mut s = session(); diff --git a/docs/mcp.md b/docs/mcp.md index a68645348e..7bc318fb44 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -7058,9 +7058,14 @@ the document; so does `object.expand`. Placing a text file (`.txt`) sets it as area type: `file.place {path | name+dataBase64, text?: {characterSet?: "unicode" | "ansi", platform?: "windows" | "mac", removeLineReturns?, removeParagraphReturns?, replaceSpaces?: n}}` (Text Import Options). Unicode reads UTF-8, or UTF-16 with a byte-order mark; ANSI reads Windows-1252 (Mac Roman on -`mac`). `removeLineReturns` joins the lines of each block into one paragraph, `removeParagraphReturns` drops blank -lines, `replaceSpaces: 3` turns runs of 3 or more spaces into tabs. The frame fills `rect`, the replaced object's -bounds, or the artboard less a 36 pt margin. +`mac`). Unicode reads a file without a byte-order mark as UTF-16 by the rule `dataset.import` uses: at least half of +its two-byte units hold a NUL byte, in the byte order that reads a line break, or for a file with one line, big-endian +when more of its units start with a NUL byte than end with one, and little-endian otherwise. Any other file is read as +UTF-8, else as the platform's 8-bit set: one with fewer NUL bytes, one with an odd length, one where both byte orders +read a line break, and one that isn't valid UTF-16 in that order or holds a NUL character. `removeLineReturns` joins +the lines of each block into one paragraph, `removeParagraphReturns` drops blank lines, `replaceSpaces: 3` turns runs +of 3 or more spaces into tabs. The frame fills `rect`, the replaced object's bounds, or the artboard less a 36 pt +margin. ```json {"name":"run_command","arguments":{"command":"file.place","params":{"path":"/tmp/notes.txt","text":{"removeLineReturns":true,"removeParagraphReturns":true}}}} diff --git a/docs/target-app-parity.md b/docs/target-app-parity.md index 8fe37830a2..f124ce3249 100644 --- a/docs/target-app-parity.md +++ b/docs/target-app-parity.md @@ -1749,6 +1749,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur - A Word paragraph justified with a low, medium or high kashida length places justified with Insert Kashidas set to Short, Medium or Long (`file.place`, `crates/engine/src/tests/textimport.rs`). - File › Place reads Word 97-2003 `.doc` documents as area type with the same formatting, read from the main text's pieces, the style sheet's paragraph and character styles and direct formatting. Table rows are paragraphs with their cells separated by tabs and fields place as their results; fields without a result, hidden and deleted text, pictures, headers, footers, notes, comments and text boxes are left out, and `warnings` names each kind left out or changed. Encrypted documents, documents last saved with fast save and Word 6.0 and Word 95 documents are refused (`file.place`, `crates/engine/src/tests/textimport.rs`). - Placing a Word 97-2003 `.doc` document works out each style's formatting once and each property run's once, and refuses property pages that are out of order, listed twice or overlap, a style sheet that holds fewer styles than it counts, styles based on each other in a loop, and more than 100,000 runs of character or of paragraph properties, 16 MiB of properties or 100,000 paragraphs, hidden ones included (`file.place`, `crates/engine/src/tests/textimport.rs`). + - File › Place of a text file with the Unicode character set reads UTF-16 without a byte-order mark by the rule Import Data Sets… uses (at least half of its two-byte units hold a NUL byte; the byte order that reads a line break; valid UTF-16 with no NUL character), instead of reading it as UTF-8 with other characters in place of those outside U+0001 to U+00FF (`file.place`, `crates/engine/src/tests/textimport.rs`). - On the desktop, File › Place reads and decodes files off the UI thread, one at a time, and places each once it is read, with "Placing" in the status bar and the busy spinner until then: the Place dialog's OK, a file dropped on the canvas, a click or drag of the place cursor, and the OK of the PDF, DXF and text import options. The files and thumbnails the place cursor is loaded with are read the same way, and the Place dialog's lines about its files are read on a thread of their own. Files are placed in the order they were started; a file is not placed while a dialog is open or a drag is under way, and a file whose document was closed meanwhile is not placed. `file.place` and `file.place.queue` over MCP and scripts reply as before. - A file read for `file.place`, or files read for the place cursor, apart from the session (`Session::place_read`, `Session::place_queue_read`) go only into the document they were read for: while another document is active, or once that document is closed, they are refused with "the document it was prepared for is no longer active" and no document, selection, tool or undo step changes (`crates/engine/src/tests/place.rs`). - On the desktop, Paste from a menu or shortcut captures another app's bitmap or reads a copied image file on the UI thread, then decodes the image on the file-reading worker. The status bar shows "Pasting" until the image is pasted. Windows device-independent bitmaps and Linux clipboard images are encoded as PNG on that worker. A Copy or Cut made while the image is read keeps its internal clipboard. `edit.paste*` over MCP and scripts reads the bitmap at once.