Merge #6491: fix(render): a non-embedded Times New Roman draws the glyphs its codes name (hayro-interpret patch 88)

# Conflicts:
#	crates/render/src/raster/tests.rs
#	vendor/README.md
This commit is contained in:
bflatastic
2026-11-09 08:07:55 +07:00
6 changed files with 241 additions and 3 deletions
+4
View File
@@ -240,6 +240,10 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
## Progress log
- **2026-10-13 (M8, redact by regex):** `redact_mark` takes `regex` (caller expression, linear-time engine, length/size/nesting caps, empty-matching refused) with optional `validate` (luhn, mod97, iban). Headless only; issue #1886.
- **2026-11-07 (M2, #6449):** A non-embedded Times New Roman set as a TrueType CID font (`TimesNewRoman,Bold`, `TimesNewRomanPSMT` and the other faces) draws the glyphs its codes name, as the non-embedded Arial does: its first 258 glyphs follow the same standard Macintosh order, which the embedded Times New Roman subsets of a public French form confirm in 941 of 942 `/ToUnicode` entries (vendored hayro-interpret patch 88). That form drew "Salarié(e) du régime général" as "Salari²*g+f w't²i lo g'ì²p²tcn" and "changement" as "ej angement". Times and Times New Roman MT Std are not read this way.
- **2026-11-05 (M2, #5889):** A non-embedded Arial set as a TrueType CID font (`CIDFontType2`, `Identity-H`, as Word and PDFMaker write it) draws the glyphs its codes name. Those codes are glyph indexes in the author's Arial, whose first 258 glyphs follow the standard Macintosh order, so the standard substitute now draws each glyph by name (vendored hayro-interpret patch 76). Before, it took each code as one of its own glyph indexes, and "Low Power" drew as "@ck 'Dck Yf". Arial Narrow, Black and Unicode MS, and `CIDFontType0` fonts, are not read this way. Copying or searching those runs still needs a `/ToUnicode`.
- **2026-11-07 (M7, #2964 / #3185):** Supported Form text can be prepared for direct editing; groups left grouped (unsafe compositing, text nested past eight levels) are reported. Legacy Thai keeps the original font programs, Regular/Bold and text settings, also across same-face subsets; the narrow THSarabunPSK map repair is checked glyph by glyph against the font program; cross-family replacement is explicit. Builds on #3067 and #2811. Unsafe groups and mixed contextual styles remain partial; estimates unchanged.
- **2026-10-28 (M10, #4119 partial):** Scan & OCR ▸ Enhance scanned file cleans scanned pages in place: background removal (level 0–100, as the camera clean-up), edge shadow removal (book gutters, copier borders) and despeckle. Each scan image is decoded, cleaned and written back into its own object at the same size and colour space (JPEG stays JPEG, others lossless, 1-bit stays 1-bit), so the OCR text layer, comments and links stay; one undo step, shared images once, unsupported images (CMYK, indexed, JPEG 2000, JBIG2, CCITT, broken data) reported. `page_enhance_scan` drives it headlessly. Not yet an option of Recognize text. Estimates unchanged.
- **2026-11-07 (M10, `ocr.ocr-web`, port onto #977):** Web OCR rebuilt on main's engines and preprocessing (#977). `OcrJob::run` is now `OcrJob::steps` a page at a time: the page renders, crops to the region and cleans each page, a Web Worker runs ocrs on the result, and `OcrShot::place` maps the words back through the region and preprocessing. Every strategy reads like this when only one engine is present, and the web build has only ocrs. The downloaded models are installed into the engine cache (`ocr::install`), so the Verify screen and the engine-status checks see them too. The batch keeps main's skipped-files bucket, low-confidence notes and cancel. The translations moved to craft-i18n (storytold/craft-i18n#141). The Verify screen starts the model download itself and runs once the models land; before, on a fresh page it failed with the desktop's "set PDFCRAFT_MODELS" message. Known limits: the cleaning steps, and the Verify screen's Run OCR and re-reads, run on the page. Estimates unchanged.
+4
View File
@@ -8821,6 +8821,10 @@ mod symbolic_truetype_cmap;
// length gives (vendored hayro-interpret patch 83).
mod truetype_loca_format;
// An Arial (#5889) or Times New Roman (#6449) the PDF doesn't embed draws the glyphs its codes
// name.
mod non_embedded_arial;
/// Render `pdf`'s first page at 72 dpi with no time limit beyond the test's own.
fn render_first_page(pdf: Vec<u8>) -> RenderedPage {
let mut r = PageRenderer::new(Arc::new(pdf), RenderConfig::default());
@@ -0,0 +1,150 @@
//! An Arial or a Times New Roman the PDF doesn't embed draws the glyphs its codes name (vendored
//! hayro-interpret patches 76 and 88, #5889, #6449). Word and PDFMaker write such a font as a
//! Type 0 `CIDFontType2` with `Identity-H`, often without `/ToUnicode`, so each code is a glyph
//! index in the author's font. Both number their first 258 glyphs in the standard Macintosh order
//! of the TrueType `post` table (`L` is 47, `o` 82, `w` 90). The standard substitute numbers its
//! glyphs another way: "Low Power" drew as "@ck 'Dck Yf", and "Salarié(e) du régime général" as
//! "Salari²*g+f w't²i lo g'ì²p²tcn".
//!
//! Each document is drawn twice, once as such a file has it and once with a `/ToUnicode` that
//! names each code's character, which draws through the substitute's glyph for that character.
//! The two must be the same pixels.
use super::render_first_page;
/// "Low É" as Arial's glyph indexes: L, o, w, space, Eacute.
const LOW_E_ACUTE: [(u16, char); 5] = [(47, 'L'), (82, 'o'), (90, 'w'), (3, ' '), (101, 'É')];
struct Font<'a> {
/// The `/BaseFont` of both the Type 0 font and its descendant.
base_font: &'a str,
/// The descendant's `/Subtype`.
subtype: &'a str,
/// `true`: the codes are 1, 2, … and a `/CIDToGIDMap` stream gives each its glyph in the author's font.
/// `false`: `/CIDToGIDMap /Identity`, and the codes are the glyph indexes themselves.
mapped: bool,
to_unicode: bool,
}
/// A page showing [`LOW_E_ACUTE`] at 40 pt in `font`, which has no font program.
fn pdf(font: &Font<'_>) -> Vec<u8> {
let codes: Vec<u16> = if font.mapped { (1..=LOW_E_ACUTE.len() as u16).collect() } else { LOW_E_ACUTE.iter().map(|(gid, _)| *gid).collect() };
let hex: String = codes.iter().map(|code| format!("{code:04X}")).collect();
let content = format!("BT /F1 40 Tf 5 15 Td <{hex}> Tj ET");
let bfchar: String = codes.iter().zip(LOW_E_ACUTE).map(|(code, (_, c))| format!("<{code:04X}> <{:04X}>\n", u32::from(c))).collect();
let cmap = format!(
"/CIDInit /ProcSet findresource begin 12 dict begin begincmap
/CIDSystemInfo << /Registry (Adobe) /Ordering (UCS) /Supplement 0 >> def
/CMapName /Adobe-Identity-UCS def /CMapType 2 def
1 begincodespacerange <0000> <FFFF> endcodespacerange
{} beginbfchar
{bfchar}endbfchar
endcmap CMapName currentdict /CMap defineresource pop end end",
codes.len()
);
// CID 0 is .notdef; CIDs 1… give the glyphs in order.
let cid_to_gid: Vec<u8> = std::iter::once(0).chain(LOW_E_ACUTE.iter().map(|(gid, _)| *gid)).flat_map(u16::to_be_bytes).collect();
let Font { base_font, subtype, .. } = font;
let to_unicode = if font.to_unicode { " /ToUnicode 8 0 R" } else { "" };
let map = if font.mapped { "9 0 R" } else { "/Identity" };
let mut out = format!(
"%PDF-1.7
1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj
2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 >> endobj
3 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 220 60] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >> endobj
4 0 obj << /Length {} >> stream
{content}
endstream endobj
5 0 obj << /Type /Font /Subtype /Type0 /BaseFont /{base_font} /Encoding /Identity-H /DescendantFonts [6 0 R]{to_unicode} >> endobj
6 0 obj << /Type /Font /Subtype /{subtype} /BaseFont /{base_font} /CIDSystemInfo << /Registry (Adobe) /Ordering (Identity) /Supplement 0 >>
/FontDescriptor 7 0 R /CIDToGIDMap {map} /DW 1000 >> endobj
7 0 obj << /Type /FontDescriptor /FontName /{base_font} /Flags 32 /FontBBox [-665 -325 2000 1040] /ItalicAngle 0 /Ascent 728 /Descent -210 /CapHeight 716 /StemV 80 >> endobj
8 0 obj << /Length {} >> stream
{cmap}
endstream endobj
9 0 obj << /Length {} >> stream
",
content.len(),
cmap.len(),
cid_to_gid.len()
)
.into_bytes();
out.extend_from_slice(&cid_to_gid);
out.extend_from_slice(b"\nendstream endobj\ntrailer << /Root 1 0 R >>\n%%EOF");
out
}
/// `font` drawn without and with a `/ToUnicode`; the second must draw ink.
fn without_and_with_unicode(font: Font<'_>) -> (Vec<u8>, Vec<u8>) {
let without = render_first_page(pdf(&Font { to_unicode: false, ..font }));
let with = render_first_page(pdf(&Font { to_unicode: true, ..font }));
assert!(with.rgba.as_chunks::<4>().0.iter().any(|pixel| pixel[0] < 128), "{}: the control must draw ink", font.base_font);
(without.rgba.to_vec(), with.rgba.to_vec())
}
/// Each Arial face, as Windows exporters name it and as Word names a face it doesn't embed, draws
/// "Low É" from its glyph indexes, with `/CIDToGIDMap /Identity` and with a map stream. The name
/// is read without regard to case, `MT` included.
#[test]
fn an_arial_the_pdf_does_not_embed_draws_the_glyphs_its_codes_name() {
for base_font in [
"ArialMT",
"Arial-BoldMT",
"Arial-ItalicMT",
"Arial-BoldItalicMT",
"Arial",
"Arial,Bold",
"Arial,BoldItalic",
"ABCDEF+Arial-BoldMT",
"arialmt",
"Arial-Boldmt",
] {
for mapped in [false, true] {
let (without, with) = without_and_with_unicode(Font { base_font, subtype: "CIDFontType2", mapped, to_unicode: false });
assert!(without == with, "{base_font} (CIDToGIDMap {}): the codes draw \"Low É\"", if mapped { "stream" } else { "/Identity" });
}
}
}
/// Each Times New Roman face, as Windows exporters name it and as Word names a face it doesn't
/// embed (`TimesNewRoman,Bold` in #6449), draws "Low É" from its glyph indexes.
#[test]
fn a_times_new_roman_the_pdf_does_not_embed_draws_the_glyphs_its_codes_name() {
for base_font in [
"TimesNewRomanPSMT",
"TimesNewRomanPS-BoldMT",
"TimesNewRomanPS-ItalicMT",
"TimesNewRomanPS-BoldItalicMT",
"TimesNewRoman",
"TimesNewRoman,Bold",
"TimesNewRoman,Italic",
"TimesNewRoman,BoldItalic",
"WFXLUF+TimesNewRoman,Bold",
] {
for mapped in [false, true] {
let (without, with) = without_and_with_unicode(Font { base_font, subtype: "CIDFontType2", mapped, to_unicode: false });
assert!(without == with, "{base_font} (CIDToGIDMap {}): the codes draw \"Low É\"", if mapped { "stream" } else { "/Identity" });
}
}
}
/// Only the faces of Arial and Times New Roman are read in that order: Arial Narrow, Arial Black,
/// Arial Unicode MS, Times and Times New Roman MT Std are other fonts, and a `CIDFontType0` font's
/// CIDs aren't TrueType glyph indexes.
#[test]
fn codes_of_other_fonts_are_not_read_in_the_macintosh_order() {
for (base_font, subtype) in [
("ArialNarrow", "CIDFontType2"),
("ArialNarrow-Bold", "CIDFontType2"),
("Arial-Black", "CIDFontType2"),
("ArialUnicodeMS", "CIDFontType2"),
("ArialMT", "CIDFontType0"),
("Times-Bold", "CIDFontType2"),
("TimesNewRomanMTStd", "CIDFontType2"),
("TimesNewRoman,Bold", "CIDFontType0"),
] {
let (without, with) = without_and_with_unicode(Font { base_font, subtype, mapped: false, to_unicode: false });
assert!(without != with, "{base_font} ({subtype}) is not read in the Macintosh order");
}
}
+2 -2
View File
@@ -794,8 +794,8 @@ milestone = "M2"
status = "partial"
commands = []
tools = []
evidence = ["crates/render/src/raster/tests.rs::huge_cid_width_ranges_terminate", "crates/render/src/raster/tests.rs::non_embedded_japanese_cid_fonts_draw_with_a_craft_fonts_face", "crates/render/src/raster/tests.rs::mincho_substitute_covers_half_width_katakana", "crates/render/src/raster/tests.rs::non_embedded_chinese_and_korean_cid_fonts_draw_with_an_installed_face", "crates/engine/src/tests.rs::installed_cjk_fonts_draw_in_the_viewer_but_not_in_an_export", "crates/render/src/text/tests.rs::identity_to_unicode_names_give_text", "crates/fonts/src/pdf/tests.rs::identity_to_unicode_names_read_codes_as_unicode", "crates/render/src/raster/tests.rs::a_soft_hyphen_a_substitute_lacks_draws_its_hyphen"]
notes = "CID-to-Unicode tables are M2.4; a non-embedded Adobe-Japan1 font is drawn in a craft-fonts Japanese face (Mincho or Gothic by name, else by the descendant font descriptor's Serif flag; bold Gothic for a Bold name or a `/FontWeight` of 600 or more; BIZ UDMincho for Mincho, which covers half-width katakana) when built with craft-fonts; the desktop viewer can fall back to an installed CJK face for GB1, CNS1, Korea1 and Japan1, including GBK-EUC-H TrueType CID fonts without ToUnicode. Windows preserves installed SimSun, SimHei, KaiTi and FangSong families, including subset-prefixed Founder aliases. Combine thumbnails and accessibility figure previews use the same on-screen fallback. System faces are display-only, disabled by PDFCRAFT_SYSTEM_FONTS=0, and not used by export, printing, OCR or automation; machines without a candidate and web builds remain unsupported. A `/ToUnicode` given as the name `/Identity-H` or `/Identity-V` (WSCAD) reads codes as Unicode for selection, search and editing, without control characters. A non-embedded Identity font whose `/ToUnicode` gives a soft hyphen (U+00AD, as Windows exporters write Arial's hyphen) draws the standard substitute's hyphen (#5888). Any other character the substitute lacks still falls back to the CID taken as one of the substitute's glyph indexes, an unrelated glyph (#3529, #5726)"
evidence = ["crates/render/src/raster/tests.rs::huge_cid_width_ranges_terminate", "crates/render/src/raster/tests.rs::non_embedded_japanese_cid_fonts_draw_with_a_craft_fonts_face", "crates/render/src/raster/tests.rs::mincho_substitute_covers_half_width_katakana", "crates/render/src/raster/tests.rs::non_embedded_chinese_and_korean_cid_fonts_draw_with_an_installed_face", "crates/engine/src/tests.rs::installed_cjk_fonts_draw_in_the_viewer_but_not_in_an_export", "crates/render/src/text/tests.rs::identity_to_unicode_names_give_text", "crates/fonts/src/pdf/tests.rs::identity_to_unicode_names_read_codes_as_unicode", "crates/render/src/raster/tests.rs::a_soft_hyphen_a_substitute_lacks_draws_its_hyphen", "crates/render/src/raster/tests/non_embedded_arial.rs::an_arial_the_pdf_does_not_embed_draws_the_glyphs_its_codes_name", "crates/render/src/raster/tests/non_embedded_arial.rs::a_times_new_roman_the_pdf_does_not_embed_draws_the_glyphs_its_codes_name", "crates/render/src/raster/tests/non_embedded_arial.rs::codes_of_other_fonts_are_not_read_in_the_macintosh_order"]
notes = "CID-to-Unicode tables are M2.4; a non-embedded Adobe-Japan1 font is drawn in a craft-fonts Japanese face (Mincho or Gothic by name, else by the descendant font descriptor's Serif flag; bold Gothic for a Bold name or a `/FontWeight` of 600 or more; BIZ UDMincho for Mincho, which covers half-width katakana) when built with craft-fonts; the desktop viewer can fall back to an installed CJK face for GB1, CNS1, Korea1 and Japan1, including GBK-EUC-H TrueType CID fonts without ToUnicode. Windows preserves installed SimSun, SimHei, KaiTi and FangSong families, including subset-prefixed Founder aliases. Combine thumbnails and accessibility figure previews use the same on-screen fallback. System faces are display-only, disabled by PDFCRAFT_SYSTEM_FONTS=0, and not used by export, printing, OCR or automation; machines without a candidate and web builds remain unsupported. A `/ToUnicode` given as the name `/Identity-H` or `/Identity-V` (WSCAD) reads codes as Unicode for selection, search and editing, without control characters. A non-embedded Identity font whose `/ToUnicode` gives a soft hyphen (U+00AD, as Windows exporters write Arial's hyphen) draws the standard substitute's hyphen (#5888). A non-embedded Arial or Times New Roman (`CIDFontType2`; ArialMT, Arial-BoldMT, Arial,Bold, TimesNewRomanPSMT, TimesNewRoman,Bold and their other faces) whose codes no Unicode route draws is read as glyph indexes of the author's font, whose first 258 glyphs follow the standard Macintosh order: the substitute draws each glyph by that name, .notdef when it has none (#5889, #5726, #6449). Its text still can't be selected, searched or copied without a `/ToUnicode`, Courier New and other faces whose numbering isn't confirmed aren't read this way, and Arial and Times New Roman glyphs past the first 258 (Cyrillic, Greek) and every other font's characters the substitute lacks still fall back to the CID taken as one of the substitute's glyph indexes, an unrelated glyph (#3529, #5726)"
[[feature]]
id = "view.render-image-codecs"
+1 -1
View File
File diff suppressed because one or more lines are too long
+80
View File
@@ -16,6 +16,7 @@ use kurbo::{BezPath, Vec2};
use skrifa::attribute::Style;
use skrifa::charmap::MapVariant;
use skrifa::raw::collections::int_set::Domain;
use skrifa::raw::tables::post::DEFAULT_GLYPH_NAMES;
use skrifa::{FontRef, GlyphId, MetadataProvider};
use std::collections::HashMap;
use std::ops::Deref;
@@ -54,6 +55,11 @@ pub(crate) struct Type0Font {
/// such text as the code's Unicode value; the identity CMap the name resolves to maps codes
/// to CIDs only and gave no text at all.
identity_to_unicode: bool,
/// PdfCraft patch: an Arial or a Times New Roman the PDF doesn't embed, as a TrueType CID
/// font (`CIDFontType2`) whose CIDs aren't of an Adobe character collection. Its CIDToGIDMap
/// gives glyph indexes in the author's font, which numbers its first 258 glyphs in the
/// standard Macintosh order of the TrueType `post` table (#5889, #6449).
mac_glyph_order: bool,
}
impl Type0Font {
@@ -163,6 +169,17 @@ impl Type0Font {
.get::<Name<'_>>(BASE_FONT)
.map(|n| strip_subset_prefix(n.as_str()).to_string());
// PdfCraft patch: the descendant names the font program itself; a Type 0 font's own
// `/BaseFont` may carry the CMap's name after it.
let mac_glyph_order = fallback
&& !adobe_collection
&& descendant_font
.get::<Name<'_>>(SUBTYPE)
.is_some_and(|s| s.deref() == CID_FONT_TYPE2)
&& descendant_font
.get::<Name<'_>>(BASE_FONT)
.is_some_and(|n| numbers_glyphs_in_mac_order(strip_subset_prefix(n.as_str())));
// Extract font flags from descriptor
let font_flags = font_descriptor
.get::<u32>(FLAGS)
@@ -186,6 +203,7 @@ impl Type0Font {
ucs2,
adobe_collection,
identity_to_unicode,
mac_glyph_order,
})
}
@@ -209,6 +227,16 @@ impl Type0Font {
return glyph;
}
// PdfCraft patch: the CID of an Arial or a Times New Roman the PDF doesn't embed names
// one of that font's glyphs, which the substitute draws by that name. Taken as the
// substitute's own glyph index it drew an unrelated glyph: "Low Power" as
// "@ck 'Dck Yf" (#5889), "Salarié(e) du régime" as "Salari²*g+f w't²i lo" (#6449).
if self.mac_glyph_order
&& let Some(glyph) = self.map_via_mac_glyph_name(cid)
{
return glyph;
}
// PdfCraft patch: a substitute that isn't CID-keyed numbers its glyphs its own way, so
// the CID of an Adobe character collection taken as its glyph index draws an unrelated
// glyph. Draw .notdef, as for a CID the font has no glyph for.
@@ -318,6 +346,27 @@ impl Type0Font {
}
}
/// PdfCraft patch: the substitute's glyph for the author's glyph that `cid` gives, found by
/// that glyph's name in the standard Macintosh order. `None` past the 258 glyphs the order
/// names, and for a substitute that isn't a name-keyed CFF font (the standard substitutes
/// are); .notdef for a name the substitute has no glyph for.
fn map_via_mac_glyph_name(&self, cid: u32) -> Option<GlyphId> {
let FontType::Cff(c) = &self.font_type else {
return None;
};
if c.is_cid() {
return None;
}
let gid = self.cid_to_gid_map.map(u16::try_from(cid).ok()?);
let name = *DEFAULT_GLYPH_NAMES.get(usize::try_from(gid.to_u32()).ok()?)?;
// `.null` and `nonmarkingreturn` draw nothing, and `nonbreakingspace` is a space.
let name = match name {
".null" | "nonmarkingreturn" | "nonbreakingspace" => "space",
name => name,
};
Some(c.glyph_index_by_name(name).unwrap_or(GlyphId::NOTDEF))
}
fn code_to_cid(&self, code: u32) -> Option<u32> {
for byte_len in 1..=4_u8 {
if let Some(cid) = self.encoding.lookup_cid_code(code, byte_len) {
@@ -643,6 +692,37 @@ fn is_variation_selector(c: char) -> bool {
matches!(c, '\u{FE00}'..='\u{FE0F}' | '\u{E0100}'..='\u{E01EF}')
}
/// PdfCraft patch: whether `name`, a `/BaseFont` without its subset tag, is one of the four
/// faces of Arial or Times New Roman as exporters name them: `ArialMT`, `Arial-BoldMT`,
/// `Arial-ItalicMT`, `Arial-BoldItalicMT`, `TimesNewRomanPSMT`, `TimesNewRomanPS-BoldMT` and so
/// on, or `Arial` or `TimesNewRoman` followed by `,Bold`, `,Italic` or `,BoldItalic`. Both
/// number their first 258 glyphs in the standard Macintosh order. Arial Narrow, Arial Black,
/// Arial Unicode MS and Arial Rounded are other fonts, numbered their own way. Letter case
/// doesn't matter, `MT` included.
fn numbers_glyphs_in_mac_order(name: &str) -> bool {
let (family, style) = name.split_once([',', '-', ' ']).unwrap_or((name, ""));
let style: String = style
.chars()
.filter(|c| !matches!(c, ',' | '-' | ' '))
.collect();
["Arial", "TimesNewRoman", "TimesNewRomanPS"]
.iter()
.any(|f| without_mt(family).eq_ignore_ascii_case(f))
&& ["", "Regular", "Bold", "Italic", "BoldItalic"]
.iter()
.any(|s| without_mt(&style).eq_ignore_ascii_case(s))
}
/// PdfCraft patch: `name` without a trailing `MT` in any letter case (Monotype's mark in
/// PostScript names such as `ArialMT`).
fn without_mt(name: &str) -> &str {
name.len()
.checked_sub(2)
.and_then(|stem| Some((name.get(..stem)?, name.get(stem..)?)))
.filter(|(_, suffix)| suffix.eq_ignore_ascii_case("MT"))
.map_or(name, |(stem, _)| stem)
}
/// PdfCraft patch: the largest CID (ISO 32000-2 §9.7.2; CIDs are at most two bytes).
const MAX_CID: u32 = 65_535;