PDF export: flattening for PDF/X-1a, X-3 and 1.3 keeps art across clipping layers

- PDF/X-1a, PDF/X-3 and PDF 1.3 files leave the layers they don't print
  out of the flattening. A flattened group no longer takes the place of
  art in a layer that doesn't print, so the art of a clipping layer over
  such a layer prints, and art of a layer that doesn't print stays out
  of the file.
- A flattened group goes into a lower layer only when that layer prints,
  shows and is a template as the clipping layer does, the clipping layer
  and the groups it leaves have no opacity, blend mode or isolation, and
  the clipping path has no fill. Otherwise the group holds only the art
  inside the clipping layer, drawn over the clipping path's fill and the
  art below, and the art below is flattened on its own. Before, the fill
  covered the group, the group lost the layer's opacity, or the clipping
  path cut the art below.
- Transparent art that a clipping path hides entirely, such as one of
  area type holding only a space, is removed. Before, the export failed
  because the art kept its transparency.

Object › Flatten Transparency follows the same rules. Documents without
clipping layers or layers that don't print export the same bytes as
before.

crates/engine/src/cmd/fileio/tests_pdfx.rs

@@ -281,6 +281,13 @@ pub(super) fn lost(reference: &vectorcraft_render::Rendered, variant: &vectorcra
     reference.iter().zip(variant).filter(|(r, v)| r[..3].iter().any(|c| *c < 200) && v[..3].iter().all(|c| *c >= 250)).count()
 }
 
+/// How many pixels of `a` and `b` differ by more than 24 in a channel.
+fn differ(a: &vectorcraft_render::Rendered, b: &vectorcraft_render::Rendered) -> usize {
+    assert_eq!((a.width, a.height), (b.width, b.height));
+    let (a, b) = (a.pixels.as_chunks::<4>().0, b.pixels.as_chunks::<4>().0);
+    a.iter().zip(b).filter(|(p, q)| p[..3].iter().zip(&q[..3]).any(|(x, y)| x.abs_diff(*y) > 24)).count()
+}
+
 /// The width and height of what each of `doc`'s images shows: its frame inside the clipping
 /// paths around it.
 pub(super) fn shown_image_sizes(doc: &vectorcraft_doc::Document) -> Vec<(f64, f64)> {
@@ -501,3 +508,194 @@ fn flat_pdf_files_of_art_that_no_clip_group_hides_are_unchanged() {
     let (mut s, _) = cropped("cmyk", 400.0, Art::Image { see_through: true });
     assert!(bytes(&mut s, &files[0]) != before(|| bytes(&mut s, &files[0])));
 }
+
+/// The options of the PDF files without transparency.
+const FLAT_FILES: [&str; 3] = [r#"{"standard": "pdfX1a"}"#, r#"{"standard": "pdfX3"}"#, r#"{"compatibility": "1.3", "preserveEditing": false}"#];
+
+/// [`cropped_by`]'s clipping layer of a gradient at 80%, with a rectangle in the layer below that
+/// crosses the crop's right edge. → the session and the clipping layer's id.
+fn clipping_layer_over_a_rectangle(mode: &str) -> (Session, u64) {
+    let (mut s, layer) = cropped_by(mode, 400.0, Art::Gradient, true);
+    run(&mut s, "transparency.set", json!({"opacity": 80}));
+    let below = s.doc().unwrap().doc.layers[0].id.0;
+    run(&mut s, "layer.setCurrent", json!({"id": below}));
+    run(&mut s, "shape.rectangle", json!({"x": 100, "y": 60, "width": 120, "height": 40}));
+    run(&mut s, "paint.setFill", json!({"color": "#406020"}));
+    (s, layer)
+}
+
+/// A clipping layer over a layer that doesn't print, and a clipping layer that doesn't print
+/// over one that does: the files without transparency print what the file with transparency
+/// prints, and nothing of the layer that doesn't print.
+#[test]
+fn flattening_for_pdfx_leaves_out_the_layers_that_do_not_print() {
+    let _g = crate::tests_colormgmt::GLOBAL.lock().unwrap_or_else(|e| e.into_inner());
+    let mut wrong = vec![];
+    for mode in ["cmyk", "rgb"] {
+        for clipping_layer_prints in [true, false] {
+            let (mut s, layer) = clipping_layer_over_a_rectangle(mode);
+            let below = s.doc().unwrap().doc.layers[0].id.0;
+            let off = if clipping_layer_prints { below } else { layer };
+            run(&mut s, "layer.setProps", json!({"id": off, "printable": false}));
+            let reference = drawn(&read_back(&mut s, &json!({})));
+            for flat in FLAT_FILES {
+                let back = drawn(&read_back(&mut s, &serde_json::from_str(flat).unwrap()));
+                let (lost, gained) = (lost(&reference, &back), lost(&back, &reference));
+                if lost + gained > 0 {
+                    wrong.push(format!("{mode}, clipping layer prints: {clipping_layer_prints}, {flat}: {lost} pixels lost, {gained} gained"));
+                }
+            }
+        }
+    }
+    assert!(wrong.is_empty(), "{wrong:#?}");
+}
+
+/// A clipping layer whose clipping path has a fill: the files without transparency show the art
+/// over the fill, not the fill over the art.
+#[test]
+fn flattening_for_pdfx_keeps_the_art_of_a_clipping_layer_over_its_clipping_path_fill() {
+    let _g = crate::tests_colormgmt::GLOBAL.lock().unwrap_or_else(|e| e.into_inner());
+    let mut wrong = vec![];
+    for mode in ["cmyk", "rgb"] {
+        let (mut s, layer) = clipping_layer_over_a_rectangle(mode);
+        let clip = s.doc().unwrap().doc.node(vectorcraft_doc::NodeId(layer)).unwrap().children().unwrap()[0].id.0;
+        run(&mut s, "paint.setFill", json!({"ids": [clip], "color": "#ffdd00"}));
+        let reference = drawn(&read_back(&mut s, &json!({})));
+        for flat in FLAT_FILES {
+            let back = drawn(&read_back(&mut s, &serde_json::from_str(flat).unwrap()));
+            // PDF/X-1a converts an RGB document's colors to CMYK.
+            let converted = mode == "rgb" && flat.contains("pdfX1a");
+            let (lost, differ) = (lost(&reference, &back), if converted { 0 } else { differ(&reference, &back) });
+            if lost > 0 || differ > 100 {
+                wrong.push(format!("{mode}, {flat}: {lost} pixels of art lost, {differ} pixels differ"));
+            }
+        }
+    }
+    assert!(wrong.is_empty(), "{wrong:#?}");
+}
+
+/// Transparent art in a clipping layer whose clipping path is area type holding only a space,
+/// which hides it: the files without transparency are written, without the art.
+#[test]
+fn flattening_for_pdfx_removes_transparent_art_that_a_clipping_layer_hides_entirely() {
+    let _g = crate::tests_colormgmt::GLOBAL.lock().unwrap_or_else(|e| e.into_inner());
+    let [x, y, w, h] = FRAME;
+    let mut s = Session::new();
+    run(&mut s, "file.new", json!({"width": 400, "height": 300, "colorMode": "cmyk"}));
+    let layer = run(&mut s, "layer.new", json!({}))["id"].as_u64().unwrap();
+    run(&mut s, "shape.rectangle", json!({"x": x, "y": y, "width": w, "height": h}));
+    let stops = json!([{"offset": 0, "color": "#ff0000"}, {"offset": 1, "color": "#0000ff"}]);
+    run(&mut s, "paint.setFill", json!({"gradient": {"kind": "linear", "stops": stops}}));
+    run(&mut s, "transparency.set", json!({"opacity": 80}));
+    run(&mut s, "text.create", json!({"x": 50, "y": 50, "text": " ", "area": {"width": 80, "height": 60}}));
+    run(&mut s, "layer.clippingMask.toggle", json!({"id": layer}));
+    let reference = drawn(&read_back(&mut s, &json!({})));
+    for flat in FLAT_FILES {
+        let p: Value = serde_json::from_str(flat).unwrap();
+        let back = match s.execute("document.exportPdf", &p) {
+            Ok(v) => vectorcraft_pdf::import(&vectorcraft_format::base64_decode(v["dataBase64"].as_str().unwrap()).unwrap()).unwrap(),
+            Err(e) => panic!("{flat}: {e}"),
+        };
+        assert_eq!(lost(&drawn(&back), &reference), 0, "{flat}: the hidden art prints");
+    }
+}
+
+/// Object › Flatten Transparency of art in a clipping layer and art in the layer below that
+/// reaches past the clipping path: when the layer below doesn't print, the clipping layer has
+/// opacity, or the clipping path has a fill (see-through too, flattened keeping alpha, in a layer
+/// with opacity or Multiply mode, or
+/// over a layer with opacity, overlapping art of a layer with opacity or art an opacity mask
+/// hides), the group stays in the clipping layer, the art below stays in its layer, and the page
+/// looks the same.
+#[test]
+fn flattening_keeps_the_group_in_the_clipping_layer_when_the_layer_below_differs() {
+    let mut wrong = vec![];
+    let cases = [
+        "the layer below doesn't print",
+        "the clipping path has a fill",
+        "the clipping path has a see-through fill, keeping alpha",
+        "the clipping path has a see-through fill, in a layer with opacity",
+        "the clipping path has a see-through fill, in a layer in Multiply mode",
+        "the clipping path has a see-through fill, over a layer with opacity",
+        "the clipping path has a see-through fill, over art that an opacity mask hides",
+        "the clipping path has a see-through fill, over overlapping art of a layer at 50%",
+        "the clipping path has no fill, in a layer with opacity",
+    ];
+    for case in cases {
+        let mut s = Session::new();
+        run(&mut s, "file.new", json!({"width": 400, "height": 300, "colorMode": "rgb"}));
+        let below = s.doc().unwrap().doc.layers[0].id.0;
+        let orange = run(&mut s, "shape.rectangle", json!({"x": 20, "y": 20, "width": 200, "height": 150}))["id"].clone();
+        run(&mut s, "paint.setFill", json!({"color": "#ff8000"}));
+        let upper = run(&mut s, "layer.new", json!({}))["id"].as_u64().unwrap();
+        let blue = run(&mut s, "shape.rectangle", json!({"x": 100, "y": 80, "width": 200, "height": 150}))["id"].clone();
+        run(&mut s, "paint.setFill", json!({"color": "#2040c0"}));
+        run(&mut s, "transparency.set", json!({"opacity": 50}));
+        let clip = run(&mut s, "shape.rectangle", json!({"x": 60, "y": 40, "width": 300, "height": 220}))["id"].clone();
+        run(&mut s, "layer.clippingMask.toggle", json!({"id": upper}));
+        if case.starts_with("the layer below") {
+            run(&mut s, "layer.setProps", json!({"id": below, "printable": false}));
+        } else if !case.contains("no fill") {
+            run(&mut s, "paint.setFill", json!({"ids": [clip], "color": "#88ffcc"}));
+        }
+        let keeping_alpha = case.ends_with("keeping alpha");
+        if case.contains("see-through fill") {
+            run(&mut s, "transparency.set", json!({"ids": [clip], "opacity": 50}));
+        }
+        if case.ends_with("in a layer with opacity") {
+            run(&mut s, "transparency.set", json!({"ids": [upper], "opacity": 80}));
+        }
+        if case.ends_with("over a layer with opacity") {
+            run(&mut s, "transparency.set", json!({"ids": [below], "opacity": 50}));
+        }
+        if case.ends_with("an opacity mask hides") {
+            run(&mut s, "layer.setCurrent", json!({"id": below}));
+            run(&mut s, "select.set", json!({"ids": [orange]}));
+            let group = run(&mut s, "object.group", json!({}))["id"].clone();
+            let mask = run(&mut s, "shape.rectangle", json!({"x": 20, "y": 20, "width": 200, "height": 150}))["id"].clone();
+            run(&mut s, "paint.setFill", json!({"color": "#000000"}));
+            run(&mut s, "select.set", json!({"ids": [group, mask]}));
+            run(&mut s, "transparency.makeOpacityMask", json!({}));
+        }
+        if case.ends_with("Multiply mode") {
+            run(&mut s, "transparency.set", json!({"ids": [upper], "blendMode": "multiply"}));
+        }
+        let mut art = vec![orange.clone()];
+        if case.ends_with("a layer at 50%") {
+            run(&mut s, "layer.setCurrent", json!({"id": below}));
+            art.push(run(&mut s, "shape.rectangle", json!({"x": 80, "y": 60, "width": 120, "height": 120}))["id"].clone());
+            run(&mut s, "paint.setFill", json!({"color": "#20a040"}));
+            run(&mut s, "transparency.set", json!({"ids": [below], "opacity": 50}));
+        }
+        art.push(blue.clone());
+        // The page as it prints.
+        let printed = |d: &vectorcraft_doc::Document| {
+            let mut d = d.clone();
+            if case.starts_with("the layer below") {
+                let mut layer = (*d.layers[0]).clone();
+                layer.visible = false;
+                d.layers[0] = layer.into();
+            }
+            drawn(&d)
+        };
+        let before = printed(&s.doc().unwrap().doc);
+        run(&mut s, "select.set", json!({"ids": art}));
+        run(&mut s, "object.flattenTransparency", json!({"preserveAlpha": keeping_alpha}));
+        let d = &s.doc().unwrap().doc;
+        let layer_of = |id: &Value| id.as_u64().and_then(|id| d.ancestry(vectorcraft_doc::NodeId(id))).map(|a| a[0].0);
+        if layer_of(&orange) != Some(below) {
+            wrong.push(format!("{case}: the art below is in layer {:?}", layer_of(&orange)));
+        }
+        let upper_art = d.node(vectorcraft_doc::NodeId(upper)).and_then(|l| l.children()).map_or(0, Vec::len);
+        if d.node(vectorcraft_doc::NodeId(blue.as_u64().unwrap())).is_some() || upper_art != 2 {
+            wrong.push(format!("{case}: the clipping layer holds {upper_art} objects"));
+        }
+        let after = printed(d);
+        // Flat regions differ from see-through art along their antialiased edges.
+        let (lost, differ) = (lost(&before, &after), differ(&before, &after));
+        if lost > 0 || differ > 1200 {
+            wrong.push(format!("{case}: {lost} pixels of art lost, {differ} pixels differ"));
+        }
+    }
+    assert!(wrong.is_empty(), "{wrong:#?}");
+}

crates/engine/src/cmd/flatten/apply.rs

@@ -1,4 +1,4 @@
-use super::analysis::{Scratch, outlined_text, overlapping, reach};
+use super::analysis::{Scratch, has_effects, masked, outlined_text, overlapping, reach};
 use super::compose::{Builder, CLEAR, Composer, Elem, Fill, Kind, Px, WHITE};
 use super::options::FlattenOptions;
 use super::{
@@ -108,30 +108,82 @@ pub(super) fn plan(src: &Document, roots: &[NodeId], o: &FlattenOptions, render:
     let mut sc = Scratch { brushes: vectorcraft_brush::library(&baked), doc: baked.clone() };
     let plain: Vec<Node> = art.iter().map(|n| sc.plain(n)).collect();
     // Each object as its clip groups and clipping layers show it.
-    let clipped: Vec<Node> = plain.iter().zip(roots).map(|(n, id)| clipped_by_ancestors(src, *id, n)).collect();
+    let clipped: Vec<Node> = plain.iter().zip(roots).map(|(n, id)| clipped_by_ancestors(src, *id, n, None)).collect();
     let reaches: Vec<Option<Rect>> = plain.iter().zip(&clipped).map(|(n, c)| reach(n).zip(reach(c)).map(|(a, b)| a.intersect(b))).collect();
     let see_through: Vec<bool> = plain.iter().map(|n| o.shows_transparency(&baked, n) || o.draws_see_through_image(n)).collect();
     let cmyk = src.color_mode == ColorMode::Cmyk;
     let mut out = Plan { flat: vec![], kept: vec![], transparent: roots.iter().zip(&see_through).filter(|(_, t)| **t).map(|(id, _)| *id).collect() };
     // Baking above stopped early when cancelled; each group is a place to stop (a plan changes nothing).
     vectorcraft_doc::cancel::check()?;
-    for g in overlapping(&reaches)? {
+    let mut queue = overlapping(&reaches)?;
+    queue.reverse();
+    while let Some(g) = queue.pop() {
         vectorcraft_doc::cancel::check()?;
+        // A group that stays inside the clip groups and clipping layers of its topmost object
+        // replaces only the objects inside them, over the others, which are flattened on their own.
+        let (g, under) = match inside_and_under(src, roots, &g) {
+            Some((inside, under)) => {
+                queue.push(under.clone());
+                (inside, under)
+            }
+            None => (g, vec![]),
+        };
+        // Composited over white, the group covers what it is drawn over, so it is composited over
+        // the fills of the clipping paths around it and the objects under it in the same kind of
+        // layer. Composited over nothing, it keeps its alpha and they show through.
+        let backdrop = !o.preserve_alpha;
+        let state = g.last().map(|&i| layer_state(src, roots[i]));
+        let top = g.last().map(|&i| roots[i]);
+        // A blend mode of a layer or group around the group alone would blend the art under it
+        // twice.
+        let blends = |i: usize| top.is_some_and(|t| own_layers(src, t, Some(roots[i])).iter().any(|a| a.blend != BlendMode::Normal));
+        // Art under an opacity mask, effects or a knockout group is left out: the group can't draw it
+        // as those do.
+        let drawn_plainly = |i: usize| {
+            own_layers(src, roots[i], top).iter().all(|a| !masked(a) && !has_effects(a) && !matches!(a.knockout, vectorcraft_doc::Knockout::On))
+        };
+        let under_shown = under.iter().copied().filter(|&i| backdrop && Some(layer_state(src, roots[i])) == state && !blends(i) && drawn_plainly(i));
+        let composited: Vec<usize> = under_shown.chain(g.iter().copied()).collect();
+        let below = composited.len() - g.len();
+        let extent = composited.iter().filter_map(|&i| reaches[i]).reduce(|a, b| a.union(b)).unwrap_or_default();
+        let shown = |nodes: &[Node]| -> Vec<Node> {
+            let mut seen = vec![];
+            let (mut under, mut out) = (vec![], vec![]);
+            for (k, &i) in composited.iter().enumerate() {
+                let n = clipped_by_ancestors(src, roots[i], &nodes[i], backdrop.then_some((&mut seen, extent)));
+                if k < below {
+                    let groups = own_layers(src, roots[i], top).into_iter().filter(|a| !draws_as_is(a)).collect();
+                    under.push((groups, n));
+                } else {
+                    out.push(n);
+                }
+            }
+            let mut all = in_groups(under, 0);
+            all.extend(out);
+            all
+        };
         let ids: Vec<NodeId> = g.iter().map(|&i| roots[i]).collect();
         let group = g.iter().any(|&i| see_through[i]).then(|| {
-            let plain: Vec<&Node> = g.iter().map(|&i| &clipped[i]).collect();
-            flatten_group(&plain, o, cmyk)
+            let plain = shown(&plain);
+            flatten_group(&plain.iter().collect::<Vec<_>>(), o, cmyk)
         });
         let Some((regions, area)) = group.flatten() else {
-            out.kept.extend(ids);
+            // Transparent art that its clip groups or clipping layers hide entirely draws nothing:
+            // it is removed.
+            let hides = |i: usize| see_through[i] && reach(&plain[i]).is_some() && reach(&clipped[i]).is_none();
+            let (hidden, left): (Vec<usize>, Vec<usize>) = g.iter().partition(|&&i| hides(i));
+            if !hidden.is_empty() {
+                let roots = hidden.iter().map(|&i| roots[i]).collect();
+                out.flat.push(Flat { roots, regions: vec![], area: None, raster: None, lines: None });
+            }
+            out.kept.extend(left.iter().map(|&i| roots[i]));
             continue;
         };
         let raster = match area.as_ref().filter(|_| render) {
             Some(a) => {
                 // Drawn as planned: the image can land outside the clip groups and clipping layers
                 // around its art.
-                let art: Vec<Node> = g.iter().map(|&i| clipped_by_ancestors(src, roots[i], &art[i])).collect();
-                Some(render_raster(&baked, &art, a.rect, a.ppi, a.clip.clone(), o)?)
+                Some(render_raster(&baked, &shown(&art), a.rect, a.ppi, a.clip.clone(), o)?)
             }
             None => None,
         };
@@ -146,8 +198,10 @@ pub(super) fn plan(src: &Document, roots: &[NodeId], o: &FlattenOptions, render:
 
 /// `plain` (object `id` of `src` as plain art) inside a clip group for each clip group and
 /// clipping layer around `id`, innermost first, each clipped by that group's or layer's clipping
-/// path. `plain` itself when nothing around it clips.
-fn clipped_by_ancestors(src: &Document, id: NodeId, plain: &Node) -> Node {
+/// path. `plain` itself when nothing around it clips. With `fills` (clipping paths, and an area),
+/// the fill of each clipping path not in them yet goes behind the art, inside the area, and the
+/// path is added to them.
+fn clipped_by_ancestors(src: &Document, id: NodeId, plain: &Node, mut fills: Option<(&mut Vec<NodeId>, Rect)>) -> Node {
     let mut out = plain.clone();
     if !clips_count() {
         return out;
@@ -170,13 +224,106 @@ fn clipped_by_ancestors(src: &Document, id: NodeId, plain: &Node) -> Node {
         if let NodeKind::Path { rule: r, .. } = &mut path.kind {
             *r = rule;
         }
-        let mut g = Node::group(above.id, vec![Arc::new(path), Arc::new(out)]);
+        let mut children = vec![Arc::new(path)];
+        // The renderer paints the fill behind the clipped art.
+        if let Some((seen, r)) = fills.as_mut()
+            && !seen.contains(&clip.id)
+            && let Some(fill) = clip.clip_paint().fill
+        {
+            let r = *r;
+            seen.push(clip.id);
+            let mut at = Node::group(above.id, vec![Arc::new(Node::path(clip.id, shapes::rectangle(r), Appearance::default())), Arc::new(fill)]);
+            at.set_clips(true);
+            children.push(Arc::new(at));
+        }
+        children.push(Arc::new(out));
+        let mut g = Node::group(above.id, children);
         g.set_clips(true);
         out = g;
     }
     out
 }
 
+/// Whether layer or group `n` draws what it holds as it is: opaque, in Normal mode, not isolated.
+fn draws_as_is(n: &Node) -> bool {
+    n.opacity >= 1.0 && n.blend == BlendMode::Normal && !n.isolate
+}
+
+/// `items` (in paint order, each an object and the layers and groups around it, outermost first,
+/// that don't draw it as it is) with the objects of each such layer or group in one group with its
+/// opacity, blend mode and isolation, from the `depth`th around them on: they then composite
+/// together as that layer or group draws them.
+fn in_groups(items: Vec<(Vec<&Node>, Node)>, depth: usize) -> Vec<Node> {
+    let mut out = vec![];
+    let mut items = items.into_iter().peekable();
+    while let Some((around, n)) = items.next() {
+        let Some(a) = around.get(depth).copied() else {
+            out.push(n);
+            continue;
+        };
+        let mut run = vec![(around, n)];
+        while let Some(next) = items.next_if(|(around, _)| around.get(depth).is_some_and(|b| b.id == a.id)) {
+            run.push(next);
+        }
+        let mut g = Node::group(a.id, in_groups(run, depth + 1).into_iter().map(Arc::new).collect());
+        (g.opacity, g.blend, g.isolate) = (a.opacity, a.blend, a.isolate);
+        out.push(g);
+    }
+    out
+}
+
+/// The layers and groups around object `id` of `src` that are not also around `other`, outermost
+/// first.
+fn own_layers(src: &Document, id: NodeId, other: Option<NodeId>) -> Vec<&Node> {
+    let mine = src.ancestry(id).unwrap_or_default();
+    let theirs = other.and_then(|t| src.ancestry(t)).unwrap_or_default();
+    let shared = mine.iter().zip(&theirs).take_while(|(a, b)| a == b).count();
+    let own = mine.get(shared..mine.len().saturating_sub(1)).unwrap_or_default();
+    own.iter().filter_map(|a| src.node(*a)).collect()
+}
+
+/// The clip groups and clipping layers around object `id` of `d`, innermost first.
+fn clip_ancestors(d: &Document, id: NodeId) -> Vec<NodeId> {
+    let mut a = d.ancestry(id).unwrap_or_default();
+    a.pop();
+    a.into_iter().rev().filter(|a| d.node(*a).is_some_and(Node::clips)).collect()
+}
+
+/// Whether the layers around object `id` of `d` all print, all show, and whether one is a
+/// template.
+fn layer_state(d: &Document, id: NodeId) -> (bool, bool, bool) {
+    let mut a = d.ancestry(id).unwrap_or_default();
+    a.pop();
+    a.iter().filter_map(|a| d.node(*a)).fold((true, true, false), |(print, show, template), n| match &n.kind {
+        NodeKind::Layer { printable, template: t, .. } => (print && *printable, show && n.visible, template || *t),
+        _ => (print, show, template),
+    })
+}
+
+/// For group `g` (indices of `roots` of `src`, in paint order) whose flattened group can't leave
+/// the clip groups and clipping layers of its topmost object ([`place_of`]): the objects inside
+/// them and the objects under them. `None` when the group can leave them, or every object is
+/// inside them.
+fn inside_and_under(src: &Document, roots: &[NodeId], g: &[usize]) -> Option<(Vec<usize>, Vec<usize>)> {
+    if !clips_count() || g.len() < 2 {
+        return None;
+    }
+    let ids: Vec<NodeId> = g.iter().map(|&i| roots[i]).collect();
+    let top = clip_ancestors(src, *ids.last()?);
+    if top.is_empty() {
+        return None;
+    }
+    let inside = |id: NodeId| {
+        let mine = clip_ancestors(src, id);
+        top.iter().all(|c| mine.contains(c))
+    };
+    if !inside(place_of(src, &ids)?) {
+        return None;
+    }
+    let (inside, under): (Vec<usize>, Vec<usize>) = g.iter().partition(|&&i| inside(roots[i]));
+    (!under.is_empty()).then_some((inside, under))
+}
+
 /// The regions and the image area replacing one group of overlapping objects (`plain`: the art as
 /// composited; `cmyk`: in a CMYK document); `None` when it paints nothing.
 fn flatten_group(plain: &[&Node], o: &FlattenOptions, cmyk: bool) -> Option<(Regions, Option<Area>)> {
@@ -493,6 +640,17 @@ pub(super) fn apply(d: &mut Document, sel: &mut vectorcraft_doc::Selection, plan
     for f in plan.flat {
         // A group always has objects.
         let Some(top) = place_of(d, &f.roots) else { continue };
+        // Art hidden entirely is removed.
+        if f.regions.is_empty() && f.area.is_none() {
+            if one_pass {
+                flat.extend(f.roots.iter().map(|r| (*r, vec![])));
+            } else {
+                for r in &f.roots {
+                    d.remove(*r)?;
+                }
+            }
+            continue;
+        }
         let mut children = vec![];
         // The image goes under the regions, reaching a pixel past its own: their edges then fall on
         // matching pixels instead of on what is behind the group.
@@ -564,19 +722,26 @@ pub(super) fn apply(d: &mut Document, sel: &mut vectorcraft_doc::Selection, plan
 
 /// The object of `roots` (in paint order) whose place their flattened group takes: the topmost one
 /// that no clip group or clipping layer clips unless it clips every object of `roots`, else the
-/// topmost one.
+/// topmost one. The group stays with the topmost object rather than move to a layer that prints,
+/// shows or is a template where the topmost object's layer differs, out from a clip group or
+/// clipping layer whose clipping path has a fill, which would then cover it, or out from a layer
+/// or group with opacity, a blend mode or isolation.
 fn place_of(d: &Document, roots: &[NodeId]) -> Option<NodeId> {
-    let top = roots.last().copied();
+    let top = roots.last().copied()?;
     if !clips_count() || roots.len() < 2 {
-        return top;
+        return Some(top);
     }
-    let clips = |id: NodeId| -> Vec<NodeId> {
-        let ancestry = d.ancestry(id).unwrap_or_default();
-        ancestry.iter().rev().skip(1).filter(|a| d.node(**a).is_some_and(Node::clips)).copied().collect()
-    };
-    let all: Vec<Vec<NodeId>> = roots.iter().map(|r| clips(*r)).collect();
+    let fills = |clip: &NodeId| d.node(*clip).and_then(|n| n.children()?.first().map(|c| c.clip_paint().fill.is_some())).unwrap_or(false);
+    let all: Vec<Vec<NodeId>> = roots.iter().map(|r| clip_ancestors(d, *r)).collect();
+    let above = all.last().map_or(&[][..], Vec::as_slice);
     let shared = |mine: &Vec<NodeId>| mine.iter().all(|c| all.iter().all(|theirs| theirs.contains(c)));
-    roots.iter().zip(&all).rev().find(|(_, mine)| shared(mine)).map(|(r, _)| *r).or(top)
+    let leaves_fill = |mine: &Vec<NodeId>| above.iter().any(|c| !mine.contains(c) && fills(c));
+    // Out of a layer or group with opacity, a blend mode or isolation, the group would lose it.
+    let leaves_compositing = |r: NodeId| own_layers(d, top, Some(r)).iter().any(|a| !draws_as_is(a));
+    let state = layer_state(d, top);
+    let place =
+        roots.iter().zip(&all).rev().find(|(r, mine)| shared(mine) && !leaves_fill(mine) && layer_state(d, **r) == state && !leaves_compositing(**r));
+    Some(place.map_or(top, |(r, _)| *r))
 }
 
 /// An image of `r` (in a clip group when it is clipped).

crates/engine/src/cmd/flatten/commands.rs

@@ -10,7 +10,7 @@ pub fn specs() -> Vec<CommandSpec> {
             "Flatten Transparency…",
             ["Object"],
             None,
-            "{preset?: \"high\"|\"medium\"|\"low\" or a saved preset's name (see flattener.presets.list; default medium), balance?: 0..100 (raster/vector balance; 0 rasterizes everything), lineArtPpi?: 1..2400, gradientPpi?: 1..2400 (areas only gradients and meshes reach), textToOutlines?, strokesToOutlines?, clipComplexRegions? (clip images to the region outlines, else rectangles), antiAlias?, preserveAlpha? (composite over nothing instead of white), preserveOverprints? (areas showing one paint keep its colour and overprint; false clears overprints), options?: {the same keys}, ids?|id?} overlapping transparent objects become one group of flat-colour regions (CMYK colours composited ink by ink in CMYK documents), plus an image where gradients, patterns, images, masks or raster effects reach (only where a clip group's or clipping layer's clipping path shows them); objects without transparency stay; a frame of threaded text that is outlined or flattened leaves its thread with the text it shows → {ids, rasterized: images made, vector: regions made, options, warning? (an image in the art couldn't be decoded, as object.rasterize says)}",
+            "{preset?: \"high\"|\"medium\"|\"low\" or a saved preset's name (see flattener.presets.list; default medium), balance?: 0..100 (raster/vector balance; 0 rasterizes everything), lineArtPpi?: 1..2400, gradientPpi?: 1..2400 (areas only gradients and meshes reach), textToOutlines?, strokesToOutlines?, clipComplexRegions? (clip images to the region outlines, else rectangles), antiAlias?, preserveAlpha? (composite over nothing instead of white), preserveOverprints? (areas showing one paint keep its colour and overprint; false clears overprints), options?: {the same keys}, ids?|id?} overlapping transparent objects become one group of flat-colour regions (CMYK colours composited ink by ink in CMYK documents), plus an image where gradients, patterns, images, masks or raster effects reach (only where a clip group's or clipping layer's clipping path shows them, over its fill; transparent objects such a path hides entirely are removed); objects without transparency stay; a frame of threaded text that is outlined or flattened leaves its thread with the text it shows → {ids, rasterized: images made, vector: regions made, options, warning? (an image in the art couldn't be decoded, as object.rasterize says)}",
             has_doc,
             flatten
         ),

docs/mcp.md

@@ -836,7 +836,8 @@ page with hint tables saying where every other page's objects are), which stays
 `compatibility: "1.3"` writes a PDF 1.3 file, which has no transparency: a copy of the document is flattened first
 with `flattenerPreset` (empty: High Resolution; `high`, `medium`, `low` or a saved preset, `flattener.presets.list`)
 and `flattener` options over it (as `object.flattenTransparency` takes them); images with see-through pixels count as
-transparency and rasterized areas are clipped to their regions. A file that would still have transparency fails the
+transparency and rasterized areas are clipped to their regions. Layers the file leaves out (non-printing ones, unless
+`includeNonPrinting` or `createLayers` is set) are left out of the flattening. A file that would still have transparency fails the
 export. PDF/X-1a and PDF/X-3 files are PDF 1.3 files too.
 A page with a side longer than 14,400 pt is written with a `UserUnit` (PDF 1.6): its longer side divided by 14,400,
 rounded up to the next thousandth, with the page's sizes and coordinates in those units, so that no side is longer than
@@ -3062,7 +3063,12 @@ Each other group becomes one group of atomic regions: paths filled with the flat
 white, which blend modes do not read; over nothing, keeping alpha, with `preserveAlpha: true`), plus one image where gradients, patterns, images,
 opacity masks or raster effects reach (clipped to those regions with `clipComplexRegions`, a rectangle without).
 The image leaves out the part of such art that the clipping path of a clip group or clipping layer hides, so art under
-the hidden part of a cropped picture stays as it is.
+the hidden part of a cropped picture stays as it is. Transparent art that such a clipping path hides entirely is
+removed. A group that holds art from inside and outside a clipping layer goes in the place of the topmost art outside
+it. When that layer prints, shows or is a template where the clipping layer differs, the clipping layer or a group
+around the art has opacity, a blend mode or isolation, or the clipping path has a fill,
+the group holds only the art inside the clipping layer, drawn over the fill and over the art below in the same kind of
+layer as its own layers draw it (over nothing with `preserveAlpha`), and the art outside is flattened on its own.
 `preset` is `high`, `medium` (the default) or `low`; option keys (`balance` 0–100, `lineArtPpi`, `gradientPpi`,
 `textToOutlines`, `strokesToOutlines`, `clipComplexRegions`, `antiAlias`, `preserveAlpha`, `preserveOverprints`)
 override it, at the top level or in `options`. `balance: 0` rasterizes everything; lower balances rasterize groups

docs/target-app-parity.md

@@ -1413,7 +1413,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur
     In a `.ai` opened from its editing data, Bas Relief in an art style takes its Detail, Smoothness and Light direction from the style's settings block, and an object other than type with it opens as itself with the live effect. A settings block that VectorCraft can't read (malformed, another effect's, a value out of range or a key it doesn't know) keeps the object's drawn look (part of #637; [raster effects reference](https://helpx.adobe.com/illustrator/desktop/special-effects-styles/apply-filter-effects/raster-effects-reference.html), [Photoshop File Formats Specification](https://www.adobe.com/devnet-apps/photoshop/fileformatashtml/), [scripting the filter gallery](https://community.adobe.com/questions-712/scripting-support-for-the-filter-gallery-1064646); `crates/eps/src/import/ai/style/tests.rs`, `crates/eps/src/import/ai/tests.rs`).
     PDF export and `.ai` saves write a page with a side longer than 14,400 pt with a `UserUnit` (PDF 1.6), its longer side divided by 14,400 and rounded up to the next thousandth, so that no side is longer than 14,400 units (ISO 32000-1, Annex C.2); a PDF 1.5 file with such a page is written as PDF 1.6 with PDF 1.5's encryption, PDF/X files and PDF 1.3 and 1.4 files write it in points (in PDF 1.3 and 1.4, a page past 32,767 pt that needs a real number past 32,767 is refused with an error naming it), a page whose short side is 3 pt or longer but would be under 3 units at that `UserUnit` gets a smaller one that keeps the short side at 3 units, and the export's warnings give each such page's size and how it is written (#6382).
     PDF export warns when a PDF/X file has no bleed on a side where the document's bleed has one, and writes the file as asked.
-    PDF/X-1a, PDF/X-3 and PDF 1.3 files keep the art under the hidden part of a picture, gradient, pattern, mesh or raster effect that a clipping mask or clipping layer crops: flattening rasterizes only the part of such art that its clipping path shows, on its own page and on the next one (`crates/engine/src/cmd/fileio/tests_pdfx.rs`).
+    PDF/X-1a, PDF/X-3 and PDF 1.3 files keep the art under the hidden part of a picture, gradient, pattern, mesh or raster effect that a clipping mask or clipping layer crops: flattening rasterizes only the part of such art that its clipping path shows, on its own page and on the next one. Layers the file leaves out are left out of the flattening, and transparent art that a clipping path hides entirely is removed (`crates/engine/src/cmd/fileio/tests_pdfx.rs`).
     Placed (linked) files in a `.ai`'s editing data come in as linked images in their place, each showing the low-resolution preview the file keeps of it until its file is found or relinked in the Links panel (a file found next to the document, at its path, or in the folder the editing data names relative to the document's, such as `src/`, is read as the document opens, and its link takes that file's size, modification time and hash). A file with them no longer falls back to its PDF part, so its guides, sublayers and hidden objects open (#1264; `crates/eps/src/import/ai/tests.rs`, `crates/engine/src/tests/links.rs`). In a file opened that way, a gradient with stops in the middle runs its whole ramp across its shape and a gradient object no longer passes its fill to the next path (#1390); text objects with an appearance keep their own text and frame, type of hidden layers stays hidden, and type on later artboards lands on its text objects (#1391); area type whose frame is too short for a line of the fallback font shows its first line, the frame made taller with a warning in the open result (#1392).
     Placed (linked) files in a `.ai`'s editing data come in as linked images in their place, each showing the low-resolution preview the file keeps of it until its file is found or relinked in the Links panel (a file found next to the document or at its path is read as the document opens). A file with them no longer falls back to its PDF part, so its guides, sublayers and hidden objects open (#1264; `crates/eps/src/import/ai/tests.rs`, `crates/engine/src/tests/links.rs`). In a file opened that way, a gradient with stops in the middle runs its whole ramp across its shape and a gradient object no longer passes its fill to the next path (#1390); text objects with an appearance keep their own text and frame, type of hidden layers stays hidden, and type on later artboards lands on its text objects (#1391); area type whose frame is too short for a line of the fallback font shows its first line, the frame made taller with a warning in the open result (#1392).
     A placed PDF or `.ai` file in a `.ai`'s editing data comes in as a placed document instead of a linked image: it shows the page and Crop to box its dictionary names, and the preview the file keeps until its file is found, then the page as vectors. A placed file in opacity-mask art keeps showing its preview (`crates/eps/src/import/ai/tests.rs`, `crates/engine/src/tests/placed_document.rs`).