Hamza Naveedandskylar taylor-barrick
97cef90e01
* Redaction: a pattern nothing paints with any more leaves the page
A tiling pattern's cell is a content stream of its own and can draw
anything, text included. Redacting a shape filled with such a pattern
removed the fill from the page's content -- the page looked right, a
black box -- but left the pattern in /Resources /Pattern, and its cell,
with the text that was under the mark, was written into the saved file
unchanged.
The pass does not enter pattern cells, so it cannot redact inside one.
What it can do is notice that nothing paints with the pattern any more.
The graphics state now carries the fill and stroke pattern a `scn` or
`SCN` selected (and `cs`, `g`, `rg`, `k` and their stroke forms clear
it), every painting operator that survives the rewrite records the
pattern it drew with, and a pattern the page selected but no surviving
paint used leaves the page's resources.
Measured on the report's file:
before applied: 1, (SECRET) in the saved bytes
after applied: 1, no /Pattern on the page, (SECRET) gone
A pattern some surviving paint still uses is kept, and so is its cell:
with two boxes filled from one pattern and only the first marked,
patterns_removed is 0 and the resources keep the entry -- the second box
still shows the word, so it is not a leak.
Report gains patterns_removed.
Fixes #3647.
* Keep a pattern every operator that can paint with it still uses
Two defects CodeRabbit found in the first version of this change, both
of which could remove a pattern the page still draws with.
Pattern use was recorded for path painting only. Text shows with the
fill pattern too -- and with the stroke pattern in the stroking render
modes -- as do image masks and anything a form draws, and a form's
content is interpreted with a graphics state of its own, so a pattern it
inherits is never seen inside it. Content like
/Pattern cs /P0 scn 20 140 100 30 re f BT (KEEP) Tj ET
with the box redacted therefore lost the fill of text still on the page.
Every operator that can paint with the current pattern now records it:
text (except mode 3, which paints nothing, and mode 7, which only
clips), inline images, and image or form XObjects. Only path painting
knows exactly what it drew and whether the paint survived, so the others
are recorded as they are met. That can keep a pattern whose paint was in
fact all redacted, which costs an unused resource entry, while dropping
one something still draws with takes paint off the page.
/Resources /Pattern may be an indirect object, like any value. Reading
the reference itself as a dictionary found none and pruned nothing, so
on such a file the cell -- with the text that was under the mark --
stayed in the saved bytes, which is the leak this change is for. It is
resolved now, and what is written back is a dictionary of this page's
own, so pruning never reaches through a /Pattern dictionary another page
shares.
The controls in the test: without the first fix the surviving text's
pattern is removed (left: 1, right: 0), and without the second the
indirect file prunes nothing (left: 0, right: 1).
* M8: refuse retained unsafe tiling patterns
* M8: guard shared appearances and narrow pattern scans
* M8: update TLS dependencies for new advisories
* Redaction: the comment on prune_patterns said more than the code does
`prune_patterns` drops the `/Pattern` resource entry; the `scn` that selected
it stays in the content, so the page names a pattern its resources no longer
hold. The comment claimed the selector went too (noted in review).
ISO 32000-1 8.6.6.2 makes that operand a key of the `/Pattern` subdictionary,
so the open question is whether a reader stops reading the stream at the stale
name. `a_page_whose_pruned_pattern_is_still_selected_renders` answers it with
the renderer rather than with prose: the redacted page is saved, reopened and
rasterized, and
* it renders with no error and the same 300x300 page box,
* the text written *after* the stale selector is still painted -- which is
exactly what a reader would drop if it gave up at the bad operand,
* the redaction overlay is black over the box the pattern filled.
Control: with the drawing after the selector removed, the pixel assertion fails
("the text after the stale selector is still painted"), so it is reading that
text and not a stray pixel.
The selector is left in place rather than cut out: removing operators is a
rewrite of the content this pass does not otherwise do, and nothing surviving
paints with that colour. `pdfcraft-render` is a dev dependency of this crate
only -- layer 4 on layer 3, which the layering allows.
* WIP: pattern guard depth bound sized from the interpreter's
* Pattern tests: keep the guard's cases reaching the guard now a hit path is removed whole
* Pattern automation tests: keep the pattern in use where they test the guard, since a hit path is now removed whole
* Redaction: a form without resources keeps the page pattern it still paints
A form with no /Resources of its own paints with its caller's patterns, but
its surviving pattern paint was recorded only in the form's own output and
never reached the page. With the page's own fill under the mark removed
whole, prune_patterns then dropped /P0 from the page while the form outside
the mark still painted with it, so that fill lost its paint in the saved file.
Pass the painted names up from a form that inherits its caller's resources.
Only the painted side goes up, so this can keep a pattern, never prune one.
* Redaction: a pruned pattern does not stay named by the page tree
prune_patterns writes only the page's own /Resources, and the guard walked
only pages' effective resources. A /Pages node that still named the pruned
pattern kept its cell reachable from Root, so a full save wrote the cell,
text included, while apply returned Ok. Three shapes did it: a page with no
/Parent inheriting from the root, a /Parent that disagrees with the /Kids
tree, and a root /Resources shadowed by a nearer holder that was retired.
Patterns pruned from a page are now unlinked from page-tree holders no page
inherits from, as unlink_doomed does for XObjects, when every /Parent agrees
with the tree. Whenever a pattern was pruned (or a selected page had any),
the guard also inspects the /Resources of every /Pages node Root reaches
down /Kids and up each page's /Parent chain, so whatever stays reachable is
proven safe or the redaction is refused, leaving the document unchanged.
* Redaction: a Type 3 glyph without resources keeps the page pattern it paints
A Type 3 font with no /Resources looks the names in its glyph descriptions
up in the resources of the page or form showing it, so a glyph outside the
mark can paint with the page's pattern. Text recorded only the fill and
stroke pattern of the graphics state, so with the page's own fill under the
mark removed whole, prune_patterns dropped the pattern and the glyph lost its
paint. Showing such a font now keeps every pattern of the scope, as a form
without /Resources keeps what it paints.
* Redaction: say what text-recorded pattern use keeps
Text, inline images and XObjects record their pattern before the hit test,
so a pattern painted only by redacted text keeps its cell. The guard refuses
a cell with text or images; a vector-only cell stays in the file. The comment
said this cost only an unused resource entry. Pin the behaviour in a test.
* Redaction: the page tree is followed up every /Parent link
A full save writes every node a /Parent link reaches, but the guard followed
/Parent only from pages and stopped at the first node it had already seen,
and unlinking never followed it. A /Pages node reached only through the
/Parent of a /Kids node -- a root /Pages with a /Parent, or an intermediate
node whose /Parent leads out of the tree -- kept naming the pruned pattern,
and its cell, text included, stayed in the saved file while apply returned
Ok.
The guard now follows /Parent from every node it visits, page or /Pages, with
the same seen-set against cycles. Unlinking treats a root with a /Parent as an
inconsistent tree, like a disagreeing link, and leaves the decision to the
guard, which inspects every node those links reach and refuses an unsafe
cell.
* Redaction: a long page tree does not exhaust the pattern guard's budget
The page-tree walk counted every node, pages included, against the guard's
16,384-object budget, so redacting a pattern page of a 17,000-page document
was refused with a limit error where it had applied before. Only the holder
nodes the guard inspects count against that budget now; the walk itself is
bounded at a million nodes, as unlink_doomed bounds the same tree.
* Redaction: a Type 3 font set by an ExtGState or shown in a form keeps the page patterns
The resourceless Type 3 check covered only a font set by Tf in the scope that
paints. A Type 3 font without /Resources set through an ExtGState /Font entry
was not flagged, and a glyph shown inside a form with /Resources of its own
recorded its patterns in the form's scope, which the page never saw. In both
cases the page's own fill under the mark was removed whole, prune_patterns
dropped the pattern, and the glyph outside the mark lost its paint.
An ExtGState font is now checked the same way, and a glyph shown in any
nested form marks the page as keeping every pattern it selected. This can
only keep patterns; whatever is kept is checked by the guard.
* Redaction: a pruned pattern a full save would still write is proven safe or refused
The walks that check retained pattern cells follow the references they know
-- resource categories, the page tree, appearances -- but a full save writes
whatever the trailer reaches by any reference. A direct /Parent dictionary, a
/Kids array on a page, or a stray resource entry (/Shading naming the pattern
stream) kept a pruned cell, text included, in the saved file while apply
returned Ok.
pdfcraft_cos gains reachable_objects, which runs the same breadth-first walk
write_full uses (factored out as full_save_order, so the two cannot drift).
Before apply succeeds, every pruned pattern that walk still reaches is
inspected like a retained cell: vector-only passes, anything else refuses
with the existing pattern error. It runs only when a pattern was pruned and
costs one walk of the reachable objects.
* Redaction: a direct /Parent dictionary is part of the page tree too
A /Parent given as a dictionary rather than a reference is written inline by
a full save, but the page-tree walk followed /Parent only as a reference, and
unlinking took such a tree for consistent. A root /Pages whose direct /Parent
held /Resources naming the pruned pattern kept its cell in the saved file.
The guard now walks a direct /Parent like any node (its /Resources, /Parent
and /Kids), and unlinking counts any /Parent that is not the reference /Kids
came from as an inconsistent tree, leaving the decision to the guard.
* Redaction: page-tree walks queue each node once and borrow /Kids
On a hostile tree whose /Pages nodes share one long /Kids array, the guard's
page-tree walk and pattern unlinking pushed every /Kids entry of every node
before checking whether it had been seen, and copied the array on each
visit. Both now queue only nodes not seen yet and read /Kids in place;
unlinking also stops at the first node reached twice, since its /Parent can
agree with at most one of the nodes above it, and leaves that tree to the
guard.
* Redaction: a Type 3 font with /Resources null is resourceless too
/Resources null is the same as no /Resources, but the check counted any
entry, so such a font's glyph outside the mark lost the page pattern it
painted with. A Type 3 glyph shown by a kept annotation appearance is a known
limit, pinned as an ignored test: appearances are not interpreted for pattern
use, and under ISO 32000 the glyph resolves in the appearance's own resources.
* Redaction: a pruned pattern is recorded along its whole reference chain
A pattern name can lead to a reference-valued object whose value is the cell
(/P0 7 0 R, object 7 being 6 0 R). Pruning recorded only object 7, so the
reachability backstop checked an object nothing reached while a stray
reference kept the cell, object 6, in the saved file. Every object along the
chain is recorded now, bounded like the other walks.
* Pattern tests: pin the file identifier where two saves are compared
main now gives a redacted rewrite a new identifier from fresh entropy, so an
incremental and a full save after Apply differ in /ID alone. The test that
checks they are the same save passes fixed entropy to both.
* Redaction: replies and pop-ups removed on other pages feed the shared-appearance check
Apply checks every page for retained unsafe pattern cells when a removed
annotation's appearance is shared with one that stays on an unselected page.
Only annotations removed under a mark fed that check. Replies and pop-ups
that go with them in step 2b (removed_everywhere), often listed on another
page, did not, so a text-bearing cell in an appearance a kept annotation
shares with them escaped the all-pages check and Apply returned Ok.
Their appearance references now go into removed_appearances too, with the
same limit. Raised by FredApps' review of the rebase.
---------
Co-authored-by: skylar taylor-barrick <hi@skylar.cc>
pdfcraft-redact
Layer L4 (may use edit): redaction, architecture §11.2.
// Marks are Redact annotations (pdfcraft-annot: Shape::Redact { quads, overlay }).
let marks: Vec<Mark> = marks(&doc); // page, areas (user space), fill, overlay text
let report = apply(&mut doc, None)?; // everything, or Some(&[pages])
clear_marks(&mut doc, None)?; // remove marks without applying
apply removes, under every marked area:
- text: glyphs are cut out of
Tj/TJ/'/"and replaced by aTJdisplacement of the same advance, so the rest of the line keeps its exact position. Widths come from/Widths,/W//DW(composite fonts, embedded CMaps with codespace and CID ranges), Type 3 font matrices, and approximations for the standard 14. A glyph goes when it overlaps an area by more than a fifth of its size (or a point) both ways; zero-size text goes when its origin is inside; - images: fully covered → removed; partly covered → a copy with the covered pixels cleared
(8/16-bit and 1/2/4-bit images, image masks; Flate/LZW/RL/A85/AHx, and CCITT fax or JBIG2 as
the last filter, decoded through
Document::decoded_image_within, a JBIG2 image with its/JBIG2Globals;/DecodeParmsgiven by reference, numbers inside included, count). The copy is Flate, or for a one-bit image of one component (a scan as a rule) CCITT Group 4, or JBIG2 when it was JBIG2, whichever is smallest as stored, coded from the cleared samples; it refers to no globals. Globals hold the shapes of every image that shares them (a scan's pages, often), those of marks under the area included, so every other image that uses the globals of a removed or replaced image is decoded and stored again in place without them, as a copy is (Flate, Group 4 or JBIG2, whichever is smallest;Report::images_restored), and they leave the file on the full rewrite; one that can't be decoded for certain refuses the redaction (RedactError::SharedJbig2Globals). Codecs redaction doesn't decode (DCT, JPX) are removed whole (fail-closed), and so is a JBIG2 image whose page isn't its size (the renderer draws the whole page into the image, so the rows cleared wouldn't be the ones shown) or of more than one bit or component, and Group 3 on bytes (/K0 or more,/EncodedByteAlign): PdfCraft's renderer draws some of its layouts (the fill ending each end-of-line code on a byte) as other rows than the decoder reads, so a mark over what is shown needn't cover what is cleared; inline images under an area are removed. Images with separate/SMaskor/Maskdata, and images whose data decodes to more than three times their declared raster and a byte per row, plus 4 KiB (ASCIIHex text broken into lines under Flate is a little over twice; for a fax image's code, about a byte per pixel), are removed whole; cleared copies drop/Alternatesso alternate pixels cannot survive; A run that would decode more of its images than one redaction may (pdfcraft_cos::ImageBudget, filters, decoders and pixels alike) is refused and changes nothing: such an image can't be cleared, and removing it whole would take what no mark covers with it. - vectors: covered paths are removed; partly covered paths and shadings are clipped so nothing paints inside the areas;
- form XObjects: rewritten recursively into new objects (pages sharing the original keep it);
- annotations whose rectangle overlaps (with their pop-ups and their replies,
/IRT), and form fields with a widget under a mark; - the marks themselves, replaced by boxes in their fill colour (and overlay text) drawn into the
page as a
/PCMark /Redactionstream.
A verification pass re-reads each redacted page and fails the operation if any glyph or inline
image is still under an area. It checks every content stream the page had, whatever its
dictionary calls it; only the overlay streams this apply added (Report::overlays) are skipped.
Text is placed with the font and size set by Tf or by an ExtGState's /Font through gs, and a
form XObject starts from the text state of the content that draws it. Content that can't be
decoded makes the operation fail rather than leave it unredacted, and so does an inline image the
renderer's tokenizer reads no image for (RedactError::Unreadable): where the page's operators
resume after it would be a guess. A Redact annotation with neither a usable /QuadPoints array nor a usable
/Rect makes it fail with RedactError::UnreadableMark before anything changes, instead of
skipping that mark. The count covers the marks on the pages being applied.
Tiling-pattern cells are separate streams. Applying a mark removes a pattern resource when no
surviving paint uses it. If a cell remains reachable after rewriting the selected pages and may
contain text, images, replacement descriptions, or unreadable content, Apply refuses with a
RedactError::Pattern and restores the document and pending marks. This includes patterns owned
by Forms and patterns inherited by Forms. Vector-only retained cells remain supported. The
bounded check is conservative: an unrelated retained pattern on a selected page can also cause
refusal when its content cannot be proven safe. If a selected page had a pattern resource, the
check covers all pages after rewriting so another page cannot keep the same cell in the saved PDF.
Before changing any page, apply prepares the redaction appearances with a shared 8 MiB
budget for their uncompressed PDF content across the selected pages. It bounds repeated-text
counts and working line buffers before allocating them, and measures and encodes each repeated
line once. Invalid or excessive appearances return RedactError::OverlayLimit without changing
the document. Reduce the marked area, disable repeated text, or apply fewer pages at a time.
This limits appearance generation; it is not a limit on the input PDF or the whole process's memory.
Structure cleanup resolves named MCIDs and indirect content references. Form content
references are updated to the cleaned copy; unchanged sibling descriptions remain. The structure
tree's object references (/OBJR) to the annotations and form fields apply removes are dropped,
as Sanitize drops them, so a full rewrite of a tagged PDF can't keep a removed comment or field value.
A tree where that can't be done (an element written in place, or one too large to walk) makes
apply fail with RedactError::Tags rather than leave the annotation reachable.
Replacement descriptions (/Alt, /ActualText, /E) are removed from marked-content spans
changed by cleanup, from every structure element above them (found through both /P and /K),
and from elements that lose an annotation reference, which also lose /T. An element written in
place above one whose descriptions go makes apply fail with RedactError::Tags. Unchanged
spans retain their descriptions; unused property entries are retired during redaction
and hidden-content cleanup, including aliases in shared/inherited resources. Descriptions still
used by surviving pages or forms are kept.
A redacted page that inherits /Resources from the page tree gets them pinned (by reference;
a direct inherited dictionary is made one shared object first) before it is rewritten. The parent's
/Resources is then retired unless another page still inherits it, so it cannot keep a removed
XObject reachable. Surviving form XObjects lose the resource entries for removed XObjects their own
content doesn't draw, for the same reason. Pages that aren't redacted are left untouched.
Remove hidden information / Sanitize
let found = sanitize::scan(&doc); // [(Hidden, count)] per category
sanitize::remove_hidden(&mut doc, &[Hidden::Comments])?; // chosen categories
sanitize::sanitize(&mut doc)?; // all of them
Categories: metadata (/Info, XMP), file attachments, comments, form fields (flattened so the
values stay visible), hidden text (render modes 3 and 7, or wholly off the page), hidden layers
(content of off optional-content groups, then the groups), bookmarks, links/actions/JavaScript,
and /PieceInfo private data. XMP metadata, private data, associated files (/AF) and
additional actions (/AA) are removed from every object reachable from the catalog that owns
them (images, fonts, Forms, annotations and fields as well as the catalog and pages), never from
name maps such as resource dictionaries. Content is read strictly: a page whose content can't be
read in full is refused, not cleaned in part. Both make the next save a full rewrite.
patterns finds phone numbers, e-mail addresses, credit card numbers (Luhn-checked), US Social
Security numbers and dates for Search & Redact.
A phone label such as Tel. or Fax: also enables Italian international geographic numbers with a slash, such as Tel. +39 0426/123456 or Fax: +390426/123456 (#3205 follow-up).
The country code and national number are matched together, with the significant initial 0 and 6 to 11 national digits described in the AGCOM numbering plan published by ITU.
This narrow case accepts at most one space after +39 and one slash immediately after the area code; international slash numbers without a phone label, or with country codes other than +39, remain unsupported.
Patterns are heuristic: dates, protocol identifiers and longer number runs are retained, and users still need to review the marks before applying them.
codes holds the redaction code sets (U.S. FOIA and U.S. Privacy Act exemptions):
CodeSet::from_id("foia")?.overlay(&["(b)(6)"]) gives the overlay text for the picked codes.
Custom code sets aren't supported yet.
Not yet: pattern locales, struct-tree/alt-text cleanup, re-encoding of DCT images (they are removed instead).