The redaction walk reads past bytes the content parser can't read: a
stray `)` or `}` in a `TJ` array, say. Text they hide is neither found nor
removed, while other readers draw it: Poppler's `pdftotext` extracts
`[(TAIL) ) (SECRET)] TJ` as TAILSECRET. The only refusal was a side effect
of the reads after the walk, which run only when the walk rewrote
something, read what it rewrote as rewritten, and in the font cleanup's
case only when the resources list a font. So Apply Redactions reported
success and left the text where the walk rewrites nothing (the only text
under the mark in such a `TJ`, or one in a form with resources of its
own), in a stream it rewrites (it wrote the array out as
`(TAIL) (SECRET) TJ`, without the stray byte or the brackets), and on a
page whose font is set only through an ExtGState. Remove Hidden
Information never counted hidden text in such a `TJ`: it found nothing to
remove, or left it beside what it removed.
The walk now records that it met such bytes (`Output::unparsed`, from
`Parsed::skipped`, nested forms included), and Apply Redactions refuses
the page with `RedactError::Unreadable`, as it does an inline image whose
end is a guess. Remove Hidden Information counts such a page in its scan,
so the removal runs and refuses it, as it does content that doesn't
decode; it refuses an inline image whose end is a guess the same way,
which it took on trust before. Apply walks only the forms that reach a
mark, so bytes in a form away from the marks don't make the walk refuse
the page; in a form without resources of its own, the check after the
walk still refuses them when it runs, as before.
Tests (`crates/redact/src/tests/unparsed.rs`): text under a mark in four
broken `TJ` arrays, alone and beside other text; the same in a stream the
walk rewrites; in a form under the mark with resources of its own; on a
page without font resources; hidden text in such a `TJ`, alone and beside
other hidden text, through the scan, Remove Hidden Information and
Sanitize Document; hidden text beside an inline image the renderer can't
end; and a form away from the marks, which is still redacted around. Two
tests in `tests.rs` now expect the walk's refusal; the check after the
walk keeps a case of its own, a form without resources away from the
mark.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Since 0a00862e9, what an image is stored again with (its size, filters and parameters) is
remembered by number and generation, so a value reached through a reference of another
generation is fetched again rather than taken from what the reference of its object's own
generation read. The README's list of what may be read again now says so. What an image names
is still remembered by object number, so a dictionary named that way isn't read again.
The same sentence said a chain of more than 32 references may be read again. `names_in` follows
33 references before it gives up on a chain, as `a_chain_of_references_is_named_to_the_limit`
says, so it is more than 33.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
The Consumers review of 64dfe1fd4. The README said a surviving form loses
the resource entries for removed XObjects its own content does not draw.
It also keeps those a form without resources of its own, drawn through
them, still draws, and since this PR a redacted page keeps them the same
way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
One conflict, in redact lib.rs: the module list, where main added `elsewhere` and
`identification` (#6714) and this branch `globals`. All three stay, in order.
Main's #2448 work also added a sixth step to `apply`, after the verification and the pattern
guard: `metadata::scrub` removes the `/Info` entries and the catalog's and pages' XMP that repeat
the removed text. An image that only such a packet refers to leaves the file then, after
`unshare` had looked for what left, so its shared dictionary stayed. `unshare` now runs after
the scrub, last of what can take something out. Storing images again changes no page's content
and no pattern cell, so the verification and the pattern guard before it check the same pages
they did. The crate's doc says the dictionary goes last. Remove Hidden Information's order is
unchanged: main's changes to `sanitize.rs` only read (`owned_values`).
Test: `an_image_only_the_scrubbed_metadata_refers_to_takes_a_shared_jbig2_dictionary_too`. An
image under the mark is also named by the catalog's XMP, which repeats the text removed beside
it. After the redaction the packet is gone, the image on the other page is stored again, and
the dictionary is gone too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Main's #1143 (863af864a), merged in 77b9cfa31, makes cos read a reference whose generation isn't
its object's as null. The walk that finds what shares a /JBIG2Globals dictionary with what left
fetched through `Document::get`, but remembered what it read by object number alone, so a
reference of another generation and the object's own one shared what the first of them read:
- What a parameters value names (`Run::names_in`): `/DecodeParms [10 1 R 10 0 R]` on an image
that left, object 10 naming dictionary 9, read 10 1 R as null, named nothing, and remembered
that for object 10. The 10 0 R after it found it, so nothing named 9, and the dictionary
stayed in the file with the image another page kept.
- A size, filters or parameters an image is stored again with (`Run::follow`): an image whose
`/Width 11 1 R` reads as null can't be stored again, and refused the run alone. Once another
image had read 11 0 R, it took that width and was stored with it.
What an image names is now read by object number, whatever generation the reference gives
(`Run::fetch_numbered`), in `names_in` and in the `/JBIG2Globals` chain alike: a reader that
doesn't compare generations draws with that object, so what an image that left names this way
may hold its shapes. That is how the chain already named `9 1 R` (the "another generation" way
out), and how both read before #1143. What is read by number is remembered by number. The
values an image is stored again with are read as cos reads them, and remembered by number and
generation, so a reference of another generation reads its own way.
`kept`'s comment said a reference reads the object of its number whatever its generation, which
is no longer so for a full save. It says how `reachable_objects` reads one now. The README and
`Run`'s comments say which reads take the number and which the generation too.
Tests:
- `a_redacted_scans_shared_jbig2_dictionary_leaves_the_file` gains three ways out:
- parameters through another generation, then their own;
- parameters through another generation alone;
- a `/JBIG2Globals` reference to a reference of another generation.
- `a_size_through_another_generation_isnt_the_one_its_own_reference_reads` stores no image
with the width another reference read. It refuses instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Conflicts, and how they were resolved:
- cos writer.rs and lib.rs: main now numbers a full save in `full_save_order` and exports
`reachable_objects`, the object numbers a full save writes, for the pattern guard. This
branch had added the same walk as `walk` and `reachable`. Main's is kept, with its alias
handling, and this branch's is dropped: `globals::kept` is now `reachable_objects`, so the
JBIG2 cleanup and the pattern guard read one walk. `kept` returns the walk's error (a save
that can't number its objects), which redaction and Remove Hidden Information pass on. The
cos test of what is reachable, and two redaction tests, read `reachable_objects` instead.
The exports keep this branch's `DECODE_PARMS_KEYS` and main's `TextChars`,
`civil_from_days` and `new_file_id`.
- redact lib.rs: main unlinks pruned patterns from `/Pages` nodes after `unlink_doomed`; this
branch replaced `restore_shared_jbig2` with `globals::unshare`. Both stay, the unlink first:
an image a pruned pattern's cell drew leaves the file with it, and is a user of a dictionary
that left too.
- redact sanitize.rs: main's `strip_outline_actions` and this branch's `unshare_unmarked` both
stay. At the end of `remove_hidden`, `unshare_unmarked` runs after main's `unlink_undrawn`,
so an image unlinked for being drawn nowhere takes a shared dictionary with it, and is
followed by main's `require_full_save_with_new_id`.
- redact image.rs: `MAX_PIXELS` is `pub(crate)` on both sides; the comment names
`globals::without_dictionary`, where this branch moved `without_globals`.
Tests: `an_undrawn_image_unlinked_takes_a_shared_jbig2_dictionary_too` (Remove Hidden
Information, through `unlink_undrawn`) and
`an_image_a_pruned_patterns_cell_drew_takes_a_shared_jbig2_dictionary_too` (a redaction,
through `unlink_pruned_patterns`) pin the order of the two new routes before `unshare`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* Redaction: a pattern nothing paints with any more leaves the page
A tiling pattern's cell is a content stream of its own and can draw
anything, text included. Redacting a shape filled with such a pattern
removed the fill from the page's content -- the page looked right, a
black box -- but left the pattern in /Resources /Pattern, and its cell,
with the text that was under the mark, was written into the saved file
unchanged.
The pass does not enter pattern cells, so it cannot redact inside one.
What it can do is notice that nothing paints with the pattern any more.
The graphics state now carries the fill and stroke pattern a `scn` or
`SCN` selected (and `cs`, `g`, `rg`, `k` and their stroke forms clear
it), every painting operator that survives the rewrite records the
pattern it drew with, and a pattern the page selected but no surviving
paint used leaves the page's resources.
Measured on the report's file:
before applied: 1, (SECRET) in the saved bytes
after applied: 1, no /Pattern on the page, (SECRET) gone
A pattern some surviving paint still uses is kept, and so is its cell:
with two boxes filled from one pattern and only the first marked,
patterns_removed is 0 and the resources keep the entry -- the second box
still shows the word, so it is not a leak.
Report gains patterns_removed.
Fixes#3647.
* Keep a pattern every operator that can paint with it still uses
Two defects CodeRabbit found in the first version of this change, both
of which could remove a pattern the page still draws with.
Pattern use was recorded for path painting only. Text shows with the
fill pattern too -- and with the stroke pattern in the stroking render
modes -- as do image masks and anything a form draws, and a form's
content is interpreted with a graphics state of its own, so a pattern it
inherits is never seen inside it. Content like
/Pattern cs /P0 scn 20 140 100 30 re f BT (KEEP) Tj ET
with the box redacted therefore lost the fill of text still on the page.
Every operator that can paint with the current pattern now records it:
text (except mode 3, which paints nothing, and mode 7, which only
clips), inline images, and image or form XObjects. Only path painting
knows exactly what it drew and whether the paint survived, so the others
are recorded as they are met. That can keep a pattern whose paint was in
fact all redacted, which costs an unused resource entry, while dropping
one something still draws with takes paint off the page.
/Resources /Pattern may be an indirect object, like any value. Reading
the reference itself as a dictionary found none and pruned nothing, so
on such a file the cell -- with the text that was under the mark --
stayed in the saved bytes, which is the leak this change is for. It is
resolved now, and what is written back is a dictionary of this page's
own, so pruning never reaches through a /Pattern dictionary another page
shares.
The controls in the test: without the first fix the surviving text's
pattern is removed (left: 1, right: 0), and without the second the
indirect file prunes nothing (left: 0, right: 1).
* M8: refuse retained unsafe tiling patterns
* M8: guard shared appearances and narrow pattern scans
* M8: update TLS dependencies for new advisories
* Redaction: the comment on prune_patterns said more than the code does
`prune_patterns` drops the `/Pattern` resource entry; the `scn` that selected
it stays in the content, so the page names a pattern its resources no longer
hold. The comment claimed the selector went too (noted in review).
ISO 32000-1 8.6.6.2 makes that operand a key of the `/Pattern` subdictionary,
so the open question is whether a reader stops reading the stream at the stale
name. `a_page_whose_pruned_pattern_is_still_selected_renders` answers it with
the renderer rather than with prose: the redacted page is saved, reopened and
rasterized, and
* it renders with no error and the same 300x300 page box,
* the text written *after* the stale selector is still painted -- which is
exactly what a reader would drop if it gave up at the bad operand,
* the redaction overlay is black over the box the pattern filled.
Control: with the drawing after the selector removed, the pixel assertion fails
("the text after the stale selector is still painted"), so it is reading that
text and not a stray pixel.
The selector is left in place rather than cut out: removing operators is a
rewrite of the content this pass does not otherwise do, and nothing surviving
paints with that colour. `pdfcraft-render` is a dev dependency of this crate
only -- layer 4 on layer 3, which the layering allows.
* WIP: pattern guard depth bound sized from the interpreter's
* Pattern tests: keep the guard's cases reaching the guard now a hit path is removed whole
* Pattern automation tests: keep the pattern in use where they test the guard, since a hit path is now removed whole
* Redaction: a form without resources keeps the page pattern it still paints
A form with no /Resources of its own paints with its caller's patterns, but
its surviving pattern paint was recorded only in the form's own output and
never reached the page. With the page's own fill under the mark removed
whole, prune_patterns then dropped /P0 from the page while the form outside
the mark still painted with it, so that fill lost its paint in the saved file.
Pass the painted names up from a form that inherits its caller's resources.
Only the painted side goes up, so this can keep a pattern, never prune one.
* Redaction: a pruned pattern does not stay named by the page tree
prune_patterns writes only the page's own /Resources, and the guard walked
only pages' effective resources. A /Pages node that still named the pruned
pattern kept its cell reachable from Root, so a full save wrote the cell,
text included, while apply returned Ok. Three shapes did it: a page with no
/Parent inheriting from the root, a /Parent that disagrees with the /Kids
tree, and a root /Resources shadowed by a nearer holder that was retired.
Patterns pruned from a page are now unlinked from page-tree holders no page
inherits from, as unlink_doomed does for XObjects, when every /Parent agrees
with the tree. Whenever a pattern was pruned (or a selected page had any),
the guard also inspects the /Resources of every /Pages node Root reaches
down /Kids and up each page's /Parent chain, so whatever stays reachable is
proven safe or the redaction is refused, leaving the document unchanged.
* Redaction: a Type 3 glyph without resources keeps the page pattern it paints
A Type 3 font with no /Resources looks the names in its glyph descriptions
up in the resources of the page or form showing it, so a glyph outside the
mark can paint with the page's pattern. Text recorded only the fill and
stroke pattern of the graphics state, so with the page's own fill under the
mark removed whole, prune_patterns dropped the pattern and the glyph lost its
paint. Showing such a font now keeps every pattern of the scope, as a form
without /Resources keeps what it paints.
* Redaction: say what text-recorded pattern use keeps
Text, inline images and XObjects record their pattern before the hit test,
so a pattern painted only by redacted text keeps its cell. The guard refuses
a cell with text or images; a vector-only cell stays in the file. The comment
said this cost only an unused resource entry. Pin the behaviour in a test.
* Redaction: the page tree is followed up every /Parent link
A full save writes every node a /Parent link reaches, but the guard followed
/Parent only from pages and stopped at the first node it had already seen,
and unlinking never followed it. A /Pages node reached only through the
/Parent of a /Kids node -- a root /Pages with a /Parent, or an intermediate
node whose /Parent leads out of the tree -- kept naming the pruned pattern,
and its cell, text included, stayed in the saved file while apply returned
Ok.
The guard now follows /Parent from every node it visits, page or /Pages, with
the same seen-set against cycles. Unlinking treats a root with a /Parent as an
inconsistent tree, like a disagreeing link, and leaves the decision to the
guard, which inspects every node those links reach and refuses an unsafe
cell.
* Redaction: a long page tree does not exhaust the pattern guard's budget
The page-tree walk counted every node, pages included, against the guard's
16,384-object budget, so redacting a pattern page of a 17,000-page document
was refused with a limit error where it had applied before. Only the holder
nodes the guard inspects count against that budget now; the walk itself is
bounded at a million nodes, as unlink_doomed bounds the same tree.
* Redaction: a Type 3 font set by an ExtGState or shown in a form keeps the page patterns
The resourceless Type 3 check covered only a font set by Tf in the scope that
paints. A Type 3 font without /Resources set through an ExtGState /Font entry
was not flagged, and a glyph shown inside a form with /Resources of its own
recorded its patterns in the form's scope, which the page never saw. In both
cases the page's own fill under the mark was removed whole, prune_patterns
dropped the pattern, and the glyph outside the mark lost its paint.
An ExtGState font is now checked the same way, and a glyph shown in any
nested form marks the page as keeping every pattern it selected. This can
only keep patterns; whatever is kept is checked by the guard.
* Redaction: a pruned pattern a full save would still write is proven safe or refused
The walks that check retained pattern cells follow the references they know
-- resource categories, the page tree, appearances -- but a full save writes
whatever the trailer reaches by any reference. A direct /Parent dictionary, a
/Kids array on a page, or a stray resource entry (/Shading naming the pattern
stream) kept a pruned cell, text included, in the saved file while apply
returned Ok.
pdfcraft_cos gains reachable_objects, which runs the same breadth-first walk
write_full uses (factored out as full_save_order, so the two cannot drift).
Before apply succeeds, every pruned pattern that walk still reaches is
inspected like a retained cell: vector-only passes, anything else refuses
with the existing pattern error. It runs only when a pattern was pruned and
costs one walk of the reachable objects.
* Redaction: a direct /Parent dictionary is part of the page tree too
A /Parent given as a dictionary rather than a reference is written inline by
a full save, but the page-tree walk followed /Parent only as a reference, and
unlinking took such a tree for consistent. A root /Pages whose direct /Parent
held /Resources naming the pruned pattern kept its cell in the saved file.
The guard now walks a direct /Parent like any node (its /Resources, /Parent
and /Kids), and unlinking counts any /Parent that is not the reference /Kids
came from as an inconsistent tree, leaving the decision to the guard.
* Redaction: page-tree walks queue each node once and borrow /Kids
On a hostile tree whose /Pages nodes share one long /Kids array, the guard's
page-tree walk and pattern unlinking pushed every /Kids entry of every node
before checking whether it had been seen, and copied the array on each
visit. Both now queue only nodes not seen yet and read /Kids in place;
unlinking also stops at the first node reached twice, since its /Parent can
agree with at most one of the nodes above it, and leaves that tree to the
guard.
* Redaction: a Type 3 font with /Resources null is resourceless too
/Resources null is the same as no /Resources, but the check counted any
entry, so such a font's glyph outside the mark lost the page pattern it
painted with. A Type 3 glyph shown by a kept annotation appearance is a known
limit, pinned as an ignored test: appearances are not interpreted for pattern
use, and under ISO 32000 the glyph resolves in the appearance's own resources.
* Redaction: a pruned pattern is recorded along its whole reference chain
A pattern name can lead to a reference-valued object whose value is the cell
(/P0 7 0 R, object 7 being 6 0 R). Pruning recorded only object 7, so the
reachability backstop checked an object nothing reached while a stray
reference kept the cell, object 6, in the saved file. Every object along the
chain is recorded now, bounded like the other walks.
* Pattern tests: pin the file identifier where two saves are compared
main now gives a redacted rewrite a new identifier from fresh entropy, so an
incremental and a full save after Apply differ in /ID alone. The test that
checks they are the same save passes fixed entropy to both.
* Redaction: replies and pop-ups removed on other pages feed the shared-appearance check
Apply checks every page for retained unsafe pattern cells when a removed
annotation's appearance is shared with one that stays on an unselected page.
Only annotations removed under a mark fed that check. Replies and pop-ups
that go with them in step 2b (removed_everywhere), often listed on another
page, did not, so a text-bearing cell in an appearance a kept annotation
shares with them escaped the all-pages check and Apply returned Ok.
Their appearance references now go into removed_appearances too, with the
same limit. Raised by FredApps' review of the rebase.
---------
Co-authored-by: skylar taylor-barrick <hi@skylar.cc>
The README said apply "refuses and changes nothing" and that remove_hidden
takes the dictionaries out "the same way", which reads as if a refused
remove_hidden changed nothing too. It has changed the document by then
(a_refused_removal_of_hidden_information_doesnt_say_nothing_changed: the
comment goes before the refusal), as its doc says, so a caller passes a copy it
can drop. Raised by CodeRabbit on #5698.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Since b1f221611 the dictionary is read within the allowance too, so the test
that gave unshare no allowance was refused reading the dictionary, before the
image was decoded: turning the image's refusal into "can't be read" left every
test green (Claims review of 5bef9b0e7). The test now also gives exactly what
reading the dictionary costs, so the image is what is refused.
The same review: the README lists the other images that can't be stored
again (more than one component, no size) and names ImageBudget::decode, not
Document::decoded_image_within, which #5569 made private;
SharedJbig2Budget's doc and remove_hidden's comment name the dictionaries read
to know copies too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Main (#5569) decodes every image through cos's ImageBudget, which counts the
JBIG2 decoder's own work and the globals it reads, through main's own
restore_shared_jbig2 and image::without_globals for the images that share a
redacted image's /JBIG2Globals too. This branch replaces both with
globals::unshare, so the merge keeps unshare and drops them.
without_dictionary now decodes through ImageBudget::decode instead of
Document::decoded_image_within, which main made private. That closes the gap
the PR noted: the decoder's own work (up to hayro-jbig2's 2^30 steps an image)
was charged one unit of the run budget. apply passes its run's allowance, what
clearing left of it, as main's restore did. Past it apply is refused with
ImagesPastAllowance, as main's restore was. remove_hidden gives unshare an
allowance of its own (sanitize::unshare_unmarked), and past it is refused with
SharedJbig2Budget: "redact fewer pages at a time" means nothing there, and
Optimize's discard runs it too. SharedJbig2Budget's text now says finding and
storing again.
Tests: main's allowance test runs on unshare, with ImageBudget::with_work(0)
refused as ImagesPastAllowance, through unshare_unmarked as SharedJbig2Budget,
and u64::MAX storing one image. Test helpers decode through ImageBudget, since
Document::decoded_image is no longer public.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
A review of ecf028965 measured that decoding an image undoes the filters
of each dictionary it names again when cos doesn't hold their bytes (20
images naming a 32 MiB dictionary: 1.2 s at 2 units each). content()
now remembers what each dictionary decoded to, or its limit when it
couldn't, and an image is charged that before it is decoded.
Names gathered from many entries are held as a set as they are read, so
a million entries naming one chain hold it once. The claim that each
object is fetched once a run for each way it is read says its exception:
a /JBIG2Globals chain named part way is followed again from there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Conflicts:
- crates/redact/src/lib.rs: main added the run's ImageBudget to Applying
and Scope::new; kept both sides (the removed-text fields and the image
budget), with main's refused-images check before the destructuring.
- crates/redact/src/tests.rs (auto-merged): this PR's two tests that build
a Scope or an Applying now pass main's ImageBudget.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyG234xeYkJfr41E6iP
A review of fee25bfe4 found three costs the budget didn't count:
- A /JBIG2Globals reference in parameters given directly was fetched
again for every entry of every stream (2,000 entries, 53 KB: 40 s, two
thousand units of four million). What a reference leads to is now
remembered for the run under it, and a chain met part way takes what
is known from there.
- A /Filter name was kept whatever it was, so a 32 MiB name referred to
nine times was held nine times and copied for each image (725 MiB from
2.8 KB). Only a name a decoder knows is kept; any other is no filter,
and a name longer than any decoder's isn't looked at.
- A decode that failed spent nothing, so a dictionary one byte past its
limit escaped the budget. A failure now costs as much as it was allowed
to decode: a dictionary MAX_JBIG2_GLOBALS, an image its allowance.
The README says what the budget counts, and that past it the redaction
is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
/FDecodeParms are the parameters of data in another file, which a reader
may draw instead of the stream's own. A dictionary that only a removed
image named there stayed in the file with Ok. It is now read with
/DecodeParms and /DP, and an image whose data is in another file counts
as one that can't be stored without the dictionary: readers draw that
file, not a copy stored here.
images_restored counts what was stored again. A dictionary that names
itself as its own globals (malformed) leaves with what used it but can't
be stored, and was counted.
The table's /Mask route gets a row in the routes test, as it was only
measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Redaction starts from the images a run replaces or removes and the forms
and soft masks under them, so a dictionary used by something that left
another way stays, with apply returning Ok: a stamp or widget appearance, a
Type 3 glyph, an /Alternates entry, a /Mask, or an image with /Filter by
reference or /DecodeParms through a reference to a reference. A
byte-identical copy stays too, which is what pdfseparate then pdfunite
leaves, and Remove Hidden Information never takes a dictionary out.
Compare what a full save would keep before and after the run instead, the
same walk write_full makes, now cos::reachable. What left names the
dictionaries, in any entry of its parameters; what stays that names one, or
a copy, is stored again without it; and the walk is taken once more, so a
dictionary still kept refuses the run, whatever keeps it. An image with
/Width, /Height or /Filter by reference is read, and is one bit in the copy,
as it was when drawn. /Alternates stays with what is stored again, and a
stream that isn't JBIG2 refuses rather than being stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
An image that shared a redacted image's /JBIG2Globals is now stored
again as the smallest of Flate, Group 4 or JBIG2 coded alone (smallest),
like a cleared copy, rather than always Flate, so the other pages of a
scan don't balloon either. The shared-globals test's second page is a
ragged scan and checks it is no longer Flate and is smaller.
A JBIG2 image's /JBIG2Globals hold the symbol dictionary of every image that
shares them (jbig2enc -s and most scanners give one to all of a scan's pages),
so the bitmaps of marks drawn only under a redaction stayed in the file while
another page's image used them. Every image using the globals of a removed or
replaced image (or of one in a removed or replaced form) is now decoded and
stored again in place as Flate without them; a full save then drops them. One
that can't be decoded for certain refuses the redaction
(RedactError::SharedJbig2Globals). Report::images_restored counts them.
#2539 review: the scrub removes the catalog's XMP packet whole when it holds
the removed text, and with it the file's PDF/A and PDF/UA claims, which
Report didn't say. The identification (pdfaid part, conformance, amd, rev;
pdfuaid part, rev) is now read by namespace (roxmltree, already in the build
through pdfcraft-xfa), each value kept only when it is the short code it must
be and doesn't hold the removed text (one-character codes always), and
written back in a packet holding nothing else, with a fixed pdfuaid extension
schema under PDF/A-1 to -3 (left out, with the PDF/UA claim, if the redacted
words are among its words). Report::identification says, per standard,
None / Kept / Lost; the engine passes it on as
EditNotice::RedactionIdentification, the Apply notification says it after the
count, and redact_apply returns identification { pdfa, pdfua }.
Tests (redact): the_identification_of_a_removed_packet_is_written_back,
a_packet_without_identification_is_only_removed,
an_identification_that_cannot_be_kept_is_reported_lost,
unsearched_text_keeps_only_one_character_codes,
the_extension_schema_is_left_out_when_it_uses_a_redacted_word; engine
applied_redactions_report_the_pdfa_identification_written_back; automation
redact_apply_reports_the_identification_written_back; ui-egui
applying_says_the_pdfa_identification_was_written_back. With the write-back
replaced by "nothing kept", the redact tests fail (Kept expected, None
found); with the notice not pushed, the engine, automation and UI tests fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
#2539 review: ISO 32000-2 §7.9.2.2.1 defines text strings in PDFDocEncoding,
UTF-16BE (FE FF) and UTF-8 (EF BB BF) only, so an FF FE string was read
through PDFDocEncoding and Cyrillic or CJK text in it could never match.
It is now read as UTF-16LE for the search (fail-closed), in the /Info scrub
and in the bookmark, attachment and comment counts.
Test: a_utf16le_string_is_searched_as_utf16le (failed before: 0 /Info
entries removed; with only the count reverted, bookmarks 0 instead of 1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
#2539 review: a decomposed "Müller" in a UTF-16BE /Title, a U+FB01 ligature
from a ToUnicode map and full-width letters never matched their plain forms.
Both sides are now normalized to NFKC and lower case (unicode-normalization,
already in the build through pdfcraft-crypt's stringprep, so no new crate).
Normalized text more than four times its input's length (U+FDFA and the like)
isn't searched: it counts as holding the text.
Tests: unicode_forms_of_the_removed_text_match (failed before: Title, Subject
and Keywords kept), a_value_that_normalizes_to_far_more_text_counts_as_holding_it
(failed with the growth limit raised to 100: 0 removed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
#2539 review: a heading tracked with TJ numbers ([(B) -300 (L) -300 ...] TJ)
was collected as "B L U E F I N", which neither held "Bluefin" nor had a word
of three letters, and a value spelled "Blue fin" or "BLUE-FIN" didn't hold the
term "bluefin". A TJ gap can't be told from a word space, so the glyph ends stay
as they are; instead every term with three letters or digits is also looked
for by its letters and digits alone, and a value is part of a run when its
letters and digits are, starting and ending at the run's word edges ("True"
still isn't part of "untrue"). It over-matches only towards removal.
Tests: a_heading_tracked_with_tj_numbers_matches_the_metadata,
a_value_spaced_or_hyphenated_otherwise_matches (both failed before: /Title
kept, metadata 0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
Conflict in crates/engine/src/document.rs: main turned `edit_notices` into
`Vec<EditNotice>`; the PR's `redaction_elsewhere` field is kept beside it in
this merge, and the next commit makes it an `EditNotice` variant.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
Report::still_holding (Elsewhere) counts the places apply leaves as they are that still hold the
removed text: bookmark titles, attachment names, file names and descriptions, the text and authors
of the comments that remain, and XMP on objects other than the catalog and the pages. Nothing is
changed; Remove hidden information removes them. The text is matched as the metadata check
matches it (one Search type now serves both), with its own work limit. When the removed text
couldn't be searched for, every place holding any text counts and `unsearched` says so. The walks
read each outline item and name-tree node once, within caps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
End-to-end tests for a removed run over 64 KiB, more than 20,000 terms, a run over 4,096
characters, and XMP that is corrupt, cut short, behind two filters, over 16 MiB or in another
encoding. Each was seen failing with its guard removed or loosened.
The corrupt and cut-short cases showed that the tolerant decoder returned the part of the packet
before the damage, which was then searched as if it were the whole packet. XMP is now decoded
strictly, and a packet behind more than one filter goes unread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
Four ways round the allowance, each measured by a review of hostile input:
A Flate stream read as zlib that fails at once, and then as raw deflate that
decodes everything before failing, gave nothing back and so was charged
nothing, here and in DecodeBudget: 110 s for a 387 KB file, all decoded. What
the retry decoded now counts, though the data is still dropped as it was.
Fax and 1-bit Flate images paid by the byte for work done by the pixel: 19
bytes can be a blank page of 100 million pixels, about a second each. Every
image decoded here now pays for its pixels first, as JBIG2 already did.
A JPEG paid nothing, and what one asks for doesn't grow with its length: 156
bytes declared 80 megapixels, 72 s for an 8 KB file. A JPEG pays for its
samples before it is decoded, in Optimize and in an export that has to decode
one for its mask.
Merging fragments paid for reading strips and not for the picture it built:
393 s for 1,154 bytes. It pays for the picture before allocating it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Filters were charged only for what they handed a JBIG2 decoder, once they
had handed it over: a chain that failed, or went past its image's limit,
cost its time and nothing else, and images that weren't JBIG2 weren't
charged at all. A 734 KB file of 200 images of Flate inside Flate took an
image export 111 s, and that grew with the file. It now takes 12 s.
Each byte read and each byte a stage decodes is now a step of the same
allowance, paid as it happens, so no stage runs past what is left. That
holds for every image read through ImageBudget, and a fax image pays for
its rows and its code. JBIG2 globals pay when they have to be read, and
kept ones don't. Optimize's two reads that went round the budget go
through it, and merging fragments shares it.
Optimize reads an image twice, to pack it and to resample it, so a refused
image is now counted once a run rather than once a read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
From a review of who reads a refusal.
The rule is to leave it out and say so, and it held on some paths only. An image whose JBIG2 mask is
refused no longer comes out opaque. The first image past the allowance reads as that and not as
corrupt. A page reports what it left out. The Office warning stopped blaming forms. Redaction refuses
the run, leaving the document as it was: removing a readable image whole would take what no mark
covers with it, unreported.
The 2 GiB an export holds is twice the document if that is more, so a large honest file loses nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
One image has its own limit, about 6 s to 17 s, but nothing bounded a file of thousands of images
made to reach it: an operation took that long for each, and an image sharing /JBIG2Globals decodes
them again.
Image::decode_within (hayro-jbig2 patch 11) spends from what a run may still do, starting with its
page's pixels, and charge_reading charges reading a stream and its globals, 4 steps a byte, before
anything is parsed, so an image that fails has paid too. pdfcraft_filters::decode_jbig2_within takes
it, and ImageBudget in cos is that allowance: 2^34 steps, about 1,000 scanned pages, and 2^13 more for
each byte of the file, so a long scan is read whole. An image past it spends all that's left, and
later ones aren't read, globals included.
Optimize leaves such an image as it is and counts it in Report::images_refused. Image export leaves it
out, with the reason, and stops at 2 GiB of files held. Office export shares one allowance, leaves the
picture out and says the page was truncated. Redaction removes it whole, as any image it can't read.
Rendering gives a page's images 2^30 steps between them, inside the watchdog, and each page starts
afresh in Context::new as its other budgets do (hayro-syntax patch 31, hayro-interpret patch 73).
It's a thread-local, unbounded until a page sets it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Review on #2539 (nitpick): the README named only the collection limit
and unreadable XMP as fail-closed cases. It now lists all of them (the
run and run-length limits, a removed glyph with unknown text, a form
XObject removed unread past the depth limit or unreadable, the term
limits, the search work limit, oversized or undecodable XMP) and the
metadata the check leaves to Sanitize: XMP on objects other than the
catalog and pages, bookmark titles, and attachments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq7HyCfwzyRXj4VfQAVVdv
Redaction wrote every cleared copy as Flate, so a redacted Group 4 or JBIG2 scan
page grew several times over until the optimizer ran. A cleared one-bit image of one
component is now stored as the smallest of Flate, CCITT Group 4 and, when it was
JBIG2, JBIG2 (PDF 1.4, which such a file already is), as stored, dictionary included:
a tiny image can lose by Group 4's parameters. Both are coded from the
cleared samples, so nothing of the original's code is carried over; rows Group 4 or
JBIG2 can't code (too wide) leave Flate. Images of more bits stay Flate.
Tests: a cleared one-bit Flate scan of ragged word blocks (where Group 4 is smaller
than Flate) comes out Group 4 and an eight-bit image Flate, each decoding to the
cleared samples (the whole buffer); a tiny image whose Group 4 code is shorter but whose
stream isn't, and rows wider than Group 4 codes, stay Flate and clear; the redacted
JBIG2 scan of #5461's test comes out JBIG2, smaller than
Flate, with no globals and no stream holding the original code. The tests' image
helpers decode through Document::decoded_image, as image codecs need.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #283. A scan stored as JBIG2 (as the optimizer writes them) is decoded
through Document::decoded_image_within, its /JBIG2Globals included, and its covered
pixels cleared in a Flate copy that refers to no globals, as CCITT fax images are.
Before, any JBIG2 image under a mark was removed whole, blanking the page.
Still removed whole (fail-closed): a JBIG2 page of another size than the image,
which decoded_image refuses (the renderer draws the whole page into the image, so
the rows cleared wouldn't be the ones shown); globals that can't be read; damaged
data; an image claiming more than one bit or one component, even where its rows
take as many bytes; a page far past the image's raster, refused before it is
decoded.
Globals no other image uses leave the file with the original on the full rewrite
(tested); globals another image still uses stay, and the README and parity note say
they may hold the shapes of marks from under the area, not where they were.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-up on #2539:
- the glyph end includes character and word spacing, so letter-spaced
redacted text is one word and still matches /Title (Bluefin);
- an XMP value wholly part of a removed run goes, as an /Info value does
(J. Li), compared as whole words (True stays with "untrue");
- when the removed text was cut short (too many runs, a run too long, too
many terms) every /Info string and XMP stream goes instead of searching
what was kept;
- XMP that is not valid UTF-8 or UTF-16 (also UTF-16 without a leading
mark) is not searched lossily but removed;
- namespace declarations and the xpacket wrapper are not searched;
comments and other processing instructions are;
- the raw-byte assertion saves without object streams so it can see /Info.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An /Info value whose words are too short to be terms ("J. Li") survived
when the sentence holding it was redacted. A value that is wholly part
of a removed run, with three letters or digits at least, now goes as
well (the reverse check #2598 proposed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Applying redactions removed the glyphs under the marks but left the same
words in the document information dictionary and the XMP metadata.
The redaction interpreter now collects the removed glyphs' text in runs by
position. After verification, /Info entries containing a run (or one of its
longer words) are removed, and the catalog's and pages' XMP streams whose
text mentions one are removed whole; unreadable or oversized XMP fails
closed. Report::metadata counts what went.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PdfCraft's bootstrap renderer draws byte-aligned Group 3 whose fill ends
each end-of-line code on a byte (as TIFF writes it) as other rows than
decode_ccitt reads (black box, at every /K, with /EndOfLine or not:
mostly the first row, then blank). Redaction cleared pixels in the
picture decode_ccitt reads, so a mark over what PdfCraft showed needn't
cover what was cleared, and content shown blank stayed in the copy. Such
an image (/K 0 or more, /EncodedByteAlign) is now removed whole, as
every CCITT image was before this PR; Group 3 off bytes and Group 4 are
still cleared.
The image's parameters come from Document::fax_params, the reading of a
fax image's /DecodeParms entry that decoded_image already did, now on
its own (it was to come with #4404, which uses it too). The redaction
README, the module doc, the redaction and core.image-filters notes and
the spec parity table's status line and revision row say so, and the
core.image-filters note now records the renderer's misdrawing.
group_3_on_bytes_is_removed_whole_and_other_fax_images_cleared: a white
16 × 2 picture, checked to decode as such, is removed whole on bytes (no
codes; codes at /K 0; at /K 1 with /EncodedByteAlign by reference) and
cleared off bytes and as Group 4 on bytes. It fails without the guard;
cargo-mutants catches every viable mutant of it.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
decode_ccitt reads byte-aligned Group 3 whose data has end-of-line codes
but none before the first row (not as T.4 sends them) on bytes. At /K 0
with the fill ending each code on a byte, about one small picture in
five is then read as other rows; at fax width almost none are, as such
pages come out short and are refused. Export already said so. The
optimizer, when it resamples such an image or stores it again as Group 4
from behind another filter, and redaction, when it clears pixels in one,
write those rows back, and nothing said that.
Black-box, PdfCraft's renderer draws such an image exactly as
decode_ccitt reads it (all 42 misread of 200 seeded pictures, behind
ASCII85Decode too), so the written image is what PdfCraft showed before.
Refusing such data instead (a code after nearly every row but none
first) would also refuse some ordinary Modified Huffman images, whose
rows can meet 11 zeros by chance (0.5 to 5% of small ones in a
black-box probe), so the limit is stated rather than guarded: in
decode_ccitt's doc, the filters, optimize and redact READMEs, and the
core.image-filters and mono-compression notes. The optimize README's and
the mono-compression note's "not yet: Group 3 to Group 4 without
resampling" now say a lone Group 3 image: one behind another filter is
stored again as Group 4.
Also from the review: docs/roadmap.md keeps #4335's status line under
this PR's, rather than replacing it; the filters README's tests and the
core.image-filters evidence name the Group 3 layout test; the filters
crate's description names Group 3 and 4 decoding.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
Mutation testing found behaviour no test held, now covered:
- a Group 4 halftone above 450 ppi isn't resampled and stays byte for
byte, alone by default and behind ASCIIHexDecode with Retain;
- under Retain a resampled Flate scan stays Flate, though Group 4 would
be smaller (ragged bands);
- sixteen-bit grey of only 0x0000 and 0xFFFF keeps its bytes;
- a busy Group 4 image behind ASCIIHexDecode, whose code is over three
times its raster, decodes within the fax bound and is stored again as
Group 4 alone. Redaction's busy-fax test is that busy again: since the
bound for other images is three times the raster, the old checkerboard
fitted it, and dropping the fax bound passed.
The redaction README says three times, and doc_optimize's description
says what Retain keeps: CCITT stays CCITT, anything else becomes Flate.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6yUtvWGPsJpMLyh6pxSXG
- Redacting a CCITT image went through decoded_image, which undoes the
filters before the fax decoder up to MAX_DECODED, so a Flate bomb in
front of a fax image cost a gigabyte while every other image stops at
twice its raster. Document::decoded_image_within takes a bound, and
redaction gives a fax image's code about a byte per pixel. A fax image
cut short, or behind a bomb, is removed whole; a busy one whose code
is bigger than twice its raster is still cleared.
- Every image is now decoded through decoded_image_within, which reads
/DecodeParms through Document::with_direct_parms: given by reference,
the numbers inside (/Predictor 7 0 R) included. That reading was first
written here and has moved down to #4304. The predictor test now covers
the dictionary, an array's entry and the number itself given by
reference; read without them, the cleared bytes drift off the covered
pixel and the covered sample stays in the copy.
From reviews with the pdfcraft-review skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Create PDF and the optimizer now store black-and-white scans as Group 4,
and redaction couldn't decode CCITT, so redacting one word on such a
scan removed the whole page image. A CCITT image (the last filter,
after any others) now decodes through Document::decoded_image and is
cleared in a Flate copy like any other image; one that can't be decoded
in full is still removed whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
