- cos: a filter chain whose first stage stops at damage and whose
second reads all it was given is damaged.
- optimize: a page damaged partway stops resampling for the whole
document, so another page drawing the same image whole doesn't get
it resampled.
- redact: after redacting content with /DecodeParms by reference, no
stream in the saved file holds the text, read with its parameters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
Optimize read a page's content up to any damage in its streams as the whole
of it. Merging a page's image strips then wrote the page's content again as
one stream, without what followed the damage, and the image pass measured
each image by its uses before the damage only, so one drawn larger past it
could be resampled below that use.
- `DecodeBudget::decode_contents_with_damage` reads a page's `/Contents` as
`decode_contents` does, each stream also saying whether it is damaged
partway; `decode_contents` is it without the flag.
- `content_bytes` gives `None` for a page with a damaged stream, as for one
that doesn't decode: the content is not all known. Its strips aren't merged
(`merge_page` already leaves such a page alone), and no image is resampled
(`effective_resolutions` gives every image `INFINITY`).
- A form damaged partway leaves the measuring incomplete in the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: JBIG2 images share a symbol dictionary, up to 16 at a time
With Monochrome Images set to JBIG2, Optimize stored each image's
letters in a symbol dictionary of its own, so a document of N pages
carried its font's shapes N times. Now up to 16 images the run stores
as JBIG2 share one dictionary in a /JBIG2Globals stream, holding the
marks found in more than one of them; each image's text regions refer
to it first, then to a dictionary of its own for the marks only it
repeats. An image is stored that way when it comes out smaller, and a
group only when what its images save is more than the globals take.
- filters: encode_jbig2_shared codes a group: the globals are a symbol
dictionary segment numbered 0 and associated with no page (7.2.6);
each page numbers its segments from 1, as a segment refers only to
lower numbers (7.2.5), and the shared symbols' IDs come first, as the
dictionaries are referred to in that order (Table 31, SBSYMS). At
most 16,383 shared marks, those in the most images, so a page keeps
room for its own under the decoder's 32,767.
- optimize: the run holds the 1-bit samples of the images it stores as
JBIG2 (16 at most, 64 MiB at most) and stores them again sharing,
weighing each with the longest reference the globals could have.
Eight pages of 10 pt text rendered at 300 dpi take 41% less in
Helvetica and 42% in Times than coded alone, 10% in Courier, and 46%
in Helvetica at 600 dpi; groups of 2, 4, 16 and 32 save 19%, 33%, 45%
and 47%. Redacting one image stores the others of its group again
without the dictionary, so 16 keeps that to 15 images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: test what the mutants found untested in shared JBIG2 dictionaries
- filters: the group goes through encode_jbig2_shared, which refuses it
for an image the encoder refuses; two marks in common are enough to
share; past the 65,536 distinct marks a group holds, a mark first seen
later isn't shared while those held count every image.
- optimize: optimizing again keeps both the shared pages and the pages
coded alone as they are; under Retain, averaged JBIG2 pages share
nothing; a last group of two shares.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: only images redaction can store again share a JBIG2 dictionary
Redacting one image of a group stores the others again without the
dictionary, which can hold the shapes of marks under the area, and
refuses the redaction for one it can't: it takes image masks and
images of one component as its own reading counts them (no DeviceN,
the family named directly), and at most 64 Mi pixels. Optimize took
DeviceN of one colourant and up to 80 million pixels, so such an image
in a group made any redaction of the others fail, where before it was
coded alone and the same redaction worked (the Consumers review).
Only an image mask or a DeviceGray image named directly, of at most
2^26 pixels, now joins a group; others are coded alone. An engine test
optimizes three pages, two DeviceGray and one DeviceN, and redacts the
first: the DeviceN page stays alone, the second is stored again, and
no image names the dictionary after a save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: optimize and redaction share one limit on the images redaction stores again
The second Consumers review: Optimize's limit on images that share a
JBIG2 dictionary copied redaction's private 64 Mi, with nothing tying
the two, so a lower one in redaction would bring back refused
redactions with no test going red. Both now read
pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS. The engine test adds an image
mask, the other kind that shares, so a redaction that stopped storing
either kind again fails it. doc_optimize's description, the README,
the parity note and ROADMAP say which images share (named directly).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: a JBIG2 group is coded before its samples would pass the limit
CodeRabbit on #5910: hold added an image to the group and then checked
SHARED_BYTES, so a group could pass the 64 MiB it is documented to stay
within by its last image, which can itself be up to 64 MiB (an image
that shares has at most 2^26 pixels, and as its rows are whole bytes,
at most that many bytes; a 1-pixel-wide one takes them all). hold_in
codes the group first when an image would take it past the limit, and
after when it has 16 images; with room for 10 bytes, images of 4 go two
to a group, where the old order made one of 12. CodeRabbit's follow-up
too: the 18-image test decodes every image and compares it with its
samples, across the boundary between the groups.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* optimize: count the globals' object framing; filters: test the shared coder's size limit
- A JBIG2 group shares a dictionary only when it saves more than the
globals' dictionary and data plus NEW_OBJECT (96 bytes: the object's
framing, its xref entry, and an encrypted file's IV and padding), so a
group that saves less than that no longer makes the file larger.
- encode_jbig2_shared is Unsupported for an image taller than the
decoder takes and codes one at the bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filters: drop the unused rows binding in encode_shared
page() no longer takes the rows since #5511.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filters: the shared coder's at-bound test says what it returns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
* M11: JBIG2 encodes only what its decoder takes: Optimize no longer loses large images
encode_jbig2 took rows of up to 2^20 samples and any number of them, while hayro-jbig2, which
draws and reads these streams, takes 65,535 on a side and 2^28 samples. The optimizer doesn't
read back what it stores, so with Monochrome Images on JBIG2, any 1-bit image with a side of
65,536 or more was replaced by a stream that draws nothing: 65,536 × 16 went from 37,875 dark
pixels to none, and was reported as recompressed.
The encoder now refuses what the decoder can't read, so such an image stays as it is. Symbol
coding's limit of 2^24 rows can't be reached any more, so it goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the JBIG2 boundary test checks a pixel
It counted rows, which a page decoded blank would pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the encoder's bound on samples is MAX_JBIG2_PIXELS, the decoder's
main brought decode_jbig2 and its public MAX_JBIG2_PIXELS into this crate,
2^28 like this branch's private MAX_JBIG2_SAMPLES and for the same reason.
One constant now bounds both, so the encoder can't drift past what the
decoder reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: an image too large for JBIG2 isn't stored as JBIG2, as the notes say
The parity note said such an image stays as it is. A 1-bit one does unless
it is averaged, but one packed from more bits may become 1-bit Flate. The
note now says it isn't stored as JBIG2 and cites
an_image_too_large_for_the_jbig2_decoder_stays_as_it_is, and the optimizer
README lists the bound under Not yet. MAX_JBIG2_PIXELS is the most the
decoder takes, not about that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the optimizer's own size limit, and tests for what JBIG2 can't take
The optimizer leaves images over 80 million pixels alone whatever the
settings, and nothing said so; the README and parity note say it now, and
that JBIG2's 2^28 in all is past it, so from Optimize the side bound is the
one that applies. The note says what the image step does: it leaves a
1-bit image past the bound as it is unless it is averaged, and may store
one packed from more bits as 1-bit Flate.
what_is_too_large_for_jbig2_is_stored_another_way tests those two cases:
eight-bit grey of black and white at 65,536 wide becomes 1-bit Flate, and
a 1-bit image at 600 ppi averaged to 70,000 wide does too. At 65,535 and
35,000 both become JBIG2.
The symbol cap's comment says why hayro-jbig2 refuses 32,768 symbols: it
caps new and exported symbols together at 65,535, and a plan exports every
symbol it codes. Measured: 32,767 decode, 32,768, 40,000 and 65,535 are
refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the 80-million-pixel limit is the image steps', tested on both sides
The README and parity note said images over 80 million pixels are left
alone whatever the settings, but Clean Up still Flates an unfiltered or
LZW one. They now say the image steps leave them alone, and the note cites
group_4_takes_images_of_one_bit_a_pixel_up_to_the_pixel_limit for it.
(The last commit said the note also gives 2^28 as past that limit; only
the README does.)
what_is_too_large_for_jbig2_is_stored_another_way covered the width
bound only: it now runs tall images too, packed and averaged, and checks
the averaged ones are one bit and half the size both ways. The older test
uses the same bars() fixture, and its comment says such a stream drew
nothing in PdfCraft, not that the picture was lost: Poppler and
Ghostscript draw it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the JBIG2 stored at the decoder's bound reads back, and grey keeps the pixel limit
- The optimize test at 65,535 on a side decoded nothing it stored: it
checked the filter only, and the coding stored there is symbols, not
the generic coding the filters test reads back. It now decodes the
stored stream and compares it with the bars.
- The README's 80-million-pixel sentence covers the colour and grey
steps too, and only the 1-bit step had a test. A grey image of
10,000 × 8,000 at 600 ppi is resampled; a row more stays as it is.
- The filters README names decode_jbig2_within, with its work, as main
renamed decode_jbig2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
* M11: the filters README names #5569's other renames too
The README named decode_jbig2_within but still decode_ccitt, which
#5569 made private behind decode_ccitt_within, and gave
Document::decoded_image as both decoders' caller, which is test-only
now: ImageBudget::decode calls them, as lib.rs's doc says. The ccitt.rs
test line keeps decode_ccitt, which those tests call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(optimize): make pictures under a stencil mask smaller
An image with any /Mask was skipped by the colour and grey settings. That is
right for a colour-key mask, which names sample values that resampling and JPEG
would change. A stencil mask is an image of its own, with its own resolution
(ISO 32000-2 8.9.6.3), so the picture can change under it and the mask stays as
it is. Some exporters put one on every picture, even where it hides nothing, so
whole drawing sets of pictures were left at full size.
A mask that can't be read as a stencil, such as a reference to nothing, still
leaves its picture alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(optimize): only a mask that says /ImageMask true counts as a stencil
A stream /Mask without /ImageMask true let its picture be replaced, although a
mask that isn't a stencil is meant to leave its picture alone. It now does.
The stencil test also checks that the hole still shows what's behind it. The first
pixel all round is left out, because the coarser picture blends into it a little
(at most 26 of 255 in this test).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(optimize): draw a coloured page behind the stencil test's picture
On a white page, a hole painted white looks the same as an open hole, so the check
on the hole couldn't tell them apart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: quincy-jc <184324679+quincy-jc@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`ImageBudget::decode` is now the one public way to decode an image's
CCITT fax or JBIG2 data: `Document::decoded_image` and
`decoded_image_within`, which decoded with no limit on the work, are
test-only in pdfcraft-cos, and pdfcraft-filters exports only
`decode_ccitt_within` and `decode_jbig2_within`, which spend from a
caller's allowance (`decode_jbig2` is gone, `decode_ccitt` private).
So an operation over many images can't decode them without sharing an
allowance; the compiler holds new callers to it. Doctests that must not
compile check both crates; with the old entry points public they fail.
Tests outside the crates decode through an unlimited ImageBudget.
Streams in general (`Stream::decoded`) are left for a follow-up.
Every colour operator in the pages' content and the Form XObjects it draws is
rewritten in the target space by the standard device formulas, source colours
read through the space in force (edit/src/colour.rs: device, ICCBased, CalRGB/
CalGray, Lab, Indexed, Separation/DeviceN through a Type 2 tint). Preserve black,
pictures too (optimize::recolour_image: 8-bit gray/RGB re-encoded, JPEG stays
JPEG, CMYK lossless), comments' and fields' appearances optionally.
Edit::ConvertColours (one undo step; refuses when nothing changes, so
Session::count_colours counts first), the Convert colours dialog, command
prepress.convert_colors, tool doc_convert_colours, rows in every language
(RGB and CMYK allowed unchanged in the Ukrainian and Kazakh checks), parity
print.convert-colors shipped.
Carries the review fixes:
- ATTRIBUTION.toml: the language files' checksums after the new rows
- Convert colours: keep RGB and CMYK as-is in the Czech and Turkish menu-label checks
- Convert colours: CMYK black start, shared objects copied, streams kept apart, guarded count
- Convert colours tests: drop an unused helper
- Format with rustfmt
- i18n(ui): carry this PR's new wordings in the catalogues that must cover them
- Parity: evidence paths follow the tests moved into tests/it
- fix(colours): comment permission for appearances; page range per document
- fix(convert-colours): say what it is not, and what it leaves behind
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Storing LZW streams with Flate decoded image streams outside the allowance,
with a limit for each stream and no total, and nothing asked whether to go
on: a 593 KB file of 100 images took 113 s with none refused, and images
refused by the image pass were still rewritten. The pass now pays for what
it decodes and for storing and checking it, at 8 steps a byte since Flate
takes up to 78 ns a byte, and streams past that stay LZW.
An image is counted as refused only when it stayed as it was: one packed
before the allowance ran out on its second read was rewritten. doc_reduce
reports images_refused as doc_optimize does, and the app's action says so
in its log. A refused image no longer reads as an unsupported filter:
FilterError::Spent says what happened.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
A link annotation listed twice in one page's /Annots, or by two pages, kept its later references after the first one stripped its external action, leaving inert links on the page. Every reference to a link that ends up with no action and no /Dest is now removed. A shared link counts once in external_links.
From a review of the spellings an indexed image can take:
- The palette was read for every 1-bit indexed image, resampled or not, and twice each. Images
share a table, so 1,000 images sharing one with 16 MiB of raw data took 16 to 60 s, none of
them resampled. It is now read only for an image about to be resampled, and a table over
4 KiB is refused before it is copied or decoded, since no palette needs that.
- A colour key can hide the dark sample a tie goes to, so a thin rule of the other was lost: a
white rule on a hidden black went from 255 to the backdrop's 128. An image with a colour key
now keeps its resolution. Grey had this since #5426 and is covered too.
- `/Indexed` by reference, and a base of `[/DeviceRGB]`, are the same colour space and are read
so. `/I`, an inline abbreviation, and a real hival aren't, and such an image is left alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
From CodeRabbit's review: default colour spaces apply to an indexed colour space's
device base, so one in force where an annotation's appearance (or a pattern, or
another image's /Alternates) draws the image may change which palette entry is darker,
and the tie a resampled image keeps. Only black-and-white RGB went through
used_elsewhere; indexed images now do too when downsampling is on, and one used beyond
the page and form resources the walk read keeps its resolution, as such RGB keeps its
colour space.
A grey image stays out of this check, since sample 0 stays dark in a grey space, and #5426's rule
for grey is unchanged: the test now has a grey image drawn on the page and in an annotation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AmsdeUCdTaD72jkDw2pJKY
#5461 made Document::decoded_image read JBIG2, page exactly the image's size, and
kept the optimizer's monochrome step off it until it could weigh JBIG2 again. Now
one_bit reads JBIG2 last after any filters, within the raster's allowance, and mono
treats it as it does any black-and-white image:
- above the threshold it is averaged, and stored as the compression asks, the
smallest of what that may store; under Retain it is stored as JBIG2 again, its kind
of compression kept;
- not resampled, it is stored again as the compression asks when that is smaller: a
generic region becomes symbols with JBIG2 asked for, and JBIG2 behind ASCIIHex is
stored again alone (as Group 4 where that is smaller);
- a JBIG2 image stored again refers to no /JBIG2Globals; one whose page isn't its
size isn't read, and stays.
jbig2_images_are_left_as_they_are becomes jbig2_scans_are_averaged_and_stored_again:
a JBIG2 scan at 600 ppi, alone, behind ASCIIHex and with globals, averages to the
same samples as its Flate copy under each compression (Retain keeping JBIG2); at 300
ppi a generic region of rendered text stays under Group 4 (larger) and Retain and
becomes symbols under JBIG2 (under half the size); behind ASCIIHex it is stored again
alone, as Group 4 where the hex is spaced out; a taller page stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#5426 left indexed images at their resolution rather than guess which sample is
dark. The palette says: dark_is_one now maps samples 0 and 1 through the decode array
to palette entries (rounded and clipped to hival, ISO 32000-2 §8.9.5.2) and compares
how light those entries are (`lightness`: the grey, or Rec. 601 luma over DeviceRGB).
Only DeviceGray and DeviceRGB bases count, the table a string or a stream (within
4 KiB), directly or by reference: there the palette's values are the colour, and
under /DefaultGray or /DefaultRGB no polarity is asked for. Other bases (CMYK,
calibrated, ICC-based, a resource name), two entries alike, an entry the table
doesn't hold or a hival past 255 still say nothing; separations and DeviceN still
aren't averaged, their tint transforms unread. #5426's tone guard applies as to any
image.
#5426's indexed thin-rule test now checks what it was for: with the palette either
way round and the decode array either way, the averaged image holds the averaged
DeviceGray one's samples (flipped where its samples are) and renders the same. A unit
test covers each base, both orders, an inverting decode array, the weights of luma,
the table by reference and in a stream, and every refusal; the older polarity test's
doc says what is known now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decode_jbig2 sits next to decode_ccitt, through hayro-jbig2 as Export ▸ Images
used it, and shares CCITT's row sink. The JBIG2 page has to be the image's size:
PdfCraft's renderer draws the whole page into the image (black box: a 4-row page
behind /Height 2 fills the image box with all four rows), so the rows of a taller
page aren't the image's. The decoder decodes a page whole, so the page is checked
before it is decoded: that size, within the caller's bound as rows of one bit, and
at most 2^28 pixels (MAX_JBIG2_PIXELS; the decoder keeps a bit a pixel, about 34 MB
there).
Document::decoded_image_within decodes a JBIG2 image as it does a fax image: the
filters before it undone first, then the decoder with the /JBIG2Globals of the
image's own /DecodeParms entry (/Crypt counts in the position; fax_params and the
globals now share last_filter_parms). Globals have a limit of their own,
MAX_JBIG2_GLOBALS (32 MiB, moved from Export), and globals that can't be read are an
error rather than ignored.
Export ▸ Images reads JBIG2 through it, so its own decoder, row sink and DecodeParms
reader go, and hayro-jbig2 leaves pdfcraft-create. JBIG2 behind another filter now
exports, and so copies (Copy Image uses the same function); a page taller than the
image, which Export cut to the image's rows, now skips the image, as one of another
width did. doc_export_all_images' description no longer lists JBIG2 inside other
filters as skipped.
The optimizer's monochrome step read images through decoded_image_within and
relied on JBIG2 failing there: it now leaves JBIG2 images alone explicitly, as
before (a resampled one would come out Flate under Retain), and the README and
parity note list them as not yet handled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bflatastic