mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-19 03:48:34 +00:00
## What this adds
Structural round-trip, property and never-crash evidence for `crates/annot`, which had no `tests/` directory — every prior test checked an in-memory object and never crossed the writer. 50 new tests take the crate from 42 to **92 passing / 0 failed**, with nothing ignored and no existing test weakened:
- `tests/round_trip.rs` (25) — annotations survive save and reopen across all three writer paths: geometry, contents text, flag bits, opacity, identifying names, pop-up back-references, link targets/URIs/dash structure, per-page order and ownership.
- `tests/properties.rs` (10) — proptest invariants over generated geometry, opacity and flags.
- `tests/malformed.rs` (15) — damaged input must never panic; each case documents the *measured* behaviour as defined degradation, not endorsed output.
- `tests/support/{bytes.rs,fixture.rs}` — a raw-byte reader and PDF skeletons, so claims can be made about the saved file rather than only about parsed objects.
`proptest = { workspace = true }` as a dev-dependency (same declaration as `cos`; style differs from the dotted form in crypt/filters). proptest 1.11.0 is already locked in the workspace, so `Cargo.lock` gains one edge and no new package.
**No product code is changed.** This pins current behaviour, including three questionable behaviours it surfaced (below), and leaves every fix to a follow-up.
## Appearance is deliberately out of scope
Nothing here asserts what an appearance stream draws, so this stays clear of #2345, #2137, #2066, #1775 and #1513 (all open, all rewriting appearance generation). Structural fidelity — where a mark sits, what it says, which rules apply — is behaviour none of them intends to alter.
## Evidence quality: mutation-tested, with credit split honestly
Ten single-site guard/arithmetic/key-rename mutants were run against `crates/annot/src/lib.rs` by two independent reviewers plus the author. All ten were caught, and they split **five and five**:
- Killed by pre-existing inline tests in `crates/annot/src/tests.rs` as well: `lib.rs:857` inverted, `:1030` inverted (six inline tests), `:1930` inverted, `:2161` tie-break swapped, `:1241` `/Parent` renamed.
- Killed **only** by tests added here: `:1030` clamp deleted, `:857` clamp deleted, opacity clamp removed at `:1024`, colour clamp in `rgb()` removed at `:824`, and a clamp introduced on the read path at `:2238`.
`lib.rs:857` and `:1030` appear on both sides deliberately — inverting either trips inline tests, deleting them trips nothing outside this branch. Confusing those two mutation families is easy enough to produce a false refutation, so the distinction is recorded in `round_trip.rs` rather than left to whoever re-runs them. A build failure was never counted as a kill in any pass. The mutants were run at `crates/annot/src/lib.rs` as of base `99941daa`. Main has since split that file into modules (`5ce4d244`); the anchor statements are byte-identical at their new homes (the `/Parent` site apart, locals renamed), but main also grew the inline suite from 42 to 74 tests — so the five-and-five split describes the base tree, and the battery was not re-run at the merge head `3122ebad`.
## Three defects found while writing this — reported, not fixed
1. **Large finite numbers are written unvalidated and become unreadable downstream.** The only style guard on the create path (`lib.rs:1018`) is `!finite(&style.color) || !style.opacity.is_finite() || !style.width.is_finite()`; the `abs() < 1e7` cap inside `finite()` (`:827-829`) applies to **colour alone**, so `width` carries no magnitude limit at all. Measured: `width = 1e9` and `1e299` are accepted and written verbatim to `/BS /W`. qpdf (`pikepdf.Job(["--check", …])`, pikepdf 9.11 / libqpdf 12.2) exits **3** — its *warning* status, printed with "operation succeeded with warnings" (0 = clean and silent here) — reporting `overflow/underflow converting <301 digits> to 64-bit integer`, and `/Annots[0]` resolves to `null` (still null after a qpdf re-save) while `pikepdf.open()` reports success. Our own `repair_log()` stays empty, which conflicts with CLAUDE.md's "never silently drop or repair data".
2. **Fill colour is accepted on create and refused on edit.** Create never screens `fill` (it is clamped component-wise by `rgb()` at `:823-825`), while `set_fill` refuses the identical value with `Invalid("invalid fill colour")` at `:2010`. Pinned by `malformed::the_same_huge_fill_is_accepted_on_create_and_refused_on_edit`.
3. **One unparseable token in a page's `/Annots` loses the whole page, silently.** For `[4 0 R notanumber]`, `page_refs()` returns `Ok([])` with `/Pages` still showing the page — so the page object itself fails to parse, every annotation on it disappears, and `repair_log()` records nothing. qpdf keeps the page and warns: `(object 3 0, offset 191): unknown token while reading object; treating as string`.
Anchors in this section read against `crates/annot/src/lib.rs` at base `99941daa`; main has since split that file (`5ce4d244`). The same code at merge head `3122ebad`: `:1018` → `create.rs:416`, `:827-829` → `create.rs:17-19`, `:823-825` → `create.rs:13-15`, `:2010` → `modify.rs:626-627`. All three described behaviours re-verified unchanged at this head.
These need a policy decision rather than a patch, so they are described here and will be filed as separate issues; nothing in this PR fixes them.
## CI status and base note
This branch was opened at `afa8e18` (then current `main`) and rebased onto `99941da`. Two jobs were red on the original base — `test (ubuntu-latest)`, `test (windows-latest)` and the FreeBSD job, all failing in `crates/automation/tests/automation.rs` at `:354` (`signing_with_translated_labels_through_tools`, which passed a 7-character password against an 8-character minimum) and at `:359` (`a_save_that_does_not_reopen_leaves_the_file_as_it_was`), `173 passed; 2 failed`. Those were upstream failures reproduced on that commit's own CI run (**38703572296**, plus the FreeBSD run **38703573428**) in a crate this change never touches; they have since been fixed on `main` by #2420 (the signing password now meets the minimum) and #2484 (save/search regression expectations updated), which is why `main`'s latest CI run is green and why this branch was rebased. The rebase is a no-op for our code: `crates/annot/src/lib.rs` at both commits is byte-identical (`fb491e19…`), so every line number cited below still resolves as written.
Since then `main` was merged in (`dc943eeb`, merge commit `3122ebad`) to clear the FreeBSD job per the CI triage on this PR: the `files_and_quit_from_the_operating_system` race was fixed on main by `12fcc641` (#2812). The merge probe was clean (zero conflicts; `Cargo.lock` the only overlapping path). Main split `crates/annot/src/lib.rs` into modules and reshaped `set_style` to take a `StyleChange`; the merge commit carries three tests-only adaptations (two `SaveOptions` literals gain `drop_unlinked: false`, the writer's default; one `set_style` call re-shaped; citations re-derived to the new file layout). No crate source diverges from main's: `git diff --stat upstream/main HEAD -- crates/annot/src/` is empty. Inline-suite counts at this head: 74 (main grew it from 42); the 42 → 92 figures above describe base `99941daa`. proptest on main's lock is now 1.12.0; one edge added, no new package.
## Where the proof stops
- The first CI run on this branch was at our original base and showed the inherited failures described above; results cited here are local macOS runs (we have no Poppler locally, so qpdf cross-checks ran through pikepdf in a workspace virtualenv).
- The probe PDFs used for qpdf measurements are **not** committed, because AGENTS.md §1 requires an `ATTRIBUTION.toml` entry per sample document — so CI cannot re-measure them. Each measurement carries a regenerate recipe and sha256 in the test comments (`malformed.rs`).
- All oracle figures are tied to one machine's build: pikepdf 9.11 over libqpdf 12.2 (`__libqpdf_version__`; the Job API rejects `--version`, so no raw qpdf version string). Poppler comparisons would need CI, as `pdftoppm` is unavailable locally.
- Mutation evidence covers single-site guard/arithmetic/key-rename mutants only — no multi-site, data-flow or off-by-one classes were attempted.
- We did not measure what any viewer displays to a user after these failures; the listed-but-null entry was observed, not rendered behaviour.
## Question for maintainers
CLAUDE.md asks new tests to be registered in `ROADMAP.md` and `parity/acrobat-features.toml`, but several open PRs above are editing both files right now, so we left them untouched to avoid collisions. Tell us where you would like these registered and we will add the lines here, or open a follow-up.
Credit: `contributors/people.toml` deliberately carries no new entry in this PR — ours is already queued on #2073.
337 lines
13 KiB
Rust
337 lines
13 KiB
Rust
//! Shared helpers compile into every integration-test binary of this crate, and not every
|
|
//! binary uses every one of them; `dead_code` would then fail `clippy -D warnings` for reasons
|
|
//! that have nothing to do with the test being written.
|
|
#![allow(dead_code)]
|
|
|
|
//! A deliberately strict reader over the *bytes* a save produced, independent of `pdfcraft-cos`.
|
|
//!
|
|
//! Reopening saved bytes with `Document::open` and comparing dictionaries proves that our writer
|
|
//! and our reader agree. It does not prove what is actually *in the file*: a lenient parser can
|
|
//! forgive output that another implementation refuses (an exponent in what should be a fixed
|
|
//! decimal number, an unterminated string, a key written twice). These helpers look at the bytes
|
|
//! themselves and reject anything outside the narrow grammar ISO 32000-2 §7.3 allows for the
|
|
//! object types annotations use, so an assertion here is a claim about the file rather than about
|
|
//! a round trip through our own pair of implementations.
|
|
//!
|
|
//! Scope: dictionary keys whose values are number arrays (`/Rect`, `/QuadPoints`, `/L`,
|
|
//! `/Vertices`, `/CL`, `/RD`), number arrays of arrays (`/InkList`), text strings (`/Contents`,
|
|
//! `/T`, `/NM`, `/M`) and names (`/Subtype`). That covers the entries these tests care about; it
|
|
//! is not a general PDF parser and does not try to be.
|
|
|
|
/// Every value token written for `/key` in the raw bytes, in file order.
|
|
///
|
|
/// A name-keyed dictionary entry is `/key` preceded and followed by delimiters, so searching for
|
|
/// `Rect` does not match inside `/MyRectangle`. The returned spans include the enclosing brackets
|
|
/// or string delimiters, which the decoders below strip.
|
|
pub fn value_tokens(bytes: &[u8], key: &[u8]) -> Vec<Vec<u8>> {
|
|
let mut out = Vec::new();
|
|
let mut i = 0;
|
|
while i + key.len() + 1 < bytes.len() {
|
|
if bytes[i] != b'/' || &bytes[i + 1..i + 1 + key.len()] != key {
|
|
i += 1;
|
|
continue;
|
|
}
|
|
let after = i + 1 + key.len();
|
|
// The key must end here: a longer name is a different key.
|
|
if after < bytes.len() && is_name_byte(bytes[after]) {
|
|
i += 1;
|
|
continue;
|
|
}
|
|
let Some((end, token)) = capture(bytes, after) else {
|
|
i += 1;
|
|
continue;
|
|
};
|
|
// A `/Key /Other` hit is ambiguous: it may be a key whose value is a name, or the *name*
|
|
// that is some earlier key's value (`/S /URI`). Re-scan from the next byte so a later real
|
|
// occurrence of `/Key` is still found; an array, string or number value is unambiguous.
|
|
let name_valued = token.starts_with(b"/");
|
|
out.push(token);
|
|
i = if name_valued { i + 1 } else { end };
|
|
}
|
|
out
|
|
}
|
|
|
|
fn is_name_byte(b: u8) -> bool {
|
|
b.is_ascii_alphanumeric() || matches!(b, b'.' | b'-' | b'_' | b'#' | b'%')
|
|
}
|
|
|
|
/// Capture the object that starts at `at`, with bracket/paren balance respected. A literal
|
|
/// string's brackets do not count; an array containing a string does nest.
|
|
fn capture(bytes: &[u8], mut at: usize) -> Option<(usize, Vec<u8>)> {
|
|
while bytes.get(at).is_some_and(|b| b.is_ascii_whitespace()) {
|
|
at += 1;
|
|
}
|
|
let first = *bytes.get(at)?;
|
|
match first {
|
|
b'[' => balanced(bytes, at, b'[', b']'),
|
|
b'(' => literal_string(bytes, at),
|
|
b'<' if bytes.get(at + 1) == Some(&b'<') => balanced(bytes, at, b'<', b'>'),
|
|
// A hex string: `<...>`, closed by the first `>` that is not part of a `>>`.
|
|
b'<' => {
|
|
let close = bytes.get(at + 1..)?.iter().position(|&b| b == b'>')?;
|
|
Some((at + close + 2, bytes[at..at + close + 2].to_vec()))
|
|
}
|
|
b'/' => {
|
|
let mut j = at + 1;
|
|
while bytes.get(j).is_some_and(|b| !is_delimiter(*b)) {
|
|
j += 1;
|
|
}
|
|
Some((j, bytes[at..j].to_vec()))
|
|
}
|
|
_ => {
|
|
let mut j = at;
|
|
while bytes.get(j).is_some_and(|b| !is_delimiter(*b)) {
|
|
j += 1;
|
|
}
|
|
Some((j, bytes[at..j].to_vec()))
|
|
}
|
|
}
|
|
}
|
|
|
|
fn is_delimiter(b: u8) -> bool {
|
|
b.is_ascii_whitespace() || matches!(b, b'(' | b')' | b'<' | b'>' | b'[' | b']' | b'{' | b'}' | b'/' | b'%')
|
|
}
|
|
|
|
/// `open` at `at`, closed by the matching `close`, skipping over literal strings so that a
|
|
/// delimiter inside `(…)` does not end the span.
|
|
fn balanced(bytes: &[u8], at: usize, open: u8, close: u8) -> Option<(usize, Vec<u8>)> {
|
|
let mut depth = 0usize;
|
|
let mut i = at;
|
|
while i < bytes.len() {
|
|
match bytes[i] {
|
|
b'(' => {
|
|
let (end, _) = literal_string(bytes, i)?;
|
|
i = end;
|
|
continue;
|
|
}
|
|
b if b == open => depth += 1,
|
|
b if b == close => {
|
|
depth -= 1;
|
|
if depth == 0 {
|
|
return Some((i + 1, bytes[at..=i].to_vec()));
|
|
}
|
|
}
|
|
_ => {}
|
|
}
|
|
i += 1;
|
|
}
|
|
None
|
|
}
|
|
|
|
fn literal_string(bytes: &[u8], at: usize) -> Option<(usize, Vec<u8>)> {
|
|
let mut i = at + 1;
|
|
while i < bytes.len() {
|
|
match bytes[i] {
|
|
b'\\' => i += 2,
|
|
b')' => return Some((i + 1, bytes[at..=i].to_vec())),
|
|
_ => i += 1,
|
|
}
|
|
}
|
|
None
|
|
}
|
|
|
|
/// The numbers of an array token, written in fixed decimal notation (§7.3.3: no exponent form).
|
|
/// `None` when the token is not an array, holds a non-number, or uses `e`.
|
|
pub fn numbers(token: &[u8]) -> Option<Vec<f64>> {
|
|
let inner = token.strip_prefix(b"[")?.strip_suffix(b"]")?.to_vec();
|
|
let mut out = Vec::new();
|
|
for word in split_words(&inner) {
|
|
if word.first() == Some(&b'[') {
|
|
// A nested array (`/InkList`): its numbers count as this one's, flattened.
|
|
out.extend(numbers(&word)?);
|
|
} else {
|
|
out.push(fixed_decimal(&word)?);
|
|
}
|
|
}
|
|
Some(out)
|
|
}
|
|
|
|
/// A single number written for `/key` (`/CA`, `/F`, a stream's `/Length`), strictly decimal.
|
|
pub fn scalars_of(bytes: &[u8], key: &str) -> Vec<f64> {
|
|
value_tokens(bytes, key.as_bytes()).iter().filter_map(|t| fixed_decimal(t)).collect()
|
|
}
|
|
|
|
/// All numbers of an ARRAY token written for `/key`, with the strict fixed-decimal rule.
|
|
///
|
|
/// ONLY valid for keys whose value is an array (`/Rect`, `/QuadPoints`, `/L`, `/IC`,
|
|
/// `/PCPictureSize`). For a SCALAR key (`/W`, `/CA`, `/F`) every token fails `numbers` and the
|
|
/// result is an empty vector — so comparing two such calls against each other is vacuous: it
|
|
/// passes whether or not the file holds anything. Use [`scalars_of`] for scalars, and always
|
|
/// assert non-emptiness when a value is expected. A scalar misuse was found and fixed in
|
|
/// `round_trip.rs` (a `/BS /W` comparison that silently compared `[] == []`).
|
|
pub fn numbers_of(bytes: &[u8], key: &str) -> Vec<Vec<f64>> {
|
|
value_tokens(bytes, key.as_bytes()).iter().filter_map(|t| numbers(t)).collect()
|
|
}
|
|
|
|
/// A number written as `[+-]?digits[.]?` or `[+-]?digits[.]digits`, with no exponent and no
|
|
/// stray characters. `str::parse::<f64>` alone would accept `1e7`, which PDF does not use.
|
|
fn fixed_decimal(word: &[u8]) -> Option<f64> {
|
|
if word.is_empty() || word.iter().any(|b| *b == b'e' || *b == b'E') {
|
|
return None;
|
|
}
|
|
let body = match word[0] {
|
|
b'+' | b'-' => &word[1..],
|
|
_ => word,
|
|
};
|
|
if body.is_empty() {
|
|
return None;
|
|
}
|
|
let mut digits = 0;
|
|
let mut dots = 0;
|
|
for &b in body {
|
|
match b {
|
|
b'0'..=b'9' => digits += 1,
|
|
b'.' => dots += 1,
|
|
_ => return None,
|
|
}
|
|
}
|
|
if digits == 0 || dots > 1 {
|
|
return None;
|
|
}
|
|
std::str::from_utf8(word).ok()?.parse::<f64>().ok()
|
|
}
|
|
|
|
/// The decoded bytes of a string token (literal with escapes, or hex), `None` if malformed.
|
|
pub fn raw_string(token: &[u8]) -> Option<Vec<u8>> {
|
|
match (token.first(), token.last()) {
|
|
(Some(b'('), Some(b')')) => {
|
|
let mut out = Vec::new();
|
|
let body = &token[1..token.len() - 1];
|
|
let mut i = 0;
|
|
while i < body.len() {
|
|
if body[i] != b'\\' {
|
|
out.push(body[i]);
|
|
i += 1;
|
|
continue;
|
|
}
|
|
let e = *body.get(i + 1)?;
|
|
match e {
|
|
b'n' => out.push(b'\n'),
|
|
b'r' => out.push(b'\r'),
|
|
b't' => out.push(b'\t'),
|
|
b'b' => out.push(0x08),
|
|
b'f' => out.push(0x0C),
|
|
b'(' | b')' | b'\\' => out.push(e),
|
|
b'0'..=b'7' => {
|
|
let mut v = 0u32;
|
|
let mut j = i + 1;
|
|
let mut n = 0;
|
|
while j < body.len() && n < 3 && (b'0'..=b'7').contains(&body[j]) {
|
|
v = v * 8 + u32::from(body[j] - b'0');
|
|
j += 1;
|
|
n += 1;
|
|
}
|
|
out.push(u8::try_from(v).ok()?);
|
|
i = j;
|
|
continue;
|
|
}
|
|
b'\n' => {}
|
|
_ => return None,
|
|
}
|
|
i += 2;
|
|
}
|
|
Some(out)
|
|
}
|
|
(Some(b'<'), Some(b'>')) => {
|
|
let hex: Vec<u8> = token[1..token.len() - 1].iter().copied().filter(|b| !b.is_ascii_whitespace()).collect();
|
|
if !hex.len().is_multiple_of(2) {
|
|
return None;
|
|
}
|
|
let mut out = Vec::with_capacity(hex.len() / 2);
|
|
for pair in hex.as_chunks::<2>().0 {
|
|
let s = std::str::from_utf8(pair).ok()?;
|
|
out.push(u8::from_str_radix(s, 16).ok()?);
|
|
}
|
|
Some(out)
|
|
}
|
|
_ => None,
|
|
}
|
|
}
|
|
|
|
/// Text as the file holds it: UTF-16BE behind a BOM, PDF 2.0's UTF-8 BOM, or PDFDocEncoding —
|
|
/// approximated by Latin-1 above 0x7F. The two differ only at a handful of bytes between 0x18
|
|
/// and 0x9F, none of which these tests assert on; ASCII and the UTF-16 cases are exact.
|
|
pub fn text_of_raw(bytes: &[u8]) -> String {
|
|
match bytes {
|
|
[0xFE, 0xFF, rest @ ..] => String::from_utf16_lossy(&rest.as_chunks::<2>().0.iter().map(|p| u16::from_be_bytes(*p)).collect::<Vec<_>>()),
|
|
[0xEF, 0xBB, 0xBF, rest @ ..] => String::from_utf8_lossy(rest).into_owned(),
|
|
b => b.iter().map(|&c| char::from(c)).collect(),
|
|
}
|
|
}
|
|
|
|
/// Every text value written for `/key`, decoded from the bytes.
|
|
pub fn texts_of(bytes: &[u8], key: &str) -> Vec<String> {
|
|
value_tokens(bytes, key.as_bytes()).iter().filter_map(|t| raw_string(t)).map(|b| text_of_raw(&b)).collect()
|
|
}
|
|
|
|
/// A name token's bytes (`/Foo` with `#hh` escapes decoded), or the bare name of a token.
|
|
pub fn name_of_token(token: &[u8]) -> Option<Vec<u8>> {
|
|
if *token.first()? != b'/' {
|
|
return None;
|
|
}
|
|
let body = &token[1..];
|
|
let mut out = Vec::new();
|
|
let mut i = 0;
|
|
while i < body.len() {
|
|
if body[i] == b'#' {
|
|
let s = std::str::from_utf8(body.get(i + 1..i + 3)?).ok()?;
|
|
out.push(u8::from_str_radix(s, 16).ok()?);
|
|
i += 3;
|
|
} else {
|
|
out.push(body[i]);
|
|
i += 1;
|
|
}
|
|
}
|
|
Some(out)
|
|
}
|
|
|
|
/// Every name written for `/key`.
|
|
pub fn names_of(bytes: &[u8], key: &str) -> Vec<String> {
|
|
value_tokens(bytes, key.as_bytes()).iter().filter_map(|t| name_of_token(t)).map(|n| String::from_utf8_lossy(&n).into_owned()).collect()
|
|
}
|
|
|
|
/// Whether the file holds the name `/name` as its own token, so searching `Rect` cannot match
|
|
/// inside `/Rectangle`. It does not know about string objects, so a name-looking sequence inside
|
|
/// literal text would also match: use it only for names distinctive enough to be unambiguous.
|
|
pub fn has_name(bytes: &[u8], name: &str) -> bool {
|
|
let n = name.as_bytes();
|
|
if n.is_empty() {
|
|
return false;
|
|
}
|
|
let mut i = 0;
|
|
while i + n.len() < bytes.len() {
|
|
if bytes[i] == b'/' && &bytes[i + 1..i + 1 + n.len()] == n {
|
|
let after = i + 1 + n.len();
|
|
if after >= bytes.len() || !is_name_byte(bytes[after]) {
|
|
return true;
|
|
}
|
|
}
|
|
i += 1;
|
|
}
|
|
false
|
|
}
|
|
|
|
/// Tokens split on whitespace, with `[` … `]` groups kept whole.
|
|
fn split_words(bytes: &[u8]) -> Vec<Vec<u8>> {
|
|
let mut out = Vec::new();
|
|
let mut i = 0;
|
|
while i < bytes.len() {
|
|
if bytes[i].is_ascii_whitespace() {
|
|
i += 1;
|
|
continue;
|
|
}
|
|
if bytes[i] == b'[' {
|
|
let (end, tok) = balanced(bytes, i, b'[', b']').expect("an opened array closes in a token we captured");
|
|
out.push(tok);
|
|
i = end;
|
|
continue;
|
|
}
|
|
let start = i;
|
|
while bytes.get(i).is_some_and(|b| !b.is_ascii_whitespace() && *b != b'[') {
|
|
i += 1;
|
|
}
|
|
out.push(bytes[start..i].to_vec());
|
|
}
|
|
out
|
|
}
|