craft-pdf: the PDF object layer from PdfCraft (cos), on filters and crypt

craft-pdf: the PDF object layer from PdfCraft (`cos`), on filters and crypt

* hayro-jbig2: bound a stream's total decode work and what a page retains; no i32 overflow

Port of pdfcraft 85c10051 (patches 5-9), 82439525 (patch 10) and 10107d94 (patch 12).

* hayro-jbig2: PdfCraft's JBIG2 follow-ups (refinement fixes, file header, work accounting, dictionary pixel cap)

Ports pdfcraft 72e763ad, b7ec3669, e3bc4e85, 01d3479f, b9e068aa, 858ce960, 07f44038, e38b9d18, 8717a085, 58c30af3 and the hayro-jbig2 part of e2464e14 (with the hayro-jbig2 part of 0ed33990 they build on).

* Add craft-stream-codecs: the codec modules of PdfCraft's stream filters

LZW, ASCII85, ASCIIHex, RunLength, the TIFF and PNG predictors, bounded CCITT
fax decoding (through hayro-ccitt) with Group 4 encoding (through fax), and
the lossless JBIG2 encoder with its MQ coder, from pdfcraft-filters at
PdfCraft revision 80e26e674d1aa33078e83c8139875047f3289eb6.

The crate sits below craft-pdf and craft-tiff: it knows no container. The
PDF filter chain, Flate and the bounded JBIG2 decoding entry are not part of
it.

* craft-pdf: the PDF filter chain from PdfCraft (`filters`)

Adds the `filters` feature: the /Filter chain of pdfcraft-filters (PdfCraft
revision 80e26e674d1aa33078e83c8139875047f3289eb6) as craft_pdf::filters,
over the codecs in craft-stream-codecs. It adds Flate, strict and tolerant
chain decoding, byte accounting and a shared work allowance, and encoding,
with the chain's unit tests, its property tests and its fuzz target.

The code is unchanged apart from the imports of the moved codecs; FilterError
is craft-stream-codecs' CodecError under its old name.

* craft-stream-codecs: bounded JBIG2 decoding (decode_jbig2_within)

The JBIG2 decoding entry of pdfcraft-filters (PdfCraft revision
80e26e674d1aa33078e83c8139875047f3289eb6, src/jbig2/decode.rs) with its tests.
It spends from the caller's work allowance through hayro-jbig2's
Image::decode_within and charge_reading, which come with
fix/sec-hayro-jbig2-followups (#886).

* vendor/README.craft-reader.md: one #831 paragraph, the follow-ups after it

* craft-pdf: filters re-exports decode_jbig2_within (and lib.rs module order)

* craft-pdf: the object layer from PdfCraft (`cos`)

Adds the `cos` feature: PdfCraft's pdfcraft-cos crate (revision
1d2af9907ba772b368f6b54811359805fa5428e9) as craft_pdf::cos, on crypt and
filters, with its unit tests, integration tests and examples. Its test helpers
come with it: pdfcraft-testkit as src/cos/testkit.rs (cfg(test)) and the qpdf
runner of pdfcraft-oracle as tests/cos/oracle.rs.

The code is unchanged apart from module paths and doc comments. The child test
of file_identifier is named by its module path, since the integration tests
are one binary.

* craft-pdf: cos unit tests build for wasm

The property tests in src/cos/writer.rs need proptest, a dev-dependency of
the non-wasm targets only, and one test in src/cos/document/tests.rs times a
load on a thread, which wasm32-unknown-unknown does not have. Both are
compiled for non-wasm targets only; the other unit tests still build for wasm.
This commit is contained in:
Dylan M. Taylor
2026-11-11 22:44:59 -05:00
committed by GitHub
parent 7dcc9950a1
commit fa389c7883
35 changed files with 15699 additions and 4 deletions
Generated
+4
View File
@@ -1629,20 +1629,24 @@ version = "0.1.0"
dependencies = [
"aes",
"bytemuck",
"chrono",
"craft-stream-codecs",
"flate2",
"hayro",
"hayro-cmap",
"hayro-interpret",
"hayro-syntax",
"js-sys",
"krilla",
"kurbo 0.13.1",
"md-5 0.11.0",
"proptest",
"serde_json",
"sha2 0.11.1",
"stringprep",
"thiserror 2.0.21",
"ttf-parser",
"wasm-bindgen",
"wasm-bindgen-test",
"zeroize",
]
+18
View File
@@ -19,6 +19,7 @@ write = ["dep:krilla", "dep:thiserror"]
write-pdf = ["write", "krilla/pdf"]
write-simple-text = ["write", "krilla/simple-text"]
filters = ["dep:craft-stream-codecs", "dep:flate2", "dep:thiserror"]
cos = ["crypt", "filters", "dep:chrono", "dep:js-sys", "dep:wasm-bindgen"]
crypt = ["dep:aes", "dep:md-5", "dep:sha2", "dep:stringprep", "dep:zeroize", "dep:thiserror"]
[dependencies]
@@ -37,8 +38,17 @@ md-5 = { version = "0.11", optional = true }
sha2 = { version = "0.11", optional = true }
stringprep = { version = "0.1", optional = true }
zeroize = { version = "1", optional = true }
# PDF date strings in the object layer (`cos`).
chrono = { workspace = true, optional = true }
# On the web, a rewrite's fresh file identifier (`cos`) takes its entropy from the browser
# (`crypto.getRandomValues`); native builds compile neither.
[target.'cfg(target_arch = "wasm32")'.dependencies]
js-sys = { version = "0.3", optional = true }
wasm-bindgen = { version = "0.2", optional = true }
[dev-dependencies]
serde_json = { workspace = true }
hayro-syntax = "=0.7.2"
hayro-interpret = { version = "=0.7.0", default-features = false }
hayro-cmap = "=0.1.0"
@@ -50,3 +60,11 @@ proptest = { workspace = true }
[target.'cfg(target_arch = "wasm32")'.dev-dependencies]
wasm-bindgen-test = "0.3"
[[example]]
name = "cos_info"
required-features = ["cos"]
[[example]]
name = "cos_protect"
required-features = ["cos"]
+45
View File
@@ -14,6 +14,7 @@ No feature is enabled by default.
| `write-simple-text` | `write` plus Krilla's optional text shaper and `draw_text` |
| `filters` | The `/Filter` chain for editing and writing PDF files: decode and encode, with limits and accounting. No reader or writer dependency |
| `crypt` | The standard security handler: decrypt, encrypt and create encryption. No reader or writer dependency |
| `cos` | `crypt` and `filters` plus the object layer: open, edit and save PDF files (incremental and full saves). No reader or writer dependency |
## Reading and rendering
@@ -111,6 +112,50 @@ read-only consumer, then repeat with `--target wasm32-unknown-unknown` and the t
These backend checks supplement the root workspace gates; excluded vendor crates are not
covered by `cargo test --workspace` alone.
## Object layer
`cos` is an object layer for apps that edit PDF files (ISO 32000-2 §7). It parses the object
graph lazily and tolerantly, keeps edits in a copy-on-write overlay, and writes the file back
incrementally (the original bytes untouched) or as a full, garbage-collected rewrite. It uses
`crypt` and `filters`; the reader (`read`) and writer (`write`) are separate and do not use it.
```rust
use craft_pdf::cos::{Document, ObjRef, Object, SaveOptions, write_full, write_incremental};
let doc = Document::open(bytes)?; // Arc<Vec<u8>>
let doc = Document::open_with_password(bytes, Some("owner"))?; // encrypted files
let obj = doc.get(ObjRef::new(12, 0)); // loaded on first use, cached
let data = stream.decoded()?; // filters applied on demand
doc.set(r, Object::Dict(d)); // edits land in the overlay
let r = doc.add(obj);
let bytes = write_incremental(&doc, &SaveOptions::default())?; // append a revision
let bytes = write_full(&doc, &SaveOptions::default())?; // rewrite what the trailer reaches
```
- Reads every cross-reference form (tables, streams, hybrid `/XRefStm`) and object streams. A
file whose cross-reference data is wrong or missing is rebuilt by scanning for object headers,
and each repair is recorded (`repair_log`). Malformed input gives `CosError`, never a panic.
- Encryption on open and save through `crypt`: revisions 2 to 6, crypt filters, `/EFF` and
`/EncryptMetadata`. `set_encryption` and `remove_encryption` change it.
- `write_incremental` appends the edited objects with a cross-reference section in the style of
the file's last one, so earlier revisions and their signatures stay as they were; it falls back
to `write_full` when an edit cannot be appended. `write_full` renumbers what the trailer
reaches and, by default, packs objects into object streams. Both return the bytes.
- `Document` is cheap to clone: the bytes, index and caches are shared, so a clone is a
snapshot for undo or a background job.
- Images: `ImageBudget::decode` gives an image's samples after its filters, CCITT fax and JBIG2
included, all images of one operation spending from one allowance. DCT and JPX are not decoded
here.
- Not here: streaming files larger than memory, linearized output.
The code comes from PdfCraft's `pdfcraft-cos` crate (revision
`1d2af9907ba772b368f6b54811359805fa5428e9`), unchanged apart from module paths and doc
comments. Its test helpers come along: `src/cos/testkit.rs` (synthetic files, from
`pdfcraft-testkit`) and `tests/cos/oracle.rs` (qpdf runs, from `pdfcraft-oracle`; a missing
`qpdf` skips those tests, or fails them when `CRAFT_PDF_ORACLES_REQUIRED` is set). Tests:
`cargo test -p craft-pdf --no-default-features --features cos`. Examples: `cos_info` and
`cos_protect` (`--features cos`).
## Stream filters
`filters` is the `/Filter` chain of a PDF stream (ISO 32000-2 §7.4) for code that edits or
+38
View File
@@ -0,0 +1,38 @@
//! Print how a file was opened: revisions, repair notes, trailer, security and the catalog.
//! `cargo run -p craft-pdf --features cos --example cos_info f.pdf [password] [object-number…]`
fn show(o: &craft_pdf::cos::Object) -> String {
let mut t = Vec::new();
craft_pdf::cos::serialize(o, &mut t);
let s = String::from_utf8_lossy(&t).into_owned();
s.chars().take(400).collect()
}
fn main() {
let mut args = std::env::args().skip(1);
let path = args.next().expect("usage: cosinfo <file.pdf> [password] [obj…]");
let password = args.next().filter(|p| !p.is_empty());
let objs: Vec<u32> = args.filter_map(|a| a.parse().ok()).collect();
let bytes = std::sync::Arc::new(std::fs::read(&path).expect("readable"));
match craft_pdf::cos::Document::open_with_password(bytes, password.as_deref()) {
Ok(doc) => {
println!("version {} revisions {:?}", doc.version(), doc.revisions());
for r in doc.repair_log() {
println!("repair: {r}");
}
println!("trailer {}", show(&craft_pdf::cos::Object::Dict(doc.trailer().clone())));
if let Some(s) = doc.security() {
println!("security: auth {:?}, key {} bytes, {:?}", s.auth(), s.file_key().len(), s.permissions());
}
if let Some(r) = doc.root() {
println!("catalog {}", show(&doc.get(r)));
}
for n in objs {
match doc.try_get(n) {
Ok(o) => println!("{n} 0 obj {}", show(&o)),
Err(e) => println!("{n} 0 obj error: {e}"),
}
}
}
Err(e) => println!("error: {e}"),
}
}
+35
View File
@@ -0,0 +1,35 @@
//! Encrypt a PDF: `cargo run -p craft-pdf --features cos --example cos_protect in.pdf out.pdf rc4-40|rc4-128|aes-128|aes-256 USER OWNER`
/// 32 fresh bytes for the file key. `RandomState` is seeded by the operating system (as the
/// engine's `entropy` uses it), so every run encrypts with a different key.
fn seed() -> [u8; 32] {
use std::hash::{BuildHasher, Hasher};
let mut out = [0u8; 32];
for (i, chunk) in out.chunks_mut(8).enumerate() {
let mut h = std::collections::hash_map::RandomState::new().build_hasher();
h.write_usize(i);
chunk.copy_from_slice(&h.finish().to_le_bytes());
}
out
}
fn main() {
let a: Vec<String> = std::env::args().skip(1).collect();
let [input, output, alg, user, owner] = &a[..] else { panic!("usage: protect in out alg user owner") };
let algorithm = match alg.as_str() {
"rc4-40" => craft_pdf::cos::Algorithm::Rc4_40,
"rc4-128" => craft_pdf::cos::Algorithm::Rc4_128,
"aes-128" => craft_pdf::cos::Algorithm::Aes128,
_ => craft_pdf::cos::Algorithm::Aes256,
};
let mut doc = craft_pdf::cos::Document::open(std::sync::Arc::new(std::fs::read(input).unwrap())).unwrap();
doc.set_encryption(&craft_pdf::cos::NewEncryption {
algorithm,
user_password: user,
owner_password: owner,
permissions: -1,
encrypt_metadata: true,
seed: seed(),
})
.unwrap();
std::fs::write(output, craft_pdf::cos::write_full(&doc, &Default::default()).unwrap()).unwrap();
}
+58
View File
@@ -0,0 +1,58 @@
//! The PDF object layer (COS), for code that edits and saves PDF files. It uses [`crate::crypt`]
//! and [`crate::filters`]; the reader (`read`) is separate and does not use it.
//!
//! Parses the COS object graph lazily and tolerantly, keeps edits in a copy-on-write overlay
//! (cheap snapshots for undo), and writes documents back either incrementally (original bytes
//! untouched) or as a full, garbage-collected rewrite. An app can use the object graph as its
//! document model.
//!
//! Done: every cross-reference form (tables, streams, hybrid `/XRefStm`), object streams, repair
//! by scanning, encryption on load and save (R2–R6, crypt filters, `/EFF`), incremental writing,
//! and full saves that pack objects into compressed object streams with a cross-reference stream.
//! Not yet: streaming very large files within a memory budget, and linearized output.
mod budget;
mod bytes;
mod document;
mod object;
pub mod page_labels;
mod parser;
mod security;
mod writer;
#[cfg(test)]
pub(crate) mod testkit;
pub use crate::crypt::{
Algorithm, Auth, Method as CryptMethod, NewEncryption, Permissions, SecurityHandler, effective_password, pdfdoc_byte, pdfdoc_char,
};
pub use budget::{DecodeBudget, ImageBudget, MAX_PAGE_DECODED, MAX_PAGE_FORMS, MAX_REDACTED_IMAGE_PIXELS};
pub use bytes::Bytes;
pub use document::{DECODE_PARMS_KEYS, Document, ReadState, Revision, SourceIdentity, XrefEntry};
pub use object::{Dict, MAX_DECODED, MAX_JBIG2_GLOBALS, Name, ObjRef, Object, PdfString, Stream, TextChars};
pub use parser::{Lexer, is_whitespace, parse_indirect};
pub use writer::{SaveOptions, civil_from_days, new_file_id, pdf_date, reachable_objects, serialize, write_full, write_incremental};
#[derive(Debug, thiserror::Error, Clone, PartialEq)]
pub enum CosError {
#[error("not a PDF file (no %PDF- header)")]
NotPdf,
#[error("syntax error at byte {offset}: {detail}")]
Syntax { offset: usize, detail: String },
#[error("object {0} is missing")]
MissingObject(u32),
#[error("object {0} is not a dictionary")]
NotADictionary(ObjRef),
#[error("stream data could not be decoded: {0}")]
Filter(String),
#[error("the document is protected by a password")]
NeedsPassword,
#[error("the password is incorrect")]
WrongPassword,
#[error("unsupported security: {0}")]
Security(String),
#[error("internal lock poisoned")]
Poisoned,
#[error("this document was opened read-only with a decode limit and cannot be saved")]
ReadOnlyLimit,
}
File diff suppressed because it is too large Load Diff
+98
View File
@@ -0,0 +1,98 @@
//! Stream data that can share a larger buffer instead of copying out of it.
use std::ops::{Deref, Range};
use std::sync::Arc;
/// A stream's bytes: a buffer of their own, or a range of a larger shared buffer. Streams read
/// from a file point into the file's bytes, which the document keeps anyway, so caching a
/// parsed stream costs its dictionary and not a second copy of its data.
#[derive(Clone, Default)]
pub struct Bytes {
buf: Arc<Vec<u8>>,
range: Range<usize>,
}
impl Bytes {
/// `range` of `buf`, without copying; `None` when the range is not within `buf`.
pub(crate) fn view(buf: &Arc<Vec<u8>>, range: Range<usize>) -> Option<Self> {
(range.start <= range.end && range.end <= buf.len()).then(|| Self { buf: buf.clone(), range })
}
/// Whether these bytes keep alive a larger buffer than themselves that isn't `owner`.
pub(crate) fn borrows_other_than(&self, owner: &Arc<Vec<u8>>) -> bool {
self.range.len() < self.buf.len() && !Arc::ptr_eq(&self.buf, owner)
}
/// Whether these bytes are (a view into) `buf`, holding no memory of their own.
pub(crate) fn shares(&self, buf: &Arc<Vec<u8>>) -> bool {
Arc::ptr_eq(&self.buf, buf)
}
/// The same bytes, in a buffer of their own.
pub(crate) fn detached(&self) -> Self {
Self::from(self.to_vec())
}
}
impl From<Vec<u8>> for Bytes {
fn from(v: Vec<u8>) -> Self {
let range = 0..v.len();
Self { buf: Arc::new(v), range }
}
}
impl From<Arc<Vec<u8>>> for Bytes {
fn from(buf: Arc<Vec<u8>>) -> Self {
let range = 0..buf.len();
Self { buf, range }
}
}
impl Deref for Bytes {
type Target = [u8];
fn deref(&self) -> &[u8] {
// `view` and `From` only build in-bounds ranges; an empty slice keeps this panic-free.
self.buf.get(self.range.clone()).unwrap_or_default()
}
}
impl AsRef<[u8]> for Bytes {
fn as_ref(&self) -> &[u8] {
self
}
}
impl PartialEq for Bytes {
fn eq(&self, other: &Self) -> bool {
(Arc::ptr_eq(&self.buf, &other.buf) && self.range == other.range) || **self == **other
}
}
impl Eq for Bytes {}
impl std::fmt::Debug for Bytes {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
(**self).fmt(f)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn views_share_their_buffer_and_compare_by_content() {
let file = Arc::new(b"0123456789".to_vec());
let view = Bytes::view(&file, 2..5).expect("in range");
assert_eq!(&*view, b"234");
assert_eq!(view.as_ptr(), file[2..].as_ptr(), "no copy");
assert_eq!(view, Bytes::from(b"234".to_vec()));
assert!(Bytes::view(&file, 5..11).is_none() && Bytes::view(&file, Range { start: 6, end: 5 }).is_none());
let other = Arc::new(Vec::new());
assert!(view.borrows_other_than(&other) && !view.borrows_other_than(&file));
let own = view.detached();
assert!(!own.borrows_other_than(&other) && own == view && own.as_ptr() != view.as_ptr());
assert!(!Bytes::from(file.clone()).borrows_other_than(&other), "a whole buffer is its own");
}
}
+98
View File
@@ -0,0 +1,98 @@
//! Document: cross-reference data, lazy object loading, and a copy-on-write overlay of edits.
//!
//! `Document` is cheap to clone (the original bytes, xref index and parse caches are shared), so
//! a clone is a snapshot: undo/redo and background jobs hold clones while the UI edits another.
//! Edits never touch the original bytes; the writer appends them (incremental save) or rewrites
//! the file (full save).
use std::borrow::Cow;
use std::collections::{BTreeMap, HashMap, HashSet};
use std::sync::{Arc, Mutex, Weak};
use crate::cos::CosError;
use crate::cos::object::{Dict, MAX_DECODED, MAX_JBIG2_GLOBALS, ObjRef, Object, Stream};
use crate::cos::parser::{EndstreamIndex, Lexer, is_whitespace, parse_indirect, parse_indirect_shared};
mod objects;
mod open;
mod reader;
mod state;
mod xref;
pub use objects::DECODE_PARMS_KEYS;
pub(crate) use objects::Held;
#[cfg(test)]
pub(crate) use reader::find;
#[cfg(test)]
use reader::rfind;
use state::{ObjStm, Slot, Unreadable};
pub(crate) use state::{ObjectReader, held_bytes};
pub use state::{ReadState, SourceIdentity};
#[cfg(test)]
use state::{SECTION_READS, objstm_member};
use xref::XrefIndex;
pub use xref::{Revision, XrefEntry};
#[derive(Clone)]
pub struct Document {
data: Arc<Vec<u8>>,
/// Where `endstream` occurs in `data`; see [`EndstreamIndex`].
endstreams: Arc<EndstreamIndex>,
source_identity: Arc<()>,
entries: Arc<XrefIndex>,
trailer: Dict,
revisions: Arc<Vec<Revision>>,
/// See [`Self::xref_oddities`].
xref_oddities: Arc<Vec<(u64, u32)>>,
/// Where the file's object headers are, as (offset, generation) of the last header of each
/// number. Filled only when the cross-reference data proved inconsistent; see
/// `fill_xref_gaps`.
located: Arc<BTreeMap<u32, (u64, u16)>>,
repair_log: Arc<Vec<String>>,
/// See [`Self::rebuilt_unlinked_update`].
unlinked_update: bool,
/// What loading each file object gave, a failure as well: a damaged object is otherwise
/// loaded again on every read, and a member of an object stream that can't be decoded
/// decodes the stream again (`Document::load`).
cache: Arc<Mutex<state::ObjectCache>>,
objstms: Arc<Mutex<HashMap<u32, Arc<ObjStm>>>>,
/// See [`Self::unreadable_objects`]. Shared by clones, like the caches: it describes the
/// source, and is reset wherever they are.
unreadable: Arc<Mutex<Unreadable>>,
/// What `/JBIG2Globals` streams of the file decode to, by object number: the images of a scan
/// share them, and each would otherwise decode them again (see `Document::jbig2_globals`).
jbig2_globals: Arc<Mutex<HashMap<u32, Arc<Vec<u8>>>>>,
stream_limit: usize,
overlay: BTreeMap<u32, Slot>,
next_num: u32,
/// Position of the `%PDF-` header (some files have junk before it).
header_offset: usize,
version: String,
/// The authenticated security handler of an encrypted document.
security: Option<Arc<crate::crypt::SecurityHandler>>,
/// Object number of the `/Encrypt` dictionary (never encrypted itself).
encrypt_num: Option<u32>,
/// What that dictionary and `/ID` refer to, which the key comes from, and so is as stored.
encrypt_members: Arc<HashSet<u32>>,
/// Object numbers a scan of the file didn't find: the file doesn't change, so it never will.
unfound: Arc<Mutex<HashSet<u32>>>,
/// Encryption was added, changed or removed since opening: only a full save can apply it.
encryption_changed: bool,
/// Set by edits that must not leave earlier revisions in the file (redaction).
full_save: bool,
/// The next full save also gives the file a new identifier (`/ID`), see
/// [`Document::require_full_save_with_new_id`].
new_id: bool,
/// The handler and `/Encrypt` object number that saves use, when encryption changed.
out_security: Option<Arc<crate::crypt::SecurityHandler>>,
out_encrypt_num: Option<u32>,
}
impl std::fmt::Debug for Document {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Document").field("bytes", &self.data.len()).field("xref", &self.entries).field("edits", &self.overlay.len()).finish()
}
}
#[cfg(test)]
mod tests;
File diff suppressed because it is too large Load Diff
+503
View File
@@ -0,0 +1,503 @@
use super::reader::find;
use super::state::{Credentials, Slot};
use super::xref::XrefIndex;
use super::{Arc, BTreeMap, CosError, Dict, Document, HashSet, ObjRef, Object};
/// More objects than `/Encrypt` and `/ID` refer to in any file, so that one which refers to the page
/// tree, say, doesn't have a whole document read to open it.
const MAX_ENCRYPT_MEMBERS: usize = 64;
impl Document {
/// A new document with an empty page tree (for split / extract / combine). It has no
/// original bytes, so saving always writes a full file.
pub fn new_empty() -> Self {
let mut doc = Document {
data: Arc::new(Vec::new()),
endstreams: Arc::default(),
source_identity: Arc::new(()),
entries: Arc::new(XrefIndex::empty()),
trailer: Dict::new(),
revisions: Arc::new(Vec::new()),
xref_oddities: Arc::default(),
located: Arc::default(),
repair_log: Arc::new(Vec::new()),
unlinked_update: false,
cache: Arc::default(),
objstms: Arc::default(),
unreadable: Arc::default(),
jbig2_globals: Arc::default(),
stream_limit: crate::cos::object::MAX_DECODED,
overlay: BTreeMap::new(),
next_num: 1,
header_offset: 0,
version: "1.7".into(),
security: None,
encrypt_num: None,
encrypt_members: Arc::default(),
unfound: Arc::default(),
encryption_changed: false,
full_save: false,
new_id: false,
out_security: None,
out_encrypt_num: None,
};
let mut pages = Dict::new();
pages.set(b"Type".to_vec(), Object::name("Pages"));
pages.set(b"Kids".to_vec(), Object::Array(Vec::new()));
pages.set(b"Count".to_vec(), Object::Int(0));
let pages = doc.add(pages);
let mut catalog = Dict::new();
catalog.set(b"Type".to_vec(), Object::name("Catalog"));
catalog.set(b"Pages".to_vec(), Object::Ref(pages));
let root = doc.add(catalog);
doc.trailer.set(b"Root".to_vec(), Object::Ref(root));
doc
}
/// Parse a document. Damaged cross-reference data is reconstructed; see `repair_log`.
/// Encrypted documents open only if their user password is empty; see `open_with_password`.
pub fn open(data: Arc<Vec<u8>>) -> Result<Self, CosError> {
Self::open_with_password(data, None)
}
/// Parse a document, authenticating encrypted ones with `password` (owner or user; `None`
/// tries the empty password). Fails with `NeedsPassword` / `WrongPassword` as appropriate.
pub fn open_with_password(data: Arc<Vec<u8>>, password: Option<&str>) -> Result<Self, CosError> {
Self::open_with_stream_limit(data, password, crate::cos::object::MAX_DECODED)
}
/// Open with a tighter decompression bound for object and cross-reference streams.
/// Inspection can use this without changing the editing reader's default limit. Such a
/// document is read-only: objects in a larger object stream read as missing, so the
/// writers refuse to save it ([`CosError::ReadOnlyLimit`]).
pub fn open_with_stream_limit(data: Arc<Vec<u8>>, password: Option<&str>, stream_limit: usize) -> Result<Self, CosError> {
Self::open_as(data, Credentials::Password(password), stream_limit)
}
/// Open `data`, an earlier revision of this file (the bytes a signature covers) or another
/// save of it, with this document's credentials: an encrypted one must have the same
/// `/Encrypt` dictionary and `/ID` as this document, whose file key is then reused, so no
/// password is needed or kept; unencrypted data opens as [`open`](Self::open) does. Data
/// encrypted differently fails with `NeedsPassword`.
pub fn open_with_same_credentials(&self, data: Arc<Vec<u8>>) -> Result<Self, CosError> {
let credentials = match &self.security {
Some(handler) => Credentials::Handler { handler, id0: self.first_id() },
None => Credentials::Password(None),
};
Self::open_as(data, credentials, self.stream_limit)
}
fn open_as(data: Arc<Vec<u8>>, credentials: Credentials<'_>, stream_limit: usize) -> Result<Self, CosError> {
// Viewers accept files whose header is missing or damaged as long as the body looks
// like PDF; so do we (a note goes to the repair log).
let header = find(&data, b"%PDF-", 0, 1024);
let headerless = header.is_none();
if headerless && find(&data, b" obj", 0, 4096).is_none() {
return Err(CosError::NotPdf);
}
let header_offset = header.unwrap_or(0);
let version = match header {
Some(h) => String::from_utf8_lossy(&data[(h + 5).min(data.len())..(h + 8).min(data.len())]).trim().to_string(),
None => "1.4".to_string(),
};
let mut doc = Document {
data,
endstreams: Arc::default(),
source_identity: Arc::new(()),
entries: Arc::new(XrefIndex::empty()),
trailer: Dict::new(),
revisions: Arc::new(Vec::new()),
xref_oddities: Arc::default(),
located: Arc::default(),
repair_log: Arc::new(Vec::new()),
unlinked_update: false,
cache: Arc::default(),
objstms: Arc::default(),
unreadable: Arc::default(),
jbig2_globals: Arc::default(),
stream_limit: stream_limit.min(crate::cos::object::MAX_DECODED),
overlay: BTreeMap::new(),
next_num: 1,
header_offset,
version,
security: None,
encrypt_num: None,
encrypt_members: Arc::default(),
unfound: Arc::default(),
encryption_changed: false,
full_save: false,
new_id: false,
out_security: None,
out_encrypt_num: None,
};
let mut log = Vec::new();
if headerless {
log.push("the %PDF- header is missing; reading the file as PDF 1.4".into());
}
let parsed = doc.read_xref_chain(&mut log);
let mut authenticated = false;
// The newest section cut the page tree off from the chain (see `update_cuts_off_the_pages`).
let mut cut_off = false;
let ok = match parsed {
Ok((entries, trailer, revisions, oddities)) => {
doc.entries = Arc::new(entries);
doc.trailer = trailer;
doc.revisions = Arc::new(revisions);
doc.xref_oddities = Arc::new(oddities);
doc.fill_xref_gaps(&mut log);
// Authenticate before checking the catalog: it may sit in an encrypted object
// stream, which reads as garbage until the key is known.
if let Some(enc) = doc.trailer.get(b"Encrypt").cloned() {
doc.authenticate(&enc, &credentials)?;
authenticated = true;
}
let catalog = doc.root_is_catalog();
cut_off = catalog && doc.update_cuts_off_the_pages();
catalog && !cut_off
}
Err(e) => {
log.push(format!("cross-reference data unreadable ({e}); rebuilding from object headers"));
false
}
};
if cut_off {
log.push(UNLINKED_UPDATE.into());
doc.unlinked_update = true;
} else if !ok && !doc.entries.is_empty() {
log.push("cross-reference data does not lead to a valid catalog; rebuilding from object headers".into());
}
if !ok {
doc.reconstruct(&mut log)?;
}
// Header generations are plaintext, including in encrypted files. Inspect only the
// bounded headers, not object bodies, and retain the compact xref representation.
doc.note_xref_generation_repairs(&mut log);
doc.repair_log = Arc::new(log);
// Reconstruction may have found a different trailer: (re)authenticate against it.
if (!ok || !authenticated)
&& let Some(enc) = doc.trailer.get(b"Encrypt").cloned()
{
doc.authenticate(&enc, &credentials)?;
}
if Arc::make_mut(&mut doc.entries).remove_exhausted() {
Arc::make_mut(&mut doc.repair_log).push("skipped exhausted xref object number 4294967295".into());
}
let max = doc.entries.max_num();
let size = u32::try_from(doc.trailer.int(b"Size").unwrap_or(0).max(0)).unwrap_or(u32::MAX);
if size == u32::MAX {
Arc::make_mut(&mut doc.repair_log).push("exhausted trailer /Size; new objects use unused object numbers".into());
}
doc.next_num = max.saturating_add(1).max(size).max(1);
Ok(doc)
}
/// Whether the chain of cross-reference sections ends at the newest one (no `/Prev`, or one
/// that isn't an offset) although the file holds earlier sections, and the catalog's `/Pages`
/// is not among the objects that one section reaches. ISO 32000-2 §7.5.6 and §7.5.8.2 require
/// the `/Prev`; without it the earlier revisions, page tree and all, are cut off, so the file
/// is rebuilt from its object headers as a damaged one is (the newest definition of each
/// object wins), and the repair log says why. (A file whose `/Root` itself is missing is
/// rebuilt anyway.)
fn update_cuts_off_the_pages(&self) -> bool {
if self.revisions.len() != 1 {
return false;
}
let Some(root) = self.root() else { return false };
let Ok(catalog) = self.try_get_reference(root) else { return false };
let Object::Dict(catalog) = &*catalog else { return false };
// The page tree first: the scan for an earlier `startxref` reads the whole file.
!catalog.get(b"Pages").is_some_and(|pages| matches!(&*self.resolve(pages), Object::Dict(_))) && earlier_sections(&self.data)
}
fn authenticate(&mut self, enc: &Object, credentials: &Credentials<'_>) -> Result<(), CosError> {
// Authentication can change how the same input is decoded.
self.source_identity = Arc::new(());
let (num, dict) = match enc {
Object::Ref(r) => (Some(r.num), self.get(*r).as_dict().cloned()),
Object::Dict(d) => (None, Some(d.clone())),
_ => (None, None),
};
let Some(dict) = dict else {
// A dangling /Encrypt: viewers treat the file as unencrypted, and so do we.
let mut log = self.repair_log.as_ref().clone();
log.push("the trailer names an /Encrypt dictionary that does not exist; reading the file as unencrypted".into());
self.repair_log = Arc::new(log);
self.trailer.remove(b"Encrypt");
return Ok(());
};
let params = crate::cos::security::encrypt_dict(self, &dict);
let id0 = self.first_id();
// Read here, while nothing is decrypted yet.
let members = self.encryption_members(enc);
let handler = match credentials {
Credentials::Password(password) => match crate::crypt::SecurityHandler::open(params.clone(), &id0, *password) {
Ok(handler) => Arc::new(handler),
// No password, or the empty one: what only the embedded files need isn't asked.
Err(crate::crypt::CryptError::WrongPassword) => match self.without_password(&dict, params) {
Some(handler) if password.is_none_or(str::is_empty) => Arc::new(handler),
_ if password.is_none() => return Err(CosError::NeedsPassword),
_ => return Err(CosError::WrongPassword),
},
Err(other) => return Err(CosError::Security(other.to_string())),
},
// The same encryption as the document whose key this is: that key opens it. A handler
// without the key opens only what it would open by itself.
Credentials::Handler { handler, id0: known } if *handler.dict() == params && *known == id0 => {
if !handler.has_key() && !crate::cos::security::efopen_only(self, &dict) {
return Err(CosError::NeedsPassword);
}
Arc::clone(handler)
}
Credentials::Handler { .. } => return Err(CosError::NeedsPassword),
};
self.security = Some(handler);
self.encrypt_num = num;
self.encrypt_members = Arc::new(members);
// Objects read while locating the catalog were read before decryption was possible.
self.cache = Arc::default();
self.objstms = Arc::default();
self.unreadable = Arc::default();
Ok(())
}
/// The handler for a document opened without a password that only its embedded files need
/// (see [`crate::cos::security::efopen_only`]). Not a repair: ISO 32000-2 §7.6.6 has such a
/// document open so; [`Document::embedded_files_locked`] tells it.
fn without_password(&self, dict: &Dict, params: crate::crypt::EncryptDict) -> Option<crate::crypt::SecurityHandler> {
crate::cos::security::efopen_only(self, dict).then(|| crate::crypt::SecurityHandler::without_key(params)).flatten()
}
/// A document opened without the password its embedded files need takes either password
/// later: with the file key its embedded files decrypt, so what was read as stored is read
/// again. Refused while an edited stream holds data read as stored, which the key would
/// encrypt a second time.
fn unlock_embedded_files(&mut self, password: &str) -> Result<(), CosError> {
let Some(current) = self.security.as_deref() else { return Ok(()) };
if self.overlay_entries().any(|(_, _, o)| matches!(o.as_deref(), Some(Object::Stream(s)) if self.reads_as_stored(s))) {
return Err(CosError::Security("an embedded file was edited before the password was given: save or undo that edit first".into()));
}
let handler = crate::crypt::SecurityHandler::open(current.dict().clone(), &self.first_id(), Some(password)).map_err(|e| match e {
crate::crypt::CryptError::WrongPassword => CosError::WrongPassword,
other => CosError::Security(other.to_string()),
})?;
self.security = Some(Arc::new(handler));
self.source_identity = Arc::new(());
self.cache = Arc::default();
self.objstms = Arc::default();
Ok(())
}
/// Whether the document opened without the password its embedded files need (`/AuthEvent
/// /EFOpen`): their data is held, and saved, as stored, and isn't to be read as the files.
pub fn embedded_files_locked(&self) -> bool {
self.security.as_deref().is_some_and(|h| !h.has_key())
}
/// Whether `s`, a stream of this document, holds encrypted data that reads as stored because the
/// document opened without the password its embedded files need (see
/// [`Document::embedded_files_locked`]). Such data isn't the file it encrypts.
pub fn reads_as_stored(&self, s: &crate::cos::Stream) -> bool {
self.security.as_deref().is_some_and(|h| crate::cos::security::stream_reads_as_stored(self, h, s))
}
/// The first file identifier, part of the encryption key (empty when there is none).
fn first_id(&self) -> Vec<u8> {
let id = self.trailer.get(b"ID").map(|id| self.resolve(id));
let first = id.as_deref().and_then(Object::as_array).and_then(|a| a.first()).map(|s| self.resolve(s));
first.as_deref().and_then(Object::as_string).map(|s| s.bytes.clone()).unwrap_or_default()
}
/// The objects `/Encrypt`, given as `enc`, and `/ID` refer to, the ones they refer to, and so
/// on, up to [`MAX_ENCRYPT_MEMBERS`]. Their strings aren't encrypted (ISO 32000-2 §7.6.2): the
/// key is made from them, so they have to read as stored, and stay so. A number with no object
/// behind it, and a stream, which those dictionaries never hold, isn't one: an object added
/// under that number later, or a page's content a hostile `/Encrypt` names, is encrypted.
fn encryption_members(&self, enc: &Object) -> HashSet<u32> {
let mut pending = Vec::new();
crate::cos::writer::collect_refs(enc, &mut |r| pending.push(r));
if let Some(id) = self.trailer.get(b"ID") {
crate::cos::writer::collect_refs(id, &mut |r| pending.push(r));
}
let mut members = HashSet::new();
while let Some(r) = pending.pop() {
if members.len() >= MAX_ENCRYPT_MEMBERS || members.contains(&r.num) {
continue;
}
let held = self.get(r);
if matches!(&*held, Object::Null | Object::Stream(_)) {
continue;
}
members.insert(r.num);
crate::cos::writer::collect_refs(&held, &mut |x| pending.push(x));
}
members
}
/// Whether object `num` is written as it is held, being one of [`Document::encryption_members`],
/// unedited, and the save writing the same `/Encrypt`. What was edited or added is encrypted
/// whatever its number.
pub(crate) fn stored_as_is(&self, num: u32) -> bool {
!self.encryption_changed && !self.is_edited(num) && self.encrypt_members.contains(&num)
}
/// Lift the restrictions of a document opened with its user password by entering the
/// owner (permissions) password (§7.6.4). Fails with `WrongPassword` for any other
/// password. Does nothing when the document is unencrypted or already opened as owner.
/// Both passwords unlock the same file key, so nothing read so far is decoded again.
pub fn unlock(&mut self, password: &str) -> Result<(), CosError> {
let Some(current) = self.security.as_deref() else { return Ok(()) };
if !current.has_key() {
return self.unlock_embedded_files(password);
}
if current.auth() == crate::crypt::Auth::Owner {
return Ok(());
}
let owner = current.authenticate_owner(&self.first_id(), password).map_err(|e| match e {
crate::crypt::CryptError::WrongPassword => CosError::WrongPassword,
other => CosError::Security(other.to_string()),
})?;
self.security = Some(Arc::new(owner));
Ok(())
}
/// The security handler, when the document is encrypted.
pub fn security(&self) -> Option<&crate::crypt::SecurityHandler> {
self.security.as_deref()
}
/// What the opening password allows (`None` for unencrypted documents: everything).
pub fn permissions(&self) -> Option<crate::crypt::Permissions> {
self.security().map(|s| s.permissions())
}
/// Make the next save a full rewrite: earlier revisions (which still hold what an edit
/// removed) must not stay in the file. Redaction requires this.
pub fn require_full_save(&mut self) {
self.full_save = true;
}
/// Like [`Document::require_full_save`], and the rewritten file also gets a new file identifier
/// (`/ID`), so that a redacted or sanitized copy never claims to be its source (ISO 32000-2
/// §14.4). An encrypted file keeps the first string of its identifier, which its key depends on.
pub fn require_full_save_with_new_id(&mut self) {
self.full_save = true;
self.new_id = true;
}
/// The next save must rewrite the whole file (see [`Document::require_full_save`]).
/// Repair-aware edits retaining two accepted generations cannot use one incremental slot.
pub fn full_save_required(&self) -> bool {
self.full_save || self.overlay.values().any(|slot| matches!(slot, Slot::RepairedSet(..)))
}
/// The next full save gives the file a new identifier (see [`Document::require_full_save_with_new_id`]).
pub fn new_id_required(&self) -> bool {
self.new_id
}
/// Security was set or removed since the document was opened (the next save applies it).
pub fn encryption_changed(&self) -> bool {
self.encryption_changed
}
/// The handler used to write: the new one after `set_encryption` / `remove_encryption`,
/// otherwise the one the document was opened with.
pub(crate) fn output_security(&self) -> (Option<&crate::crypt::SecurityHandler>, Option<u32>) {
if self.encryption_changed { (self.out_security.as_deref(), self.out_encrypt_num) } else { (self.security.as_deref(), self.encrypt_num) }
}
/// The security the next save writes: protection applied with `set_encryption`, none after
/// `remove_encryption`, otherwise the security the document was opened with.
pub fn output_handler(&self) -> Option<&crate::crypt::SecurityHandler> {
self.output_security().0
}
/// Protect the document with a password (§7.6.4). Takes effect on the next save, which is
/// always a full rewrite. Returns the handler (authenticated as owner).
pub fn set_encryption(&mut self, params: &crate::crypt::NewEncryption) -> Result<(), CosError> {
// The file identifier is part of the key; make sure it exists and keep it.
let id = self.trailer.get(b"ID").map(|id| self.resolve(id));
let id0 = match id.as_deref() {
Some(Object::Array(a)) if a.len() == 2 => self.first_id(),
_ => {
let id = crate::crypt::file_id(&params.seed);
let s = Object::String(crate::cos::PdfString { bytes: id.clone(), hex: true });
self.trailer.set(b"ID".to_vec(), Object::Array(vec![s.clone(), s]));
id
}
};
let h = crate::crypt::create(params, &id0).map_err(|e| CosError::Security(e.to_string()))?;
let d = h.dict();
let s = |b: &[u8]| Object::String(crate::cos::PdfString { bytes: b.to_vec(), hex: true });
let mut e = Dict::new();
e.set(b"Filter".to_vec(), Object::name("Standard"));
e.set(b"V".to_vec(), Object::Int(d.v));
e.set(b"R".to_vec(), Object::Int(d.r));
e.set(b"Length".to_vec(), Object::Int(d.length_bits));
e.set(b"O".to_vec(), s(&d.o));
e.set(b"U".to_vec(), s(&d.u));
e.set(b"P".to_vec(), Object::Int(i64::from(d.p)));
if d.v >= 4 {
let mut cf = Dict::new();
let mut std_cf = Dict::new();
std_cf.set(b"Type".to_vec(), Object::name("CryptFilter"));
let (cfm, len) = if d.v >= 5 { ("AESV3", 32) } else { ("AESV2", 16) };
std_cf.set(b"CFM".to_vec(), Object::name(cfm));
std_cf.set(b"Length".to_vec(), Object::Int(len));
std_cf.set(b"AuthEvent".to_vec(), Object::name("DocOpen"));
cf.set(b"StdCF".to_vec(), Object::Dict(std_cf));
e.set(b"CF".to_vec(), Object::Dict(cf));
e.set(b"StmF".to_vec(), Object::name("StdCF"));
e.set(b"StrF".to_vec(), Object::name("StdCF"));
if !d.encrypt_metadata {
e.set(b"EncryptMetadata".to_vec(), Object::Bool(false));
}
}
if d.v >= 5 {
e.set(b"OE".to_vec(), s(&d.oe));
e.set(b"UE".to_vec(), s(&d.ue));
e.set(b"Perms".to_vec(), s(&d.perms));
}
if let Some(Object::Ref(old)) = self.trailer.get(b"Encrypt").cloned() {
self.free(old);
}
let r = self.add(e);
self.trailer.set(b"Encrypt".to_vec(), Object::Ref(r));
// Objects are still read with the original handler; saves use the new one.
self.out_security = Some(Arc::new(h));
self.out_encrypt_num = Some(r.num);
self.encryption_changed = true;
Ok(())
}
/// Remove password protection (requires the document to be open — any authenticated
/// password; callers should require the owner password, §7.6.4). Applied by a full save.
pub fn remove_encryption(&mut self) {
if let Some(Object::Ref(r)) = self.trailer.get(b"Encrypt").cloned() {
self.free(r);
}
self.trailer.remove(b"Encrypt");
self.out_security = None;
self.out_encrypt_num = None;
self.encryption_changed = true;
}
/// Decrypt a freshly parsed indirect object if the document is encrypted.
pub(super) fn decrypted(&self, id: ObjRef, o: Object) -> Object {
match &self.security {
Some(h) if Some(id.num) != self.encrypt_num && !self.encrypt_members.contains(&id.num) => {
crate::cos::security::transform(self, h, &o, id.num, id.generation, crate::cos::security::Pass::Decrypt)
}
_ => o,
}
}
}
/// The repair-log line for a file rebuilt because its last update has no `/Prev`.
const UNLINKED_UPDATE: &str = "the file's last update has no /Prev link to its earlier revisions, so their objects (the page tree among them) are unreachable through the cross-reference chain; rebuilding from object headers";
/// Whether `data` has more than one `startxref`: an update was appended to it, or it is
/// linearized (whose first-page section links to the main one).
fn earlier_sections(data: &[u8]) -> bool {
find(data, b"startxref", 0, data.len()).and_then(|at| find(data, b"startxref", at.saturating_add(9), data.len())).is_some()
}
+957
View File
@@ -0,0 +1,957 @@
#[cfg(test)]
use super::state::SECTION_READS;
use super::state::Unreadable;
use super::xref::{
MAX_COMPACT_XREF_BYTES, MAX_COMPACT_XREF_ROWS, Revision, XrefEntry, XrefIndex, XrefRange, XrefRevisionSection, XrefSection, XrefSectionData,
};
use super::{Arc, BTreeMap, CosError, Dict, Document, HashMap, HashSet, Lexer, ObjRef, Object, is_whitespace, parse_indirect, parse_indirect_shared};
impl Document {
#[allow(clippy::type_complexity)]
pub(super) fn read_xref_chain(&self, log: &mut Vec<String>) -> Result<(XrefIndex, Dict, Vec<Revision>, Vec<(u64, u32)>), CosError> {
let mut compact_log = Vec::new();
if let Ok(Some((index, trailer, revisions))) = self.read_compact_xref_chain(&mut compact_log) {
let size = trailer.int(b"Size").unwrap_or(0).max(0);
if size > 0 && size <= i64::from(u32::MAX) && u64::from(index.max_num()) < size as u64 {
log.extend(compact_log);
// The compact reader declines any section with set-aside entries, so there are none here.
return Ok((index, trailer, revisions, Vec::new()));
}
}
let (entries, trailer, revisions, oddities, generation_repairs) = self.read_xref_chain_materialized(log)?;
Ok((XrefIndex::Materialized(Arc::new(entries), Arc::new(generation_repairs)), trailer, revisions, oddities))
}
fn read_compact_xref_chain(&self, log: &mut Vec<String>) -> Result<Option<(XrefIndex, Dict, Vec<Revision>)>, CosError> {
let data = &self.data;
let tail_start = data.len().saturating_sub(4096);
let sx = rfind(data, b"startxref", tail_start).ok_or(CosError::Syntax { offset: data.len(), detail: "no startxref".into() })?;
let mut lx = Lexer::new(data, sx + 9);
let first = match lx.object() {
Ok(Object::Int(n)) if n >= 0 => n as u64,
_ => return Err(CosError::Syntax { offset: sx, detail: "bad startxref value".into() }),
};
let mut sections = Vec::new();
let mut retained_rows = 0usize;
let mut retained_bytes = 0usize;
let mut trailer: Option<Dict> = None;
let mut revisions = Vec::new();
let mut visited = HashSet::new();
let mut next = Some(first);
while let Some(off) = next.take() {
if !visited.insert(off) || visited.len() > 10_000 {
log.push(format!("cross-reference chain loops at offset {off}; stopped"));
break;
}
let ((primary, sect_trailer, is_stream), header_relative) = match self.read_compact_section(off as usize) {
Ok(Some(v)) => (v, false),
Ok(None) => return Ok(None),
Err(e) if self.header_offset > 0 => {
log.push(format!("xref at {off} unreadable ({e}); retrying relative to the header"));
match self.read_compact_section(off as usize + self.header_offset)? {
Some(v) => (v, true),
None => return Ok(None),
}
}
// An unreadable older section loses every object only it locates: the
// materialized reader logs it and the file is rebuilt from object headers (#1656).
Err(e) => return Err(e),
};
revisions.push(Revision { xref_offset: off, is_stream, header_relative });
if primary.renumbered {
log.push(format!("xref subsection at {off} is numbered from 1 but begins with object 0's entry; read from 0"));
}
let Some(rows) = retained_rows.checked_add(primary.row_count).filter(|rows| *rows <= MAX_COMPACT_XREF_ROWS) else { return Ok(None) };
let Some(bytes) = retained_bytes.checked_add(primary.retained_bytes()).filter(|bytes| *bytes <= MAX_COMPACT_XREF_BYTES) else {
return Ok(None);
};
retained_rows = rows;
retained_bytes = bytes;
let hybrid = if let Some(x) = sect_trailer.int(b"XRefStm").filter(|x| *x >= 0) {
match self.read_compact_section(x as usize) {
Ok(Some((section, _, _))) => {
let Some(rows) = retained_rows.checked_add(section.row_count).filter(|rows| *rows <= MAX_COMPACT_XREF_ROWS) else {
return Ok(None);
};
let Some(bytes) = retained_bytes.checked_add(section.retained_bytes()).filter(|bytes| *bytes <= MAX_COMPACT_XREF_BYTES)
else {
return Ok(None);
};
retained_rows = rows;
retained_bytes = bytes;
Some(section)
}
Ok(None) => return Ok(None),
Err(e) => {
log.push(format!("hybrid /XRefStm at {x} unreadable: {e}"));
None
}
}
} else {
None
};
if let Some(hybrid) = &hybrid {
// A hybrid entry that disagrees with the table is set aside by the materialized reader.
let mut numbers = Vec::new();
hybrid.extend_numbers(&mut numbers);
for num in numbers {
let Some(entry) = hybrid.lookup(num, &self.data) else { continue };
if primary.lookup(num, &self.data).is_some_and(|old| !matches!(old, XrefEntry::Free { .. }) && old != entry) {
return Ok(None);
}
}
}
sections.push(XrefRevisionSection { primary, hybrid });
next = prev_offset(data, &sect_trailer, off, |num| sections.iter().find_map(|s| s.lookup(num, data)), log);
if trailer.is_none() {
trailer = Some(sect_trailer);
}
}
revisions.reverse();
let mut trailer = trailer.ok_or(CosError::Syntax { offset: 0, detail: "no trailer".into() })?;
for k in [&b"Prev"[..], b"XRefStm", b"Type", b"W", b"Index", b"Filter", b"DecodeParms", b"Length"] {
trailer.remove(k);
}
let mut max_num = 0;
let mut has_entries = false;
for revision in &sections {
max_num = max_num.max(revision.primary.max_num);
has_entries |= revision.primary.has_entries;
if let Some(hybrid) = &revision.hybrid {
max_num = max_num.max(hybrid.max_num);
has_entries |= hybrid.has_entries;
}
}
Ok(Some((XrefIndex::Compact { revisions: sections, max_num, has_entries }, trailer, revisions)))
}
fn read_compact_section(&self, off: usize) -> Result<Option<(XrefSection, Dict, bool)>, CosError> {
let data = &self.data;
let mut lx = Lexer::new(data, off);
if lx.eat_keyword(b"xref") {
let mut ranges = Vec::new();
let mut layout_ok = true;
let mut max_num = 0;
let mut has_entries = false;
let mut previous_end = None;
let mut row_count = 0usize;
let mut renumbered = false;
loop {
lx.skip_ws();
if lx.eat_keyword(b"trailer") {
let t = lx.object()?;
let d = t.as_dict().cloned().ok_or(CosError::Syntax { offset: lx.pos, detail: "trailer is not a dictionary".into() })?;
if !layout_ok {
return Ok(None);
}
return Ok(Some((XrefSection { data: XrefSectionData::Table, ranges, row_count, max_num, has_entries, renumbered }, d, false)));
}
let (Ok(Object::Int(start)), Ok(Object::Int(count))) = (lx.object(), lx.object()) else {
return Err(CosError::Syntax { offset: lx.pos, detail: "bad xref subsection header".into() });
};
if !valid_xref_range(start, count) || count > 10_000_000 {
return Err(CosError::Syntax { offset: lx.pos, detail: "invalid xref object range".into() });
}
let shifted = subsection_start(data, self.header_offset, lx.pos, start, count);
renumbered |= shifted != start;
let start = shifted;
let Some(next_rows) = usize::try_from(count).ok().and_then(|count| row_count.checked_add(count)) else { return Ok(None) };
if next_rows > MAX_COMPACT_XREF_ROWS {
return Ok(None);
}
row_count = next_rows;
let mut range_offset = None;
let mut stride = None;
let mut previous_record = None;
for i in 0..count {
lx.skip_ws();
let record = lx.pos;
if range_offset.is_none() {
range_offset = Some(record);
}
if let Some(previous) = previous_record {
let Some(delta) = record.checked_sub(previous) else {
layout_ok = false;
continue;
};
if stride.is_some_and(|known| known != delta) {
layout_ok = false;
}
stride.get_or_insert(delta);
}
previous_record = Some(record);
let o = lx.token();
lx.skip_ws();
let _generation = lx.token();
lx.skip_ws();
let kind = lx.token();
let num = (start + i) as u32;
let offset = std::str::from_utf8(o).ok().and_then(|s| s.parse::<u64>().ok());
match (kind, offset) {
(b"n", Some(offset)) if offset != 0 => {
has_entries = true;
max_num = max_num.max(num);
}
(b"n", Some(_)) => {
// Set aside by the materialized reader, which records it.
if num != 0 {
return Ok(None);
}
}
(b"f", _) => {
has_entries = true;
max_num = max_num.max(num);
}
_ => return Err(CosError::Syntax { offset: lx.pos, detail: "bad xref entry".into() }),
}
}
if count > 0 {
let range_start = start as u32;
let range_count = count as u32;
let end = u64::from(range_start) + u64::from(range_count);
if previous_end.is_some_and(|previous| u64::from(range_start) < previous) {
layout_ok = false;
}
previous_end = Some(end);
ranges.push(XrefRange {
start: range_start,
count: range_count,
offset: range_offset.unwrap_or(lx.pos),
stride: stride.unwrap_or(1),
});
}
}
}
let (_, obj) = parse_indirect(data, off, &|_| None)?;
let Object::Stream(stream) = obj else { return Err(CosError::Syntax { offset: off, detail: "expected xref table or stream".into() }) };
if stream.dict.name(b"Type") != Some(b"XRef") {
return Err(CosError::Syntax { offset: off, detail: "stream at startxref is not /Type /XRef".into() });
}
let w: Vec<usize> = stream
.dict
.get(b"W")
.and_then(Object::as_array)
.map(|a| a.iter().map(|x| x.as_int().unwrap_or(0).clamp(0, 8) as usize).collect())
.unwrap_or_default();
if w.len() < 3 {
return Err(CosError::Syntax { offset: off, detail: "xref stream /W invalid".into() });
}
let widths = [w[0], w[1], w[2]];
let row = widths.iter().sum::<usize>();
if row == 0 {
return Err(CosError::Syntax { offset: off, detail: "xref stream row width 0".into() });
}
let size = stream.dict.int(b"Size").unwrap_or(0).max(0);
let index: Vec<i64> = match stream.dict.get(b"Index").and_then(Object::as_array) {
Some(a) => a.iter().filter_map(Object::as_int).collect(),
None => vec![0, size],
};
let mut requested_rows = 0usize;
let mut previous_end = None;
for pair in index.chunks(2) {
let [start, count] = pair else { break };
if !valid_xref_range(*start, *count) {
return Err(CosError::Syntax { offset: off, detail: "invalid xref object range".into() });
}
let Ok(count) = usize::try_from(*count) else { return Ok(None) };
let Some(next) = requested_rows.checked_add(count).filter(|rows| *rows <= MAX_COMPACT_XREF_ROWS) else { return Ok(None) };
if count > 0 {
let end = (*start as u64).saturating_add(count as u64);
if previous_end.is_some_and(|previous| (*start as u64) < previous) {
return Ok(None);
}
previous_end = Some(end);
}
requested_rows = next;
}
if requested_rows.checked_mul(row).is_none_or(|bytes| bytes > MAX_COMPACT_XREF_BYTES) {
return Ok(None);
}
let raw = stream.decoded_within(MAX_XREF_OBJECTS.saturating_mul(row).min(self.stream_limit).min(MAX_COMPACT_XREF_BYTES))?;
if raw.capacity() > MAX_COMPACT_XREF_BYTES {
return Ok(None);
}
let mut ranges = Vec::new();
let mut row_start = 0usize;
let available_rows = raw.len() / row;
for pair in index.chunks(2) {
let [start, count] = pair else { break };
let range_start = *start as u32;
let Ok(requested) = usize::try_from(*count) else { return Ok(None) };
let range_count = requested.min(available_rows.saturating_sub(row_start));
if range_count > 0 {
let Ok(range_count_u32) = u32::try_from(range_count) else { return Ok(None) };
let Some(byte_offset) = row_start.checked_mul(row) else { return Ok(None) };
ranges.push(XrefRange { start: range_start, count: range_count_u32, offset: byte_offset, stride: row });
}
let Some(next) = row_start.checked_add(range_count) else { return Ok(None) };
row_start = next;
}
// Rows the materialized reader sets aside (reserved types, offset 0) are left to it.
for range in &ranges {
for i in 0..range.count as usize {
let at = range.offset + i * row;
let Some(r) = raw.get(at..at + row) else { break };
let field = |from: usize, width: usize, default: u64| -> u64 {
if width == 0 {
return default;
}
r[from..from + width].iter().fold(0u64, |acc, byte| acc << 8 | u64::from(*byte))
};
let kind = field(0, widths[0], 1);
let offset = field(widths[0], widths[1], 0);
if kind > 2 || (kind == 1 && offset == 0 && range.start + i as u32 != 0) {
return Ok(None);
}
}
}
let max_num = ranges.iter().filter_map(|range| range.start.checked_add(range.count.saturating_sub(1))).max().unwrap_or(0);
let row_count = row_start;
let section = XrefSection {
data: XrefSectionData::Stream { bytes: Arc::new(raw), widths },
ranges,
row_count,
max_num,
has_entries: false,
renumbered: false,
};
let has_entries = section.has_any_entry();
let section = XrefSection { has_entries, ..section };
Ok(Some((section, stream.dict.clone(), true)))
}
/// Read the cross-reference chain from `startxref` into one map of entries, newer sections
/// winning, with the trailer, the revisions oldest first and the set-aside entries. A chain
/// that lists more than [`MAX_XREF_OBJECTS`] rows in all is refused with
/// [`CosError::Syntax`], and the caller rebuilds the file from object headers (#5082).
#[allow(clippy::type_complexity)]
pub(super) fn read_xref_chain_materialized(
&self,
log: &mut Vec<String>,
) -> Result<(BTreeMap<u32, XrefEntry>, Dict, Vec<Revision>, Vec<(u64, u32)>, Vec<String>), CosError> {
self.read_xref_chain_materialized_with_limit(log, MAX_XREF_OBJECTS)
}
/// [`Self::read_xref_chain_materialized`] refusing a chain past `row_limit` rows in all
/// instead of [`MAX_XREF_OBJECTS`], so tests can reach the limit with a few rows.
#[allow(clippy::type_complexity)]
pub(super) fn read_xref_chain_materialized_with_limit(
&self,
log: &mut Vec<String>,
row_limit: usize,
) -> Result<(BTreeMap<u32, XrefEntry>, Dict, Vec<Revision>, Vec<(u64, u32)>, Vec<String>), CosError> {
let data = &self.data;
let tail_start = data.len().saturating_sub(4096);
let sx = rfind(data, b"startxref", tail_start).ok_or(CosError::Syntax { offset: data.len(), detail: "no startxref".into() })?;
let mut lx = Lexer::new(data, sx + 9);
let first = match lx.object() {
Ok(Object::Int(n)) if n >= 0 => n as u64,
_ => return Err(CosError::Syntax { offset: sx, detail: "bad startxref value".into() }),
};
let mut entries = BTreeMap::new();
let mut trailer: Option<Dict> = None;
let mut revisions = Vec::new();
let mut oddities = Vec::new();
let mut visited = HashSet::new();
// Hybrid streams already read, by offset: their entries, or `None` when unreadable. A newer
// section has merged every entry of one into `entries`, so an older section that names it
// again only has its own table compared with them, without decoding the stream again.
// At most MAX_XREF_OBJECTS entries are kept in all; a stream past that is read again.
let mut hybrids: HashMap<i64, Option<BTreeMap<u32, XrefEntry>>> = HashMap::new();
let (mut hybrid_rows, mut noted) = (0usize, HashSet::new());
let mut generation_repairs = Vec::new();
let mut recorded = HashSet::new();
let mut diagnose = true;
// Every row the sections (and their hybrid streams) list, counted by `read_section` as it
// reads them, before any are merged or deduplicated.
let mut merged_rows = 0usize;
let mut next = Some(first);
while let Some(off) = next.take() {
if !visited.insert(off) || visited.len() > 10_000 {
log.push(format!("cross-reference chain loops at offset {off}; stopped"));
break;
}
// This section's own entries; older sections only fill what newer ones left out.
let mut sect = BTreeMap::new();
// Object numbers whose entry in this section was set aside ([`Self::xref_oddities`]).
let mut odd = Vec::new();
let rows_before = merged_rows;
let ((sect_trailer, is_stream), header_relative) = match self.read_section(off as usize, &mut sect, &mut odd, &mut merged_rows, log) {
Ok(v) => (v, false),
Err(e) if self.header_offset > 0 => {
log.push(format!("xref at {off} unreadable ({e}); retrying relative to the header"));
sect.clear();
odd.clear();
merged_rows = rows_before;
(self.read_section(off as usize + self.header_offset, &mut sect, &mut odd, &mut merged_rows, log)?, true)
}
Err(e) => {
// An unreadable older section loses every object only it locates (the page
// tree, often): rebuild from object headers, as readers do (#1656).
if !revisions.is_empty() {
log.push(format!("older cross-reference section at {off} is unreadable: {e}"));
}
return Err(e);
}
};
revisions.push(Revision { xref_offset: off, is_stream, header_relative });
for (&num, &entry) in &sect {
if !diagnose {
break;
}
diagnose = self.inspect_xref_generation_repair(num, entry, &mut recorded, &mut generation_repairs);
}
// Hybrid files (§7.5.8.4): the /XRefStm entries supplement the table and replace the
// free entries it gives objects only a cross-reference stream can locate.
let hybrid = sect_trailer.int(b"XRefStm").filter(|x| *x >= 0);
if let Some((x, read)) = hybrid.and_then(|x| hybrids.get(&x).map(|read| (x, read))) {
if noted.insert(x) {
log.push(format!("hybrid /XRefStm at {x} named again by the section at {off}; read once"));
}
if let Some(stm) = read {
for (num, old) in &sect {
// The table places the object somewhere other than the stream does.
if !matches!(old, XrefEntry::Free { .. }) && stm.get(num).is_some_and(|e| e != old) {
odd.push(*num);
}
}
}
} else if let Some(x) = hybrid {
let mut stm = BTreeMap::new();
let read = match self.read_section(x as usize, &mut stm, &mut odd, &mut merged_rows, log) {
Ok(_) => {
for (&num, &entry) in &stm {
if !diagnose {
break;
}
diagnose = self.inspect_xref_generation_repair(num, entry, &mut recorded, &mut generation_repairs);
}
for (&num, &e) in &stm {
match sect.get(&num) {
None | Some(XrefEntry::Free { .. }) => {
sect.insert(num, e);
}
// The table already places the object somewhere else.
Some(old) if *old != e => odd.push(num),
Some(_) => {}
}
}
Some(stm)
}
Err(e) => {
log.push(format!("hybrid /XRefStm at {x} unreadable: {e}"));
None
}
};
let rows = read.as_ref().map_or(0, BTreeMap::len);
if let Some(total) = hybrid_rows.checked_add(rows).filter(|total| *total <= MAX_XREF_OBJECTS) {
hybrid_rows = total;
hybrids.insert(x, read);
}
}
// Each section's rows are capped, so the chain is too: at most `row_limit` rows
// (MAX_XREF_OBJECTS outside tests) read in all, or the file is rebuilt from object
// headers, as when one section's stream decodes past that. This counts every row
// read, before deduplication, not the map's size: a chain that relists the same
// objects, or a stream whose `/Index` repeats a range (`[0 4 0 4]`), costs as much to
// merge (#5082). Rows set aside are among the rows read, and a row is set aside at
// most twice (in its section and against a hybrid stream), so they stay bounded too. A
// section's rows (its hybrid stream's included) never outnumber the rows the compact
// reader counts for it, and that reader stops at about a quarter of MAX_XREF_OBJECTS:
// no chain within its limit is refused.
if merged_rows > row_limit {
return Err(CosError::Syntax {
offset: usize::try_from(off).unwrap_or(usize::MAX),
detail: format!("cross-reference chain lists more than {row_limit} entries"),
});
}
oddities.extend(odd.into_iter().map(|num| (off, num)));
for (num, e) in sect {
entries.entry(num).or_insert(e);
}
next = prev_offset(data, &sect_trailer, off, |num| entries.get(&num).copied(), log);
if trailer.is_none() {
trailer = Some(sect_trailer);
}
}
revisions.reverse();
let mut trailer = trailer.ok_or(CosError::Syntax { offset: 0, detail: "no trailer".into() })?;
for k in [&b"Prev"[..], b"XRefStm", b"Type", b"W", b"Index", b"Filter", b"DecodeParms", b"Length"] {
trailer.remove(k);
}
Ok((entries, trailer, revisions, oddities, generation_repairs))
}
/// Read one section at `off`, adding entries not already known (newer sections win). `odd`
/// collects the object numbers whose entry here is set aside ([`Self::xref_oddities`]), and
/// `counted` grows by every row read, repeated and set-aside rows included (#5082).
fn read_section(
&self,
off: usize,
entries: &mut BTreeMap<u32, XrefEntry>,
odd: &mut Vec<u32>,
counted: &mut usize,
log: &mut Vec<String>,
) -> Result<(Dict, bool), CosError> {
#[cfg(test)]
SECTION_READS.with(|c| c.set(c.get() + 1));
let data = &self.data;
let mut lx = Lexer::new(data, off);
if lx.eat_keyword(b"xref") {
loop {
lx.skip_ws();
if lx.eat_keyword(b"trailer") {
let t = lx.object()?;
let d = t.as_dict().cloned().ok_or(CosError::Syntax { offset: lx.pos, detail: "trailer is not a dictionary".into() })?;
return Ok((d, false));
}
let (Ok(Object::Int(start)), Ok(Object::Int(count))) = (lx.object(), lx.object()) else {
return Err(CosError::Syntax { offset: lx.pos, detail: "bad xref subsection header".into() });
};
if !valid_xref_range(start, count) || count > 10_000_000 {
return Err(CosError::Syntax { offset: lx.pos, detail: "invalid xref object range".into() });
}
let shifted = subsection_start(data, self.header_offset, lx.pos, start, count);
if shifted != start {
log.push(format!("xref subsection at {off} is numbered from 1 but begins with object 0's entry; read from 0"));
}
let start = shifted;
for i in 0..count {
*counted = counted.saturating_add(1);
lx.skip_ws();
let o = lx.token();
lx.skip_ws();
let g = lx.token();
lx.skip_ws();
let kind = lx.token();
let num = (start + i) as u32;
let offset = std::str::from_utf8(o).ok().and_then(|s| s.parse::<u64>().ok());
let generation = std::str::from_utf8(g).ok().and_then(|s| s.parse::<u32>().ok()).unwrap_or(0).min(u16::MAX as u32) as u16;
let entry = match (kind, offset) {
(b"n", Some(offset)) => XrefEntry::InFile { offset, generation },
(b"f", _) => XrefEntry::Free { next_generation: generation },
_ => return Err(CosError::Syntax { offset: lx.pos, detail: "bad xref entry".into() }),
};
// Offset 0 "in use" entries are a known producer bug: ignore them. (Object 0
// is never a real object, so its entry cannot mislead anyone.)
if matches!(entry, XrefEntry::InFile { offset: 0, .. }) {
if num != 0 {
odd.push(num);
}
continue;
}
if *entries.entry(num).or_insert(entry) != entry {
odd.push(num);
}
}
}
}
// Cross-reference stream.
let (_, obj) = parse_indirect(data, off, &|_| None)?;
let Object::Stream(s) = obj else { return Err(CosError::Syntax { offset: off, detail: "expected xref table or stream".into() }) };
if s.dict.name(b"Type") != Some(b"XRef") {
return Err(CosError::Syntax { offset: off, detail: "stream at startxref is not /Type /XRef".into() });
}
let w: Vec<usize> = s
.dict
.get(b"W")
.and_then(Object::as_array)
.map(|a| a.iter().map(|x| x.as_int().unwrap_or(0).clamp(0, 8) as usize).collect())
.unwrap_or_default();
if w.len() < 3 {
return Err(CosError::Syntax { offset: off, detail: "xref stream /W invalid".into() });
}
let size = s.dict.int(b"Size").unwrap_or(0).max(0);
let index: Vec<i64> = match s.dict.get(b"Index").and_then(Object::as_array) {
Some(a) => a.iter().filter_map(Object::as_int).collect(),
None => vec![0, size],
};
let row = w[0] + w[1] + w[2];
if row == 0 {
return Err(CosError::Syntax { offset: off, detail: "xref stream row width 0".into() });
}
// One row per object, and a document has at most 8,388,607 indirect objects (the classic
// implementation limit): decoding more than that (about 200 MB at the widest rows) is a
// decompression bomb, not cross-reference data. The section is then reconstructed.
let raw = s.decoded_within(MAX_XREF_OBJECTS.saturating_mul(row).min(self.stream_limit))?;
let field = |r: &[u8], from: usize, len: usize, default: u64| -> u64 {
if len == 0 {
return default;
}
r[from..from + len].iter().fold(0u64, |acc, b| acc << 8 | *b as u64)
};
let mut rows = raw.chunks_exact(row);
for pair in index.chunks(2) {
let [start, count] = pair else { break };
if !valid_xref_range(*start, *count) {
return Err(CosError::Syntax { offset: off, detail: "invalid xref object range".into() });
}
for i in 0..*count {
let Some(r) = rows.next() else { break };
*counted = counted.saturating_add(1);
let t = field(r, 0, w[0], 1);
let a = field(r, w[0], w[1], 0);
let b = field(r, w[0] + w[1], w[2], 0);
let num = (*start + i) as u32;
let entry = match t {
0 => XrefEntry::Free { next_generation: b.min(u16::MAX as u64) as u16 },
1 => XrefEntry::InFile { offset: a, generation: b.min(u16::MAX as u64) as u16 },
2 => XrefEntry::InStream { stream: a.min(u32::MAX as u64) as u32, index: b.min(u32::MAX as u64) as u32 },
// Reserved types are treated as null references (§7.5.8.3).
_ => {
odd.push(num);
continue;
}
};
if matches!(entry, XrefEntry::InFile { offset: 0, .. }) {
if num != 0 {
odd.push(num);
}
continue;
}
if *entries.entry(num).or_insert(entry) != entry {
odd.push(num);
}
}
}
Ok((s.dict.clone(), true))
}
/// Cross-reference data that leaves out objects the file contains: a subsection numbered one
/// too low (object 0 in use, the last object without an entry), or an in-use entry at offset 0
/// that nothing else replaces. (A number no section lists is not evidence: it is a null
/// object, §7.3.10, and valid files have them below an oversized `/Size`.) Viewers rebuild
/// the table and show these objects; reading them as null would drop them from every copy.
/// When one of those shows, the file is scanned once for object headers, and wrong offsets
/// are looked up there (`load`). Only the numbers the damage leaves out take the last
/// header's location: one whose entry was set aside, or the last object of a subsection
/// numbered one too low. Any other unlisted number stays null, and existing
/// entries are never replaced: an object freed by an update stays freed.
pub(super) fn fill_xref_gaps(&mut self, log: &mut Vec<String>) {
let entries = &self.entries;
let shifted = matches!(entries.lookup(0, &self.data), Some(XrefEntry::InFile { .. }));
let set_aside = self.xref_oddities.iter().any(|(_, num)| entries.lookup(*num, &self.data).is_none());
if !(shifted || set_aside) {
return;
}
let mut located = BTreeMap::new();
for (num, offset, generation, _) in object_headers(&self.data) {
located.insert(num, (offset, generation));
}
let mut filled = entries.to_map(&self.data);
// A subsection `0 N` whose rows are those of objects 1 to N leaves out object N, whatever
// newer sections say about its neighbours.
let past_shift: HashSet<u32> = if shifted {
self.revisions
.iter()
.filter(|r| !r.is_stream)
.filter_map(|r| usize::try_from(r.xref_offset).ok().map(|o| if r.header_relative { o.saturating_add(self.header_offset) } else { o }))
.filter_map(|at| shifted_subsection_end(&self.data, at))
.collect()
} else {
HashSet::new()
};
let odd: HashSet<u32> = self.xref_oddities.iter().map(|(_, n)| *n).collect();
let mut found = 0usize;
for (&num, &(offset, generation)) in &located {
let left_out = past_shift.contains(&num) || odd.contains(&num);
if num != 0 && !filled.contains_key(&num) && left_out {
filled.insert(num, XrefEntry::InFile { offset, generation });
found += 1;
}
}
if found > 0 {
log.push(format!("the cross-reference data leaves out {found} objects the file contains; located them by their object headers"));
self.entries = Arc::new(XrefIndex::Materialized(Arc::new(filled), Arc::default()));
}
self.located = Arc::new(located);
}
/// Rebuild the cross-reference index by scanning for `N G obj` headers (§7.5.4 recovery).
pub(super) fn reconstruct(&mut self, log: &mut Vec<String>) -> Result<(), CosError> {
let data = self.data.clone();
let mut entries: BTreeMap<u32, XrefEntry> = BTreeMap::new();
let mut objstms = Vec::new();
// Offsets of objects that may be cross-reference streams: their dictionaries are the
// trailers of files that have no `trailer` keyword (§7.5.8).
let mut xref_streams = Vec::new();
let headers = object_headers(&data);
// Where `/XRef` occurs in the file, found in one pass.
let mut xref_marks = Vec::new();
let mut mark_from = 0;
while let Some(p) = find(&data, b"/XRef", mark_from, data.len()) {
xref_marks.push(p);
mark_from = p + 1;
}
for &(num, offset, generation, body) in &headers {
entries.insert(num, XrefEntry::InFile { offset, generation });
objstms.push(num);
// `/XRef` within the 1024 bytes after the header (the first one that is there).
let first = xref_marks.partition_point(|&p| p < body);
if xref_marks.get(first).is_some_and(|&p| p + 5 <= body.saturating_add(1024).min(data.len()))
&& let Ok(at) = usize::try_from(offset)
{
xref_streams.push(at);
}
}
if entries.is_empty() {
return Err(CosError::Syntax { offset: 0, detail: "no objects found".into() });
}
self.entries = Arc::new(XrefIndex::Materialized(Arc::new(entries.clone()), Arc::default()));
self.cache.lock().map_err(|_| CosError::Poisoned)?.clear();
*self.unreadable.lock().map_err(|_| CosError::Poisoned)? = Unreadable::default();
// Objects inside object streams.
for num in objstms {
if let Ok(o) = self.try_get(num)
&& let Object::Stream(s) = &*o
&& s.dict.name(b"Type") == Some(b"ObjStm")
&& let Ok(stm) = self.objstm(num)
{
for (idx, member) in stm.index.iter().enumerate() {
if let Some((inner, _)) = member {
entries.entry(*inner).or_insert(XrefEntry::InStream { stream: num, index: idx as u32 });
}
}
}
}
let object_count = entries.len();
self.entries = Arc::new(XrefIndex::Materialized(Arc::new(entries), Arc::default()));
// Failures from the scan stand: the members added change no scanned entry, and a
// `/Length` found only now cannot make a stream unreadable.
self.cache.lock().map_err(|_| CosError::Poisoned)?.clear();
// Trailer: the last `trailer` dictionary or cross-reference stream dictionary with a
// usable /Root (last in the file, whichever kind), else the catalog itself. One whose
// /Root is a catalog (as `root_is_catalog` judges it) beats one whose /Root is another
// dictionary, such as /Info, wherever it is; an FDF's /Root (no /Type, no /Pages) still
// counts when nothing better is there.
// A file whose only trailers are cross-reference streams (and whose `startxref` is gone
// or points nowhere) keeps its /Info, /ID and /Encrypt this way.
let rank = |at: usize, d: &Dict| {
let root = self.try_get_reference(d.reference(b"Root")?).ok()?;
let Object::Dict(c) = &*root else { return None };
Some((c.name(b"Type") == Some(b"Catalog") || c.contains(b"Pages"), at))
};
let mut trailer: Option<((bool, usize), Dict)> = None;
let mut from = 0;
while let Some(p) = find(&data, b"trailer", from, data.len()) {
from = p + 7;
let mut lx = Lexer::new(&data, from);
if let Ok(Object::Dict(d)) = lx.object()
&& let Some(r) = rank(p, &d)
&& trailer.as_ref().is_none_or(|(best, _)| r > *best)
{
trailer = Some((r, d));
}
}
for at in xref_streams {
if trailer.as_ref().is_some_and(|((catalog, p), _)| *catalog && *p > at) {
continue;
}
if let Ok((_, Object::Stream(s))) = parse_indirect_shared(&data, at, &|_| None, &self.endstreams)
&& s.dict.name(b"Type") == Some(b"XRef")
&& let Some(r) = rank(at, &s.dict)
&& trailer.as_ref().is_none_or(|(best, _)| r > *best)
{
trailer = Some((r, s.dict));
}
}
let trailer = match trailer {
Some((_, t)) => t,
None => {
let catalog = self
.entries
.object_numbers(&self.data)
.into_iter()
.find(|n| matches!(self.try_get(*n).as_deref(), Ok(Object::Dict(d)) if d.name(b"Type") == Some(b"Catalog")));
let Some(c) = catalog else { return Err(CosError::Syntax { offset: 0, detail: "no document catalog found".into() }) };
let mut t = Dict::new();
t.set(b"Root".to_vec(), Object::Ref(ObjRef::new(c, self.generation(c))));
t
}
};
let mut trailer = trailer;
// What a cross-reference stream's dictionary carries for itself goes, as the chain
// readers drop it: a save writes its own.
for k in [&b"Prev"[..], b"XRefStm", b"Type", b"W", b"Index", b"Filter", b"DecodeParms", b"Length"] {
trailer.remove(k);
}
self.trailer = trailer;
self.revisions = Arc::new(Vec::new());
log.push(format!("rebuilt cross-reference index from {object_count} object headers"));
Ok(())
}
}
/// For a classic table at `at` whose first subsection is `0 N` and opens with an in-use row
/// (the rows of objects 1 to N), the number N its rows really end at.
fn shifted_subsection_end(data: &[u8], at: usize) -> Option<u32> {
let mut lx = Lexer::new(data, at);
if !lx.eat_keyword(b"xref") {
return None;
}
let (Ok(Object::Int(0)), Ok(Object::Int(count))) = (lx.object(), lx.object()) else { return None };
lx.skip_ws();
let offset = std::str::from_utf8(lx.token()).ok().and_then(|s| s.parse::<u64>().ok());
lx.skip_ws();
lx.token();
lx.skip_ws();
match (offset, lx.token()) {
(Some(offset), b"n") if offset != 0 => u32::try_from(count).ok().filter(|n| *n > 0),
_ => None,
}
}
/// Every `N G obj` header in `data`, in file order, as (number, offset, generation, position
/// after `obj`).
fn object_headers(data: &[u8]) -> Vec<(u32, u64, u16, usize)> {
let mut headers = Vec::new();
let mut i = 0;
while i < data.len() {
// Object headers start at a line start (or after whitespace).
let at_start = i == 0 || is_whitespace(data[i - 1]) || data[i - 1] == b'>';
if at_start && data[i].is_ascii_digit() {
let mut lx = Lexer::new(data, i);
let n = lx.token();
lx.skip_ws();
let g = lx.token();
if !n.is_empty() && !g.is_empty() && n.iter().all(u8::is_ascii_digit) && g.iter().all(u8::is_ascii_digit) && lx.eat_keyword(b"obj") {
let num = std::str::from_utf8(n).ok().and_then(|s| s.parse::<u32>().ok());
let generation = std::str::from_utf8(g).ok().and_then(|s| s.parse::<u16>().ok()).unwrap_or(0);
if let Some(num) = num {
headers.push((num, i as u64, generation, lx.pos));
}
i = lx.pos;
continue;
}
}
i += 1;
}
headers
}
/// The offset of the section before the one whose trailer this is (`/Prev`), if any. Some writers
/// give it as a reference, `N 0 R`. When the sections read so far (`lookup`) place that object
/// and it holds an offset (iText's `indirectPrev.pdf`), that is the offset; otherwise, as the
/// tiff2pdf files in JHOVE's PDF-HUL-157 cases need, the sections before are at N. Either way the
/// reference is noted. Any other value that is not an offset is noted and ends the chain.
fn prev_offset(data: &[u8], trailer: &Dict, off: u64, lookup: impl Fn(u32) -> Option<XrefEntry>, log: &mut Vec<String>) -> Option<u64> {
match trailer.get(b"Prev") {
Some(Object::Int(p)) if *p >= 0 => Some(*p as u64),
Some(Object::Ref(r)) => {
let held = match lookup(r.num) {
Some(XrefEntry::InFile { offset, generation }) if generation == r.generation => {
usize::try_from(offset).ok().and_then(|at| parse_indirect(data, at, &|_| None).ok()).and_then(|(at, value)| match value {
Object::Int(p) if at == *r && p >= 0 => Some(p as u64),
_ => None,
})
}
_ => None,
};
match held {
Some(p) => {
log.push(format!(
"the trailer at {off} gives /Prev as the reference {} {} R; reading the offset {p} it holds",
r.num, r.generation
));
Some(p)
}
None => {
log.push(format!(
"the trailer at {off} gives /Prev as the reference {} {} R; reading it as offset {}",
r.num, r.generation, r.num
));
Some(u64::from(r.num))
}
}
}
Some(_) => {
log.push(format!("the trailer at {off} has a /Prev that is not an offset; the sections before it are unreachable"));
None
}
None => None,
}
}
pub(crate) fn find(hay: &[u8], needle: &[u8], from: usize, to: usize) -> Option<usize> {
let to = to.min(hay.len());
if from >= to || needle.is_empty() || to - from < needle.len() {
return None;
}
let hay = &hay[from..to];
// Look for the needle's rarest byte first, a plain byte scan, and compare the rest only there.
const COMMON: &[u8] = b" etaoinsrhldcumfpgwybvkxjqz";
let (k, rare) =
needle.iter().copied().enumerate().max_by_key(|&(_, b)| COMMON.iter().position(|&c| c == b).unwrap_or(COMMON.len())).unwrap_or((0, 0));
let mut i = k;
while let Some(p) = hay.get(i..).and_then(|rest| position_of(rest, rare)) {
let start = i + p - k;
if hay.get(start..start + needle.len()) == Some(needle) {
return Some(start + from);
}
i += p + 1;
}
None
}
/// The first position of `byte` in `hay`. Whole 32-byte blocks are tested without branching, which
/// the compiler turns into vector compares; only a block that holds the byte is searched.
fn position_of(hay: &[u8], byte: u8) -> Option<usize> {
const BLOCK: usize = 32;
let (blocks, rest) = hay.as_chunks::<BLOCK>();
for (i, block) in blocks.iter().enumerate() {
if block.iter().fold(false, |any, &b| any | (b == byte)) {
return block.iter().position(|&b| b == byte).map(|p| i * BLOCK + p);
}
}
rest.iter().position(|&b| b == byte).map(|p| blocks.len() * BLOCK + p)
}
pub(super) fn rfind(hay: &[u8], needle: &[u8], from: usize) -> Option<usize> {
if hay.len() < needle.len() {
return None;
}
(from..=hay.len() - needle.len()).rev().find(|&i| &hay[i..i + needle.len()] == needle)
}
/// The object number a classic xref subsection really starts at. A common producer bug heads the
/// table `1 n` and then writes object 0's free entry (`0000000000 65535 f`) as its first row, so
/// every row would be read one number too high and object 1 as free. Such a subsection is read
/// from 0 when its second row, read that way, locates an object that is indeed numbered 1, at
/// that offset or, as `load` reads offsets, relative to a header shifted by `header_offset`.
/// `rows` is the position just after the subsection header.
fn subsection_start(data: &[u8], header_offset: usize, rows: usize, start: i64, count: i64) -> i64 {
if start != 1 || count < 2 {
return start;
}
let mut lx = Lexer::new(data, rows);
let mut row = || {
lx.skip_ws();
let offset = std::str::from_utf8(lx.token()).ok().and_then(|s| s.parse::<u64>().ok());
lx.skip_ws();
let generation = std::str::from_utf8(lx.token()).ok().and_then(|s| s.parse::<u32>().ok());
lx.skip_ws();
(offset, generation, lx.token() == b"f")
};
let (Some(0), Some(65535), true) = row() else { return start };
let (Some(second), _, false) = row() else { return start };
let object_one = |at: usize| {
if at >= data.len() {
return false;
}
let mut obj = Lexer::new(data, at);
matches!((obj.object(), obj.object()), (Ok(Object::Int(1)), Ok(Object::Int(_)))) && obj.eat_keyword(b"obj")
};
let Ok(second) = usize::try_from(second) else { return start };
if object_one(second) || (header_offset != 0 && object_one(second.saturating_add(header_offset))) { 0 } else { start }
}
/// Object numbers must fit the representation used by references and the xref map.
/// Validate the whole subsection before adding entries, so neither arithmetic overflow nor
/// truncation can turn a damaged range into entries for unrelated objects.
fn valid_xref_range(start: i64, count: i64) -> bool {
u32::try_from(start).is_ok() && count >= 0 && start.checked_add(count).is_some_and(|end| end <= i64::from(u32::MAX) + 1)
}
/// The most indirect objects a document can have (the classic implementation limit, ISO
/// 32000-1 Annex C): it bounds how much data a cross-reference stream can hold.
const MAX_XREF_OBJECTS: usize = 8_388_607;
+505
View File
@@ -0,0 +1,505 @@
use super::xref::XrefEntry;
use super::{Arc, BTreeMap, CosError, Dict, Document, HashMap, ObjRef, Object, Weak};
thread_local! {
/// The objects being loaded on this thread, innermost last (see `Document::try_get`).
static LOADING: std::cell::RefCell<Vec<u32>> = const { std::cell::RefCell::new(Vec::new()) };
/// Indirect `/Length` values resolved during the current outermost load: a damaged file
/// retries a parse several ways, and each retry would resolve the same lengths again
/// (exponential in the chain of streams whose lengths point at further streams). Each is
/// kept with whether resolving it read unsaved edits.
pub(super) static LENGTHS: std::cell::RefCell<HashMap<ObjRef, Length>> = std::cell::RefCell::new(HashMap::new());
/// What the loads running on this thread have read ([`ReadFrame`]): how many frames are
/// open, the object numbers read inside them, and how often that list was full.
static READS: std::cell::RefCell<(u32, Vec<u32>, u64)> = const { std::cell::RefCell::new((0, Vec::new(), 0)) };
/// Unsaved edits read on this thread (`Document::try_get`, and a `/Length` in `Document::load`).
static OVERLAY_READS: std::cell::Cell<u64> = const { std::cell::Cell::new(0) };
/// How deep `Document::scan_for` is nested on this thread, and how many indirect `/Length`
/// values the outermost scan and those inside it have resolved ([`ScanLengths`]).
static SCAN_LENGTHS: std::cell::Cell<(u32, u32)> = const { std::cell::Cell::new((0, 0)) };
}
/// The indirect `/Length` values a scan of the file for an object may resolve, shared with the
/// scans it leads to. Each header in the file that starts with the object's number is parsed, and
/// a `/Length` that names an object the cross-reference table cannot place scans the file for that
/// one: a damaged file with many such headers, each naming another missing object, would read the
/// whole file once per header. Past the budget a `/Length` is left unresolved, and the stream is
/// read as before resolution, by its `endstream`.
pub(super) struct ScanLengths;
impl ScanLengths {
/// A real object is defined once per revision, and its `/Length` once per definition.
pub(super) const MAX: u32 = 32;
/// Marks a scan as running; the outermost one starts the budget afresh.
pub(super) fn enter() -> Self {
SCAN_LENGTHS.with(|c| {
let (depth, used) = c.get();
c.set((depth.saturating_add(1), if depth == 0 { 0 } else { used }));
});
ScanLengths
}
/// Whether one more `/Length` may be resolved, counting it if so.
pub(super) fn take(&self) -> bool {
SCAN_LENGTHS.with(|c| {
let (depth, used) = c.get();
if used >= Self::MAX {
return false;
}
c.set((depth, used.saturating_add(1)));
true
})
}
}
impl Drop for ScanLengths {
fn drop(&mut self) {
SCAN_LENGTHS.with(|c| {
let (depth, used) = c.get();
c.set((depth.saturating_sub(1), used));
});
}
}
/// A resolved `/Length` in [`LENGTHS`]: its value, whether resolving it read unsaved edits, and
/// what it read.
pub(super) type Length = (Option<i64>, bool, Reads);
/// The other objects a cached load read: an object stream for a member, an indirect `/Length`,
/// and what those read in turn. Clones share the caches, so a document whose edits include one of
/// them reads the object again instead (`Document::try_get`); `None` is a load that read more
/// than [`Reads::MAX`], which such a document always reads again.
#[derive(Clone, Debug, Default)]
pub(super) struct Reads(Option<Arc<[u32]>>);
impl Reads {
/// Objects one entry keeps: a member's stream and a length or two, and what they read.
const MAX: usize = 64;
/// Object numbers the open frames keep between them.
const HELD: usize = 4096;
/// Whether `overlay` (a document's edits) holds any of these objects.
pub(super) fn touch(&self, overlay: &BTreeMap<u32, Slot>) -> bool {
!overlay.is_empty() && self.0.as_ref().is_none_or(|nums| nums.iter().any(|n| overlay.contains_key(n)))
}
fn bytes(&self) -> usize {
self.0.as_ref().map_or(0, |nums| nums.len().saturating_mul(std::mem::size_of::<u32>()))
}
}
/// Counts `num` as read by the loads open on this thread, with what loading it read (`reads`).
pub(super) fn note_read(num: u32, reads: Option<&Reads>) {
READS.with(|r| {
let (depth, held, full) = &mut *r.borrow_mut();
if *depth == 0 {
return;
}
let more = reads.map_or(Some(&[][..]), |r| r.0.as_deref());
match more {
Some(more) if held.len().saturating_add(more.len()) < Reads::HELD => {
held.push(num);
held.extend_from_slice(more);
}
_ => *full = full.wrapping_add(1),
}
});
}
/// Collects what one load reads, for [`Reads`]; frames nest, and each read counts for every
/// frame open around it.
pub(super) struct ReadFrame {
start: usize,
full: u64,
}
impl ReadFrame {
pub(super) fn enter() -> Self {
READS.with(|r| {
let (depth, held, full) = &mut *r.borrow_mut();
*depth = depth.saturating_add(1);
ReadFrame { start: held.len(), full: *full }
})
}
/// What was read since the frame opened.
pub(super) fn reads(&self) -> Reads {
READS.with(|r| {
let (_, held, full) = &*r.borrow();
if *full != self.full {
return Reads(None);
}
let mut nums = held.get(self.start..).unwrap_or_default().to_vec();
nums.sort_unstable();
nums.dedup();
Reads((nums.len() <= Reads::MAX).then(|| Arc::from(nums)))
})
}
}
impl Drop for ReadFrame {
fn drop(&mut self) {
READS.with(|r| {
let (depth, held, _) = &mut *r.borrow_mut();
*depth = depth.saturating_sub(1);
if *depth == 0 {
held.clear();
}
});
}
}
/// Unsaved edits read so far on this thread: a load during which this changes depended on one.
pub(super) fn overlay_reads() -> u64 {
OVERLAY_READS.with(std::cell::Cell::get)
}
pub(super) fn note_overlay_read() {
OVERLAY_READS.with(|c| c.set(c.get().wrapping_add(1)));
}
/// Marks an object as being loaded on this thread. Loading can re-enter (indirect stream
/// `/Length`, object streams): an object already being loaded is a cycle, and the nesting is
/// bounded as well.
pub(super) struct LoadGuard;
impl LoadGuard {
const MAX: usize = 32;
pub(super) fn enter(num: u32) -> Option<Self> {
LOADING.with(|l| {
let mut l = l.borrow_mut();
if l.len() >= Self::MAX || l.contains(&num) {
return None;
}
l.push(num);
Some(LoadGuard)
})
}
}
/// Whether an object is being loaded on this thread: [`LENGTHS`] holds only while one is, since
/// it is cleared when the outermost load ends.
pub(super) fn loading() -> bool {
LOADING.with(|l| !l.borrow().is_empty())
}
/// What loading a file object gave, kept in the document's cache either way.
pub(super) type Loaded = Result<Arc<Object>, CosError>;
/// The bytes a load result holds of its own ([`held_bytes`]); a failure holds none.
pub(super) fn loaded_bytes(loaded: &Loaded, data: &Arc<Vec<u8>>) -> usize {
loaded.as_ref().map_or(0, |o| held_bytes(o, data, 0))
}
impl Drop for LoadGuard {
fn drop(&mut self) {
let outermost = LOADING.with(|l| {
let mut l = l.borrow_mut();
l.pop();
l.is_empty()
});
if outermost {
LENGTHS.with(|m| m.borrow_mut().clear());
}
}
}
#[derive(Clone, Debug)]
pub(super) struct ObjStm {
/// One slot per member in stream order, since cross-reference entries name a member by its
/// position: (object number, byte offset within `data`), or `None` for a member whose number
/// or offset is unusable ([`objstm_member`]).
pub(super) index: Vec<Option<(u32, usize)>>,
pub(super) data: Arc<Vec<u8>>,
/// What parsing it read: the stream itself and its `/Length`, `/DecodeParms` and so on.
pub(super) reads: Reads,
}
/// Where member `num` of an object stream starts: its `offset` past `/First` (`first`), when the
/// number is an object number and the start lies inside the decoded `len` bytes. Checked on every
/// target, so a 32-bit build neither truncates nor wraps a large `/First` or offset (#1815).
pub(super) fn objstm_member(num: i64, offset: i64, first: Option<usize>, len: usize) -> Option<(u32, usize)> {
let num = u32::try_from(num).ok()?;
let start = first?.checked_add(usize::try_from(offset).ok()?)?;
(start < len).then_some((num, start))
}
/// Objects the reader could not read, each with the reason (see `Document::unreadable_objects`).
/// Bounded: a document records past [`Unreadable::MAX`] of the objects it shows only the fact that
/// there were more. Entries a clone has edited are hidden from it and do not count, so the whole
/// list holds at most `MAX` plus the objects edited in clones.
#[derive(Debug, Default)]
pub(super) struct Unreadable {
pub(super) objects: BTreeMap<u32, (u16, String)>,
pub(super) more: bool,
}
impl Unreadable {
pub(super) const MAX: usize = 100;
/// Characters of a reason kept: a parser's message can quote the bytes it stopped at.
const REASON: usize = 160;
/// Whether `num` is to be recorded: not listed yet and there is room among the entries the
/// asking document still shows (`hidden` are the ones it has edited, as in [`Self::lines`]),
/// so a clone's edits never use up its own allowance. A full list notes that there were more,
/// so a caller asks this before building the reason.
pub(super) fn wants(&mut self, num: u32, hidden: impl Fn(u32) -> bool) -> bool {
if self.objects.contains_key(&num) {
return false;
}
if self.objects.keys().filter(|n| !hidden(**n)).count() >= Self::MAX {
self.more = true;
return false;
}
true
}
/// Record `num` once; the first reason given is kept.
pub(super) fn note(&mut self, num: u32, generation: u16, reason: &str, hidden: impl Fn(u32) -> bool) {
if self.wants(num, hidden) {
self.objects.insert(num, (generation, reason.chars().take(Self::REASON).collect()));
}
}
/// One line per recorded object not `hidden`, then one saying there were more, if there were.
pub(super) fn lines(&self, hidden: impl Fn(u32) -> bool) -> Vec<String> {
// Clones share the list and each hides its own edits, so the list may hold more than
// `MAX` entries in all; each document still shows at most `MAX`.
let mut shown = self.objects.iter().filter(|(num, _)| !hidden(**num));
let mut lines: Vec<String> = shown
.by_ref()
.take(Self::MAX)
.map(|(num, (generation, reason))| format!("object {num} {generation} could not be read: {reason}"))
.collect();
if self.more || shown.next().is_some() {
lines.push(format!("more objects could not be read; only the first {} are listed", Self::MAX));
}
lines
}
}
/// Where an overlay slot came from.
#[derive(Clone, Debug, PartialEq)]
pub(super) enum Slot {
Set(u16, Arc<Object>),
/// Both generations accepted from a damaged live source slot survive a repair-aware edit.
/// A full rewrite is required: a new xref slot cannot encode two generations.
RepairedSet(u16, u16, Arc<Object>),
Freed(u16),
}
/// An immutable input/decode context shared by document clones. Holding this identity
/// retains neither the input bytes nor the parsed document. Weak ownership prevents an
/// expired identity from matching a later allocation at the same address.
#[derive(Clone, Debug)]
pub struct SourceIdentity(pub(super) Weak<()>);
impl PartialEq for SourceIdentity {
fn eq(&self, other: &Self) -> bool {
self.0.ptr_eq(&other.0)
}
}
impl Eq for SourceIdentity {}
/// What a document reads as: its source ([`SourceIdentity`]), its trailer and every object changed
/// since it was opened. A clone, or a snapshot that undo restores, has the same state, and every
/// change makes a different one. Changed objects are compared as allocations, which the state
/// keeps alive, so a later object can't take an address one of them had.
#[derive(Clone)]
pub struct ReadState {
pub(super) source: SourceIdentity,
pub(super) trailer: Dict,
pub(super) overlay: BTreeMap<u32, Slot>,
}
impl ReadState {
/// Whether `doc` reads as this state.
pub fn matches(&self, doc: &Document) -> bool {
self.source == doc.source_identity()
&& self.trailer == doc.trailer
&& self.overlay.len() == doc.overlay.len()
&& self.overlay.iter().zip(&doc.overlay).all(|((a, x), (b, y))| {
a == b
&& match (x, y) {
(Slot::Set(g, o), Slot::Set(h, p)) => g == h && Arc::ptr_eq(o, p),
(Slot::RepairedSet(g, a, o), Slot::RepairedSet(h, b, p)) => g == h && a == b && Arc::ptr_eq(o, p),
(Slot::Freed(g), Slot::Freed(h)) => g == h,
_ => false,
}
})
}
}
impl std::fmt::Debug for ReadState {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ReadState").field("changed_objects", &self.overlay.len()).finish_non_exhaustive()
}
}
/// A temporary reader for whole-document work. Its caches never populate the editor or
/// undo snapshots. Returned objects survive cache eviction; a stream read from the file is a
/// view into the input bytes (it keeps the whole input alive while held, not a copy).
pub(crate) struct ObjectReader {
pub(crate) document: Document,
pub(super) decoded_count: usize,
}
impl ObjectReader {
pub(super) const OBJECTS: usize = 128;
pub(super) const STREAM_BYTES: usize = 8 * 1024 * 1024;
pub(crate) fn get(&mut self, reference: ObjRef) -> Arc<Object> {
self.try_get(reference).unwrap_or_else(|_| Arc::new(Object::Null))
}
pub(crate) fn try_get(&mut self, reference: ObjRef) -> Result<Arc<Object>, CosError> {
let object = self.document.try_get_reference(reference);
// A missed eviction here only costs a re-parse, so never fail the read on a poisoned
// lock. The editing-cache evictors propagate instead; they guard user edits.
if let Ok(mut streams) = self.document.objstms.lock() {
// Only insertion changes this private cache between reads. Most neighboring
// objects reuse a decoded stream: do not sum the whole store for every object.
if streams.len() != self.decoded_count {
let bytes = streams.values().fold(0usize, |bytes, stream| {
bytes
.saturating_add(stream.data.capacity())
.saturating_add(stream.index.capacity().saturating_mul(std::mem::size_of::<Option<(u32, usize)>>()))
});
if bytes > Self::STREAM_BYTES || streams.len() >= Self::OBJECTS {
// Retain one oversized active stream so its next object does not require
// decoding it again. That stream still obeys the document's decode cap.
let current = match self.document.xref_entry(reference.num) {
Some(XrefEntry::InStream { stream, .. }) => Some(stream),
_ => None,
};
streams.retain(|num, _| Some(*num) == current);
}
self.decoded_count = streams.len();
}
}
// Parsed objects stay cached (an indirect /Length or a dictionary read again costs a
// lookup, not a parse) until the cache passes its object count or holds more than
// STREAM_BYTES of data of its own. Streams read from the file are views into the
// input bytes, which the document keeps anyway, so they don't count; decrypted
// streams and strings do. Checked after failed loads too: resolving a /Length may
// have cached large objects.
if let Ok(mut cache) = self.document.cache.lock()
&& (cache.len() >= Self::OBJECTS || cache.owned() > Self::STREAM_BYTES)
{
cache.clear();
}
object
}
}
#[cfg(test)]
thread_local! {
/// Cross-reference sections read on this thread.
pub(super) static SECTION_READS: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
}
/// What loading each object gave (a failure as well), by number, each with what it holds ([`held_bytes`]) measured once, when it goes
/// in, and their sum kept beside them.
///
/// Both caches are bounded by those bytes. Summing them on every read walked every cached object
/// whole each time: a file whose page dictionaries carry kilobytes of private data (InDesign's
/// `/PieceInfo`) took seconds to open, almost all of it re-measuring objects already measured
/// (#5873).
#[derive(Default)]
pub(crate) struct ObjectCache {
entries: HashMap<u32, (Loaded, usize, Reads)>,
owned: usize,
}
impl ObjectCache {
/// What loading `num` gave, and what that load read.
pub(super) fn get(&self, num: &u32) -> Option<(&Loaded, &Reads)> {
self.entries.get(num).map(|(object, _, reads)| (object, reads))
}
#[cfg(test)]
pub(crate) fn contains_key(&self, num: &u32) -> bool {
self.entries.contains_key(num)
}
/// Keep what loading `num` gave, which holds `owned` bytes ([`loaded_bytes`]), and what it read.
pub(super) fn insert(&mut self, num: u32, object: Loaded, owned: usize, reads: Reads) {
let owned = owned.saturating_add(reads.bytes());
if let Some((_, before, _)) = self.entries.insert(num, (object, owned, reads)) {
self.owned = self.owned.saturating_sub(before);
}
self.owned = self.owned.saturating_add(owned);
}
pub(crate) fn len(&self) -> usize {
self.entries.len()
}
#[cfg(test)]
pub(crate) fn is_empty(&self) -> bool {
self.entries.is_empty()
}
/// The bytes the cached objects hold between them.
pub(crate) fn owned(&self) -> usize {
self.owned
}
pub(crate) fn clear(&mut self) {
self.entries.clear();
self.owned = 0;
}
pub(crate) fn retain(&mut self, mut keep: impl FnMut(&u32) -> bool) {
self.entries.retain(|num, _| keep(num));
self.owned = self.entries.values().fold(0usize, |n, (_, owned, _)| n.saturating_add(*owned));
}
#[cfg(test)]
pub(crate) fn values(&self) -> impl Iterator<Item = &Arc<Object>> {
self.entries.values().filter_map(|(object, _, _)| object.as_ref().ok())
}
}
/// Nesting [`held_bytes`] walks: past the parser's own nesting limit, so every parsed object
/// is measured in full.
const OWNED_BYTES_DEPTH: usize = 256;
/// What keeping `object` parsed costs: string and stream data that is not a view into `data`
/// (the input file), and the parsed tree around it -- an [`Object`] per value and a key per
/// dictionary entry. An object nested deeper than [`OWNED_BYTES_DEPTH`] cannot be measured and
/// counts as too large to keep.
///
/// The tree counts. Measured by its strings and streams alone, a page dictionary carrying
/// InDesign's `/PieceInfo` -- thousands of small numeric arrays, parsed into tens of thousands of
/// values -- weighed nothing, so the caches kept a thousand of them: a 35 MB file held 1.5 GB
/// (#5873).
pub(crate) fn held_bytes(object: &Object, data: &Arc<Vec<u8>>, depth: usize) -> usize {
if depth > OWNED_BYTES_DEPTH {
return usize::MAX;
}
let node = std::mem::size_of::<Object>();
// A key, its slot beside the value, and room in the index a large dictionary keeps.
let entry = |key: &[u8]| key.len().saturating_add(std::mem::size_of::<(crate::cos::Name, Object)>()).saturating_add(16);
let dict = |d: &crate::cos::Dict, start: usize| {
d.iter().fold(start, |n, (k, v)| n.saturating_add(entry(k)).saturating_add(held_bytes(v, data, depth + 1)))
};
match object {
Object::String(s) => node.saturating_add(s.bytes.len()),
Object::Name(n) => node.saturating_add(n.len()),
Object::Stream(s) => dict(&s.dict, node.saturating_add(if s.raw.shares(data) { 0 } else { s.raw.len() })),
Object::Array(a) => a.iter().fold(node, |n, v| n.saturating_add(held_bytes(v, data, depth + 1))),
Object::Dict(d) => dict(d, node),
_ => node,
}
}
/// How an encrypted document is opened: with a password, or with the file key of a document
/// that has the same encryption (an earlier revision of the same file).
pub(super) enum Credentials<'a> {
Password(Option<&'a str>),
Handler { handler: &'a Arc<crate::crypt::SecurityHandler>, id0: Vec<u8> },
}
File diff suppressed because it is too large Load Diff
+275
View File
@@ -0,0 +1,275 @@
use super::{Arc, BTreeMap, Lexer};
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum XrefEntry {
Free { next_generation: u16 },
InFile { offset: u64, generation: u16 },
InStream { stream: u32, index: u32 },
}
#[derive(Clone, Copy, Debug)]
pub(super) struct XrefRange {
pub(super) start: u32,
pub(super) count: u32,
pub(super) offset: usize,
pub(super) stride: usize,
}
#[derive(Clone, Debug)]
pub(super) enum XrefSectionData {
Table,
Stream { bytes: Arc<Vec<u8>>, widths: [usize; 3] },
}
#[derive(Clone, Debug)]
pub(super) struct XrefSection {
pub(super) data: XrefSectionData,
pub(super) ranges: Vec<XrefRange>,
pub(super) row_count: usize,
pub(super) max_num: u32,
pub(super) has_entries: bool,
/// A table subsection was read one number lower than it says ([`subsection_start`]).
pub(super) renumbered: bool,
}
#[derive(Clone, Debug)]
pub(super) struct XrefRevisionSection {
pub(super) primary: XrefSection,
pub(super) hybrid: Option<XrefSection>,
}
#[derive(Clone, Debug)]
pub(super) enum XrefIndex {
Compact { revisions: Vec<XrefRevisionSection>, max_num: u32, has_entries: bool },
Materialized(Arc<BTreeMap<u32, XrefEntry>>, Arc<Vec<String>>),
}
pub(super) const MAX_COMPACT_XREF_ROWS: usize = 2_000_000;
pub(super) const MAX_COMPACT_XREF_BYTES: usize = 64 * 1024 * 1024;
impl XrefSection {
pub(super) fn has_any_entry(&self) -> bool {
match &self.data {
XrefSectionData::Table => self.has_entries,
XrefSectionData::Stream { bytes, widths } => {
let row_width = widths.iter().sum::<usize>();
if row_width == 0 {
return false;
}
for range in &self.ranges {
let available = bytes.len().saturating_sub(range.offset) / row_width;
for i in 0..available.min(range.count as usize) {
let Some(offset) = i.checked_mul(row_width).and_then(|delta| range.offset.checked_add(delta)) else { continue };
let Some(row) = bytes.get(offset..offset.saturating_add(row_width)) else { continue };
let read_field = |from: usize, width: usize, default: u64| -> Option<u64> {
if width == 0 {
return Some(default);
}
let end = from.checked_add(width)?;
Some(row.get(from..end)?.iter().fold(0u64, |acc, byte| acc.wrapping_shl(8) | u64::from(*byte)))
};
let Some(kind) = read_field(0, widths[0], 1) else { continue };
if kind == 0 || kind == 2 {
return true;
}
if kind == 1 && read_field(widths[0], widths[1], 0).is_some_and(|offset| offset != 0) {
return true;
}
}
}
false
}
}
}
pub(super) fn retained_bytes(&self) -> usize {
match &self.data {
XrefSectionData::Table => 0,
XrefSectionData::Stream { bytes, .. } => bytes.capacity(),
}
}
pub(super) fn lookup(&self, num: u32, file: &[u8]) -> Option<XrefEntry> {
let after = self.ranges.partition_point(|range| range.start <= num);
let range = self.ranges.get(after.checked_sub(1)?)?;
let end = u64::from(range.start).checked_add(u64::from(range.count))?;
if u64::from(num) >= end {
return None;
}
let row = usize::try_from(num.checked_sub(range.start)?).ok()?;
let offset = range.offset.checked_add(row.checked_mul(range.stride)?)?;
match &self.data {
XrefSectionData::Table => {
let mut lexer = Lexer::new(file, offset);
lexer.skip_ws();
let object_offset = lexer.token();
lexer.skip_ws();
let generation = lexer.token();
lexer.skip_ws();
let kind = lexer.token();
match kind {
b"n" => {
let object_offset = std::str::from_utf8(object_offset).ok()?.parse::<u64>().ok()?;
if object_offset == 0 {
return None;
}
let generation = std::str::from_utf8(generation).ok()?.parse::<u32>().unwrap_or(0).min(u16::MAX as u32) as u16;
Some(XrefEntry::InFile { offset: object_offset, generation })
}
b"f" => {
let generation = std::str::from_utf8(generation).ok()?.parse::<u32>().unwrap_or(0).min(u16::MAX as u32) as u16;
Some(XrefEntry::Free { next_generation: generation })
}
_ => None,
}
}
XrefSectionData::Stream { bytes, widths } => {
let row_width = widths.iter().sum::<usize>();
let end = offset.checked_add(row_width)?;
let row = bytes.get(offset..end)?;
let field = |from: usize, width: usize, default: u64| -> Option<u64> {
if width == 0 {
return Some(default);
}
let end = from.checked_add(width)?;
Some(row.get(from..end)?.iter().fold(0u64, |acc, byte| acc.wrapping_shl(8) | u64::from(*byte)))
};
let kind = field(0, widths[0], 1)?;
let a = field(widths[0], widths[1], 0)?;
let b = field(widths[0] + widths[1], widths[2], 0)?;
let entry = match kind {
0 => XrefEntry::Free { next_generation: b.min(u16::MAX as u64) as u16 },
1 => XrefEntry::InFile { offset: a, generation: b.min(u16::MAX as u64) as u16 },
2 => XrefEntry::InStream { stream: a.min(u32::MAX as u64) as u32, index: b.min(u32::MAX as u64) as u32 },
_ => return None,
};
(!matches!(entry, XrefEntry::InFile { offset: 0, .. })).then_some(entry)
}
}
}
pub(super) fn numbers(&self) -> impl Iterator<Item = u32> + '_ {
self.ranges.iter().flat_map(|range| {
let available = match &self.data {
XrefSectionData::Table => range.count as usize,
XrefSectionData::Stream { bytes, widths } => {
let row_width = widths.iter().sum::<usize>();
bytes.len().saturating_sub(range.offset).checked_div(row_width).unwrap_or(0)
}
};
(0..available.min(range.count as usize)).filter_map(move |i| u32::try_from(i).ok().and_then(|i| range.start.checked_add(i)))
})
}
pub(super) fn extend_numbers(&self, numbers: &mut Vec<u32>) {
numbers.extend(self.numbers());
}
}
impl XrefRevisionSection {
pub(super) fn lookup(&self, num: u32, file: &[u8]) -> Option<XrefEntry> {
let primary = self.primary.lookup(num, file);
let hybrid = self.hybrid.as_ref().and_then(|section| section.lookup(num, file));
match primary {
None | Some(XrefEntry::Free { .. }) => hybrid.or(primary),
entry => entry,
}
}
fn extend_numbers(&self, numbers: &mut Vec<u32>) {
self.primary.extend_numbers(numbers);
if let Some(hybrid) = &self.hybrid {
hybrid.extend_numbers(numbers);
}
}
}
impl XrefIndex {
pub(super) fn empty() -> Self {
Self::Materialized(Arc::new(BTreeMap::new()), Arc::default())
}
pub(super) fn lookup(&self, num: u32, file: &[u8]) -> Option<XrefEntry> {
match self {
Self::Compact { revisions, .. } => revisions.iter().find_map(|revision| revision.lookup(num, file)),
Self::Materialized(entries, _) => entries.get(&num).copied(),
}
}
pub(super) fn is_empty(&self) -> bool {
match self {
Self::Compact { has_entries, .. } => !has_entries,
Self::Materialized(entries, _) => entries.is_empty(),
}
}
/// Drop an entry for object number 4294967295, which no incremental `/Size` can cover.
/// Only a materialized index can hold one: the compact reader requires `max_num < /Size`.
pub(super) fn remove_exhausted(&mut self) -> bool {
match self {
Self::Compact { .. } => false,
Self::Materialized(entries, _) => Arc::make_mut(entries).remove(&u32::MAX).is_some(),
}
}
/// Every entry, newest section first, as the materialized reader keeps them.
pub(super) fn to_map(&self, file: &[u8]) -> BTreeMap<u32, XrefEntry> {
match self {
Self::Materialized(entries, _) => entries.as_ref().clone(),
Self::Compact { revisions, .. } => {
let mut numbers = Vec::new();
for revision in revisions {
revision.extend_numbers(&mut numbers);
}
numbers.into_iter().filter_map(|num| self.lookup(num, file).map(|entry| (num, entry))).collect()
}
}
}
pub(super) fn max_num(&self) -> u32 {
match self {
Self::Compact { max_num, .. } => *max_num,
Self::Materialized(entries, _) => entries.keys().next_back().copied().unwrap_or(0),
}
}
pub(super) fn object_numbers(&self, file: &[u8]) -> Vec<u32> {
match self {
Self::Materialized(entries, _) => {
entries.iter().filter_map(|(&num, entry)| (!matches!(entry, XrefEntry::Free { .. })).then_some(num)).collect()
}
Self::Compact { revisions, .. } => {
if let [revision] = revisions.as_slice()
&& revision.hybrid.is_none()
{
let mut numbers = Vec::with_capacity(revision.primary.row_count);
revision.primary.extend_numbers(&mut numbers);
numbers.retain(|num| self.lookup(*num, file).is_some_and(|entry| !matches!(entry, XrefEntry::Free { .. })));
return numbers;
}
let mut numbers = Vec::new();
for revision in revisions {
revision.extend_numbers(&mut numbers);
}
numbers.sort_unstable();
numbers.dedup();
numbers.retain(|num| self.lookup(*num, file).is_some_and(|entry| !matches!(entry, XrefEntry::Free { .. })));
numbers
}
}
}
}
/// One cross-reference section of the file (one per save; the first is the original).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Revision {
/// Offset of the xref section (`startxref` value).
pub xref_offset: u64,
/// `true` if the section is a cross-reference stream.
pub is_stream: bool,
/// `true` if `xref_offset` counts from the `%PDF-` header rather than from the start of the
/// file. The two differ only in a file with bytes before its header, and an update appended
/// to such a file must count the same way as the section it points back to.
pub header_relative: bool,
}
+833
View File
@@ -0,0 +1,833 @@
//! The PDF object model (ISO 32000-2 §7.3).
//!
//! Dictionaries keep their key order so rewritten objects stay recognisable in diffs and
//! byte-stable when re-serialized. Streams keep their *encoded* bytes; decoding is on demand.
use crate::filters::{Filter, Params};
use crate::cos::{Bytes, CosError};
/// An indirect reference: object number and generation.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct ObjRef {
pub num: u32,
pub generation: u16,
}
impl ObjRef {
pub const fn new(num: u32, generation: u16) -> Self {
Self { num, generation }
}
}
impl std::fmt::Display for ObjRef {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{} {} R", self.num, self.generation)
}
}
/// A string object. `hex` records how it was written, so it is re-emitted the same way.
#[derive(Clone, Debug, PartialEq)]
pub struct PdfString {
pub bytes: Vec<u8>,
pub hex: bool,
}
impl PdfString {
pub fn literal(bytes: impl Into<Vec<u8>>) -> Self {
Self { bytes: bytes.into(), hex: false }
}
/// A text string (ISO 32000-2 §7.9.2.2): PDFDocEncoding when possible, else UTF-16BE with BOM.
pub fn text(s: &str) -> Self {
if s.chars().all(|c| (' '..='~').contains(&c) || c == '\n' || c == '\r' || c == '\t') {
return Self::literal(s.as_bytes().to_vec());
}
let mut bytes = vec![0xFE, 0xFF];
for u in s.encode_utf16() {
bytes.extend_from_slice(&u.to_be_bytes());
}
Self { bytes, hex: false }
}
/// Decode a text string: UTF-16BE (BOM), UTF-8 (BOM, PDF 2.0) or PDFDocEncoding.
///
/// Lenient beyond the spec, as readers are: invalid sequences after a BOM decode lossily
/// instead of dropping the string, a BOM-less string whose bytes are valid UTF-8 with
/// non-ASCII bytes (written by some producers) is read as UTF-8 rather than as
/// PDFDocEncoding mojibake, and stray leading U+FEFF (doubled BOMs) are removed.
pub fn to_text(&self) -> String {
self.text_chars().collect()
}
/// The characters [`PdfString::to_text`] decodes, one at a time, without building a string.
pub fn text_chars(&self) -> TextChars<'_> {
let decode = match self.bytes.as_slice() {
[0xFE, 0xFF, rest @ ..] => {
let units: fn(&[u8; 2]) -> u16 = |c| u16::from_be_bytes(*c);
Decode::Utf16(char::decode_utf16(rest.as_chunks::<2>().0.iter().map(units)))
}
[0xEF, 0xBB, 0xBF, rest @ ..] => Decode::Utf8Lossy { chunks: rest.utf8_chunks(), valid: "".chars(), replace: false },
b => match std::str::from_utf8(b) {
// Pure ASCII is identical either way; only non-ASCII UTF-8 needs the switch.
Ok(utf8) if !b.is_ascii() => Decode::Utf8(utf8.chars()),
_ => Decode::PdfDoc(b.iter()),
},
};
TextChars { decode, leading: true }
}
/// Whether the string decodes to `s` ([`PdfString::to_text`]), compared without allocating.
pub fn text_eq(&self, s: &str) -> bool {
self.text_chars().eq(s.chars())
}
}
/// The characters of a text string: [`PdfString::text_chars`].
pub struct TextChars<'a> {
decode: Decode<'a>,
/// Still at the start, where stray U+FEFF (doubled BOMs) are dropped.
leading: bool,
}
/// The UTF-16BE code units after a BOM (an odd last byte is left out).
type Utf16Units<'a> = std::iter::Map<std::slice::Iter<'a, [u8; 2]>, fn(&[u8; 2]) -> u16>;
enum Decode<'a> {
Utf16(std::char::DecodeUtf16<Utf16Units<'a>>),
/// After a UTF-8 BOM: each invalid sequence is one U+FFFD, as `String::from_utf8_lossy` has it.
Utf8Lossy {
chunks: std::str::Utf8Chunks<'a>,
valid: std::str::Chars<'a>,
replace: bool,
},
Utf8(std::str::Chars<'a>),
PdfDoc(std::slice::Iter<'a, u8>),
}
impl Iterator for Decode<'_> {
type Item = char;
fn next(&mut self) -> Option<char> {
match self {
Decode::Utf16(units) => units.next().map(|u| u.unwrap_or(char::REPLACEMENT_CHARACTER)),
Decode::Utf8Lossy { chunks, valid, replace } => loop {
if let Some(c) = valid.next() {
return Some(c);
}
if *replace {
*replace = false;
return Some(char::REPLACEMENT_CHARACTER);
}
let chunk = chunks.next()?;
*valid = chunk.valid().chars();
*replace = !chunk.invalid().is_empty();
},
Decode::Utf8(chars) => chars.next(),
Decode::PdfDoc(bytes) => bytes.next().map(|&c| crate::crypt::pdfdoc_char(c)),
}
}
}
impl Iterator for TextChars<'_> {
type Item = char;
fn next(&mut self) -> Option<char> {
loop {
let c = self.decode.next()?;
if self.leading && c == '\u{FEFF}' {
continue;
}
self.leading = false;
return Some(c);
}
}
}
/// A name object (without the leading `/`, with `#xx` escapes already decoded).
pub type Name = Vec<u8>;
#[cfg(test)]
thread_local! {
/// Key comparisons and index lookups made by [`Dict`] on this thread.
static KEY_COMPARES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
}
/// Entries from which a [`Dict`] keeps a key index. Smaller dictionaries (nearly all of them)
/// search their entries; a parsed file controls the key count, and a linear search per
/// inserted key would make a dictionary of many distinct keys quadratic to build.
const INDEX_FROM: usize = 32;
/// An ordered dictionary.
#[derive(Clone, Default)]
pub struct Dict {
entries: Vec<(Name, Object)>,
/// Key to position in `entries`, kept once the dictionary reaches [`INDEX_FROM`] entries.
/// Boxed so that every `Dict`, and so every `Object`, grows by one pointer, not by a map.
#[allow(clippy::box_collection)]
index: Option<Box<std::collections::HashMap<Name, usize>>>,
}
impl PartialEq for Dict {
fn eq(&self, other: &Self) -> bool {
self.entries == other.entries
}
}
impl std::fmt::Debug for Dict {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Dict").field("entries", &self.entries).finish()
}
}
impl Dict {
pub fn new() -> Self {
Self::default()
}
fn position(&self, key: &[u8]) -> Option<usize> {
if let Some(index) = &self.index {
#[cfg(test)]
KEY_COMPARES.with(|c| c.set(c.get() + 1));
return index.get(key).copied();
}
self.entries.iter().position(|(k, _)| {
#[cfg(test)]
KEY_COMPARES.with(|c| c.set(c.get() + 1));
k == key
})
}
pub fn get(&self, key: &[u8]) -> Option<&Object> {
self.position(key).and_then(|i| self.entries.get(i)).map(|(_, v)| v)
}
pub fn get_mut(&mut self, key: &[u8]) -> Option<&mut Object> {
let i = self.position(key)?;
self.entries.get_mut(i).map(|(_, v)| v)
}
/// Insert or replace, keeping the position of an existing key.
pub fn set(&mut self, key: impl Into<Name>, value: impl Into<Object>) {
let key = key.into();
let value = value.into();
match self.position(&key).and_then(|i| self.entries.get_mut(i)) {
Some((_, v)) => *v = value,
None => {
let at = self.entries.len();
if let Some(index) = &mut self.index {
index.insert(key.clone(), at);
} else if at + 1 >= INDEX_FROM {
let index = self.entries.iter().enumerate().map(|(i, (k, _))| (k.clone(), i)).chain([(key.clone(), at)]).collect();
self.index = Some(Box::new(index));
}
self.entries.push((key, value));
}
}
}
pub fn remove(&mut self, key: &[u8]) -> Option<Object> {
let i = self.position(key)?;
if let Some(index) = &mut self.index {
index.remove(key);
index.values_mut().filter(|at| **at > i).for_each(|at| *at -= 1);
}
Some(self.entries.remove(i).1)
}
pub fn contains(&self, key: &[u8]) -> bool {
self.get(key).is_some()
}
pub fn iter(&self) -> impl Iterator<Item = (&Name, &Object)> {
self.entries.iter().map(|(k, v)| (k, v))
}
pub fn len(&self) -> usize {
self.entries.len()
}
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
pub fn name(&self, key: &[u8]) -> Option<&[u8]> {
self.get(key).and_then(Object::as_name)
}
pub fn int(&self, key: &[u8]) -> Option<i64> {
self.get(key).and_then(Object::as_int)
}
pub fn reference(&self, key: &[u8]) -> Option<ObjRef> {
self.get(key).and_then(Object::as_ref)
}
}
impl FromIterator<(Name, Object)> for Dict {
fn from_iter<T: IntoIterator<Item = (Name, Object)>>(iter: T) -> Self {
let mut d = Dict::new();
for (k, v) in iter {
d.set(k, v);
}
d
}
}
/// A stream: dictionary plus its data *as stored* (still filtered).
#[derive(Clone, Debug, PartialEq)]
pub struct Stream {
pub dict: Dict,
pub raw: Bytes,
}
/// Upper bound for a single decoded stream (decompression-bomb defence).
pub const MAX_DECODED: usize = 1 << 30;
/// The most an image's `/JBIG2Globals` stream may decode to: the symbol and pattern dictionaries an
/// encoder shares between pages, themselves coded, so far less than any stream may decode to
/// ([`MAX_DECODED`]). They are decoded before the image's JBIG2 page is read, so the page's size
/// can't bound them: they have a limit of their own.
pub const MAX_JBIG2_GLOBALS: usize = 32 << 20;
impl Stream {
/// A new stream from already-encoded bytes (`/Length` is set at write time).
pub fn from_raw(dict: Dict, raw: Vec<u8>) -> Self {
Self { dict, raw: raw.into() }
}
/// A new stream holding `data` compressed with Flate.
pub fn flate(mut dict: Dict, data: &[u8]) -> Self {
dict.set(b"Filter".to_vec(), Object::Name(b"FlateDecode".to_vec()));
dict.remove(b"DecodeParms");
Self { dict, raw: crate::filters::encode_flate(data).into() }
}
/// The filter chain declared in the dictionary.
pub fn filters(&self) -> Vec<(Filter, Params)> {
let names: Vec<&[u8]> = match self.dict.get(b"Filter") {
Some(Object::Name(n)) => vec![n],
Some(Object::Array(a)) => a.iter().filter_map(Object::as_name).collect(),
_ => Vec::new(),
};
let parms: Vec<Option<&Dict>> = match self.dict.get(b"DecodeParms") {
Some(Object::Dict(d)) => vec![Some(d)],
Some(Object::Array(a)) => a.iter().map(Object::as_dict).collect(),
_ => Vec::new(),
};
names
.into_iter()
.enumerate()
// `/Crypt` is applied by the security handler when the object is loaded (§7.4.10).
.filter(|(_, n)| *n != b"Crypt")
.map(|(i, n)| {
let p = parms.get(i).copied().flatten();
let get = |k: &[u8], d: i64| p.and_then(|p| p.int(k)).unwrap_or(d);
let params = Params {
predictor: get(b"Predictor", 1),
colors: get(b"Colors", 1),
bits_per_component: get(b"BitsPerComponent", 8),
columns: get(b"Columns", 1),
early_change: get(b"EarlyChange", 1),
};
(Filter::from_name(n), params)
})
.collect()
}
/// [`Stream::filters`], or an error when `/Filter` is present but is not a name or an
/// array of names (an indirect reference, say): the data is not unfiltered, so it must not
/// read as decoded.
pub(crate) fn readable_filters(&self) -> Result<Vec<(Filter, Params)>, crate::filters::FilterError> {
let readable = match self.dict.get(b"Filter") {
None | Some(Object::Null | Object::Name(_)) => true,
Some(Object::Array(a)) => a.iter().all(|f| f.as_name().is_some()),
Some(_) => false,
};
if !readable {
return Err(crate::filters::FilterError::Unsupported("/Filter is not a name or an array of names".into()));
}
Ok(self.filters())
}
/// Decoded data, tolerating truncated or corrupt encodings the way viewers do (the data
/// decoded before the damage is returned). Decoding past
/// [`crate::cos::budget::MAX_PAGE_DECODED`] (256 MiB), the most the renderer reads from one
/// stream, is an error. Streams with image codecs fail with
/// `CosError::Filter` (keep them encoded). Use `decoded_strict` to detect damage.
pub fn decoded(&self) -> Result<Vec<u8>, CosError> {
self.decoded_within(crate::cos::budget::MAX_PAGE_DECODED)
}
/// Like [`Stream::decoded`], for a stream whose decoded size has a tighter bound than
/// [`MAX_DECODED`]: decoding past `max` bytes is an error.
pub fn decoded_within(&self, max: usize) -> Result<Vec<u8>, CosError> {
let decode =
|chain: &[(Filter, Params)], data: &[u8], max| crate::filters::decode_tolerant(chain, data, max).map(|(v, _)| (v, 0)).map_err(|e| (e, 0));
self.decode_chain(max, decode).map(|(v, _)| v).map_err(|(e, _)| CosError::Filter(e.to_string()))
}
/// [`Stream::decoded_within`] for a caller charging an allowance: the filter error is kept
/// typed, so a caller can tell a spent limit from corrupt data, `max` bounds all filter
/// stages together, and the result gives whether the data stopped at damage (it is then what
/// decoded before it) and the bytes the stages decoded (the work done, on success and on
/// failure).
pub(crate) fn decoded_within_counted(&self, max: usize) -> Result<(Vec<u8>, bool, usize), (crate::filters::FilterError, usize)> {
let mut damaged = false;
let decode = |chain: &[(Filter, Params)], data: &[u8], max| {
crate::filters::decode_tolerant_counted(chain, data, max).map(|(v, partial, spent)| {
damaged = partial;
(v, spent)
})
};
self.decode_chain(max, decode).map(|(v, spent)| (v, damaged, spent))
}
/// [`Stream::decoded_within_counted`] with every filter stage also held to `each`, as
/// [`Stream::decoded_within`] holds it, while `max` bounds the stages together
/// ([`crate::filters::decode_tolerant_counted_each`]). Without filters the data is held to
/// both.
pub(crate) fn decoded_within_each_counted(
&self,
each: usize,
max: usize,
) -> Result<(Vec<u8>, bool, usize), (crate::filters::FilterError, usize)> {
let mut damaged = false;
let decode = |chain: &[(Filter, Params)], data: &[u8], max| {
crate::filters::decode_tolerant_counted_each(chain, data, each.min(MAX_DECODED), max).map(|(v, partial, spent)| {
damaged = partial;
(v, spent)
})
};
let max = if self.readable_filters().is_ok_and(|chain| chain.is_empty()) { max.min(each) } else { max };
self.decode_chain(max, decode).map(|(v, spent)| (v, damaged, spent))
}
/// [`Stream::decoded_within`] spending from `work`, what an operation may still do across a
/// document's images ([`crate::filters::decode_tolerant_within`]).
pub(crate) fn decoded_spending(&self, max: usize, work: &mut u64) -> Result<Vec<u8>, CosError> {
let filter = |e: crate::filters::FilterError| CosError::Filter(e.to_string());
let chain = self.readable_filters().map_err(filter)?;
crate::filters::decode_tolerant_within(&chain, &self.raw, max.min(MAX_DECODED), work).map_err(filter)
}
/// [`Stream::decoded_spending`] for the samples of an image `rows` high (its `/Height`):
/// [`crate::filters::decode_image_tolerant_within`].
pub(crate) fn decoded_image_rows_spending(&self, max: usize, rows: u32, work: &mut u64) -> Result<Vec<u8>, CosError> {
let filter = |e: crate::filters::FilterError| CosError::Filter(e.to_string());
let chain = self.readable_filters().map_err(filter)?;
crate::filters::decode_image_tolerant_within(&chain, &self.raw, max.min(MAX_DECODED), rows, work).map_err(filter)
}
/// The stream decoded through its filters by `decode`, or copied when it has none, within
/// `max` and [`MAX_DECODED`].
fn decode_chain(
&self,
max: usize,
decode: impl FnOnce(&[(Filter, Params)], &[u8], usize) -> Result<(Vec<u8>, usize), (crate::filters::FilterError, usize)>,
) -> Result<(Vec<u8>, usize), (crate::filters::FilterError, usize)> {
let max = max.min(MAX_DECODED);
let chain = self.readable_filters().map_err(|e| (e, 0))?;
if chain.is_empty() {
return self
.raw_within_typed(max)
.map(|v| {
let n = v.len();
(v, n)
})
.map_err(|e| (e, 0));
}
decode(&chain, &self.raw, max)
}
/// Decoded data, bounded by [`MAX_DECODED`]; any corruption is an error.
pub fn decoded_strict(&self) -> Result<Vec<u8>, CosError> {
self.decoded_strict_within(MAX_DECODED)
}
/// Like [`Stream::decoded_strict`], bounded by `max` (and [`MAX_DECODED`]): for data that
/// will be copied elsewhere, where a damaged stream must not pass for a shorter one.
pub fn decoded_strict_within(&self, max: usize) -> Result<Vec<u8>, CosError> {
let decode = |chain: &[(Filter, Params)], data: &[u8], max| crate::filters::decode(chain, data, max).map(|v| (v, 0)).map_err(|e| (e, 0));
self.decode_chain(max, decode).map(|(v, _)| v).map_err(|(e, _)| CosError::Filter(e.to_string()))
}
/// [`Stream::decoded_strict`] bounded by `max` and counted like
/// [`Stream::decoded_within_counted`].
pub(crate) fn decoded_strict_within_counted(&self, max: usize) -> Result<(Vec<u8>, usize), (crate::filters::FilterError, usize)> {
self.decode_chain(max, crate::filters::decode_counted)
}
/// Data after the first filter alone, bounded by `max`: the rest of the chain stays encoded.
/// Any corruption in that first filter is an error.
pub fn decoded_first_filter_strict(&self, max: usize) -> Result<Vec<u8>, CosError> {
let mut chain = self.readable_filters().map_err(|e| CosError::Filter(e.to_string()))?;
chain.truncate(1);
if chain.is_empty() {
return self.raw_within(max);
}
crate::filters::decode(&chain, &self.raw, max.min(MAX_DECODED)).map_err(|e| CosError::Filter(e.to_string()))
}
fn raw_within(&self, max: usize) -> Result<Vec<u8>, CosError> {
self.raw_within_typed(max).map_err(|e| CosError::Filter(e.to_string()))
}
fn raw_within_typed(&self, max: usize) -> Result<Vec<u8>, crate::filters::FilterError> {
let max = max.min(MAX_DECODED);
// An empty decoder chain still produces an owned output buffer: check its limit
// before cloning, including streams whose /Crypt filter was applied on load.
if self.raw.len() > max {
return Err(crate::filters::FilterError::LimitExceeded(max));
}
Ok(self.raw.to_vec())
}
}
#[derive(Clone, Debug, PartialEq)]
pub enum Object {
Null,
Bool(bool),
Int(i64),
Real(f64),
String(PdfString),
Name(Name),
Array(Vec<Object>),
Dict(Dict),
Stream(Stream),
Ref(ObjRef),
}
impl Object {
pub fn name(n: &str) -> Self {
Object::Name(n.as_bytes().to_vec())
}
pub fn as_int(&self) -> Option<i64> {
match self {
Object::Int(i) => Some(*i),
Object::Real(r) if r.fract() == 0.0 && r.is_finite() => Some(*r as i64),
_ => None,
}
}
pub fn as_f64(&self) -> Option<f64> {
match self {
Object::Int(i) => Some(*i as f64),
Object::Real(r) => Some(*r),
_ => None,
}
}
pub fn as_name(&self) -> Option<&[u8]> {
match self {
Object::Name(n) => Some(n),
_ => None,
}
}
pub fn as_ref(&self) -> Option<ObjRef> {
match self {
Object::Ref(r) => Some(*r),
_ => None,
}
}
pub fn as_dict(&self) -> Option<&Dict> {
match self {
Object::Dict(d) => Some(d),
Object::Stream(s) => Some(&s.dict),
_ => None,
}
}
pub fn as_dict_mut(&mut self) -> Option<&mut Dict> {
match self {
Object::Dict(d) => Some(d),
Object::Stream(s) => Some(&mut s.dict),
_ => None,
}
}
pub fn as_array(&self) -> Option<&Vec<Object>> {
match self {
Object::Array(a) => Some(a),
_ => None,
}
}
pub fn as_string(&self) -> Option<&PdfString> {
match self {
Object::String(s) => Some(s),
_ => None,
}
}
}
impl From<i64> for Object {
fn from(v: i64) -> Self {
Object::Int(v)
}
}
impl From<i32> for Object {
fn from(v: i32) -> Self {
Object::Int(v as i64)
}
}
impl From<f64> for Object {
fn from(v: f64) -> Self {
Object::Real(v)
}
}
impl From<bool> for Object {
fn from(v: bool) -> Self {
Object::Bool(v)
}
}
impl From<ObjRef> for Object {
fn from(v: ObjRef) -> Self {
Object::Ref(v)
}
}
impl From<Dict> for Object {
fn from(v: Dict) -> Self {
Object::Dict(v)
}
}
impl From<Vec<Object>> for Object {
fn from(v: Vec<Object>) -> Self {
Object::Array(v)
}
}
impl From<PdfString> for Object {
fn from(v: PdfString) -> Self {
Object::String(v)
}
}
impl From<Stream> for Object {
fn from(v: Stream) -> Self {
Object::Stream(v)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_filter_that_is_not_a_direct_name_is_not_decoded_as_absent() {
let stream = |filter: Object| {
let mut d = Dict::new();
d.set(b"Filter".to_vec(), filter);
Stream::from_raw(d, b"48 69>".to_vec())
};
let hex = Object::Name(b"ASCIIHexDecode".to_vec());
assert_eq!(stream(hex.clone()).decoded_strict().ok(), Some(b"Hi".to_vec()));
assert_eq!(stream(Object::Array(vec![hex.clone()])).decoded().ok(), Some(b"Hi".to_vec()));
assert_eq!(stream(Object::Null).decoded_strict().ok(), Some(b"48 69>".to_vec()));
for filter in [
Object::Ref(ObjRef::new(5, 0)),
Object::Array(vec![Object::Ref(ObjRef::new(5, 0))]),
Object::Int(3),
Object::Array(vec![hex, Object::Int(1)]),
] {
let s = stream(filter.clone());
assert!(s.decoded_strict().is_err(), "{filter:?}");
assert!(s.decoded().is_err(), "{filter:?}");
assert!(s.decoded_first_filter_strict(1 << 20).is_err(), "{filter:?}");
}
}
#[test]
fn parsing_many_distinct_keys_is_not_quadratic() {
let n = 20_000;
let body: String = (0..n).map(|i| format!("/K{i} {i} ")).collect();
let text = format!("<< {body}/K7 true >>");
KEY_COMPARES.with(|c| c.set(0));
let Ok(Object::Dict(d)) = crate::cos::parser::Lexer::new(text.as_bytes(), 0).object() else { panic!("not a dict") };
let compares = KEY_COMPARES.with(|c| c.get());
assert!(compares < 8 * n, "{compares} key comparisons for {n} keys");
// Order and replace-in-place semantics stay those of a plain ordered dictionary.
assert_eq!(d.len(), n);
assert_eq!(d.iter().next().map(|(k, _)| k.as_slice()), Some(&b"K0"[..]));
assert_eq!(d.get(b"K7"), Some(&Object::Bool(true)));
assert_eq!(d.iter().nth(7).map(|(_, v)| v), Some(&Object::Bool(true)));
let mut d = d;
assert_eq!(d.remove(b"K3"), Some(Object::Int(3)));
assert_eq!(d.get(b"K4"), Some(&Object::Int(4)));
assert_eq!(d.iter().nth(3).map(|(k, _)| k.as_slice()), Some(&b"K4"[..]));
d.set(b"K3".to_vec(), Object::Null);
assert_eq!(d.iter().last().map(|(k, _)| k.as_slice()), Some(&b"K3"[..]));
assert!(d.get(b"K20000").is_none());
}
#[test]
fn text_strings_round_trip() {
for s in ["Plain ASCII", "Café — naïve", "日本語", "emoji 📄"] {
assert_eq!(PdfString::text(s).to_text(), s);
}
assert!(!PdfString::text("abc").bytes.starts_with(&[0xFE, 0xFF]));
}
#[test]
fn text_string_encodings() {
// UTF-16BE with BOM.
let mut utf16 = vec![0xFE, 0xFF];
utf16.extend("第一章 概述".encode_utf16().flat_map(u16::to_be_bytes));
assert_eq!(PdfString::literal(utf16).to_text(), "第一章 概述");
// UTF-8 with BOM (PDF 2.0): the BOM is not part of the text.
let mut utf8 = vec![0xEF, 0xBB, 0xBF];
utf8.extend_from_slice("第一章".as_bytes());
assert_eq!(PdfString::literal(utf8).to_text(), "第一章");
// Invalid UTF-8 after the BOM decodes lossily instead of vanishing.
assert_eq!(PdfString::literal(vec![0xEF, 0xBB, 0xBF, b'A', 0xFF, b'B']).to_text(), "A\u{FFFD}B");
// BOM-less raw UTF-8 from producers that skip the BOM.
assert_eq!(PdfString::literal("目录".as_bytes()).to_text(), "目录");
// PDFDocEncoding Latin-1 (not valid UTF-8) stays PDFDocEncoding.
assert_eq!(PdfString::literal(vec![b'C', b'a', b'f', 0xE9]).to_text(), "Café");
// Doubled BOMs leave no U+FEFF behind.
assert_eq!(PdfString::literal(vec![0xFE, 0xFF, 0xFE, 0xFF, 0x00, b'A']).to_text(), "A");
assert_eq!(PdfString::literal(vec![0xEF, 0xBB, 0xBF, 0xEF, 0xBB, 0xBF, b'A']).to_text(), "A");
// Odd trailing byte and empty strings never panic.
assert_eq!(PdfString::literal(vec![0xFE, 0xFF, 0x00, b'A', 0x00]).to_text(), "A");
assert_eq!(PdfString::literal(vec![0xFE, 0xFF]).to_text(), "");
assert_eq!(PdfString::literal(Vec::new()).to_text(), "");
}
#[test]
fn text_eq_compares_what_to_text_decodes() {
let utf16 = |s: &str| [0xFE, 0xFF].into_iter().chain(s.encode_utf16().flat_map(u16::to_be_bytes)).collect::<Vec<u8>>();
// Each invalid UTF-8 sequence is one U+FFFD, as String::from_utf8_lossy has it.
let lossy = [0xEF, 0xBB, 0xBF, 0xC3, 0x28, b'x', 0xF0, 0x9F, 0x92, 0xFF];
assert_eq!(PdfString::literal(lossy.to_vec()).to_text(), String::from_utf8_lossy(&lossy[3..]));
let cases: Vec<Vec<u8>> = vec![
Vec::new(),
b"Plain".to_vec(),
vec![b'C', b'a', b'f', 0xE9, 0x18],
"目录".as_bytes().to_vec(),
utf16("第一章"),
vec![0xFE, 0xFF, 0xD8, 0x00, 0x00, b'A'],
vec![0xFE, 0xFF, 0xFE, 0xFF, 0x00, b'A', 0xFE, 0xFF],
lossy.to_vec(),
];
for bytes in cases {
let s = PdfString::literal(bytes);
let text = s.to_text();
assert!(s.text_eq(&text), "{text:?}");
assert!(!s.text_eq(&format!("{text}x")), "{text:?}");
assert!(text.is_empty() || !s.text_eq(&text[..text.len() - text.chars().last().map_or(0, char::len_utf8)]), "{text:?}");
}
// A U+FEFF after the first character is text and stays.
assert!(PdfString::literal(utf16("A\u{FEFF}")).text_eq("A\u{FEFF}"));
}
#[test]
fn pdfdoc_encoding_high_range() {
assert_eq!(PdfString::literal(vec![0x84, 0x92, 0xA0, b'A']).to_text(), "—™€A");
}
#[test]
fn dict_preserves_order_and_replaces_in_place() {
let mut d = Dict::new();
d.set(b"B".to_vec(), 1);
d.set(b"A".to_vec(), 2);
d.set(b"B".to_vec(), 3);
let keys: Vec<_> = d.iter().map(|(k, _)| k.clone()).collect();
assert_eq!(keys, vec![b"B".to_vec(), b"A".to_vec()]);
assert_eq!(d.int(b"B"), Some(3));
}
#[test]
fn raw_streams_refuse_decoding_past_a_caller_limit() {
let stream = Stream::from_raw(Dict::new(), b"abc".to_vec());
for max in [0, 2] {
assert!(matches!(stream.decoded_within(max), Err(CosError::Filter(_))));
}
}
#[test]
fn strict_decoding_is_bounded_by_the_callers_remaining_allowance() {
let stream = Stream::flate(Dict::new(), &vec![0u8; 1 << 20]);
assert!(matches!(stream.decoded_strict_within(1000), Err(CosError::Filter(_))));
assert_eq!(stream.decoded_strict_within(1 << 20).map(|v| v.len()).ok(), Some(1 << 20));
}
#[test]
fn empty_filter_arrays_refuse_decoding_past_a_caller_limit() {
let mut dict = Dict::new();
dict.set(b"Filter".to_vec(), Object::Array(Vec::new()));
let stream = Stream::from_raw(dict, b"abc".to_vec());
assert!(matches!(stream.decoded_within(2), Err(CosError::Filter(_))));
}
#[test]
fn crypt_only_streams_refuse_decoding_past_a_caller_limit() {
let mut dict = Dict::new();
dict.set(b"Filter".to_vec(), Object::name("Crypt"));
let mut parms = Dict::new();
parms.set(b"Name".to_vec(), Object::name("Identity"));
dict.set(b"DecodeParms".to_vec(), Object::Dict(parms));
let stream = Stream::from_raw(dict, b"abc".to_vec());
assert!(matches!(stream.decoded_within(2), Err(CosError::Filter(_))));
}
#[test]
fn identity_streams_preserve_bytes_within_a_caller_limit() {
let mut array = Dict::new();
array.set(b"Filter".to_vec(), Object::Array(Vec::new()));
let mut crypt = Dict::new();
crypt.set(b"Filter".to_vec(), Object::name("Crypt"));
for dict in [Dict::new(), array, crypt] {
let stream = Stream::from_raw(dict.clone(), b"abc".to_vec());
assert_eq!(stream.decoded_within(3).unwrap(), b"abc");
assert_eq!(stream.decoded_within(usize::MAX).unwrap(), b"abc");
assert_eq!(stream.decoded().unwrap(), b"abc");
assert_eq!(stream.decoded_strict().unwrap(), b"abc");
assert!(Stream::from_raw(dict, Vec::new()).decoded_within(0).unwrap().is_empty());
}
}
#[test]
fn filtered_streams_preserve_the_caller_limit() {
let stream = Stream::flate(Dict::new(), b"abc");
assert!(matches!(stream.decoded_within(2), Err(CosError::Filter(_))));
assert_eq!(stream.decoded_within(3).unwrap(), b"abc");
assert_eq!(stream.decoded_strict().unwrap(), b"abc");
}
#[test]
fn raw_flate_fallback_streams_preserve_the_caller_limit() {
// Raw deflate with a stored byte and a final empty block. 78 01 also
// passes the zlib header check, so decoding must retry the whole input.
let mut dict = Dict::new();
dict.set(b"Filter".to_vec(), Object::name("FlateDecode"));
let stream = Stream::from_raw(dict, vec![0x78, 0x01, 0x00, 0xfe, 0xff, b'A', 0x01, 0x00, 0x00, 0xff, 0xff]);
let result = stream.decoded_within(0);
assert!(matches!(&result, Err(CosError::Filter(detail)) if detail.contains("exceeds the limit of 0 bytes")), "{result:?}");
assert_eq!(stream.decoded_within(1).unwrap(), b"A");
assert_eq!(stream.decoded_strict().unwrap(), b"A");
}
#[test]
fn decoded_stops_at_renderer_cap_of_256_mib() {
let data = vec![0u8; (256 << 20) + 1];
let stream = Stream::flate(Dict::new(), &data);
assert!(matches!(stream.decoded(), Err(CosError::Filter(_))));
}
#[test]
fn first_filter_decodes_only_the_outer_wrapper() {
let data = "0 0 m 100 100 l S\n".repeat(64).into_bytes();
let inner = Stream::flate(Dict::new(), &data).raw.to_vec();
for (outer, name, corrupt) in [(Filter::Ascii85, "ASCII85Decode", &b"{~>"[..]), (Filter::AsciiHex, "ASCIIHexDecode", &b"G0>"[..])] {
let encoded = crate::filters::encode(&outer, &Params::default(), &inner).unwrap();
let mut dict = Dict::new();
dict.set(b"Filter".to_vec(), Object::Array(vec![Object::name(name), Object::name("FlateDecode")]));
let stream = Stream::from_raw(dict.clone(), encoded);
assert_eq!(stream.decoded_first_filter_strict(usize::MAX).unwrap(), inner);
assert_eq!(stream.decoded_strict().unwrap(), data);
assert!(matches!(stream.decoded_first_filter_strict(inner.len() - 1), Err(CosError::Filter(_))));
assert!(matches!(Stream::from_raw(dict, corrupt.to_vec()).decoded_first_filter_strict(usize::MAX), Err(CosError::Filter(_))));
}
}
}
+96
View File
@@ -0,0 +1,96 @@
//! Bounded numbering for page labels (ISO 32000-2 §12.4.2).
//! Pure helpers shared by inspection and editing; no object graph or renderer dependency.
/// A complete custom label, including its prefix, may contain this many UTF-8 bytes.
pub const MAX_LABEL_BYTES: usize = 1024;
/// Raw prefixes are bounded before decoding; this admits a full UTF-16BE ASCII label.
pub const MAX_PREFIX_BYTES: usize = 2 * MAX_LABEL_BYTES + 2;
/// Sum of the UTF-8 bytes of custom labels for one document.
pub const MAX_LABEL_TOTAL_BYTES: usize = 4 << 20;
/// Sum of the UTF-8 bytes of the decoded prefixes of one `/PageLabels` tree. Each range decodes
/// its own prefix, and ranges may share one prefix object, so without this a small file could ask
/// for a copy per range: [`MAX_LABEL_TREE_WORK`] ranges naming one 1,024-byte prefix would hold
/// about 200 MiB. Real documents stay far below it.
pub const MAX_PREFIX_TOTAL_BYTES: usize = 4 << 20;
/// The most number-tree entries (nodes, kids and range pairs) read from one `/PageLabels`
/// tree. Scaled for real documents: one range per page on a ten-thousand-page PDF happens
/// (prefix labels; #307's file has 9,156 pages), so the cap is twenty times that while still
/// bounding hostile trees.
pub const MAX_LABEL_TREE_WORK: usize = 200_000;
/// Maximum nesting of a label number tree.
pub const MAX_LABEL_TREE_DEPTH: usize = 32;
/// Lowercase Roman numerals, or `None` if the output exceeds the remaining label budget.
/// The budget is capped at [`MAX_LABEL_BYTES`], including when `n` is hostile.
pub fn roman(mut n: u64, max_bytes: usize) -> Option<String> {
const PARTS: [(u64, &str); 13] = [
(1000, "m"),
(900, "cm"),
(500, "d"),
(400, "cd"),
(100, "c"),
(90, "xc"),
(50, "l"),
(40, "xl"),
(10, "x"),
(9, "ix"),
(5, "v"),
(4, "iv"),
(1, "i"),
];
let max_bytes = max_bytes.min(MAX_LABEL_BYTES);
let mut out = String::new();
for (value, part) in PARTS {
let count = usize::try_from(n / value).ok()?;
let bytes = count.checked_mul(part.len())?;
if bytes > max_bytes.saturating_sub(out.len()) {
return None;
}
// The quotient is checked against the tiny byte budget before repeating.
out.push_str(&part.repeat(count));
n %= value;
}
Some(out)
}
/// a..z, aa..zz, aaa.. (the repeated-letter numbering specified by PDF).
/// Returns `None` instead of truncating a label that exceeds the remaining byte budget.
pub fn alpha(n: u64, max_bytes: usize) -> Option<String> {
let n = n.max(1) - 1;
let count = usize::try_from(n / 26 + 1).ok()?;
if count > max_bytes.min(MAX_LABEL_BYTES) {
return None;
}
let letter = char::from(b'a' + (n % 26) as u8);
Some(std::iter::repeat_n(letter, count).collect())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn bounded_page_labels_follow_pdf_numbering() {
assert_eq!(roman(4, MAX_LABEL_BYTES).as_deref(), Some("iv"));
assert_eq!(roman(1994, MAX_LABEL_BYTES).as_deref(), Some("mcmxciv"));
for (n, expected) in [(0, "a"), (1, "a"), (26, "z"), (27, "aa"), (52, "zz"), (53, "aaa")] {
assert_eq!(alpha(n, MAX_LABEL_BYTES).as_deref(), Some(expected));
}
}
#[test]
fn bounded_page_labels_check_lengths_before_repeating() {
assert_eq!(roman(1_024_000, MAX_LABEL_BYTES).unwrap().len(), MAX_LABEL_BYTES);
assert!(roman(1_024_001, MAX_LABEL_BYTES).is_none());
assert_eq!(alpha(26 * 1024, MAX_LABEL_BYTES).unwrap().len(), MAX_LABEL_BYTES);
assert!(alpha(26 * 1024 + 1, MAX_LABEL_BYTES).is_none());
assert_eq!(roman(4, 2).as_deref(), Some("iv"));
assert!(roman(4, 1).is_none());
assert_eq!(alpha(27, 2).as_deref(), Some("aa"));
assert!(alpha(27, 1).is_none());
for max in [0, MAX_LABEL_BYTES, usize::MAX] {
assert!(roman(u64::MAX, max).is_none());
assert!(alpha(u64::MAX, max).is_none());
}
}
}
+886
View File
@@ -0,0 +1,886 @@
//! Lexer and object parser (ISO 32000-2 §7.2–7.3), tolerant of common real-world damage.
//!
//! The parser works on a byte slice and a position; it never allocates for skipped data and
//! never panics on malformed input (every index is bounds-checked). Depth is limited to defend
//! against pathological nesting.
use std::sync::Arc;
use crate::cos::object::{Dict, ObjRef, Object, PdfString, Stream};
use crate::cos::{Bytes, CosError};
const MAX_DEPTH: usize = 128;
/// Objects one top-level [`Lexer::object`] call may build. Real arrays (page trees, parent
/// trees, widths) stay far below; a decompressed object stream of `0 0 0 ...` does not.
const MAX_NODES: usize = 1 << 22;
/// One of the six PDF whitespace characters (ISO 32000-2 §7.2.3, Table 1).
pub fn is_whitespace(b: u8) -> bool {
matches!(b, b'\0' | b'\t' | b'\n' | b'\x0C' | b'\r' | b' ')
}
pub(crate) fn is_delimiter(b: u8) -> bool {
matches!(b, b'(' | b')' | b'<' | b'>' | b'[' | b']' | b'{' | b'}' | b'/' | b'%')
}
fn is_regular(b: u8) -> bool {
!is_whitespace(b) && !is_delimiter(b)
}
/// A cursor over PDF bytes.
pub struct Lexer<'a> {
pub data: &'a [u8],
pub pos: usize,
/// Objects built by the current top-level [`Lexer::object`] call.
nodes: usize,
}
impl<'a> Lexer<'a> {
pub fn new(data: &'a [u8], pos: usize) -> Self {
Self { data, pos: pos.min(data.len()), nodes: 0 }
}
fn peek(&self) -> Option<u8> {
self.data.get(self.pos).copied()
}
/// Skip whitespace and comments.
pub fn skip_ws(&mut self) {
while let Some(b) = self.peek() {
if is_whitespace(b) {
self.pos += 1;
} else if b == b'%' {
while let Some(c) = self.peek() {
if c == b'\n' || c == b'\r' {
break;
}
self.pos += 1;
}
} else {
break;
}
}
}
/// Read a run of regular characters (a keyword or number).
pub fn token(&mut self) -> &'a [u8] {
let start = self.pos.min(self.data.len());
while self.peek().is_some_and(is_regular) {
self.pos += 1;
}
&self.data[start..self.pos]
}
/// `true` (and consumes it) if the next token is exactly `kw`.
pub fn eat_keyword(&mut self, kw: &[u8]) -> bool {
self.skip_ws();
let end = self.pos + kw.len();
if self.data.get(self.pos..end) == Some(kw) && self.data.get(end).is_none_or(|b| !is_regular(*b)) {
self.pos = end;
true
} else {
false
}
}
fn err(&self, what: &str) -> CosError {
CosError::Syntax { offset: self.pos, detail: what.to_string() }
}
/// Parse one direct object. References `N G R` are recognised.
pub fn object(&mut self) -> Result<Object, CosError> {
self.nodes = 0;
self.object_depth(0)
}
fn over_budget(&self) -> bool {
self.nodes > MAX_NODES
}
fn object_depth(&mut self, depth: usize) -> Result<Object, CosError> {
if depth > MAX_DEPTH {
return Err(self.err("nesting too deep"));
}
self.nodes = self.nodes.saturating_add(1);
if self.over_budget() {
return Err(self.err("object too large"));
}
self.skip_ws();
let Some(b) = self.peek() else { return Err(self.err("unexpected end of data")) };
match b {
b'/' => {
self.pos += 1;
Ok(Object::Name(self.name_body()))
}
b'(' => {
self.pos += 1;
Ok(Object::String(PdfString { bytes: self.literal_string()?, hex: false }))
}
b'<' if self.data.get(self.pos + 1) == Some(&b'<') => {
self.pos += 2;
Ok(Object::Dict(self.dict_body(depth)?))
}
b'<' => {
self.pos += 1;
Ok(Object::String(PdfString { bytes: self.hex_string()?, hex: true }))
}
b'[' => {
self.pos += 1;
let mut items = Vec::new();
loop {
self.skip_ws();
match self.peek() {
None => return Err(self.err("unterminated array")),
Some(b']') => {
self.pos += 1;
break;
}
_ => {
let before = self.pos;
match self.object_depth(depth + 1) {
Ok(o) => items.push(o),
Err(e) if self.over_budget() => return Err(e),
// Tolerate junk inside arrays by skipping one byte.
Err(_) if self.pos == before => self.pos += 1,
Err(e) => return Err(e),
}
}
}
}
Ok(Object::Array(items))
}
b'+' | b'-' | b'.' | b'0'..=b'9' => self.number_or_ref(),
_ => {
let start = self.pos;
let tok = self.token();
match tok {
b"true" => Ok(Object::Bool(true)),
b"false" => Ok(Object::Bool(false)),
b"null" => Ok(Object::Null),
b"" => {
self.pos = start + 1;
Err(CosError::Syntax { offset: start, detail: format!("unexpected byte 0x{b:02x}") })
}
other => Err(CosError::Syntax { offset: start, detail: format!("unexpected keyword {:?}", String::from_utf8_lossy(other)) }),
}
}
}
}
fn name_body(&mut self) -> Vec<u8> {
let mut out = Vec::new();
while let Some(b) = self.peek() {
if !is_regular(b) {
break;
}
self.pos += 1;
if b == b'#'
&& let (Some(h), Some(l)) = (self.data.get(self.pos).and_then(|c| hexval(*c)), self.data.get(self.pos + 1).and_then(|c| hexval(*c)))
{
out.push(h << 4 | l);
self.pos += 2;
continue;
}
out.push(b);
}
out
}
fn literal_string(&mut self) -> Result<Vec<u8>, CosError> {
let mut out = Vec::new();
let mut depth = 1usize;
while let Some(b) = self.peek() {
self.pos += 1;
match b {
b'(' => {
depth += 1;
out.push(b);
}
b')' => {
depth -= 1;
if depth == 0 {
return Ok(out);
}
out.push(b);
}
b'\\' => {
let Some(e) = self.peek() else { break };
self.pos += 1;
match e {
b'n' => out.push(b'\n'),
b'r' => out.push(b'\r'),
b't' => out.push(b'\t'),
b'b' => out.push(8),
b'f' => out.push(12),
b'(' | b')' | b'\\' => out.push(e),
b'\r' => {
if self.peek() == Some(b'\n') {
self.pos += 1;
}
}
b'\n' => {}
b'0'..=b'7' => {
let mut v = (e - b'0') as u32;
for _ in 0..2 {
match self.peek() {
Some(d @ b'0'..=b'7') => {
v = v * 8 + (d - b'0') as u32;
self.pos += 1;
}
_ => break,
}
}
out.push((v & 0xFF) as u8);
}
other => out.push(other),
}
}
// End-of-line markers inside strings are normalised to \n (§7.3.4.2).
b'\r' => {
if self.peek() == Some(b'\n') {
self.pos += 1;
}
out.push(b'\n');
}
other => out.push(other),
}
}
Err(self.err("unterminated string"))
}
fn hex_string(&mut self) -> Result<Vec<u8>, CosError> {
let mut out = Vec::new();
let mut hi: Option<u8> = None;
while let Some(b) = self.peek() {
self.pos += 1;
if b == b'>' {
if let Some(h) = hi {
out.push(h << 4);
}
return Ok(out);
}
if is_whitespace(b) {
continue;
}
let Some(v) = hexval(b) else { return Err(self.err("invalid hex digit")) };
match hi.take() {
None => hi = Some(v),
Some(h) => out.push(h << 4 | v),
}
}
Err(self.err("unterminated hex string"))
}
fn dict_body(&mut self, depth: usize) -> Result<Dict, CosError> {
let mut d = Dict::new();
loop {
self.skip_ws();
match self.peek() {
None => return Err(self.err("unterminated dictionary")),
Some(b'>') if self.data.get(self.pos + 1) == Some(&b'>') => {
self.pos += 2;
return Ok(d);
}
Some(b'/') => {
self.pos += 1;
let key = self.name_body();
self.skip_ws();
// A key directly followed by `>>` has no value: treat as null. (A following
// `/` is a name *value*, so it cannot signal a missing value.)
if self.peek() == Some(b'>') && self.data.get(self.pos + 1) == Some(&b'>') {
d.set(key, Object::Null);
continue;
}
let value = self.object_depth(depth + 1)?;
d.set(key, value);
}
Some(_) => {
// Tolerate garbage between entries (seen in damaged files): skip a token.
let before = self.pos;
let _ = self.object_depth(depth + 1);
if self.over_budget() {
return Err(self.err("object too large"));
}
if self.pos == before {
self.pos += 1;
}
}
}
}
}
fn number_or_ref(&mut self) -> Result<Object, CosError> {
let start = self.pos;
let first = self.number()?;
// Look ahead for "G R" to form a reference.
if let Object::Int(n) = first
&& n >= 0
{
let save = self.pos;
self.skip_ws();
if self.peek().is_some_and(|b| b.is_ascii_digit()) {
let gstart = self.pos;
let g = self.token();
if g.iter().all(u8::is_ascii_digit) && self.eat_keyword(b"R") {
let generation = std::str::from_utf8(g).ok().and_then(|s| s.parse::<u32>().ok()).unwrap_or(0);
if n <= u32::MAX as i64 {
return Ok(Object::Ref(ObjRef::new(n as u32, generation.min(u16::MAX as u32) as u16)));
}
}
let _ = gstart;
}
self.pos = save;
}
let _ = start;
Ok(first)
}
fn number(&mut self) -> Result<Object, CosError> {
let start = self.pos;
let tok = self.token();
let s = std::str::from_utf8(tok).map_err(|_| self.err("bad number"))?;
if s.contains('.') {
// Tolerate forms like "--5", "5-" or "1.2.3" by parsing the longest valid prefix.
// Only a doubled sign needs a cleaned copy: a page of InDesign `/PieceInfo` is
// thousands of plain reals, and building a copy and a substring search for each
// was a large part of reading one (#5873).
let t = s.trim_start_matches('+');
let cleaned = if t.as_bytes().windows(2).any(|w| w == b"--") { std::borrow::Cow::Owned(t.replacen("--", "-", 1)) } else { t.into() };
let v = cleaned.get(..real_prefix_len(cleaned.as_bytes())).and_then(|p| parse_counted::<f64>(p).ok());
return Ok(Object::Real(v.unwrap_or(0.0)));
}
// Tolerate a doubled sign ("--280") like the real branch does.
let t = s.trim_start_matches('+');
let cleaned = t.strip_prefix('-').filter(|r| r.starts_with('-')).unwrap_or(t);
match cleaned.parse::<i64>() {
Ok(v) => Ok(Object::Int(v)),
Err(_) => {
// Out-of-range integers are treated as reals (§7.3.3 allows implementation limits).
cleaned.parse::<f64>().map(Object::Real).map_err(|_| CosError::Syntax { offset: start, detail: format!("bad number {s:?}") })
}
}
}
}
/// Length of the longest prefix of `s` that reads as a decimal number: a sign, digits with at
/// most one `.`, and an exponent with digits. Zero when that prefix has no digit. Found in one
/// pass so a long damaged token is converted once, not once per shorter prefix.
fn real_prefix_len(s: &[u8]) -> usize {
let digits = |from: usize| s.get(from..).map_or(0, |r| r.iter().take_while(|b| b.is_ascii_digit()).count());
let mut at = usize::from(matches!(s.first(), Some(b'+' | b'-')));
let int = digits(at);
at += int;
let mut frac = 0;
if s.get(at) == Some(&b'.') {
frac = digits(at + 1);
at += 1 + frac;
}
if int + frac == 0 {
return 0;
}
if matches!(s.get(at), Some(b'e' | b'E')) {
let sign = usize::from(matches!(s.get(at + 1), Some(b'+' | b'-')));
let exp = digits(at + 1 + sign);
if exp > 0 {
at += 1 + sign + exp;
}
}
at
}
#[cfg(test)]
thread_local! {
/// Bytes searched for `endstream` on this thread.
pub(crate) static ENDSTREAM_SCAN: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
/// Bytes handed to number conversion on this thread.
static NUMBER_BYTES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
}
fn parse_counted<T: std::str::FromStr>(s: &str) -> Result<T, T::Err> {
#[cfg(test)]
NUMBER_BYTES.with(|c| c.set(c.get() + s.len()));
s.parse()
}
fn hexval(b: u8) -> Option<u8> {
match b {
b'0'..=b'9' => Some(b - b'0'),
b'a'..=b'f' => Some(b - b'a' + 10),
b'A'..=b'F' => Some(b - b'A' + 10),
_ => None,
}
}
/// Parse an indirect object at `offset`: `N G obj <object> [stream…endstream] endobj`.
/// `resolve_length` resolves an indirect `/Length`. Returns the object and its (num, gen).
pub fn parse_indirect(data: &[u8], offset: usize, resolve_length: &dyn Fn(ObjRef) -> Option<i64>) -> Result<(ObjRef, Object), CosError> {
parse_indirect_in(data, None, offset, resolve_length, None)
}
/// [`parse_indirect`] for a document's own bytes: stream data points into `data` instead of
/// being copied out of it.
/// `endstreams` is the document's [`EndstreamIndex`] for `data`.
pub(crate) fn parse_indirect_shared(
data: &Arc<Vec<u8>>,
offset: usize,
resolve_length: &dyn Fn(ObjRef) -> Option<i64>,
endstreams: &EndstreamIndex,
) -> Result<(ObjRef, Object), CosError> {
parse_indirect_in(data, Some(data), offset, resolve_length, Some(endstreams))
}
/// Where `endstream` occurs in a document's bytes, found once, the first time a stream's
/// /Length is wrong or missing. Searching the rest of the file for each such stream instead
/// makes many of them ending at one late `endstream` quadratic to read.
pub(crate) type EndstreamIndex = std::sync::OnceLock<Vec<usize>>;
fn endstream_offsets(data: &[u8]) -> Vec<usize> {
#[cfg(test)]
ENDSTREAM_SCAN.with(|c| c.set(c.get() + data.len()));
data.windows(9).enumerate().filter(|(i, w)| *w == b"endstream" && endstream_at(data, *i)).map(|(i, _)| i).collect()
}
fn parse_indirect_in(
data: &[u8],
shared: Option<&Arc<Vec<u8>>>,
offset: usize,
resolve_length: &dyn Fn(ObjRef) -> Option<i64>,
endstreams: Option<&EndstreamIndex>,
) -> Result<(ObjRef, Object), CosError> {
let mut lx = Lexer::new(data, offset);
lx.skip_ws();
let num_tok = lx.token();
lx.skip_ws();
let gen_tok = lx.token();
let parse_u = |t: &[u8]| std::str::from_utf8(t).ok().and_then(|s| s.parse::<u64>().ok());
let (Some(num), Some(generation)) = (parse_u(num_tok), parse_u(gen_tok)) else {
return Err(CosError::Syntax { offset, detail: "expected object header".into() });
};
if !lx.eat_keyword(b"obj") {
return Err(CosError::Syntax { offset: lx.pos, detail: "expected 'obj'".into() });
}
let id = ObjRef::new(num.min(u32::MAX as u64) as u32, generation.min(u16::MAX as u64) as u16);
let obj = match lx.object() {
Ok(o) => o,
// "N G obj endobj" (empty) is treated as null.
Err(_) if lx.eat_keyword(b"endobj") => return Ok((id, Object::Null)),
Err(e) => return Err(e),
};
lx.skip_ws();
if let Object::Dict(dict) = &obj
&& lx.eat_keyword(b"stream")
{
// The keyword is followed by CRLF or LF (a lone CR is tolerated). Microsoft Reporting
// Services writes a space first (`stream \r\n`); spaces and tabs count only when a line
// break follows, so they are never taken from the data.
let mut eol = lx.pos;
while matches!(data.get(eol), Some(b' ' | b'\t')) {
eol += 1;
}
if matches!(data.get(eol), Some(b'\r' | b'\n')) {
lx.pos = eol;
}
match (data.get(lx.pos), data.get(lx.pos + 1)) {
(Some(b'\r'), Some(b'\n')) => lx.pos += 2,
(Some(b'\n'), _) | (Some(b'\r'), _) => lx.pos += 1,
_ => {}
}
let start = lx.pos;
let declared = match dict.get(b"Length") {
Some(Object::Int(n)) => Some(*n),
Some(Object::Ref(r)) => resolve_length(*r),
_ => None,
};
// A /Length that fits in the file (not negative, and no wrap on a 32-bit target).
let fits = declared.and_then(|l| usize::try_from(l).ok()).filter(|l| start.checked_add(*l).is_some_and(|end| end <= data.len()));
// What follows the /Length bytes, white space aside.
let after = |len: usize| {
let mut after = Lexer::new(data, start.saturating_add(len));
after.skip_ws();
data.get(after.pos..).unwrap_or_default()
};
let len = match fits {
// The /Length ends at `endstream`, or where it would be: misspelt (`endstrea`), or
// left out before `endobj` or the end of the file. The stream is the bytes its /Length
// gives, "how many bytes of the PDF file are used for the stream's data" (ISO 32000-2
// §7.3.8.2), as other readers take it. A producer that wrote `endstrea` after every
// stream lost every font and every page's content, and searching on would run into a
// later object's `endstream`.
Some(l) if at_stream_end(after(l)) => l,
// Wrong or missing /Length: search for "endstream" (§7.3.8.1 recovery). With none
// left in the file, a /Length that fits is still the stream's extent.
_ => match find_endstream(data, start, endstreams) {
Some(end) => end - start,
None => fits.ok_or_else(|| CosError::Syntax { offset: start, detail: "stream without endstream".into() })?,
},
};
let range = start..start + len;
let raw = match shared {
Some(buf) => Bytes::view(buf, range),
None => data.get(range).map(|r| Bytes::from(r.to_vec())),
}
.ok_or_else(|| CosError::Syntax { offset: start, detail: "stream data past the end of the file".into() })?;
let mut dict = dict.clone();
dict.set(b"Length".to_vec(), Object::Int(raw.len() as i64));
return Ok((id, Object::Stream(Stream { dict, raw })));
}
Ok((id, obj))
}
/// Whether `rest`, what follows a stream's /Length bytes with white space skipped, is where the
/// stream ends. A whole `endstream` token is: a token runs on through regular characters (ISO
/// 32000-2 §7.2.3), so payload such as `endstreamer` is not one. A misspelt `endstream`
/// (`endstrea`), an `endobj` with the `endstream` left out, or both, end it only where the object
/// ends as well, before the end of the data, another object's `N G obj` header, or `xref`,
/// `trailer` or `startxref`: payload such as `endobj more` or `endobject`, where a wrong /Length
/// stops, is not, and the search for `endstream` runs instead. So does anything else but the end
/// of the data.
fn at_stream_end(rest: &[u8]) -> bool {
let mut lx = Lexer::new(rest, 0);
if lx.eat_keyword(b"endstream") {
return true;
}
let misspelt = lx.eat_keyword(b"endstrea");
let endobj = lx.eat_keyword(b"endobj");
lx.skip_ws();
if lx.pos >= rest.len() {
return true;
}
if !(misspelt || endobj) {
return false;
}
if [&b"xref"[..], b"trailer", b"startxref"].iter().any(|kw| Lexer::new(rest, lx.pos).eat_keyword(kw)) {
return true;
}
let number = |lx: &mut Lexer| {
lx.skip_ws();
let t = lx.token();
!t.is_empty() && t.iter().all(u8::is_ascii_digit)
};
number(&mut lx) && number(&mut lx) && lx.eat_keyword(b"obj")
}
/// Whether a whole `endstream` token starts at `at` in `data`: one followed by a regular
/// character (`endstreamer`) is payload (ISO 32000-2 §7.2.3).
fn endstream_at(data: &[u8], at: usize) -> bool {
data.get(at..).is_some_and(|d| d.starts_with(b"endstream")) && !data.get(at.saturating_add(9)).copied().is_some_and(is_regular)
}
/// Offset where stream data ends (before the EOL preceding "endstream").
fn find_endstream(data: &[u8], from: usize, endstreams: Option<&EndstreamIndex>) -> Option<usize> {
let at = match endstreams {
Some(index) => {
let index = index.get_or_init(|| endstream_offsets(data));
index.get(index.partition_point(|&p| p < from)).copied()?
}
None => {
let hay = data.get(from..)?;
let found = hay.windows(9).enumerate().position(|(i, w)| w == b"endstream" && endstream_at(hay, i));
#[cfg(test)]
ENDSTREAM_SCAN.with(|c| c.set(c.get() + found.map_or(hay.len(), |p| p + 9)));
found? + from
}
};
let mut end = at;
if end > from && data[end - 1] == b'\n' {
end -= 1;
}
if end > from && data[end - 1] == b'\r' {
end -= 1;
}
Some(end)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn numbers_with_multibyte_garbage_do_not_panic() {
// From `cargo xtask fuzz`: "2 0 obj\n.\u{2014}" (an em dash after a dot).
let doc = crate::cos::Document::open(std::sync::Arc::new("2 0 obj\n.\u{2014}\n".as_bytes().to_vec()));
assert!(doc.is_err());
for t in [".\u{2014}", "1.\u{e9}5", "-.\u{1F600}", "+.5\u{2014}"] {
let _ = Lexer::new(t.as_bytes(), 0).object();
}
assert_eq!(Lexer::new("1.5\u{2014}".as_bytes(), 0).object().unwrap(), Object::Real(1.5));
}
#[test]
fn a_damaged_real_is_converted_once() {
let token = format!("1.{}{}", "5".repeat(2_000), "x".repeat(20_000));
NUMBER_BYTES.with(|c| c.set(0));
let Ok(Object::Real(v)) = Lexer::new(token.as_bytes(), 0).object() else { panic!("not a real") };
let converted = NUMBER_BYTES.with(|c| c.get());
assert!(converted <= token.len(), "{converted} bytes converted for a {}-byte token", token.len());
assert!((v - 1.5555).abs() < 1e-3);
// The longest numeric prefix wins, as before.
for (t, want) in
[("1.2.3", 1.2), ("--5.", -5.0), ("+.5x", 0.5), ("5.-", 5.0), ("1.5e3", 1500.0), ("1.5e+", 1.5), ("-.x", 0.0), (".", 0.0), ("-inf.", 0.0)]
{
assert_eq!(Lexer::new(t.as_bytes(), 0).object().ok(), Some(Object::Real(want)), "{t}");
}
}
fn obj(s: &str) -> Object {
Lexer::new(s.as_bytes(), 0).object().unwrap_or_else(|e| panic!("{s}: {e}"))
}
#[test]
fn doubled_sign_integer_reads_like_the_real_form() {
// Some writers emit a TJ kerning as "--280"; the real form "--280.0" is already read.
assert_eq!(obj("--280.0"), Object::Real(-280.0));
assert_eq!(obj("--280"), Object::Int(-280));
match obj("[(Hello) --280 (World)]") {
Object::Array(items) => {
assert_eq!(items.len(), 3);
assert_eq!(items[1], Object::Int(-280));
}
other => panic!("not an array: {other:?}"),
}
}
#[test]
fn scalars() {
assert_eq!(obj("true"), Object::Bool(true));
assert_eq!(obj("null"), Object::Null);
assert_eq!(obj(" -17 "), Object::Int(-17));
assert_eq!(obj("+42"), Object::Int(42));
assert_eq!(obj("-.5"), Object::Real(-0.5));
assert_eq!(obj("4."), Object::Real(4.0));
assert_eq!(obj("99999999999999999999"), Object::Real(1e20));
assert_eq!(obj("12 0 R"), Object::Ref(ObjRef::new(12, 0)));
assert_eq!(obj("12 0 obj").as_int(), Some(12), "12 followed by 0 obj is not a reference");
}
#[test]
fn names_decode_hex_escapes() {
assert_eq!(obj("/A#20B"), Object::Name(b"A B".to_vec()));
assert_eq!(obj("/Adobe#"), Object::Name(b"Adobe#".to_vec()));
assert_eq!(obj("/"), Object::Name(Vec::new()));
}
#[test]
fn literal_strings() {
let s = |x: &str| obj(x).as_string().expect("string").bytes.clone();
assert_eq!(s("(a (nested) b)"), b"a (nested) b");
assert_eq!(s(r"(\n\t\\\(\))"), b"\n\t\\()");
assert_eq!(s(r"(\101\1011)"), b"AA1");
assert_eq!(s("(line\\\ncontinued)"), b"linecontinued");
assert_eq!(s("(cr\r\nlf)"), b"cr\nlf");
}
#[test]
fn hex_strings() {
let s = obj("<48 65 6C6c6f7>");
assert_eq!(s.as_string().unwrap().bytes, b"Hellop");
assert!(s.as_string().unwrap().hex);
}
#[test]
fn arrays_and_dicts() {
let o = obj("<< /Type /Page /Kids [1 0 R 2 0 R] /Box [0 0 612.5 792] /Nested << /A true >> /Empty >>");
let d = o.as_dict().unwrap();
assert_eq!(d.name(b"Type"), Some(&b"Page"[..]));
assert_eq!(d.get(b"Kids").unwrap().as_array().unwrap().len(), 2);
assert_eq!(d.get(b"Box").unwrap().as_array().unwrap()[2], Object::Real(612.5));
assert_eq!(d.get(b"Empty"), Some(&Object::Null));
}
#[test]
fn comments_are_whitespace() {
assert_eq!(obj("% hello\n [1 % two\n 3]"), Object::Array(vec![Object::Int(1), Object::Int(3)]));
}
#[test]
fn indirect_stream_with_wrong_length_recovers() {
let data = b"7 0 obj\n<< /Length 999 >>\nstream\nHELLO\nendstream\nendobj\n";
let (id, o) = parse_indirect(data, 0, &|_| None).unwrap();
assert_eq!(id, ObjRef::new(7, 0));
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(&s.raw[..], b"HELLO");
assert_eq!(s.dict.int(b"Length"), Some(5));
}
/// With no `endstream` left in the file, as a producer that writes `endstrea` leaves every
/// stream, a stream is the bytes its /Length gives, "how many bytes of the PDF file are used
/// for the stream's data" (ISO 32000-2 §7.3.8.2): it was an error, and the object null, so
/// such a file lost every font and page content. A /Length past the file, or none, is still
/// no stream.
#[test]
fn indirect_stream_without_endstream_is_its_length() {
for end in [&b"\r\nendstrea\rendobj\r\n"[..], b"\nendobj\n", b""] {
let mut data = b"7 0 obj\n<< /Length 5 >>\nstream\r\nHELLO".to_vec();
data.extend_from_slice(end);
let (_, o) = parse_indirect(&data, 0, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(&s.raw[..], b"HELLO", "{end:?}");
}
assert!(parse_indirect(b"7 0 obj\n<< /Length 6 >>\nstream\nHELLO", 0, &|_| None).is_err(), "past the file");
assert!(parse_indirect(b"7 0 obj\n<< >>\nstream\nHELLO\nendstrea\nendobj\n", 0, &|_| None).is_err(), "no /Length");
}
/// A stream ending in `endstrea` is its /Length even when a later object does have an
/// `endstream`: the search for one would otherwise run on into that object and take its
/// header and data as this stream's.
#[test]
fn a_stream_without_endstream_is_not_read_into_a_later_one() {
let data = b"7 0 obj\n<< /Length 5 >>\nstream\r\nHELLO\r\nendstrea\rendobj\r\n8 0 obj\n<< /Length 5 >>\nstream\nWORLD\nendstream\nendobj\n";
let second = data.windows(7).position(|w| w == b"8 0 obj").unwrap();
for (offset, want) in [(0, &b"HELLO"[..]), (second, b"WORLD")] {
let (_, o) = parse_indirect(data, offset, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected at {offset}") };
assert_eq!(&s.raw[..], want, "at {offset}");
}
// Through the loader's shared index of `endstream` offsets as well.
let index = EndstreamIndex::new();
let (_, o) = parse_indirect_in(data, None, 0, &|_| None, Some(&index)).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(&s.raw[..], b"HELLO");
}
/// The same for a stream whose `endstream` is left out before `endobj`: the later object's
/// `endstream` is not this stream's end.
#[test]
fn a_stream_ending_at_endobj_is_not_read_into_a_later_one() {
let data = b"7 0 obj\n<< /Length 5 >>\nstream\nHELLO\nendobj\n8 0 obj\n<< /Length 5 >>\nstream\nWORLD\nendstream\nendobj\n";
let (_, o) = parse_indirect(data, 0, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(&s.raw[..], b"HELLO");
}
/// A wrong /Length that stops before payload bytes beginning like an end marker (`endobject`,
/// `endstreak`, `endstreamer`) is not taken for the stream's end: a token runs on through
/// regular characters (ISO 32000-2 §7.2.3), so the search for `endstream` finds the whole
/// payload, and skips `endstreamer` too, by itself and through the loader's shared index.
#[test]
fn a_length_ending_inside_a_longer_word_is_not_an_end_marker() {
for word in ["endobject", "endstreak", "endstreamer"] {
let pdf = format!("7 0 obj\n<< /Length 5 >>\nstream\nHELLO {word} more\nendstream\nendobj\n");
let (_, o) = parse_indirect(pdf.as_bytes(), 0, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(s.raw[..], *format!("HELLO {word} more").as_bytes(), "{word}");
let index = EndstreamIndex::new();
let (_, o) = parse_indirect_in(pdf.as_bytes(), None, 0, &|_| None, Some(&index)).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(s.raw[..], *format!("HELLO {word} more").as_bytes(), "{word}, indexed");
}
// Whole markers still end it, followed by white space, a delimiter or the end of the data.
for tail in ["endobj\n", "endstrea%", "endobj"] {
let pdf = format!("7 0 obj\n<< /Length 5 >>\nstream\nHELLO\n{tail}");
let (_, o) = parse_indirect(pdf.as_bytes(), 0, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(&s.raw[..], b"HELLO", "{tail:?}");
}
}
/// A whole `endobj` or `endstrea` token where a wrong /Length stops is payload, not the
/// stream's end, unless the object ends there too: what follows is the end of the data,
/// another object's header, or `xref`, `trailer` or `startxref`. Otherwise the search for
/// `endstream` finds the whole payload.
#[test]
fn an_end_marker_inside_the_payload_does_not_end_the_stream() {
for payload in ["endobj more", "endstrea more", "endstrea endobj more", "endobj 12 more"] {
let pdf = format!("7 0 obj\n<< /Length 5 >>\nstream\nHELLO {payload}\nendstream\nendobj\n");
let (_, o) = parse_indirect(pdf.as_bytes(), 0, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(s.raw[..], *format!("HELLO {payload}").as_bytes(), "{payload}");
}
// Where the object does end, the /Length is the stream, also with an `endstream` later.
for next in ["8 0 obj\n<< >>", "%c\n12 0 obj", "xref", "trailer", "startxref"] {
for marker in ["endobj", "endstrea\nendobj", "endstrea"] {
let pdf = format!("7 0 obj\n<< /Length 5 >>\nstream\nHELLO\n{marker}\n{next}\nstream\nWORLD\nendstream\nendobj\n");
let (_, o) = parse_indirect(pdf.as_bytes(), 0, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(&s.raw[..], b"HELLO", "{marker:?} then {next:?}");
}
}
}
/// Without an `endstream`, the /Length is trusted as written when it fits, also when it is
/// wrong, and an indirect one is resolved first; one that doesn't fit, a negative one and one
/// too large for any file are errors rather than wrapped into a short read.
#[test]
fn a_stream_without_endstream_takes_its_length_only_when_it_fits() {
let stream = |length: &str, tail: &str| format!("7 0 obj\n<< /Length {length} >>\nstream\nHELLO{tail}");
let raw = |pdf: &str, lookup: &dyn Fn(ObjRef) -> Option<i64>| match parse_indirect(pdf.as_bytes(), 0, lookup) {
Ok((_, Object::Stream(s))) => Ok(s.raw.to_vec()),
Ok((_, other)) => panic!("{pdf:?} is {other:?}"),
Err(e) => Err(e),
};
let none = |_: ObjRef| None;
// Wrong but fitting: the bytes it gives, not the bytes up to `endstrea`.
assert_eq!(raw(&stream("3", "\nendstrea\nendobj\n"), &none).unwrap(), b"HEL");
// Exactly to the end of the file, and nothing at all.
assert_eq!(raw(&stream("5", ""), &none).unwrap(), b"HELLO");
assert_eq!(raw(&stream("0", "\nendstrea\nendobj\n"), &none).unwrap(), b"");
// Indirect, resolved through the lookup.
let lookup = |r: ObjRef| (r == ObjRef::new(9, 0)).then_some(4);
assert_eq!(raw(&stream("9 0 R", "\nendstrea\nendobj\n"), &lookup).unwrap(), b"HELL");
assert!(raw(&stream("9 0 R", "\nendstrea\nendobj\n"), &none).is_err(), "an unresolved /Length");
// One past the end, negative, the largest integer and one beyond it.
for length in ["6", "-1", "-9223372036854775808", "9223372036854775807", "18446744073709551621"] {
assert!(raw(&stream(length, ""), &none).is_err(), "/Length {length}");
}
}
#[test]
fn indirect_stream_with_a_space_before_the_line_break() {
// `stream \r\n` as Microsoft Reporting Services writes it: the data starts after the
// line break, so the /Length is still right and the data keeps no leading space.
for eol in [&b" \r\n"[..], b" \n", b"\t \r"] {
let mut data = b"7 0 obj\n<< /Length 5 >>\nstream".to_vec();
data.extend_from_slice(eol);
data.extend_from_slice(b"HELLO\r\nendstream\nendobj\n");
let (_, o) = parse_indirect(&data, 0, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(&s.raw[..], b"HELLO", "{eol:?}");
}
// Spaces with no line break after them are data, not a keyword terminator.
let data = b"7 0 obj\n<< /Length 7 >>\nstream HELLO\nendstream\nendobj\n";
let (_, o) = parse_indirect(data, 0, &|_| None).unwrap();
let Object::Stream(s) = o else { panic!("stream expected") };
assert_eq!(&s.raw[..], b" HELLO");
}
#[test]
fn indirect_length_is_resolved() {
let data = b"7 0 obj\n<< /Length 8 0 R >>\nstream\r\nAB\r\nendstream\nendobj\n";
let (_, o) = parse_indirect(data, 0, &|r| (r.num == 8).then_some(2)).unwrap();
let Object::Stream(s) = o else { panic!() };
assert_eq!(&s.raw[..], b"AB");
}
#[test]
fn garbage_never_panics() {
for input in ["<<", "[", "(", "<", "<< /A", "[1 2", ")", "]", ">>", "<<>>>", "1 0", "\0\0", "/A#", "<zz>", "((("] {
let _ = Lexer::new(input.as_bytes(), 0).object();
}
let deep = "[".repeat(10_000);
assert!(Lexer::new(deep.as_bytes(), 0).object().is_err());
}
#[test]
fn one_object_has_a_node_budget() {
// A wide array (from a tiny compressed object stream) must not grow without bound.
let wide = |n: usize| format!("[{}]", "0 ".repeat(n));
let at_cap = Lexer::new(wide(MAX_NODES - 1).as_bytes(), 0).object();
assert!(matches!(at_cap, Ok(Object::Array(ref items)) if items.len() == MAX_NODES - 1));
assert!(Lexer::new(wide(MAX_NODES).as_bytes(), 0).object().is_err());
// The same inside a dictionary value and behind tolerated junk.
assert!(Lexer::new(format!("<< /A {} >>", wide(MAX_NODES)).as_bytes(), 0).object().is_err());
assert!(Lexer::new(format!("[[{}] ]", "0 ".repeat(MAX_NODES)).as_bytes(), 0).object().is_err());
// The budget is per top-level object: a lexer reused for many operands keeps working.
let two = wide(MAX_NODES - 1).repeat(2);
let mut lx = Lexer::new(two.as_bytes(), 0);
assert!(lx.object().is_ok() && lx.object().is_ok());
}
/// A plain real is read without a cleaned copy of its token; the forms that need one read
/// as they did (#5873).
#[test]
fn reals_read_the_same_with_and_without_a_doubled_sign() {
for (token, want) in [("1.5", 1.5), ("-375.993", -375.993), ("+2.25", 2.25), ("--5.5", -5.5), ("1.2.3", 1.2), ("-.5", -0.5), ("0.0", 0.0)] {
assert_eq!(Lexer::new(token.as_bytes(), 0).object().unwrap(), Object::Real(want), "{token}");
}
}
}
+376
View File
@@ -0,0 +1,376 @@
//! Bridging the COS object model and [`crate::crypt`]: reading `/Encrypt` dictionaries and
//! decrypting / encrypting the strings and streams of an object (§7.6.2).
//!
//! Not encrypted, per the spec: the `/Encrypt` dictionary itself, cross-reference streams, the
//! trailer `/ID`, objects inside object streams (the object stream is encrypted as a whole),
//! the `/Contents` of signature dictionaries, and metadata streams when `/EncryptMetadata` is
//! false. Streams naming a crypt filter (`/Crypt` in `/Filter`) use that filter; embedded files
//! use `/EFF`.
use std::cell::Cell;
use crate::crypt::{EncryptDict, Method, SecurityHandler, StreamKind};
use crate::filters::MAX_CHAIN;
use crate::cos::document::Held;
use crate::cos::{Dict, Document, Object, PdfString, Stream};
/// How many references one read or write of an object may follow, with all that it loads on the
/// way. How an object is treated can be given by reference, and what that loads can do the same: a
/// file could make one read a tree of them, each object read again once the cache has let it go.
/// An object takes a few of these, and two dozen if every part of it is given by reference; past
/// this they read as absent. The parameters of an object stream, read together, count as one.
const MAX_FOLLOWED: u32 = 32;
std::thread_local! {
/// How many [`FollowScope`]s this thread is inside.
static SCOPES: Cell<u32> = const { Cell::new(0) };
/// The references followed in the outermost of them.
static FOLLOWED: Cell<u32> = const { Cell::new(0) };
/// The references refused in the outermost of them, past [`MAX_FOLLOWED`].
static REFUSED: Cell<u32> = const { Cell::new(0) };
}
/// One read or write of an object, with everything it loads on the way: what [`MAX_FOLLOWED`] counts.
pub(crate) struct FollowScope;
impl FollowScope {
pub(crate) fn enter() -> Self {
if SCOPES.replace(SCOPES.get().saturating_add(1)) == 0 {
FOLLOWED.set(0);
REFUSED.set(0);
}
FollowScope
}
}
impl Drop for FollowScope {
fn drop(&mut self) {
SCOPES.set(SCOPES.get().saturating_sub(1));
}
}
/// What `f` gives, counted against [`MAX_FOLLOWED`], or `spent` past it. Called inside a
/// [`FollowScope`], which is what starts the count again.
pub(crate) fn following<T>(spent: T, f: impl FnOnce() -> T) -> T {
if FOLLOWED.get() >= MAX_FOLLOWED {
REFUSED.set(REFUSED.get().saturating_add(1));
return spent;
}
FOLLOWED.set(FOLLOWED.get() + 1);
f()
}
/// `o`, or the object it refers to, for what decides how a stream or dictionary is treated. Past
/// [`MAX_FOLLOWED`] a reference reads as absent.
fn followed<'o>(doc: &Document, o: &'o Object) -> Held<'o> {
match o {
Object::Ref(_) => following(Held::Direct(&Object::Null), || doc.held(o)),
o => Held::Direct(o),
}
}
fn bytes_of(doc: &Document, d: &Dict, key: &[u8]) -> Vec<u8> {
d.get(key).map(|v| doc.resolve(v)).and_then(|v| v.as_string().map(|s| s.bytes.clone())).unwrap_or_default()
}
/// Read the handler parameters from an `/Encrypt` dictionary.
pub(crate) fn encrypt_dict(doc: &Document, d: &Dict) -> EncryptDict {
let int = |k: &[u8], default: i64| d.get(k).map(|v| doc.resolve(v)).and_then(|v| v.as_int()).unwrap_or(default);
let name = |d: &Dict, k: &[u8]| d.get(k).and_then(|v| doc.resolve(v).as_name().map(<[u8]>::to_vec));
let mut crypt_filters = Vec::new();
if let Some(cf) = d.get(b"CF").map(|c| doc.resolve(c)).and_then(|c| c.as_dict().cloned()) {
for (k, v) in cf.iter() {
let Some(f) = doc.resolve(v).as_dict().cloned() else { continue };
let method = match name(&f, b"CFM").as_deref() {
Some(b"V2") => Method::Rc4,
Some(b"AESV2") => Method::Aes128,
Some(b"AESV3") => Method::Aes256,
_ => Method::Identity,
};
crypt_filters.push((k.clone(), method));
}
}
let v = int(b"V", 0);
// V4: the key length comes from the standard crypt filter's /Length (bytes, or bits in
// some files); /Length on the dictionary is often a stale 40. A length that isn't
// positive means nothing and reads as if absent.
let cf_len = d
.get(b"CF")
.map(|c| doc.resolve(c))
.and_then(|c| c.as_dict().and_then(|c| c.get(b"StdCF").map(|f| doc.resolve(f))))
.and_then(|f| f.as_dict().and_then(|f| f.get(b"Length").map(|n| doc.resolve(n))))
.and_then(|n| n.as_int())
.filter(|n| *n > 0);
let length_bits = match (v, cf_len) {
(5.., _) => 256,
(4, Some(n)) if n <= 32 => n * 8,
(4, Some(n)) => n,
(4, None) => 128,
_ => int(b"Length", 40),
};
EncryptDict {
filter: name(d, b"Filter").unwrap_or_default(),
v,
r: int(b"R", 0),
length_bits,
o: bytes_of(doc, d, b"O"),
u: bytes_of(doc, d, b"U"),
oe: bytes_of(doc, d, b"OE"),
ue: bytes_of(doc, d, b"UE"),
perms: bytes_of(doc, d, b"Perms"),
p: int(b"P", -1) as i32,
encrypt_metadata: !matches!(d.get(b"EncryptMetadata").map(|v| doc.resolve(v)).as_deref(), Some(Object::Bool(false))),
crypt_filters,
stm_f: name(d, b"StmF").unwrap_or_else(|| b"Identity".to_vec()),
str_f: name(d, b"StrF").unwrap_or_else(|| b"Identity".to_vec()),
ef_f: name(d, b"EFF").unwrap_or_default(),
}
}
/// Whether `d`, an `/Encrypt` dictionary, has crypt filters that encrypt (`/CFM` `V2`, `AESV2` or
/// `AESV3`), each authorised only when an embedded file is opened (`/AuthEvent /EFOpen`, ISO
/// 32000-2 §7.6.6, Table 25): what a document asks for its embedded files alone, which it opens
/// without (§7.6.6: "EFOpen Authorization shall be required when accessing embedded files"). The
/// files use them through `/EFF` or through a `/Crypt` filter of their own. Its strings and streams
/// have to be in the clear too, which [`SecurityHandler::without_key`] checks.
pub(crate) fn efopen_only(doc: &Document, d: &Dict) -> bool {
let name = |d: &Dict, k: &[u8]| d.get(k).and_then(|v| doc.resolve(v).as_name().map(<[u8]>::to_vec));
let Some(cf) = d.get(b"CF").map(|c| doc.resolve(c)).and_then(|c| c.as_dict().cloned()) else { return false };
let mut encrypting = Vec::new();
for (k, v) in cf.iter() {
let Some(f) = doc.resolve(v).as_dict().cloned() else { continue };
if !matches!(name(&f, b"CFM").as_deref(), Some(b"V2" | b"AESV2" | b"AESV3")) {
continue;
}
if name(&f, b"AuthEvent").as_deref() != Some(b"EFOpen") {
return false;
}
encrypting.push(k.clone());
}
!encrypting.is_empty()
}
/// Whether `s`, read from a document whose handler has no file key, holds data that handler would
/// decrypt, which therefore reads as stored: still encrypted.
pub(crate) fn stream_reads_as_stored(doc: &Document, h: &SecurityHandler, s: &Stream) -> bool {
if h.has_key() {
return false;
}
let _scope = FollowScope::enter();
let named = named_crypt_filter(doc, &s.dict);
let ty = s.dict.get(b"Type").map(|t| followed(doc, t));
let kind = match (&named, ty.as_deref().and_then(Object::as_name)) {
(Some(n), _) => StreamKind::Named(n),
(None, Some(b"EmbeddedFile")) => StreamKind::EmbeddedFile,
_ => StreamKind::Normal,
};
h.decrypts(kind)
}
/// Whether `f`, an entry of a `/Filter` chain or its only filter, is `/Crypt`.
fn is_crypt(doc: &Document, f: &Object) -> bool {
followed(doc, f).as_name() == Some(b"Crypt")
}
/// The crypt filter a stream names in its `/Filter` chain (`/Crypt` + `/DecodeParms /Name`), each
/// given directly or by reference, in an array or not. A chain longer than any decoder takes names
/// none: nothing can read that stream anyway, and every stream of a file can share one long array.
/// A `/Name` that reads as absent means `/Identity`, unless a reference on the way to it was refused
/// (see [`MAX_FOLLOWED`]): that says nothing about the stream, so it names none, and the document's
/// own filter is used.
fn named_crypt_filter(doc: &Document, dict: &Dict) -> Option<Vec<u8>> {
let refused = REFUSED.get();
let filter = followed(doc, dict.get(b"Filter")?);
let i = match &*filter {
Object::Array(a) if a.len() <= MAX_CHAIN + 1 => a.iter().position(|f| is_crypt(doc, f))?,
f if is_crypt(doc, f) => 0,
_ => return None,
};
let parms = dict.get(b"DecodeParms").map(|p| followed(doc, p));
let entry = match parms.as_deref() {
Some(Object::Array(a)) => a.get(i).map(|p| followed(doc, p)),
Some(p) if i == 0 => Some(Held::Direct(p)),
_ => None,
};
let name = entry.as_deref().and_then(Object::as_dict).and_then(|p| p.get(b"Name")).map(|n| followed(doc, n));
match name.as_deref().and_then(Object::as_name) {
Some(name) => Some(name.to_vec()),
// /Identity here would read ciphertext as data, and write the stream in the clear.
None if REFUSED.get() > refused => None,
None => Some(b"Identity".to_vec()),
}
}
/// A signature dictionary (field signature, document timestamp or usage-rights signature),
/// whose `/Contents` is never encrypted (ISO 32000-2 §7.6.2, §12.8.1): the signer writes a
/// placeholder, hashes the file around it and patches the value in place, so encrypting it
/// would make signing impossible. Identified as other readers do: by `/Type`, or by a string
/// `/Contents` alongside an array `/ByteRange` (`/Type` is optional). A signature dictionary
/// has no `/Subtype`; a dictionary with one (an annotation, whose `/Contents` is the comment
/// text) or with another `/Type` is never exempt, whatever else it carries.
/// With `doc` its entries are read through references; a dictionary nested in another object
/// has none (see [`transform`]).
fn is_signature_dict(doc: Option<&Document>, d: &Dict) -> bool {
// All this decides is what happens to such a /Contents.
if !matches!(d.get(b"Contents"), Some(Object::String(_))) {
return false;
}
// A null entry, or one that refers to nothing, is an absent one (ISO 32000-2 §7.3.7, §7.3.10).
let entry = |key: &[u8]| {
let value = d.get(key)?;
let held = doc.map_or(Held::Direct(value), |doc| followed(doc, value));
(!matches!(*held, Object::Null)).then_some(held)
};
if entry(b"Subtype").is_some() {
return false;
}
match entry(b"Type") {
Some(t) => matches!(t.as_name(), Some(b"Sig" | b"DocTimeStamp")),
None => matches!(entry(b"ByteRange").as_deref(), Some(Object::Array(_))),
}
}
/// Longer than any filter's name. A name can be of any length, and a chain every stream shares can
/// hold a long one.
const MAX_FILTER_NAME: usize = 127;
/// The dictionary of `source` without the `/Crypt` entries of its filter chain and their decode
/// parameters. The two come out direct when given by reference, which every stream of a file can
/// share, so of what was shared only small values are copied: filter names, and the parameters
/// decoders read ([`Document::with_direct_parms`]). The chain is short, as [`named_crypt_filter`]
/// found a filter in it.
fn drop_crypt_filter(doc: &Document, source: &Stream) -> Dict {
let entry = |f: &Object| match f {
f if is_crypt(doc, f) => Object::name("Crypt"),
Object::Name(n) if n.len() <= MAX_FILTER_NAME => f.clone(),
Object::Ref(_) => f.clone(),
_ => Object::Null,
};
let filter = source.dict.get(b"Filter").map(|f| followed(doc, f));
let chain: Vec<Object> = match filter.as_deref() {
Some(Object::Array(a)) => a.iter().map(entry).collect(),
Some(f) => vec![entry(f)],
None => Vec::new(),
};
let crypt: Vec<bool> = chain.iter().map(|f| f.as_name() == Some(b"Crypt")).collect();
let mut chained = Stream::from_raw(source.dict.clone(), Vec::new());
chained.dict.set(b"Filter".to_vec(), Object::Array(chain.clone()));
// A lone parameter dictionary belongs to the first filter (as `named_crypt_filter` reads it).
let parms = match doc.with_direct_parms(&chained).dict.get(b"DecodeParms") {
Some(Object::Array(p)) => Some(Object::Array(p.iter().zip(&crypt).filter(|(_, c)| !**c).map(|(p, _)| p.clone()).collect())),
Some(Object::Dict(_)) if crypt.first() == Some(&true) => None,
_ => source.dict.get(b"DecodeParms").cloned(),
};
let mut dict = source.dict.clone();
match parms {
Some(p) => dict.set(b"DecodeParms".to_vec(), p),
None => {
dict.remove(b"DecodeParms");
}
}
let filters: Vec<Object> = chain.into_iter().zip(&crypt).filter(|(_, c)| !**c).map(|(f, _)| f).collect();
if filters.is_empty() {
dict.remove(b"Filter");
if matches!(dict.get(b"DecodeParms"), Some(Object::Array(p)) if p.is_empty()) {
dict.remove(b"DecodeParms");
}
} else {
dict.set(b"Filter".to_vec(), Object::Array(filters));
}
dict
}
/// What [`transform`] does to an object.
#[derive(Clone, Copy, PartialEq, Eq)]
pub(crate) enum Pass {
/// Decrypt, with the handler the document was opened with.
Decrypt,
/// Encrypt with the handler the document was opened with: streams keep their crypt filters.
Encrypt,
/// Encrypt with protection applied by `set_encryption`. A stream's crypt filter came from
/// the source's `/Encrypt`; unless the new one defines that name it is dropped, so the
/// stream follows the new policy instead of reading as `/Identity` and staying in clear.
EncryptReplacing,
}
/// Decrypt or encrypt every string and stream in an indirect object, except a signature
/// dictionary's `/Contents` (see [`is_signature_dict`]).
///
/// `o` is as `doc` holds it: what a stream's dictionary gives by reference is read from `doc`, so
/// its references have to be the document's, not a save's renumbered ones.
pub(crate) fn transform(doc: &Document, h: &SecurityHandler, o: &Object, num: u32, generation: u16, pass: Pass) -> Object {
let _scope = FollowScope::enter();
let decrypt = pass == Pass::Decrypt;
let strings = |s: &PdfString| {
let bytes = if decrypt { h.decrypt_string(num, generation, &s.bytes) } else { h.encrypt_string(num, generation, &s.bytes) };
PdfString { bytes, hex: s.hex || !decrypt }
};
// References are followed here only, for the object itself, never for what is nested in it.
// `walk` goes deep, and a load from down there adds that depth again for each object loaded,
// which overflowed the stack on a 10 KB file. A stream can't be nested, and a dictionary that
// is reads as it says directly.
match o {
Object::Stream(s) => stream(doc, s, &strings, h, num, generation, pass),
Object::Dict(d) if is_signature_dict(Some(doc), d) => signature_dict(d, &strings),
o => walk(o, &strings),
}
}
/// `d`, a signature dictionary, with the strings of everything but its `/Contents` transformed.
fn signature_dict(d: &Dict, strings: &dyn Fn(&PdfString) -> PdfString) -> Object {
Object::Dict(
d.iter()
.map(|(k, v)| match (k.as_slice(), v) {
(b"Contents", Object::String(_)) => (k.clone(), v.clone()),
_ => (k.clone(), walk(v, strings)),
})
.collect(),
)
}
fn walk(o: &Object, strings: &dyn Fn(&PdfString) -> PdfString) -> Object {
match o {
Object::String(s) => Object::String(strings(s)),
Object::Array(a) => Object::Array(a.iter().map(|x| walk(x, strings)).collect()),
Object::Dict(d) if is_signature_dict(None, d) => signature_dict(d, strings),
Object::Dict(d) => Object::Dict(d.iter().map(|(k, v)| (k.clone(), walk(v, strings))).collect()),
other => other.clone(),
}
}
fn stream(
doc: &Document,
s: &Stream,
strings: &dyn Fn(&PdfString) -> PdfString,
h: &SecurityHandler,
num: u32,
generation: u16,
pass: Pass,
) -> Object {
let decrypt = pass == Pass::Decrypt;
let walked = |d: &Dict| -> Dict { d.iter().map(|(k, v)| (k.clone(), walk(v, strings))).collect() };
let ty = s.dict.get(b"Type").map(|t| followed(doc, t));
let ty = ty.as_deref().and_then(Object::as_name);
// Cross-reference streams are never encrypted (§7.5.8), but only reading may trust the
// label: a stream a producer calls /XRef stays plaintext if writing believes it, and the
// writer builds its own cross-reference streams outside this transform.
if (decrypt && ty == Some(b"XRef")) || (ty == Some(b"Metadata") && !h.encrypts_metadata()) {
return Object::Stream(Stream { dict: walked(&s.dict), raw: s.raw.clone() });
}
let mut named = named_crypt_filter(doc, &s.dict);
// Before the walk, which then encrypts what is in the dictionary this leaves.
let foreign = pass == Pass::EncryptReplacing && named.as_deref().is_some_and(|n| !h.dict().crypt_filters.iter().any(|(f, _)| f == n));
let dropped = foreign.then(|| drop_crypt_filter(doc, s));
if dropped.is_some() {
named = None;
}
let dict = walked(dropped.as_ref().unwrap_or(&s.dict));
let kind = match (&named, ty) {
(Some(n), _) => StreamKind::Named(n),
(None, Some(b"EmbeddedFile")) => StreamKind::EmbeddedFile,
_ => StreamKind::Normal,
};
let raw = if decrypt { h.decrypt_stream(num, generation, &s.raw, kind) } else { h.encrypt_stream(num, generation, &s.raw, kind) };
Object::Stream(Stream { dict, raw: raw.into() })
}
+358
View File
@@ -0,0 +1,358 @@
//! Synthetic PDF fixtures for the object layer's tests: one way to write the bytes, so tests
//! differ only where they mean to (from PdfCraft's `pdfcraft-testkit`).
//!
//! [`Pdf`] writes the envelope around the object bodies a test gives it: object numbers, offsets,
//! a classic cross-reference table and a trailer. It hands back bytes rather than a parsed
//! document, so a test checks the parser rather than the builder.
//!
//! ```text
//! let bytes = Pdf::new()
//! .obj("<< /Type /Catalog /Pages 2 0 R >>")
//! .obj("<< /Type /Pages /Kids [3 0 R] /Count 1 >>")
//! .obj("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 300] /Contents 4 0 R >>")
//! .stream("", b"BT /F1 12 Tf (hi) Tj ET")
//! .build();
//! ```
//!
//! Test scaffolding: compiled into the unit tests (`cfg(test)`) and into the `cos` integration
//! test binary, never into the library.
#![allow(dead_code)]
/// A small PDF with a classic cross-reference table, built one object at a time.
///
/// Objects are numbered from 1 in the order they are added, all at generation 0, so the first
/// object added is `1 0 R`. The trailer names `1 0 R` as `/Root`, which is what every caller of
/// this builder wants; [`Pdf::trailer_entry`] adds the rest.
#[derive(Debug, Clone)]
pub struct Pdf {
version: String,
objects: Vec<Vec<u8>>,
trailer: Vec<String>,
}
impl Default for Pdf {
fn default() -> Self {
Self::new()
}
}
impl Pdf {
/// An empty PDF 1.7 file. Add objects with [`Pdf::obj`].
pub fn new() -> Self {
Self { version: "1.7".to_owned(), objects: Vec::new(), trailer: Vec::new() }
}
/// Set the header version, as it appears after `%PDF-` (for example `"1.4"`).
#[must_use]
pub fn version(mut self, version: &str) -> Self {
self.version = version.to_owned();
self
}
/// Append one object body — everything between `N 0 obj` and `endobj`.
#[must_use]
pub fn obj(mut self, body: impl AsRef<[u8]>) -> Self {
self.objects.push(body.as_ref().to_vec());
self
}
/// Append several object bodies, in order.
#[must_use]
pub fn objs<I: IntoIterator<Item = T>, T: AsRef<[u8]>>(mut self, bodies: I) -> Self {
self.objects.extend(bodies.into_iter().map(|b| b.as_ref().to_vec()));
self
}
/// Append a stream object: `dict` holds any entries besides `/Length`, which is computed.
#[must_use]
pub fn stream(self, dict: &str, data: impl AsRef<[u8]>) -> Self {
self.obj(stream_obj(dict, data))
}
/// Add an entry to the trailer dictionary, such as `"/Info 9 0 R"`. `/Size` and `/Root` are
/// always written.
#[must_use]
pub fn trailer_entry(mut self, entry: &str) -> Self {
self.trailer.push(entry.to_owned());
self
}
/// The number the next object added will get, so a caller can refer forward to it.
///
/// ```
/// # use crate::cos::testkit::Pdf;
/// let mut p = Pdf::new().obj("<< /Type /Catalog /Pages 2 0 R >>");
/// assert_eq!(p.next_obj(), 2);
/// ```
pub fn next_obj(&self) -> usize {
self.objects.len().saturating_add(1)
}
/// Serialise the file.
pub fn build(&self) -> Vec<u8> {
let mut out = format!("%PDF-{}\n", self.version).into_bytes();
let mut offsets = Vec::with_capacity(self.objects.len());
for (i, body) in self.objects.iter().enumerate() {
offsets.push(out.len());
out.extend_from_slice(format!("{} 0 obj\n", i.saturating_add(1)).as_bytes());
out.extend_from_slice(body);
out.extend_from_slice(b"\nendobj\n");
}
let startxref = out.len();
let size = self.objects.len().saturating_add(1);
out.extend_from_slice(format!("xref\n0 {size}\n0000000000 65535 f \n").as_bytes());
for offset in offsets {
out.extend_from_slice(format!("{offset:010} 00000 n \n").as_bytes());
}
let extra = self.trailer.iter().map(|e| format!(" {e}")).collect::<String>();
out.extend_from_slice(format!("trailer\n<< /Size {size} /Root 1 0 R{extra} >>\nstartxref\n{startxref}\n%%EOF\n").as_bytes());
out
}
}
/// A stream object body with `/Length` filled in, for callers assembling object lists themselves.
///
/// `dict` holds any entries besides `/Length`; an empty one gives `<< /Length N >>` rather than
/// the doubled space the hand-rolled copies produced, which parsers accept but readers trip over.
pub fn stream_obj(dict: &str, data: impl AsRef<[u8]>) -> Vec<u8> {
let data = data.as_ref();
let dict = dict.trim();
let sep = if dict.is_empty() { "" } else { " " };
let mut out = format!("<<{sep}{dict} /Length {} >>\nstream\n", data.len()).into_bytes();
out.extend_from_slice(data);
out.extend_from_slice(b"\nendstream");
out
}
/// A three-page file (200, 300 and 400 points wide, in that order, the second with a link to
/// `https://example.com/`) whose page tree sits in an object stream of its first revision, plus an update that rewrites the catalog and the first
/// page in a cross-reference stream: with `/Prev` when `linked`, without it otherwise (the shape
/// of the signed PDFBOX-2720 and PDFBOX-2942 files). Without `/Prev` a reader that follows the
/// chain (ISO 32000-2 §7.5.6, §7.5.8.2) finds no page tree, and a scan for page dictionaries
/// outside object streams finds only the rewritten page.
pub fn update_without_prev(linked: bool) -> Vec<u8> {
let (mut out, eight) = unlinked_original();
// The update: the catalog and the first page again, and a section for them and itself.
let (catalog, first) = unlinked_rewrites(&mut out);
let nine = out.len();
let mut rows = Vec::new();
xref_row(&mut rows, 1, catalog, 0);
xref_row(&mut rows, 1, first, 0);
xref_row(&mut rows, 1, nine, 0);
let prev = if linked { format!(" /Prev {eight}") } else { String::new() };
out.extend_from_slice(
format!("9 0 obj\n<< /Type /XRef /Size 11 /Root 1 0 R /Index [1 1 3 1 9 1] /W [1 4 2]{prev} /Length {} >>\nstream\n", rows.len()).as_bytes(),
);
out.extend_from_slice(&rows);
out.extend_from_slice(format!("\nendstream\nendobj\nstartxref\n{nine}\n%%EOF\n").as_bytes());
out
}
/// [`update_without_prev`] with the update written as a hybrid-reference section (ISO 32000-2
/// §7.5.8.4): a classic table whose trailer names a cross-reference stream with `/XRefStm`. That
/// stream belongs to the same update; it is not an earlier section, so without `/Prev` the page
/// tree is cut off just the same.
pub fn hybrid_update_without_prev(linked: bool) -> Vec<u8> {
let (mut out, eight) = unlinked_original();
let (catalog, first) = unlinked_rewrites(&mut out);
// The update's cross-reference stream, locating only itself.
let nine = out.len();
let mut rows = Vec::new();
xref_row(&mut rows, 1, nine, 0);
out.extend_from_slice(format!("9 0 obj\n<< /Type /XRef /Size 11 /Index [9 1] /W [1 4 2] /Length {} >>\nstream\n", rows.len()).as_bytes());
out.extend_from_slice(&rows);
out.extend_from_slice(b"\nendstream\nendobj\n");
let table = out.len();
out.extend_from_slice(format!("xref\n1 1\n{catalog:010} 00000 n \n3 1\n{first:010} 00000 n \n").as_bytes());
let prev = if linked { format!(" /Prev {eight}") } else { String::new() };
out.extend_from_slice(format!("trailer\n<< /Size 11 /Root 1 0 R /XRefStm {nine}{prev} >>\nstartxref\n{table}\n%%EOF\n").as_bytes());
out
}
/// One row of a cross-reference stream with `/W [1 4 2]`.
fn xref_row(out: &mut Vec<u8>, kind: u8, field: usize, index: u16) {
out.push(kind);
out.extend_from_slice(&u32::try_from(field).unwrap_or(u32::MAX).to_be_bytes());
out.extend_from_slice(&index.to_be_bytes());
}
/// The update both fixtures append: the catalog and the first page again. Their offsets.
fn unlinked_rewrites(out: &mut Vec<u8>) -> (usize, usize) {
let catalog = out.len();
out.extend_from_slice(b"1 0 obj\n<< /Type /Catalog /Pages 2 0 R /Lang (en) >>\nendobj\n");
let first = out.len();
out.extend_from_slice(b"3 0 obj\n<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 4 0 R /Annots [] >>\nendobj\n");
(catalog, first)
}
/// The first revision of [`update_without_prev`], and the offset of its cross-reference stream.
fn unlinked_original() -> (Vec<u8>, usize) {
let row = xref_row;
let page = |width: u32, extra: &str| format!("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 {width} 200] /Contents 4 0 R{extra} >>");
let mut out = b"%PDF-1.7\n".to_vec();
let one = out.len();
out.extend_from_slice(b"1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n");
let four = out.len();
out.extend_from_slice(b"4 0 obj\n<< /Length 14 >>\nstream\n0 0 50 50 re f\nendstream\nendobj\n");
// Objects 2 (the page tree), 3, 6 and 7 (the pages) and 10 (the link) in object stream 5.
let members = [
(2, "<< /Type /Pages /Kids [3 0 R 6 0 R 7 0 R] /Count 3 >>".to_string()),
(3, page(200, "")),
(6, page(300, " /Annots [10 0 R]")),
(7, page(400, "")),
(10, "<< /Type /Annot /Subtype /Link /Rect [10 10 60 60] /A << /S /URI /URI (https://example.com/) >> >>".to_string()),
];
let (mut header, mut body) = (String::new(), String::new());
for (num, member) in &members {
header.push_str(&format!("{num} {} ", body.len()));
body.push_str(member);
body.push(' ');
}
let five = out.len();
out.extend_from_slice(
format!(
"5 0 obj\n<< /Type /ObjStm /N 5 /First {} /Length {} >>\nstream\n{header}{body}\nendstream\nendobj\n",
header.len(),
header.len() + body.len()
)
.as_bytes(),
);
let eight = out.len();
let mut rows = Vec::new();
row(&mut rows, 0, 0, 65535);
row(&mut rows, 1, one, 0);
row(&mut rows, 2, 5, 0);
row(&mut rows, 2, 5, 1);
row(&mut rows, 1, four, 0);
row(&mut rows, 1, five, 0);
row(&mut rows, 2, 5, 2);
row(&mut rows, 2, 5, 3);
row(&mut rows, 1, eight, 0);
row(&mut rows, 0, 0, 0);
row(&mut rows, 2, 5, 4);
out.extend_from_slice(format!("8 0 obj\n<< /Type /XRef /Size 11 /Root 1 0 R /W [1 4 2] /Length {} >>\nstream\n", rows.len()).as_bytes());
out.extend_from_slice(&rows);
out.extend_from_slice(format!("\nendstream\nendobj\nstartxref\n{eight}\n%%EOF\n").as_bytes());
(out, eight)
}
#[cfg(test)]
mod tests {
use super::*;
/// The serialiser the workspace had 111 copies of, kept here verbatim as the thing [`Pdf`] has
/// to agree with. Migrating a crate to the builder must not change a single byte of its
/// fixtures, or the test that fixture feeds silently starts testing something else.
fn hand_rolled(version: &str, objs: &[Vec<u8>], trailer_extra: &str) -> Vec<u8> {
let mut out = format!("%PDF-{version}\n").into_bytes();
let mut offs = Vec::new();
for (i, o) in objs.iter().enumerate() {
offs.push(out.len());
out.extend_from_slice(format!("{} 0 obj\n", i + 1).as_bytes());
out.extend_from_slice(o);
out.extend_from_slice(b"\nendobj\n");
}
let x = out.len();
out.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
for o in offs {
out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
}
out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R{trailer_extra} >>\nstartxref\n{x}\n%%EOF\n", objs.len() + 1).as_bytes());
out
}
fn sample_objects() -> Vec<Vec<u8>> {
vec![
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 612 792] >>".to_vec(),
b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R >>".to_vec(),
stream_obj("", b"BT /F1 24 Tf 20 150 Td (Page 1) Tj ET"),
]
}
#[test]
fn the_builder_is_byte_identical_to_the_serialiser_it_replaces() {
let objs = sample_objects();
for version in ["1.4", "1.7", "2.0"] {
let built = Pdf::new().version(version).objs(objs.clone()).build();
assert_eq!(built, hand_rolled(version, &objs, ""), "version {version}");
}
// ...including the trailer entries the copies varied on.
let built = Pdf::new().objs(objs.clone()).trailer_entry("/Info 5 0 R").build();
assert_eq!(built, hand_rolled("1.7", &objs, " /Info 5 0 R"));
}
/// Offsets must point at the `N 0 obj` that starts each object, counted from the file's first
/// byte. A fixture whose table is wrong tests the repair path, not the path it meant to.
#[test]
fn every_offset_in_the_table_points_at_its_object() {
let bytes = Pdf::new().objs(sample_objects()).build();
let text = String::from_utf8_lossy(&bytes);
let table = text.split("xref\n0 ").nth(1).expect("an xref table");
let rows: Vec<&str> = table.lines().skip(1).take_while(|l| l.ends_with(" n ") || l.ends_with(" f ")).collect();
assert_eq!(rows.len(), 5, "free entry plus four objects: {rows:?}");
for (i, row) in rows.iter().enumerate().skip(1) {
let offset: usize = row.split_whitespace().next().and_then(|n| n.parse().ok()).expect("an offset");
let expected = format!("{i} 0 obj\n");
assert!(bytes.get(offset..).is_some_and(|rest| rest.starts_with(expected.as_bytes())), "object {i} at {offset}");
}
}
/// `/Size` is one more than the object count (object 0 is the free head), and `startxref`
/// points at the table.
#[test]
fn the_trailer_agrees_with_the_body() {
let bytes = Pdf::new().objs(sample_objects()).build();
let text = String::from_utf8_lossy(&bytes);
assert!(text.contains("/Size 5"), "{text}");
let startxref: usize = text.rsplit("startxref\n").next().and_then(|t| t.lines().next()).and_then(|n| n.parse().ok()).expect("startxref");
assert!(bytes.get(startxref..).is_some_and(|rest| rest.starts_with(b"xref\n")), "startxref {startxref}");
}
#[test]
fn stream_length_counts_the_data_not_the_keywords() {
let obj = stream_obj("/Filter /FlateDecode", b"abc");
assert_eq!(obj, b"<< /Filter /FlateDecode /Length 3 >>\nstream\nabc\nendstream".to_vec());
// Binary data, including NUL and bytes that are not UTF-8, survives untouched.
let data = [0u8, 0xFF, b'\n', 0x80];
let obj = stream_obj("", data);
assert!(obj.ends_with(b"stream\n\0\xff\n\x80\nendstream"), "{obj:?}");
// An empty dict gives one space, not two.
assert!(obj.starts_with(b"<< /Length 4 >>"), "{obj:?}");
assert_eq!(stream_obj(" /Type /XObject ", b""), stream_obj("/Type /XObject", b""));
}
/// An empty file is still a parseable shape rather than something that panics or truncates.
#[test]
fn a_file_with_no_objects_is_still_well_formed() {
let bytes = Pdf::new().build();
let text = String::from_utf8_lossy(&bytes);
assert!(text.starts_with("%PDF-1.7\n"), "{text}");
assert!(text.contains("xref\n0 1\n0000000000 65535 f \n"), "{text}");
assert!(text.contains("/Size 1 /Root 1 0 R"), "{text}");
assert!(text.ends_with("%%EOF\n"), "{text}");
}
#[test]
fn next_obj_tracks_what_has_been_added() {
let p = Pdf::new();
assert_eq!(p.next_obj(), 1);
let p = p.obj("<< >>").obj("<< >>");
assert_eq!(p.next_obj(), 3);
// Forward references are the reason this exists: the page can name a stream not yet added.
let n = p.next_obj();
let bytes = p.obj(format!("<< /Contents {n} 0 R >>")).build();
assert!(String::from_utf8_lossy(&bytes).contains("/Contents 3 0 R"));
}
/// Hostile and awkward object bodies must not panic or corrupt the offsets.
#[test]
fn odd_object_bodies_keep_the_table_honest() {
let bodies: Vec<Vec<u8>> = vec![Vec::new(), b"\n".to_vec(), vec![0u8; 1024], "<< /T (\u{1F4A5}) >>".as_bytes().to_vec(), b"endobj".to_vec()];
let bytes = Pdf::new().objs(bodies.clone()).build();
assert_eq!(bytes, hand_rolled("1.7", &bodies, ""));
// The offsets still land on their objects even when a body contains `endobj` itself.
let text = String::from_utf8_lossy(&bytes);
assert!(text.contains("/Size 6"), "{text}");
}
}
File diff suppressed because it is too large Load Diff
+4 -4
View File
@@ -5,9 +5,9 @@
//! TIFF/PNG predictors, `ASCIIHexDecode`, `ASCII85Decode`, `RunLengthDecode`).
//! Image codecs (`DCTDecode`, `JPXDecode`, `JBIG2Decode`, `CCITTFaxDecode`),
//! `Crypt` and unknown filters are reported as [`FilterError::Unsupported`] so
//! callers keep such data encoded. CCITT fax images decode on their own
//! ([`decode_ccitt_within`]), spending from an allowance the caller shares across a run of
//! images.
//! callers keep such data encoded. CCITT fax and JBIG2 images decode on their own
//! ([`decode_ccitt_within`], [`decode_jbig2_within`]), each spending from an allowance the
//! caller shares across a run of images.
//!
//! The codecs themselves are `craft-stream-codecs`, re-exported here; this module adds Flate,
//! the chain (stage order, limits across stages, strict and tolerant decoding) and its
@@ -24,7 +24,7 @@ use std::borrow::Cow;
pub use craft_stream_codecs::{
CcittParams, CodecError as FilterError, MAX_CCITT_COLUMNS, MAX_G4_COLUMNS, MAX_JBIG2_COLUMNS, MAX_JBIG2_PIXELS, Params, SharedJbig2,
decode_ccitt_within, encode_g4, encode_jbig2, encode_jbig2_generic, encode_jbig2_shared,
decode_ccitt_within, decode_jbig2_within, encode_g4, encode_jbig2, encode_jbig2_generic, encode_jbig2_shared,
};
use craft_stream_codecs::{Failure, Step, pay};
use craft_stream_codecs::{ascii85, asciihex, lzw, predictor, runlength};
+2
View File
@@ -4,6 +4,8 @@
//! Advanced backend re-exports are coupled to the pinned backend, not a universal document model.
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
#[cfg(feature = "cos")]
pub mod cos;
#[cfg(feature = "crypt")]
pub mod crypt;
#[cfg(feature = "filters")]
+236
View File
@@ -0,0 +1,236 @@
//! Full rewrites must preserve the graph when temporary parse/packing caches are evicted.
//! Fixtures are generated here; no third-party PDF assets are used.
use std::collections::{BTreeMap, HashSet};
use std::sync::Arc;
use craft_pdf::cos::{Algorithm, Dict, Document, NewEncryption, ObjRef, Object, PdfString, SaveOptions, XrefEntry, write_full};
const NODES: usize = 260;
const EDITED: usize = 129;
fn node(i: usize) -> ObjRef {
ObjRef::new(301 + i as u32, (i % 5 + 1) as u16)
}
fn stream_body(i: usize) -> Vec<u8> {
format!("opaque stream {i}: preserve these bytes\n").into_bytes()
}
/// Nonzero generations, sparse numbers, shared references, a cycle and interleaved streams.
/// A raw classic xref avoids depending on the writer under test to construct the input.
fn fixture() -> Vec<u8> {
let mut objects = vec![
(ObjRef::new(10, 2), format!("<< /Type /Catalog /Pages 20 0 R /Extension {} /Shared 42 3 R /Freed 43 8 R /Missing 9000 0 R >>", node(0))),
(ObjRef::new(20, 0), "<< /Type /Pages /Kids [] /Count 0 >>".into()),
(ObjRef::new(42, 3), "<< /Token (shared extension value) >>".into()),
(ObjRef::new(43, 8), "<< /Token (deleted extension value) >>".into()),
(ObjRef::new(70, 4), "<< /Title (original title) >>".into()),
];
for i in 0..NODES {
let fields = format!("/Value {i} /Next {} /Shared 42 3 R /Opaque << /Nested [42 3 R (retained data)] >>", node((i + 1) % NODES));
let body = if i % 19 == 0 {
let raw = stream_body(i);
format!("<< {fields} /Length {} >>\nstream\n{}endstream", raw.len(), String::from_utf8(raw).unwrap())
} else {
format!("<< {fields} >>")
};
objects.push((node(i), body));
}
let mut bytes = b"%PDF-1.7\n".to_vec();
let mut offsets = BTreeMap::new();
for (reference, body) in objects {
offsets.insert(reference.num, (bytes.len(), reference.generation));
bytes.extend_from_slice(format!("{} {} obj\n{body}\nendobj\n", reference.num, reference.generation).as_bytes());
}
let size = offsets.last_key_value().unwrap().0 + 1;
let xref = bytes.len();
bytes.extend_from_slice(format!("xref\n0 {size}\n").as_bytes());
for n in 0..size {
match offsets.get(&n) {
Some((offset, generation)) => bytes.extend_from_slice(format!("{offset:010} {generation:05} n \n").as_bytes()),
None => bytes.extend_from_slice(b"0000000000 65535 f \n"),
}
}
bytes.extend_from_slice(format!("trailer\n<< /Size {size} /Root 10 2 R /Info 70 4 R >>\nstartxref\n{xref}\n%%EOF\n").as_bytes());
bytes
}
fn edited_document() -> Document {
let mut doc = Document::open(Arc::new(fixture())).unwrap();
assert!(doc.repair_log().is_empty());
assert_eq!(doc.root(), Some(ObjRef::new(10, 2)));
assert_eq!(doc.generation(node(EDITED).num), node(EDITED).generation);
doc.update_dict(node(EDITED), |d| d.set(b"Value".to_vec(), Object::Int(8129))).unwrap();
doc.free(ObjRef::new(43, 8));
let added = doc.add(Object::String(PdfString::text("added extension value")));
doc.update_dict(doc.root().unwrap(), |d| d.set(b"Added".to_vec(), Object::Ref(added))).unwrap();
doc
}
fn check_graph(doc: &Document) {
assert!(doc.repair_log().is_empty(), "{:?}", doc.repair_log());
let catalog = doc.get(doc.root().unwrap());
let catalog = catalog.as_dict().unwrap();
let shared = catalog.reference(b"Shared").unwrap();
let shared_object = doc.get(shared);
assert_eq!(shared_object.as_dict().unwrap().get(b"Token").unwrap().as_string().unwrap().to_text(), "shared extension value");
assert_eq!(catalog.get(b"Freed"), Some(&Object::Null));
assert_eq!(catalog.get(b"Missing"), Some(&Object::Null));
assert_eq!(doc.resolve(catalog.get(b"Added").unwrap()).as_string().unwrap().to_text(), "added extension value");
let first = catalog.reference(b"Extension").unwrap();
let mut current = first;
let mut visited = HashSet::new();
for i in 0..NODES {
assert!(visited.insert(current), "cycle closed before node {i}");
let object = doc.get(current);
let dict = object.as_dict().unwrap();
assert_eq!(dict.int(b"Value"), Some(if i == EDITED { 8129 } else { i as i64 }));
assert_eq!(dict.reference(b"Shared"), Some(shared));
let opaque = dict.get(b"Opaque").unwrap().as_dict().unwrap().get(b"Nested").unwrap().as_array().unwrap();
assert_eq!(opaque.first(), Some(&Object::Ref(shared)));
assert_eq!(opaque.get(1).unwrap().as_string().unwrap().to_text(), "retained data");
if i % 19 == 0 {
let Object::Stream(stream) = object.as_ref() else { panic!("node {i} lost its stream") };
assert_eq!(stream.decoded().unwrap(), stream_body(i));
} else {
assert!(matches!(object.as_ref(), Object::Dict(_)));
}
current = dict.reference(b"Next").unwrap();
}
assert_eq!(current, first, "the extension cycle is preserved");
assert!(doc.object_numbers().into_iter().all(|n| doc.generation(n) == 0));
}
#[test]
fn full_rewrites_preserve_graph_across_multiple_object_streams_and_evictions() {
for object_streams in [false, true] {
let doc = edited_document();
let opts = SaveOptions { object_streams, mod_date: Some("D:20261008000000Z".into()), ..SaveOptions::default() };
let saved = write_full(&doc, &opts).unwrap();
assert_eq!(saved, write_full(&doc, &opts).unwrap(), "unchanged inputs produce identical bytes");
let back = Document::open(Arc::new(saved)).unwrap();
check_graph(&back);
let info = back.resolve(back.trailer().get(b"Info").unwrap());
let info = info.as_dict().unwrap();
assert_eq!(info.get(b"Title").unwrap().as_string().unwrap().to_text(), "original title");
assert_eq!(info.get(b"ModDate").unwrap().as_string().unwrap().to_text(), "D:20261008000000Z");
let containers: HashSet<_> = back
.object_numbers()
.into_iter()
.filter_map(|num| match back.xref_entry(num) {
Some(XrefEntry::InStream { stream, .. }) => Some(stream),
_ => None,
})
.collect();
if object_streams {
assert!(containers.len() >= 3, "fixture must span multiple packed-object chunks");
} else {
assert!(containers.is_empty());
}
// Read an already packed input as well: eviction must not lose decoded ObjStm data.
check_graph(&Document::open(Arc::new(write_full(&back, &opts).unwrap())).unwrap());
// Writing must not apply the date stamp or consume the caller's edits.
let original_info = doc.resolve(doc.trailer().get(b"Info").unwrap());
assert!(!original_info.as_dict().unwrap().contains(b"ModDate"));
assert!(doc.is_modified());
assert_eq!(doc.get(node(EDITED)).as_dict().unwrap().int(b"Value"), Some(8129));
}
}
fn protect(doc: &mut Document, algorithm: Algorithm) {
doc.set_encryption(&NewEncryption {
algorithm,
user_password: "user",
owner_password: "owner",
permissions: -4,
encrypt_metadata: true,
seed: [23; 32],
})
.unwrap();
}
#[test]
fn encrypted_full_rewrites_preserve_data_across_evictions() {
for algorithm in [Algorithm::Rc4_40, Algorithm::Rc4_128, Algorithm::Aes128, Algorithm::Aes256] {
let mut doc = edited_document();
protect(&mut doc, algorithm);
let encrypted = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap());
let cold = Document::open_with_password(encrypted, Some("user")).unwrap();
// The second rewrite must decrypt lazily from the input, then encrypt the new streams.
let rewritten = Arc::new(write_full(&cold, &SaveOptions::default()).unwrap());
assert!(Document::open_with_password(rewritten.clone(), Some("wrong")).is_err());
let back = Document::open_with_password(rewritten, Some("owner")).unwrap();
assert!(back.security().is_some());
check_graph(&back);
}
}
#[test]
fn encryption_dependencies_stay_outside_object_streams_across_evictions() {
let mut doc = edited_document();
protect(&mut doc, Algorithm::Aes256);
let encrypt = doc.trailer().reference(b"Encrypt").unwrap();
let filters = doc.get(encrypt).as_dict().unwrap().get(b"CF").cloned().unwrap();
let filters = doc.add(filters);
let mut next = filters;
for i in (0..140).rev() {
let mut dict = Dict::new();
dict.set(b"Value".to_vec(), Object::Int(i));
dict.set(b"Next".to_vec(), Object::Ref(next));
next = doc.add(dict);
}
doc.update_dict(encrypt, |dict| {
dict.set(b"CF".to_vec(), Object::Ref(filters));
dict.set(b"PrivateChain".to_vec(), Object::Ref(next));
})
.unwrap();
let saved = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap());
let cold = Document::open_with_password(saved, Some("user")).unwrap();
let rewritten = Arc::new(write_full(&cold, &SaveOptions::default()).unwrap());
let back = Document::open_with_password(rewritten, Some("user")).unwrap();
check_graph(&back);
let encrypt = back.trailer().reference(b"Encrypt").unwrap();
assert!(matches!(back.xref_entry(encrypt.num), Some(XrefEntry::InFile { .. })));
let encrypt = back.get(encrypt);
let encrypt = encrypt.as_dict().unwrap();
let filters = encrypt.reference(b"CF").unwrap();
assert!(matches!(back.xref_entry(filters.num), Some(XrefEntry::InFile { .. })));
let mut current = encrypt.reference(b"PrivateChain").unwrap();
for i in 0..140 {
assert!(matches!(back.xref_entry(current.num), Some(XrefEntry::InFile { .. })));
let object = back.get(current);
let dict = object.as_dict().unwrap();
assert_eq!(dict.int(b"Value"), Some(i));
current = dict.reference(b"Next").unwrap();
}
assert_eq!(current, filters);
}
#[test]
fn large_packed_strings_round_trip_across_byte_limited_batches() {
let mut doc = Document::new_empty();
let mut references = Vec::new();
// More than 8 MiB in fewer than 100 objects: a byte cap may flush before the count cap.
for i in 0..70u8 {
let mut bytes = vec![b'a' + i % 26; 192 * 1024];
bytes[0] = i;
references.push(Object::Ref(doc.add(Object::String(PdfString::literal(bytes)))));
}
doc.update_dict(doc.root().unwrap(), |dict| dict.set(b"Extension".to_vec(), Object::Array(references))).unwrap();
let source = Arc::new(write_full(&doc, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap());
drop(doc);
let cold = Document::open(source).unwrap();
let back = Document::open(Arc::new(write_full(&cold, &SaveOptions::default()).unwrap())).unwrap();
assert!(back.repair_log().is_empty());
let catalog = back.get(back.root().unwrap());
let values = catalog.as_dict().unwrap().get(b"Extension").unwrap().as_array().unwrap();
assert_eq!(values.len(), 70);
for (i, reference) in values.iter().enumerate() {
let value = back.resolve(reference);
let bytes = &value.as_string().unwrap().bytes;
assert_eq!(bytes.len(), 192 * 1024);
assert_eq!(bytes[0], i as u8);
assert!(bytes[1..].iter().all(|&b| b == b'a' + i as u8 % 26));
}
}
+180
View File
@@ -0,0 +1,180 @@
//! Corpus parity: every file hayro-syntax can open must also open with our reader, agree on the
//! page count, and survive an incremental save that touches the catalog (the appended revision
//! must be readable by both readers and leave the original bytes untouched).
//!
//! Runs over `corpus/pdfjs/test/pdfs` at the repository root when present (PdfCraft's
//! `cargo xtask corpus` fetches and verifies it; copy or link its `corpus/pdfjs`); when it is
//! absent these are a no-op, so CI without the corpus stays green. Run with
//! `--ignored --nocapture`.
//!
//! A corpus that is present but **unverified** is a failure, not a skip. `xtask corpus` writes
//! `corpus/pdfjs/.pdfcraft-corpus.json` only after the checkout matches the pinned commit and
//! manifest sha256, so these tests refuse to run against a tree nobody has checked. Without that,
//! a half-finished or tampered checkout would quietly produce green results.
use std::sync::Arc;
use craft_pdf::cos::{Document, Object, SaveOptions, write_full, write_incremental};
/// `corpus/pdfjs`, or `None` when nothing has been fetched.
///
/// Panics when the checkout exists without the stamp that `cargo xtask corpus` writes on a
/// successful verification: an unverified corpus must fail loudly rather than skip.
fn checkout() -> Option<std::path::PathBuf> {
let dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../corpus/pdfjs");
if !dir.is_dir() {
return None;
}
let stamp = dir.join(".pdfcraft-corpus.json");
assert!(
stamp.is_file(),
"{} exists but {} does not: the corpus has not been verified against the pin. \
Run `cargo xtask corpus` (it fetches and verifies), or delete corpus/pdfjs to skip these tests.",
dir.display(),
stamp.display()
);
Some(dir)
}
/// Leaf pages reachable from the catalog (cycle-safe), like viewers count them.
fn page_count(doc: &Document) -> Option<usize> {
let root = doc.get(doc.root()?);
let mut stack = vec![root.as_dict()?.get(b"Pages")?.clone()];
let (mut seen, mut n) = (std::collections::HashSet::new(), 0);
while let Some(o) = stack.pop() {
if let Object::Ref(r) = o
&& !seen.insert(r)
{
continue;
}
let node = doc.resolve(&o);
let Some(d) = node.as_dict() else { continue };
match d.get(b"Kids").map(|k| doc.resolve(k)) {
Some(k) if d.name(b"Type") != Some(b"Page") => stack.extend(k.as_array().cloned().unwrap_or_default().into_iter().rev()),
_ => n += 1,
}
}
Some(n)
}
#[test]
#[ignore = "needs corpus/; run with --ignored"]
fn corpus_parity() {
let Some(root) = checkout() else {
eprintln!("corpus not fetched; skipping (cargo xtask corpus)");
return;
};
let dir = root.join("test/pdfs");
let entries = std::fs::read_dir(&dir).unwrap_or_else(|e| panic!("verified corpus has no {}: {e}", dir.display()));
let mut files: Vec<_> = entries.flatten().map(|e| e.path()).filter(|p| p.extension().is_some_and(|e| e == "pdf")).collect();
files.sort();
let (mut checked, mut encrypted, mut failures) = (0, 0, Vec::new());
for path in &files {
let bytes = Arc::new(std::fs::read(path).unwrap());
let Ok(reference) = std::panic::catch_unwind(|| hayro_syntax::Pdf::new(bytes.clone())) else { continue };
let Ok(reference) = reference else { continue }; // hayro can't open it either: not a parity case
let name = path.file_name().unwrap().to_string_lossy().into_owned();
if std::env::var("COS_ONLY").is_ok_and(|o| o != name) {
continue;
}
if std::env::var_os("COS_TRACE").is_some() {
eprintln!("{name}");
}
let expected = reference.pages().len();
let result = std::panic::catch_unwind(|| -> Result<(), String> {
let mut doc = match Document::open(bytes.clone()) {
Ok(d) => d,
Err(craft_pdf::cos::CosError::NeedsPassword) => return Err("encrypted".into()),
Err(e) => return Err(format!("open: {e}")),
};
// hayro may repair a broken /Count; compare against the leaf walk count loosely.
if page_count(&doc) != Some(expected) && expected > 0 {
return Err(format!("page count {:?} vs hayro {expected}", page_count(&doc)));
}
let root = doc.root().ok_or("no root")?;
doc.update_dict(root, |d| d.set(b"PdfCraftTest".to_vec(), Object::Bool(true))).map_err(|e| e.to_string())?;
let saved = write_incremental(&doc, &SaveOptions::default()).map_err(|e| format!("save: {e}"))?;
// Reconstructed files are rewritten in full; everything else must keep its bytes.
if !doc.revisions().is_empty() && saved.get(..bytes.len()) != Some(&bytes[..]) {
return Err("prefix modified".into());
}
let again = Document::open(Arc::new(saved.clone())).map_err(|e| format!("reopen: {e}"))?;
let flag = again.get(again.root().ok_or("no root after save")?).as_dict().and_then(|d| d.get(b"PdfCraftTest").cloned());
if flag != Some(Object::Bool(true)) {
return Err("edit lost after reopen".into());
}
let hay = hayro_syntax::Pdf::new(Arc::new(saved)).map_err(|e| format!("hayro reopen: {e:?}"))?;
if hay.pages().len() != expected {
return Err(format!("hayro sees {} pages after save, expected {expected}", hay.pages().len()));
}
let full = write_full(&doc, &SaveOptions::default()).map_err(|e| format!("full: {e}"))?;
let hay = hayro_syntax::Pdf::new(Arc::new(full)).map_err(|e| format!("hayro full: {e:?}"))?;
if hay.pages().len() != expected {
return Err(format!("hayro sees {} pages after full save, expected {expected}", hay.pages().len()));
}
Ok(())
});
match result {
Ok(Ok(())) => checked += 1,
Ok(Err(e)) if e == "encrypted" => encrypted += 1,
Ok(Err(e)) => failures.push(format!("{name}: {e}")),
Err(_) => failures.push(format!("{name}: PANIC")),
}
}
eprintln!("cos corpus parity: {checked} ok, {encrypted} encrypted (skipped), {} failures", failures.len());
for f in &failures {
eprintln!(" {f}");
}
assert!(failures.iter().all(|f| !f.ends_with("PANIC")), "no panics");
}
/// Password-protected corpus files (passwords from pdf.js's test manifest): the right password
/// opens them and the edit/save round trip still works; no or a wrong password is reported.
#[test]
#[ignore = "needs corpus/; run with --ignored"]
fn corpus_passwords() {
let Some(root) = checkout() else {
eprintln!("corpus not fetched; skipping (cargo xtask corpus)");
return;
};
let dir = root.join("test");
let manifest =
std::fs::read_to_string(dir.join("test_manifest.json")).unwrap_or_else(|e| panic!("verified corpus has no test_manifest.json: {e}"));
let entries: Vec<serde_json::Value> = serde_json::from_str(&manifest).expect("manifest is JSON");
let cases: Vec<(String, String)> =
entries.iter().filter_map(|e| Some((e.get("file")?.as_str()?.to_string(), e.get("password")?.as_str()?.to_string()))).collect();
assert!(cases.len() >= 5, "manifest parsed ({} cases)", cases.len());
let mut failures = Vec::new();
for (file, pw) in &cases {
let Ok(bytes) = std::fs::read(dir.join(file)) else { continue };
let bytes = Arc::new(bytes);
if !matches!(Document::open(bytes.clone()), Err(craft_pdf::cos::CosError::NeedsPassword)) {
failures.push(format!("{file}: opened without a password"));
}
if !matches!(Document::open_with_password(bytes.clone(), Some("definitely wrong")), Err(craft_pdf::cos::CosError::WrongPassword)) {
failures.push(format!("{file}: wrong password not rejected"));
}
match Document::open_with_password(bytes.clone(), Some(pw)) {
Ok(mut doc) => {
let pages = page_count(&doc);
if pages.unwrap_or(0) == 0 {
failures.push(format!("{file}: no pages after decryption"));
continue;
}
let root = doc.root().unwrap();
doc.update_dict(root, |d| d.set(b"PdfCraftTest".to_vec(), Object::Bool(true))).unwrap();
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
match Document::open_with_password(Arc::new(saved.clone()), Some(pw)) {
Ok(again) if page_count(&again) == pages => {}
other => failures.push(format!("{file}: reopen after save failed ({:?})", other.err())),
}
}
Err(e) => failures.push(format!("{file}: {e}")),
}
}
eprintln!("password corpus: {} cases, {} failures", cases.len(), failures.len());
for f in &failures {
eprintln!(" {f}");
}
assert!(failures.is_empty());
}
+224
View File
@@ -0,0 +1,224 @@
//! A document that encrypts only its embedded files, authorised when one is opened (`/AuthEvent
//! /EFOpen`, ISO 32000-2 §7.6.6, Table 25): it opens without a password, its pages read as stored,
//! and its embedded files keep their encrypted bytes, flagged as such, through every save.
use std::sync::Arc;
use craft_pdf::cos::{Algorithm, CosError, Document, NewEncryption, ObjRef, Object, SaveOptions, write_full, write_incremental};
use craft_pdf::crypt::StreamKind;
const PAGE: &str = "BT /F1 12 Tf 20 100 Td (Visible page) Tj ET";
const ATTACHED: &[u8] = b"the attached file's own words";
const PASSWORD: &str = "attach pw";
const EMBEDDED: ObjRef = ObjRef { num: 6, generation: 0 };
const CONTENT: ObjRef = ObjRef { num: 4, generation: 0 };
fn hex(bytes: &[u8]) -> String {
bytes.iter().map(|b| format!("{b:02X}")).collect()
}
/// One page in the clear and one attachment encrypted with `/StdCF`, whose `/AuthEvent` is
/// `auth_event`: AES-256 (V5, R6) or AES-128 (V4, R4), `/StmF` and `/StrF` `/Identity`.
fn fixture(algorithm: Algorithm, auth_event: &str) -> Vec<u8> {
fixture_with(algorithm, auth_event, false)
}
/// [`fixture`], the attachment naming `/StdCF` in a `/Crypt` filter of its own when `named`, with
/// no `/EFF` (ISO 32000-2 §7.4.10).
fn fixture_with(algorithm: Algorithm, auth_event: &str, named: bool) -> Vec<u8> {
let id0 = [7u8; 16];
let params =
NewEncryption { algorithm, user_password: PASSWORD, owner_password: "owner pw", permissions: -4, encrypt_metadata: true, seed: [3; 32] };
let handler = craft_pdf::crypt::create(&params, &id0).unwrap();
let d = handler.dict();
let kind = if named { StreamKind::Named(b"StdCF") } else { StreamKind::EmbeddedFile };
let cipher = handler.encrypt_stream(EMBEDDED.num, 0, ATTACHED, kind);
let (eff, crypt) = if named { ("", " /Filter [/Crypt] /DecodeParms [<< /Name /StdCF >>]") } else { (" /EFF /StdCF", "") };
let (version, cfm, r6) = match algorithm {
Algorithm::Aes256 => ("/V 5 /R 6", "/AESV3 /Length 32", format!("/OE <{}> /UE <{}> /Perms <{}>", hex(&d.oe), hex(&d.ue), hex(&d.perms))),
_ => ("/V 4 /R 4 /Length 128", "/AESV2 /Length 16", String::new()),
};
let encrypt = format!(
"<< /Filter /Standard {version} /O <{}> /U <{}> {r6} /P {} /CF << /StdCF << /AuthEvent /{auth_event} /CFM {cfm} >> >> /StmF /Identity /StrF /Identity{eff} >>",
hex(&d.o),
hex(&d.u),
d.p
);
let mut objs: Vec<Vec<u8>> = vec![
b"<< /Type /Catalog /Pages 2 0 R /Names << /EmbeddedFiles << /Names [(note.txt) 5 0 R] >> >> >>".to_vec(),
b"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 300 200] >>".to_vec(),
b"<< /Type /Page /Parent 2 0 R /Contents 4 0 R >>".to_vec(),
format!("<< /Length {} >>\nstream\n{PAGE}\nendstream", PAGE.len()).into_bytes(),
b"<< /Type /Filespec /F (note.txt) /UF (note.txt) /EF << /F 6 0 R >> >>".to_vec(),
];
let mut stream = format!("<< /Type /EmbeddedFile{crypt} /Length {} >>\nstream\n", cipher.len()).into_bytes();
stream.extend_from_slice(&cipher);
stream.extend_from_slice(b"\nendstream");
objs.push(stream);
objs.push(encrypt.into_bytes());
let mut out = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (i, o) in objs.iter().enumerate() {
offsets.push(out.len());
out.extend_from_slice(format!("{} 0 obj\n", i + 1).as_bytes());
out.extend_from_slice(o);
out.extend_from_slice(b"\nendobj\n");
}
let xref = out.len();
out.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
for o in offsets {
out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
}
let trailer = format!(
"trailer\n<< /Size {} /Root 1 0 R /Encrypt 7 0 R /ID [<{}> <{}>] >>\nstartxref\n{xref}\n%%EOF\n",
objs.len() + 1,
hex(&id0),
hex(&id0)
);
out.extend_from_slice(trailer.as_bytes());
out
}
fn stream(doc: &Document, r: ObjRef) -> craft_pdf::cos::Stream {
match &*doc.get(r) {
Object::Stream(s) => s.clone(),
other => panic!("object {} isn't a stream: {other:?}", r.num),
}
}
/// The attachment as the password reads it from `bytes`.
fn attachment_with_password(bytes: Vec<u8>) -> Vec<u8> {
let doc = Document::open_with_password(Arc::new(bytes), Some(PASSWORD)).unwrap();
assert!(!doc.embedded_files_locked());
let embedded = doc.root().and_then(|r| doc.get(r).as_dict().cloned()).unwrap();
assert!(embedded.get(b"Names").is_some());
stream(&doc, EMBEDDED).decoded().unwrap()
}
#[test]
fn a_document_encrypting_only_its_embedded_files_opens_without_a_password() {
let bytes = fixture(Algorithm::Aes256, "EFOpen");
let doc = Document::open(Arc::new(bytes.clone())).unwrap();
assert!(doc.embedded_files_locked());
// Not a repair: the specification has it open so.
assert!(doc.repair_log().is_empty(), "{:?}", doc.repair_log());
assert!(doc.trailer().get(b"Encrypt").is_some());
// The page reads as stored; the attachment keeps its encrypted bytes and says so.
let content = stream(&doc, CONTENT);
assert_eq!(content.decoded().unwrap(), PAGE.as_bytes());
assert!(!doc.reads_as_stored(&content));
let embedded = stream(&doc, EMBEDDED);
assert!(doc.reads_as_stored(&embedded));
assert_ne!(embedded.decoded().unwrap(), ATTACHED);
// The password still opens it, and reads the attachment.
assert_eq!(attachment_with_password(bytes.clone()), ATTACHED);
// A wrong password is still refused.
assert!(matches!(Document::open_with_password(Arc::new(bytes), Some("nope")), Err(CosError::WrongPassword)));
}
#[test]
fn a_crypt_filter_authorised_when_the_document_opens_still_needs_the_password() {
let bytes = fixture(Algorithm::Aes256, "DocOpen");
assert!(matches!(Document::open(Arc::new(bytes.clone())), Err(CosError::NeedsPassword)));
assert_eq!(attachment_with_password(bytes), ATTACHED);
}
#[test]
fn saves_keep_the_encrypted_embedded_files_as_stored() {
for algorithm in [Algorithm::Aes256, Algorithm::Aes128] {
let mut doc = Document::open(Arc::new(fixture(algorithm, "EFOpen"))).unwrap();
assert!(doc.embedded_files_locked(), "{algorithm:?}");
let root = doc.root().unwrap();
doc.update_dict(root, |c| c.set(b"PageMode".to_vec(), Object::name("UseAttachments"))).unwrap();
// An incremental save keeps /Encrypt, and the attachment decrypts with the password.
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert_eq!(attachment_with_password(saved), ATTACHED, "{algorithm:?}");
// A full save renumbers: AES-256 keys don't depend on the number, AES-128 keys do.
match algorithm {
Algorithm::Aes256 => {
let full = write_full(&doc, &SaveOptions::default()).unwrap();
let reopened = Document::open(Arc::new(full.clone())).unwrap();
assert!(reopened.embedded_files_locked());
let doc = Document::open_with_password(Arc::new(full), Some(PASSWORD)).unwrap();
let names = doc.root().and_then(|r| doc.get(r).as_dict().cloned()).unwrap();
let tree = doc.resolve(names.get(b"Names").unwrap()).as_dict().cloned().unwrap();
let files = doc.resolve(tree.get(b"EmbeddedFiles").unwrap()).as_dict().cloned().unwrap();
let list = doc.resolve(files.get(b"Names").unwrap()).as_array().cloned().unwrap();
let spec = doc.resolve(&list[1]).as_dict().cloned().unwrap();
let ef = doc.resolve(spec.get(b"EF").unwrap()).as_dict().cloned().unwrap();
let file = match &*doc.resolve(ef.get(b"F").unwrap()) {
Object::Stream(s) => s.decoded().unwrap(),
other => panic!("{other:?}"),
};
assert_eq!(file, ATTACHED);
}
_ => assert!(matches!(write_full(&doc, &SaveOptions::default()), Err(CosError::Security(_))), "{algorithm:?}"),
}
}
}
#[test]
fn security_cannot_change_without_the_embedded_files_password() {
let mut doc = Document::open(Arc::new(fixture(Algorithm::Aes256, "EFOpen"))).unwrap();
doc.remove_encryption();
assert!(matches!(write_full(&doc, &SaveOptions::default()), Err(CosError::Security(_))));
assert!(matches!(write_incremental(&doc, &SaveOptions::default()), Err(CosError::Security(_))));
}
/// From the review of the EFOpen change: the empty password opens it too, and the credentials of a
/// document opened without the key open no other.
#[test]
fn the_empty_password_opens_it_and_keyless_credentials_open_nothing_more() {
let bytes = fixture(Algorithm::Aes256, "EFOpen");
let doc = Document::open_with_password(Arc::new(bytes.clone()), Some("")).unwrap();
assert!(doc.embedded_files_locked());
assert!(doc.open_with_same_credentials(Arc::new(bytes)).unwrap().embedded_files_locked());
// The same /Encrypt values and /ID, authorised when the document opens.
let doc_open = fixture(Algorithm::Aes256, "DocOpen");
assert!(matches!(doc.open_with_same_credentials(Arc::new(doc_open.clone())), Err(CosError::NeedsPassword)));
assert!(matches!(Document::open_with_password(Arc::new(doc_open), Some("")), Err(CosError::WrongPassword)));
}
/// An embedded file that names an EFOpen crypt filter of its own, with no `/EFF`, opens the same way.
#[test]
fn an_embedded_file_naming_its_own_crypt_filter_opens_without_a_password() {
let bytes = fixture_with(Algorithm::Aes256, "EFOpen", true);
let doc = Document::open(Arc::new(bytes.clone())).unwrap();
assert!(doc.embedded_files_locked());
assert!(doc.reads_as_stored(&stream(&doc, EMBEDDED)));
assert_eq!(stream(&doc, CONTENT).decoded().unwrap(), PAGE.as_bytes());
assert_eq!(attachment_with_password(bytes), ATTACHED);
}
/// A save can't encrypt without the key: removing `/Encrypt` through the trailer, or putting in an
/// embedded file, is refused rather than written in the clear.
#[test]
fn saves_refuse_what_would_need_the_key() {
let open = || Document::open(Arc::new(fixture(Algorithm::Aes256, "EFOpen"))).unwrap();
let mut doc = open();
doc.trailer_mut().remove(b"Encrypt");
assert!(matches!(write_incremental(&doc, &SaveOptions::default()), Err(CosError::Security(_))));
assert!(matches!(write_full(&doc, &SaveOptions::default()), Err(CosError::Security(_))));
let mut doc = open();
let mut replaced = stream(&doc, EMBEDDED);
replaced.raw = b"new words".to_vec().into();
doc.set(EMBEDDED, Object::Stream(replaced));
assert!(matches!(write_incremental(&doc, &SaveOptions::default()), Err(CosError::Security(_))));
}
/// The password given after opening unlocks the embedded files; a wrong one doesn't.
#[test]
fn the_password_given_later_unlocks_the_embedded_files() {
let mut doc = Document::open(Arc::new(fixture(Algorithm::Aes256, "EFOpen"))).unwrap();
assert!(matches!(doc.unlock("nope"), Err(CosError::WrongPassword)));
assert!(doc.embedded_files_locked());
doc.unlock(PASSWORD).unwrap();
assert!(!doc.embedded_files_locked());
assert_eq!(stream(&doc, EMBEDDED).decoded().unwrap(), ATTACHED);
assert_eq!(stream(&doc, CONTENT).decoded().unwrap(), PAGE.as_bytes());
// Not while an embedded file read as stored has been edited.
let mut doc = Document::open(Arc::new(fixture(Algorithm::Aes256, "EFOpen"))).unwrap();
let edited = stream(&doc, EMBEDDED);
doc.set(EMBEDDED, Object::Stream(edited));
assert!(matches!(doc.unlock(PASSWORD), Err(CosError::Security(_))));
}
File diff suppressed because it is too large Load Diff
+277
View File
@@ -0,0 +1,277 @@
//! The file identifier (`/ID`, ISO 32000-2 §14.4) of a full save. An ordinary save keeps the
//! identifier the file has and writes the same bytes twice. A rewrite the document asked for (after
//! redaction or sanitizing) gives the file a new pair. An encrypted file keeps its first string,
//! because its file key is derived from it.
use std::sync::Arc;
use crate::testkit::Pdf;
use craft_pdf::cos::{Algorithm, Document, NewEncryption, Object, PdfString, SaveOptions, write_full, write_incremental};
const ID_HEX: &str = "/ID [<00112233445566778899AABBCCDDEEFF> <00112233445566778899AABBCCDDEEFF>]";
const ID_BYTES: [u8; 16] = [0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF];
/// One page whose content holds a secret, with `entry` added to the trailer (for example an `/ID`).
fn source_with(entry: Option<&str>) -> Vec<u8> {
let pdf = Pdf::new()
.obj("<< /Type /Catalog /Pages 2 0 R >>")
.obj("<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>")
.obj("<< /Type /Page /Parent 2 0 R /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>")
.stream("", "BT /F1 12 Tf 20 100 Td (TOPSECRET) Tj ET")
.obj("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>");
match entry {
Some(entry) => pdf.trailer_entry(entry).build(),
None => pdf.build(),
}
}
/// [`source_with`] with the standard identifier, or none.
fn source(with_id: bool) -> Vec<u8> {
source_with(with_id.then_some(ID_HEX))
}
/// A one-page file whose 70,000-byte content stream ends in `last`. Two of these built with
/// different last bytes have the same length and differ only past the first 64 KiB.
fn large_source(last: u8) -> Vec<u8> {
let mut content = vec![b' '; 70_000];
content.push(last);
Pdf::new()
.obj("<< /Type /Catalog /Pages 2 0 R >>")
.obj("<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>")
.obj("<< /Type /Page /Parent 2 0 R /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>")
.stream("", content)
.obj("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
.build()
}
fn reopen(bytes: Vec<u8>) -> Document {
Document::open(Arc::new(bytes)).unwrap()
}
/// The strings of the trailer's `/ID` (empty when there is none).
fn id_strings(doc: &Document) -> Vec<Vec<u8>> {
match doc.trailer().get(b"ID") {
Some(Object::Array(a)) => a.iter().map(|s| s.as_string().map(|s| s.bytes.clone()).unwrap_or_default()).collect(),
_ => Vec::new(),
}
}
#[test]
fn an_ordinary_save_keeps_the_identifier_and_writes_the_same_bytes() {
let doc = reopen(source(true));
let opts = SaveOptions::default();
let saved = write_full(&doc, &opts).unwrap();
assert_eq!(saved, write_full(&doc, &opts).unwrap(), "unchanged inputs produce identical bytes");
assert_eq!(id_strings(&reopen(saved)), vec![ID_BYTES.to_vec(), ID_BYTES.to_vec()]);
let appended = reopen(write_incremental(&doc, &opts).unwrap());
assert_eq!(id_strings(&appended), vec![ID_BYTES.to_vec(), ID_BYTES.to_vec()]);
}
#[test]
fn an_ordinary_save_of_a_file_without_an_identifier_creates_the_same_one_every_time() {
let doc = reopen(source(false));
let opts = SaveOptions::default();
let saved = write_full(&doc, &opts).unwrap();
assert_eq!(saved, write_full(&doc, &opts).unwrap(), "unchanged inputs produce identical bytes");
let ids = id_strings(&reopen(saved));
assert_eq!(ids.len(), 2);
assert!(ids.iter().all(|s| s.len() == 16), "{ids:?}");
}
#[test]
fn a_rewrite_the_document_asked_for_gets_a_new_identifier() {
let mut doc = reopen(source(true));
doc.require_full_save_with_new_id();
assert!(doc.full_save_required() && doc.new_id_required());
let opts = SaveOptions::default();
for saved in [write_incremental(&doc, &opts).unwrap(), write_full(&doc, &opts).unwrap()] {
let ids = id_strings(&reopen(saved));
assert_eq!(ids.len(), 2);
for id in &ids {
assert_eq!(id.len(), 16);
assert_ne!(id.as_slice(), ID_BYTES.as_slice(), "the copy must not claim the source's identifier");
}
}
// Every save is a new identity, not the same one again.
let first = id_strings(&reopen(write_full(&doc, &opts).unwrap()));
let second = id_strings(&reopen(write_full(&doc, &opts).unwrap()));
assert_ne!(first, second);
}
#[test]
fn a_rewrite_without_an_identifier_still_gets_one() {
let mut doc = reopen(source(false));
doc.require_full_save_with_new_id();
let ids = id_strings(&reopen(write_full(&doc, &SaveOptions::default()).unwrap()));
assert_eq!(ids.len(), 2);
assert!(ids.iter().all(|s| s.len() == 16), "{ids:?}");
}
#[test]
fn caller_entropy_makes_the_new_identifier_reproducible() {
let mut doc = reopen(source(true));
doc.require_full_save_with_new_id();
let with = |entropy: u8| SaveOptions { id_entropy: Some([entropy; 16]), ..SaveOptions::default() };
let first = write_full(&doc, &with(1)).unwrap();
assert_eq!(first, write_full(&doc, &with(1)).unwrap(), "the same entropy gives the same bytes");
let other = write_full(&doc, &with(2)).unwrap();
assert_ne!(id_strings(&reopen(first)), id_strings(&reopen(other)), "different entropy gives a different identifier");
}
#[test]
fn an_encrypted_rewrite_keeps_the_first_identifier_string_its_key_depends_on() {
for algorithm in [Algorithm::Rc4_128, Algorithm::Aes128, Algorithm::Aes256] {
let mut doc = reopen(source(true));
doc.set_encryption(&NewEncryption {
algorithm,
user_password: "user",
owner_password: "owner",
permissions: -1,
encrypt_metadata: true,
seed: [5; 32],
})
.unwrap();
let protected = write_full(&doc, &SaveOptions::default()).unwrap();
let before = Document::open_with_password(Arc::new(protected), Some("user")).unwrap();
let mut doc = before.clone();
doc.require_full_save_with_new_id();
let out = write_incremental(&doc, &SaveOptions::default()).unwrap();
// The output opens with the same password only if its first identifier string is unchanged.
let back = Document::open_with_password(Arc::new(out), Some("user")).unwrap();
let (old, new) = (id_strings(&before), id_strings(&back));
assert_eq!(new.len(), 2, "{algorithm:?}");
assert_eq!(new.first(), old.first(), "{algorithm:?}: the key depends on the first string");
assert_ne!(new.get(1), old.get(1), "{algorithm:?}: the second string is new");
}
}
/// The `/ID` shapes whose key the reader takes as the empty string: none at all, one entry, and a
/// first entry that is not a string.
fn unusable_ids() -> Vec<(&'static str, Option<Object>)> {
let empty = Object::String(PdfString::literal(Vec::new()));
vec![
("absent", None),
("one entry", Some(Object::Array(vec![empty.clone()]))),
("first entry not a string", Some(Object::Array(vec![Object::Int(7), empty]))),
]
}
/// An encrypted file whose key is derived from the empty string, with its trailer `/ID` replaced by `id`.
fn protected_with_id(algorithm: Algorithm, id: Option<Object>) -> Vec<u8> {
let mut doc = reopen(source_with(Some("/ID [<> <>]")));
doc.set_encryption(&NewEncryption {
algorithm,
user_password: "user",
owner_password: "owner",
permissions: -1,
encrypt_metadata: true,
seed: [5; 32],
})
.unwrap();
match id {
Some(id) => doc.trailer_mut().set(b"ID".to_vec(), id),
None => {
doc.trailer_mut().remove(b"ID");
}
}
write_full(&doc, &SaveOptions::default()).unwrap()
}
#[test]
fn an_encrypted_rewrite_without_a_usable_identifier_still_gets_a_new_one() {
// The key is derived from the first string of /ID, and a missing /ID, a one-entry /ID and a
// first entry that is not a string all count as the empty string. The rewrite keeps that
// (empty) first string, so the file still opens with its password, and writes a new second one.
for algorithm in [Algorithm::Rc4_128, Algorithm::Aes128, Algorithm::Aes256] {
for (shape, id) in unusable_ids() {
let before = Document::open_with_password(Arc::new(protected_with_id(algorithm, id)), Some("user")).unwrap();
let mut doc = before.clone();
doc.require_full_save_with_new_id();
let out = write_incremental(&doc, &SaveOptions::default()).unwrap();
let back = Document::open_with_password(Arc::new(out), Some("user")).unwrap();
let ids = id_strings(&back);
let previous = id_strings(&before);
assert_eq!(ids.len(), 2, "{algorithm:?}, {shape}: the rewrite writes a pair");
assert!(ids.first().is_some_and(|s| s.is_empty()), "{algorithm:?}, {shape}: the first string is the one the key uses");
assert_eq!(ids.get(1).map(Vec::len), Some(16), "{algorithm:?}, {shape}");
assert_ne!(ids.get(1), previous.get(1), "{algorithm:?}, {shape}: the second string is new");
}
}
}
/// The standard fixture rewritten for a new identifier, with `id_entropy` of 0x42.
fn fixed_entropy_rewrite() -> Vec<u8> {
let mut doc = reopen(source(true));
doc.require_full_save_with_new_id();
write_full(&doc, &SaveOptions { id_entropy: Some([0x42; 16]), ..SaveOptions::default() }).unwrap()
}
#[test]
fn a_fixed_entropy_gives_the_same_identifier_every_time() {
// The formula itself is pinned on fixed bytes in the writer's tests. The written file is not the
// same on every platform, so no identifier of a rewrite can be pinned here.
let first = id_strings(&reopen(fixed_entropy_rewrite()));
assert_eq!(first.len(), 2);
assert!(first.iter().all(|s| s.len() == 16), "{first:?}");
assert_eq!(first, id_strings(&reopen(fixed_entropy_rewrite())));
}
#[test]
fn the_identifier_depends_on_the_whole_rewritten_file() {
// Same caller entropy and same length; the two rewrites differ only in one byte of a stream
// that lies past the first 64 KiB of the output. Their identifiers must still differ.
let opts = SaveOptions { id_entropy: Some([9; 16]), ..SaveOptions::default() };
let rewrite = |last: u8| {
let mut doc = reopen(large_source(last));
doc.require_full_save_with_new_id();
write_full(&doc, &opts).unwrap()
};
let (a, b) = (rewrite(b'A'), rewrite(b'B'));
assert_eq!(a.len(), b.len(), "the two rewrites have the same length");
let first_difference = a.iter().zip(&b).position(|(x, y)| x != y).unwrap_or(usize::MAX);
assert!(first_difference > 64 * 1024, "the rewrites first differ at byte {first_difference}, inside the first 64 KiB");
assert_ne!(id_strings(&reopen(a)), id_strings(&reopen(b)), "content past 64 KiB must change the identifier");
}
/// Where [`stand_in_for_a_fresh_process`] writes its rewrite.
const ID_PROBE: &str = "CRAFT_PDF_TEST_ID_PROBE";
/// The child half of [`separate_fresh_processes_get_different_identifiers`]: it runs alone, in its
/// own process, and writes a default-path rewrite of the standard fixture to the file the parent
/// named. Without the variable it is a plain no-op.
#[test]
fn stand_in_for_a_fresh_process() {
let Some(path) = std::env::var_os(ID_PROBE) else { return };
let mut doc = reopen(source(true));
doc.require_full_save_with_new_id();
std::fs::write(path, write_full(&doc, &SaveOptions::default()).unwrap()).unwrap();
}
#[test]
fn separate_fresh_processes_get_different_identifiers() {
// The default identifier is meant to differ between fresh instances of the whole save pipeline,
// not only between two calls in one process (which the counter already separates). On the native
// test target a fresh instance is a fresh process, whose hash keys the operating system seeds;
// the fix under test runs on WebAssembly, where no browser test harness exists — there the
// browser's `crypto.getRandomValues` stands in for this seeding, and this test cannot see it.
let exe = std::env::current_exe().unwrap();
let probe = |name: &str| {
let path = std::env::temp_dir().join(format!("craft-pdf-id-probe-{}-{name}.pdf", std::process::id()));
let out = std::process::Command::new(&exe)
// The test binary holds every `cos` test module, so the child test is named by its path.
.args(["--exact", "file_identifier::stand_in_for_a_fresh_process", "--nocapture", "--"])
.env(ID_PROBE, &path)
.output()
.expect("run this test's own binary as a child");
assert!(out.status.success(), "{name}: the child failed: {}", String::from_utf8_lossy(&out.stderr));
let saved = std::fs::read(&path).unwrap_or_else(|e| panic!("{name}: the child wrote its file: {e}"));
let ids = id_strings(&reopen(saved));
assert_eq!(ids.len(), 2, "{name}");
assert!(ids.iter().all(|s| s.len() == 16), "{name}: {ids:?}");
let _ = std::fs::remove_file(&path);
ids
};
let (first, second) = (probe("a"), probe("b"));
assert_ne!(first, second, "two fresh processes must not share an identifier");
}
+510
View File
@@ -0,0 +1,510 @@
//! An indirect reference identifies an object number AND its generation (ISO 32000-2 §7.3.10).
use std::sync::Arc;
use craft_pdf::cos::{Document, ObjRef, Object, SaveOptions, XrefEntry, write_full, write_incremental};
use hayro_syntax::object::{Dict as HayroDict, ObjectIdentifier};
/// Object 3 was reused: only 3 1 R refers to the current value, never 3 0 R.
fn reused_object() -> Arc<Vec<u8>> {
object_file(&[
(1, 0, "<< /Type /Catalog /Pages 2 0 R /Stale 3 0 R /Current 3 1 R >>"),
(2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"),
(3, 1, "42"),
])
}
fn object_file(objects: &[(u32, u16, &str)]) -> Arc<Vec<u8>> {
object_file_with_entry(objects, None)
}
fn object_file_with_entry(objects: &[(u32, u16, &str)], changed: Option<(u32, u16, bool)>) -> Arc<Vec<u8>> {
let mut bytes = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (num, generation, value) in objects {
offsets.push((*num, bytes.len(), *generation));
bytes.extend_from_slice(format!("{num} {generation} obj\n{value}\nendobj\n").as_bytes());
}
let xref = bytes.len();
let size = objects.len() + 1;
bytes.extend_from_slice(format!("xref\n0 {size}\n0000000000 65535 f \n").as_bytes());
for (num, offset, generation) in offsets {
let (generation, free) = changed.filter(|(n, _, _)| *n == num).map_or((generation, false), |(_, g, free)| (g, free));
let (offset, kind) = if free { (0, 'f') } else { (offset, 'n') };
bytes.extend_from_slice(format!("{offset:010} {generation:05} {kind} \n").as_bytes());
}
bytes.extend_from_slice(format!("trailer\n<< /Size {size} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n").as_bytes());
Arc::new(bytes)
}
fn contradictory_entry(free: bool) -> Arc<Vec<u8>> {
object_file_with_entry(
&[(1, 0, "<< /Type /Catalog /Pages 2 0 R /Stale 3 0 R /Current 3 1 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), (3, 1, "42")],
Some((3, if free { 2 } else { 0 }, free)),
)
}
#[test]
fn contradictory_xref_generation_recovers_the_matching_header_and_records_repair() {
for warm in [false, true] {
let doc = Document::open(contradictory_entry(false)).unwrap();
if warm {
assert_eq!(*doc.try_get(3).unwrap(), Object::Int(42));
}
assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42));
assert_eq!(*doc.resolve(&Object::Ref(ObjRef::new(3, 1))), Object::Int(42));
// Preserve the existing xref-matching behavior, but never accept an unrelated generation.
assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Int(42));
assert_eq!(*doc.get(ObjRef::new(3, 2)), Object::Null);
assert!(!doc.xref_entry_is_exact(3), "a contradictory generation is not an exact xref entry");
assert!(
doc.repair_log()
.iter()
.any(|line| { line.contains("object 3") && line.contains("xref generation 0") && line.contains("header generation 1") }),
"{:?}",
doc.repair_log()
);
let repairs = doc.repair_log().to_vec();
for _ in 0..3 {
assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42));
}
assert_eq!(doc.repair_log(), repairs);
let mut edited = doc.clone();
edited.free(ObjRef::new(3, 1));
for generation in [0, 1, 2] {
assert_eq!(*edited.get(ObjRef::new(3, generation)), Object::Null);
}
edited.set(ObjRef::new(3, 2), Object::Int(99));
assert_eq!(*edited.get(ObjRef::new(3, 1)), Object::Null);
assert_eq!(*edited.get(ObjRef::new(3, 2)), Object::Int(99));
assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42));
}
}
/// A damaged row may name generation zero while the retained object and catalog use one.
fn contradictory_dictionary() -> Arc<Vec<u8>> {
object_file_with_entry(
&[(1, 0, "<< /Type /Catalog /Pages 2 0 R /Current 3 1 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), (3, 1, "<< /Value 42 >>")],
Some((3, 0, false)),
)
}
fn assert_repaired_dictionary(bytes: Vec<u8>, value: i32) {
let bytes = Arc::new(bytes);
let oracle = hayro_syntax::Pdf::new(bytes.clone()).unwrap();
let catalog = oracle.xref().get::<HayroDict<'_>>(ObjectIdentifier::new(1, 0)).unwrap();
assert_eq!(catalog.get::<HayroDict<'_>>(b"Current").unwrap().get::<i32>(b"Value"), Some(value));
let back = Document::open(bytes).unwrap();
let catalog = back.get(back.root().unwrap());
let current = back.resolve(catalog.as_dict().unwrap().get(b"Current").unwrap());
assert_eq!(current.as_dict().unwrap().int(b"Value"), Some(i64::from(value)));
}
#[test]
fn repaired_header_references_survive_full_saves_and_dictionary_edits() {
let original = contradictory_dictionary();
let doc = Document::open(original.clone()).unwrap();
for object_streams in [false, true] {
assert_repaired_dictionary(write_full(&doc, &SaveOptions { object_streams, ..SaveOptions::default() }).unwrap(), 42);
}
let mut edited = doc.clone();
edited.update_dict(ObjRef::new(3, 1), |dict| dict.set(b"Value".to_vec(), Object::Int(77))).unwrap();
assert_eq!(edited.generation(3), 1);
let incremental = write_incremental(&edited, &SaveOptions::default()).unwrap();
assert!(edited.full_save_required(), "repair-aware dictionary edits normalize on full rewrite");
assert_repaired_dictionary(incremental, 77);
for object_streams in [false, true] {
assert_repaired_dictionary(write_full(&edited, &SaveOptions { object_streams, ..SaveOptions::default() }).unwrap(), 77);
}
let mut restored = doc.clone();
restored.overlay_changes_from(&edited).unwrap();
assert_repaired_dictionary(write_incremental(&restored, &SaveOptions::default()).unwrap(), 77);
}
#[test]
fn free_xref_slots_never_revive_retained_object_headers() {
for warm in [false, true] {
let doc = Document::open(contradictory_entry(true)).unwrap();
assert!(matches!(doc.xref_entry(3), Some(XrefEntry::Free { next_generation: 2 })));
if warm {
assert_eq!(*doc.try_get(3).unwrap(), Object::Null);
}
for generation in [0, 1, 2] {
assert_eq!(*doc.get(ObjRef::new(3, generation)), Object::Null);
}
assert_eq!(*doc.resolve(&Object::Ref(ObjRef::new(3, 1))), Object::Null);
assert!(doc.repair_log().is_empty(), "{:?}", doc.repair_log());
}
// Control: the retained bytes really are a readable object when its slot is live.
let live = Document::open(contradictory_entry(false)).unwrap();
assert_eq!(*live.try_get(3).unwrap(), Object::Int(42));
}
#[test]
fn incremental_generation_identity_is_visible_to_an_independent_reader() {
let original = reused_object();
let mut doc = Document::open(original.clone()).unwrap();
doc.set(ObjRef::new(3, 1), Object::Int(77));
let bytes = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert!(bytes.starts_with(&original));
let oracle = hayro_syntax::Pdf::new(Arc::new(bytes)).unwrap();
let catalog = oracle.xref().get::<HayroDict<'_>>(ObjectIdentifier::new(1, 0)).unwrap();
assert_eq!(catalog.get::<i32>(b"Current"), Some(77));
assert_eq!(catalog.get::<i32>(b"Stale"), None);
}
#[test]
fn generation_repair_diagnostics_are_bounded_without_disabling_recovery() {
let mut objects = vec![(1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>")];
objects.extend((3..303).map(|num| (num, 1, "42")));
let bytes = object_file(&objects);
let bytes = String::from_utf8(bytes.as_ref().clone()).unwrap().replace("00001 n \n", "00000 n \n");
for materialized in [false, true] {
let bytes = if materialized { bytes.replace("/Size 303", "/Size 302") } else { bytes.clone() };
let doc = Document::open(Arc::new(bytes.into_bytes())).unwrap();
assert_eq!(doc.repair_log().len(), 257);
assert!(doc.repair_log().last().unwrap().contains("omitted after 256 objects"));
assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42));
assert_eq!(*doc.get(ObjRef::new(302, 1)), Object::Int(42));
assert_eq!(*doc.get(ObjRef::new(302, 2)), Object::Null);
assert_eq!(doc.repair_log().len(), 257);
}
}
#[test]
fn references_require_the_current_generation_even_with_a_warm_cache() {
let bytes = reused_object();
let oracle = hayro_syntax::Pdf::new(bytes.clone()).unwrap();
assert_eq!(oracle.xref().get::<i32>(ObjectIdentifier::new(3, 0)), None);
assert_eq!(oracle.xref().get::<i32>(ObjectIdentifier::new(3, 1)), Some(42));
let doc = Document::open(bytes).unwrap();
assert!(doc.repair_log().is_empty());
assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Null);
assert_eq!(*doc.try_get(3).unwrap(), Object::Int(42));
assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42));
assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Null);
assert_eq!(*doc.resolve(&Object::Ref(ObjRef::new(3, 0))), Object::Null);
assert_eq!(*doc.get(ObjRef::new(3, 2)), Object::Null);
}
#[test]
fn overlay_generation_changes_do_not_retarget_old_references_or_clones() {
let mut doc = Document::open(reused_object()).unwrap();
let snapshot = doc.clone();
let old = ObjRef::new(3, 1);
let new = ObjRef::new(3, 2);
doc.free(old);
assert_eq!(*doc.get(old), Object::Null);
doc.set(new, Object::Int(99));
assert_eq!(*doc.get(old), Object::Null);
assert_eq!(*doc.get(new), Object::Int(99));
assert_eq!(*snapshot.get(old), Object::Int(42));
assert_eq!(*snapshot.get(new), Object::Null);
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
let back = Document::open(Arc::new(saved)).unwrap();
assert!(back.repair_log().is_empty());
assert_eq!(*back.get(old), Object::Null);
assert_eq!(*back.get(new), Object::Int(99));
}
#[test]
fn restoring_snapshot_preserves_changes_to_nonzero_generation_objects() {
for generation in [1, 2] {
let mut saved = Document::open(reused_object()).unwrap();
let mut current = saved.clone();
let reference = ObjRef::new(3, generation);
current.set(reference, Object::Int(99));
current.update_dict(current.root().unwrap(), |catalog| catalog.set(b"Current".to_vec(), Object::Ref(reference))).unwrap();
saved.overlay_changes_from(&current).unwrap();
assert_eq!(*saved.get(reference), Object::Int(99), "generation {generation}");
let back = Document::open(Arc::new(write_incremental(&saved, &SaveOptions::default()).unwrap())).unwrap();
assert!(back.repair_log().is_empty());
assert_eq!(*back.get(reference), Object::Int(99), "generation {generation}");
let catalog = back.get(back.root().unwrap());
assert_eq!(*back.resolve(catalog.as_dict().unwrap().get(b"Current").unwrap()), Object::Int(99));
}
}
#[test]
fn full_save_does_not_turn_stale_references_into_live_ones() {
let doc = Document::open(reused_object()).unwrap();
for object_streams in [false, true] {
let saved = write_full(&doc, &SaveOptions { object_streams, ..SaveOptions::default() }).unwrap();
let saved = Arc::new(saved);
let oracle = hayro_syntax::Pdf::new(saved.clone()).unwrap();
let catalog = oracle.xref().get::<HayroDict<'_>>(ObjectIdentifier::new(1, 0)).unwrap();
assert_eq!(catalog.get::<i32>(b"Current"), Some(42));
assert_eq!(catalog.get::<i32>(b"Stale"), None);
let back = Document::open(saved).unwrap();
assert!(back.repair_log().is_empty());
let catalog = back.get(back.root().unwrap());
let catalog = catalog.as_dict().unwrap();
assert_eq!(catalog.get(b"Stale"), Some(&Object::Null));
assert_eq!(*back.resolve(catalog.get(b"Current").unwrap()), Object::Int(42));
}
}
#[test]
fn compressed_objects_only_accept_generation_zero() {
let doc = Document::open(reused_object()).unwrap();
let saved = write_full(&doc, &SaveOptions::default()).unwrap();
let back = Document::open(Arc::new(saved)).unwrap();
let catalog = back.get(back.root().unwrap());
let current = catalog.as_dict().unwrap().reference(b"Current").unwrap();
assert_eq!(current.generation, 0);
assert!(matches!(back.xref_entry(current.num), Some(XrefEntry::InStream { .. })));
assert_eq!(*back.get(current), Object::Int(42));
assert_eq!(*back.get(ObjRef::new(current.num, 1)), Object::Null);
}
#[test]
fn indirect_stream_lengths_require_the_referenced_generation() {
for generation in [0, 1] {
// The embedded endstream is data when /Length resolves, but the existing missing-
// length recovery stops there when the reference is stale.
let stream = format!("<< /Length 4 {generation} R >>\nstream\nA\nendstream\nB\nendstream");
let doc = Document::open(object_file(&[
(1, 0, "<< /Type /Catalog /Pages 2 0 R >>"),
(2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"),
(3, 0, &stream),
(4, 1, "13"),
]))
.unwrap();
let object = doc.get(ObjRef::new(3, 0));
let Object::Stream(stream) = &*object else { panic!("expected stream") };
assert_eq!(&stream.raw[..], if generation == 1 { b"A\nendstream\nB".as_slice() } else { b"A".as_slice() });
}
}
/// A newer valid row must not hide damage in a retained older revision.
fn shadowed_generation_disagreement(materialized: bool, shifted: bool) -> Arc<Vec<u8>> {
let original = contradictory_dictionary();
let text = std::str::from_utf8(&original).unwrap();
let previous: usize = text.rsplit("startxref\n").next().unwrap().lines().next().unwrap().parse().unwrap();
let mut bytes = (*original).clone();
let offset = bytes.len();
bytes.extend_from_slice(b"3 1 obj\n<< /Value 77 >>\nendobj\n");
let xref = bytes.len();
let size = if materialized { 3 } else { 4 };
bytes.extend_from_slice(
format!("xref\n3 1\n{offset:010} 00001 n \ntrailer\n<< /Size {size} /Root 1 0 R /Prev {previous} >>\nstartxref\n{xref}\n%%EOF\n").as_bytes(),
);
if shifted {
bytes.splice(..0, b"junk before the header\n".iter().copied());
}
Arc::new(bytes)
}
#[test]
fn retained_generation_disagreements_are_reported_in_materialized_mode_too() {
for materialized in [false, true] {
for shifted in [false, true] {
let doc = Document::open(shadowed_generation_disagreement(materialized, shifted)).unwrap();
assert_eq!(doc.revisions().len(), 2, "the xref chain is retained");
assert!(!doc.reconstructed());
assert_eq!(doc.get(ObjRef::new(3, 1)).as_dict().unwrap().int(b"Value"), Some(77));
assert!(
doc.repair_log().iter().any(|line| line.contains("object 3: xref generation 0 disagrees with object header generation 1")),
"materialized={materialized}, shifted={shifted}: {:?}",
doc.repair_log()
);
}
}
}
#[test]
fn repair_aware_edits_keep_both_aliases_and_full_save_normalizes_them() {
let mut doc = Document::open(contradictory_entry(false)).unwrap();
let snapshot = doc.clone();
doc.set_existing(ObjRef::new(3, 0), Object::Int(77)).unwrap();
for generation in [0, 1] {
assert_eq!(*doc.get(ObjRef::new(3, generation)), Object::Int(77));
assert_eq!(*snapshot.get(ObjRef::new(3, generation)), Object::Int(42));
}
let state = doc.read_state();
assert!(doc.set_existing(ObjRef::new(3, 2), Object::Int(99)).is_err());
assert!(state.matches(&doc));
assert_eq!(*doc.get(ObjRef::new(3, 2)), Object::Null);
let mut restored = snapshot.clone();
restored.overlay_changes_from(&doc).unwrap();
for changed in [&doc, &restored] {
assert!(changed.full_save_required());
for bytes in [
write_incremental(changed, &SaveOptions::default()).unwrap(),
write_full(changed, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap(),
] {
let bytes = Arc::new(bytes);
let oracle = hayro_syntax::Pdf::new(bytes.clone()).unwrap();
let catalog = oracle.xref().get::<HayroDict<'_>>(ObjectIdentifier::new(1, 0)).unwrap();
assert_eq!(catalog.get::<i32>(b"Current"), Some(77));
assert_eq!(catalog.get::<i32>(b"Stale"), Some(77), "both source aliases stay live");
let back = Document::open(bytes).unwrap();
assert!(back.repair_log().is_empty(), "normalized output has no generation repairs");
}
}
// Explicit replacement keeps its narrower, caller-supplied generation contract.
doc.set(ObjRef::new(3, 2), Object::Int(99));
assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Null);
assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Null);
assert_eq!(*doc.get(ObjRef::new(3, 2)), Object::Int(99));
}
#[test]
fn hybrid_generation_diagnostics_observe_rows_before_the_table_hides_them() {
let mut bytes = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (num, generation, body) in [(1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), (3, 1, "42")] {
offsets.push(bytes.len());
bytes.extend_from_slice(format!("{num} {generation} obj\n{body}\nendobj\n").as_bytes());
}
let hybrid = bytes.len();
offsets.push(hybrid);
bytes.extend_from_slice(b"4 0 obj\n<< /Type /XRef /Size 5 /W [1 4 2] /Index [3 1] /Length 7 >>\nstream\n");
bytes.push(1);
bytes.extend_from_slice(&(offsets[2] as u32).to_be_bytes());
bytes.extend_from_slice(&0u16.to_be_bytes());
bytes.extend_from_slice(b"\nendstream\nendobj\n");
let xref = bytes.len();
bytes.extend_from_slice(b"xref\n0 5\n0000000000 65535 f \n");
for (i, offset) in offsets.into_iter().enumerate() {
let generation = u16::from(i == 2);
bytes.extend_from_slice(format!("{offset:010} {generation:05} n \n").as_bytes());
}
bytes.extend_from_slice(format!("trailer\n<< /Size 5 /Root 1 0 R /XRefStm {hybrid} >>\nstartxref\n{xref}\n%%EOF\n").as_bytes());
let doc = Document::open(Arc::new(bytes)).unwrap();
assert!(!doc.reconstructed());
assert!(doc.xref_oddities().iter().any(|(_, num)| *num == 3), "the conflicting hybrid row is set aside");
assert_eq!(*doc.get(ObjRef::new(3, 1)), Object::Int(42));
assert_eq!(*doc.get(ObjRef::new(3, 0)), Object::Null, "the table still determines accepted references");
assert!(doc.repair_log().iter().any(|line| line.contains("object 3: xref generation 0 disagrees with object header generation 1")));
}
#[test]
fn discarded_xref_repair_notes_do_not_survive_object_header_reconstruction() {
let bytes =
String::from_utf8((*contradictory_dictionary()).clone()).unwrap().replace("/Root 1 0 R", "/Root 99 0 R").replace("/Size 4", "/Size 3");
let doc = Document::open(Arc::new(bytes.into_bytes())).unwrap();
assert!(doc.reconstructed());
assert!(doc.repair_log().iter().all(|line| !line.contains("xref generation")), "{:?}", doc.repair_log());
}
#[test]
fn restoring_equal_values_still_restores_the_accepted_generation_aliases() {
for edited in [false, true] {
for value in [42, 77] {
let mut current = Document::open(contradictory_entry(false)).unwrap();
if edited {
current.set_existing(ObjRef::new(3, 0), Object::Int(42)).unwrap();
}
let mut saved = Document::open(object_file(&[
(1, 0, "<< /Type /Catalog /Pages 2 0 R /Stale 3 0 R /Current 3 1 R >>"),
(2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"),
(3, 0, &value.to_string()),
]))
.unwrap();
assert_eq!(*saved.get(ObjRef::new(3, 1)), Object::Null);
saved.overlay_changes_from(&current).unwrap();
for generation in [0, 1] {
assert_eq!(*saved.get(ObjRef::new(3, generation)), Object::Int(42), "edited={edited}, value={value}");
}
assert!(saved.full_save_required());
}
}
}
#[test]
fn repaired_page_tree_aliases_keep_one_identity_in_full_saves() {
let source = object_file_with_entry(
&[
(1, 0, "<< /Type /Catalog /Pages 2 1 R /Stale 2 2 R >>"),
(2, 1, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"),
(3, 0, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 10 10] >>"),
],
Some((2, 0, false)),
);
for edited in [false, true] {
let mut doc = Document::open(source.clone()).unwrap();
if edited {
doc.update_dict(ObjRef::new(2, 0), |dict| dict.set(b"Edited".to_vec(), Object::Bool(true))).unwrap();
}
let mut restored = Document::open(source.clone()).unwrap();
restored.overlay_changes_from(&doc).unwrap();
for current in [&doc, &restored] {
for object_streams in [false, true] {
let opts = SaveOptions { object_streams, ..SaveOptions::default() };
let mut saves = vec![write_full(current, &opts).unwrap()];
if edited {
saves.push(write_incremental(current, &opts).unwrap());
}
for bytes in saves {
let back = Document::open(Arc::new(bytes)).unwrap();
let catalog = back.get(back.root().unwrap());
let catalog = catalog.as_dict().unwrap();
let pages_ref = catalog.reference(b"Pages").unwrap();
let pages = back.get(pages_ref);
let child_ref = pages.as_dict().unwrap().get(b"Kids").unwrap().as_array().unwrap()[0].as_ref().unwrap();
let child = back.get(child_ref);
assert_eq!(child.as_dict().unwrap().reference(b"Parent"), Some(pages_ref), "edited={edited}, packed={object_streams}");
assert!(matches!(catalog.get(b"Stale"), Some(Object::Null)), "unrelated generations stay null");
assert!(back.repair_log().is_empty());
}
}
}
}
}
#[test]
fn restoring_explicit_sets_removes_old_generation_aliases() {
for value in [42, 77] {
let mut current = Document::open(contradictory_entry(false)).unwrap();
current.set(ObjRef::new(3, 0), Object::Int(42));
for repaired_overlay in [false, true] {
let mut saved = Document::open(contradictory_entry(false)).unwrap();
if repaired_overlay {
saved.set_existing(ObjRef::new(3, 0), Object::Int(value)).unwrap();
}
saved.overlay_changes_from(&current).unwrap();
assert_eq!(*saved.get(ObjRef::new(3, 0)), Object::Int(42));
assert_eq!(*saved.get(ObjRef::new(3, 1)), Object::Null, "value={value}, overlay={repaired_overlay}");
}
}
}
/// `accepted_generations` lists exactly the generations `get` resolves, for every kind of slot:
/// a clean file slot, a damaged one (row and header disagree), a repair-aware edit, a plain
/// edit, a freed slot, a free row, an object stream member and an unknown number.
#[test]
fn accepted_generations_are_the_generations_get_resolves() {
let agrees = |doc: &Document, num: u32, expected: &[u16]| {
assert_eq!(doc.accepted_generations(num), expected, "object {num}");
for generation in 0..8 {
let live = !matches!(*doc.get(ObjRef::new(num, generation)), Object::Null);
assert_eq!(live, expected.contains(&generation), "object {num} generation {generation}");
}
};
let doc = Document::open(contradictory_entry(false)).unwrap();
agrees(&doc, 1, &[0]);
agrees(&doc, 3, &[0, 1]);
agrees(&doc, 9, &[]);
let mut repaired = doc.clone();
repaired.set_existing(ObjRef::new(3, 0), Object::Int(77)).unwrap();
agrees(&repaired, 3, &[0, 1]);
let mut narrowed = doc.clone();
narrowed.set(ObjRef::new(3, 2), Object::Int(99));
agrees(&narrowed, 3, &[2]);
let mut freed = doc.clone();
freed.free(ObjRef::new(3, 0));
agrees(&freed, 3, &[]);
agrees(&Document::open(contradictory_entry(true)).unwrap(), 3, &[]);
let mut doc =
Document::open(object_file(&[(1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>")])).unwrap();
let member = doc.add(Object::Dict(Default::default()));
doc.update_dict(ObjRef::new(1, 0), |c| c.set(b"Extra".to_vec(), Object::Ref(member))).unwrap();
let packed = Document::open(Arc::new(write_full(&doc, &SaveOptions { object_streams: true, ..SaveOptions::default() }).unwrap())).unwrap();
assert!(matches!(packed.xref_entry(member.num), Some(XrefEntry::InStream { .. })), "the member is in an object stream");
agrees(&packed, member.num, &[0]);
}
+17
View File
@@ -0,0 +1,17 @@
//! Tests of the `cos` feature, linked into one test binary. Not built for the web: proptest and
//! the oracle processes are not there.
#![cfg(all(feature = "cos", not(target_arch = "wasm32")))]
mod bounded_writer;
mod corpus;
mod efopen;
mod encryption;
mod file_identifier;
mod generations;
mod oracle;
mod proptests;
mod serialize_format;
mod shared_streams;
#[path = "../../src/cos/testkit.rs"]
mod testkit;
mod writer;
+231
View File
@@ -0,0 +1,231 @@
//! External oracle processes: independent implementations the object layer's output is checked
//! against (from PdfCraft's `pdfcraft-oracle`).
//!
//! **A missing oracle is either a declared skip or a failure, never a silent pass.**
//! [`REQUIRED_VAR`] decides which: set it where the tools are installed, so their absence fails
//! the run; without it, a missing tool prints why and the test returns.
//!
//! Test scaffolding, compiled into this test binary only.
#![allow(dead_code)]
use std::fmt;
use std::process::Command;
/// Setting this environment variable to any value turns a missing oracle from a skip into a
/// failure. CI sets it on the runners that install the tools.
///
/// Any value counts, including `0` — the same convention as `CRAFT_FONTS_REQUIRED`, which this
/// deliberately mirrors. The surprising half of that convention errs toward *requiring* an oracle,
/// which is the safe direction for a gate whose whole purpose is to stop absence passing quietly.
pub const REQUIRED_VAR: &str = "CRAFT_PDF_ORACLES_REQUIRED";
/// An external program found on `PATH`, ready to run.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Oracle {
program: &'static str,
version: String,
}
impl Oracle {
/// A fresh [`Command`] for the program. Callers add their own arguments.
pub fn command(&self) -> Command {
Command::new(self.program)
}
/// The program name, as it is looked up on `PATH`.
pub fn program(&self) -> &'static str {
self.program
}
/// The first line the program printed when asked for its version, for test output. Empty when
/// it printed nothing recognisable; presence of the binary is what decides availability, not
/// whether its version could be parsed.
pub fn version(&self) -> &str {
&self.version
}
}
/// An oracle that could not be used, and whether that is fatal.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Absent {
program: &'static str,
cause: String,
required: bool,
}
impl Absent {
/// Whether this absence must fail the run rather than skip it, i.e. whether [`REQUIRED_VAR`]
/// was set when the probe ran.
pub fn required(&self) -> bool {
self.required
}
/// The program that was looked for.
pub fn program(&self) -> &'static str {
self.program
}
/// How to install it, so a skip notice tells the reader what to do about it.
fn install_hint(program: &str) -> &'static str {
match program {
"qpdf" => "apt install qpdf / brew install qpdf",
"pdftoppm" | "pdftotext" => "apt install poppler-utils / brew install poppler",
_ => "install it and put it on PATH",
}
}
}
impl fmt::Display for Absent {
/// Reads as an instruction either way: required absences are failures and name the gate that
/// made them one, optional absences name the install command.
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
if self.required {
write!(
f,
"oracle `{}` is required but {}: {} is set, so this is a failure rather than a skip ({})",
self.program,
self.cause,
REQUIRED_VAR,
Self::install_hint(self.program),
)
} else {
write!(
f,
"skipping: oracle `{}` {} ({}); set {} to make this a failure instead",
self.program,
self.cause,
Self::install_hint(self.program),
REQUIRED_VAR,
)
}
}
}
/// Look for `program` on `PATH`, asking it for its version to confirm it runs.
///
/// Presence is decided by whether the program could be started at all, not by its exit status: a
/// tool that runs but reports its version oddly is still an oracle.
pub fn find(program: &'static str, version_arg: &str) -> Result<Oracle, Absent> {
let required = std::env::var_os(REQUIRED_VAR).is_some();
match Command::new(program).arg(version_arg).output() {
Ok(out) => {
// Poppler's tools print their version to stderr, qpdf to stdout.
let text = if out.stdout.is_empty() { &out.stderr } else { &out.stdout };
let version = String::from_utf8_lossy(text).lines().next().unwrap_or_default().trim().to_owned();
Ok(Oracle { program, version })
}
Err(e) => Err(Absent { program, cause: format!("is not on PATH ({e})"), required }),
}
}
/// qpdf: structural validation (`--check`) and the independent owner-password oracle.
pub fn qpdf() -> Result<Oracle, Absent> {
find("qpdf", "--version")
}
/// Poppler's `pdftoppm`: renders a page to pixels, for comparing against our own raster output.
pub fn pdftoppm() -> Result<Oracle, Absent> {
find("pdftoppm", "-v")
}
/// Poppler's `pdftotext`: the text-extraction oracle (`cargo xtask text-oracle` scores word F1
/// against it over the corpus).
pub fn pdftotext() -> Result<Oracle, Absent> {
find("pdftotext", "-v")
}
/// Binds an oracle, or returns from the calling test after saying on stderr why it was skipped.
///
/// When [`REQUIRED_VAR`] is set the absence fails the test instead. The panic is deliberate and is
/// how a test reports failure; it expands into the caller, which is test code, so the workspace's
/// never-panic rule is not weakened — this crate itself contains no panicking path.
///
/// The macro expands to a `return`, so it belongs in a test function returning `()`. A test that
/// returns `Result` should match on the probe itself.
macro_rules! or_skip {
($probe:expr) => {
match $probe {
::core::result::Result::Ok(oracle) => oracle,
::core::result::Result::Err(absent) => {
let absent: $crate::oracle::Absent = absent;
::core::assert!(!$crate::oracle::Absent::required(&absent), "{absent}");
::std::eprintln!("{absent}");
return;
}
}
};
}
pub(crate) use or_skip;
#[cfg(test)]
mod tests {
use super::*;
/// The one behaviour the crate exists for: absence is never silent, and which of the two loud
/// forms it takes is decided by the environment variable.
#[test]
fn a_missing_oracle_is_fatal_exactly_when_it_is_required() {
let optional = Absent { program: "qpdf", cause: "is not on PATH".into(), required: false };
let required = Absent { required: true, ..optional.clone() };
assert!(!optional.required());
assert!(required.required());
// Both say what is wrong and how to fix it; only the required one calls itself a failure.
let (opt, req) = (optional.to_string(), required.to_string());
assert!(opt.contains("skipping"), "{opt}");
assert!(opt.contains("apt install qpdf"), "{opt}");
assert!(opt.contains(REQUIRED_VAR), "{opt}");
assert!(req.contains("required"), "{req}");
assert!(req.contains("failure"), "{req}");
assert!(!req.contains("skipping"), "{req}");
}
/// A tool that does not exist is reported as absent rather than hanging or panicking, and the
/// reason names the program.
#[test]
fn a_program_that_does_not_exist_is_absent_with_a_reason() {
let Err(absent) = find("pdfcraft-no-such-oracle-binary", "--version") else {
panic!("a nonexistent program must not be reported as present");
};
assert_eq!(absent.program(), "pdfcraft-no-such-oracle-binary");
let text = absent.to_string();
assert!(text.contains("pdfcraft-no-such-oracle-binary"), "{text}");
assert!(text.contains("not on PATH"), "{text}");
}
/// Hostile and odd program names must not panic, hang or be mistaken for a working oracle.
#[test]
fn hostile_program_names_are_absent_not_a_panic() {
for name in ["", " ", "\0bad", "../../etc/passwd", &"x".repeat(4096)] {
// `find` takes a 'static str; leak the generated ones, which is fine in a test.
let name: &'static str = Box::leak(name.to_owned().into_boxed_str());
let found = find(name, "--version");
if let Ok(o) = found {
// Nothing by these names should run; if one somehow does, it must still be sane.
assert_eq!(o.program(), name);
}
}
}
/// The install hint is specific per tool, so a skip notice is actionable.
#[test]
fn every_known_oracle_has_an_install_hint() {
for p in ["qpdf", "pdftoppm", "pdftotext"] {
let hint = Absent::install_hint(p);
assert!(hint.contains("install"), "{p}: {hint}");
assert_ne!(hint, Absent::install_hint("something-else"), "{p} needs its own hint");
}
}
/// Probing is read-only and repeatable: asking twice gives the same answer.
#[test]
fn probing_is_stable() {
let a = qpdf().map(|o| o.program());
let b = qpdf().map(|o| o.program());
assert_eq!(a.is_ok(), b.is_ok());
if let (Ok(a), Ok(b)) = (a, b) {
assert_eq!(a, b);
}
}
}
+464
View File
@@ -0,0 +1,464 @@
//! Property tests for the object layer's save fidelity — proptest coverage of two areas raised in
//! pdfcraft #1187. **This does not close pdfcraft #1187.** That issue asks for five Hegel properties; this adds
//! save/reopen and edit-history coverage using `proptest`, which is already a dev-dependency of this
//! crate, so no new dependency is needed to review it. Left for follow-ups:
//! #1187's reference-following comparison, its snapshot/undo state machine, the password property (the
//! one whose pilot found #1185) and its damaged-file byte splices. The object-level serialise → parse
//! round trip it also lists already exists inline in `src/writer.rs`.
//!
//! Covered:
//!
//! 1. **A full save keeps every value it wrote into the catalog**, in both output styles (objects
//! packed into object streams, §7.5.7; the classic cross-reference table otherwise), keeps the
//! catalog's own structure intact (`/Type /Catalog`, and a `/Pages` node whose `/Kids` array and
//! `/Count` agree — the fixture's tree is empty, so this pins shape and agreement, not rendered
//! pages), preserves dictionary
//! key order (which `README.md:75` promises is byte-stable), emits spec-shaped §7.5.4 cross-reference
//! rows, and reopens with nothing to repair.
//! 2. **An incremental save is a strict append**: the original bytes come out untouched at the front
//! (§7.5.6), the file keeps two revisions whose first still points at the base cross-reference and
//! opens standalone, the edit is visible while every existing value stays put in order, and it
//! reopens with nothing to repair.
//! 3. **A write/delete sequence equals a plain map**, checked after every step and again once saved and
//! reopened. A deleted key must stay gone — not return its old value, not read as null either.
//! Deletes are generated against keys that were just written, so the delete path cannot be skipped.
//!
//! ## Scope, stated honestly
//! Generated values are pure data — no streams and no indirect references — so this says nothing about
//! stream payloads or the renumbering a full save legitimately performs. Strings are compared as bytes,
//! so literal ↔ hexadecimal re-emission is not treated as a change (§7.3.6); the *choice* of spelling
//! is therefore not asserted, and `PdfString.hex` is not pinned here. Dictionary keys at the catalog
//! level *are* compared in order, so reversing them fails; nested dictionary order inside a value is
//! sorted away by [`canon`] and is not pinned. Generated reals stay within ±1e6: `write_real`
//! (`src/writer.rs:46`) turns a non-finite real into `0` and an integral real below 1e15 into integer
//! syntax, so [`canon`] applies the writer's documented normalisation to **both** sides rather than
//! forgiving a mismatch afterwards.
use std::sync::Arc;
use proptest::prelude::*;
use proptest::test_runner::TestCaseError;
use craft_pdf::cos::{Dict, Document, Object, PdfString, SaveOptions, write_full, write_incremental};
/// The prefix every generated key carries. `Document::new_empty` puts `/Type` and `/Pages` in the
/// catalog; property 1 asserts those two structurally rather than ignoring them.
const KEY: &str = "px";
/// Generated keys go into the catalog in this deliberately unsorted sequence. Ascending insertion would make
/// "preserves dictionary key order" and "every dictionary sorted by key" the same observable, so a writer
/// that silently sorted would rewrite every saved byte and stay green. This scramble catches sorting of the
/// keys it names; it does NOT on its own pin where structural entries like `/Type` sit, because `saved`
/// filters to the `px` prefix - that gap is closed separately by [`catalog_keys`] asserting the full
/// sequence. Nested dictionary order is deliberately not pinned (see `canon`). Every permutation of the
/// generated keys is NOT drawn here either: one fixed order per case, chosen to make sorting detectable.
const INSERTION_ORDER: [usize; 8] = [5, 0, 7, 2, 6, 1, 4, 3];
/// A value reduced to a comparable spelling, with the writer's documented normalisations applied.
fn canon(o: &Object) -> String {
match o {
Object::Null => "null".into(),
Object::Bool(b) => format!("b{b}"),
Object::Int(i) => format!("i{i}"),
Object::Real(r) => {
if !r.is_finite() {
"i0".into()
} else if r.fract() == 0.0 && r.abs() < 1e15 {
format!("i{}", *r as i64)
} else {
format!("r{r}")
}
}
Object::Name(n) => format!("n{}", String::from_utf8_lossy(n)),
// Bytes, because a literal may be re-emitted as hexadecimal or the reverse; both spellings
// denote the same bytes (§7.3.6). The `hex` flag itself is deliberately not compared — see
// the scope note at the top.
Object::String(s) => format!("s{}", s.bytes.iter().map(|b| format!("{b:02x}")).collect::<String>()),
Object::Array(a) => format!("[{}]", a.iter().map(canon).collect::<Vec<_>>().join(",")),
Object::Dict(d) => {
// Sorted inside a value: nested key order is not pinned here. Catalog-level order is, and
// it is asserted in iteration order by `saved`.
let mut pairs: Vec<(String, String)> = d.iter().map(|(k, v)| (String::from_utf8_lossy(k).into_owned(), canon(v))).collect();
pairs.sort();
format!("{{{}}}", pairs.iter().map(|(k, v)| format!("{k}=>{v}")).collect::<Vec<_>>().join(","))
}
// Unreachable under the generators below, which build no streams and no references. Kept total
// so this compiles if someone widens them — at which point it should compare decoded bytes and
// resolved targets instead of a constant.
Object::Stream(_) => "stream".into(),
Object::Ref(r) => format!("ref{r}"),
}
}
fn leaf() -> impl Strategy<Value = Object> {
prop_oneof![
Just(Object::Null),
any::<bool>().prop_map(Object::Bool),
(-1_000_000i64..1_000_000i64).prop_map(Object::Int),
(-1e6f64..1e6f64).prop_map(Object::Real),
// Non-finite reals, drawn explicitly. `write_real` (`src/writer.rs:46`) replaces them with
// `0`, and [`canon`] spells both sides that way, so this is what makes the normalisation
// described in the module header actually exercised rather than merely asserted in prose.
prop::sample::select(vec![f64::NAN, f64::INFINITY, f64::NEG_INFINITY]).prop_map(Object::Real),
// Raw bytes, including ones a literal string must escape and the NUL byte.
prop::collection::vec(any::<u8>(), 0..12).prop_map(|b| Object::String(PdfString::literal(b))),
"[A-Za-z][A-Za-z0-9]{0,6}".prop_map(|s| Object::name(&s)),
]
}
/// Nested values to a bounded depth, so generation cannot recurse without limit.
fn value(depth: usize) -> BoxedStrategy<Object> {
if depth == 0 {
return leaf().boxed();
}
prop_oneof![
leaf().boxed(),
prop::collection::vec(value(depth - 1), 0..5).prop_map(Object::Array).boxed(),
prop::collection::btree_map("[a-z]{1,4}", value(depth - 1), 0..5)
.prop_map(|m| {
let mut d = Dict::new();
for (k, v) in m {
d.set(k.into_bytes(), v);
}
Object::Dict(d)
})
.boxed(),
]
.boxed()
}
fn find(hay: &[u8], needle: &[u8]) -> Option<usize> {
hay.windows(needle.len()).position(|w| w == needle)
}
/// Every catalog key, in the dictionary's own order - structural entries included. `saved` filters to
/// `px*`, which is why sorting of generated keys is visible there but `/Type` moving to the end was not;
/// this covers that. Read from memory before a save and from the parsed file afterwards, so the two sides
/// share only `Dict` iteration, not the serialiser under test.
fn catalog_keys(doc: &Document) -> Vec<String> {
let root = match doc.root() {
Some(r) => r,
None => return Vec::new(),
};
match doc.get(root).as_dict() {
Some(d) => d.iter().map(|(k, _)| String::from_utf8_lossy(k).into_owned()).collect(),
None => Vec::new(),
}
}
/// The catalog's `px*` entries in **iteration order**, so a reordered save fails.
fn saved(doc: &Document) -> Result<Vec<(String, String)>, TestCaseError> {
let root = doc.root().ok_or_else(|| TestCaseError::fail("no catalog in the saved file"))?;
let cat = doc.get(root);
let d = match cat.as_dict() {
Some(d) => d,
None => return Err(TestCaseError::fail("catalog is not a dictionary")),
};
Ok(d.iter().filter(|(k, _)| k.starts_with(KEY.as_bytes())).map(|(k, v)| (String::from_utf8_lossy(k).into_owned(), canon(v))).collect())
}
/// The catalog's structure, which a reader needs whether or not it holds our test keys.
fn assert_catalog_structure(doc: &Document, style: &str) -> Result<(), TestCaseError> {
let root = doc.root().ok_or_else(|| TestCaseError::fail(format!("no catalog after the {style} save")))?;
let cat = doc.get(root);
let d = match cat.as_dict() {
Some(d) => d,
None => return Err(TestCaseError::fail(format!("catalog is not a dictionary after the {style} save"))),
};
prop_assert_eq!(d.name(b"Type"), Some(&b"Catalog"[..]), "catalog /Type changed on the {} save", style);
let pages = match d.get(b"Pages") {
Some(p) => p,
None => return Err(TestCaseError::fail(format!("catalog lost /Pages on the {style} save"))),
};
let resolved = doc.resolve(pages);
let pd = match resolved.as_dict() {
Some(d) => d,
None => return Err(TestCaseError::fail(format!("/Pages does not resolve to a dictionary on the {style} save"))),
};
prop_assert_eq!(pd.name(b"Type"), Some(&b"Pages"[..]), "/Pages /Type changed on the {} save", style);
let kids = match pd.get(b"Kids").and_then(Object::as_array) {
Some(k) => k,
None => return Err(TestCaseError::fail(format!("/Pages /Kids is missing or not an array on the {style} save"))),
};
// /Count has to agree with /Kids. A page tree whose count lies still opens, then misnumbers every
// page downstream, so checking presence alone would not be enough. This fixture's tree is empty, so
// what is pinned here is shape and agreement — not the existence of rendered pages.
prop_assert_eq!(pd.int(b"Count"), Some(kids.len() as i64), "/Pages /Count disagrees with its {} kids on the {} save", kids.len(), style);
Ok(())
}
/// §7.5.4: every entry of a classic cross-reference table is exactly 20 bytes, closed by a two-byte
/// end of line. Unpadded offsets still open in lenient readers and corrupt strict ones, so the byte
/// shape is asserted rather than inferred from a successful reopen.
fn assert_classic_xref_rows(bytes: &[u8]) -> Result<(), TestCaseError> {
// Anchored on the file's own startxref (§7.5.3) rather than the first "xref" text in the buffer:
// a generated document name could otherwise point the scan at prose. A startxref that does not land
// on the `xref` keyword means a cross-reference stream, which has no table to check here.
let Some(xoff) = startxref(bytes) else { return Err(TestCaseError::fail("no readable startxref in a classic-table save")) };
let x = xoff as usize;
if bytes.get(x..x + 4) != Some(&b"xref"[..]) {
return Ok(());
}
let nl = match bytes[x..].iter().position(|b| *b == b'\n') {
Some(n) => n,
None => return Err(TestCaseError::fail("xref keyword not followed by a newline")),
};
let rest = &bytes[x + nl + 1..];
let end = match find(rest, b"trailer") {
Some(e) => e,
None => return Err(TestCaseError::fail("cross-reference table with no trailer")),
};
let mut rows = 0usize;
let mut expected_rows = 0usize;
for line in rest[..end].split(|b| *b == b'\n') {
if line.is_empty() {
continue;
}
// A CR before the line feed belongs to the two-byte terminator (§7.5.4), not to the keyword:
// stripped here so a CR+LF row is still recognised as a legal row rather than reported (wrongly)
// as "no cross-reference rows found". Producers do emit CR+LF terminators.
//
// What catches what: classification keys off the keyword, so a row whose keyword is garbage never
// reaches the width and length asserts below. Those rows are named by the `else` arm at the end
// of this loop; the row-count compare after it is only a backstop. The length assert is not the
// safety net for malformed keywords, and this comment used to imply that it was.
let fields: Vec<&[u8]> = line
.split(|b| *b == b' ')
.filter(|f| !f.is_empty())
.map(|f| match f.last() {
Some(b'\r') => &f[..f.len() - 1],
_ => f,
})
.collect();
if fields.len() == 2 && fields.iter().all(|f| f.iter().all(u8::is_ascii_digit)) {
// `<first> <count>` subsection header.
expected_rows += std::str::from_utf8(fields[1]).ok().and_then(|s| s.parse::<usize>().ok()).unwrap_or(0);
continue;
}
if fields.len() == 3 && matches!(fields[2], b"n" | b"f") {
// Classified by keyword, not by width: an unpadded row must still be recognised as a row, so
// the failure names the real crime (wrong width) instead of the misleading "no rows found".
// Both keywords are legal — `n` in use, `f` free (the head of the free chain carries
// generation 65535). §7.5.4 fixes each field's width and the whole entry at 20 bytes.
prop_assert_eq!(
fields[0].len(),
10,
"a cross-reference offset must be 10 digits (§7.5.4), got {:?} in row {:?}",
String::from_utf8_lossy(fields[0]),
String::from_utf8_lossy(line)
);
prop_assert_eq!(
fields[1].len(),
5,
"a cross-reference generation must be 5 digits (§7.5.4), got {:?} in row {:?}",
String::from_utf8_lossy(fields[1]),
String::from_utf8_lossy(line)
);
// Exactly 20 bytes per entry in the file (§7.5.4). Both legal terminators land on 19 bytes
// for the line as split here: SP+LF, which is what this writer emits today, and CR+LF, whose
// terminator is the pair itself. A 21-byte entry (SP+CRLF, or a doubled CR) fails here.
prop_assert_eq!(
line.len(),
19,
"a cross-reference row is {} bytes; §7.5.4 wants exactly 20 in the file (SP+LF or CR+LF): {:?}",
line.len(),
String::from_utf8_lossy(line)
);
rows += 1;
} else {
// Inside the table region, a line is either a subsection header or an entry. Anything else is
// named here with its bytes quoted, instead of surfacing later as a bare row-count mismatch
// ("no cross-reference rows found") that blames the table for one bad line.
prop_assert_eq!(
fields.len(),
3,
"unrecognised line in the cross-reference table, want `<offset> <generation> n|f`: {:?}",
String::from_utf8_lossy(line)
);
prop_assert!(
matches!(fields[2], b"n" | b"f"),
"illegal cross-reference keyword {:?} in row {:?}",
String::from_utf8_lossy(fields[2]),
String::from_utf8_lossy(line)
);
}
}
prop_assert!(rows > 0, "no cross-reference rows found in a classic-table save");
prop_assert_eq!(rows, expected_rows, "the subsection headers promised {} rows, the table holds {}", expected_rows, rows);
Ok(())
}
/// The `startxref` value of a file, which the next incremental update must point back at via `/Prev`.
/// The keyword is followed by whitespace, then the offset (§7.5.3), so leading whitespace is skipped.
fn startxref(bytes: &[u8]) -> Option<u64> {
let at = find(bytes, b"startxref")?;
let tail = &bytes[at + 9..];
let digits: String = tail.iter().skip_while(|b| b.is_ascii_whitespace()).take_while(|b| b.is_ascii_digit()).map(|b| char::from(*b)).collect();
digits.parse().ok()
}
proptest! {
// 256 cases per property; the suite runs in well under a second, so coverage is free here.
#![proptest_config(ProptestConfig::with_cases(256))]
/// 1. A full save keeps the values it wrote, their order, the catalog's structure and the shape of
/// its cross-reference table — in both output styles — and reopens with nothing to repair.
#[test]
fn full_save_keeps_the_catalog_intact(map in prop::collection::btree_map(0usize..8, value(3), 1..6)) {
let mut doc = Document::new_empty();
let root = doc.root().expect("an empty document has a catalog");
doc.update_dict(root, |d| {
for i in INSERTION_ORDER.into_iter().filter(|&i| map.contains_key(&i)) {
d.set(format!("{KEY}{i}").into_bytes(), map[&i].clone());
}
})
.map_err(|e| TestCaseError::fail(format!("cannot write the generated catalog: {e}")))?;
// Insertion order is this scrambled sequence filtered to the drawn keys: what must come back.
let expected: Vec<(String, String)> = INSERTION_ORDER.into_iter().filter(|&i| map.contains_key(&i)).map(|i| (format!("{KEY}{i}"), canon(&map[&i]))).collect();
// Full key sequence taken from memory BEFORE any byte is written - /Type and /Pages included.
let order_before = catalog_keys(&doc);
// Right before any save, or the later assertion would prove nothing about saving.
prop_assert_eq!(saved(&doc)?, expected.clone(), "the in-memory catalog is wrong before saving");
for object_streams in [true, false] {
let style: &str = if object_streams { "object-stream" } else { "classic-table" };
let opts = SaveOptions { object_streams, ..SaveOptions::default() };
let bytes = write_full(&doc, &opts).map_err(|e| TestCaseError::fail(format!("full save failed: {e}")))?;
if !object_streams {
assert_classic_xref_rows(&bytes).map_err(|e| TestCaseError::fail(format!("{style}: {e}")))?;
}
let again = Document::open(Arc::new(bytes)).map_err(|e| TestCaseError::fail(format!("our own full save must reopen: {e}")))?;
prop_assert!(again.repair_log().is_empty(), "a file we wrote must not need repair ({}); log: {:?}", style, again.repair_log());
// Values *and* order: README.md promises dictionaries keep their key order.
prop_assert_eq!(saved(&again)?, expected.clone(), "the catalog changed on a {} save (values or order)", style);
// Structural position too: /Type and /Pages must not drift to the end or swap places.
prop_assert_eq!(catalog_keys(&again), order_before.clone(), "the catalog's full key sequence changed on a {} save", style);
assert_catalog_structure(&again, style)?;
}
}
/// 2. An incremental save leaves the original bytes untouched, chains back through `/Prev`, and
/// reopens carrying every intended value plus the edit. The base is built **both** ways -
/// cross-reference stream and classic table - because the two are serialised by different code, and a
/// path that never runs cannot fail here. Expected contents are computed from the values we put in,
/// not read back out of the writer: comparing against our own earlier output would pass a writer that
/// is consistently wrong, which is the one thing a fidelity test must not do.
#[test]
fn incremental_save_appends_without_touching_the_original(map in prop::collection::btree_map(0usize..8, value(2), 1..5), marker in -1_000_000i64..1_000_000i64) {
let mut doc = Document::new_empty();
let root = doc.root().expect("an empty document has a catalog");
doc.update_dict(root, |d| {
for i in INSERTION_ORDER.into_iter().filter(|&i| map.contains_key(&i)) {
d.set(format!("{KEY}{i}").into_bytes(), map[&i].clone());
}
})
.map_err(|e| TestCaseError::fail(format!("cannot write the generated catalog: {e}")))?;
// Our own record of what the catalog should hold, derived from the inputs before any byte is
// written. The edit key joins it below; order follows INSERTION_ORDER, not sorted order (§7.5.4).
let intended: Vec<(String, String)> = INSERTION_ORDER.into_iter().filter(|&i| map.contains_key(&i)).map(|i| (format!("{KEY}{i}"), canon(&map[&i]))).collect();
for object_streams in [true, false] {
let style: &str = if object_streams { "object-stream" } else { "classic-table" };
let opts = SaveOptions { object_streams, ..SaveOptions::default() };
let base = write_full(&doc, &opts).map_err(|e| TestCaseError::fail(format!("{}: full save failed: {e}", style)))?;
let base_xref = startxref(&base).ok_or_else(|| TestCaseError::fail(format!("{}: the base file has no readable startxref", style)))?;
let mut again = Document::open(Arc::new(base.clone())).map_err(|e| TestCaseError::fail(format!("{}: the saved file must reopen: {e}", style)))?;
// Independent of the writer's own readings: the reopen must show what we intended to write.
prop_assert_eq!(saved(&again)?, intended.clone(), "{}: the base catalog differs from what we put in", style);
let root2 = again.root().expect("the reopened document has a catalog");
again.update_dict(root2, |d| d.set(b"pxMark".to_vec(), Object::Int(marker))).map_err(|e| TestCaseError::fail(format!("{}: cannot apply the incremental edit: {e}", style)))?;
let bytes = write_incremental(&again, &opts).map_err(|e| TestCaseError::fail(format!("{}: incremental save failed: {e}", style)))?;
prop_assert!(bytes.len() > base.len(), "{}: an incremental save must add bytes ({} over a base of {})", style, bytes.len(), base.len());
prop_assert_eq!(bytes.get(..base.len()), Some(&base[..]), "{}: an incremental save must not rewrite the original bytes", style);
let out = Document::open(Arc::new(bytes.clone())).map_err(|e| TestCaseError::fail(format!("{}: the incremental file must reopen: {e}", style)))?;
prop_assert!(out.repair_log().is_empty(), "{}: an increment we wrote must not need repair; log: {:?}", style, out.repair_log());
// The revision chain in the crate's own terms (Revision { xref_offset, is_stream }): two
// revisions, the first still pointing at the base cross-reference. Without that link no other
// reader can reach the earlier revision's objects, which is the classic incremental corruption.
// /Prev sits in the trailer for a table base and inside the cross-reference stream for a
// stream base, so it is not read here; `revisions` covers both (§7.5.3-7.5.6).
let revs = out.revisions();
prop_assert_eq!(revs.len(), 2, "{}: an incremental save must leave two revisions: {:?}", style, revs);
prop_assert_eq!(revs[0].xref_offset, base_xref, "{}: the first revision must still point at the base xref {}", style, base_xref);
prop_assert!(revs[1].xref_offset > base_xref, "{}: the second revision must sit after the first: {:?}", style, revs);
let ends = out.revision_ends();
prop_assert_eq!(ends.last().copied(), Some(bytes.len()), "{}: the last revision must run to the end of the file: {:?}", style, ends);
// The base revision opens on its own. "Nothing was rewritten" buys the user this: cut the file
// at the append point and the earlier document still reads, with our intended values.
if let Some(end0) = ends.first().copied() {
let before_doc = Document::open(Arc::new(bytes[..end0].to_vec())).map_err(|e| TestCaseError::fail(format!("{}: the base revision must open alone: {e}", style)))?;
prop_assert_eq!(saved(&before_doc)?, intended.clone(), "{}: the base revision lost or changed values when opened without the increment", style);
}
assert_catalog_structure(&out, style)?;
let mut expected = intended.clone();
expected.push(("pxMark".into(), canon(&Object::Int(marker))));
prop_assert_eq!(saved(&out)?, expected.clone(), "{}: the increment lost a value, reordered the catalog, or dropped the edit", style);
}
}
/// 3. Writes then deletes against a plain ordered model, checked after every step and again after
/// save and reopen. Deletes always target a key that exists, so the delete path is never empty.
#[test]
fn edit_sequence_equals_a_plain_map(plan in prop::collection::btree_map(0usize..8, value(2), 1..6).prop_flat_map(|sets| {
let keys: Vec<usize> = sets.keys().copied().collect();
(Just(sets), prop::collection::vec(0usize..keys.len(), 1..=keys.len())).prop_map(move |(s, picks)| {
let mut dels: Vec<usize> = picks.into_iter().map(|i| keys[i]).collect();
// Deduplicated: the same key drawn twice would make the second delete target a key that is
// already gone, which is legal PDF but not what this property means to assert. `picks` draws
// 1..=keys.len() over a non-empty key set, so `dels` cannot come back empty — the guarantee
// lives in the generator, not in an assume that could never fire.
dels.sort_unstable();
dels.dedup();
(s, dels)
})
})) {
let (sets, deletes) = plan;
let mut doc = Document::new_empty();
let root = doc.root().expect("an empty document has a catalog");
let mut model: Vec<(String, String)> = Vec::new();
for (i, o) in &sets {
let key = format!("{KEY}{i}");
doc.update_dict(root, |d| d.set(key.clone().into_bytes(), o.clone())).map_err(|e| TestCaseError::fail(format!("cannot write {key}: {e}")))?;
model.push((key.clone(), canon(o)));
prop_assert_eq!(saved(&doc)?, model.clone(), "after writing {} the catalog is wrong before saving", key);
}
for i in &deletes {
let key = format!("{KEY}{i}");
prop_assert!(model.iter().any(|(k, _)| k == &key), "the generator tried to delete a key that was never written: {}", key);
doc.update_dict(root, |d| {
d.remove(key.as_bytes());
})
.map_err(|e| TestCaseError::fail(format!("cannot delete {key}: {e}")))?;
model.retain(|(k, _)| k != &key);
prop_assert_eq!(saved(&doc)?, model.clone(), "after deleting {} it is still reachable", key);
}
let bytes = write_full(&doc, &SaveOptions::default()).map_err(|e| TestCaseError::fail(format!("full save failed: {e}")))?;
let again = Document::open(Arc::new(bytes)).map_err(|e| TestCaseError::fail(format!("our own save must reopen: {e}")))?;
let got = saved(&again)?;
prop_assert_eq!(got.clone(), model.clone(), "the reopened catalog differs from the edit history (values or order)");
for i in 0..8usize {
let key = format!("{KEY}{i}");
if !model.iter().any(|(k, _)| k == &key) {
prop_assert!(!got.iter().any(|(g, _)| *g == key), "a deleted key came back after save+reopen: {}", key);
}
}
}
}
+116
View File
@@ -0,0 +1,116 @@
//! `serialize` writes integers, references, hexadecimal strings, octal escapes and `#xx` name
//! escapes with its own digit loops. They must give the bytes the standard formatter gives.
use craft_pdf::cos::{ObjRef, Object, PdfString, serialize};
use proptest::prelude::*;
fn bytes_of(o: &Object) -> Vec<u8> {
let mut v = Vec::new();
serialize(o, &mut v);
v
}
#[test]
fn integer_extremes_and_digit_boundaries() {
let mut cases = vec![0i64, 1, -1, 9, 10, 99, 100, i64::MAX, i64::MIN, i64::MIN + 1, i64::from(i32::MAX), i64::from(i32::MIN)];
let mut p = 1i64;
while let Some(next) = p.checked_mul(10) {
cases.extend([next - 1, next, next + 1, -next]);
p = next;
}
for i in cases {
assert_eq!(bytes_of(&Object::Int(i)), i.to_string().into_bytes(), "{i}");
}
}
#[test]
fn reference_extremes() {
for (n, g) in [(0, 0), (1, 0), (u32::MAX, u16::MAX), (10, 65535), (4_000_000_000, 7)] {
assert_eq!(bytes_of(&Object::Ref(ObjRef::new(n, g))), format!("{n} {g} R").into_bytes());
}
}
#[test]
fn every_byte_value_in_hex_strings_octal_escapes_and_names() {
let all: Vec<u8> = (0..=255).collect();
let hex: String = all.iter().map(|b| format!("{b:02X}")).collect();
assert_eq!(bytes_of(&Object::String(PdfString { bytes: all.clone(), hex: true })), format!("<{hex}>").into_bytes());
// A literal string with a few control bytes escapes them in octal.
let lit = bytes_of(&Object::String(PdfString { bytes: b"abcdefghijkl\x01\x7f\xffz".to_vec(), hex: false }));
assert_eq!(lit, b"(abcdefghijkl\\001\\177\\377z)");
let name = bytes_of(&Object::Name(all.clone()));
let mut want = b"/".to_vec();
for &b in &all {
let regular = (b'!'..=b'~').contains(&b) && !matches!(b, b'(' | b')' | b'<' | b'>' | b'[' | b']' | b'{' | b'}' | b'/' | b'%' | b'#');
if regular {
want.push(b);
} else {
want.extend_from_slice(format!("#{b:02X}").as_bytes());
}
}
assert_eq!(name, want);
}
#[test]
fn whole_real_at_i32_min_stays_a_real_token() {
// -2^31 is outside the Annex C integer range (±2^31-1), so it keeps its trailing point.
assert_eq!(bytes_of(&Object::Real(f64::from(i32::MIN))), b"-2147483648.");
}
proptest! {
#[test]
fn integers_match_the_formatter(i in any::<i64>()) {
prop_assert_eq!(bytes_of(&Object::Int(i)), format!("{i}").into_bytes());
}
#[test]
fn whole_reals_match_the_formatter(i in -i32::MAX..=i32::MAX) {
prop_assert_eq!(bytes_of(&Object::Real(f64::from(i))), format!("{i}").into_bytes());
}
#[test]
fn references_match_the_formatter(n in any::<u32>(), g in any::<u16>()) {
prop_assert_eq!(bytes_of(&Object::Ref(ObjRef::new(n, g))), format!("{n} {g} R").into_bytes());
}
#[test]
fn hex_strings_match_the_formatter(bytes in proptest::collection::vec(any::<u8>(), 0..300)) {
let mut want = String::from("<");
for b in &bytes {
want.push_str(&format!("{b:02X}"));
}
want.push('>');
prop_assert_eq!(bytes_of(&Object::String(PdfString { bytes, hex: true })), want.into_bytes());
}
#[test]
fn binary_literal_strings_match_the_formatter(bytes in proptest::collection::vec(any::<u8>(), 0..300)) {
// Mostly-binary input is written as hex; mostly-text input with escapes is a literal.
let got = bytes_of(&Object::String(PdfString { bytes: bytes.clone(), hex: false }));
let binary = bytes.iter().filter(|b| !(b' '..=b'~').contains(*b) && !matches!(b, b'\n' | b'\r' | b'\t')).count();
let mut want = Vec::new();
if binary * 4 > bytes.len().max(1) {
want.push(b'<');
for b in &bytes {
want.extend_from_slice(format!("{b:02X}").as_bytes());
}
want.push(b'>');
} else {
want.push(b'(');
for &b in &bytes {
match b {
b'(' | b')' | b'\\' => want.extend_from_slice(&[b'\\', b]),
b'\n' => want.extend_from_slice(b"\\n"),
b'\r' => want.extend_from_slice(b"\\r"),
b'\t' => want.extend_from_slice(b"\\t"),
0x08 => want.extend_from_slice(b"\\b"),
0x0C => want.extend_from_slice(b"\\f"),
b' '..=b'~' => want.push(b),
_ => want.extend_from_slice(format!("\\{b:03o}").as_bytes()),
}
}
want.push(b')');
}
prop_assert_eq!(got, want);
}
}
+78
View File
@@ -0,0 +1,78 @@
//! Stream data read from a file points into the file's bytes instead of being copied: the
//! document's object cache holds parsed dictionaries, not a second copy of every stream.
use std::sync::Arc;
use craft_pdf::cos::{Document, ObjRef, Object, SaveOptions, write_full, write_incremental};
const CONTENT: &[u8] = b"0 0 1 rg 10 10 30 20 re f";
fn fixture() -> Vec<u8> {
let objs = [
"<< /Type /Catalog /Pages 2 0 R >>".to_string(),
"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 100 50] >>".to_string(),
"<< /Type /Page /Parent 2 0 R /Contents 4 0 R >>".to_string(),
format!("<< /Length {} >>\nstream\n{}\nendstream", CONTENT.len(), String::from_utf8_lossy(CONTENT)),
];
let mut out = b"%PDF-1.4\n".to_vec();
let mut offsets = Vec::new();
for (i, o) in objs.iter().enumerate() {
offsets.push(out.len());
out.extend_from_slice(format!("{} 0 obj\n{o}\nendobj\n", i + 1).as_bytes());
}
let xref = out.len();
out.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
for o in offsets {
out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
}
out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes());
out
}
const CONTENTS: ObjRef = ObjRef { num: 4, generation: 0 };
fn stream_bytes(doc: &Document, r: ObjRef) -> (Vec<u8>, *const u8) {
match &*doc.get(r) {
Object::Stream(s) => (s.raw.to_vec(), s.raw.as_ptr()),
other => panic!("{other:?}"),
}
}
#[test]
fn cached_streams_are_the_files_own_bytes() {
let file = Arc::new(fixture());
let doc = Document::open(file.clone()).unwrap();
let (data, at) = stream_bytes(&doc, CONTENTS);
assert_eq!(data, CONTENT);
assert!(file.as_ptr_range().contains(&at), "not a copy");
// A full save reads and caches every object; the cache still holds no copy of stream data.
let full = write_full(&doc, &SaveOptions::default()).unwrap();
assert!(file.as_ptr_range().contains(&stream_bytes(&doc, CONTENTS).1));
// Saves are unchanged: the stream is written as it was read.
let back = Document::open(Arc::new(full)).unwrap();
assert_eq!(stream_bytes(&back, ObjRef::new(4, 0)).0, CONTENT);
}
#[test]
fn edits_within_a_document_keep_pointing_into_its_file() {
let file = Arc::new(fixture());
let mut doc = Document::open(file.clone()).unwrap();
doc.update_dict(CONTENTS, |d| d.set(b"Marked".to_vec(), Object::Bool(true))).unwrap();
assert!(file.as_ptr_range().contains(&stream_bytes(&doc, CONTENTS).1));
let updated = write_incremental(&doc, &SaveOptions::default()).unwrap();
let back = Document::open(Arc::new(updated)).unwrap();
assert_eq!(stream_bytes(&back, CONTENTS).0, CONTENT);
}
#[test]
fn streams_moved_to_another_document_get_their_own_bytes() {
let file = Arc::new(fixture());
let source = Document::open(file.clone()).unwrap();
let moved = (*source.get(CONTENTS)).clone();
let mut target = Document::new_empty();
let r = target.add(moved);
drop(source);
// The target keeps just the stream's bytes, not the whole source file.
assert_eq!(Arc::strong_count(&file), 1);
assert_eq!(stream_bytes(&target, r).0, CONTENT);
}
+622
View File
@@ -0,0 +1,622 @@
//! Full saves with object streams and cross-reference streams (M1.9), read back by our parser
//! and by hayro, and incremental saves stacked on top of them.
use std::sync::Arc;
use crate::testkit::Pdf;
use craft_pdf::cos::{Dict, Document, ObjRef, Object, SaveOptions, Stream, write_full, write_incremental};
/// Four pages, a shared font, document info with a non-ASCII title.
fn fixture() -> Vec<u8> {
let kids: Vec<String> = (0..4).map(|i| format!("{} 0 R", 4 + 2 * i)).collect();
let mut pdf = Pdf::new()
.version("1.4")
.obj("<< /Type /Catalog /Pages 2 0 R >>")
.obj(format!("<< /Type /Pages /Kids [{}] /Count 4 /MediaBox [0 0 200 300] >>", kids.join(" ")))
.obj("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>");
for i in 0..4 {
pdf = pdf
.obj(format!("<< /Type /Page /Parent 2 0 R /Contents {} 0 R /Resources << /Font << /F1 3 0 R >> >> >>", 5 + 2 * i))
.stream("", format!("BT /F1 24 Tf 20 150 Td (Page {}) Tj ET", i + 1));
}
let info = pdf.next_obj();
pdf.obj("<< /Title (Caf\\351 \\(draft\\)) >>").trailer_entry(&format!("/Info {info} 0 R")).build()
}
fn title(doc: &Document) -> Vec<u8> {
let info = doc.resolve(doc.trailer().get(b"Info").unwrap());
info.as_dict().unwrap().get(b"Title").unwrap().as_string().unwrap().bytes.clone()
}
fn hayro_pages(bytes: &[u8]) -> usize {
hayro_syntax::Pdf::new(Arc::new(bytes.to_vec())).expect("hayro opens").pages().len()
}
#[test]
fn full_save_packs_objects_into_object_streams() {
let doc = Document::open(Arc::new(fixture())).unwrap();
let packed = write_full(&doc, &SaveOptions::default()).unwrap();
let text = String::from_utf8_lossy(&packed);
assert!(packed.starts_with(b"%PDF-1.5"), "object streams need PDF 1.5");
assert!(text.contains("/ObjStm") && text.contains("/XRef"));
assert!(!text.contains("\nxref\n"), "no classic table");
assert!(!text.contains("/Type /Catalog") && !text.contains("/Type/Catalog"), "the catalog is compressed");
let back = Document::open(Arc::new(packed.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(title(&back), b"Caf\xe9 (draft)");
assert_eq!(hayro_pages(&packed), 4);
let classic = write_full(&doc, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap();
assert!(classic.starts_with(b"%PDF-1.4"));
assert!(String::from_utf8_lossy(&classic).contains("\nxref\n"));
assert_eq!(hayro_pages(&classic), 4);
}
#[test]
fn incremental_saves_stack_on_an_xref_stream_file() {
let doc = Document::open(Arc::new(fixture())).unwrap();
let packed = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap());
let mut doc = Document::open(packed.clone()).unwrap();
// Edit an object that lives in an object stream.
let info = doc.trailer().reference(b"Info").unwrap();
doc.update_dict(info, |d: &mut Dict| {
d.set(b"Title".to_vec(), Object::String(craft_pdf::cos::PdfString { bytes: b"Second".to_vec(), hex: false }))
})
.unwrap();
let added = doc.add(Object::Int(42));
let updated = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert!(updated.starts_with(&packed), "an incremental save only appends");
let tail = String::from_utf8_lossy(&updated[packed.len()..]);
assert!(tail.contains("/XRef") && tail.contains("/Prev"), "the update uses an xref stream too");
let back = Document::open(Arc::new(updated.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(title(&back), b"Second");
assert_eq!(*back.get(ObjRef::new(added.num, 0)), Object::Int(42));
assert_eq!(back.revisions().len(), 2);
assert_eq!(hayro_pages(&updated), 4);
}
/// The revision is joined to the original once, at its exact size. A revision of more than a few
/// kilobytes used to outgrow the buffer: the whole original was copied a second time, and the
/// result kept up to as much again unused for as long as it was the document's working bytes.
#[test]
fn incremental_saves_are_allocated_at_their_exact_size() {
let table = Arc::new(fixture());
let stream = Arc::new(write_full(&Document::open(table.clone()).unwrap(), &SaveOptions::default()).unwrap());
let no_newline = Arc::new(fixture().trim_ascii_end().to_vec());
for original in [table, stream, no_newline] {
let mut doc = Document::open(original.clone()).unwrap();
let data: Vec<u8> = (0..100_000u32).map(|i| (i % 251) as u8).collect();
let added = doc.add(Object::Stream(Stream { dict: Dict::new(), raw: data.clone().into() }));
let updated = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert_eq!(updated.capacity(), updated.len());
assert!(updated.starts_with(&original), "an incremental save only appends");
let back = Document::open(Arc::new(updated.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
match &*back.get(added) {
Object::Stream(s) => assert_eq!(*s.raw, data),
other => panic!("{other:?}"),
}
assert_eq!(hayro_pages(&updated), 4);
}
}
/// A full save grows its buffer as it writes, so it used to end with up to as much again unused,
/// kept alive with the document's working bytes. It now ends at its exact size.
#[test]
fn full_saves_are_allocated_at_their_exact_size() {
let mut doc = Document::open(Arc::new(fixture())).unwrap();
let data: Vec<u8> = (0..100_000u32).map(|i| (i % 251) as u8).collect();
let added = doc.add(Object::Stream(Stream { dict: Dict::new(), raw: Arc::new(data.clone()).into() }));
let root = doc.root().unwrap();
doc.update_dict(root, |d| d.set(b"Extra".to_vec(), Object::Ref(added))).unwrap();
for object_streams in [true, false] {
let full = write_full(&doc, &SaveOptions { object_streams, ..SaveOptions::default() }).unwrap();
assert_eq!(full.capacity(), full.len(), "object streams: {object_streams}");
let back = Document::open(Arc::new(full.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
let extra = back.dict(&Object::Ref(back.root().unwrap())).and_then(|d| d.get(b"Extra").cloned()).unwrap();
match &*back.resolve(&extra) {
Object::Stream(s) => assert_eq!(*s.raw, data),
other => panic!("{other:?}"),
}
assert_eq!(hayro_pages(&full), 4);
}
}
/// A one-page classic-table file with no `/ID`, the case that used to append `xref` and then
/// `trailer` with no subsection (#1339).
fn classic_table() -> Vec<u8> {
let objects = [
"<< /Type /Catalog /Pages 2 0 R >>",
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Resources << >> /Contents 4 0 R >>",
"<< /Length 0 >>\nstream\n\nendstream",
];
let mut bytes = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (i, object) in objects.iter().enumerate() {
offsets.push(bytes.len());
bytes.extend_from_slice(format!("{} 0 obj\n{object}\nendobj\n", i + 1).as_bytes());
}
let xref = bytes.len();
bytes.extend_from_slice(b"xref\n0 5\n0000000000 65535 f \n");
for offset in offsets {
bytes.extend_from_slice(format!("{offset:010} 00000 n \n").as_bytes());
}
bytes.extend_from_slice(format!("trailer\n<< /Size 5 /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n").as_bytes());
bytes
}
fn qpdf_check(qpdf: &crate::oracle::Oracle, bytes: &[u8]) -> std::process::Output {
static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
let n = NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let path = std::env::temp_dir().join(format!("pdfcraft-xref-{}-{n}.pdf", std::process::id()));
std::fs::write(&path, bytes).unwrap();
let check = qpdf.command().arg("--check").arg(&path).output().unwrap();
let _ = std::fs::remove_file(&path);
check
}
fn page(doc: &Document) -> Dict {
let root = doc.resolve(doc.trailer().get(b"Root").unwrap());
let pages = doc.resolve(root.as_dict().unwrap().get(b"Pages").unwrap());
let kids = pages.as_dict().unwrap().get(b"Kids").unwrap().as_array().unwrap();
doc.resolve(kids.first().unwrap()).as_dict().cloned().unwrap()
}
fn assert_qpdf_clean(bytes: &[u8]) {
let qpdf = crate::oracle::or_skip!(crate::oracle::qpdf());
let check = qpdf_check(&qpdf, bytes);
let report = format!("{}{}", String::from_utf8_lossy(&check.stdout), String::from_utf8_lossy(&check.stderr));
assert!(check.status.success() && !report.contains("WARNING"), "{report}");
}
#[test]
fn unchanged_incremental_save_of_a_classic_table_is_a_valid_xref() {
let input = classic_table();
let doc = Document::open(Arc::new(input.clone())).unwrap();
assert!(!doc.is_modified());
assert!(doc.repair_log().is_empty());
assert_eq!(doc.revisions().len(), 1);
assert!(!doc.revisions()[0].is_stream);
let unchanged = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert!(unchanged.starts_with(&input));
let appended = &unchanged[input.len()..];
assert!(appended.windows(8).any(|w| w == b"xref\n0 0"), "a section with no entries still needs a subsection");
assert!(!appended.windows(13).any(|w| w == b"xref\ntrailer"));
let back = Document::open(Arc::new(unchanged.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(back.revisions().len(), 2);
assert_eq!(page(&back).name(b"Type"), Some(b"Page".as_slice()));
assert!(back.trailer().get(b"ID").is_some(), "a file without an identifier still receives one");
assert_eq!(hayro_pages(&unchanged), 1);
assert_qpdf_clean(&unchanged);
let mut edited = doc.clone();
edited.update_dict(ObjRef::new(3, 0), |d| d.set(b"Rotate".to_vec(), Object::Int(90))).unwrap();
let edited = write_incremental(&edited, &SaveOptions::default()).unwrap();
let edited_back = Document::open(Arc::new(edited.clone())).unwrap();
assert!(edited_back.repair_log().is_empty(), "{:?}", edited_back.repair_log());
assert_eq!(edited_back.get(ObjRef::new(3, 0)).as_dict().unwrap().int(b"Rotate"), Some(90));
assert_qpdf_clean(&edited);
let stamped = write_incremental(&doc, &SaveOptions { mod_date: Some("D:20260101000000Z".into()), ..SaveOptions::default() }).unwrap();
let stamped_back = Document::open(Arc::new(stamped.clone())).unwrap();
assert!(stamped_back.repair_log().is_empty(), "{:?}", stamped_back.repair_log());
let info = stamped_back.resolve(stamped_back.trailer().get(b"Info").unwrap());
assert_eq!(
info.as_dict().unwrap().get(b"ModDate").and_then(|d| d.as_string()).map(|s| s.bytes.as_slice()),
Some(b"D:20260101000000Z".as_slice())
);
assert_qpdf_clean(&stamped);
}
#[test]
fn incremental_save_keeps_a_trailer_only_edit() {
let input = classic_table();
let mut doc = Document::open(Arc::new(input)).unwrap();
assert!(!doc.is_modified());
doc.trailer_mut().set(b"Custom".to_vec(), Object::String(craft_pdf::cos::PdfString::literal(b"keep me".to_vec())));
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
let back = Document::open(Arc::new(saved.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(back.trailer().get(b"Custom").and_then(|o| o.as_string()).map(|s| s.bytes.as_slice()), Some(b"keep me".as_slice()));
assert_eq!(hayro_pages(&saved), 1);
assert_qpdf_clean(&saved);
}
#[test]
fn incremental_save_ignores_a_negative_trailer_size() {
let mut doc = Document::open(Arc::new(classic_table())).unwrap();
doc.trailer_mut().set(b"Size".to_vec(), Object::Int(-1));
let next = doc.add(Object::Int(5)).num + 1;
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
let back = Document::open(Arc::new(saved)).unwrap();
assert_eq!(back.trailer().int(b"Size"), Some(i64::from(next)));
}
/// What one edit added and a later one unlinked (a deleted comment, a cleared redaction mark) is
/// not written: the file never held it, and writing it would keep what was removed. Additions the
/// update still refers to, from the trailer or a changed object and through other additions, are.
#[test]
fn incremental_save_leaves_out_additions_nothing_refers_to() {
let mut doc = Document::open(Arc::new(classic_table())).unwrap();
let text = |s: &str| Object::String(craft_pdf::cos::PdfString::literal(s.as_bytes().to_vec()));
let dict = |entries: &[(&str, Object)]| {
let mut d = Dict::new();
entries.iter().for_each(|(k, v)| d.set(k.as_bytes().to_vec(), v.clone()));
Object::Dict(d)
};
let page_obj = ObjRef::new(3, 0);
let leaf = doc.add(text("kept leaf"));
let kept = doc.add(dict(&[("Leaf", Object::Ref(leaf))]));
doc.update_dict(page_obj, |d| d.set(b"PieceInfo".to_vec(), Object::Ref(kept))).unwrap();
let by_trailer = doc.add(text("kept by the trailer"));
doc.trailer_mut().set(b"Custom".to_vec(), Object::Ref(by_trailer));
// Linked from the page and unlinked again, as clearing a mark does.
let dropped_leaf = doc.add(text("dropped leaf"));
let dropped = doc.add(dict(&[("Leaf", Object::Ref(dropped_leaf))]));
doc.update_dict(page_obj, |d| d.set(b"Annots".to_vec(), Object::Array(vec![Object::Ref(dropped)]))).unwrap();
doc.update_dict(page_obj, |d| {
d.remove(b"Annots");
})
.unwrap();
// Unlinked additions that refer to each other.
let first = doc.add(Object::Null);
let second = doc.add(dict(&[("Next", Object::Ref(first)), ("Note", text("dropped cycle"))]));
doc.set(first, dict(&[("Next", Object::Ref(second))]));
// Off by default: a writer may add what only it finds again, by scanning.
let kept_all = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert!(kept_all.windows(12).any(|w| w == b"dropped leaf"));
let saved = write_incremental(&doc, &SaveOptions { drop_unlinked: true, ..SaveOptions::default() }).unwrap();
let holds = |s: &str| saved.windows(s.len()).any(|w| w == s.as_bytes());
assert!(holds("kept leaf") && holds("kept by the trailer"));
assert!(!holds("dropped leaf") && !holds("dropped cycle"), "unlinked additions are written");
let back = Document::open(Arc::new(saved.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(back.get(leaf).as_string().map(|s| s.bytes.as_slice()), Some(b"kept leaf".as_slice()));
for gone in [dropped, dropped_leaf, first, second] {
assert_eq!(*back.get(gone), Object::Null);
}
assert!(!page(&back).contains(b"Annots"));
assert_qpdf_clean(&saved);
// Freed beforehand, they are written the same way without the option, and what is added
// after that, nothing referring to it either (a document timestamp), is kept.
let mut freed = doc.clone();
freed.free_unlinked_additions();
assert_eq!(write_incremental(&freed, &SaveOptions::default()).unwrap(), saved, "the same objects, freed the same way");
freed.add(text("added after"));
let stamped = write_incremental(&freed, &SaveOptions::default()).unwrap();
assert!(stamped.windows(11).any(|w| w == b"added after") && !stamped.windows(12).any(|w| w == b"dropped leaf"));
}
#[test]
fn qpdf_accepts_object_stream_output() {
let qpdf = crate::oracle::or_skip!(crate::oracle::qpdf());
let doc = Document::open(Arc::new(fixture())).unwrap();
let packed = write_full(&doc, &SaveOptions::default()).unwrap();
let path = std::env::temp_dir().join(format!("pdfcraft-objstm-{}.pdf", std::process::id()));
std::fs::write(&path, &packed).unwrap();
let check = qpdf.command().arg("--check").arg(&path).output().unwrap();
let _ = std::fs::remove_file(&path);
let report = String::from_utf8_lossy(&check.stdout);
assert!(check.status.success() && report.contains("No syntax or stream encoding errors"), "{report}");
}
#[test]
fn the_catalog_of_an_encrypted_document_stays_outside_object_streams() {
// Acrobat reports an encrypted file as damaged when its catalog is in an object stream,
// before asking for the password (#774). Unencrypted output still packs the catalog.
fn root_is_packed(bytes: &[u8], password: Option<&str>) -> bool {
let doc = Document::open_with_password(Arc::new(bytes.to_vec()), password).unwrap();
let root = doc.trailer().reference(b"Root").unwrap();
let direct = format!("\n{} 0 obj", root.num);
!bytes.windows(direct.len()).any(|w| w == direct.as_bytes())
}
let plain = Document::open(Arc::new(fixture())).unwrap();
assert!(root_is_packed(&write_full(&plain, &SaveOptions::default()).unwrap(), None), "unencrypted: catalog in an object stream");
let mut enc = plain.clone();
enc.set_encryption(&craft_pdf::cos::NewEncryption {
algorithm: craft_pdf::cos::Algorithm::Aes256,
user_password: "user",
owner_password: "owner",
permissions: -4,
encrypt_metadata: true,
seed: [7; 32],
})
.unwrap();
let packed = write_full(&enc, &SaveOptions::default()).unwrap();
assert!(!root_is_packed(&packed, Some("user")), "encrypted: catalog must be a standalone object");
let back = Document::open_with_password(Arc::new(packed.clone()), Some("user")).expect("reopens");
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(title(&back), b"Caf\xe9 (draft)");
}
#[test]
fn objects_referenced_by_the_encrypt_dictionary_stay_outside_object_streams() {
// Encrypt a document, then move its crypt filters into indirect objects, as some producers
// do (pdf.js issue7665). A full save must keep them readable before decryption.
let doc = Document::open(Arc::new(fixture())).unwrap();
let mut enc = doc.clone();
enc.set_encryption(&craft_pdf::cos::NewEncryption {
algorithm: craft_pdf::cos::Algorithm::Aes256,
user_password: "",
owner_password: "owner",
permissions: -4,
encrypt_metadata: true,
seed: [7; 32],
})
.unwrap();
let bytes = Arc::new(write_full(&enc, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap());
let mut doc = Document::open(bytes).unwrap();
let enc_ref = doc.trailer().reference(b"Encrypt").expect("indirect /Encrypt");
let cf = doc.get(enc_ref).as_dict().unwrap().get(b"CF").cloned().unwrap();
let cf_ref = doc.add(cf);
doc.update_dict(enc_ref, |d| d.set(b"CF".to_vec(), Object::Ref(cf_ref))).unwrap();
// Visit this alias from Root before the encryption walk reaches /CF's other alias.
doc.update_dict(doc.root().unwrap(), |d| d.set(b"CFProbe".to_vec(), Object::Ref(cf_ref))).unwrap();
let classic = write_full(&doc, &SaveOptions { object_streams: false, ..SaveOptions::default() }).unwrap();
let saved = Document::open(Arc::new(classic.clone())).unwrap();
let cf_num = saved.get(saved.root().unwrap()).as_dict().unwrap().reference(b"CFProbe").unwrap().num;
for damaged in [false, true] {
let mut bytes = classic.clone();
if damaged {
// Equal-length edits preserve the xref offsets and its generation-zero row.
for (from, to) in [(format!("\n{cf_num} 0 obj"), format!("\n{cf_num} 1 obj")), (format!("/CF {cf_num} 0 R"), format!("/CF {cf_num} 1 R"))]
{
let start = bytes.windows(from.len()).position(|w| w == from.as_bytes()).unwrap();
bytes[start..start + from.len()].copy_from_slice(to.as_bytes());
}
}
let doc = Document::open(Arc::new(bytes)).unwrap();
let packed = write_full(&doc, &SaveOptions::default()).unwrap();
let back = Document::open(Arc::new(packed.clone())).expect("reopens");
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(title(&back), b"Caf\xe9 (draft)");
assert_eq!(hayro_pages(&packed), 4);
let probe = back.get(back.root().unwrap()).as_dict().unwrap().reference(b"CFProbe").unwrap();
let encrypt = back.get(back.trailer().reference(b"Encrypt").unwrap());
assert_eq!(encrypt.as_dict().unwrap().reference(b"CF"), Some(probe));
assert!(
matches!(back.xref_entry(probe.num), Some(craft_pdf::cos::XrefEntry::InFile { .. })),
"crypt filters must stay outside object streams"
);
}
}
#[test]
fn page_tree_loops_in_object_streams_do_not_crash_readers() {
// pdf.js Pages-tree-refs.pdf: /Kids 3 → 4 → 5 → 3. Written faithfully into object streams,
// it used to overflow hayro-syntax's stack (vendored patch: cycle guard in `resolve_pages`).
let objs = [
"<< /Type /Catalog /Pages 2 0 R >>",
"<< /Type /Pages /Kids [6 0 R 3 0 R] /Count 2 /MediaBox [0 0 595 842] >>",
"<< /Type /Pages /Kids [4 0 R] /Count 1 >>",
"<< /Type /Pages /Kids [5 0 R] /Count 1 >>",
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
"<< /Type /Page /Parent 2 0 R >>",
];
let mut out = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (i, o) in objs.iter().enumerate() {
offsets.push(out.len());
out.extend_from_slice(format!("{} 0 obj\n{o}\nendobj\n", i + 1).as_bytes());
}
let xref = out.len();
out.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
for o in offsets {
out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
}
out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes());
let doc = Document::open(Arc::new(out)).unwrap();
let packed = write_full(&doc, &SaveOptions::default()).unwrap();
assert!(String::from_utf8_lossy(&packed).contains("/ObjStm"));
assert_eq!(hayro_pages(&packed), 1);
}
/// Catalog, empty page tree, and trailer entries a reader must not invent: a direct name, a
/// direct string, and a string object reachable only from the trailer.
fn trailer_fixture() -> Vec<u8> {
let mut out = b"%PDF-1.4\n".to_vec();
let mut offsets = Vec::new();
for (i, body) in ["<< /Type /Catalog /Pages 2 0 R >>", "<< /Type /Pages /Kids [] /Count 0 >>", "(only from the trailer)"].iter().enumerate() {
offsets.push(out.len());
out.extend_from_slice(format!("{} 0 obj\n{body}\nendobj\n", i + 1).as_bytes());
}
let xref = out.len();
out.extend_from_slice(b"xref\n0 4\n0000000000 65535 f \n");
for offset in offsets {
out.extend_from_slice(format!("{offset:010} 00000 n \n").as_bytes());
}
out.extend_from_slice(
format!("trailer\n<< /Size 4 /Root 1 0 R /Version /1.7 /Custom (keep me) /Note 3 0 R >>\nstartxref\n{xref}\n%%EOF\n").as_bytes(),
);
out
}
fn trailer_custom(doc: &Document) -> Option<Vec<u8>> {
doc.trailer().get(b"Custom").and_then(Object::as_string).map(|s| s.bytes.clone())
}
fn trailer_note(doc: &Document) -> Option<Vec<u8>> {
doc.trailer().get(b"Note").map(|o| doc.resolve(o)).and_then(|o| o.as_string().map(|s| s.bytes.clone()))
}
#[test]
fn full_save_preserves_unknown_trailer_data() {
let input = trailer_fixture();
let doc = Document::open(Arc::new(input)).unwrap();
assert_eq!(trailer_custom(&doc), Some(b"keep me".to_vec()));
assert_eq!(trailer_note(&doc).as_deref(), Some(b"only from the trailer".as_slice()));
assert_eq!(doc.trailer().get(b"Version").and_then(Object::as_name), Some(&b"1.7"[..]));
assert!(doc.repair_log().is_empty());
assert!(!doc.full_save_required());
let inc = Document::open(Arc::new(write_incremental(&doc, &SaveOptions::default()).unwrap())).unwrap();
assert_eq!(trailer_custom(&inc), trailer_custom(&doc));
assert_eq!(trailer_note(&inc), trailer_note(&doc));
assert_eq!(inc.trailer().get(b"Version"), doc.trailer().get(b"Version"));
assert!(inc.repair_log().is_empty());
for object_streams in [true, false] {
let opts = SaveOptions { object_streams, ..SaveOptions::default() };
let bytes = write_full(&doc, &opts).unwrap();
let back = Document::open(Arc::new(bytes)).unwrap();
assert!(back.root().is_some());
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(trailer_custom(&back), trailer_custom(&doc));
assert_eq!(trailer_note(&back), trailer_note(&doc));
assert_eq!(back.trailer().get(b"Version"), doc.trailer().get(b"Version"));
}
let mut required = doc.clone();
required.require_full_save();
let back = Document::open(Arc::new(write_incremental(&required, &SaveOptions::default()).unwrap())).unwrap();
assert!(back.root().is_some());
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(trailer_custom(&back), trailer_custom(&doc));
assert_eq!(trailer_note(&back), trailer_note(&doc));
assert_eq!(back.trailer().get(b"Version"), doc.trailer().get(b"Version"));
}
/// The `startxref` value at the end of `bytes`.
fn last_startxref(bytes: &[u8]) -> usize {
let at = bytes.windows(9).rposition(|w| w == b"startxref").unwrap();
String::from_utf8_lossy(&bytes[at + 9..]).split_whitespace().next().unwrap().parse().unwrap()
}
#[test]
fn incremental_saves_of_a_file_with_bytes_before_its_header_count_from_the_header() {
// Offsets written before the prefix was added count from `%PDF-`, as in the KillerPDF file
// of #2364; the appended section must count the same way as the `/Prev` it points back to.
let prefix = vec![b'#'; 128];
let as_stream = write_full(&Document::open(Arc::new(fixture())).unwrap(), &SaveOptions::default()).unwrap();
for (name, body, marker) in [("table", classic_table(), &b"xref"[..]), ("stream", as_stream, &b" 0 obj"[..])] {
let input = [prefix.as_slice(), &body].concat();
let mut doc = Document::open(Arc::new(input.clone())).unwrap();
assert!(doc.revisions().last().unwrap().header_relative, "{name}");
doc.update_dict(ObjRef::new(1, 0), |d| d.set(b"Lang".to_vec(), Object::String(craft_pdf::cos::PdfString::literal(b"en".to_vec())))).unwrap();
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert!(saved.starts_with(&input), "{name}");
let xref = last_startxref(&saved) + prefix.len();
let section = saved.get(xref..).unwrap_or_default();
assert!(section.windows(marker.len()).position(|w| w == marker).is_some_and(|p| p < 16), "{name}: startxref misses its section");
let back = Document::open(Arc::new(saved.clone())).unwrap();
assert_eq!(back.revisions().len(), doc.revisions().len() + 1, "{name}");
assert!(back.revisions().iter().all(|r| r.header_relative), "{name}: one convention across the chain");
let edited = back.get(ObjRef::new(1, 0));
assert_eq!(edited.as_dict().unwrap().get(b"Lang").and_then(|o| o.as_string()).map(|s| s.bytes.as_slice()), Some(b"en".as_slice()));
assert!(back.xref_entry_is_exact(1), "{name}");
assert_qpdf_clean(&saved);
}
}
#[test]
fn incremental_saves_of_a_file_whose_offsets_count_from_its_first_byte_keep_counting_so() {
// The other convention: the same file, but its offsets include the bytes before the header.
let prefix = vec![b'#'; 128];
let mut input = [prefix.as_slice(), &classic_table()].concat();
let xref = input.windows(6).rposition(|w| w == b"\nxref\n").unwrap() + 1;
let table: String = String::from_utf8_lossy(&input[xref..])
.lines()
.map(|line| match line.strip_suffix(" 00000 n ").and_then(|o| o.parse::<usize>().ok()) {
Some(o) => format!("{:010} 00000 n \n", o + prefix.len()),
None if line.parse::<usize>().is_ok() => format!("{xref}\n"),
None => format!("{line}\n"),
})
.collect();
input.truncate(xref);
input.extend_from_slice(table.as_bytes());
let mut doc = Document::open(Arc::new(input.clone())).unwrap();
assert!(doc.repair_log().is_empty(), "{:?}", doc.repair_log());
assert!(!doc.revisions().last().unwrap().header_relative);
doc.update_dict(ObjRef::new(1, 0), |d| d.set(b"Lang".to_vec(), Object::String(craft_pdf::cos::PdfString::literal(b"en".to_vec())))).unwrap();
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert!(saved.get(last_startxref(&saved)..).unwrap_or_default().starts_with(b"xref"));
let back = Document::open(Arc::new(saved.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_eq!(back.revisions().len(), 2);
assert!(back.xref_entry_is_exact(1));
// No qpdf check here: qpdf counts every offset from the header in a file with bytes before
// it, so it already reconstructs this input before anything is appended.
}
/// One page whose catalog carries a PDF 2.0 document part tree (`/DPartRoot`, ISO 32000-2
/// 14.12) and whose structure tree root lists structure namespaces (`/Namespaces`, 14.7.4),
/// both through indirect objects that nothing else references.
fn pdf2_fixture() -> Vec<u8> {
Pdf::new()
.version("2.0")
.obj("<< /Type /Catalog /Pages 2 0 R /DPartRoot 6 0 R /StructTreeRoot 8 0 R >>")
.obj("<< /Type /Pages /Kids [4 0 R] /Count 1 /MediaBox [0 0 200 300] >>")
.obj("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
.obj("<< /Type /Page /Parent 2 0 R /Contents 5 0 R /Resources << /Font << /F1 3 0 R >> >> >>")
.stream("", "BT /F1 24 Tf 20 150 Td (Part) Tj ET")
.obj("<< /Type /DPartRoot /DPartRootNode 7 0 R /NodeNameList [/Chapter] >>")
.obj("<< /Type /DPart /Parent 6 0 R /Start 4 0 R /End 4 0 R /DPM << /Kind (cover) >> >>")
.obj("<< /Type /StructTreeRoot /K [] /Namespaces [9 0 R] >>")
.obj("<< /Type /Namespace /NS (http://iso.org/pdf2/ssn) /Schema << /Type /Filespec /F (ssn.rnc) >> >>")
.build()
}
/// What `pdf2_fixture` declares, read back through the catalog.
fn assert_dpart_and_namespaces(doc: &Document) {
let catalog = doc.get(doc.root().unwrap());
let catalog = catalog.as_dict().unwrap();
let root = doc.resolve(catalog.get(b"DPartRoot").expect("catalog keeps /DPartRoot"));
let root = root.as_dict().unwrap();
let node = doc.resolve(root.get(b"DPartRootNode").expect("/DPartRootNode kept"));
let node = node.as_dict().unwrap();
assert_eq!(node.name(b"Type"), Some(&b"DPart"[..]));
// Both ends of the part are still the one page, found through /Pages (references are
// renumbered by a full save, so they are compared inside this document).
let pages = doc.resolve(catalog.get(b"Pages").unwrap());
let kids = pages.as_dict().unwrap().get(b"Kids").unwrap().as_array().unwrap();
let page = kids.first().and_then(|k| k.as_ref()).expect("the page is still the first kid");
assert_eq!(node.reference(b"Start"), Some(page), "the part still starts at its page");
assert_eq!(node.reference(b"End"), Some(page), "the part still ends at its page");
assert!(doc.resolve(node.get(b"DPM").unwrap()).as_dict().unwrap().get(b"Kind").is_some());
let tree = doc.resolve(catalog.get(b"StructTreeRoot").unwrap());
let namespaces = doc.resolve(tree.as_dict().unwrap().get(b"Namespaces").expect("/Namespaces kept"));
let namespace = doc.resolve(&namespaces.as_array().unwrap()[0]);
let namespace = namespace.as_dict().unwrap();
assert_eq!(namespace.get(b"NS").unwrap().as_string().unwrap().bytes, b"http://iso.org/pdf2/ssn");
assert!(namespace.get(b"Schema").is_some(), "/Schema is retained");
}
#[test]
fn document_parts_and_structure_namespaces_survive_full_and_incremental_saves() {
let original = Arc::new(pdf2_fixture());
let doc = Document::open(original.clone()).unwrap();
assert_dpart_and_namespaces(&doc);
for object_streams in [true, false] {
let opts = SaveOptions { object_streams, ..SaveOptions::default() };
let full = write_full(&doc, &opts).unwrap();
let back = Document::open(Arc::new(full.clone())).unwrap();
assert!(back.repair_log().is_empty(), "{:?}", back.repair_log());
assert_dpart_and_namespaces(&back);
// A second generation, rewritten from the rewritten file.
let second = Document::open(Arc::new(write_full(&back, &opts).unwrap())).unwrap();
assert!(second.repair_log().is_empty(), "{:?}", second.repair_log());
assert_dpart_and_namespaces(&second);
}
// An incremental save of an unrelated edit leaves the file's bytes, and so both, alone.
let mut edited = Document::open(original.clone()).unwrap();
edited.add(Object::Int(1));
let updated = write_incremental(&edited, &SaveOptions::default()).unwrap();
assert!(updated.starts_with(&original));
let reopened = Document::open(Arc::new(updated)).unwrap();
assert!(reopened.repair_log().is_empty(), "{:?}", reopened.repair_log());
assert_dpart_and_namespaces(&reopened);
}