mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-11 10:46:28 +00:00
fix(render): bound the objects lopdf's fallback parses from object streams and arrays (#5083)
## What this changes lopdf's fallback loader (used by `inspect` when the lazy reader fails, and always by `attachment_data`) parsed every object stream eagerly, capping only decoded bytes. A stream of `0 0 0 …` inside the 256 MiB limit built ~120M `Object`s (10+ GB) and aborted. This bounds what it builds: - **Per top-level object:** at most 4,194,304 objects (`parser::MAX_OBJECT_NODES`, the same as cos's `MAX_NODES`), counting every array element and dictionary value at any depth. One budget is threaded through `array` / `_dictionary` / `_direct_object`. An object is charged once it is built, so failed `alt` branches cost nothing, and an exhausted budget is a nom `Failure`. - **Per object stream:** at most 1,000,000 index entries (cos's `/N` clamp), and the index list is no longer allocated from the whole block. - **Per load:** at most 8,388,608 objects across all object streams, in an atomic `NodeBudget` shared by rayon workers. A member reserves up to its own byte length (every object takes at least one byte) and refunds what it doesn't use, so parallel members can't hold the budget between them. Once the budget is spent, the remaining members are skipped with a warning. - **`convert` in `crates/render/src/structure.rs`:** the depth limit is now 128, one level per array or dictionary, matching cos's parser. Before, it was 100 with two levels per dictionary, so ~51 nested dictionaries alone forced the lopdf fallback. Logged as lopdf patch (7) in `vendor/README.md`. Fixes #5083 ## How you verified it `cargo fmt --check` on `pdfcraft-render` and `cargo clippy -p pdfcraft-render --all-targets -- -D warnings` (which compiles the vendored lopdf). Regression tests in `crates/render/src/inspect/tests.rs`, where lopdf patches are tested in-tree (the vendored lopdf's own tests need its missing `assets/example.pdf`): - `object_stream_members_over_the_node_cap_are_skipped` (croquete1): a member that is one array of 4.3M elements is skipped at the per-object cap; a small member is kept. - `object_stream_index_entries_past_the_clamp_are_ignored`: of 1,000,001 distinct members, the one past the 1,000,000-entry clamp is never built. - `object_streams_of_one_load_share_their_node_budget`, `object_streams_loaded_in_parallel_share_their_node_budget` and `a_malformed_object_stream_member_refunds_its_reservation` (Harrison-Booth): a small shared `NodeBudget` bounds the members of several streams, serially and across threads, and a malformed member refunds what it didn't build. `NodeBudget` and `ObjectStream::new_within` are now public for this. - [x] `cargo fmt --check` - [x] clippy with `-D warnings` on the crates I changed - [x] tests pass on the crates I changed (`cargo test -p pdfcraft-render --lib object_stream`: 9 passed, including the five new tests) ## Never-crash - [x] No new `unwrap`/`expect`/`panic!`/`unreachable!`/`todo!` outside tests - [x] No indexing or slicing with input-derived positions; the budget uses `checked_sub` and a `min`-bounded reservation; index entries and objects built are capped; nesting stays bounded (128 in `convert`) - [x] No `unsafe` - [x] The abort this fixes has synthetic regression tests (above) ## Provenance - [x] No assets, fonts or copied code; the limits follow `pdfcraft_cos`'s own caps ## Security - [x] No change to `--root`, MCP, the control channel or network access ## Parity and reach - Not applicable: an internal robustness fix to the fallback loader, with no user-facing feature or parity entry. The vendored-patch log (`vendor/README.md`, lopdf (7)) is updated. ## Anything a reviewer should look at first - `vendor/README.md` now lists the tests above for patch (7). - Ordinary (not compressed) indirect objects are capped at 4M objects each but aren't charged to the per-load budget; their total is bounded only by file size. - `get_compressed_object` still decodes and re-parses the whole stream per lookup (pre-existing); it now gets one object's budget. - Objects nested 101–128 deep now reach the Inspector through the lazy reader rather than being skipped by the fallback. The Inspector's own walkers cap depth at 32 or less. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Added limits to PDF object parsing and object-stream processing. Loading now parses only selected members, while lookups parse only the requested member. Members that exceed available limits are skipped in a deterministic order. * Increased the supported nesting depth for dictionary and object conversion, allowing more deeply nested content to be processed. * Documented that conversion failures can trigger eager loading as a fallback. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
@@ -249,6 +249,8 @@ fn object_stream_member_extents_bound_owned_strings() {
|
||||
assert!(!parsed.objects.contains_key(&(201, 0)));
|
||||
}
|
||||
|
||||
/// Members are read at their indexed offsets, in any order and with any whitespace around
|
||||
/// them; a number listed twice keeps its last entry.
|
||||
#[test]
|
||||
fn object_stream_member_extents_preserve_index_order_and_whitespace() {
|
||||
let members = b" \n42\n /Name \n[true 7]\n << /Key (value) >>\n (last)";
|
||||
@@ -262,6 +264,170 @@ fn object_stream_member_extents_preserve_index_order_and_whitespace() {
|
||||
assert_eq!(parsed.objects.get(&(12, 0)).unwrap().as_dict().unwrap().get(b"Key").unwrap().as_str().unwrap(), b"value");
|
||||
}
|
||||
|
||||
/// lopdf patch (7), #5083: a decoded object-stream member could build any number of objects
|
||||
/// (`0 0 0 ...` inside the decompression limit built ~120M and aborted). A member now builds at
|
||||
/// most the per-object node cap (1 << 22), so one array with more elements than that is skipped
|
||||
/// while a small member is kept. Without the node charge in the vendored parser the oversize
|
||||
/// array builds in full, so object 11 is present and the last assertion fails.
|
||||
#[test]
|
||||
fn object_stream_members_over_the_node_cap_are_skipped() {
|
||||
let small = b"[ 1 2 3 ]";
|
||||
let mut big = vec![b'['];
|
||||
for _ in 0..4_300_000 {
|
||||
big.extend_from_slice(b"0 ");
|
||||
}
|
||||
big.push(b']');
|
||||
let mut members = small.to_vec();
|
||||
let big_off = members.len();
|
||||
members.extend_from_slice(&big);
|
||||
let stream = indexed_object_stream(&[(10, 0), (11, big_off)], &members);
|
||||
let parsed = lopdf::ObjectStream::new_with_limit(&stream, Some(16 << 20)).unwrap();
|
||||
assert!(parsed.objects.contains_key(&(10, 0)), "the small member is kept");
|
||||
assert!(!parsed.objects.contains_key(&(11, 0)), "the oversize member is skipped");
|
||||
}
|
||||
|
||||
/// lopdf patch (7), #5083: an object stream reads at most 1,000,000 index entries (cos's `/N`
|
||||
/// clamp), so a huge index can't allocate an entry list or parse a member per entry. Of
|
||||
/// 1,000,001 distinct one-byte members, the last is never built.
|
||||
#[test]
|
||||
fn object_stream_index_entries_past_the_clamp_are_ignored() {
|
||||
let count = 1_000_001_u32;
|
||||
let mut members = Vec::with_capacity(2 * count as usize);
|
||||
let pairs: Vec<(u32, usize)> = (1..=count)
|
||||
.map(|id| {
|
||||
let at = members.len();
|
||||
members.extend_from_slice(b"0 ");
|
||||
(id, at)
|
||||
})
|
||||
.collect();
|
||||
let parsed = lopdf::ObjectStream::new_with_limit(&indexed_object_stream(&pairs, &members), None).unwrap();
|
||||
assert_eq!(parsed.objects.len(), 1_000_000);
|
||||
assert!(parsed.objects.contains_key(&(1_000_000, 0)));
|
||||
assert!(!parsed.objects.contains_key(&(count, 0)), "the entry past the clamp is ignored");
|
||||
}
|
||||
|
||||
/// Six object streams, each holding one nested member of seven objects.
|
||||
fn nested_object_streams() -> Vec<lopdf::Stream> {
|
||||
(0..6).map(|i| indexed_object_stream(&[(100 + i, 0)], b"<< /A [1 [2]] /B << /C 3 >> >>")).collect()
|
||||
}
|
||||
|
||||
/// lopdf patch (7), #5083: one load's object streams share a single node budget (the load-wide
|
||||
/// `MAX_LOAD_NODES`, here 12 nodes so the test stays small), so later streams are skipped once
|
||||
/// earlier ones spent it. A stream given a budget of its own would load all six.
|
||||
#[test]
|
||||
fn object_streams_of_one_load_share_their_node_budget() {
|
||||
let streams = nested_object_streams();
|
||||
let budget = lopdf::NodeBudget::new(12);
|
||||
let loaded: usize = streams.iter().map(|s| lopdf::ObjectStream::new_within(s, None, &budget).unwrap().objects.len()).sum();
|
||||
assert!(loaded > 0);
|
||||
assert!(loaded < streams.len(), "the budget is shared by all streams");
|
||||
assert!(budget.spent());
|
||||
}
|
||||
|
||||
/// The shared budget holds across threads too, as rayon's workers load object streams in
|
||||
/// parallel: members reserve their share before parsing, so together they can't overshoot.
|
||||
#[test]
|
||||
fn object_streams_loaded_in_parallel_share_their_node_budget() {
|
||||
let streams = nested_object_streams();
|
||||
let budget = &lopdf::NodeBudget::new(12);
|
||||
let loaded: usize = std::thread::scope(|scope| {
|
||||
let workers: Vec<_> =
|
||||
streams.iter().map(|s| scope.spawn(move || lopdf::ObjectStream::new_within(s, None, budget).unwrap().objects.len())).collect();
|
||||
workers.into_iter().map(|w| w.join().unwrap()).sum()
|
||||
});
|
||||
assert!(loaded > 0);
|
||||
assert!(loaded < streams.len(), "the budget is shared by all streams");
|
||||
}
|
||||
|
||||
/// A member that fails to parse gives back the nodes it reserved and did not build, so a
|
||||
/// malformed member doesn't use up the budget of the members after it.
|
||||
#[test]
|
||||
fn a_malformed_object_stream_member_refunds_its_reservation() {
|
||||
let stream = indexed_object_stream(&[(10, 0), (11, 3)], b"[1 [2]");
|
||||
let budget = lopdf::NodeBudget::new(5);
|
||||
let parsed = lopdf::ObjectStream::new_within(&stream, None, &budget).unwrap();
|
||||
assert!(!parsed.objects.contains_key(&(10, 0)));
|
||||
assert!(parsed.objects.contains_key(&(11, 0)));
|
||||
assert!(!budget.spent(), "unused reservation from the malformed member is refunded");
|
||||
}
|
||||
|
||||
/// A budget too small for every member builds them in index order, so the members a spent
|
||||
/// budget skips are the same on every load. Here the first member takes four of the five
|
||||
/// nodes, the second the last one, and the third is skipped. Parsed in parallel from the shared
|
||||
/// budget, the second and third could find it held by the first and be skipped as well.
|
||||
#[test]
|
||||
fn a_tight_budget_builds_object_stream_members_in_index_order() {
|
||||
let stream = indexed_object_stream(&[(10, 0), (11, 8), (12, 10)], b"[1 2 3] 7 8");
|
||||
let budget = lopdf::NodeBudget::new(5);
|
||||
let parsed = lopdf::ObjectStream::new_within(&stream, None, &budget).unwrap();
|
||||
assert!(parsed.objects.contains_key(&(10, 0)));
|
||||
assert!(parsed.objects.contains_key(&(11, 0)), "the second member is built from what the first left");
|
||||
assert!(!parsed.objects.contains_key(&(12, 0)), "the third member finds the budget spent");
|
||||
assert!(budget.spent());
|
||||
}
|
||||
|
||||
/// A PDF with a cross-reference stream whose Info dictionary (object 4) sits in object stream 3
|
||||
/// after `first`, a member listed before it.
|
||||
fn info_in_object_stream_pdf(first: &[u8]) -> Vec<u8> {
|
||||
let info = b"<< /Title (Kept) >>";
|
||||
let index = format!("5 0 4 {} ", first.len() + 1);
|
||||
let mut data = index.clone().into_bytes();
|
||||
data.extend_from_slice(first);
|
||||
data.push(b' ');
|
||||
data.extend_from_slice(info);
|
||||
let mut pdf = b"%PDF-1.5\n".to_vec();
|
||||
let mut offsets = [0_u32; 7];
|
||||
let mut append = |pdf: &mut Vec<u8>, id: usize, head: &[u8], body: &[u8], tail: &[u8]| {
|
||||
offsets[id] = pdf.len() as u32;
|
||||
pdf.extend_from_slice(format!("{id} 0 obj\n").as_bytes());
|
||||
pdf.extend_from_slice(head);
|
||||
pdf.extend_from_slice(body);
|
||||
pdf.extend_from_slice(tail);
|
||||
pdf.extend_from_slice(b"\nendobj\n");
|
||||
};
|
||||
append(&mut pdf, 1, b"<< /Type /Catalog /Pages 2 0 R >>", b"", b"");
|
||||
append(&mut pdf, 2, b"<< /Type /Pages /Kids [] /Count 0 >>", b"", b"");
|
||||
let head = format!("<< /Type /ObjStm /N 2 /First {} /Length {} >>\nstream\n", index.len(), data.len());
|
||||
append(&mut pdf, 3, head.as_bytes(), &data, b"\nendstream");
|
||||
let xref_offset = pdf.len() as u32;
|
||||
offsets[6] = xref_offset;
|
||||
let mut records = Vec::new();
|
||||
for (id, &offset) in offsets.iter().enumerate() {
|
||||
let (kind, field, index) = match id {
|
||||
0 => (0_u8, 0_u32, 0xffff_u16),
|
||||
4 => (2, 3, 1),
|
||||
5 => (2, 3, 0),
|
||||
_ => (1, offset, 0),
|
||||
};
|
||||
records.push(kind);
|
||||
records.extend_from_slice(&field.to_be_bytes());
|
||||
records.extend_from_slice(&index.to_be_bytes());
|
||||
}
|
||||
let head = format!("<< /Type /XRef /Size 7 /Root 1 0 R /Info 4 0 R /W [1 4 2] /Length {} >>\nstream\n", records.len());
|
||||
pdf.extend_from_slice(format!("6 0 obj\n{head}").as_bytes());
|
||||
pdf.extend_from_slice(&records);
|
||||
pdf.extend_from_slice(format!("\nendstream\nendobj\nstartxref\n{xref_offset}\n%%EOF\n").as_bytes());
|
||||
pdf
|
||||
}
|
||||
|
||||
/// lopdf patch (7), #5083: a lookup in an object stream (lopdf's metadata reader, or an indirect
|
||||
/// `/Length` read while loading) parses only the member it wants. It used to parse every member
|
||||
/// from one object's budget, so a member before it of more than `MAX_OBJECT_NODES` objects spent
|
||||
/// that budget and the wanted member was skipped: the Info dictionary read as missing.
|
||||
#[test]
|
||||
fn a_lookup_in_an_object_stream_parses_only_the_member_it_wants() {
|
||||
let mut big = vec![b'['];
|
||||
for _ in 0..4_300_000 {
|
||||
big.extend_from_slice(b"0 ");
|
||||
}
|
||||
big.push(b']');
|
||||
let metadata = Document::load_metadata_mem(&info_in_object_stream_pdf(&big)).unwrap();
|
||||
assert_eq!(metadata.title.as_deref(), Some("Kept"));
|
||||
// The fixture itself reads: with a small member before it, the Info dictionary was read before too.
|
||||
let metadata = Document::load_metadata_mem(&info_in_object_stream_pdf(b"[1 2 3]")).unwrap();
|
||||
assert_eq!(metadata.title.as_deref(), Some("Kept"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn view_and_user_space_round_trip_for_every_rotation() {
|
||||
for rotation in [0u16, 90, 180, 270] {
|
||||
|
||||
@@ -166,6 +166,8 @@ impl Structure for LazyStructure {
|
||||
fn encrypted(&self) -> bool {
|
||||
self.source.security().is_some() || self.trailer.has(b"Encrypt")
|
||||
}
|
||||
/// The object `id`, read from the lazy source and converted once, then kept; a failed
|
||||
/// conversion marks the structure so the caller can fall back to the eager loader.
|
||||
fn get_object(&self, id: ObjectId) -> Result<&Object> {
|
||||
let missing = || Error::ObjectNotFound(id);
|
||||
let slot = self.objects.binary_search_by_key(&id.0, |(n, _)| *n).ok().and_then(|i| self.objects.get(i)).ok_or_else(missing)?;
|
||||
@@ -185,8 +187,15 @@ impl Structure for LazyStructure {
|
||||
}
|
||||
}
|
||||
|
||||
/// Nesting `convert` accepts: the same as `pdfcraft_cos`'s parser (each array or dictionary
|
||||
/// one level), so every object cos parsed converts. A lower limit would turn deep but valid
|
||||
/// nesting into a lazy-reader failure and force the eager lopdf fallback.
|
||||
const CONVERT_DEPTH: usize = 128;
|
||||
|
||||
/// Convert a cos dictionary to lopdf's, without `/PieceInfo`. `depth` is the nesting level of
|
||||
/// the dictionary itself; its values sit one level deeper.
|
||||
fn convert_dict(dict: &pdfcraft_cos::Dict, depth: usize) -> Result<Dictionary> {
|
||||
if depth > 100 {
|
||||
if depth > CONVERT_DEPTH {
|
||||
return Err(Error::RecursionLimit);
|
||||
}
|
||||
dict.iter()
|
||||
@@ -203,8 +212,10 @@ fn convert_dict(dict: &pdfcraft_cos::Dict, depth: usize) -> Result<Dictionary> {
|
||||
/// untouched; this is the read-only view inspection walks.
|
||||
const PRIVATE_DATA: &[u8] = b"PieceInfo";
|
||||
|
||||
/// Convert a cos object to lopdf's, failing past [`CONVERT_DEPTH`] levels of arrays and
|
||||
/// dictionaries.
|
||||
fn convert(object: &CosObject, depth: usize) -> Result<Object> {
|
||||
if depth > 100 {
|
||||
if depth > CONVERT_DEPTH {
|
||||
return Err(Error::RecursionLimit);
|
||||
}
|
||||
Ok(match object {
|
||||
@@ -218,8 +229,8 @@ fn convert(object: &CosObject, depth: usize) -> Result<Object> {
|
||||
}
|
||||
CosObject::Ref(value) => Object::Reference((value.num, value.generation)),
|
||||
CosObject::Array(value) => Object::Array(value.iter().map(|o| convert(o, depth + 1)).collect::<Result<_>>()?),
|
||||
CosObject::Dict(value) => Object::Dictionary(convert_dict(value, depth + 1)?),
|
||||
CosObject::Stream(value) => Object::Stream(lopdf::Stream::new(convert_dict(&value.dict, depth + 1)?, value.raw.to_vec())),
|
||||
CosObject::Dict(value) => Object::Dictionary(convert_dict(value, depth)?),
|
||||
CosObject::Stream(value) => Object::Stream(lopdf::Stream::new(convert_dict(&value.dict, depth)?, value.raw.to_vec())),
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+1
-1
@@ -47,7 +47,7 @@ pub use encryption::{EncryptionState, EncryptionVersion, Permissions};
|
||||
pub use error::{DecompressError, Error, ParseError, Result};
|
||||
pub use incremental_document::IncrementalDocument;
|
||||
pub use load_options::{FilterFunc, LoadOptions};
|
||||
pub use object_stream::{ObjectStream, ObjectStreamBuilder, ObjectStreamConfig};
|
||||
pub use object_stream::{NodeBudget, ObjectStream, ObjectStreamBuilder, ObjectStreamConfig};
|
||||
pub use outlines::Outline;
|
||||
pub use reader::{PdfMetadata, Reader};
|
||||
pub use save_options::{SaveOptions, SaveOptionsBuilder};
|
||||
|
||||
Vendored
+163
-22
@@ -1,13 +1,78 @@
|
||||
use crate::parser;
|
||||
use crate::{Document, Error, Object, ObjectId, Result, Stream};
|
||||
use std::cell::Cell;
|
||||
use std::collections::BTreeMap;
|
||||
use std::num::TryFromIntError;
|
||||
use std::str::FromStr;
|
||||
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
|
||||
|
||||
use log::warn;
|
||||
#[cfg(feature = "rayon")]
|
||||
use rayon::prelude::*;
|
||||
|
||||
/// PdfCraft patch: index entries read from one object stream (the same clamp `pdfcraft_cos`
|
||||
/// applies to `/N`). Entries past it are ignored, so a huge index block cannot allocate an entry
|
||||
/// list or parse a member per entry.
|
||||
pub(crate) const MAX_OBJECT_STREAM_MEMBERS: usize = 1_000_000;
|
||||
|
||||
/// PdfCraft patch: objects (every member and everything nested in it) one load may build from
|
||||
/// all of its object streams together, about one per object of the largest cross-reference
|
||||
/// table `pdfcraft_cos` accepts (8,388,607). Each stream is bounded by the decompression limit,
|
||||
/// but a file may hold many, and every two decoded bytes can be an object of a hundred bytes or
|
||||
/// more; this keeps the worst case near a gigabyte instead of tens.
|
||||
pub(crate) const MAX_LOAD_NODES: usize = 1 << 23;
|
||||
|
||||
/// PdfCraft patch: the objects parsing may still build, shared by every object stream one load
|
||||
/// parses (and by rayon workers, hence atomic). Each member reserves its share before parsing
|
||||
/// and refunds what it did not use, so concurrent members cannot overshoot the total. Public so
|
||||
/// PdfCraft's tests can load object streams from a small budget.
|
||||
///
|
||||
/// The loaders parse their object streams one after another in object-number order, and a
|
||||
/// stream whose members could run the budget out parses them in index order (see
|
||||
/// [`ObjectStream::new_within`]), so the members skipped once it is spent are the same on every
|
||||
/// load of a file.
|
||||
pub struct NodeBudget(AtomicUsize);
|
||||
|
||||
impl NodeBudget {
|
||||
/// A budget of `nodes` objects.
|
||||
pub fn new(nodes: usize) -> Self {
|
||||
Self(AtomicUsize::new(nodes))
|
||||
}
|
||||
|
||||
/// Take up to `want` nodes and return how many were granted (0 once the budget is spent).
|
||||
fn reserve(&self, want: usize) -> usize {
|
||||
// A compare-exchange loop rather than `fetch_update`, which newer toolchains deprecate
|
||||
// in favour of a `try_update` older than this crate's minimum Rust version.
|
||||
let mut left = self.0.load(Ordering::Relaxed);
|
||||
loop {
|
||||
let take = left.min(want);
|
||||
match self
|
||||
.0
|
||||
.compare_exchange_weak(left, left - take, Ordering::Relaxed, Ordering::Relaxed)
|
||||
{
|
||||
Ok(_) => return take,
|
||||
Err(now) => left = now,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Give back nodes [`Self::reserve`] granted and parsing did not use. Refunds never exceed
|
||||
/// what was taken, so the count stays within the starting budget and cannot overflow.
|
||||
fn refund(&self, nodes: usize) {
|
||||
self.0.fetch_add(nodes, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// The nodes not yet reserved.
|
||||
fn left(&self) -> usize {
|
||||
self.0.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// Whether nothing is left to build from.
|
||||
pub fn spent(&self) -> bool {
|
||||
self.left() == 0
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct ObjectStream {
|
||||
pub objects: BTreeMap<ObjectId, Object>,
|
||||
@@ -50,7 +115,33 @@ impl ObjectStream {
|
||||
/// Parse an existing object stream without modifying it, rejecting the
|
||||
/// decoded content if it would exceed `max_decompressed_size` bytes. `None`
|
||||
/// means no limit (the behavior of [`ObjectStream::new`]).
|
||||
///
|
||||
/// PdfCraft patch: members are built from a budget of `MAX_LOAD_NODES` objects of their own.
|
||||
pub fn new_with_limit(stream: &Stream, max_decompressed_size: Option<usize>) -> Result<ObjectStream> {
|
||||
Self::new_within(stream, max_decompressed_size, &NodeBudget::new(MAX_LOAD_NODES))
|
||||
}
|
||||
|
||||
/// PdfCraft patch: [`ObjectStream::new_with_limit`], building members from `budget`, which a
|
||||
/// load shares between all of its object streams. A member that would exceed its share, or
|
||||
/// any member once the budget is spent, is skipped with a warning.
|
||||
///
|
||||
/// When the budget holds enough for every member, they are parsed in parallel from a share
|
||||
/// reserved for this stream alone, which no member can run out; otherwise they are parsed in
|
||||
/// index order from `budget`. Either way which members are built doesn't depend on the order
|
||||
/// in which rayon's workers reach them.
|
||||
pub fn new_within(
|
||||
stream: &Stream, max_decompressed_size: Option<usize>, budget: &NodeBudget,
|
||||
) -> Result<ObjectStream> {
|
||||
Self::new_members_within(stream, max_decompressed_size, budget, |_| true)
|
||||
}
|
||||
|
||||
/// PdfCraft patch: [`ObjectStream::new_within`], parsing only the members whose object
|
||||
/// number `keep` accepts. The others are never parsed, so they spend nothing from `budget`:
|
||||
/// a loader passes the members it would add (not placed in another stream by the
|
||||
/// cross-reference table, and not already loaded), and a lookup the one member it wants.
|
||||
pub(crate) fn new_members_within(
|
||||
stream: &Stream, max_decompressed_size: Option<usize>, budget: &NodeBudget, keep: impl Fn(u32) -> bool,
|
||||
) -> Result<ObjectStream> {
|
||||
let content = match max_decompressed_size {
|
||||
// Object streams are decoded while the document is loaded, so
|
||||
// enforcing the limit here bounds the memory a single stream can use.
|
||||
@@ -75,8 +166,10 @@ impl ObjectStream {
|
||||
let index_block = content.get(..first_offset).ok_or(Error::InvalidOffset(first_offset))?;
|
||||
|
||||
let numbers_str = std::str::from_utf8(index_block).map_err(|e| Error::InvalidObjectStream(e.to_string()))?;
|
||||
// PdfCraft patch: read at most `MAX_OBJECT_STREAM_MEMBERS` index entries.
|
||||
let numbers: Vec<_> = numbers_str
|
||||
.split_whitespace()
|
||||
.take(2 * MAX_OBJECT_STREAM_MEMBERS)
|
||||
.map(|number| u32::from_str(number).ok())
|
||||
.collect();
|
||||
let len = numbers.len() / 2 * 2; // Ensure only pairs.
|
||||
@@ -85,6 +178,9 @@ impl ObjectStream {
|
||||
if numbers.len().try_into().ok() != n.checked_mul(2) {
|
||||
warn!("object stream: the object stream dictionary specifies a wrong number of objects")
|
||||
}
|
||||
if usize::try_from(n).is_ok_and(|n| n > MAX_OBJECT_STREAM_MEMBERS) {
|
||||
warn!("object stream: /N {n} exceeds {MAX_OBJECT_STREAM_MEMBERS}; later members are ignored")
|
||||
}
|
||||
|
||||
// PdfCraft patch: bound members by the next distinct offset and parse each offset
|
||||
// only once, so overlapping or duplicate entries cannot amplify decoded content.
|
||||
@@ -103,44 +199,89 @@ impl ObjectStream {
|
||||
end = offset;
|
||||
}
|
||||
|
||||
let chunks_filter_map = |(index, chunk): (usize, &[_])| {
|
||||
let id = chunk[0]?;
|
||||
let offset = first_offset.saturating_add(chunk[1]? as usize);
|
||||
// PdfCraft patch: find each member's bytes first, in index order, so the members can be
|
||||
// charged to the budget before any is parsed.
|
||||
let mut members = Vec::new();
|
||||
for (index, chunk) in numbers[..len].chunks(2).enumerate() {
|
||||
let &[Some(id), Some(offset)] = chunk else { continue };
|
||||
let offset = first_offset.saturating_add(offset as usize);
|
||||
|
||||
if offset >= content.len() {
|
||||
warn!("out-of-bounds offset in object stream");
|
||||
return None;
|
||||
continue;
|
||||
}
|
||||
let &(first_index, extent_end) = extents.get(&offset)?;
|
||||
let Some(&(first_index, extent_end)) = extents.get(&offset) else {
|
||||
continue;
|
||||
};
|
||||
if index != first_index {
|
||||
warn!("duplicate offset in object stream");
|
||||
return None;
|
||||
continue;
|
||||
}
|
||||
if !keep(id) {
|
||||
continue;
|
||||
}
|
||||
// Skip leading whitespace — some PDFs emit newlines before objects in ObjStm
|
||||
let mut start = offset;
|
||||
while start < extent_end && content[start].is_ascii_whitespace() {
|
||||
while start < extent_end && content.get(start).is_some_and(u8::is_ascii_whitespace) {
|
||||
start += 1;
|
||||
}
|
||||
if start >= extent_end {
|
||||
warn!("only whitespace after offset in object stream");
|
||||
continue;
|
||||
}
|
||||
members.push((id, start, extent_end));
|
||||
}
|
||||
|
||||
// PdfCraft patch: each member builds at most `MAX_OBJECT_NODES` objects, reserved from
|
||||
// `from`; once that is spent the remaining members are skipped unparsed. Every object
|
||||
// takes at least one byte, so a member reserves no more than its length.
|
||||
let share = |&(_, start, end): &(u32, usize, usize)| parser::MAX_OBJECT_NODES.min(end - start);
|
||||
let skipped = AtomicBool::new(false);
|
||||
let parse_member = |from: &NodeBudget, member: &(u32, usize, usize)| -> Option<(ObjectId, Object)> {
|
||||
let &(id, start, end) = member;
|
||||
let granted = from.reserve(share(member));
|
||||
if granted == 0 {
|
||||
skipped.store(true, Ordering::Relaxed);
|
||||
return None;
|
||||
}
|
||||
let object = parser::direct_object(&content[start..extent_end])?;
|
||||
|
||||
Some(((id, 0), object))
|
||||
let nodes = Cell::new(granted);
|
||||
let object = content
|
||||
.get(start..end)
|
||||
.and_then(|bytes| parser::direct_object_within(bytes, &nodes));
|
||||
from.refund(nodes.get());
|
||||
if object.is_none() && nodes.get() == 0 {
|
||||
warn!("object stream: member {id} exceeds the object budget and was skipped");
|
||||
}
|
||||
Some(((id, 0), object?))
|
||||
};
|
||||
#[cfg(feature = "rayon")]
|
||||
let objects = numbers[..len]
|
||||
.par_chunks(2)
|
||||
.enumerate()
|
||||
.filter_map(chunks_filter_map)
|
||||
.collect();
|
||||
#[cfg(not(feature = "rayon"))]
|
||||
let objects = numbers[..len]
|
||||
.chunks(2)
|
||||
.enumerate()
|
||||
.filter_map(chunks_filter_map)
|
||||
.collect();
|
||||
|
||||
let want = members.iter().map(&share).fold(0, usize::saturating_add);
|
||||
let granted = budget.reserve(want);
|
||||
let objects: BTreeMap<ObjectId, Object> = if granted == want {
|
||||
// Every member fits, so they are parsed in parallel from this stream's own share:
|
||||
// its members' reservations add up to it, so none is refused whatever the order.
|
||||
let own = NodeBudget::new(granted);
|
||||
#[cfg(feature = "rayon")]
|
||||
let objects = members
|
||||
.par_iter()
|
||||
.filter_map(|member| parse_member(&own, member))
|
||||
.collect();
|
||||
#[cfg(not(feature = "rayon"))]
|
||||
let objects = members.iter().filter_map(|member| parse_member(&own, member)).collect();
|
||||
budget.refund(own.left());
|
||||
objects
|
||||
} else {
|
||||
// The budget may run out part-way, so the members are parsed in index order and the
|
||||
// ones skipped are the same on every load.
|
||||
budget.refund(granted);
|
||||
members
|
||||
.iter()
|
||||
.filter_map(|member| parse_member(budget, member))
|
||||
.collect()
|
||||
};
|
||||
if skipped.load(Ordering::Relaxed) {
|
||||
warn!("object stream: the object budget is spent; remaining members were skipped");
|
||||
}
|
||||
|
||||
Ok(ObjectStream {
|
||||
objects,
|
||||
|
||||
Vendored
+82
-20
@@ -3,6 +3,7 @@ use crate::Error;
|
||||
use crate::content::*;
|
||||
use crate::error;
|
||||
use crate::xref::*;
|
||||
use std::cell::Cell;
|
||||
use std::collections::HashSet;
|
||||
use std::str::{self, FromStr};
|
||||
|
||||
@@ -27,6 +28,19 @@ pub(crate) type NomError<'a> = nom::error::Error<ParserInput<'a>>;
|
||||
|
||||
pub(crate) type NomResult<'a, O, E = NomError<'a>> = IResult<ParserInput<'a>, O, E>;
|
||||
|
||||
/// PdfCraft patch: objects one top-level parse may build, counting every array element and
|
||||
/// dictionary value at any depth (the same cap as `pdfcraft_cos`'s `MAX_NODES`). Each object
|
||||
/// costs a hundred bytes or more, so a decoded stream of `0 0 0 ...` otherwise built one per
|
||||
/// two input bytes: gigabytes from a stream inside the decompression limit. Real arrays (page
|
||||
/// trees, parent trees, widths) stay far below.
|
||||
pub(crate) const MAX_OBJECT_NODES: usize = 1 << 22;
|
||||
|
||||
/// PdfCraft patch: the budget of a container parsed outside `_direct_object` (an indirect
|
||||
/// object's body, a content-stream operand, a CMap or inline-image dictionary). Its contents are
|
||||
/// charged but the container itself isn't, so it starts one short and the whole object stays
|
||||
/// within `MAX_OBJECT_NODES`. (`MAX_OBJECT_NODES` is a nonzero constant, so this can't underflow.)
|
||||
const MAX_NESTED_NODES: usize = MAX_OBJECT_NODES - 1;
|
||||
|
||||
#[inline]
|
||||
fn strip_nom<O>(r: NomResult<O>) -> Option<O> {
|
||||
r.ok().map(|(_, o)| o)
|
||||
@@ -270,33 +284,48 @@ fn boolean(input: ParserInput) -> NomResult<Object> {
|
||||
.parse(input)
|
||||
}
|
||||
|
||||
/// `null`.
|
||||
fn null(input: ParserInput) -> NomResult<Object> {
|
||||
map(tag(&b"null"[..]), |_| Object::Null).parse(input)
|
||||
}
|
||||
|
||||
fn array(depth: usize) -> impl Fn(ParserInput) -> NomResult<Vec<Object>> {
|
||||
/// An array, nested at most `depth` levels. PdfCraft patch: every element is charged to `nodes`
|
||||
/// (see `_direct_object`), which bounds the element count of one array and of all arrays nested
|
||||
/// in it.
|
||||
fn array(depth: usize, nodes: &Cell<usize>) -> impl Fn(ParserInput) -> NomResult<Vec<Object>> {
|
||||
move |input| {
|
||||
delimited(
|
||||
pair(tag(&b"["[..]), space),
|
||||
many0(_direct_object(depth)),
|
||||
many0(_direct_object(depth, nodes)),
|
||||
tag(&b"]"[..]),
|
||||
)
|
||||
.parse(input)
|
||||
}
|
||||
}
|
||||
|
||||
/// A dictionary, from a node budget of its own (`MAX_NESTED_NODES`).
|
||||
pub(crate) fn dictionary(input: ParserInput) -> NomResult<Dictionary> {
|
||||
_dictionary(crate::reader::MAX_NESTING_DEPTH)(input)
|
||||
let nodes = Cell::new(MAX_NESTED_NODES);
|
||||
_dictionary(crate::reader::MAX_NESTING_DEPTH, &nodes)(input)
|
||||
}
|
||||
|
||||
fn _dictionary(depth: usize) -> impl Fn(ParserInput) -> NomResult<Dictionary> {
|
||||
move |input| delimited(pair(tag(&b"<<"[..]), space), inner_dictionary(depth), tag(&b">>"[..])).parse(input)
|
||||
/// `<< ... >>`, nested at most `depth` levels, its values charged to `nodes`.
|
||||
fn _dictionary(depth: usize, nodes: &Cell<usize>) -> impl Fn(ParserInput) -> NomResult<Dictionary> {
|
||||
move |input| {
|
||||
delimited(
|
||||
pair(tag(&b"<<"[..]), space),
|
||||
inner_dictionary(depth, nodes),
|
||||
tag(&b">>"[..]),
|
||||
)
|
||||
.parse(input)
|
||||
}
|
||||
}
|
||||
|
||||
fn inner_dictionary(depth: usize) -> impl Fn(ParserInput) -> NomResult<Dictionary> {
|
||||
/// A dictionary's key-value pairs, without its delimiters; each value is charged to `nodes`.
|
||||
fn inner_dictionary(depth: usize, nodes: &Cell<usize>) -> impl Fn(ParserInput) -> NomResult<Dictionary> {
|
||||
move |input| {
|
||||
fold_many0(
|
||||
pair(terminated(name, space), _direct_object(depth)),
|
||||
pair(terminated(name, space), _direct_object(depth, nodes)),
|
||||
Dictionary::new,
|
||||
|mut dict, (key, value)| {
|
||||
dict.set(key, value);
|
||||
@@ -307,7 +336,9 @@ fn inner_dictionary(depth: usize) -> impl Fn(ParserInput) -> NomResult<Dictionar
|
||||
}
|
||||
}
|
||||
|
||||
/// A CMap's `N dict dup begin /Key value def ...` dictionary, from a node budget of its own.
|
||||
pub(crate) fn dict_dup(input: ParserInput) -> NomResult<Dictionary> {
|
||||
let nodes = Cell::new(MAX_NESTED_NODES);
|
||||
delimited(
|
||||
(
|
||||
digit1,
|
||||
@@ -323,7 +354,7 @@ pub(crate) fn dict_dup(input: ParserInput) -> NomResult<Dictionary> {
|
||||
terminated(
|
||||
pair(
|
||||
terminated(name, space),
|
||||
_direct_object(crate::reader::MAX_NESTING_DEPTH),
|
||||
_direct_object(crate::reader::MAX_NESTING_DEPTH, &nodes),
|
||||
),
|
||||
pair(tag(&b"def"[..]), multispace1),
|
||||
),
|
||||
@@ -434,7 +465,8 @@ fn reference(input: ParserInput) -> NomResult<Object> {
|
||||
map(terminated(object_id, tag(&b"R"[..])), Object::Reference).parse(input)
|
||||
}
|
||||
|
||||
fn _direct_objects(depth: usize) -> impl Fn(ParserInput) -> NomResult<Object> {
|
||||
/// Any direct object, its nested arrays and dictionaries at most `depth` levels deep.
|
||||
fn _direct_objects(depth: usize, nodes: &Cell<usize>) -> impl Fn(ParserInput) -> NomResult<Object> {
|
||||
move |input| {
|
||||
alt((
|
||||
null,
|
||||
@@ -445,34 +477,55 @@ fn _direct_objects(depth: usize) -> impl Fn(ParserInput) -> NomResult<Object> {
|
||||
map(name, Object::Name),
|
||||
map(literal_string, Object::string_literal),
|
||||
hexadecimal_string,
|
||||
map(array(depth), Object::Array),
|
||||
map(_dictionary(depth), Object::Dictionary),
|
||||
map(array(depth, nodes), Object::Array),
|
||||
map(_dictionary(depth, nodes), Object::Dictionary),
|
||||
))
|
||||
.parse(input)
|
||||
}
|
||||
}
|
||||
|
||||
fn _direct_object(depth: usize) -> impl Fn(ParserInput) -> NomResult<Object> {
|
||||
/// A direct object and the whitespace after it, charged one node once built; a spent budget
|
||||
/// or a depth of 0 is a nom `Failure`.
|
||||
fn _direct_object(depth: usize, nodes: &Cell<usize>) -> impl Fn(ParserInput) -> NomResult<Object> {
|
||||
move |input| {
|
||||
if depth == 0 {
|
||||
return Err(nom::Err::Failure(NomError::from_error_kind(input, ErrorKind::TooLarge)));
|
||||
}
|
||||
terminated(_direct_objects(depth - 1), space).parse(input)
|
||||
let (rest, object) = terminated(_direct_objects(depth - 1, nodes), space).parse(input)?;
|
||||
// PdfCraft patch: charge each object once it is built, so a failed alternative costs
|
||||
// nothing; its children were charged as they were built. A spent budget is a Failure,
|
||||
// which ends the whole parse instead of backtracking into another alternative.
|
||||
let Some(left) = nodes.get().checked_sub(1) else {
|
||||
return Err(nom::Err::Failure(NomError::from_error_kind(input, ErrorKind::TooLarge)));
|
||||
};
|
||||
nodes.set(left);
|
||||
Ok((rest, object))
|
||||
}
|
||||
}
|
||||
|
||||
/// One direct object from a fresh node budget. PdfCraft patch: object streams now parse through
|
||||
/// `direct_object_within`; only tests use this.
|
||||
#[cfg(test)]
|
||||
pub fn direct_object(input: ParserInput) -> Option<Object> {
|
||||
strip_nom(_direct_object(crate::reader::MAX_NESTING_DEPTH)(input))
|
||||
direct_object_within(input, &Cell::new(MAX_OBJECT_NODES))
|
||||
}
|
||||
|
||||
/// PdfCraft patch: parse one direct object, building at most `nodes` objects (the object itself
|
||||
/// and everything nested in it) and leaving the unspent count in `nodes`.
|
||||
pub(crate) fn direct_object_within(input: ParserInput, nodes: &Cell<usize>) -> Option<Object> {
|
||||
strip_nom(_direct_object(crate::reader::MAX_NESTING_DEPTH, nodes)(input))
|
||||
}
|
||||
|
||||
/// An indirect object's body (a stream or a direct object), from a node budget of its own.
|
||||
fn object<'a>(
|
||||
input: ParserInput<'a>, reader: &Reader, already_seen: &mut HashSet<ObjectId>, recover_stream_length: bool,
|
||||
recovery_bound: Option<usize>,
|
||||
) -> NomResult<'a, Object> {
|
||||
let nodes = Cell::new(MAX_NESTED_NODES);
|
||||
terminated(
|
||||
alt((
|
||||
|input| stream(input, reader, already_seen, recover_stream_length, recovery_bound),
|
||||
_direct_objects(crate::reader::MAX_NESTING_DEPTH),
|
||||
_direct_objects(crate::reader::MAX_NESTING_DEPTH, &nodes),
|
||||
)),
|
||||
space,
|
||||
)
|
||||
@@ -689,7 +742,9 @@ fn operator(input: ParserInput) -> NomResult<String> {
|
||||
.parse(input)
|
||||
}
|
||||
|
||||
/// One content-stream operand, from a node budget of its own.
|
||||
fn operand(input: ParserInput) -> NomResult<Object> {
|
||||
let nodes = Cell::new(MAX_NESTED_NODES);
|
||||
terminated(
|
||||
alt((
|
||||
null,
|
||||
@@ -699,7 +754,7 @@ fn operand(input: ParserInput) -> NomResult<Object> {
|
||||
map(name, Object::Name),
|
||||
map(literal_string, Object::string_literal),
|
||||
hexadecimal_string,
|
||||
map(array(crate::reader::MAX_NESTING_DEPTH), Object::Array),
|
||||
map(array(crate::reader::MAX_NESTING_DEPTH, &nodes), Object::Array),
|
||||
map(dictionary, Object::Dictionary),
|
||||
)),
|
||||
content_space,
|
||||
@@ -724,8 +779,10 @@ fn inline_image(input: ParserInput) -> NomResult<(Vec<Object>, String)> {
|
||||
preceded(pair(tag(&b"BI"[..]), content_space), cut(inline_image_impl)).parse(input)
|
||||
}
|
||||
|
||||
/// An inline image after `BI`: its dictionary (from a node budget of its own) and data.
|
||||
fn inline_image_impl(input: ParserInput) -> NomResult<(Vec<Object>, String)> {
|
||||
let (input, stream_dict) = inner_dictionary(crate::reader::MAX_NESTING_DEPTH).parse(input)?;
|
||||
let nodes = Cell::new(MAX_NESTED_NODES);
|
||||
let (input, stream_dict) = inner_dictionary(crate::reader::MAX_NESTING_DEPTH, &nodes).parse(input)?;
|
||||
let (input, _) = pair(tag(&b"ID"[..]), content_space).parse(input)?;
|
||||
match image_data_stream(input, stream_dict) {
|
||||
Ok((input, stream)) => {
|
||||
@@ -1299,6 +1356,7 @@ EI";
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
/// Arrays nested past the depth limit are an error, not a stack overflow.
|
||||
#[test]
|
||||
fn deeply_nested_array_is_rejected() {
|
||||
on_big_stack(|| {
|
||||
@@ -1306,11 +1364,13 @@ EI";
|
||||
let mut input = Vec::with_capacity(depth * 2);
|
||||
input.extend(std::iter::repeat_n(b'[', depth));
|
||||
input.extend(std::iter::repeat_n(b']', depth));
|
||||
let result = _direct_object(crate::reader::MAX_NESTING_DEPTH)(input.as_slice());
|
||||
let result =
|
||||
_direct_object(crate::reader::MAX_NESTING_DEPTH, &Cell::new(MAX_OBJECT_NODES))(input.as_slice());
|
||||
assert!(result.is_err());
|
||||
});
|
||||
}
|
||||
|
||||
/// Dictionaries nested past the depth limit are an error, not a stack overflow.
|
||||
#[test]
|
||||
fn deeply_nested_dictionary_is_rejected() {
|
||||
on_big_stack(|| {
|
||||
@@ -1322,15 +1382,17 @@ EI";
|
||||
for _ in 0..depth {
|
||||
input.extend_from_slice(b">>");
|
||||
}
|
||||
let result = _direct_object(crate::reader::MAX_NESTING_DEPTH)(input.as_slice());
|
||||
let result =
|
||||
_direct_object(crate::reader::MAX_NESTING_DEPTH, &Cell::new(MAX_OBJECT_NODES))(input.as_slice());
|
||||
assert!(result.is_err());
|
||||
});
|
||||
}
|
||||
|
||||
/// Nesting well within the limit still parses.
|
||||
#[test]
|
||||
fn modestly_nested_array_still_parses() {
|
||||
let input = b"[[[[[1]]]]]";
|
||||
let obj = _direct_object(crate::reader::MAX_NESTING_DEPTH)(test_span(input));
|
||||
let obj = _direct_object(crate::reader::MAX_NESTING_DEPTH, &Cell::new(MAX_OBJECT_NODES))(test_span(input));
|
||||
assert!(obj.is_ok());
|
||||
}
|
||||
}
|
||||
|
||||
Vendored
+83
-42
@@ -1,13 +1,13 @@
|
||||
use log::{error, warn};
|
||||
use std::cmp;
|
||||
use std::collections::{BTreeMap, HashMap, HashSet};
|
||||
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
|
||||
use std::convert::TryInto;
|
||||
#[cfg(not(feature = "async"))]
|
||||
use std::fs::File;
|
||||
#[cfg(not(feature = "async"))]
|
||||
use std::io::Read;
|
||||
use std::path::Path;
|
||||
use std::sync::Mutex;
|
||||
use std::sync::{Mutex, PoisonError};
|
||||
|
||||
#[cfg(feature = "rayon")]
|
||||
use rayon::prelude::*;
|
||||
@@ -22,7 +22,7 @@ use crate::common_data_structures;
|
||||
use crate::encryption::{self, EncryptionState};
|
||||
use crate::error::{ParseError, XrefError};
|
||||
use crate::load_options::{FilterFunc, LoadOptions};
|
||||
use crate::object_stream::ObjectStream;
|
||||
use crate::object_stream::{MAX_LOAD_NODES, NodeBudget, ObjectStream};
|
||||
use crate::parser;
|
||||
use crate::xref::{Xref, XrefEntry, XrefType};
|
||||
use crate::{Dictionary, Document, Error, IncrementalDocument, Object, ObjectId, Result};
|
||||
@@ -838,6 +838,8 @@ impl Reader<'_> {
|
||||
Ok(self.document)
|
||||
}
|
||||
|
||||
/// Load an encrypted document: read every object's raw bytes first, then decrypt and parse
|
||||
/// them, object streams from one shared node budget.
|
||||
fn load_encrypted_document(&mut self, _filter_func: Option<FilterFunc>) -> Result<()> {
|
||||
// First, extract all raw object bytes without parsing
|
||||
let entries: Vec<_> = self
|
||||
@@ -849,6 +851,8 @@ impl Reader<'_> {
|
||||
.collect();
|
||||
|
||||
let mut object_streams = Vec::new();
|
||||
// PdfCraft patch: one object budget for every object stream this load parses.
|
||||
let node_budget = NodeBudget::new(MAX_LOAD_NODES);
|
||||
|
||||
for (obj_num, entry) in entries {
|
||||
match entry {
|
||||
@@ -894,8 +898,9 @@ impl Reader<'_> {
|
||||
}
|
||||
}
|
||||
|
||||
let mut streams_to_process: std::collections::HashMap<u32, Vec<(u32, u16)>> =
|
||||
std::collections::HashMap::new();
|
||||
// PdfCraft patch: object streams in object-number order, so the members a spent node
|
||||
// budget skips are the same on every load.
|
||||
let mut streams_to_process: BTreeMap<u32, Vec<(u32, u16)>> = BTreeMap::new();
|
||||
for (obj_num, container_id, index) in object_streams {
|
||||
streams_to_process
|
||||
.entry(container_id)
|
||||
@@ -907,7 +912,11 @@ impl Reader<'_> {
|
||||
if let Some(container_obj) = self.document.objects.get(&(container_id, 0))
|
||||
&& let Ok(stream) = container_obj.as_stream()
|
||||
{
|
||||
match ObjectStream::new_with_limit(stream, self.max_decompressed_size) {
|
||||
// PdfCraft patch: parse only the members the table places in this stream, so
|
||||
// the others spend nothing from the budget.
|
||||
let listed: BTreeSet<u32> = objects_in_stream.iter().map(|&(obj_num, _)| obj_num).collect();
|
||||
let keep = |obj_num: u32| listed.contains(&obj_num);
|
||||
match ObjectStream::new_members_within(stream, self.max_decompressed_size, &node_budget, keep) {
|
||||
Ok(object_stream) => {
|
||||
for (obj_num, _index) in objects_in_stream {
|
||||
let obj_id = (obj_num, 0);
|
||||
@@ -960,10 +969,12 @@ impl Reader<'_> {
|
||||
self.document.reference_table = xref;
|
||||
}
|
||||
|
||||
/// Load every object in the cross-reference table, expanding object streams from one node
|
||||
/// budget shared by the whole load.
|
||||
fn load_objects_raw(&mut self, filter_func: Option<FilterFunc>) -> Result<()> {
|
||||
let is_encrypted = self.document.trailer.get(b"Encrypt").is_ok();
|
||||
let zero_length_streams = Mutex::new(vec![]);
|
||||
let object_streams = Mutex::new(vec![]);
|
||||
let object_stream_ids = Mutex::new(vec![]);
|
||||
|
||||
// Build a map of which container each compressed object belongs to
|
||||
// according to the xref. This prevents stale ObjStm copies (e.g., from
|
||||
@@ -1014,36 +1025,10 @@ impl Reader<'_> {
|
||||
|
||||
if let Ok(stream) = object.as_stream() {
|
||||
if stream.dict.has_type(b"ObjStm") && !is_encrypted {
|
||||
let obj_stream = match ObjectStream::new_with_limit(stream, self.max_decompressed_size) {
|
||||
Ok(obj_stream) => obj_stream,
|
||||
// Not an encryption-related loss, so it obeys the same
|
||||
// strict-mode policy as any other load error.
|
||||
Err(e) => {
|
||||
error!("Object stream load error for {object_id:?}: {e:?}");
|
||||
return if self.strict { Err(e) } else { Ok(None) };
|
||||
}
|
||||
};
|
||||
let container_id = object_id.0;
|
||||
let mut object_streams = object_streams.lock().unwrap();
|
||||
if let Some(filter_func) = filter_func {
|
||||
let objects: BTreeMap<(u32, u16), Object> = obj_stream
|
||||
.objects
|
||||
.into_iter()
|
||||
.filter(|((obj_num, _), _)| {
|
||||
compressed_obj_containers
|
||||
.get(obj_num)
|
||||
.is_none_or(|&c| c == container_id)
|
||||
})
|
||||
.filter_map(|(object_id, mut object)| filter_func(object_id, &mut object))
|
||||
.collect();
|
||||
object_streams.extend(objects);
|
||||
} else {
|
||||
object_streams.extend(obj_stream.objects.into_iter().filter(|((obj_num, _), _)| {
|
||||
compressed_obj_containers
|
||||
.get(obj_num)
|
||||
.is_none_or(|&c| c == container_id)
|
||||
}));
|
||||
}
|
||||
// PdfCraft patch: object streams are expanded after this parallel pass,
|
||||
// one at a time in object-number order (below).
|
||||
let mut object_stream_ids = object_stream_ids.lock().unwrap_or_else(PoisonError::into_inner);
|
||||
object_stream_ids.push(object_id);
|
||||
} else if stream.content.is_empty() {
|
||||
let mut zero_length_streams = zero_length_streams.lock().unwrap();
|
||||
zero_length_streams.push(object_id);
|
||||
@@ -1079,9 +1064,59 @@ impl Reader<'_> {
|
||||
.collect::<Result<BTreeMap<_, _>>>()?;
|
||||
}
|
||||
|
||||
// Only add entries, but never replace entries
|
||||
for (id, entry) in object_streams.into_inner().unwrap() {
|
||||
self.document.objects.entry(id).or_insert(entry);
|
||||
// PdfCraft patch: expand the object streams one at a time in object-number order, from one
|
||||
// node budget for the whole load, so many streams, each inside the decompression limit,
|
||||
// cannot together build billions of objects, and the members a spent budget skips are the
|
||||
// same on every load. A stream parses only the members it would add: none the table
|
||||
// places in another stream (stale copies), and none already loaded, as an ordinary object
|
||||
// or from an earlier stream, since a member never replaces an object. Those spend nothing.
|
||||
// Members the table doesn't list at all are still parsed (a rebuilt table lists no
|
||||
// compressed objects) and share the budget.
|
||||
let node_budget = NodeBudget::new(MAX_LOAD_NODES);
|
||||
let mut object_stream_ids = object_stream_ids.into_inner().unwrap_or_else(PoisonError::into_inner);
|
||||
object_stream_ids.sort_unstable();
|
||||
object_stream_ids.dedup();
|
||||
for object_id in object_stream_ids {
|
||||
let container_id = object_id.0;
|
||||
let loaded = &self.document.objects;
|
||||
let keep = |obj_num: u32| {
|
||||
compressed_obj_containers
|
||||
.get(&obj_num)
|
||||
.is_none_or(|&c| c == container_id)
|
||||
&& !loaded.contains_key(&(obj_num, 0))
|
||||
};
|
||||
let parsed = match loaded.get(&object_id).map(Object::as_stream) {
|
||||
Some(Ok(stream)) => {
|
||||
ObjectStream::new_members_within(stream, self.max_decompressed_size, &node_budget, keep)
|
||||
}
|
||||
_ => continue,
|
||||
};
|
||||
match parsed {
|
||||
Ok(obj_stream) => {
|
||||
let members: BTreeMap<_, _> = match filter_func {
|
||||
Some(filter_func) => obj_stream
|
||||
.objects
|
||||
.into_iter()
|
||||
.filter_map(|(object_id, mut object)| filter_func(object_id, &mut object))
|
||||
.collect(),
|
||||
None => obj_stream.objects,
|
||||
};
|
||||
// Only add entries, but never replace entries
|
||||
for (id, entry) in members {
|
||||
self.document.objects.entry(id).or_insert(entry);
|
||||
}
|
||||
}
|
||||
// Not an encryption-related loss, so it obeys the same strict-mode policy as
|
||||
// any other load error; a lenient load drops the stream, as it drops any object
|
||||
// that fails to load.
|
||||
Err(e) => {
|
||||
error!("Object stream load error for {object_id:?}: {e:?}");
|
||||
if self.strict {
|
||||
return Err(e);
|
||||
}
|
||||
self.document.objects.remove(&object_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for object_id in zero_length_streams.into_inner().unwrap() {
|
||||
@@ -1174,8 +1209,14 @@ impl Reader<'_> {
|
||||
let container_id = (container_id, 0);
|
||||
let container_obj = self.get_object(container_id, already_seen)?;
|
||||
let container_stream = container_obj.as_stream()?;
|
||||
let object_stream = ObjectStream::new_with_limit(container_stream, self.max_decompressed_size)?;
|
||||
object_stream.objects.get(&id).cloned().ok_or(Error::MissingXrefEntry)
|
||||
// PdfCraft patch: a lookup parses only the member it wants, from one object's budget
|
||||
// rather than a load's, so the members before it can't spend that budget first.
|
||||
let budget = NodeBudget::new(parser::MAX_OBJECT_NODES);
|
||||
let mut object_stream =
|
||||
ObjectStream::new_members_within(container_stream, self.max_decompressed_size, &budget, |obj_num| {
|
||||
obj_num == id.0
|
||||
})?;
|
||||
object_stream.objects.remove(&id).ok_or(Error::MissingXrefEntry)
|
||||
}
|
||||
|
||||
pub fn get_object(&self, id: ObjectId, already_seen: &mut HashSet<ObjectId>) -> Result<Object> {
|
||||
|
||||
Reference in New Issue
Block a user