Commit Graph
106 Commits
Author SHA1 Message Date
FredApps cd845444f6 Merge remote-tracking branch 'origin/codex/dng-capture-info' into w1653 2026-11-16 08:52:13 +01:00
FredApps d8a2de2224 Merge remote-tracking branch 'origin/codex/dng-metadata' into w1648 2026-11-16 08:52:09 +01:00
FredApps 2686ac1f95 Merge remote-tracking branch 'origin/codex/shared-dng-opcode-envelope' into w1647 2026-11-16 08:52:05 +01:00
FredApps 456e71da87 Merge remote-tracking branch 'origin/codex/shared-dng-raw' into w1645 2026-11-16 08:52:01 +01:00
FredApps 8a067d58d0 raw: dng_lines checks unpack sizes, budgets each chunk's real decode allocation, and allows subsampled lossy JPEG
- unpack: checked sample, row-bit and output products, capped at MAX_SAMPLES before allocating
- every chunk is charged what its decoder may allocate (lossless: coded rows + 1/8; lossy: the
  32-pixel MCU-padded tile) against the aggregate budget before decoding, the lossless decoder
  is held to that, and truncated capacity is released
- the lossy encoded-size floor allows 64 x spp samples per two-bit block (chroma subsampling)
2026-11-16 08:51:58 +01:00
FredApps 250fc97fa8 raw: opcode_params reads areas as the shared opcodes::Area and clamps them (Area::clamp); document how it differs from the typed opcodes model 2026-11-16 08:36:26 +01:00
FredApps 0e4b33fdf0 Merge remote-tracking branch 'origin/codex/dng-capture-info' into w1653 2026-11-16 08:35:30 +01:00
FredApps 7feff2a02d raw: dng_metadata::Timecode is the crate Timecode; share the CinemaDNG frame-rate range with dng.rs
Removes the duplicated Timecode type and BCD decoder (Sonar duplication 28%) and the copied
frame-rate plausibility check; capture_info is tested against the full reader's CinemaTags.
2026-11-16 08:35:12 +01:00
FredApps 77125b817f Merge remote-tracking branch 'origin/codex/dng-metadata' into w1648 2026-11-16 08:34:03 +01:00
FredApps 6225363b1f raw: dng_metadata tests against hand-built DNGs and the full reader; document why it keeps its own tag reads 2026-11-16 08:33:43 +01:00
FredApps b611b03a29 Merge remote-tracking branch 'origin/codex/shared-dng-opcode-envelope' into w1647 2026-11-16 08:32:30 +01:00
FredApps 89e96d2790 raw: opcodes::parse_list/write_list use the opcode_raw envelope walker and encoder
One opcode-list envelope implementation: opcode_raw::Records walks the records (stopping at a
truncated one) for both the strict opcode_raw::parse and the typed opcodes::parse_list, and
opcodes::write_list encodes through opcode_raw::encode instead of truncating lengths with as.
2026-11-16 08:32:10 +01:00
FredApps 456b7d5b35 Merge remote-tracking branch 'origin/codex/shared-dng-raw' into w1645 2026-11-16 08:30:57 +01:00
FredApps 59d28093e6 raw: dng_lines reuses the DNG reader's unpackers and JPEG decoder, drops the image dependency
- uncompressed rows go through crate::unpack (read_u16s / unpack_msb), as in dng.rs
- lossy JPEG tiles (34892) use tiffraw::lossy_jpeg_with_tile (zune-jpeg) with its tile-size checks
- line-dng now implies dng instead of pulling in image
- tests: every layout and compression read against the full reader, refusals, unpack depths
2026-11-16 08:27:55 +01:00
FredApps 3fd0383c81 Merge remote-tracking branch 'origin/main' into w1644 2026-11-16 08:20:40 +01:00
Dylan M. Taylor 4374003c36 raw: reject zero-pitch public opcode area iterators 2026-11-15 23:26:23 -05:00
Dylan M. Taylor a8fc359bc5 raw: share CinemaDNG capture metadata 2026-11-15 21:49:46 -05:00
Dylan M. Taylor 61a9330f4e Adapt attributed FilmCraft metadata to bounded shared reader
Preserve vangel76 original FilmCraft source commit 28247afae7806590dcdb149d744408739fb738df before the shared metadata extraction. Final metadata implementation is unchanged from the existing PR.
2026-11-15 21:49:46 -05:00
Dylan M. Taylor b00cb4533a raw: use fixed-size opcode record chunks 2026-11-15 21:49:46 -05:00
vangel76andClaude Opus 5.5 79c6602a35 #1065: clean-room DNG / CinemaDNG decoder crate
Problem: camera RAW is at 0% (docs/codec-parity.md, #383). CinemaDNG,
the one cinema RAW format with a public specification, could not be
read at all.

Fix: new L0 crate filmcraft-dng, written from the public DNG 1.4-1.7,
TIFF 6.0 / TIFF-EP, ITU-T T.81 and CinemaDNG specifications only (no
other decoder's source read; editions in the crate README):
- TIFF container with SubIFDs, bounded (64 IFDs, 4 levels, cycles end
  the walk) and checked offsets; the main raw IFD is the largest
  NewSubFileType 0 CFA / LinearRaw image.
- Raw data: 1-16-bit uncompressed (MSB-first packing) and lossless JPEG
  (process 14, predictors 1-7, 1-4 components, point transform, line
  restarts), strips and tiles, tiles decoded in parallel; allocations
  capped by size and by the data present.
- Processing per the DNG model: ActiveArea, LinearizationTable, per-site
  black levels with deltas, white levels, as-shot white balance with
  neutral highlight clipping, Malvar-He-Cutler Bayer demosaic (bilinear
  for other patterns), DefaultCrop, ColorMatrix / CameraCalibration /
  ForwardMatrix with illuminant interpolation, Bradford to linear
  Rec. 709, BaselineExposure; output scene-linear RGBA floats.
- CinemaDNG FrameRate, TimeCodes and ReelName tags.
- A test-only writer (feature `writer`) synthesises DNGs. Tests: raw
  samples bit-exact for every depth / predictor / layout, processing
  against hand-computed values, hostile files (every truncation, 12k
  mutations, cyclic IFDs, huge sizes, bad offsets), and LibRaw's
  unprocessed_raw as an external oracle reading the same samples
  (skipped when not installed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 21:49:46 -05:00
vangel76andClaude Opus 5.5 440e14c7f2 #1065: clean-room DNG / CinemaDNG decoder crate
Problem: camera RAW is at 0% (docs/codec-parity.md, #383). CinemaDNG,
the one cinema RAW format with a public specification, could not be
read at all.

Fix: new L0 crate filmcraft-dng, written from the public DNG 1.4-1.7,
TIFF 6.0 / TIFF-EP, ITU-T T.81 and CinemaDNG specifications only (no
other decoder's source read; editions in the crate README):
- TIFF container with SubIFDs, bounded (64 IFDs, 4 levels, cycles end
  the walk) and checked offsets; the main raw IFD is the largest
  NewSubFileType 0 CFA / LinearRaw image.
- Raw data: 1-16-bit uncompressed (MSB-first packing) and lossless JPEG
  (process 14, predictors 1-7, 1-4 components, point transform, line
  restarts), strips and tiles, tiles decoded in parallel; allocations
  capped by size and by the data present.
- Processing per the DNG model: ActiveArea, LinearizationTable, per-site
  black levels with deltas, white levels, as-shot white balance with
  neutral highlight clipping, Malvar-He-Cutler Bayer demosaic (bilinear
  for other patterns), DefaultCrop, ColorMatrix / CameraCalibration /
  ForwardMatrix with illuminant interpolation, Bradford to linear
  Rec. 709, BaselineExposure; output scene-linear RGBA floats.
- CinemaDNG FrameRate, TimeCodes and ReelName tags.
- A test-only writer (feature `writer`) synthesises DNGs. Tests: raw
  samples bit-exact for every depth / predictor / layout, processing
  against hand-computed values, hostile files (every truncation, 12k
  mutations, cyclic IFDs, huge sizes, bad offsets), and LibRaw's
  unprocessed_raw as an external oracle reading the same samples
  (skipped when not installed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 21:49:46 -05:00
Dylan M. Taylor fab94c80fe raw: bound opcode area count validation 2026-11-15 21:49:46 -05:00
Dylan M. Taylor 75cff558e5 raw: parse DNG opcode parameters into typed values 2026-11-15 21:49:46 -05:00
vangel76andClaude Opus 5.5 cb3f76848b #1065: DNG opcode lists (OpcodeList1/2/3)
Problem: DNG opcode lists were ignored, so lens shading gain maps
(phones and many cameras), bad pixel maps, vignette and distortion
corrections and trims that a file asks for were silently left out,
and an opcode a reader must apply could not be refused.

Fix (DNG 1.7.1.0 chapter 7): a big-endian opcode list parser (at most
256 opcodes, every count checked against the parameter bytes before
allocating) and the opcodes WarpRectilinear, WarpFisheye,
WarpRectilinear2 (with the skip rule), FixVignetteRadial,
FixBadPixelsConstant / List, TrimBounds, MapTable, MapPolynomial,
GainMap, DeltaPerRow / Column and ScalePerRow / Column. OpcodeList1 runs
on the stored samples (0-65535), OpcodeList2 on the linear values
before white balance (the fused normalise splits only then),
OpcodeList3 on the demosaiced camera values; values are clipped after
each opcode. Unknown opcodes or ones newer than 1.7.1.0 are skipped
when optional and an error naming them when mandatory; preview-only
opcodes are skipped on the reduced-resolution path, and a mandatory
OpcodeList3 keeps full resolution. Interpretations where the spec
leaves room are recorded in the README. Files without opcode lists
develop exactly as before.

Tests: unit tests per opcode with known results (tables, polynomials,
deltas, scales, gain map interpolation and edge replication, vignette,
warps reproducing a ramp, fisheye, the skip rule, bad pixels, trim,
hostile parameters) and whole-file tests of each list's stage, the
optional flag, the preview path, and 3000 mutations under catch_unwind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 21:49:46 -05:00
FredApps 58d6844dad raw: carry the linearization table through DngImage views
Main's writer (#1745) reads linearized/linearization_table from the image;
the borrowed view needs the same fields so the merge compiles.
2026-11-16 01:58:21 +01:00
FredApps c18762fb57 Merge remote-tracking branch 'origin/main' into t1770 2026-11-16 01:56:53 +01:00
Dylan M. Taylor 2d271b95bd raw: expose opcode encoding as a fallible API without panics 2026-11-15 19:45:27 -05:00
Dylan M. Taylor cbc6172375 raw: reject unrepresentable opcode envelope lengths before encoding 2026-11-15 19:44:12 -05:00
Dylan M. Taylor c48cf22a03 Adapt attributed FilmCraft opcode envelopes to shared bounded API
Reuse vangel76 source from filmcraft commit 7e6f075ef7d0e9d2517628a7f48f648b5aff2f1b. Keep shared extraction code unchanged while retaining corrected parent provenance.
2026-11-15 19:41:21 -05:00
vangel76andClaude Opus 5.5 c31578e1ff #1065: DNG opcode lists (OpcodeList1/2/3)
Problem: DNG opcode lists were ignored, so lens shading gain maps
(phones and many cameras), bad pixel maps, vignette and distortion
corrections and trims that a file asks for were silently left out,
and an opcode a reader must apply could not be refused.

Fix (DNG 1.7.1.0 chapter 7): a big-endian opcode list parser (at most
256 opcodes, every count checked against the parameter bytes before
allocating) and the opcodes WarpRectilinear, WarpFisheye,
WarpRectilinear2 (with the skip rule), FixVignetteRadial,
FixBadPixelsConstant / List, TrimBounds, MapTable, MapPolynomial,
GainMap, DeltaPerRow / Column and ScalePerRow / Column. OpcodeList1 runs
on the stored samples (0-65535), OpcodeList2 on the linear values
before white balance (the fused normalise splits only then),
OpcodeList3 on the demosaiced camera values; values are clipped after
each opcode. Unknown opcodes or ones newer than 1.7.1.0 are skipped
when optional and an error naming them when mandatory; preview-only
opcodes are skipped on the reduced-resolution path, and a mandatory
OpcodeList3 keeps full resolution. Interpretations where the spec
leaves room are recorded in the README. Files without opcode lists
develop exactly as before.

Tests: unit tests per opcode with known results (tables, polynomials,
deltas, scales, gain map interpolation and edge replication, vignette,
warps reproducing a ramp, fisheye, the skip rule, bad pixels, trim,
hostile parameters) and whole-file tests of each list's stage, the
optional flag, the preview path, and 3000 mutations under catch_unwind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 19:41:21 -05:00
FredApps 00dbe7449a raw: initialize linearization_table in the dngwrite test image 2026-11-15 19:31:39 -05:00
FredApps 1a83fd70a3 Merge pull request #1761 from storytold/extract/dcp-wire
craft-raw: extract DCP camera-profile reader and writer
2026-11-16 01:30:25 +01:00
Dylan M. Taylor e6a9bcf338 Adapt attributed FilmCraft DNG raw history to shared line decoder
Preserve path-scoped source ancestry, original contributor authors, messages and dates; retain reviewed shared APIs and final integration tree.
2026-11-15 19:19:49 -05:00
FredApps 1f1211fe64 Merge pull request #1077 from vangel76/dng-decode-plus
#1065: DNG raw settings, faster decode, opcode lists, lossy JPEG DNG
2026-11-15 19:19:49 -05:00
Dylan M. Taylor 5e5c0cf874 dng: reject zero or over-16-bit sample depths before unpacking (#4192)
unpack accepted a bit depth of 0: the input requirement became zero bytes,
so one source byte produced a header-sized buffer of zero samples. Depths
outside 1..=16 (the u16 output range) are now rejected up front.
2026-11-15 19:19:49 -05:00
vangel76andClaude Sonnet 5.5 41801059b3 Address review: lossy JPEG channel check, opcode opt-in in develop_with, writer errors
- raw: a lossy JPEG tile must have as many components as SamplesPerPixel.
- process::develop_with: use the opcode lists only when apply_opcodes is set.
- writer: lossy_jpeg returns an error instead of an empty chunk or an encoder panic; new try_write rejects unsupported sample counts and bits/CFA options for lossy JPEG.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-15 19:19:49 -05:00
vangel76andClaude Sonnet 5.5 b8a3f828a1 Merge upstream/main into dng-decode-plus
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-15 19:19:49 -05:00
vangel76andClaude Sonnet 5.5 2f093b0e2b dng: restore rayon import in raw.rs after merge
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-15 19:19:49 -05:00
vangel76andClaude Sonnet 5.5 206f2b0485 Merge upstream/main into dng-decode-plus
Move the new DNG integration tests into the shared tests/it binary.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-15 19:19:49 -05:00
vangel76andClaude Opus 5.5 589d84d0f4 #1065: lossy JPEG DNG (Compression 34892)
Problem: lossy DNGs (DNG 1.4 Compression 34892, used for proxy and
"lossy compressed" DNGs, 8-bit LinearRaw tiles usually with a
LinearizationTable) were refused as unsupported.

Fix: decode each tile or strip with the baseline JPEG decoder the
workspace already depends on (`image` with only its jpeg feature, as in
crates/codecs and crates/media; no new crate in Cargo.lock, builds for
wasm32), one component as greyscale or three as colour, the samples
filling the chunk in raster order as with lossless JPEG. As the spec
allows it only for 8-bit LinearRaw, CFA or other depths are refused with
a message naming lossy JPEG. The decoder runs under catch_unwind with an
allocation limit, its picture must match the chunk, and a tiny file
cannot claim a huge picture (at least two bits per 8 x 8 block). Info
reports "lossy JPEG". The test writer gains a lossy JPEG path through
`image`'s encoder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 19:19:49 -05:00
vangel76andClaude Opus 5.5 eb5844d4da #1065: faster DNG decode and development, reduced-resolution path
Problem: a 4096 x 2160 12-bit lossless-JPEG CinemaDNG frame took about
50 ms (tiles) to 115 ms (one strip) on 16 threads and about 200 ms on
one, too slow for smooth 4K playback, and every frame was developed at
full size even for small monitors and proxies.

Fix, without changing the output (f32 bit patterns compared):
- Lossless JPEG: one 13-bit lookup gives a code and its difference
  bits together; the bit reader takes eight bytes at once when they
  hold no 0xFF; per-predictor line loops.
- Tiles decode in parallel by bands straight into place; a single
  strip covering the image is used without a copy.
- Normalisation: without black-level deltas, a lookup table per
  CFA / black-level site replaces the per-sample float arithmetic
  (each entry computed by the same expression).
- Demosaic, crop and colour run row by row from the normalised plane
  (no padded copy, no RGB intermediate), with 5 x 5 windows sliding
  along the five mirrored rows; the bilinear (Fast) Bayer demosaic
  sums in the reference order.
- ProcessOptions::target_size now develops Bayer, LinearRaw and mono
  pictures at 1/2, 1/4 or 1/8 size (the smallest covering the
  target) by averaging each colour over the blocks, without demosaic.

The first version is kept as `reference` (feature `writer`, tests and
benchmarks only): tests/equivalence.rs checks identical raw samples and
developed bits over all Bayer phases, both qualities, odd crops,
active areas, linearisation, black deltas, codes above the bit depth,
LinearRaw, mono, other CFAs and tiny pictures; examples/bench.rs times
both interleaved at 1/4/16 threads. Best of 9 under load ~12: tiles
2.5x (1 thread) to 2.9x (16), one strip 2.1-2.6x; half size 3-4x.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 19:19:49 -05:00
c6f4234682 dng: bound lossless-JPEG decoding by the data it has and reject shared tile bytes (#3680)
* dng: bound lossless-JPEG decoding by the data it has and reject shared tile bytes (#2776)

Remaining amplification holes after the decoded-sample cap: the per-chunk
decode cap had 256 Ki samples of slack on top of the chunk's size, a patched
frame header could ask for far more samples than the entropy-coded bytes can
hold (zero bits were fed past the end), and strips/tiles pointing at the same
bytes were each decoded in full.

ljpeg::decode now fails with Truncated when width*height*components needs more
than one bit per sample of the remaining data, the caller caps at exactly the
chunk's sample count, and raw::chunks rejects overlapping strip/tile ranges.

Tested with a tiny lj92 stream whose SOF is patched to 512x512 and a tiled
file whose second tile reuses the first tile's bytes.

Fixes #2776

* dng: keep the slack for a lossless JPEG coded larger than its chunk

The entropy-length bound already caps the allocation; a last strip coded at
full RowsPerStrip holds more samples than the chunk needs and is truncated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Brandon Thomas <echelon@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 19:19:48 -05:00
Earl Co 56ac43a7f1 security(dng): bound tile decoding and IFD traversal (#2952) 2026-11-15 19:19:48 -05:00
vangel76andClaude Opus 5.5 953cb7ef3c #1065: clean-room DNG / CinemaDNG decoder crate
Problem: camera RAW is at 0% (docs/codec-parity.md, #383). CinemaDNG,
the one cinema RAW format with a public specification, could not be
read at all.

Fix: new L0 crate filmcraft-dng, written from the public DNG 1.4-1.7,
TIFF 6.0 / TIFF-EP, ITU-T T.81 and CinemaDNG specifications only (no
other decoder's source read; editions in the crate README):
- TIFF container with SubIFDs, bounded (64 IFDs, 4 levels, cycles end
  the walk) and checked offsets; the main raw IFD is the largest
  NewSubFileType 0 CFA / LinearRaw image.
- Raw data: 1-16-bit uncompressed (MSB-first packing) and lossless JPEG
  (process 14, predictors 1-7, 1-4 components, point transform, line
  restarts), strips and tiles, tiles decoded in parallel; allocations
  capped by size and by the data present.
- Processing per the DNG model: ActiveArea, LinearizationTable, per-site
  black levels with deltas, white levels, as-shot white balance with
  neutral highlight clipping, Malvar-He-Cutler Bayer demosaic (bilinear
  for other patterns), DefaultCrop, ColorMatrix / CameraCalibration /
  ForwardMatrix with illuminant interpolation, Bradford to linear
  Rec. 709, BaselineExposure; output scene-linear RGBA floats.
- CinemaDNG FrameRate, TimeCodes and ReelName tags.
- A test-only writer (feature `writer`) synthesises DNGs. Tests: raw
  samples bit-exact for every depth / predictor / layout, processing
  against hand-computed values, hostile files (every truncation, 12k
  mutations, cyclic IFDs, huge sizes, bad offsets), and LibRaw's
  unprocessed_raw as an external oracle reading the same samples
  (skipped when not installed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 19:19:48 -05:00
FredApps 920c745f42 raw: initialize linearization_table in the dngwrite test image 2026-11-16 01:13:47 +01:00
Dylan M. Taylor c599f64832 Merge pull request #1662 from vangel76/perf/raw-ljpeg-nef-1655
CR2, DNG, ARW lossless-JPEG and NEF/RW2 Huffman raw decoding is up to 2x as fast
2026-11-15 18:49:54 -05:00
FredApps b24a1ee25c Merge pull request #1745 from storytold/fix/raw-dng-lin-before-opcodes
raw: leave LinearizationTable unapplied when OpcodeList1 is present
2026-11-16 00:44:49 +01:00
Dylan M. Taylor f60aec302b Merge pull request #1659 from vangel76/perf/raw-dng-parallel-copy-1651
DNG files decode up to 2.4x faster on many cores (5x uncompressed)
2026-11-15 18:27:55 -05:00
Dylan M. Taylor ba240b89a3 Merge pull request #1677 from vangel76/perf/raw-raf-1656
Fujifilm compressed RAF files decode about 1.4x as fast
2026-11-15 18:25:44 -05:00
Dylan M. Taylor 4608875afb Merge pull request #1683 from vangel76/perf/raw-cr3-1654
Canon C-RAW (CR3) files decode about 1.5x as fast, lossless CR3 up to 1.4x
2026-11-15 18:23:39 -05:00