From fdcc7158293c19e0a224a01e2009243110e4a9f5 Mon Sep 17 00:00:00 2001 From: Valentin Klein Date: Fri, 6 Nov 2026 17:37:23 +0100 Subject: [PATCH] Viewer exposure on the full float range of 32 bpc frames (#2937) * Viewer exposure on the full float range of 32 bpc frames (#2797) 32 bpc viewer frames on the desktop are rendered without the HDR step (skip_hdr_step, #2916) and kept as half floats in the RAM preview. A display pass on egui-wgpu's device turns each frame into one RGBA8 texture with mips per viewer, in this order: decode, exposure in linear light, Compand / Tone Map, encode, display colour, Show Channel, rounding. Before, exposure was an 8-bit lookup applied after tone mapping, clamping and the display conversion, so -2 EV could not bring values above 1.0 back. - engine: viewer::DisplayPass describes the pass as data; its pixel() is the CPU fallback (no device) and the shader's reference. It replaces exposure_lut / display_transform. FrameEncoding records a frame's curve and the HDR step it still needs. 32 bpc snapshots are taken without the HDR step; snapshot_rgba8 applies it. - ui-egui: FrameImage::CpuF16 (half crate, F16C at run time), byte accounting at 8 bytes a pixel in the cache and the prefetch estimates; display_gpu.rs + display_gpu.wgsl; main viewer, snapshot and passive viewers draw through the pass; the 8-bit image other panels read is made on demand with the HDR step; the Info panel shows float values. - Half-float frames are not written to the disk cache yet. Co-Authored-By: Claude Opus 5.5 * Viewer display pass: f16 disk cache entries, window-device gating, review fixes - Disk cache: half-float frame entries (format byte in the entry header, byte planes + LZ4); 8-bit entries keep format 0 and stay valid. 32 bpc viewer frames are written and read again, and the timeline's disk bar sees them. A damaged half-float entry is deleted. - The display pass runs on the window's egui-wgpu device, with or without the GPU compositor (still building, failed, GL fallback). Its pipelines are made inside error scopes; a backend that rejects them gets the CPU pass instead of a blank viewer. - A new LUT of the same size (another ICC profile, Preserve RGB, another display) runs the pass again. - f32 to f16: one pass per L1-sized block, the NaN / infinity fix-up on the bits, no fill of the buffer (4K: 4.7 ms, as the 8-bit conversion). - Mips of odd-sized levels include the last column / row; the snapshot's half floats and texture go with the snapshot; no placeholder texture ids; u64 size checks. - Tests: LUT profiles and negative values against the shader, Bake reruns, the passive viewer's colour, f16 disk round trips; exact comparisons allow 1 LSB on software rasterizers. Docs: the display pass and the f16 disk entries in architecture.md. Co-Authored-By: Claude Opus 5.5 * Viewer f16 disk-cache test: temp dir removed on failure too, wait for the queued disk write Review points: the test's directory was only removed by its last statement (now tempfile::TempDir), and the frame is released before its disk write is queued, so flush() could return early (now waits for the entry). Co-Authored-By: Claude Sonnet 5.5 * Viewer: oversized half-float frames use the CPU display pass, so exposure/HDR come before the averaging down Co-Authored-By: Claude Sonnet 5.5 * Viewer: fmt and a debug assert for the half-float upload path Co-Authored-By: Claude Sonnet 5.5 * Disk cache: persist_missing also writes half-float frames Co-Authored-By: Claude Sonnet 5.5 * Passive viewer: oversized half-float frames take the CPU display pass instead of leaving a stale texture Co-Authored-By: Claude Sonnet 5.5 --------- Co-authored-by: Claude Opus 5.5 --- Cargo.lock | 3 + Cargo.toml | 1 + crates/engine/src/cmd/viewer_cmds.rs | 17 +- crates/engine/src/tests.rs | 1 + crates/engine/src/tests/display_pass.rs | 206 +++++++ crates/engine/src/tests/viewer.rs | 30 +- crates/engine/src/viewer.rs | 71 +-- crates/engine/src/viewer/display_pass.rs | 404 ++++++++++++ crates/render/src/disk_cache.rs | 179 +++++- crates/ui-egui/Cargo.toml | 6 + crates/ui-egui/src/app.rs | 7 + crates/ui-egui/src/app/gpu.rs | 25 +- crates/ui-egui/src/app/prefetch.rs | 10 +- crates/ui-egui/src/app/viewer_frames.rs | 43 +- crates/ui-egui/src/color_picker.rs | 91 ++- crates/ui-egui/src/display_gpu.rs | 492 +++++++++++++++ crates/ui-egui/src/display_gpu.wgsl | 314 ++++++++++ crates/ui-egui/src/display_gpu/tests.rs | 582 ++++++++++++++++++ crates/ui-egui/src/frames.rs | 102 ++- crates/ui-egui/src/frames/f16.rs | 91 +++ crates/ui-egui/src/frames/tests.rs | 184 +++++- crates/ui-egui/src/lib.rs | 23 +- crates/ui-egui/src/panels/info.rs | 44 +- crates/ui-egui/src/panels/viewer.rs | 1 + crates/ui-egui/src/panels/viewer/frame.rs | 18 +- crates/ui-egui/src/panels/viewer_tools.rs | 223 ++++--- crates/ui-egui/src/panels/viewers.rs | 97 ++- crates/ui-egui/tests/it/ui_gpu.rs | 39 ++ crates/ui-egui/tests/it/ui_web_gpu_storage.rs | 2 +- crates/ui-egui/tests/it/ui_web_offload.rs | 2 +- docs/architecture.md | 23 +- 31 files changed, 3081 insertions(+), 250 deletions(-) create mode 100644 crates/engine/src/tests/display_pass.rs create mode 100644 crates/engine/src/viewer/display_pass.rs create mode 100644 crates/ui-egui/src/display_gpu.rs create mode 100644 crates/ui-egui/src/display_gpu.wgsl create mode 100644 crates/ui-egui/src/display_gpu/tests.rs create mode 100644 crates/ui-egui/src/frames/f16.rs diff --git a/Cargo.lock b/Cargo.lock index 7d58b050d..5b426c8f8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1970,6 +1970,7 @@ dependencies = [ name = "effectcraft-ui-egui" version = "0.21.0" dependencies = [ + "bytemuck", "chrono", "craft-quit", "craft-ui", @@ -1992,6 +1993,7 @@ dependencies = [ "egui_extras", "egui_kittest", "exr", + "half", "image", "kurbo", "log", @@ -2000,6 +2002,7 @@ dependencies = [ "serde", "serde_json", "sys-locale", + "tempfile", "toml", "web-time", ] diff --git a/Cargo.toml b/Cargo.toml index 8ffe7b988..abb2256c2 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -112,6 +112,7 @@ filmcraft-geom = { git = "https://github.com/storytold/filmcraft", rev = "d5ead4 wgpu = { version = "30.0.1", features = ["fragile-send-sync-non-atomic-wasm"] } bytemuck = "1" pollster = "1" +half = "2.7" # ui # Shared frontend components; keep this dependency above the engine boundary. diff --git a/crates/engine/src/cmd/viewer_cmds.rs b/crates/engine/src/cmd/viewer_cmds.rs index cba8a418c..698a46f02 100644 --- a/crates/engine/src/cmd/viewer_cmds.rs +++ b/crates/engine/src/cmd/viewer_cmds.rs @@ -159,9 +159,14 @@ fn take_snapshot(s: &mut Session, p: &Value) -> Result { let comp = s.project.comp(cid).ok_or(EngineError::NoComp)?; let t = s.time_of(cid); let scale = f_p(p, "scale").unwrap_or(1.0).clamp(0.05, 1.0); - let img = s.render(cid, t, RenderOpts { scale, guides: true, view: s.view_camera(cid).filter(|_| comp.has_3d()), ..Default::default() }); + // 32 bpc: without the HDR step, so viewer exposure reaches the over-range values (the + // display pass applies the step, as for viewer frames). + let skip_hdr_step = s.project.settings.bit_depth.levels().is_none(); + let opts = RenderOpts { scale, guides: true, view: s.view_camera(cid).filter(|_| comp.has_3d()), skip_hdr_step, ..Default::default() }; + let img = s.render(cid, t, opts); let (w, h) = (img.width, img.height); - s.snapshot = Some(Snapshot { comp: cid, time: t, scale, image: std::sync::Arc::new(img) }); + let encoding = crate::viewer::FrameEncoding::of(&s.project.settings, skip_hdr_step); + s.snapshot = Some(Snapshot { comp: cid, time: t, scale, image: std::sync::Arc::new(img), encoding }); s.events.push(crate::Event::ProjectChanged { revision: s.revision }); Ok(json!({"width": w, "height": h, "time": t.seconds()})) } @@ -298,15 +303,17 @@ fn new_path(s: &mut Session, p: &Value) -> Result { Ok(json!({"layer": lid.0, "group": group, "path": path_uid})) } -/// Snapshot image as straight 8-bit RGBA rows (for frontends / `--out`). +/// Snapshot image as straight 8-bit RGBA rows (for frontends / `--out`), with the HDR step a +/// 32 bpc snapshot was taken without. pub fn snapshot_rgba8(s: &Session) -> Option<(u32, u32, Vec)> { let snap = s.snapshot.as_ref()?; let img = &snap.image; let mut out = Vec::with_capacity(img.data.len() * 4); for p in &img.data { let a = p[3].clamp(0.0, 1.0); - let un = |c: f32| if a > 0.0 { (c / a).clamp(0.0, 1.0) } else { 0.0 }; - out.extend([un(p[0]), un(p[1]), un(p[2]), a].map(|v| (v * 255.0 + 0.5) as u8)); + let un = |c: f32| if a > 0.0 { c / a } else { 0.0 }; + let c = snap.encoding.finish([un(p[0]), un(p[1]), un(p[2])]).map(|v| v.clamp(0.0, 1.0)); + out.extend([c[0], c[1], c[2], a].map(|v| (v * 255.0 + 0.5) as u8)); } Some((img.width, img.height, out)) } diff --git a/crates/engine/src/tests.rs b/crates/engine/src/tests.rs index 527381551..19953053e 100644 --- a/crates/engine/src/tests.rs +++ b/crates/engine/src/tests.rs @@ -11,6 +11,7 @@ mod color_view; mod crypto_pick; mod delete_keys; mod disk_cache; +mod display_pass; mod effects; mod essential; mod essential_more; diff --git a/crates/engine/src/tests/display_pass.rs b/crates/engine/src/tests/display_pass.rs new file mode 100644 index 000000000..8daeea136 --- /dev/null +++ b/crates/engine/src/tests/display_pass.rs @@ -0,0 +1,206 @@ +//! The viewer's display pass (`viewer::DisplayPass`): exposure on the float range of 32 bpc +//! frames, the HDR step after exposure, the display colour after both, and the frame encodings. + +use effectcraft_color::{ColorSpace, linear_to_srgb}; +use effectcraft_project::HdrMode; +use effectcraft_render::RenderOpts; +use effectcraft_render::color::{compand, tone_map}; +use effectcraft_time::Tick; +use serde_json::json; + +use crate::Session; +use crate::viewer::{Channel, DisplayColor, DisplayPass, FrameEncoding, PassCurve, SimProfile, Simulation}; + +const SRGB: FrameEncoding = FrameEncoding { curve: PassCurve::Srgb, hdr: HdrMode::Clip }; + +fn rgb(encoding: FrameEncoding, stops: f32) -> DisplayPass { + DisplayPass::new(Channel::Rgb, false, stops, encoding, None) +} + +/// An opaque grey of linear light `l`, sRGB-encoded as 32 bpc frames store it. +fn grey(l: f32) -> [f32; 4] { + let e = PassCurve::Srgb.encode(l); + [e, e, e, 1.0] +} + +fn q(v: f32) -> u8 { + (v.clamp(0.0, 1.0) * 255.0 + 0.5).floor() as u8 +} + +/// The issue's case: over-range values come back with negative exposure (4.0 at -2 EV is white) +/// and 0.5 at +1 EV is white; values of the 8-bit range keep today's bytes. +#[test] +fn exposure_reaches_over_range_values() { + let table: [(f32, [u8; 5]); 6] = [ + (0.25, [137, 99, 71, 34, 188]), + (0.5, [188, 137, 99, 49, 255]), + (1.0, [255, 188, 137, 71, 255]), + (2.0, [255, 255, 188, 99, 255]), + (4.0, [255, 255, 255, 137, 255]), + (16.0, [255, 255, 255, 255, 255]), + ]; + for (l, want) in table { + for (stops, w) in [0.0, -1.0, -2.0, -4.0, 1.0].into_iter().zip(want) { + assert_eq!(rgb(SRGB, stops).pixel(grey(l))[0], w, "linear {l} at {stops} EV"); + } + } + assert_eq!(rgb(SRGB, -2.0).pixel(grey(4.0)), [255, 255, 255, 255]); + assert_eq!(rgb(SRGB, 1.0).pixel(grey(0.5)), [255, 255, 255, 255]); + // Premultiplied: alpha stays, colour scales with it. + let e = PassCurve::Srgb.encode(4.0); + let half = rgb(SRGB, -2.0).pixel([e * 0.5, e * 0.5, e * 0.5, 0.5]); + assert!(half[3] == 128 && half[..3].iter().all(|c| (127..=128).contains(c)), "{half:?}"); +} + +/// Compand / Tone Map run after exposure: a value under the knee that exposure pushes over it +/// is companded (not clipped), and highlights darkened by exposure are not companded first. +#[test] +fn the_hdr_step_runs_after_exposure() { + let compand_enc = FrameEncoding { curve: PassCurve::Srgb, hdr: HdrMode::Compand }; + // 0.6 is below the 0.8 knee: unchanged at 0 EV, companded to about 0.97 at +1 EV. + assert_eq!(rgb(compand_enc, 0.0).pixel(grey(0.6))[0], q(linear_to_srgb(0.6))); + let up = rgb(compand_enc, 1.0).pixel(grey(0.6))[0]; + assert_eq!(up, q(linear_to_srgb(compand(1.2)))); + assert!(up < 255, "companded after exposure, not clipped: {up}"); + // 3.0 at -2 EV is 0.75, under the knee: compand first would have made it 0.25. + let down = rgb(compand_enc, -2.0).pixel(grey(3.0))[0]; + assert_eq!(down, q(linear_to_srgb(0.75))); + assert!(down > q(linear_to_srgb(compand(3.0) * 0.25)) + 50, "{down}"); + // Tone Map (luminance): the same order. + let tm = FrameEncoding { curve: PassCurve::Srgb, hdr: HdrMode::ToneMap }; + for (l, stops) in [(0.6f32, 1.0f32), (3.0, -2.0), (16.0, -1.0), (0.1, 0.0)] { + let k = 2f32.powf(stops); + let want = q(linear_to_srgb(tone_map([l * k; 3])[0])); + assert_eq!(rgb(tm, stops).pixel(grey(l))[0], want, "{l} at {stops}"); + } +} + +/// The display colour converts the exposed output numbers: exposure first, then the monitor +/// conversion (before: the conversion of the 8-bit frame, then exposure). +#[test] +fn display_colour_follows_exposure() { + let dc = + DisplayColor::new(Some(ColorSpace::Srgb), false, ColorSpace::Srgb, true, ColorSpace::DisplayP3, Simulation::default(), &Default::default()).unwrap(); + let pass = DisplayPass::new(Channel::Rgb, false, 1.0, SRGB, Some(&dc)); + let red = [PassCurve::Srgb.encode(0.4), 0.0, 0.0, 1.0]; + let exposed = [linear_to_srgb(0.8), 0.0, 0.0]; + let want = dc.apply_straight(exposed).map(q); + assert_eq!(pass.pixel(red)[..3], want); + // The conversion is not applied to a single channel's view. + assert!(DisplayPass::new(Channel::Red, false, 0.0, SRGB, Some(&dc)).color.is_none()); +} + +/// The pass's display colour is `DisplayColor::apply_straight`: the same conversions in the same +/// order (a few ULPs apart where the pass uses `f32` for PQ / HLG). +#[test] +fn pass_colour_matches_display_color() { + let custom = Default::default(); + let mut cases = vec![]; + for display in [ColorSpace::DisplayP3, ColorSpace::Rec709, ColorSpace::Rec2020, ColorSpace::Srgb] { + for profile in SimProfile::ALL { + for preserve_rgb in [false, true] { + let sim = Simulation { profile, preserve_rgb }; + cases.extend(DisplayColor::new(Some(ColorSpace::Srgb), false, ColorSpace::Srgb, true, display, sim, &custom)); + } + } + cases.extend(DisplayColor::new(Some(ColorSpace::Rec2020), true, ColorSpace::Srgb, false, display, Simulation::default(), &custom)); + } + // A LUT-based profile (an uneven, channel-mixing grid). + cases.push(DisplayColor::from_lut(std::sync::Arc::new(test_lut()))); + assert!(cases.len() > 40); + for dc in &cases { + let pass = dc.pass(); + for i in 0..=20 { + for c in [[i as f32 / 20.0, 0.3, 0.9], [0.05, i as f32 / 20.0, 0.5], [1.0, 1.0, i as f32 / 20.0]] { + let (a, b) = (pass.apply(c), dc.apply_straight(c)); + for k in 0..3 { + // The simulation's 8-bit step can land either side of a tie. + assert!((a[k] - b[k]).abs() < 1e-4 || ((a[k] - b[k]).abs() < 1.5 / 255.0 && c != [1.0; 3]), "{dc:?} {c:?}: {a:?} vs {b:?}"); + } + } + } + } +} + +/// A 17³ LUT that mixes channels and bends the curve (as an ICC profile's device does). +pub(crate) fn test_lut() -> effectcraft_color::icc::Lut3d { + effectcraft_color::icc::Lut3d::bake(17, |c| [c[0].powf(1.3) * 0.9 + c[1] * 0.1, c[1] * 0.8 + c[2] * 0.2, (c[2] * 0.7 + c[0] * 0.3).sqrt()]) +} + +/// Frames keep their encoding: the output space's curve (the Working Gamma's when unmanaged) and +/// the HDR step only when it was skipped and applies (managed, SDR output). +#[test] +fn frame_encodings() { + let mut s = Session::default(); + s.execute("comp.new", json!({"width": 8, "height": 8, "duration": 1})).unwrap(); + assert_eq!(FrameEncoding::of(&s.project.settings, true), SRGB, "unmanaged: sRGB at Working Gamma 2.2, no HDR step"); + s.execute("file.projectSettings", json!({"bitDepth": 32, "workingSpace": "srgb", "hdr": "compand"})).unwrap(); + let settings = s.project.settings.clone(); + assert_eq!(FrameEncoding::of(&settings, true), FrameEncoding { curve: PassCurve::Srgb, hdr: HdrMode::Compand }); + assert_eq!(FrameEncoding::of(&settings, false), SRGB, "rendered with the step: nothing pending"); + s.execute("file.projectSettings", json!({"outputSpace": "rec2100pq"})).unwrap(); + assert_eq!(FrameEncoding::of(&s.project.settings, true), FrameEncoding { curve: PassCurve::Pq, hdr: HdrMode::Clip }, "HDR outputs never tone map"); + s.execute("file.projectSettings", json!({"outputSpace": "rec709"})).unwrap(); + assert_eq!(FrameEncoding::of(&s.project.settings, true).curve, PassCurve::Gamma(2.4)); +} + +/// A 32 bpc Compand project with an over-range layer: `FrameEncoding::finish` on the frame +/// rendered without the HDR step gives the frame rendered with it, and Take Snapshot (now taken +/// without the step) exports the same bytes as before, within a rounding step. +#[test] +fn the_skipped_hdr_step_finishes_like_the_render() { + let mut s = Session::default(); + s.execute("comp.new", json!({"width": 16, "height": 8, "duration": 1})).unwrap(); + s.execute("file.projectSettings", json!({"bitDepth": 32, "workingSpace": "srgb", "hdr": "compand"})).unwrap(); + let lid = s.execute("layer.newSolid", json!({"color": "#c08040", "width": 16, "height": 8})).unwrap()["layer"].as_u64().unwrap(); + s.execute("effect.apply", json!({"layers": [lid], "effect": "Exposure"})).unwrap(); + s.execute("prop.set", json!({"layer": lid, "path": "effects/#1/master/exposure", "value": 1.5})).unwrap(); + let cid = s.active_comp_id().unwrap(); + let with = s.render(cid, Tick::ZERO, RenderOpts::default()); + let without = s.render(cid, Tick::ZERO, RenderOpts { skip_hdr_step: true, ..Default::default() }); + let enc = FrameEncoding::of(&s.project.settings, true); + assert_eq!(enc.hdr, HdrMode::Compand); + let (w, o) = (with.data[0], without.data[0]); + assert!(o[0] > 1.0 && w[0] < 1.0, "over range without the step, companded with it: {o:?} {w:?}"); + let f = enc.finish([o[0], o[1], o[2]]); + for k in 0..3 { + assert!((f[k] - w[k]).abs() < 1e-4, "{f:?} vs {w:?}"); + } + s.execute("view.takeSnapshot", json!({})).unwrap(); + assert_eq!(s.snapshot.as_ref().unwrap().encoding, enc); + let (_, _, rgba) = crate::commands::viewer_cmds::snapshot_rgba8(&s).unwrap(); + let before = w.map(|v| (v.clamp(0.0, 1.0) * 255.0 + 0.5) as u8); + for k in 0..4 { + assert!((rgba[k] as i32 - before[k] as i32).abs() <= 1, "{:?} vs {before:?}", &rgba[..4]); + } + // 8 bpc snapshots are rendered with the step, as before. + s.execute("file.projectSettings", json!({"bitDepth": 8})).unwrap(); + s.execute("view.takeSnapshot", json!({})).unwrap(); + assert_eq!(s.snapshot.as_ref().unwrap().encoding.hdr, HdrMode::Clip); +} + +/// The 8-bit CPU path's tables give what the pass gives pixel by pixel: every value of every +/// channel view, opaque and not, with and without exposure. +#[test] +fn the_8_bit_tables_are_the_pass() { + for (channel, colorized) in + [(Channel::Rgb, false), (Channel::RgbStraight, false), (Channel::Red, false), (Channel::Green, true), (Channel::Blue, false), (Channel::Alpha, false)] + { + for stops in [0.0f32, 1.0, -2.5] { + let pass = DisplayPass::new(channel, colorized, stops, SRGB, None); + let px: Vec<[u8; 4]> = (0..256 * 4) + .map(|i| { + let v = (i % 256) as u8; + let a = [255u8, 255, 128, 3][i / 256]; + let c = |k: u8| ((v as u32 * k as u32) % 256 * a as u32 / 255) as u8; + [c(1), c(7), c(13), a] + }) + .collect(); + let mut got = px.clone(); + pass.apply_rgba8(&mut got); + for (p, g) in px.iter().zip(&got) { + assert_eq!(*g, pass.pixel(p.map(|v| v as f32 / 255.0)), "{channel:?} {colorized} {stops}: {p:?}"); + } + } + } +} diff --git a/crates/engine/src/tests/viewer.rs b/crates/engine/src/tests/viewer.rs index 9a0b9cc58..31197b2a8 100644 --- a/crates/engine/src/tests/viewer.rs +++ b/crates/engine/src/tests/viewer.rs @@ -8,7 +8,7 @@ use effectcraft_time::Tick; use serde_json::json; use crate::Session; -use crate::viewer::{Channel, PRI_GRID, PRI_GUIDE, PRI_LAYER, PRI_VERTEX, SnapKind, SnapSource, SnapTarget, display_transform, snap}; +use crate::viewer::{Channel, DisplayPass, FrameEncoding, PRI_GRID, PRI_GUIDE, PRI_LAYER, PRI_VERTEX, PassCurve, SnapKind, SnapSource, SnapTarget, snap}; fn comp() -> Session { let mut s = Session::default(); @@ -225,15 +225,19 @@ fn snapping_options_command_sets_and_toggles() { // ---------------------------------------------------------------- channels, exposure, snapshot +/// The display pass on 8-bit frames (sRGB, no HDR step): Show Channel and exposure. +fn display_rgba8(px: [u8; 4], channel: Channel, colorized: bool, stops: f32) -> [u8; 4] { + let encoding = FrameEncoding { curve: PassCurve::Srgb, hdr: effectcraft_project::HdrMode::Clip }; + let mut v = [px]; + DisplayPass::new(channel, colorized, stops, encoding, None).apply_rgba8(&mut v); + v[0] +} + #[test] fn channel_and_exposure_pixel_transforms() { // Premultiplied: straight (200, 100, 50) at 50% alpha. let px = [100u8, 50, 25, 128]; - let run = |c: Channel, col: bool, stops: f32| { - let mut v = [px]; - display_transform(&mut v, c, col, stops); - v[0] - }; + let run = |c: Channel, col: bool, stops: f32| display_rgba8(px, c, col, stops); assert_eq!(run(Channel::Rgb, false, 0.0), px); assert_eq!(run(Channel::Red, false, 0.0), [100, 100, 100, 255]); assert_eq!(run(Channel::Green, true, 0.0), [0, 50, 0, 255]); @@ -242,13 +246,13 @@ fn channel_and_exposure_pixel_transforms() { let st = run(Channel::RgbStraight, false, 0.0); assert!((st[0] as i32 - 199).abs() <= 1 && (st[1] as i32 - 100).abs() <= 1 && st[3] == 255, "{st:?}"); // +1 stop doubles linear light: sRGB 50% gray → about 69%; -1 stop darkens; 0 is identity. - let mut g = [[128u8, 128, 128, 255]]; - display_transform(&mut g, Channel::Rgb, false, 1.0); - assert!((174..=178).contains(&g[0][0]), "{:?}", g[0]); - let mut g = [[128u8, 128, 128, 255]]; - display_transform(&mut g, Channel::Rgb, false, -1.0); - assert!((92..=96).contains(&g[0][0]), "{:?}", g[0]); - assert_eq!(crate::viewer::exposure_lut(0.0)[200], 200); + let g = display_rgba8([128, 128, 128, 255], Channel::Rgb, false, 1.0); + assert!((174..=178).contains(&g[0]), "{g:?}"); + let g = display_rgba8([128, 128, 128, 255], Channel::Rgb, false, -1.0); + assert!((92..=96).contains(&g[0]), "{g:?}"); + for v in 0..=255u8 { + assert_eq!(display_rgba8([v, v, v, 255], Channel::Rgb, false, 0.0), [v, v, v, 255]); + } // Exposure applies to a single channel too, and keeps alpha untouched in RGB. assert!(run(Channel::Red, false, 1.0)[0] > 100); assert_eq!(run(Channel::Rgb, false, 2.0)[3], 128); diff --git a/crates/engine/src/viewer.rs b/crates/engine/src/viewer.rs index 1889b6a83..f8e4c7740 100644 --- a/crates/engine/src/viewer.rs +++ b/crates/engine/src/viewer.rs @@ -1,5 +1,5 @@ //! Composition viewer maths shared by the frontends and tests: snapping (targets, tolerance and -//! priority), the Show Channel / Exposure display transforms, and the viewer snapshot. +//! priority), the display pass (Show Channel, exposure, display colour), and the viewer snapshot. //! //! Nothing here touches the project; edits go through commands. @@ -15,6 +15,9 @@ use effectcraft_render::{EvalCtx, ExprHost}; use effectcraft_time::Tick; use serde::{Deserialize, Serialize}; +mod display_pass; +pub use display_pass::{DisplayPass, FrameEncoding, PassColor, PassConv, PassCurve, SHADER_EXPONENTS, hdr_step, lut_eval}; + // ---------------------------------------------------------------- snapping /// What a snap target is: a point (snaps both axes) or an axis-aligned line. @@ -500,63 +503,6 @@ fn linear_to_srgb(v: f32) -> f32 { if v <= 0.003_130_8 { v * 12.92 } else { 1.055 * v.powf(1.0 / 2.4) - 0.055 } } -/// 8-bit exposure lookup table: `stops` applied in linear light. -pub fn exposure_lut(stops: f32) -> [u8; 256] { - let k = 2f32.powf(stops); - let mut lut = [0u8; 256]; - for (i, o) in lut.iter_mut().enumerate() { - let lin = srgb_to_linear(i as f32 / 255.0) * k; - *o = (linear_to_srgb(lin).clamp(0.0, 1.0) * 255.0 + 0.5) as u8; - } - lut -} - -/// Apply the viewer's channel and exposure to premultiplied 8-bit RGBA pixels in place. -/// RGB shows the image (exposure only); Red/Green/Blue/Alpha show one channel as an opaque -/// grayscale image (or tinted in that channel's colour when `colorized`); RGB Straight shows the -/// unpremultiplied colour, opaque. -pub fn display_transform(px: &mut [[u8; 4]], channel: Channel, colorized: bool, stops: f32) { - let lut = (stops != 0.0).then(|| exposure_lut(stops)); - let ex = |v: u8| lut.as_ref().map(|l| l[v as usize]).unwrap_or(v); - for p in px.iter_mut() { - let [r, g, b, a] = *p; - *p = match channel { - Channel::Rgb => { - if lut.is_none() { - continue; - } - // Exposure on straight colour, re-premultiplied. - let un = |c: u8| if a == 0 { 0 } else { ((c as u32 * 255 + a as u32 / 2) / a as u32).min(255) as u8 }; - let pm = |c: u8| ((ex(un(c)) as u32 * a as u32 + 127) / 255) as u8; - [pm(r), pm(g), pm(b), a] - } - Channel::Red | Channel::Green | Channel::Blue => { - let (i, c) = match channel { - Channel::Red => (0, r), - Channel::Green => (1, g), - _ => (2, b), - }; - let v = ex(c); - if colorized { - let mut o = [0, 0, 0, 255]; - o[i] = v; - o - } else { - [v, v, v, 255] - } - } - Channel::Alpha => { - let v = ex(a); - [v, v, v, 255] - } - Channel::RgbStraight => { - let un = |c: u8| if a == 0 { 0 } else { ((c as u32 * 255 + a as u32 / 2) / a as u32).min(255) as u8 }; - [ex(un(r)), ex(un(g)), ex(un(b)), 255] - } - }; - } -} - // ---------------------------------------------------------------- display colour management /// View ▸ Simulate Output: the output device a preview imitates. @@ -999,6 +945,12 @@ impl DisplayColor { ) } + /// A conversion that is a baked 3D LUT alone, as a LUT-based My Custom RGB profile makes + /// (tests of the display pass). + pub fn from_lut(lut: Arc) -> DisplayColor { + DisplayColor { sim: None, display: None, lut: Some(lut) } + } + /// Whether the simulation runs a LUT-based ICC profile (baked into a 3D LUT). pub fn is_lut(&self) -> bool { self.lut.is_some() @@ -1097,6 +1049,9 @@ pub struct Snapshot { /// Render scale the image was taken at. pub scale: f64, pub image: Arc, + /// How `image` is encoded: rendered without the HDR step in 32 bpc, so viewer exposure + /// reaches its over-range values; the display pass applies the step. + pub encoding: FrameEncoding, } #[cfg(test)] diff --git a/crates/engine/src/viewer/display_pass.rs b/crates/engine/src/viewer/display_pass.rs new file mode 100644 index 000000000..f3daf579c --- /dev/null +++ b/crates/engine/src/viewer/display_pass.rs @@ -0,0 +1,404 @@ +//! The viewer's display pass as plain data: what turns a rendered frame into the pixels the +//! Composition panel shows. The frontend runs it in a shader. The same steps on the CPU +//! ([`DisplayPass::pixel`]) are its fallback without a GPU and the reference the shader is tested +//! against, so both use the same `f32` operations in the same order. +//! +//! Steps, per pixel of a premultiplied frame encoded in the output space: +//! 1. Unpremultiply (NaN alpha counts as 0, NaN colour as 0). +//! 2. When exposure or an HDR step applies: decode with the frame's curve, multiply by 2^stops +//! in linear light, run the pending HDR step (Project Settings ▸ HDR: Compand / Tone Map, for +//! frames rendered without it), encode again. +//! 3. Clamp to 0..1: the output's numbers. +//! 4. The display colour conversion (Use Display Color Management, Simulate Output), on RGB and +//! RGB Straight only. +//! 5. Show Channel, premultiply (without step 2 or a display conversion: the frame's own +//! premultiplied values, clamped to alpha), and round to 8 bits with `floor(v · 255 + 0.5)`. +//! +//! Exposure therefore acts on the full float range of 32 bpc frames, before tone mapping and +//! before the display conversion. With neither exposure nor an HDR step, step 2 is skipped, so +//! 8-bit frames pass through unchanged. + +use std::sync::Arc; + +use effectcraft_color::icc::Lut3d; +use effectcraft_color::space; +use effectcraft_project::{HdrMode, ProjectSettings}; +use effectcraft_render::color::Pipe; + +use super::{Channel, Conv, Curve, DisplayColor}; + +/// A transfer curve of the display pass. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum PassCurve { + Linear, + Srgb, + /// A pure power curve (BT.1886 2.4, Mac 1.8, custom gammas). + Gamma(f32), + Pq, + Hlg, +} + +/// SMPTE ST 2084 and BT.2100 HLG constants (`effectcraft_color::space`), in `f32`. +const PQ_M1: f32 = 2610.0 / 16384.0; +const PQ_M2: f32 = 2523.0 / 4096.0 * 128.0; +const PQ_C1: f32 = 3424.0 / 4096.0; +const PQ_C2: f32 = 2413.0 / 4096.0 * 32.0; +const PQ_C3: f32 = 2392.0 / 4096.0 * 32.0; +/// Linear 1.0 as a share of PQ's 10 000 cd/m² (the 203 cd/m² reference white). +const PQ_WHITE: f32 = 203.0 / 10_000.0; +const HLG_A: f32 = 0.178_832_77; +const HLG_B: f32 = 0.284_668_92; +const HLG_C: f32 = 0.559_910_7; +const HLG_REFERENCE: f32 = 0.264_962_56; + +/// The exponents the shader is handed (`vals2.yzw`): 1 / 2.4, 1 / PQ m2, 1 / PQ m1. +pub const SHADER_EXPONENTS: [f32; 3] = [1.0 / 2.4, 1.0 / PQ_M2, 1.0 / PQ_M1]; + +/// `a^g` for `a ≥ 0`, 0 at 0 (WGSL leaves `pow(0, g)` to the implementation). +fn power(a: f32, g: f32) -> f32 { + if a <= 0.0 { 0.0 } else { a.powf(g) } +} + +impl PassCurve { + /// The curve a colour space encodes with. + pub fn of(curve: space::Curve) -> PassCurve { + match curve { + space::Curve::Linear => PassCurve::Linear, + space::Curve::Srgb => PassCurve::Srgb, + space::Curve::Gamma24 => PassCurve::Gamma(2.4), + space::Curve::Pq => PassCurve::Pq, + space::Curve::Hlg => PassCurve::Hlg, + } + } + + fn of_viewer(c: Curve) -> PassCurve { + match c { + Curve::Srgb => PassCurve::Srgb, + Curve::Gamma(g) => PassCurve::Gamma(g), + Curve::Linear => PassCurve::Linear, + Curve::Space(c) => PassCurve::of(c), + } + } + + /// The shader's curve id. + pub fn id(self) -> u32 { + match self { + PassCurve::Linear => 0, + PassCurve::Srgb => 1, + PassCurve::Gamma(_) => 2, + PassCurve::Pq => 3, + PassCurve::Hlg => 4, + } + } + + /// The shader's parameter to decode with: the gamma (1 for other curves). + pub fn decode_param(self) -> f32 { + if let PassCurve::Gamma(g) = self { g } else { 1.0 } + } + + /// The shader's parameter to encode with: 1 / the gamma, as [`PassCurve::encode`] divides. + pub fn encode_param(self) -> f32 { + if let PassCurve::Gamma(g) = self { 1.0 / g } else { 1.0 } + } + + /// Encoded → linear light, mirrored around zero. + pub fn decode(self, v: f32) -> f32 { + let a = v.abs(); + let l = match self { + PassCurve::Linear => a, + PassCurve::Srgb => { + if a <= 0.04045 { + a / 12.92 + } else { + power((a + 0.055) / 1.055, 2.4) + } + } + PassCurve::Gamma(g) => power(a, g), + PassCurve::Pq => { + let p = power(a.min(1.0), 1.0 / PQ_M2); + power((p - PQ_C1).max(0.0) / (PQ_C2 - PQ_C3 * p), 1.0 / PQ_M1) / PQ_WHITE + } + PassCurve::Hlg => { + let e = if a <= 0.5 { a * a / 3.0 } else { (((a - HLG_C) / HLG_A).exp() + HLG_B) / 12.0 }; + e / HLG_REFERENCE + } + }; + l.copysign(v) + } + + /// Linear light → encoded, mirrored around zero. + pub fn encode(self, v: f32) -> f32 { + let a = v.abs(); + let e = match self { + PassCurve::Linear => a, + PassCurve::Srgb => { + if a <= 0.003_130_8 { + a * 12.92 + } else { + 1.055 * power(a, 1.0 / 2.4) - 0.055 + } + } + PassCurve::Gamma(g) => power(a, 1.0 / g), + PassCurve::Pq => { + let y = power(a * PQ_WHITE, PQ_M1); + power((PQ_C1 + PQ_C2 * y) / (1.0 + PQ_C3 * y), PQ_M2) + } + PassCurve::Hlg => { + let e = a * HLG_REFERENCE; + if e <= 1.0 / 12.0 { (3.0 * e).sqrt() } else { HLG_A * (12.0 * e - HLG_B).ln() + HLG_C } + } + }; + e.copysign(v) + } +} + +/// A conversion of the display colour: decode, an optional linear-light matrix, encode. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct PassConv { + pub from: PassCurve, + /// Rows of the linear RGB matrix (`None`: the same primaries). + pub matrix: Option<[[f32; 3]; 3]>, + pub to: PassCurve, +} + +impl PassConv { + fn of(c: &Conv) -> PassConv { + PassConv { from: PassCurve::of_viewer(c.from), matrix: c.m, to: PassCurve::of_viewer(c.to) } + } + + pub fn apply(&self, c: [f32; 3]) -> [f32; 3] { + let l = c.map(|v| self.from.decode(v)); + let l = match &self.matrix { + Some(m) => [0, 1, 2].map(|i| m[i][0] * l[0] + m[i][1] * l[1] + m[i][2] * l[2]), + None => l, + }; + l.map(|v| self.to.encode(v)) + } +} + +/// The display colour conversion ([`DisplayColor`]) as data for the display pass. +#[derive(Clone, Debug)] +pub enum PassColor { + /// One conversion; with Simulate Output, the conversion to the simulated device, its 8-bit + /// numbers (`quantize`) and the conversion from the device to the display. + Convs { first: PassConv, quantize: bool, second: Option }, + /// A LUT-based My Custom RGB profile: the whole simulation baked into a 3D LUT. + Lut(Arc), +} + +impl PassColor { + /// One straight colour (0..1) to the display's numbers (0..1). + pub fn apply(&self, c: [f32; 3]) -> [f32; 3] { + let o = match self { + PassColor::Lut(l) => lut_eval(l, c), + PassColor::Convs { first, quantize, second } => { + let mut o = first.apply(c); + if *quantize { + o = o.map(|v| (v.clamp(0.0, 1.0) * 255.0 + 0.5).floor() / 255.0); + } + match second { + Some(s) => s.apply(o), + None => o, + } + } + }; + o.map(|v| v.clamp(0.0, 1.0)) + } +} + +/// [`Lut3d::eval`], written as the shader samples the LUT (the same products and sums). +pub fn lut_eval(l: &Lut3d, c: [f32; 3]) -> [f32; 3] { + let n = l.size.max(2); + let mut k = [0usize; 3]; + let mut f = [0f32; 3]; + for i in 0..3 { + let x = c[i].clamp(0.0, 1.0) * (n - 1) as f32; + k[i] = (x.floor() as usize).min(n - 2); + f[i] = x - k[i] as f32; + } + let at = |r: usize, g: usize, b: usize| l.data.get((r * n + g) * n + b).copied().unwrap_or_default(); + let mut out = [0f32; 3]; + for corner in 0..8 { + let (dr, dg, db) = (corner >> 2 & 1, corner >> 1 & 1, corner & 1); + let w = (if dr == 1 { f[0] } else { 1.0 - f[0] }) * (if dg == 1 { f[1] } else { 1.0 - f[1] }) * (if db == 1 { f[2] } else { 1.0 - f[2] }); + let v = at(k[0] + dr, k[1] + dg, k[2] + db); + for i in 0..3 { + out[i] += w * v[i]; + } + } + out +} + +impl DisplayColor { + /// The conversion as data for the display pass. + pub fn pass(&self) -> PassColor { + if let Some(l) = &self.lut { + return PassColor::Lut(l.clone()); + } + match (&self.sim, &self.display) { + (Some((to, show)), _) => PassColor::Convs { first: PassConv::of(to), quantize: true, second: show.as_ref().map(PassConv::of) }, + (None, Some(d)) => PassColor::Convs { first: PassConv::of(d), quantize: false, second: None }, + (None, None) => { + PassColor::Convs { first: PassConv { from: PassCurve::Linear, matrix: None, to: PassCurve::Linear }, quantize: false, second: None } + } + } + } +} + +/// How a rendered frame's pixels are encoded: the curve of its output space, and the HDR step +/// (Project Settings ▸ HDR) it was rendered without and the display pass still runs (`Clip`: +/// none). Kept with the frame, so a later settings change doesn't reinterpret older frames. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct FrameEncoding { + pub curve: PassCurve, + pub hdr: HdrMode, +} + +impl FrameEncoding { + /// A frame of a project with `settings`; `skipped_hdr_step`: rendered with + /// `RenderOpts::skip_hdr_step`. + pub fn of(settings: &ProjectSettings, skipped_hdr_step: bool) -> FrameEncoding { + let pipe = Pipe::of(settings); + let curve = pipe.output_linear().decode.map_or(PassCurve::Linear, |s| PassCurve::of(s.curve())); + let hdr = if skipped_hdr_step && pipe.output_hdr().is_some() { settings.hdr } else { HdrMode::Clip }; + FrameEncoding { curve, hdr } + } + + /// The frame as it would have been rendered with the HDR step: one straight colour. + pub fn finish(&self, c: [f32; 3]) -> [f32; 3] { + if self.hdr == HdrMode::Clip { + return c; + } + let l = hdr_step(self.hdr, c.map(|v| self.curve.decode(v))); + l.map(|v| self.curve.encode(v)) + } +} + +/// Project Settings ▸ HDR on linear light (`effectcraft_render::color::output_hdr`). +pub fn hdr_step(mode: HdrMode, l: [f32; 3]) -> [f32; 3] { + match mode { + HdrMode::Clip => l, + HdrMode::Compand => l.map(effectcraft_render::color::compand), + HdrMode::ToneMap => effectcraft_render::color::tone_map(l), + } +} + +/// The whole display pass of one viewer: Show Channel, exposure, the frame's encoding and the +/// display colour conversion. +#[derive(Clone, Debug)] +pub struct DisplayPass { + pub channel: Channel, + pub colorized: bool, + /// 2^stops of the viewer's exposure. + pub gain: f32, + pub encoding: FrameEncoding, + pub color: Option, +} + +impl DisplayPass { + /// `stops` of exposure on frames encoded as `encoding`. + pub fn new(channel: Channel, colorized: bool, stops: f32, encoding: FrameEncoding, color: Option<&DisplayColor>) -> DisplayPass { + let gain = if stops == 0.0 || !stops.is_finite() { 1.0 } else { 2f32.powf(stops) }; + // The display conversion applies to the colour views only (as before). + let color = color.filter(|_| matches!(channel, Channel::Rgb | Channel::RgbStraight)).map(DisplayColor::pass); + DisplayPass { channel, colorized, gain, encoding, color } + } + + /// Step 2 of the module docs runs. + pub fn changes_light(&self) -> bool { + self.gain != 1.0 || self.encoding.hdr != HdrMode::Clip + } + + /// Nothing changes the frame's pixels (beyond clamping and rounding). + pub fn is_identity(&self) -> bool { + self.channel == Channel::Rgb && !self.changes_light() && self.color.is_none() + } + + /// Step 2: exposure and the pending HDR step on straight colour. + fn light(&self, c: [f32; 3]) -> [f32; 3] { + if !self.changes_light() { + return c; + } + let curve = self.encoding.curve; + let l = hdr_step(self.encoding.hdr, c.map(|v| curve.decode(v) * self.gain)); + l.map(|v| curve.encode(v)) + } + + /// One premultiplied pixel (encoded in the output space) → premultiplied RGBA8 on screen. + pub fn pixel(&self, p: [f32; 4]) -> [u8; 4] { + let q = |v: f32| (v.clamp(0.0, 1.0) * 255.0 + 0.5).floor() as u8; + let a = if p[3].is_nan() { 0.0 } else { p[3].clamp(0.0, 1.0) }; + let un = |v: f32| if a > 0.0 && !v.is_nan() { v / a } else { 0.0 }; + let c = self.light([un(p[0]), un(p[1]), un(p[2])]).map(|v| v.clamp(0.0, 1.0)); + // A premultiplied channel: without exposure or HDR step straight from the frame (clamped + // to alpha, as `display_pixel`), so nothing changes it by a division's rounding. + let premul = |i: usize| if self.changes_light() || p[i].is_nan() { c[i] * a } else { p[i].clamp(0.0, a) }; + match self.channel { + Channel::Rgb | Channel::RgbStraight => { + let c = match &self.color { + Some(dc) => dc.apply(c), + None => c, + }; + if self.channel == Channel::RgbStraight { + [q(c[0]), q(c[1]), q(c[2]), 255] + } else if self.color.is_none() { + [q(premul(0)), q(premul(1)), q(premul(2)), q(a)] + } else { + [q(c[0] * a), q(c[1] * a), q(c[2] * a), q(a)] + } + } + Channel::Red | Channel::Green | Channel::Blue => { + let i = match self.channel { + Channel::Red => 0, + Channel::Green => 1, + _ => 2, + }; + let v = q(premul(i)); + if self.colorized { + let mut o = [0, 0, 0, 255]; + o[i] = v; + o + } else { + [v, v, v, 255] + } + } + Channel::Alpha => { + // Exposure lightens or darkens the alpha view as an encoded grey. + let v = if self.gain != 1.0 { self.encoding.curve.encode(self.encoding.curve.decode(a) * self.gain) } else { a }; + let v = q(v); + [v, v, v, 255] + } + } + } + + /// The pass on premultiplied 8-bit pixels in place (the CPU path for 8-bit frames). Without + /// a display conversion or HDR step each channel of an opaque pixel goes through the same + /// curve on its own, so those pixels (and the Alpha view, which reads alpha only) look the + /// result up in a table of [`DisplayPass::pixel`] (the browser has no threads to share + /// the per-pixel work). + pub fn apply_rgba8(&self, px: &mut [[u8; 4]]) { + use rayon::prelude::*; + if self.is_identity() { + return; + } + let f = |v: u8| v as f32 / 255.0; + if self.channel == Channel::Alpha { + let t: [[u8; 4]; 256] = std::array::from_fn(|a| self.pixel([0.0, 0.0, 0.0, f(a as u8)])); + px.par_iter_mut().for_each(|p| *p = t[p[3] as usize]); + return; + } + let table = (self.color.is_none() && self.encoding.hdr == HdrMode::Clip) + .then(|| -> [[u8; 4]; 256] { std::array::from_fn(|v| self.pixel([f(v as u8), f(v as u8), f(v as u8), 1.0])) }); + px.par_iter_mut().for_each(|p| { + *p = match &table { + Some(t) if p[3] == 255 => match self.channel { + Channel::Rgb | Channel::RgbStraight => [t[p[0] as usize][0], t[p[1] as usize][1], t[p[2] as usize][2], 255], + Channel::Red => t[p[0] as usize], + Channel::Green => t[p[1] as usize], + _ => t[p[2] as usize], + }, + _ => self.pixel(p.map(f)), + } + }); + } +} diff --git a/crates/render/src/disk_cache.rs b/crates/render/src/disk_cache.rs index ef41fdf66..427198ad2 100644 --- a/crates/render/src/disk_cache.rs +++ b/crates/render/src/disk_cache.rs @@ -6,7 +6,9 @@ //! files' size and modification time included) plus frame, scale and view; layer buffers by //! the layer cache's content key salted with the project's footage fingerprint. A changed //! project therefore never reads stale pixels: its keys simply change. -//! * Payloads are LZ4-compressed (`lz4_flex`, MIT): frames as 8-bit premultiplied RGBA, layer +//! * Payloads are LZ4-compressed (`lz4_flex`, MIT): frames as 8-bit premultiplied RGBA (or, for +//! 32 bpc viewer frames, half-float RGBA with the bytes shuffled into planes; the entry +//! header's format byte tells them apart, 8-bit entries keep format 0), layer //! buffers as `f32` RGBA with the bytes shuffled into planes (exact, so cached and fresh //! renders are bit-identical), Roto Brush / Object Matte segmentations as //! `FrameSeg::to_bytes` (keyed by their content-keyed chain key and scale, so they are @@ -76,6 +78,19 @@ pub struct Frame8 { pub rgba: Vec, } +/// A cached 32 bpc viewer frame: premultiplied RGBA half floats (IEEE binary16 bits), four +/// per pixel. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FrameF16 { + pub width: u32, + pub height: u32, + pub rgba: Vec, +} + +/// The entry header's format byte of frame entries. +const FORMAT_RGBA8: u8 = 0; +const FORMAT_F16: u8 = 1; + /// Counters for tests, the Info panel and `cache.diskStats`. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub struct DiskStats { @@ -226,7 +241,8 @@ impl LayerPrefetch { #[cfg_attr(target_arch = "wasm32", allow(dead_code))] enum Job { - Write(Kind, u128, Vec), + /// Kind, key, header format byte, payload. + Write(Kind, u128, u8, Vec), } struct Queue { @@ -245,7 +261,7 @@ impl Queue { #[cfg(not(target_arch = "wasm32"))] fn pop(&mut self) -> Option { if let Some(j) = self.frames.pop_front() { - let Job::Write(_, _, p) = &j; + let Job::Write(_, _, _, p) = &j; self.frame_bytes = self.frame_bytes.saturating_sub(p.len()); return Some(j); } @@ -520,16 +536,21 @@ impl DiskCache { } fn read(&self, kind: Kind, key: u128) -> Option> { + self.read_entry(kind, key).map(|(_, p)| p) + } + + /// An entry's payload and its header's format byte. + fn read_entry(&self, kind: Kind, key: u128) -> Option<(u8, Vec)> { if !self.contains(kind, key) { self.misses.fetch_add(1, Ordering::Relaxed); return None; } let path = self.path(kind, key); let data = std::fs::read(&path).ok(); - let ok = data.as_deref().and_then(|d| verify(d, kind)); + let ok = data.as_deref().and_then(|d| verify(d, kind).map(|p| (d[5], p))); match ok { - Some(payload) => { - let payload = payload.to_vec(); + Some((format, payload)) => { + let payload = (format, payload.to_vec()); self.hits.fetch_add(1, Ordering::Relaxed); if let Ok(mut g) = self.index.lock() { g.touch(kind, key); @@ -552,6 +573,12 @@ impl DiskCache { } } + /// A valid file whose payload does not decode: delete it, as a damaged one. + fn discard(&self, kind: Kind, key: u128) { + let _ = std::fs::remove_file(self.path(kind, key)); + self.forget(kind, key); + } + fn forget(&self, kind: Kind, key: u128) { if let Ok(mut g) = self.index.lock() { g.remove(kind, key); @@ -560,9 +587,14 @@ impl DiskCache { /// Queue a write; `false` when it was dropped (a full queue). fn enqueue(&self, kind: Kind, key: u128, payload: Vec) -> bool { + self.enqueue_format(kind, key, FORMAT_RGBA8, payload) + } + + /// [`DiskCache::enqueue`] with the entry's header format byte. + fn enqueue_format(&self, kind: Kind, key: u128, format: u8, payload: Vec) -> bool { #[cfg(target_arch = "wasm32")] { - self.write_now(kind, key, &payload); + self.write_format(kind, key, format, &payload); true } #[cfg(not(target_arch = "wasm32"))] @@ -580,9 +612,9 @@ impl DiskCache { if kind == Kind::Frame { q.pending_frames.insert(key); q.frame_bytes = q.frame_bytes.saturating_add(payload.len()); - q.frames.push_back(Job::Write(kind, key, payload)); + q.frames.push_back(Job::Write(kind, key, format, payload)); } else { - q.jobs.push_back(Job::Write(kind, key, payload)); + q.jobs.push_back(Job::Write(kind, key, format, payload)); } cv.notify_all(); true @@ -591,6 +623,10 @@ impl DiskCache { /// Write an entry now (atomic rename), then evict down to the limit. fn write_now(&self, kind: Kind, key: u128, payload: &[u8]) { + self.write_format(kind, key, FORMAT_RGBA8, payload); + } + + fn write_format(&self, kind: Kind, key: u128, format: u8, payload: &[u8]) { if self.contains(kind, key) { return; } @@ -600,7 +636,7 @@ impl DiskCache { self.write_failed(&e); return; } - let data = seal(kind, payload); + let data = seal_format(kind, format, payload); // An entry that cannot fit would be evicted straight away. if data.len() as u64 > self.limit_bytes() { return; @@ -646,11 +682,43 @@ impl DiskCache { // ------------------------------------------------------------ typed entries + /// An 8-bit frame (`None` for a half-float entry of that key, which [`DiskCache::get_frame_f16`] + /// reads). pub fn get_frame(&self, key: u128) -> Option { - let p = self.read(Kind::Frame, key)?; + let (format, p) = self.read_entry(Kind::Frame, key)?; + if format != FORMAT_RGBA8 { + return None; + } decode_frame(&p) } + /// A half-float frame (32 bpc viewer frames). A damaged one is deleted. + pub fn get_frame_f16(&self, key: u128) -> Option { + let (format, p) = self.read_entry(Kind::Frame, key)?; + if format != FORMAT_F16 { + return None; + } + let f = decode_frame_f16(&p); + if f.is_none() { + self.discard(Kind::Frame, key); + } + f + } + + /// Queue a half-float frame for writing (`width * height * 4` binary16 values). `true` when + /// it is on disk or queued; `false` when refused (wrong size, a full queue). + pub fn put_frame_f16(&self, key: u128, width: u32, height: u32, rgba: &[u16]) -> bool { + if rgba.len() as u64 != width as u64 * height as u64 * 4 { + return false; + } + self.contains(Kind::Frame, key) || self.enqueue_format(Kind::Frame, key, FORMAT_F16, encode_frame_f16(width, height, rgba)) + } + + /// Write a half-float frame synchronously (tests). + pub fn put_frame_f16_now(&self, key: u128, width: u32, height: u32, rgba: &[u16]) { + self.write_format(Kind::Frame, key, FORMAT_F16, &encode_frame_f16(width, height, rgba)); + } + /// Queue a frame for writing (8-bit premultiplied RGBA, `width * height * 4` bytes). `true` /// when it is on disk or queued; `false` when refused (wrong size, a full queue). pub fn put_frame(&self, key: u128, width: u32, height: u32, rgba: &[u8]) -> bool { @@ -762,8 +830,8 @@ fn writer(dc: std::sync::Weak, q: Arc<(Mutex, Condvar)>) { } }; let Some(cache) = dc.upgrade() else { return }; - let Job::Write(kind, key, payload) = job; - cache.write_now(kind, key, &payload); + let Job::Write(kind, key, format, payload) = job; + cache.write_format(kind, key, format, &payload); drop(cache); if let Ok(mut g) = m.lock() { g.busy = false; @@ -775,12 +843,17 @@ fn writer(dc: std::sync::Weak, q: Arc<(Mutex, Condvar)>) { } } -/// An entry file's bytes: header (magic, kind), payload, checksum. +/// An entry file's bytes: header (magic, kind, format 0), payload, checksum. pub fn seal(kind: Kind, payload: &[u8]) -> Vec { + seal_format(kind, FORMAT_RGBA8, payload) +} + +/// [`seal`] with the header's format byte (frames: 0 = 8-bit, 1 = half float). +fn seal_format(kind: Kind, format: u8, payload: &[u8]) -> Vec { let mut data = Vec::with_capacity(payload.len() + 16); data.extend_from_slice(MAGIC); data.push(kind.tag()); - data.extend_from_slice(&[0, 0, 0]); + data.extend_from_slice(&[format, 0, 0]); data.extend_from_slice(payload); let sum = checksum(&data); data.extend_from_slice(&sum.to_le_bytes()); @@ -879,6 +952,46 @@ fn decode_frame(p: &[u8]) -> Option { (rgba.len() == w as usize * h as usize * 4).then_some(Frame8 { width: w, height: h, rgba }) } +/// Half floats with their low and high bytes in planes (per channel), then LZ4: the high bytes +/// (sign, exponent) repeat a lot, mixed with the low ones LZ4 finds little. +fn encode_frame_f16(w: u32, h: u32, rgba: &[u16]) -> Vec { + use rayon::prelude::*; + let n = rgba.len() / 4; + let mut planes = vec![0u8; n * 8]; + // Plane (byte k, channel c) at (k · 4 + c) · n. + planes.par_chunks_mut(n.max(1)).enumerate().for_each(|(plane, out)| { + let (k, c) = (plane / 4, plane % 4); + for (o, px) in out.iter_mut().zip(rgba.as_chunks::<4>().0) { + *o = px[c].to_le_bytes()[k]; + } + }); + let mut out = Vec::with_capacity(planes.len() / 3 + 16); + out.extend_from_slice(&w.to_le_bytes()); + out.extend_from_slice(&h.to_le_bytes()); + out.extend_from_slice(&lz4_flex::compress_prepend_size(&planes)); + out +} + +fn decode_frame_f16(p: &[u8]) -> Option { + if p.len() < 8 { + return None; + } + let w = u32::from_le_bytes(p[0..4].try_into().ok()?); + let h = u32::from_le_bytes(p[4..8].try_into().ok()?); + let n = (w as usize).checked_mul(h as usize)?; + let planes = lz4_flex::decompress_size_prepended(&p[8..]).ok()?; + if planes.len() != n.checked_mul(8)? { + return None; + } + let mut rgba = vec![0u16; n * 4]; + for (i, px) in rgba.as_chunks_mut::<4>().0.iter_mut().enumerate() { + for (c, v) in px.iter_mut().enumerate() { + *v = u16::from_le_bytes([planes[c * n + i], planes[(4 + c) * n + i]]); + } + } + Some(FrameF16 { width: w, height: h, rgba }) +} + fn encode_layer(b: &Buf) -> Vec { let (w, h) = (b.img.width, b.img.height); let n = b.img.data.len(); @@ -1175,6 +1288,42 @@ mod tests { assert_eq!(parse_entry_name(&format!("{key:032x}.tmp.1")), None); } + /// Half-float frames (32 bpc viewer frames) round-trip bit for bit, over-range values, + /// negatives and odd sizes included; they and 8-bit frames never read as each other; a + /// damaged half-float entry is a miss and is deleted; they count toward the size limit. + #[test] + fn half_float_frames_round_trip() { + let dir = tmpdir("f16"); + let dc = DiskCache::open(&dir, 1 << 30).unwrap(); + let (w, h) = (7u32, 5u32); + // 4.0, 16.0, -0.25, 65504 and a spread of bit patterns. + let rgba: Vec = (0..w * h * 4).map(|i| [0x4400, 0x4c00, 0xb400, 0x7bff][i as usize % 4] ^ ((i * 37) as u16 & 0x03ff)).collect(); + dc.put_frame_f16_now(1, w, h, &rgba); + assert_eq!(dc.get_frame_f16(1), Some(FrameF16 { width: w, height: h, rgba: rgba.clone() })); + assert!(dc.get_frame(1).is_none(), "not an 8-bit frame"); + dc.put_frame_now(2, 2, 2, &[9; 16]); + assert!(dc.get_frame_f16(2).is_none() && dc.contains(Kind::Frame, 2), "an 8-bit entry stays valid"); + assert_eq!(dc.get_frame(2).unwrap().rgba, vec![9; 16]); + assert!(dc.stats().bytes > 0); + // Reopened, still there (format byte persisted). + drop(dc); + let dc = DiskCache::open(&dir, 1 << 30).unwrap(); + assert_eq!(dc.get_frame_f16(1).unwrap().rgba, rgba); + // A sealed entry with a payload of the wrong size: deleted on read. + let p = dc.path(Kind::Frame, 3); + std::fs::create_dir_all(p.parent().unwrap()).unwrap(); + std::fs::write(&p, seal_format(Kind::Frame, FORMAT_F16, &encode_frame_f16(w, h + 1, &rgba[..])[..])).unwrap(); + dc.rescan(); + assert!(dc.contains(Kind::Frame, 3)); + assert!(dc.get_frame_f16(3).is_none()); + assert!(!dc.contains(Kind::Frame, 3) && !p.exists()); + // Wrong lengths are not written. + dc.put_frame_f16(4, w, h, &rgba[1..]); + dc.flush(); + assert!(!dc.contains(Kind::Frame, 4)); + let _ = std::fs::remove_dir_all(&dir); + } + /// Segmentations round-trip by (chain key, scale), survive a reopen (rescan), are bounded by /// the size limit like every entry, and a damaged file is a miss. #[test] diff --git a/crates/ui-egui/Cargo.toml b/crates/ui-egui/Cargo.toml index 59d437b26..1576907cb 100644 --- a/crates/ui-egui/Cargo.toml +++ b/crates/ui-egui/Cargo.toml @@ -33,11 +33,16 @@ rayon = { workspace = true } web-time = { workspace = true } kurbo = { workspace = true } image = { workspace = true } +# Half-float viewer frames of 32 bpc projects (F16C is picked at run time). +half = { workspace = true } +bytemuck = { workspace = true } [target.'cfg(not(target_arch = "wasm32"))'.dependencies] # Floating panels open in OS windows of their own on desktop; the web keeps them in the page. craft-ui = { workspace = true, features = ["native-windows"] } craft-update = { workspace = true } +# Waits for the display pass's error scopes (immediately ready on native wgpu). +pollster = { workspace = true } sys-locale = { workspace = true } [dev-dependencies] @@ -50,6 +55,7 @@ effectcraft-segment = { workspace = true } effectcraft-host = { workspace = true } effectcraft-media = { workspace = true } exr = { workspace = true } +tempfile = { workspace = true } [build-dependencies] serde_json = { workspace = true } diff --git a/crates/ui-egui/src/app.rs b/crates/ui-egui/src/app.rs index 9b0e1583e..9d32ba4cd 100644 --- a/crates/ui-egui/src/app.rs +++ b/crates/ui-egui/src/app.rs @@ -79,6 +79,13 @@ impl EffectcraftApp { viewer_shown: None, crypto_probe_busy: Default::default(), viewer_image: None, + viewer_f16: None, + window_wgpu: None, + display_gpu: None, + display_gpu_failed: false, + viewer_bake: Default::default(), + snapshot_bake: Default::default(), + snapshot_f16: None, viewer_readback: Default::default(), wgpu: None, gpu: None, diff --git a/crates/ui-egui/src/app/gpu.rs b/crates/ui-egui/src/app/gpu.rs index 225a8727c..c9849e921 100644 --- a/crates/ui-egui/src/app/gpu.rs +++ b/crates/ui-egui/src/app/gpu.rs @@ -73,14 +73,30 @@ impl EffectcraftApp { renderer.free_texture(id); } for (_, texture) in self.passive_tex.values() { - if let panels::viewers::PassiveTexture::Gpu(id, _) = texture { - renderer.free_texture(id); + match texture { + panels::viewers::PassiveTexture::Gpu(id, _) => renderer.free_texture(id), + panels::viewers::PassiveTexture::Baked(b) => { + if let Some(id) = b.id() { + renderer.free_texture(&id); + } + } + panels::viewers::PassiveTexture::Cpu(_) => {} } } + drop(renderer); } + // The display pass's textures: half-float frames go through the CPU from now on. + self.viewer_bake.free(self.window_wgpu.as_ref()); + self.snapshot_bake.free(self.window_wgpu.as_ref()); + self.display_gpu = None; + self.window_wgpu = None; self.viewer_native = None; self.passive_tex.retain(|_, (_, texture)| matches!(texture, panels::viewers::PassiveTexture::Cpu(_))); - self.viewer_shown = self.viewer_tex.as_ref().map(|(texture, key, _)| (texture.id(), *key)); + self.viewer_shown = match &self.viewer_f16 { + // A half-float frame stays on screen through the CPU pass. + Some(_) => self.viewer_shown.map(|(_, key)| (None, key)), + None => self.viewer_tex.as_ref().map(|(texture, key, _)| (Some(texture.id()), *key)), + }; self.viewer_image = None; // Old completion closures retain only their detached slot, not the replacement. self.viewer_readback = Default::default(); @@ -152,6 +168,9 @@ impl EffectcraftApp { return; } let Some(rs) = frame.wgpu_render_state() else { return }; + // The display pass needs only this device, not the compositor (which may take tens of + // seconds to build, or fail). + self.window_wgpu = Some(rs.clone()); // Help ▸ System Compatibility Report names the adapter and backend the window got. let info = rs.adapter.get_info(); self.session.window_adapter = Some(format!( diff --git a/crates/ui-egui/src/app/prefetch.rs b/crates/ui-egui/src/app/prefetch.rs index f00d44014..4af91782f 100644 --- a/crates/ui-egui/src/app/prefetch.rs +++ b/crates/ui-egui/src/app/prefetch.rs @@ -221,10 +221,7 @@ impl EffectcraftApp { self.playback.start_frame = cur; // `prefetch_playback` requests every uncached frame of offsets 0..window from here. let window = (ahead * 3).min(fit - 1); - let need = self.frames.get(&FrameKey { frame: cur, ..series }).map(|img| { - let [w, h] = img.size(); - startup_frames(ahead, window, self.frames.budget(), w.saturating_mul(h).saturating_mul(4)) - }); + let need = self.frames.get(&FrameKey { frame: cur, ..series }).map(|img| startup_frames(ahead, window, self.frames.budget(), img.bytes())); if need.is_some() && self.playback.first_cached.is_none() { self.playback.first_cached = Some(now); } @@ -448,10 +445,11 @@ impl EffectcraftApp { ((self.frames.budget() as f64 * 0.9 / self.frame_bytes(comp, scale)) as i64).max(1) } - /// Bytes of one comp-sized viewer frame at `scale` (Extended Viewer frames are larger). + /// Bytes of one comp-sized viewer frame at `scale` (Extended Viewer frames are larger; 32 bpc + /// frames are half floats, twice the bytes of 8-bit ones). fn frame_bytes(&self, comp: &effectcraft_engine::project::Comp, scale: f64) -> f64 { let side = |n: u32| (n as f64 * scale).ceil().max(1.0); - side(comp.width) * side(comp.height) * 4.0 + side(comp.width) * side(comp.height) * frames::frame_pixel_bytes(&self.session.project.settings) as f64 } fn frames_parallelism(&self) -> usize { diff --git a/crates/ui-egui/src/app/viewer_frames.rs b/crates/ui-egui/src/app/viewer_frames.rs index e58655817..a1a0faaf5 100644 --- a/crates/ui-egui/src/app/viewer_frames.rs +++ b/crates/ui-egui/src/app/viewer_frames.rs @@ -7,11 +7,18 @@ impl EffectcraftApp { } /// The viewer's current pixels as 8-bit premultiplied RGBA, reading a GPU frame back the - /// first time something asks for it. + /// first time something asks for it. A half-float frame (32 bpc) is converted the first time: + /// as rendered with Project Settings ▸ HDR, without the viewer's exposure or display colour. /// /// The browser can't wait for the GPU: there the first call starts an asynchronous readback /// and returns `None`; a later call (the next frame) has the pixels. pub fn viewer_pixels(&mut self) -> Option> { + if self.viewer_image.is_none() + && let Some(f) = &self.viewer_f16 + { + let pass = effectcraft_engine::viewer::DisplayPass::new(effectcraft_engine::viewer::Channel::Rgb, false, 0.0, f.encoding, None); + self.viewer_image = Some(Arc::new(f.to_color_image(&pass))); + } let to_image = |bytes: Vec, w: u32, h: u32| { let px = bytes.as_chunks::<4>().0.iter().map(|c| egui::Color32::from_rgba_premultiplied(c[0], c[1], c[2], c[3])).collect(); Arc::new(egui::ColorImage::new([w as usize, h as usize], px)) @@ -44,6 +51,36 @@ impl EffectcraftApp { self.viewer_image.clone() } + /// The half-float frame on screen (32 bpc projects), for float readouts. + pub fn viewer_f16(&self) -> Option<&Arc> { + self.viewer_f16.as_ref() + } + + /// The texture the Composition viewer's display pass drew last (half-float frames, or 8-bit + /// frames with exposure, a channel view or display colour), when it ran on the GPU. + pub fn viewer_display_texture(&self) -> Option { + self.viewer_bake.id() + } + + /// The display pass's pipelines on egui-wgpu's device, made the first time (desktop; the + /// browser's viewer converts on the CPU for now). `None` without a device. + pub(crate) fn display_gpu(&mut self) -> Option> { + if cfg!(target_arch = "wasm32") || self.display_gpu_failed { + return None; + } + if self.display_gpu.is_none() { + let rs = self.window_wgpu.as_ref()?; + match crate::display_gpu::DisplayGpu::try_new(&rs.device) { + Ok(g) => self.display_gpu = Some(Arc::new(g)), + Err(e) => { + log::warn!("the viewer's display pass runs on the CPU: {e}"); + self.display_gpu_failed = true; + } + } + } + self.display_gpu.clone() + } + /// The viewer's Resolution: the active composition's (saved with the project). pub fn viewer_res(&self) -> state::Resolution { self.session.active_comp().map(|c| c.resolution).unwrap_or_default() @@ -173,7 +210,9 @@ impl EffectcraftApp { backend: effectcraft_engine::render::Backend::Auto, nested_switches: self.session.prefs.general.switches_affect_nested_comps, draft_shadows: self.session.prefs.three_d.realtime_shadows, - skip_hdr_step: false, + // 32 bpc: half-float frames without the HDR step; the display pass applies it after + // the viewer's exposure (#2797). + skip_hdr_step: frames::f16_frames(&self.session.project.settings), proxy: Default::default(), } } diff --git a/crates/ui-egui/src/color_picker.rs b/crates/ui-egui/src/color_picker.rs index c75cc0242..d434538e6 100644 --- a/crates/ui-egui/src/color_picker.rs +++ b/crates/ui-egui/src/color_picker.rs @@ -710,7 +710,8 @@ mod tests { options.exposure = 1.0; let exposed = press_sampling_viewer(&mut harness).unwrap(); let mut expected = [[32, 64, 96, 255]]; - effectcraft_engine::viewer::display_transform(&mut expected, effectcraft_engine::viewer::Channel::Rgb, false, 1.0); + let encoding = effectcraft_engine::viewer::FrameEncoding::of(&harness.state().session.project.settings, false); + effectcraft_engine::viewer::DisplayPass::new(effectcraft_engine::viewer::Channel::Rgb, false, 1.0, encoding, None).apply_rgba8(&mut expected); for (actual, expected) in exposed.iter().zip(expected[0]) { assert!((*actual - expected as f32 / 255.0).abs() < 0.01, "{exposed:?}"); } @@ -736,6 +737,94 @@ mod tests { assert_eq!(press_sampling_viewer(&mut harness), None, "wireframe must not sample an old pixel frame"); } + /// A 32 bpc project whose solid is about 4.0 in linear light (white +2 stops), in half-float viewer + /// frames; `hdr`: Project Settings ▸ HDR (with an sRGB working space). + fn hdr_viewer(hdr: &str) -> egui_kittest::Harness<'static, crate::EffectcraftApp> { + let mut s = effectcraft_engine::Session::default(); + s.execute("comp.new", serde_json::json!({"width": 320, "height": 180, "duration": 2})).unwrap(); + s.execute("file.projectSettings", serde_json::json!({"bitDepth": 32, "workingSpace": "srgb", "hdr": hdr})).unwrap(); + let lid = s.execute("layer.newSolid", serde_json::json!({"color": "#ffffff"})).unwrap()["layer"].as_u64().unwrap(); + s.execute("effect.apply", serde_json::json!({"layers": [lid], "effect": "Exposure"})).unwrap(); + s.execute("prop.set", serde_json::json!({"layer": lid, "path": "effects/#1/master/exposure", "value": 2.0})).unwrap(); + let mut harness = egui_kittest::Harness::builder().with_size(vec2(1200.0, 800.0)).build_eframe(|_| crate::EffectcraftApp::new(s)); + for _ in 0..400 { + harness.step(); + if harness.state().frames.inflight() == 0 && harness.state().viewer_f16.is_some() { + harness.run_steps(3); + return harness; + } + std::thread::sleep(std::time::Duration::from_millis(5)); + } + panic!("the half-float viewer frame did not arrive"); + } + + /// The linear light of the frame's first pixel (sRGB-encoded). + fn linear_of(harness: &egui_kittest::Harness<'_, crate::EffectcraftApp>) -> f32 { + let p = harness.state().viewer_f16.as_ref().unwrap().pixel(0).unwrap(); + effectcraft_engine::color::srgb_to_linear(p[0] / p[3]) + } + + fn exposed(harness: &mut egui_kittest::Harness<'_, crate::EffectcraftApp>, stops: f32) -> f32 { + harness.state_mut().session.state.viewer.exposure = stops; + harness.run_steps(2); + press_sampling_viewer(harness).unwrap()[0] + } + + /// #2797: in 32 bpc the viewer's exposure reaches values above 1.0. White +2 stops (about 4.0) + /// is white at 0 EV and still white at -2 EV (the 8-bit viewer showed 0.25 there), and at -3 EV + /// it is an eighth of its linear light. + #[test] + fn viewer_exposure_reaches_over_range_values_in_32_bpc() { + let mut harness = hdr_viewer("clip"); + let l = linear_of(&harness); + assert!(l > 3.9, "the frame keeps the over-range value: {l}"); + let srgb = |l: f32| effectcraft_engine::color::linear_to_srgb(l); + assert!((exposed(&mut harness, 0.0) - 1.0).abs() < 0.01); + assert!((exposed(&mut harness, -2.0) - 1.0).abs() < 0.01, "-2 EV"); + let v = exposed(&mut harness, -3.0); + assert!((v - srgb(l / 8.0)).abs() < 0.01 && v < 0.8, "-3 EV: {v}"); + // The 8-bit image other panels read: as rendered (clipped), without the exposure. + let img = harness.state_mut().viewer_pixels().unwrap(); + assert_eq!(img.pixels[0].to_array(), [255, 255, 255, 255]); + // A 32 bpc snapshot keeps the range too. + harness.state_mut().session.execute("view.takeSnapshot", serde_json::json!({})).unwrap(); + harness.state_mut().session.state.viewer.show_snapshot = true; + let v = exposed(&mut harness, -3.0); + assert!((v - srgb(l / 8.0)).abs() < 0.01, "snapshot at -3 EV: {v}"); + } + + /// Compand runs after the exposure: at -3 EV the 4.0 white is about 0.5, under the knee, so it + /// shows as that (companding first left 0.99, darkened to 0.12). At 0 EV it is companded as + /// before. + #[test] + fn compand_follows_the_viewer_exposure() { + let mut harness = hdr_viewer("compand"); + assert_eq!(harness.state().viewer_f16.as_ref().unwrap().encoding.hdr, effectcraft_engine::project::HdrMode::Compand); + let srgb = |l: f32| effectcraft_engine::color::linear_to_srgb(l); + let l = linear_of(&harness); + let at0 = exposed(&mut harness, 0.0); + let want = srgb(effectcraft_engine::render::color::compand(l)); + assert!((at0 - want).abs() < 0.01, "0 EV: {at0} vs {want}"); + let v = exposed(&mut harness, -3.0); + assert!((v - srgb(l / 8.0)).abs() < 0.01 && v < 0.8, "-3 EV: {v}"); + // Other panels' 8-bit image is the companded frame, as the render was. + let img = harness.state_mut().viewer_pixels().unwrap(); + assert!((img.pixels[0].r() as f32 / 255.0 - want).abs() < 0.01); + // A passive viewer gets the half-float frame through the same pass (no channel or exposure + // of its own; on the CPU without a device): the companded frame. + let ctx = harness.ctx.clone(); + let (key, f) = (harness.state().viewer_shown.unwrap().1, harness.state().viewer_f16.clone().unwrap()); + crate::panels::viewers::set_texture(harness.state_mut(), &ctx, 7, key, crate::frames::FrameImage::CpuF16(f), None, 0); + let Some((_, crate::panels::viewers::PassiveTexture::Cpu(tex))) = harness.state().passive_tex.get(&7) else { panic!("no passive texture") }; + assert_eq!(tex.size(), [320, 180]); + let f = harness.state().viewer_f16.clone().unwrap(); + let crate::frames::FrameImage::Cpu(shown) = crate::panels::viewers::cpu_passive_image(harness.state(), crate::frames::FrameImage::CpuF16(f), None) + else { + panic!("an 8-bit image") + }; + assert!((shown.pixels[0].r() as f32 / 255.0 - want).abs() < 0.01, "the passive viewer shows the companded frame: {:?}", shown.pixels[0]); + } + #[test] fn a_new_modal_backdrop_cannot_deliver_a_pending_sample() { let mut harness = sampling_viewer(); diff --git a/crates/ui-egui/src/display_gpu.rs b/crates/ui-egui/src/display_gpu.rs new file mode 100644 index 000000000..f9ddc4aad --- /dev/null +++ b/crates/ui-egui/src/display_gpu.rs @@ -0,0 +1,492 @@ +//! The viewer's display pass on egui-wgpu's device (#2797): a rendered frame (half floats for 32 +//! bpc projects, RGBA8 otherwise) goes through Show Channel, exposure, the pending HDR step and +//! the display colour (`effectcraft_engine::viewer::DisplayPass`) into one premultiplied RGBA8 +//! texture with mips per viewer, which egui draws like the GPU compositor's frames (native +//! texture, the Viewer Zoom Quality sampler). +//! +//! This is not the compositor's `Gpu` (which may be missing): it needs only egui-wgpu's device. +//! Without one (no adapter, headless tests, the device retired) the same pass runs on the CPU +//! (`DisplayPass::pixel`). The shader (`display_gpu.wgsl`) repeats the CPU steps operation by +//! operation; the tests below hold it to them. + +use std::sync::Arc; + +use effectcraft_engine::viewer::{Channel, DisplayPass, PassColor, PassConv, PassCurve}; +use eframe::egui_wgpu; +use eframe::wgpu; + +use crate::frames::F16Frame; + +const UNIFORM_BYTES: usize = 12 * 16; + +/// A frame for the display pass. +#[derive(Clone)] +pub enum Source { + F16(Arc), + Rgba8(Arc), +} + +impl Source { + fn same(&self, o: &Source) -> bool { + match (self, o) { + (Source::F16(a), Source::F16(b)) => Arc::ptr_eq(a, b), + (Source::Rgba8(a), Source::Rgba8(b)) => Arc::ptr_eq(a, b), + _ => false, + } + } + + pub fn size(&self) -> [usize; 2] { + match self { + Source::F16(f) => f.size(), + Source::Rgba8(c) => c.size, + } + } + + /// A half-float frame past the texture limit goes through the CPU pass: the display + /// conversion (exposure, HDR) comes before the averaging down, as with the 8-bit mips, and + /// the GPU would average the scene-linear values first. + pub fn needs_cpu_pass(&self, max_side: usize) -> bool { + matches!(self, Source::F16(_)) && crate::frames::fit_factor(self.size(), max_side, 1) > 1 + } +} + +/// The pass's uniform block (`Params` in the shader). +pub fn uniform(pass: &DisplayPass) -> [u8; UNIFORM_BYTES] { + let channel = match pass.channel { + Channel::Rgb => 0u32, + Channel::Red => 1, + Channel::Green => 2, + Channel::Blue => 3, + Channel::Alpha => 4, + Channel::RgbStraight => 5, + }; + let hdr = match pass.encoding.hdr { + effectcraft_engine::project::HdrMode::Clip => 0u32, + effectcraft_engine::project::HdrMode::Compand => 1, + effectcraft_engine::project::HdrMode::ToneMap => 2, + }; + let curve = pass.encoding.curve; + let none = PassConv { from: PassCurve::Linear, matrix: None, to: PassCurve::Linear }; + let (kind, quantize, first, second, lut_n) = match &pass.color { + None => (0u32, false, none, None, 2u32), + Some(PassColor::Convs { first, quantize, second }) => (1, *quantize, *first, *second, 2), + Some(PassColor::Lut(l)) => (2, false, none, None, l.size.max(2) as u32), + }; + let s = second.unwrap_or(none); + let mat = |m: Option<[[f32; 3]; 3]>| { + let m = m.unwrap_or([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]); + m.map(|r| [r[0], r[1], r[2], 0.0]) + }; + let [x24, xm2, xm1] = effectcraft_engine::viewer::SHADER_EXPONENTS; + let u = |v: u32| v.to_le_bytes(); + let f = |v: f32| v.to_le_bytes(); + let mut b = Vec::with_capacity(UNIFORM_BYTES); + for v in [ + channel, + pass.colorized as u32, + curve.id(), + hdr, + kind, + quantize as u32, + second.is_some() as u32, + lut_n, + first.from.id(), + first.to.id(), + s.from.id(), + s.to.id(), + ] { + b.extend(u(v)); + } + for v in [ + pass.gain, + curve.decode_param(), + first.matrix.is_some() as u32 as f32, + s.matrix.is_some() as u32 as f32, + first.from.decode_param(), + first.to.encode_param(), + s.from.decode_param(), + s.to.encode_param(), + curve.encode_param(), + x24, + xm2, + xm1, + ] { + b.extend(f(v)); + } + for row in mat(first.matrix).into_iter().chain(mat(s.matrix)) { + for v in row { + b.extend(f(v)); + } + } + let mut out = [0u8; UNIFORM_BYTES]; + out.copy_from_slice(&b); + out +} + +/// Mip levels of a full chain for a `w` × `h` texture. +fn mip_levels(w: u32, h: u32) -> u32 { + 32 - w.max(h).max(1).leading_zeros() +} + +/// The display pass's pipelines on one device (made once, kept with the app). +pub struct DisplayGpu { + bake: wgpu::RenderPipeline, + mip: wgpu::RenderPipeline, + bake_layout: wgpu::BindGroupLayout, + mip_layout: wgpu::BindGroupLayout, + no_lut: wgpu::TextureView, + max_side: u32, +} + +impl DisplayGpu { + /// The pipelines, or why the device would not make them (a shader a backend rejects): the + /// viewer then runs the pass on the CPU instead of showing nothing. + pub fn try_new(device: &wgpu::Device) -> Result { + #[cfg(not(target_arch = "wasm32"))] + { + let internal = device.push_error_scope(wgpu::ErrorFilter::Internal); + let validation = device.push_error_scope(wgpu::ErrorFilter::Validation); + let made = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| DisplayGpu::new(device))); + let validation = pollster::block_on(validation.pop()); + let internal = pollster::block_on(internal.pop()); + if let Some(e) = validation.or(internal) { + return Err(format!("the display pass's pipelines: {e}")); + } + made.map_err(|_| "the display pass's pipelines: the backend panicked".to_string()) + } + #[cfg(target_arch = "wasm32")] + { + Ok(DisplayGpu::new(device)) + } + } + + pub fn new(device: &wgpu::Device) -> DisplayGpu { + let module = device.create_shader_module(wgpu::ShaderModuleDescriptor { + label: Some("viewer display pass"), + source: wgpu::ShaderSource::Wgsl(include_str!("display_gpu.wgsl").into()), + }); + let tex = |binding: u32, dim: wgpu::TextureViewDimension, filterable: bool| wgpu::BindGroupLayoutEntry { + binding, + visibility: wgpu::ShaderStages::FRAGMENT, + ty: wgpu::BindingType::Texture { sample_type: wgpu::TextureSampleType::Float { filterable }, view_dimension: dim, multisampled: false }, + count: None, + }; + let uniform = wgpu::BindGroupLayoutEntry { + binding: 0, + visibility: wgpu::ShaderStages::FRAGMENT, + ty: wgpu::BindingType::Buffer { ty: wgpu::BufferBindingType::Uniform, has_dynamic_offset: false, min_binding_size: None }, + count: None, + }; + let bake_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor { + label: Some("viewer display pass"), + entries: &[uniform, tex(1, wgpu::TextureViewDimension::D2, false), tex(2, wgpu::TextureViewDimension::D3, false)], + }); + let mip_layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor { + label: Some("viewer display mips"), + entries: &[tex(1, wgpu::TextureViewDimension::D2, false)], + }); + let pipeline = |label: &str, layout: &wgpu::BindGroupLayout, entry: &str| { + let pl = + device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor { label: Some(label), bind_group_layouts: &[Some(layout)], immediate_size: 0 }); + device.create_render_pipeline(&wgpu::RenderPipelineDescriptor { + label: Some(label), + layout: Some(&pl), + vertex: wgpu::VertexState { module: &module, entry_point: Some("vs"), buffers: &[], compilation_options: Default::default() }, + fragment: Some(wgpu::FragmentState { + module: &module, + entry_point: Some(entry), + targets: &[Some(wgpu::ColorTargetState { format: wgpu::TextureFormat::Rgba8Unorm, blend: None, write_mask: wgpu::ColorWrites::ALL })], + compilation_options: Default::default(), + }), + primitive: wgpu::PrimitiveState::default(), + depth_stencil: None, + multisample: wgpu::MultisampleState::default(), + multiview_mask: None, + cache: None, + }) + }; + let no_lut = device.create_texture(&wgpu::TextureDescriptor { + label: Some("viewer display LUT (none)"), + size: wgpu::Extent3d { width: 1, height: 1, depth_or_array_layers: 1 }, + mip_level_count: 1, + sample_count: 1, + dimension: wgpu::TextureDimension::D3, + format: wgpu::TextureFormat::Rgba32Float, + usage: wgpu::TextureUsages::TEXTURE_BINDING, + view_formats: &[], + }); + DisplayGpu { + bake: pipeline("viewer display pass", &bake_layout, "fs"), + mip: pipeline("viewer display mips", &mip_layout, "mip"), + bake_layout, + mip_layout, + no_lut: no_lut.create_view(&Default::default()), + max_side: device.limits().max_texture_dimension_2d, + } + } + + /// The largest texture side the device takes. + pub fn max_side(&self) -> usize { + self.max_side as usize + } + + /// `src` in a texture (half floats or RGBA8), averaged down by `k` first when `k > 1` + /// (8-bit frames past the texture limit, #201; half-float ones never get here, see + /// [`Source::needs_cpu_pass`]): `reuse` when it has the size and format (a playing + /// preview's frames), else a new one. + fn upload(&self, device: &wgpu::Device, queue: &wgpu::Queue, src: &Source, k: usize, reuse: Option) -> wgpu::Texture { + let write = |w: u32, h: u32, format: wgpu::TextureFormat, bytes: &[u8]| { + let (w, h) = (w.max(1), h.max(1)); + let fits = |t: &wgpu::Texture| t.width() == w && t.height() == h && t.format() == format; + let tex = reuse.clone().filter(fits).unwrap_or_else(|| { + device.create_texture(&wgpu::TextureDescriptor { + label: Some("viewer frame (display pass input)"), + size: wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 }, + mip_level_count: 1, + sample_count: 1, + dimension: wgpu::TextureDimension::D2, + format, + usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST, + view_formats: &[], + }) + }); + let px = if format == wgpu::TextureFormat::Rgba16Float { 8 } else { 4 }; + if bytes.len() as u64 >= w as u64 * h as u64 * px as u64 { + queue.write_texture( + wgpu::TexelCopyTextureInfo { texture: &tex, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All }, + bytes, + wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(w * px), rows_per_image: Some(h) }, + wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 }, + ); + } + tex + }; + let f16 = wgpu::TextureFormat::Rgba16Float; + let rgba8 = wgpu::TextureFormat::Rgba8Unorm; + match src { + Source::F16(f) => { + debug_assert_eq!(k, 1, "half-float frames past the texture limit take the CPU pass"); + write(f.width, f.height, f16, bytemuck::cast_slice(f.bits())) + } + Source::Rgba8(c) if k > 1 => { + let c = crate::frames::fit_texture_by((**c).clone(), usize::MAX, k); + write(c.size[0] as u32, c.size[1] as u32, rgba8, c.as_raw()) + } + Source::Rgba8(c) => write(c.size[0] as u32, c.size[1] as u32, rgba8, c.as_raw()), + } + } + + fn lut(&self, device: &wgpu::Device, queue: &wgpu::Queue, lut: &effectcraft_engine::color::icc::Lut3d) -> wgpu::Texture { + let n = lut.size.max(2) as u32; + let tex = device.create_texture(&wgpu::TextureDescriptor { + label: Some("viewer display LUT"), + size: wgpu::Extent3d { width: n, height: n, depth_or_array_layers: n }, + mip_level_count: 1, + sample_count: 1, + dimension: wgpu::TextureDimension::D3, + format: wgpu::TextureFormat::Rgba32Float, + usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_DST, + view_formats: &[], + }); + // data[(r · n + g) · n + b]: x = b, y = g, z = r. + let mut texels: Vec = Vec::with_capacity((n * n * n * 4) as usize); + for i in 0..(n * n * n) as usize { + let v = lut.data.get(i).copied().unwrap_or_default(); + texels.extend([v[0], v[1], v[2], 1.0]); + } + queue.write_texture( + wgpu::TexelCopyTextureInfo { texture: &tex, mip_level: 0, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All }, + bytemuck::cast_slice(&texels), + wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(n * 16), rows_per_image: Some(n) }, + wgpu::Extent3d { width: n, height: n, depth_or_array_layers: n }, + ); + tex + } + + /// The display texture for `input`: premultiplied RGBA8, a full mip chain. + fn output(device: &wgpu::Device, w: u32, h: u32) -> wgpu::Texture { + device.create_texture(&wgpu::TextureDescriptor { + label: Some("viewer frame (displayed)"), + size: wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 }, + mip_level_count: mip_levels(w, h), + sample_count: 1, + dimension: wgpu::TextureDimension::D2, + format: wgpu::TextureFormat::Rgba8Unorm, + usage: wgpu::TextureUsages::RENDER_ATTACHMENT | wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_SRC, + view_formats: &[], + }) + } + + /// Run the pass from `input` into level 0 of `out`, then fill its mips. + fn run( + &self, + device: &wgpu::Device, + queue: &wgpu::Queue, + input: &wgpu::Texture, + out: &wgpu::Texture, + params: &[u8; UNIFORM_BYTES], + lut: Option<&wgpu::Texture>, + ) { + use wgpu::util::DeviceExt as _; + let buf = device.create_buffer_init(&wgpu::util::BufferInitDescriptor { + label: Some("viewer display pass"), + contents: params, + usage: wgpu::BufferUsages::UNIFORM, + }); + let input_view = input.create_view(&Default::default()); + let lut_view = lut.map(|l| l.create_view(&Default::default())); + let bake = device.create_bind_group(&wgpu::BindGroupDescriptor { + label: Some("viewer display pass"), + layout: &self.bake_layout, + entries: &[ + wgpu::BindGroupEntry { binding: 0, resource: buf.as_entire_binding() }, + wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::TextureView(&input_view) }, + wgpu::BindGroupEntry { binding: 2, resource: wgpu::BindingResource::TextureView(lut_view.as_ref().unwrap_or(&self.no_lut)) }, + ], + }); + let level = |l: u32| out.create_view(&wgpu::TextureViewDescriptor { base_mip_level: l, mip_level_count: Some(1), ..Default::default() }); + let mut enc = device.create_command_encoder(&wgpu::CommandEncoderDescriptor { label: Some("viewer display pass") }); + let pass = |enc: &mut wgpu::CommandEncoder, target: &wgpu::TextureView, pipeline: &wgpu::RenderPipeline, bg: &wgpu::BindGroup| { + let mut p = enc.begin_render_pass(&wgpu::RenderPassDescriptor { + label: Some("viewer display pass"), + color_attachments: &[Some(wgpu::RenderPassColorAttachment { + view: target, + depth_slice: None, + resolve_target: None, + ops: wgpu::Operations { load: wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT), store: wgpu::StoreOp::Store }, + })], + ..Default::default() + }); + p.set_pipeline(pipeline); + p.set_bind_group(0, bg, &[]); + p.draw(0..3, 0..1); + }; + pass(&mut enc, &level(0), &self.bake, &bake); + for l in 1..out.mip_level_count() { + let above = level(l - 1); + let bg = device.create_bind_group(&wgpu::BindGroupDescriptor { + label: Some("viewer display mips"), + layout: &self.mip_layout, + entries: &[wgpu::BindGroupEntry { binding: 1, resource: wgpu::BindingResource::TextureView(&above) }], + }); + pass(&mut enc, &level(l), &self.mip, &bg); + } + queue.submit([enc.finish()]); + } +} + +/// One viewer's displayed texture (the Composition viewer, a passive viewer, the snapshot): the +/// frame it was made from and the pass's uniform, so it runs again only when either changes. +#[derive(Default)] +pub struct Bake { + source: Option, + /// Averaged down by this for the texture limit (1: not). + factor: usize, + input: Option, + output: Option, + id: Option, + params: Option<[u8; UNIFORM_BYTES]>, + lut: Option<(Arc, wgpu::Texture)>, + smooth: Option, + /// How often the pass ran (tests). + runs: u64, +} + +impl Bake { + /// The egui texture showing `src` through `pass` (sampled with the Viewer Zoom Quality's + /// sampler for `smooth`), running the pass again when the frame or the pass changed. + pub fn show(&mut self, rs: &egui_wgpu::RenderState, gpu: &DisplayGpu, src: &Source, pass: &DisplayPass, smooth: bool) -> Option { + if src.needs_cpu_pass(gpu.max_side()) { + return None; + } + let (device, queue) = (&rs.device, &rs.queue); + let params = uniform(pass); + let new_source = !self.source.as_ref().is_some_and(|s| s.same(src)); + if new_source { + self.factor = crate::frames::fit_factor(src.size(), gpu.max_side(), 1); + self.input = Some(gpu.upload(device, queue, src, self.factor, self.input.take())); + self.source = Some(src.clone()); + } + let lut = match &pass.color { + Some(PassColor::Lut(l)) => Some(l.clone()), + _ => None, + }; + // A new LUT (another profile, Preserve RGB, another display) has the same size, so the + // uniform alone doesn't tell: it reruns the pass itself. + let new_lut = match (&lut, &self.lut) { + (Some(l), Some((have, _))) if Arc::ptr_eq(l, have) => false, + (Some(l), _) => { + self.lut = Some((l.clone(), gpu.lut(device, queue, l))); + true + } + (None, had) => { + let changed = had.is_some(); + self.lut = None; + changed + } + }; + let input = self.input.as_ref()?; + let (w, h) = (input.width(), input.height()); + let resized = self.output.as_ref().is_none_or(|o| o.width() != w || o.height() != h); + if resized { + self.output = Some(DisplayGpu::output(device, w, h)); + } + let output = self.output.as_ref()?; + if new_source || resized || new_lut || self.params != Some(params) { + gpu.run(device, queue, input, output, ¶ms, self.lut.as_ref().map(|(_, t)| t)); + self.params = Some(params); + self.runs += 1; + } + let view = output.create_view(&Default::default()); + let sampler = || crate::panels::viewer::zoom_sampler(smooth); + let mut ren = rs.renderer.write(); + let id = match self.id { + Some(id) if !resized && self.smooth == Some(smooth) => id, + Some(id) => { + ren.update_egui_texture_from_wgpu_texture_with_sampler_options(device, &view, sampler(), id); + id + } + None => ren.register_native_texture_with_sampler_options(device, &view, sampler()), + }; + self.id = Some(id); + self.smooth = Some(smooth); + Some(id) + } + + /// The texture egui draws, once [`Bake::show`] made it. + pub fn id(&self) -> Option { + self.id + } + + /// The frame shown (the one last given to [`Bake::show`]). + pub fn source(&self) -> Option<&Source> { + self.source.as_ref() + } + + /// The texture coordinates of the frame in its texture (see `frames::fitted_uv`). + pub fn uv(&self) -> egui::Rect { + let size = self.source.as_ref().map_or([1, 1], Source::size); + crate::frames::fitted_uv(size, self.factor.max(1)) + } + + /// Give the texture back to the renderer. + pub fn free(&mut self, rs: Option<&egui_wgpu::RenderState>) { + if let (Some(id), Some(rs)) = (self.id.take(), rs) { + rs.renderer.write().free_texture(&id); + } + *self = Bake::default(); + } + + /// How often the pass ran for this slot. + pub fn runs(&self) -> u64 { + self.runs + } + + /// The displayed texture (tests read it back). + pub fn texture(&self) -> Option<&wgpu::Texture> { + self.output.as_ref() + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/ui-egui/src/display_gpu.wgsl b/crates/ui-egui/src/display_gpu.wgsl new file mode 100644 index 000000000..7e8163431 --- /dev/null +++ b/crates/ui-egui/src/display_gpu.wgsl @@ -0,0 +1,314 @@ +// The viewer's display pass (#2797): a rendered frame (premultiplied, encoded in the output +// space; Rgba16Float for 32 bpc frames, Rgba8Unorm otherwise) → the premultiplied RGBA8 texture +// the viewer draws. Line by line the same operations as `effectcraft_engine::viewer::DisplayPass` +// (its `pixel` is this shader's CPU reference and fallback): +// unpremultiply → decode, × 2^stops, Compand / Tone Map, encode → clamp → display colour → +// Show Channel, premultiply → floor(v · 255 + 0.5). + +struct Params { + // channel (0 RGB, 1 R, 2 G, 3 B, 4 A, 5 RGB Straight), colorized, frame curve, HDR step + // (0 none, 1 Compand, 2 Tone Map) + ids0: vec4, + // display colour (0 none, 1 conversions, 2 LUT), 8-bit device numbers, a second conversion, + // LUT size + ids1: vec4, + // curves: first conversion's decode and encode, second conversion's decode and encode + ids2: vec4, + // gain (2^stops), frame curve's gamma, first / second conversion has a matrix (1.0) + vals0: vec4, + // gammas of the ids2 curves (encoding: 1 / gamma) + vals1: vec4, + // 1 / the frame curve's gamma, 1 / 2.4, 1 / PQ m2, 1 / PQ m1: divided on the CPU, as the + // reference does, so the exponents are the same bits + vals2: vec4, + m1: array, 3>, + m2: array, 3>, +}; + +@group(0) @binding(0) var P: Params; +@group(0) @binding(1) var frame: texture_2d; +@group(0) @binding(2) var lut: texture_3d; + +@vertex +fn vs(@builtin(vertex_index) i: u32) -> @builtin(position) vec4 { + // One triangle covering the target. + let x = f32((i << 1u) & 2u); + let y = f32(i & 2u); + return vec4(x * 2.0 - 1.0, 1.0 - y * 2.0, 0.0, 1.0); +} + +// The curves take the gamma to decode with, or 1 / gamma to encode with. +// SMPTE ST 2084 and BT.2100 HLG (effectcraft_color::space), linear 1.0 = reference white. +const PQ_M1: f32 = 0.1593017578125; +const PQ_M2: f32 = 78.84375; +const PQ_C1: f32 = 0.8359375; +const PQ_C2: f32 = 18.8515625; +const PQ_C3: f32 = 18.6875; +const PQ_WHITE: f32 = 0.0203; +const HLG_A: f32 = 0.17883277; +const HLG_B: f32 = 0.28466892; +const HLG_C: f32 = 0.55991073; +const HLG_REFERENCE: f32 = 0.26496256; + +fn power(a: f32, g: f32) -> f32 { + if (a <= 0.0) { + return 0.0; + } + return pow(a, g); +} + +fn copysign(m: f32, v: f32) -> f32 { + if (v < 0.0) { + return -m; + } + return m; +} + +fn decode(curve: u32, g: f32, v: f32) -> f32 { + let a = abs(v); + var l = a; + switch curve { + case 1u: { + if (a <= 0.04045) { + l = a / 12.92; + } else { + l = power((a + 0.055) / 1.055, 2.4); + } + } + case 2u: { + l = power(a, g); + } + case 3u: { + let p = power(min(a, 1.0), P.vals2.z); + l = power(max(p - PQ_C1, 0.0) / (PQ_C2 - PQ_C3 * p), P.vals2.w) / PQ_WHITE; + } + case 4u: { + var e: f32; + if (a <= 0.5) { + e = a * a / 3.0; + } else { + e = (exp((a - HLG_C) / HLG_A) + HLG_B) / 12.0; + } + l = e / HLG_REFERENCE; + } + default: {} + } + return copysign(l, v); +} + +fn encode(curve: u32, g: f32, v: f32) -> f32 { + let a = abs(v); + var e = a; + switch curve { + case 1u: { + if (a <= 0.0031308) { + e = a * 12.92; + } else { + e = 1.055 * power(a, P.vals2.y) - 0.055; + } + } + case 2u: { + e = power(a, g); + } + case 3u: { + let y = power(a * PQ_WHITE, PQ_M1); + e = power((PQ_C1 + PQ_C2 * y) / (1.0 + PQ_C3 * y), PQ_M2); + } + case 4u: { + let s = a * HLG_REFERENCE; + if (s <= 1.0 / 12.0) { + e = sqrt(3.0 * s); + } else { + e = HLG_A * log(12.0 * s - HLG_B) + HLG_C; + } + } + default: {} + } + return copysign(e, v); +} + +fn decode3(curve: u32, g: f32, c: vec3) -> vec3 { + return vec3(decode(curve, g, c.x), decode(curve, g, c.y), decode(curve, g, c.z)); +} + +fn encode3(curve: u32, g: f32, c: vec3) -> vec3 { + return vec3(encode(curve, g, c.x), encode(curve, g, c.y), encode(curve, g, c.z)); +} + +fn mat_mul(m: array, 3>, l: vec3) -> vec3 { + return vec3( + m[0].x * l.x + m[0].y * l.y + m[0].z * l.z, + m[1].x * l.x + m[1].y * l.y + m[1].z * l.z, + m[2].x * l.x + m[2].y * l.y + m[2].z * l.z, + ); +} + +// effectcraft_render::color::compand +fn compand(v: f32) -> f32 { + let k = 0.8; + if (v <= k) { + return v; + } + return k + (1.0 - k) * (1.0 - exp(-(v - k) / (1.0 - k))); +} + +// effectcraft_render::color::tone_map (extended Reinhard of Rec. 709 luminance, white 4.0) +fn tone_map(c: vec3) -> vec3 { + let w = 4.0; + let l = 0.2126 * c.x + 0.7152 * c.y + 0.0722 * c.z; + if (l <= 0.0) { + return c; + } + let lm = l * (1.0 + l / (w * w)) / (1.0 + l); + let k = lm / l; + return c * k; +} + +fn lut_at(n: u32, r: u32, g: u32, b: u32) -> vec3 { + return textureLoad(lut, vec3(b, g, r), 0).xyz; +} + +// effectcraft_engine::viewer::lut_eval: trilinear in the same order. +fn lut_eval(c: vec3) -> vec3 { + let n = P.ids1.w; + let x = clamp(c, vec3(0.0), vec3(1.0)) * f32(n - 1u); + let k = min(vec3(floor(x)), vec3(n - 2u)); + let f = x - vec3(k); + var out = vec3(0.0); + for (var corner = 0u; corner < 8u; corner++) { + let d = vec3((corner >> 2u) & 1u, (corner >> 1u) & 1u, corner & 1u); + var wr = 1.0 - f.x; + if (d.x == 1u) { + wr = f.x; + } + var wg = 1.0 - f.y; + if (d.y == 1u) { + wg = f.y; + } + var wb = 1.0 - f.z; + if (d.z == 1u) { + wb = f.z; + } + let w = wr * wg * wb; + out = out + w * lut_at(n, k.x + d.x, k.y + d.y, k.z + d.z); + } + return out; +} + +fn display_colour(c: vec3) -> vec3 { + var o = c; + if (P.ids1.x == 2u) { + o = lut_eval(c); + } else if (P.ids1.x == 1u) { + o = decode3(P.ids2.x, P.vals1.x, o); + if (P.vals0.z == 1.0) { + o = mat_mul(P.m1, o); + } + o = encode3(P.ids2.y, P.vals1.y, o); + if (P.ids1.y == 1u) { + o = floor(clamp(o, vec3(0.0), vec3(1.0)) * 255.0 + 0.5) / 255.0; + } + if (P.ids1.z == 1u) { + o = decode3(P.ids2.z, P.vals1.z, o); + if (P.vals0.w == 1.0) { + o = mat_mul(P.m2, o); + } + o = encode3(P.ids2.w, P.vals1.w, o); + } + } + return clamp(o, vec3(0.0), vec3(1.0)); +} + +fn q(v: f32) -> f32 { + return floor(clamp(v, 0.0, 1.0) * 255.0 + 0.5) / 255.0; +} + +@fragment +fn fs(@builtin(position) pos: vec4) -> @location(0) vec4 { + let s = textureLoad(frame, vec2(floor(pos.xy)), 0); + // Frames are never NaN here: `frames::F16Frame` and 8-bit frames hold finite values only + // (the NaN test is the reference's, kept for safety; WGSL may drop it). Alpha is clamped to + // 0..1 first, as `display_pixel` and `DisplayPass::pixel` do (the compositor's alpha never + // exceeds 1). + var a = s.w; + if (!(a == a)) { + a = 0.0; + } + a = clamp(a, 0.0, 1.0); + var c = vec3(0.0); + if (a > 0.0) { + c = s.xyz / a; + } + let channel = P.ids0.x; + let curve = P.ids0.z; + let g = P.vals0.y; + let gain = P.vals0.x; + let light = gain != 1.0 || P.ids0.w != 0u; + // Premultiplied channels without exposure or HDR step: straight from the frame. + var pm = clamp(s.xyz, vec3(0.0), vec3(a)); + if (light) { + var l = decode3(curve, g, c) * gain; + if (P.ids0.w == 1u) { + l = vec3(compand(l.x), compand(l.y), compand(l.z)); + } else if (P.ids0.w == 2u) { + l = tone_map(l); + } + c = encode3(curve, P.vals2.x, l); + } + c = clamp(c, vec3(0.0), vec3(1.0)); + if (light) { + pm = c * a; + } + if (channel == 0u || channel == 5u) { + if (P.ids1.x != 0u) { + c = display_colour(c); + pm = c * a; + } + if (channel == 5u) { + return vec4(q(c.x), q(c.y), q(c.z), 1.0); + } + return vec4(q(pm.x), q(pm.y), q(pm.z), q(a)); + } + if (channel == 4u) { + var v = a; + if (gain != 1.0) { + v = encode(curve, P.vals2.x, decode(curve, g, a) * gain); + } + let e = q(v); + return vec4(e, e, e, 1.0); + } + let v = q(pm[channel - 1u]); + if (P.ids0.y == 1u) { + var o = vec4(0.0, 0.0, 0.0, 1.0); + o[channel - 1u] = v; + return o; + } + return vec4(v, v, v, 1.0); +} + +// The next mip level of the displayed texture (`frame` = the level above): each texel the mean +// of the premultiplied texels it covers, 2 × 2, and at the right / bottom edge of a level above +// of odd size also its last column / row (3 wide or high), so no texel of the frame is dropped +// (the level sizes are the GPU's, rounded down). Rounded as the pass rounds. +@fragment +fn mip(@builtin(position) pos: vec4) -> @location(0) vec4 { + let above = vec2(textureDimensions(frame)); + let size = max(above / 2, vec2(1)); + let p = vec2(floor(pos.xy)); + let lo = min(p * 2, above - vec2(1)); + var hi = min(p * 2 + vec2(2), above); + if (p.x == size.x - 1) { + hi.x = above.x; + } + if (p.y == size.y - 1) { + hi.y = above.y; + } + var s = vec4(0.0); + for (var y = lo.y; y < hi.y; y++) { + for (var x = lo.x; x < hi.x; x++) { + s = s + textureLoad(frame, vec2(x, y), 0); + } + } + let n = f32((hi.x - lo.x) * (hi.y - lo.y)); + return floor(s / n * 255.0 + 0.5) / 255.0; +} diff --git a/crates/ui-egui/src/display_gpu/tests.rs b/crates/ui-egui/src/display_gpu/tests.rs new file mode 100644 index 000000000..5b26cf5f9 --- /dev/null +++ b/crates/ui-egui/src/display_gpu/tests.rs @@ -0,0 +1,582 @@ +//! The display pass on a real device (any wgpu adapter; CI runs lavapipe), held to its CPU +//! reference `DisplayPass::pixel`. Skips without an adapter. + +use std::sync::{Arc, Mutex}; + +use effectcraft_engine::project::HdrMode; +use effectcraft_engine::render::Image; +use effectcraft_engine::viewer::{Channel, DisplayColor, DisplayPass, FrameEncoding, PassCurve, SimProfile, Simulation}; +use eframe::egui_wgpu; +use eframe::wgpu; + +use super::{DisplayGpu, Source}; +use crate::frames::F16Frame; + +/// One device at a time (tests run in parallel). +static DEVICE: Mutex<()> = Mutex::new(()); + +struct Dev { + instance: wgpu::Instance, + adapter: wgpu::Adapter, + device: wgpu::Device, + queue: wgpu::Queue, + /// A software rasterizer (lavapipe / llvmpipe, as CI has): its pow / exp / division round + /// differently again, so exact comparisons allow 1 LSB on a few bytes there. + cpu: bool, +} + +fn device() -> Option { + let instance = wgpu::Instance::new(wgpu::InstanceDescriptor::new_without_display_handle().with_env()); + let adapter = pollster::block_on(instance.request_adapter(&wgpu::RequestAdapterOptions::default())).ok()?; + let (device, queue) = + pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor { label: Some("display pass tests"), ..Default::default() })).ok()?; + let info = adapter.get_info(); + eprintln!("display pass tests on {:?} ({:?})", info.name, info.device_type); + Some(Dev { cpu: info.device_type == wgpu::DeviceType::Cpu, instance, adapter, device, queue }) +} + +/// `compare`'s result is exact, or (software rasterizers) at most 1 LSB on at most 1 % of bytes. +fn exact_enough(d: &Dev, (n, max): (usize, i32), bytes: usize) -> bool { + (n, max) == (0, 0) || d.cpu && max <= 1 && n * 100 <= bytes +} + +/// egui-wgpu's render state on the test device, as eframe hands it to the app. +fn render_state(d: &Dev) -> egui_wgpu::RenderState { + let renderer = egui_wgpu::Renderer::new(&d.device, wgpu::TextureFormat::Rgba8Unorm, egui_wgpu::RendererOptions::default()); + egui_wgpu::RenderState { + adapter: d.adapter.clone(), + available_adapters: vec![d.adapter.clone()], + instance: d.instance.clone(), + device: d.device.clone(), + queue: d.queue.clone(), + target_format: wgpu::TextureFormat::Rgba8Unorm, + renderer: Arc::new(egui::mutex::RwLock::new(renderer)), + } +} + +/// A 17³ LUT that mixes channels and bends the curve (as an ICC profile's device does). +fn test_lut(k: f64) -> Arc { + Arc::new(effectcraft_engine::color::icc::Lut3d::bake(17, |c| { + [c[0].powf(1.3) * 0.9 + c[1] * 0.1, c[1] * 0.8 + c[2] * 0.2, (c[2] * 0.7 + c[0] * 0.3).sqrt() * k] + })) +} + +fn read(d: &Dev, tex: &wgpu::Texture, level: u32) -> (Vec<[u8; 4]>, [u32; 2]) { + let (w, h) = ((tex.width() >> level).max(1), (tex.height() >> level).max(1)); + let stride = (w * 4).div_ceil(256) * 256; + let buf = d.device.create_buffer(&wgpu::BufferDescriptor { + label: None, + size: (stride * h) as u64, + usage: wgpu::BufferUsages::MAP_READ | wgpu::BufferUsages::COPY_DST, + mapped_at_creation: false, + }); + let mut e = d.device.create_command_encoder(&Default::default()); + e.copy_texture_to_buffer( + wgpu::TexelCopyTextureInfo { texture: tex, mip_level: level, origin: wgpu::Origin3d::ZERO, aspect: wgpu::TextureAspect::All }, + wgpu::TexelCopyBufferInfo { buffer: &buf, layout: wgpu::TexelCopyBufferLayout { offset: 0, bytes_per_row: Some(stride), rows_per_image: Some(h) } }, + wgpu::Extent3d { width: w, height: h, depth_or_array_layers: 1 }, + ); + d.queue.submit([e.finish()]); + buf.slice(..).map_async(wgpu::MapMode::Read, |r| r.expect("map")); + d.device.poll(wgpu::PollType::wait_indefinitely()).expect("poll"); + let data = buf.slice(..).get_mapped_range().expect("mapped").to_vec(); + let mut out = Vec::with_capacity((w * h) as usize); + for y in 0..h { + let row = &data[(y * stride) as usize..][..(w * 4) as usize]; + out.extend(row.as_chunks::<4>().0.iter().copied()); + } + (out, [w, h]) +} + +/// The pass on the device: level 0 of the displayed texture, and the texture. +fn bake(d: &Dev, gpu: &DisplayGpu, src: &Source, pass: &DisplayPass) -> (Vec<[u8; 4]>, wgpu::Texture) { + let k = crate::frames::fit_factor(src.size(), gpu.max_side(), 1); + let input = gpu.upload(&d.device, &d.queue, src, k, None); + let out = DisplayGpu::output(&d.device, input.width(), input.height()); + let lut = match &pass.color { + Some(effectcraft_engine::viewer::PassColor::Lut(l)) => Some(gpu.lut(&d.device, &d.queue, l)), + _ => None, + }; + gpu.run(&d.device, &d.queue, &input, &out, &super::uniform(pass), lut.as_ref()); + (read(d, &out, 0).0, out) +} + +fn frame(px: &[[f32; 4]], w: u32, encoding: FrameEncoding) -> Arc { + let mut img = Image::new(w, px.len() as u32 / w); + img.data.copy_from_slice(px); + Arc::new(F16Frame::from_image(&img, encoding)) +} + +/// GPU output against the reference on the frame's own (half-float) values: the number of +/// bytes that differ and the largest difference. +fn compare(got: &[[u8; 4]], f: &F16Frame, pass: &DisplayPass) -> (usize, i32) { + let (mut n, mut max) = (0, 0); + for (i, g) in got.iter().enumerate() { + let want = pass.pixel(f.pixel(i).expect("pixel")); + for c in 0..4 { + let d = (g[c] as i32 - want[c] as i32).abs(); + if d > 0 { + n += 1; + max = max.max(d); + } + } + } + (n, max) +} + +const SRGB: FrameEncoding = FrameEncoding { curve: PassCurve::Srgb, hdr: HdrMode::Clip }; + +fn grey(l: f32) -> [f32; 4] { + let e = PassCurve::Srgb.encode(l); + [e, e, e, 1.0] +} + +/// The issue's values (4.0 at -2 EV is white, 0.5 at +1 EV is white) and the HDR golden table, +/// bit for bit on the GPU. +#[test] +fn hdr_values_on_the_gpu() { + let _g = DEVICE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(d) = device() else { return eprintln!("no wgpu adapter: skipping") }; + let gpu = DisplayGpu::new(&d.device); + let ls = [0.25f32, 0.5, 1.0, 2.0, 4.0, 16.0]; + let f = frame(&ls.map(grey), 6, SRGB); + let want: [[u8; 6]; 5] = [ + [137, 188, 255, 255, 255, 255], + [99, 137, 188, 255, 255, 255], + [71, 99, 137, 188, 255, 255], + [34, 49, 71, 99, 137, 255], + [188, 255, 255, 255, 255, 255], + ]; + for (stops, w) in [0.0f32, -1.0, -2.0, -4.0, 1.0].into_iter().zip(want) { + let pass = DisplayPass::new(Channel::Rgb, false, stops, SRGB, None); + let (got, _) = bake(&d, &gpu, &Source::F16(f.clone()), &pass); + let r: Vec = got.iter().map(|p| p[0]).collect(); + assert!(r.iter().zip(w).all(|(g, w)| *g == w || d.cpu && g.abs_diff(w) <= 1), "{stops} EV: {r:?} vs {w:?}"); + let c = compare(&got, &f, &pass); + assert!(c == (0, 0) || d.cpu && c.1 <= 1, "{stops} EV: {c:?}"); + } +} + +/// Exposure 0 on half floats: bit for bit the reference, and the bytes the 8-bit viewer showed +/// for values in 0..1 (at least 97.9 % equal, never more than 1 apart: the half-float storage). +#[test] +fn exposure_zero_parity_on_the_gpu() { + let _g = DEVICE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(d) = device() else { return eprintln!("no wgpu adapter: skipping") }; + let gpu = DisplayGpu::new(&d.device); + let pass = DisplayPass::new(Channel::Rgb, false, 0.0, SRGB, None); + for alpha in [1.0f32, 0.5] { + let img = crate::frames::tests::parity_frame(alpha); + let f = Arc::new(F16Frame::from_image(&img, SRGB)); + let (got, _) = bake(&d, &gpu, &Source::F16(f.clone()), &pass); + let c = compare(&got, &f, &pass); + assert!(exact_enough(&d, c, got.len() * 4), "alpha {alpha}: {c:?}"); + let old = crate::frames::to_color_image(&img); + let (mut same, mut max) = (0, 0); + for (g, o) in got.iter().zip(&old.pixels) { + let o = o.to_array(); + for c in 0..3 { + same += (g[c] == o[c]) as usize; + max = max.max((g[c] as i32 - o[c] as i32).abs()); + } + assert_eq!(g[3], o[3]); + } + let share = same as f64 / (got.len() * 3) as f64; + eprintln!("GPU, alpha {alpha}: {:.2} % of colour bytes as before, max {max}", share * 100.0); + assert!(share >= 0.979 && max <= 1, "alpha {alpha}: {share}, max {max}"); + } + // The issue's two values, exactly. + let f = frame(&[grey(4.0), grey(0.5)], 2, SRGB); + let (m2, _) = bake(&d, &gpu, &Source::F16(f.clone()), &DisplayPass::new(Channel::Rgb, false, -2.0, SRGB, None)); + let (p1, _) = bake(&d, &gpu, &Source::F16(f), &DisplayPass::new(Channel::Rgb, false, 1.0, SRGB, None)); + let white = |p: [u8; 4]| p.iter().all(|c| *c == 255 || d.cpu && *c >= 254); + assert!(white(m2[0]) && white(p1[1]), "{:?} {:?}", m2[0], p1[1]); +} + +/// Every pass setting the viewer has, against the reference on the same half-float input: +/// exposure, HDR step (after exposure), Show Channel, curves and display colour. +#[test] +fn the_shader_is_the_reference() { + let _g = DEVICE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(d) = device() else { return eprintln!("no wgpu adapter: skipping") }; + let gpu = DisplayGpu::new(&d.device); + // 64 × 64: colours 0..2 in every channel, alpha 1, 0.5 and a few others, some over range. + let px: Vec<[f32; 4]> = (0..64 * 64) + .map(|i| { + let (x, y) = (i % 64, i / 64); + let a = [1.0, 0.5, 0.25, 0.8, 1.0, 1.0, 0.0, 1.0][y % 8]; + // Row 5: negative values too (32 bpc frames can hold them). + let lo = if y % 8 == 5 { 0.75 } else { 0.0 }; + let v = |k: usize| ((x * 7 + y * k) % 64) as f32 / 32.0 - lo; + [v(3) * a, v(5) * a, v(11) * a, a] + }) + .collect(); + let dcs = [ + None, + DisplayColor::new( + Some(effectcraft_engine::project::ColorSpace::Srgb), + false, + effectcraft_engine::project::ColorSpace::Srgb, + true, + effectcraft_engine::project::ColorSpace::DisplayP3, + Simulation::default(), + &Default::default(), + ), + DisplayColor::new( + Some(effectcraft_engine::project::ColorSpace::Srgb), + false, + effectcraft_engine::project::ColorSpace::Srgb, + true, + effectcraft_engine::project::ColorSpace::Srgb, + Simulation { profile: SimProfile::Rec709, preserve_rgb: false }, + &Default::default(), + ), + DisplayColor::new( + Some(effectcraft_engine::project::ColorSpace::Srgb), + false, + effectcraft_engine::project::ColorSpace::Srgb, + true, + effectcraft_engine::project::ColorSpace::DisplayP3, + Simulation { profile: SimProfile::Mac18, preserve_rgb: true }, + &Default::default(), + ), + // A LUT-based My Custom RGB profile. + Some(DisplayColor::from_lut(test_lut(1.0))), + ]; + let (mut report, mut off, mut total) = (vec![], 0usize, 0usize); + for curve in [PassCurve::Srgb, PassCurve::Gamma(2.4), PassCurve::Linear, PassCurve::Pq, PassCurve::Hlg] { + for hdr in [HdrMode::Clip, HdrMode::Compand, HdrMode::ToneMap] { + let enc = FrameEncoding { curve, hdr }; + let f = frame(&px, 64, enc); + for stops in [0.0f32, 1.0, -2.0, 2.5] { + for (ci, dc) in dcs.iter().enumerate() { + for (channel, colorized) in + [(Channel::Rgb, false), (Channel::RgbStraight, false), (Channel::Red, false), (Channel::Green, true), (Channel::Alpha, false)] + { + if ci > 0 && !matches!(channel, Channel::Rgb) { + continue; + } + let pass = DisplayPass::new(channel, colorized, stops, enc, dc.as_ref()); + let (got, _) = bake(&d, &gpu, &Source::F16(f.clone()), &pass); + let (n, max) = compare(&got, &f, &pass); + // The device's pow / exp / ln / division are not the CPU's to the last + // bit: a value on a rounding tie can land either side of it. + // PQ's exponent of 78.8 magnifies those last bits most. + // Software rasterizers (CI's lavapipe) differ more. + let share = if matches!(curve, PassCurve::Pq | PassCurve::Hlg) { 5 } else { 1 } * if d.cpu { 5 } else { 1 }; + assert!(max <= 1 && n * 100 <= got.len() * 4 * share, "{curve:?} {hdr:?} {stops} EV {channel:?} dc {ci}: {n} bytes off, max {max}"); + off += n; + total += got.len() * 4; + if n > 0 { + report.push(format!("{curve:?} {hdr:?} {stops} {channel:?} {ci}: {n}")); + } + } + } + } + } + } + eprintln!("{off} of {total} bytes 1 LSB off the reference: {report:?}"); +} + +/// 8-bit frames go through the same pass (exposure before the display colour there too), equal +/// to the CPU pass on their bytes. +#[test] +fn rgba8_frames_match_the_cpu_pass() { + let _g = DEVICE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(d) = device() else { return eprintln!("no wgpu adapter: skipping") }; + let gpu = DisplayGpu::new(&d.device); + let px: Vec = (0..256 * 16) + .map(|i| { + let (x, y) = (i % 256, i / 256); + let a = [255u8, 128, 64, 200][y % 4]; + let c = |k: usize| ((x * k + y) % 256) as u32 * a as u32 / 255; + egui::Color32::from_rgba_premultiplied(c(1) as u8, c(3) as u8, c(7) as u8, a) + }) + .collect(); + let img = Arc::new(egui::ColorImage::new([256, 16], px)); + let dc = DisplayColor::new( + Some(effectcraft_engine::project::ColorSpace::Srgb), + false, + effectcraft_engine::project::ColorSpace::Srgb, + true, + effectcraft_engine::project::ColorSpace::DisplayP3, + Simulation::default(), + &Default::default(), + ); + for stops in [0.0f32, 1.0, -1.5] { + for dc in [None, dc.as_ref()] { + let pass = DisplayPass::new(Channel::Rgb, false, stops, SRGB, dc); + let (got, _) = bake(&d, &gpu, &Source::Rgba8(img.clone()), &pass); + let mut want: Vec<[u8; 4]> = img.pixels.iter().map(|c| c.to_array()).collect(); + pass.apply_rgba8(&mut want); + let off = got.iter().zip(&want).flat_map(|(g, w)| (0..4).map(move |c| (g[c] as i32 - w[c] as i32).abs())).filter(|d| *d > 0).collect::>(); + eprintln!("8-bit, {stops} EV, dc {}: {} bytes 1 LSB off", dc.is_some(), off.len()); + assert!(off.iter().all(|d| *d <= 1) && off.len() * 100 <= got.len() * 4, "{stops} EV, dc {}: {off:?}", dc.is_some()); + if stops == 0.0 && dc.is_none() { + assert_eq!(got, want, "nothing to change: the frame's bytes"); + } + } + } +} + +/// The mips: each texel the rounded mean of the 2 × 2 texels above it, and at the edge of an +/// odd-sized level above also of its last column / row, so no frame pixel is dropped. +#[test] +fn mips_average_the_level_above() { + let _g = DEVICE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(d) = device() else { return eprintln!("no wgpu adapter: skipping") }; + let gpu = DisplayGpu::new(&d.device); + let px: Vec<[f32; 4]> = (0..37 * 21).map(|i| [(i % 37) as f32 / 36.0, (i / 37) as f32 / 20.0, 0.3, 1.0]).collect(); + let f = frame(&px, 37, SRGB); + let (_, tex) = bake(&d, &gpu, &Source::F16(f), &DisplayPass::new(Channel::Rgb, false, 0.5, SRGB, None)); + assert_eq!(tex.mip_level_count(), 6); + for l in 1..tex.mip_level_count() { + let (above, [aw, ah]) = read(&d, &tex, l - 1); + let (got, [w, h]) = read(&d, &tex, l); + for y in 0..h { + for x in 0..w { + let (x0, y0) = ((x * 2).min(aw - 1), (y * 2).min(ah - 1)); + let x1 = if x == w - 1 { aw } else { (x * 2 + 2).min(aw) }; + let y1 = if y == h - 1 { ah } else { (y * 2 + 2).min(ah) }; + let n = ((x1 - x0) * (y1 - y0)) as f32; + let want: [u8; 4] = std::array::from_fn(|c| { + let s: f32 = + (y0..y1).flat_map(|yy| (x0..x1).map(move |xx| (xx, yy))).map(|(xx, yy)| above[(yy * aw + xx) as usize][c] as f32 / 255.0).sum(); + (s / n * 255.0 + 0.5).floor() as u8 + }); + let g = got[(y * w + x) as usize]; + assert!((0..4).all(|c| (g[c] as i32 - want[c] as i32).abs() <= 1), "level {l} ({x}, {y}): {g:?} vs {want:?}"); + } + } + } +} + +/// One egui frame through egui_wgpu's renderer into an offscreen target (eframe minus the +/// swapchain): RGBA8 of the framebuffer. +fn render_egui( + d: &Dev, + renderer: &mut egui_wgpu::Renderer, + ctx: &egui::Context, + size_pts: [f32; 2], + ppp: f32, + ui_fn: &mut dyn FnMut(&mut egui::Ui), +) -> (Vec<[u8; 4]>, [u32; 2]) { + ctx.set_pixels_per_point(ppp); + let raw = egui::RawInput { screen_rect: Some(egui::Rect::from_min_size(egui::Pos2::ZERO, egui::vec2(size_pts[0], size_pts[1]))), ..Default::default() }; + let mut first = ctx.run_ui(raw.clone(), |ui| ui_fn(ui)); + let mut out = ctx.run_ui(raw, |ui| ui_fn(ui)); + for o in [&mut first, &mut out] { + for (id, delta) in o.textures_delta.set.iter().flat_map(|(id, v)| v.iter().map(move |d| (id, d))) { + renderer.update_texture(&d.device, &d.queue, *id, delta); + } + for id in &o.textures_delta.free { + renderer.free_texture(id); + } + o.textures_delta.clear(); + } + let jobs = ctx.tessellate(out.shapes, out.pixels_per_point); + let size = [(size_pts[0] * ppp).round() as u32, (size_pts[1] * ppp).round() as u32]; + let sd = egui_wgpu::ScreenDescriptor { size_in_pixels: size, pixels_per_point: out.pixels_per_point }; + let target = d.device.create_texture(&wgpu::TextureDescriptor { + label: Some("framebuffer"), + size: wgpu::Extent3d { width: size[0], height: size[1], depth_or_array_layers: 1 }, + mip_level_count: 1, + sample_count: 1, + dimension: wgpu::TextureDimension::D2, + format: wgpu::TextureFormat::Rgba8Unorm, + usage: wgpu::TextureUsages::RENDER_ATTACHMENT | wgpu::TextureUsages::COPY_SRC, + view_formats: &[], + }); + let view = target.create_view(&Default::default()); + let mut enc = d.device.create_command_encoder(&Default::default()); + let user = renderer.update_buffers(&d.device, &d.queue, &mut enc, &jobs, &sd); + { + let pass = enc.begin_render_pass(&wgpu::RenderPassDescriptor { + label: Some("egui"), + color_attachments: &[Some(wgpu::RenderPassColorAttachment { + view: &view, + depth_slice: None, + resolve_target: None, + ops: wgpu::Operations { load: wgpu::LoadOp::Clear(wgpu::Color::TRANSPARENT), store: wgpu::StoreOp::Store }, + })], + ..Default::default() + }); + let mut pass = pass.forget_lifetime(); + renderer.render(&mut pass, &jobs, &sd); + } + d.queue.submit(user.into_iter().chain([enc.finish()])); + (read(d, &target, 0).0, size) +} + +/// The displayed texture drawn by egui as the viewer draws it (native texture, the Faster +/// sampler: nearest): at 2 pixels per point, zoomed 8× and 13.37× with the frame past every +/// edge of a clip rect, each screen pixel shows the texel under its centre; and over egui's +/// checkerboard, half-transparent pixels blend premultiplied. +#[test] +fn the_displayed_texture_lands_where_the_viewer_draws_it() { + let _g = DEVICE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(d) = device() else { return eprintln!("no wgpu adapter: skipping") }; + let gpu = DisplayGpu::new(&d.device); + let mut renderer = + egui_wgpu::Renderer::new(&d.device, wgpu::TextureFormat::Rgba8Unorm, egui_wgpu::RendererOptions { dithering: false, ..Default::default() }); + let ctx = egui::Context::default(); + let (tw, th) = (64u32, 32u32); + let px: Vec<[f32; 4]> = (0..tw * th).map(|i| [(i % tw) as f32 / 63.0, (i / tw) as f32 / 31.0, 1.0, 1.0]).collect(); + let f = frame(&px, tw, SRGB); + let pass = DisplayPass::new(Channel::Rgb, false, -1.0, SRGB, None); + let (texels, tex) = bake(&d, &gpu, &Source::F16(f), &pass); + let id = + renderer.register_native_texture_with_sampler_options(&d.device, &tex.create_view(&Default::default()), crate::panels::viewer::zoom_sampler(false)); + let ppp = 2.0f32; + let clip = egui::Rect::from_min_max(egui::pos2(50.0, 40.0), egui::pos2(350.0, 260.0)); + for (zoom, origin) in [(2.0f32, egui::pos2(80.0, 60.0)), (8.0, egui::pos2(-100.0, -60.0)), (13.37, egui::pos2(-151.3, -77.7))] { + let rect = egui::Rect::from_min_size(origin, egui::vec2(tw as f32 * zoom, th as f32 * zoom)); + let mut ui_fn = |ui: &mut egui::Ui| { + let p = ui.painter().with_clip_rect(clip); + p.image(id, rect, egui::Rect::from_min_max(egui::pos2(0.0, 0.0), egui::pos2(1.0, 1.0)), egui::Color32::WHITE); + }; + let (out, sz) = render_egui(&d, &mut renderer, &ctx, [400.0, 300.0], ppp, &mut ui_fn); + let (mut bad, mut drawn) = (0, 0); + for y in 0..sz[1] { + for x in 0..sz[0] { + let (cx, cy) = ((x as f32 + 0.5) / ppp, (y as f32 + 0.5) / ppp); + let (u, v) = ((cx - rect.min.x) / zoom, (cy - rect.min.y) / zoom); + // Centres within a hair of a texel or clip edge can go either way. + let near = |t: f32| (t - t.round()).abs() < 0.01; + if near(u) + || near(v) + || near(cx * ppp - clip.min.x * ppp) + || !clip.contains(egui::pos2(cx, cy)) + || u < 0.0 + || v < 0.0 + || u >= tw as f32 + || v >= th as f32 + { + continue; + } + drawn += 1; + let want = texels[(v as u32 * tw + u as u32) as usize]; + let got = out[(y * sz[0] + x) as usize]; + if (0..4).any(|c| (got[c] as i32 - want[c] as i32).abs() > 1) { + bad += 1; + } + } + } + assert!(drawn > 1000, "{drawn}"); + assert_eq!(bad, 0, "zoom {zoom}, origin {origin:?}: {bad} of {drawn} pixels wrong"); + } + renderer.free_texture(&id); + // Alpha 0.5 over a checkerboard. + let src = [0.5f32, 0.25, 0.0, 0.5]; + let f = frame(&[src; 256], 16, SRGB); + let (texels, tex) = bake(&d, &gpu, &Source::F16(f), &DisplayPass::new(Channel::Rgb, false, 0.0, SRGB, None)); + let id = + renderer.register_native_texture_with_sampler_options(&d.device, &tex.create_view(&Default::default()), crate::panels::viewer::zoom_sampler(false)); + let (c1, c2) = (egui::Color32::from_gray(0x66), egui::Color32::from_gray(0x99)); + let mut ui_fn = |ui: &mut egui::Ui| { + let p = ui.painter(); + for j in 0..8 { + for i in 0..8 { + let r = egui::Rect::from_min_size(egui::pos2(i as f32 * 8.0, j as f32 * 8.0), egui::vec2(8.0, 8.0)); + p.rect_filled(r, 0.0, if (i + j) % 2 == 0 { c1 } else { c2 }); + } + } + p.image( + id, + egui::Rect::from_min_size(egui::Pos2::ZERO, egui::vec2(64.0, 64.0)), + egui::Rect::from_min_max(egui::pos2(0.0, 0.0), egui::pos2(1.0, 1.0)), + egui::Color32::WHITE, + ); + }; + let (out, sz) = render_egui(&d, &mut renderer, &ctx, [64.0, 64.0], 2.0, &mut ui_fn); + let s = texels[0]; + let mut bad = 0; + for y in 0..sz[1] { + for x in 0..sz[0] { + let (i, j) = (x / 16, y / 16); + let dst = if (i + j) % 2 == 0 { 0x66 } else { 0x99 } as f32; + let want: [u8; 3] = std::array::from_fn(|c| (s[c] as f32 + dst * (1.0 - s[3] as f32 / 255.0)).round() as u8); + let got = out[(y * sz[0] + x) as usize]; + if (0..3).any(|c| (got[c] as i32 - want[c] as i32).abs() > 1) { + bad += 1; + } + } + } + assert_eq!(bad, 0, "alpha over the checkerboard: {bad} pixels off"); +} + +/// Timings at 4K (`cargo test -- --ignored display_pass_timings --nocapture`): the f32 → half +/// conversion of a frame, today's f32 → RGBA8 conversion, and the display pass. +#[test] +#[ignore = "timing, not a check"] +fn display_pass_timings() { + let _g = DEVICE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let img = Image::filled(3840, 2160, [0.7, 1.9, 0.2, 1.0]); + let time = |f: &mut dyn FnMut()| { + let mut v: Vec = (0..9) + .map(|_| { + let t = std::time::Instant::now(); + f(); + t.elapsed().as_secs_f64() * 1000.0 + }) + .collect(); + v.sort_by(f64::total_cmp); + v[4] + }; + let f16 = time(&mut || drop(F16Frame::from_image(&img, SRGB))); + let rgba8 = time(&mut || drop(crate::frames::to_color_image(&img))); + eprintln!("4K: f32 → f16 {f16:.2} ms, f32 → RGBA8 {rgba8:.2} ms (medians of 9)"); + let Some(d) = device() else { return }; + let gpu = DisplayGpu::new(&d.device); + let f = Arc::new(F16Frame::from_image(&img, SRGB)); + let input = gpu.upload(&d.device, &d.queue, &Source::F16(f), 1, None); + let out = DisplayGpu::output(&d.device, 3840, 2160); + let params = super::uniform(&DisplayPass::new(Channel::Rgb, false, -1.0, FrameEncoding { curve: PassCurve::Srgb, hdr: HdrMode::ToneMap }, None)); + let pass = time(&mut || { + gpu.run(&d.device, &d.queue, &input, &out, ¶ms, None); + d.device.poll(wgpu::PollType::wait_indefinitely()).expect("poll"); + }); + let f = Arc::new(F16Frame::from_image(&img, SRGB)); + let mut kept = Some(gpu.upload(&d.device, &d.queue, &Source::F16(f.clone()), 1, None)); + let upload = time(&mut || { + kept = Some(gpu.upload(&d.device, &d.queue, &Source::F16(f.clone()), 1, kept.take())); + d.device.poll(wgpu::PollType::wait_indefinitely()).expect("poll"); + }); + let cpu = time(&mut || drop(f.to_color_image(&DisplayPass::new(Channel::Rgb, false, -1.0, SRGB, None)))); + eprintln!("4K: half-float upload into the kept texture {upload:.2} ms; the pass on the CPU (no device) {cpu:.2} ms"); + eprintln!("4K: display pass with mips, submit to idle {pass:.2} ms"); +} + +/// `Bake::show` on egui-wgpu's render state: it runs the pass again when the frame, the pass's +/// settings or the LUT change (a new LUT of the same size included: another ICC profile, +/// Preserve RGB, another display), and only then; the texture id stays. +#[test] +fn bakes_run_again_when_frame_pass_or_lut_change() { + let _g = DEVICE.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let Some(d) = device() else { return eprintln!("no wgpu adapter: skipping") }; + let gpu = DisplayGpu::new(&d.device); + let rs = render_state(&d); + let px: Vec<[f32; 4]> = (0..16).map(|i| [i as f32 / 15.0, 0.4, 0.7, 1.0]).collect(); + let f = frame(&px, 4, SRGB); + let lut = |k: f64| DisplayColor::from_lut(test_lut(k)); + let (l1, l2) = (lut(1.0), lut(0.5)); + let pass = |stops: f32, dc: &DisplayColor| DisplayPass::new(Channel::Rgb, false, stops, SRGB, Some(dc)); + let mut b = super::Bake::default(); + let id = b.show(&rs, &gpu, &Source::F16(f.clone()), &pass(0.0, &l1), true).unwrap(); + let first = read(&d, b.texture().unwrap(), 0).0; + assert_eq!(b.runs(), 1); + b.show(&rs, &gpu, &Source::F16(f.clone()), &pass(0.0, &l1), true); + assert_eq!(b.runs(), 1, "nothing changed"); + assert_eq!(b.show(&rs, &gpu, &Source::F16(f.clone()), &pass(0.0, &l2), true), Some(id)); + assert_eq!(b.runs(), 2, "a new LUT of the same size"); + let second = read(&d, b.texture().unwrap(), 0).0; + assert_ne!(first, second); + let want: Vec<[u8; 4]> = (0..16).map(|i| pass(0.0, &l2).pixel(f.pixel(i).unwrap())).collect(); + assert!(second.iter().zip(&want).all(|(g, w)| (0..4).all(|c| g[c].abs_diff(w[c]) <= 1)), "{second:?} vs {want:?}"); + b.show(&rs, &gpu, &Source::F16(f.clone()), &pass(-1.0, &l2), true); + assert_eq!(b.runs(), 3, "another exposure"); + b.show(&rs, &gpu, &Source::F16(frame(&px, 4, SRGB)), &pass(-1.0, &l2), true); + assert_eq!(b.runs(), 4, "another frame"); + b.free(Some(&rs)); + assert!(b.id().is_none()); +} diff --git a/crates/ui-egui/src/frames.rs b/crates/ui-egui/src/frames.rs index 788af49e8..001790116 100644 --- a/crates/ui-egui/src/frames.rs +++ b/crates/ui-egui/src/frames.rs @@ -23,6 +23,10 @@ //! to the GPU) frames stay on the GPU: the compositor writes an RGBA8 texture that the viewer //! registers with egui-wgpu and draws directly; pixels are read back only when something needs //! them (Info panel, eyedroppers, histograms). +//! +//! 32 bpc projects keep half-float frames on the desktop ([`F16Frame`], rendered without the HDR +//! step): twice the bytes of an 8-bit frame, so the RAM preview holds half as many. The disk +//! cache keeps them as half-float entries. use std::collections::{BTreeSet, HashMap, HashSet, VecDeque}; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; @@ -47,10 +51,14 @@ const MIN_GPU_BUDGET: usize = 64 << 20; const GPU_PRESSURE_COOLDOWN: std::time::Duration = std::time::Duration::from_secs(5); const GPU_RECOVERY_INTERVAL: std::time::Duration = std::time::Duration::from_secs(30); -/// A rendered viewer frame: CPU pixels or a GPU texture. +mod f16; +pub use f16::{F16_PIXEL_BYTES, F16Frame}; + +/// A rendered viewer frame: 8-bit CPU pixels, half-float CPU pixels (32 bpc) or a GPU texture. #[derive(Clone)] pub enum FrameImage { Cpu(Arc), + CpuF16(Arc), Gpu(Arc), } @@ -58,11 +66,14 @@ impl FrameImage { pub fn size(&self) -> [usize; 2] { match self { FrameImage::Cpu(c) => c.size, + FrameImage::CpuF16(f) => f.size(), FrameImage::Gpu(f) => [f.width as usize, f.height as usize], } } - fn bytes(&self) -> usize { - self.size()[0] * self.size()[1] * 4 + /// Bytes the frame holds in the RAM preview (GPU frames: without their mips). + pub fn bytes(&self) -> usize { + let [w, h] = self.size(); + w * h * if matches!(self, FrameImage::CpuF16(_)) { F16_PIXEL_BYTES } else { 4 } } fn is_gpu(&self) -> bool { matches!(self, FrameImage::Gpu(_)) @@ -615,6 +626,29 @@ impl Default for Frames { } } +/// Whether viewer frames of a project with `settings` are half floats ([`F16Frame`], rendered +/// with `RenderOpts::skip_hdr_step`): 32 bpc on the desktop. The browser's frames (its frame +/// worker's RGBA8, the GPU compositor's display textures) stay 8-bit for now. +pub fn f16_frames(settings: &effectcraft_engine::project::ProjectSettings) -> bool { + cfg!(not(target_arch = "wasm32")) && settings.bit_depth.levels().is_none() +} + +/// Bytes per pixel of the viewer frames of a project with `settings` (see [`f16_frames`]). +pub fn frame_pixel_bytes(settings: &effectcraft_engine::project::ProjectSettings) -> usize { + if f16_frames(settings) { F16_PIXEL_BYTES } else { 4 } +} + +/// A rendered frame as a cached viewer frame: half floats for `skip_hdr_step` renders (see +/// [`f16_frames`]), else 8-bit. +pub fn cpu_frame(img: &effectcraft_engine::render::Image, project: &Project, opts: &RenderOpts) -> FrameImage { + if opts.skip_hdr_step { + let encoding = effectcraft_engine::viewer::FrameEncoding::of(&project.settings, true); + FrameImage::CpuF16(Arc::new(F16Frame::from_image(img, encoding))) + } else { + FrameImage::Cpu(Arc::new(to_color_image(img))) + } +} + /// Premultiplied f32 → egui premultiplied Color32 (alpha kept, so the viewer can show the /// transparency grid or the comp background colour underneath). pub fn to_color_image(img: &effectcraft_engine::render::Image) -> egui::ColorImage { @@ -974,7 +1008,7 @@ impl Frames { self.request_content_key(src.project.clone(), base.revision, base.comp); return true; }; - let (missing, converting): (Vec<(u128, Arc)>, bool) = { + let (missing, converting): (Vec<(u128, FrameImage)>, bool) = { let (Ok(c), Ok(mut queued), Ok(mut written)) = (self.cache.lock(), self.persisted.lock(), self.written.lock()) else { return false }; if queued.len() > 100_000 { queued.clear(); @@ -988,7 +1022,10 @@ impl Frames { .iter() .filter(|(k, _)| k.same_series(base)) .filter_map(|(k, img)| { - let FrameImage::Cpu(img) = img else { return None }; + // (GPU frames have no CPU pixels to write) + if img.is_gpu() { + return None; + } let dk = disk_cache::frame_key(content, k.frame, k.scale, k.view ^ oh); let tries = written.1.get(&dk).copied().unwrap_or(0); (tries < 2 && !dc.contains(disk_cache::Kind::Frame, dk) && !dc.frame_pending(dk) && queued.insert(dk)).then(|| (dk, img.clone())) @@ -1002,12 +1039,17 @@ impl Frames { let (persisted, written) = (self.persisted.clone(), self.written.clone()); let work = move || { for (dk, img) in missing { - let rgba: Vec = img.pixels.iter().flat_map(|c| c.to_array()).collect(); // (accepted: the disk cache tracks it until written, and it counts as a try; // refused by a full queue: retried, without counting) - if dc.put_frame(dk, img.size[0] as u32, img.size[1] as u32, &rgba) - && let Ok(mut w) = written.lock() - { + let accepted = match &img { + FrameImage::Cpu(img) => { + let rgba: Vec = img.pixels.iter().flat_map(|c| c.to_array()).collect(); + dc.put_frame(dk, img.size[0] as u32, img.size[1] as u32, &rgba) + } + FrameImage::CpuF16(f) => dc.put_frame_f16(dk, f.width, f.height, f.bits()), + FrameImage::Gpu(_) => false, + }; + if accepted && let Ok(mut w) = written.lock() { let n = w.1.entry(dk).or_default(); *n = n.saturating_add(1); } @@ -1563,10 +1605,20 @@ impl Worker { .disk .as_ref() .map(|dc| (dc.clone(), frame_disk_key(content_keys, &job.src.project, &job.key, opts_hash(&job.opts), epoch, self.virtual_id.as_deref()))); - let from_disk = disk.as_ref().and_then(|(dc, k)| dc.get_frame(*k)); + // Half-float frames (32 bpc) have entries of their own (their key has the options). + let from_disk = disk.as_ref().and_then(|(dc, k)| { + if job.opts.skip_hdr_step { + let f = dc.get_frame_f16(*k)?; + let encoding = effectcraft_engine::viewer::FrameEncoding::of(&job.src.project.settings, true); + Some(FrameImage::CpuF16(Arc::new(F16Frame::from_bits(f.width, f.height, f.rgba, encoding)))) + } else { + let f = dc.get_frame(*k)?; + Some(FrameImage::Cpu(Arc::new(egui::ColorImage::from_rgba_premultiplied([f.width as usize, f.height as usize], &f.rgba)))) + } + }); let disk_hit = from_disk.is_some(); let ci = match from_disk { - Some(f) => FrameImage::Cpu(Arc::new(egui::ColorImage::from_rgba_premultiplied([f.width as usize, f.height as usize], &f.rgba))), + Some(f) => f, None => match self.render_guarded(&job).filter(|_| !job.cancel.load(Ordering::Relaxed)) { Some(ci) => ci, // Called off (frames it read may be missing) or failed: not cached, and not @@ -1580,10 +1632,20 @@ impl Worker { // Publish the shared frame and wake the viewer before optional RGBA conversion // and compression. Persistence still runs on this worker, using the same snapshot. self.finish(&job, ci.clone(), t0); - if !disk_hit && let (Some((dc, k)), FrameImage::Cpu(img)) = (&disk, &ci) { - let rgba: Vec = img.pixels.iter().flat_map(|c| c.to_array()).collect(); - dc.put_frame(*k, img.size[0] as u32, img.size[1] as u32, &rgba); - if let Some(ctx) = &self.ctx { + if !disk_hit && let Some((dc, k)) = &disk { + let written = match &ci { + FrameImage::Cpu(img) => { + let rgba: Vec = img.pixels.iter().flat_map(|c| c.to_array()).collect(); + dc.put_frame(*k, img.size[0] as u32, img.size[1] as u32, &rgba); + true + } + FrameImage::CpuF16(f) => { + dc.put_frame_f16(*k, f.width, f.height, f.bits()); + true + } + FrameImage::Gpu(_) => false, + }; + if written && let Some(ctx) = &self.ctx { ctx.request_repaint(); } } @@ -1745,18 +1807,18 @@ impl Worker { } } let t0 = web_time::Instant::now(); - let img = to_color_image(&r.comp_frame_cpu(job.comp, job.t)); + let img = cpu_frame(&r.comp_frame_cpu(job.comp, job.t), &job.src.project, &job.opts); if let Some((p, key)) = auto { p.record(key, false, t0.elapsed().as_secs_f64() * 1000.0); } - return FrameImage::Cpu(Arc::new(img)); + return img; } if let Some(g) = gpu { // The GPU composites and the frame is read back. return match g.within_memory(|| r.comp_frame(job.comp, job.t)) { Ok(img) => { self.gpu_succeeded(); - FrameImage::Cpu(Arc::new(to_color_image(&img))) + cpu_frame(&img, &job.src.project, &job.opts) } Err(e) => { self.gpu_attempt_failed(g, &e); @@ -1764,7 +1826,7 @@ impl Worker { } }; } - FrameImage::Cpu(Arc::new(to_color_image(&r.comp_frame(job.comp, job.t)))) + cpu_frame(&r.comp_frame(job.comp, job.t), &job.src.project, &job.opts) } } @@ -1863,4 +1925,4 @@ pub(crate) fn render_scale(thousandths: u32, factor: Option) -> f64 { } #[cfg(test)] -mod tests; +pub(crate) mod tests; diff --git a/crates/ui-egui/src/frames/f16.rs b/crates/ui-egui/src/frames/f16.rs new file mode 100644 index 000000000..5d00f0ca6 --- /dev/null +++ b/crates/ui-egui/src/frames/f16.rs @@ -0,0 +1,91 @@ +//! Half-float viewer frames (32 bpc projects on the desktop, #2797): the RAM preview keeps the +//! rendered frame's full range, without the HDR step, so viewer exposure, Compand / Tone Map and +//! the display colour run on it in the display pass (`crate::display_gpu`), in that order. + +use effectcraft_engine::render::Image; +use effectcraft_engine::viewer::{DisplayPass, FrameEncoding}; +use half::f16; +use half::slice::{HalfBitsSliceExt, HalfFloatSliceExt}; +use rayon::prelude::*; + +/// Values converted per piece: 16 KB of input, so the piece stays in L1 while its non-finite +/// values are fixed up. +const BLOCK: usize = 4096; + +/// A premultiplied RGBA frame in half floats, encoded in the output space as rendered (values +/// above 1.0 kept), with how it is encoded. Never NaN or infinite (see [`F16Frame::from_image`]): +/// the display pass relies on it. +pub struct F16Frame { + pub width: u32, + pub height: u32, + /// RGBA, row by row, as IEEE binary16 bits. + data: Vec, + pub encoding: FrameEncoding, +} + +/// Bytes per pixel of a half-float frame. +pub const F16_PIXEL_BYTES: usize = 8; + +/// Half floats that are NaN become 0, infinities ±65504 (one pass, no branches on the data). +fn finite(bits: &mut [u16]) { + for b in bits { + let v = *b; + let special = v & 0x7c00 == 0x7c00; + let fixed = if v & 0x03ff != 0 { 0 } else { (v & 0x8000) | 0x7bff }; + *b = if special { fixed } else { v }; + } +} + +impl F16Frame { + /// A rendered frame in half floats. NaN becomes 0 (NaN alpha is transparent, as in + /// `display_pixel`) and values past the half-float range become ±65504. + pub fn from_image(img: &Image, encoding: FrameEncoding) -> F16Frame { + let src = img.data.as_flattened(); + // A zeroed allocation: fresh pages from the OS, no fill pass. + let mut data = vec![0u16; src.len()]; + data.par_chunks_mut(BLOCK).zip(src.par_chunks(BLOCK)).for_each(|(out, inp)| { + out.reinterpret_cast_mut::().convert_from_f32_slice(inp); + finite(out); + }); + F16Frame { width: img.width, height: img.height, data, encoding } + } + + /// A frame from binary16 bits (the disk cache), made finite as [`F16Frame::from_image`] does. + pub fn from_bits(width: u32, height: u32, mut data: Vec, encoding: FrameEncoding) -> F16Frame { + data.resize(width as usize * height as usize * 4, 0); + data.par_chunks_mut(BLOCK).for_each(finite); + F16Frame { width, height, data, encoding } + } + + /// The pixels as binary16 bits (RGBA, row by row). + pub fn bits(&self) -> &[u16] { + &self.data + } + + pub fn size(&self) -> [usize; 2] { + [self.width as usize, self.height as usize] + } + + pub fn bytes(&self) -> usize { + self.data.len() * 2 + } + + /// Pixel `i` (row-major) as `f32`. + pub fn pixel(&self, i: usize) -> Option<[f32; 4]> { + let p = self.data.get(i.checked_mul(4)?..i * 4 + 4)?; + Some([0, 1, 2, 3].map(|k| f16::from_bits(p[k]).to_f32())) + } + + /// The frame through `pass` on the CPU (no GPU, or a single sample). + pub fn to_color_image(&self, pass: &DisplayPass) -> egui::ColorImage { + let px: Vec = self + .data + .par_chunks(4) + .map(|p| { + let [r, g, b, a] = pass.pixel([0, 1, 2, 3].map(|k| f16::from_bits(p[k]).to_f32())); + egui::Color32::from_rgba_premultiplied(r, g, b, a) + }) + .collect(); + egui::ColorImage::new(self.size(), px) + } +} diff --git a/crates/ui-egui/src/frames/tests.rs b/crates/ui-egui/src/frames/tests.rs index be96ecf30..df7b3900c 100644 --- a/crates/ui-egui/src/frames/tests.rs +++ b/crates/ui-egui/src/frames/tests.rs @@ -1136,6 +1136,175 @@ fn timing_markers_and_expressions_still_change_the_frames() { assert_ne!(at(&s), o2, "an expression can read the marker, so the frames change"); } +const SRGB: effectcraft_engine::viewer::FrameEncoding = + effectcraft_engine::viewer::FrameEncoding { curve: effectcraft_engine::viewer::PassCurve::Srgb, hdr: effectcraft_engine::project::HdrMode::Clip }; + +/// Half floats keep over-range values (the point of #2797); NaN becomes 0 and values past the +/// half-float range ±65504. +#[test] +fn half_float_frames_keep_over_range_values() { + let mut img = effectcraft_engine::render::Image::new(3, 1); + img.data.copy_from_slice(&[[4.0, 16.0, 1.825, 1.0], [70_000.0, f32::NAN, -f32::INFINITY, 0.5], [0.1, -0.25, 1000.0, f32::NAN]]); + let f = F16Frame::from_image(&img, SRGB); + assert_eq!(f.pixel(0), Some([4.0, 16.0, half::f16::from_f32(1.825).to_f32(), 1.0])); + assert_eq!(f.pixel(1), Some([65504.0, 0.0, -65504.0, 0.5])); + let p = f.pixel(2).unwrap(); + assert_eq!((p[2], p[3]), (1000.0, 0.0)); + assert!((p[0] - 0.1).abs() < 1e-4 && (p[1] + 0.25).abs() < 1e-4); + assert_eq!(f.bytes(), 3 * F16_PIXEL_BYTES); +} + +/// Oversized half-float frames take the CPU pass: display conversion, then averaging down. Linear +/// 8 and 0 alternating in Clip mode at -2 EV is white and black (byte 128 averaged); averaging the +/// scene values first would give 4.0 x 0.25 = 1.0, byte 255. +#[test] +fn oversized_half_float_frames_convert_before_averaging() { + use crate::display_gpu::Source; + use effectcraft_engine::viewer::{Channel, DisplayPass}; + let mut img = effectcraft_engine::render::Image::new(2, 1); + img.data.copy_from_slice(&[[8.0, 8.0, 8.0, 1.0], [0.0, 0.0, 0.0, 1.0]]); + let lin = + effectcraft_engine::viewer::FrameEncoding { curve: effectcraft_engine::viewer::PassCurve::Linear, hdr: effectcraft_engine::project::HdrMode::Clip }; + let f = Arc::new(F16Frame::from_image(&img, lin)); + let pass = DisplayPass::new(Channel::Rgb, false, -2.0, lin, None); + let src = Source::F16(f.clone()); + assert!(src.needs_cpu_pass(1) && !src.needs_cpu_pass(2)); + assert!(!Source::Rgba8(Arc::new(egui::ColorImage::new([2, 1], vec![egui::Color32::WHITE; 2]))).needs_cpu_pass(1), "8-bit frames are averaged on the GPU"); + let out = fit_texture_by(f.to_color_image(&pass), 1, 1); + assert_eq!(out.size, [1, 1]); + assert_eq!(out.pixels[0].to_array(), [128, 128, 128, 255]); +} + +/// 65 536 colours spread over 0..1 (each channel on its own 16-bit grid), premultiplied by +/// `alpha`: the 8-bit parity input. +pub(crate) fn parity_frame(alpha: f32) -> effectcraft_engine::render::Image { + let mut img = effectcraft_engine::render::Image::new(256, 256); + for (i, p) in img.data.iter_mut().enumerate() { + let v = |k: usize| ((i * k) % 65_536) as f32 / 65_535.0; + *p = [v(1) * alpha, v(7_919) * alpha, v(104_729) * alpha, alpha]; + } + img +} + +/// At 0 EV the half-float viewer shows the bytes of the old 8-bit frames for values in 0..1: at +/// least 97.9 % of the colour bytes equal (alpha all), never more than 1 apart. The rest is the +/// half-float storage. +#[test] +fn half_float_frames_show_what_8_bit_frames_showed() { + use effectcraft_engine::viewer::{Channel, DisplayPass}; + let pass = DisplayPass::new(Channel::Rgb, false, 0.0, SRGB, None); + for alpha in [1.0f32, 0.5] { + let img = parity_frame(alpha); + let f = F16Frame::from_image(&img, SRGB); + let new = f.to_color_image(&pass); + let old = to_color_image(&img); + let (mut same, mut total, mut max) = (0, 0, 0); + for (n, o) in new.pixels.iter().zip(&old.pixels) { + let (n, o) = (n.to_array(), o.to_array()); + assert_eq!(n[3], o[3]); + for c in 0..3 { + total += 1; + same += (n[c] == o[c]) as usize; + max = max.max((n[c] as i32 - o[c] as i32).abs()); + } + } + let share = same as f64 / total as f64; + eprintln!("alpha {alpha}: {:.2} % of colour bytes equal, max {max}", share * 100.0); + assert!(share >= 0.979 && max <= 1, "alpha {alpha}: {share} equal, max {max}"); + } +} + +/// The RAM preview counts half-float frames at 8 bytes a pixel: in a budget of three 8-bit +/// frames, a half-float frame of the same size takes the room of two. +#[test] +fn half_float_frames_count_twice_the_bytes() { + let f16 = || FrameImage::CpuF16(Arc::new(F16Frame::from_image(&effectcraft_engine::render::Image::new(2, 2), SRGB))); + assert_eq!(f16().bytes(), 32); + assert_eq!(img().bytes(), 16); + let frames = Frames::default(); + frames.set_budget(48); + for i in 0..3 { + frames.cache.lock().unwrap().insert(key(1, i, 0), img()); + } + assert_eq!(frames.cached_frames(&key(1, 0, 0)).len(), 3); + frames.cache.lock().unwrap().insert(key(2, 0, 0), f16()); + let c = frames.cache.lock().unwrap(); + assert!(c.bytes <= 48, "{}", c.bytes); + assert_eq!(c.map.len(), 2, "the half-float frame took two 8-bit frames' room"); +} + +/// 32 bpc viewer frames go to the disk cache as half floats and come back from it bit for bit +/// in a new session (over-range values kept); the timeline's disk bar sees them. +#[test] +fn half_float_frames_survive_in_the_disk_cache() { + use effectcraft_engine::Session; + use serde_json::json; + // Removed on drop, a failed assertion included. + let dir = tempfile::tempdir().unwrap(); + let disk = DiskCache::open(dir.path(), 1 << 30).unwrap(); + let mut s = Session::default(); + let cid = ItemId(s.execute("comp.new", json!({"width": 16, "height": 8, "duration": 1})).unwrap()["comp"].as_u64().unwrap()); + s.execute("file.projectSettings", json!({"bitDepth": 32, "workingSpace": "srgb", "hdr": "compand"})).unwrap(); + let lid = s.execute("layer.newSolid", json!({"color": "#ffffff"})).unwrap()["layer"].as_u64().unwrap(); + s.execute("effect.apply", json!({"layers": [lid], "effect": "Exposure"})).unwrap(); + s.execute("prop.set", json!({"layer": lid, "path": "effects/#1/master/exposure", "value": 2.0})).unwrap(); + let src = |disk: &Arc| RenderSource { + project: s.project.clone(), + footage: s.footage.clone(), + expr: None, + layer_cache: Arc::default(), + gpu: None, + gpu_display: false, + disk: Some(disk.clone()), + }; + let opts = RenderOpts { skip_hdr_step: true, ..Default::default() }; + let shown = |f: &Frames, disk: &Arc| { + let key = FrameKey { + revision: s.revision, + content: f.content_of(&s.project, s.revision, cid), + comp: cid.0, + frame: 0, + scale: 1000, + view: 0, + opts: opts_hash(&opts), + }; + f.request_urgent(&src(disk), key, cid, Tick::ZERO, opts); + wait_for_renders(f); + match f.get(&key) { + Some(FrameImage::CpuF16(img)) => (key, img), + _ => panic!("a half-float frame"), + } + }; + let first = Frames::default(); + let (key, rendered) = shown(&first, &disk); + assert!(rendered.pixel(0).unwrap()[0] > 1.0, "over range, without the HDR step"); + // The worker releases the frame before it queues the disk write: wait for the entry, not + // only for the queue to drain. + for _ in 0..400 { + disk.flush(); + if disk.stats().entries >= 1 { + break; + } + std::thread::sleep(std::time::Duration::from_millis(5)); + } + assert_eq!(disk.stats().entries, 1, "written as a frame entry"); + // The timeline's blue bar: on disk, not in this (new) RAM cache. + let again = Frames::default(); + let key2 = FrameKey { content: again.content_of(&s.project, s.revision, cid), ..key }; + for _ in 0..200 { + if !again.disk_frames(&src(&disk), &key2, 0..1, &opts).is_empty() { + break; + } + std::thread::sleep(std::time::Duration::from_millis(5)); + } + assert_eq!(again.disk_frames(&src(&disk), &key2, 0..1, &opts), vec![0]); + let hits = disk.stats().hits; + let (_, read) = shown(&again, &disk); + assert_eq!(disk.stats().hits, hits + 1, "read from the disk cache"); + assert_eq!(read.bits(), rendered.bits()); + assert_eq!(read.encoding, rendered.encoding); +} + /// The oracle for [`Comp::changed_spans`]: every frame an edit keeps renders the same in the edited /// project. Moves, trims, splits, deletes and duplicates of moving clips, under adjustment /// layers, with seeded noise, motion blur, mattes, parents and effects that read other times. @@ -1415,9 +1584,19 @@ fn disk_keys_survive_save_and_reopen() { /// Roto Brush model stamp in its opts; the disk keys are the worker's, without it. #[test] fn ram_frames_missing_on_disk_are_written() { + ram_frames_missing_on_disk_are_written_as(false); +} + +/// Same for a 32 bpc project, whose RAM frames are half-float. +#[test] +fn half_float_ram_frames_missing_on_disk_are_written() { + ram_frames_missing_on_disk_are_written_as(true); +} + +fn ram_frames_missing_on_disk_are_written_as(half: bool) { use effectcraft_engine::Session; use serde_json::json; - let dir = std::env::temp_dir().join(format!("ec-persist-{}", std::process::id())); + let dir = std::env::temp_dir().join(format!("ec-persist-{}-{half}", std::process::id())); let _ = std::fs::remove_dir_all(&dir); let disk = DiskCache::open(&dir, 1 << 30).unwrap(); let mut s = Session::default(); @@ -1436,7 +1615,8 @@ fn ram_frames_missing_on_disk_are_written() { let series = FrameKey { revision: 1, content: f.content_of(&src.project, 1, cid), comp: cid.0, opts: opts_hash(&RenderOpts::default()) ^ 1, ..key(0, 0, 0) }; for fr in 0..10 { - insert(&f, FrameKey { frame: fr, ..series }); + let frame = if half { FrameImage::CpuF16(Arc::new(F16Frame::from_image(&effectcraft_engine::render::Image::new(8, 8), SRGB))) } else { img() }; + f.cache.lock().unwrap().insert(FrameKey { frame: fr, ..series }, frame); } let on_disk = |fr: i64| { disk.contains( diff --git a/crates/ui-egui/src/lib.rs b/crates/ui-egui/src/lib.rs index 063d1201f..88f99a657 100644 --- a/crates/ui-egui/src/lib.rs +++ b/crates/ui-egui/src/lib.rs @@ -24,6 +24,7 @@ pub mod bench; pub mod color_picker; pub mod control; pub mod credits; +pub mod display_gpu; pub mod dock; pub mod dock_ui; mod expression_diagnostics; @@ -314,12 +315,28 @@ pub struct EffectcraftApp { /// The last GPU frame shown: its egui texture id (registered with egui-wgpu), key and texture. pub(crate) viewer_native: Option<(egui::TextureId, FrameKey, Arc)>, /// The texture the viewer draws and its frame key (CPU or GPU frame). - pub(crate) viewer_shown: Option<(egui::TextureId, FrameKey)>, + /// (`None` texture: a half-float frame, drawn by the display pass.) + pub(crate) viewer_shown: Option<(Option, FrameKey)>, /// A Cryptomatte eyedropper probe is being built (one at a time, re-arming included). pub(crate) crypto_probe_busy: Arc, - /// Last displayed image (Info panel colour, eyedroppers, histograms). GPU frames fill it on - /// demand ([`EffectcraftApp::viewer_pixels`]). + /// Last displayed image (Info panel colour, eyedroppers, histograms). GPU frames and + /// half-float frames fill it on demand ([`EffectcraftApp::viewer_pixels`]). pub(crate) viewer_image: Option>, + /// The half-float frame on screen (32 bpc projects; its key is `viewer_shown`'s). + pub(crate) viewer_f16: Option>, + /// The window's egui-wgpu device (with or without a GPU compositor on it): the display pass + /// runs there. + pub(crate) window_wgpu: Option, + /// The display pass on that device (made with the first frame that needs it), and whether + /// making it failed (the pass then runs on the CPU). + pub(crate) display_gpu: Option>, + pub(crate) display_gpu_failed: bool, + /// The Composition viewer's and the snapshot's displayed textures (display pass). + pub(crate) viewer_bake: display_gpu::Bake, + pub(crate) snapshot_bake: display_gpu::Bake, + /// The snapshot in half floats for the display pass (the image it was made from, not kept + /// alive by this). + pub(crate) snapshot_f16: Option<(std::sync::Weak, Arc)>, /// wasm32: an asynchronous readback of a GPU viewer frame (key asked for, result). viewer_readback: (Option, Arc)>>>), /// egui-wgpu's device, once the first frame arrives (desktop and WebGPU). diff --git a/crates/ui-egui/src/panels/info.rs b/crates/ui-egui/src/panels/info.rs index a02134191..4fb686d86 100644 --- a/crates/ui-egui/src/panels/info.rs +++ b/crates/ui-egui/src/panels/info.rs @@ -18,16 +18,28 @@ pub fn show(app: &mut EffectcraftApp, ui: &mut egui::Ui, rect: Rect) { let p = ui.painter().with_clip_rect(rect); let x0 = rect.min.x + 12.0; let mut y = rect.min.y + 14.0 - scroll.offset; - if app.pointer_comp.is_some() { + let comp_size = app.session.active_comp().map_or([1.0, 1.0], |c| [c.width as f32, c.height as f32]); + let region = || super::viewer_tools::shown_region(ui.ctx(), comp_size); + // 32 bpc (half-float frames): the frame's float values, straight colour as rendered (before + // Project Settings ▸ HDR and the viewer's exposure), as After Effects shows them in 32 bpc. + let floats = match (app.pointer_comp, app.viewer_f16()) { + (Some([cx, cy]), Some(f)) => { + let p = super::fx_widgets::frame_pixel_index(f.size(), region(), [cx as f64, cy as f64]).and_then(|i| f.pixel(i)); + let un = |p: [f32; 4]| { + let a = p[3].clamp(0.0, 1.0); + if a > 0.0 { [p[0] / a, p[1] / a, p[2] / a, a] } else { [0.0; 4] } + }; + Some((p.map(un), (cx, cy))) + } + _ => None, + }; + if app.pointer_comp.is_some() && floats.is_none() { // GPU frames: read the shown frame back for sampling. app.viewer_pixels(); } let (rgba, xy) = match (app.pointer_comp, &app.viewer_image) { - (Some([cx, cy]), Some(img)) => { - let comp = app.session.active_comp().map_or([1.0, 1.0], |c| [c.width as f32, c.height as f32]); - let region = super::viewer_tools::shown_region(ui.ctx(), comp); - (super::fx_widgets::frame_pixel(img, region, [cx as f64, cy as f64]), Some((cx, cy))) - } + _ if floats.is_some() => (None, floats.map(|(_, xy)| xy)), + (Some([cx, cy]), Some(img)) => (super::fx_widgets::frame_pixel(img, region(), [cx as f64, cy as f64]), Some((cx, cy))), _ => (None, None), }; let row = |p: &egui::Painter, y: f32, k: &str, v: String, col: Color32| { @@ -43,20 +55,26 @@ pub fn show(app: &mut EffectcraftApp, ui: &mut egui::Ui, rect: Rect) { [f(c.r()), f(c.g()), f(c.b()), a] } }; - let px = rgba.map(unpre).unwrap_or([0, 0, 0, 0]); - row(&p, y, "R :", format!("{}", px[0]), Color32::from_rgb(0xe0, 0x60, 0x60)); - row(&p, y + 18.0, "G :", format!("{}", px[1]), Color32::from_rgb(0x60, 0xd0, 0x60)); - row(&p, y + 36.0, "B :", format!("{}", px[2]), Color32::from_rgb(0x60, 0x90, 0xf0)); - row(&p, y + 54.0, "A :", format!("{}", px[3]), t.text_dim); + let (texts, swatch): ([String; 4], Option<[u8; 4]>) = match floats { + Some((p, _)) => (p.unwrap_or([0.0; 4]).map(|v| format!("{v:.4}")), p.map(|p| p.map(|v| (v.clamp(0.0, 1.0) * 255.0 + 0.5) as u8))), + None => { + let px = rgba.map(unpre).unwrap_or([0, 0, 0, 0]); + (px.map(|v| format!("{v}")), rgba.map(|_| px)) + } + }; + let [tr_, tg, tb, ta] = texts; + row(&p, y, "R :", tr_, Color32::from_rgb(0xe0, 0x60, 0x60)); + row(&p, y + 18.0, "G :", tg, Color32::from_rgb(0x60, 0xd0, 0x60)); + row(&p, y + 36.0, "B :", tb, Color32::from_rgb(0x60, 0x90, 0xf0)); + row(&p, y + 54.0, "A :", ta, t.text_dim); let cx = rect.min.x + rect.width() * 0.5; if let Some((x, yy)) = xy { p.text(pos2(cx, y), Align2::LEFT_CENTER, format!("X : {x:.0}"), Tokens::mono(12.0), t.text); p.text(pos2(cx, y + 18.0), Align2::LEFT_CENTER, format!("Y : {yy:.0}"), Tokens::mono(12.0), t.text); } - if let Some(c) = rgba { + if let Some(px) = swatch { let sw = egui::Rect::from_min_size(pos2(cx, y + 34.0), vec2(36.0, 30.0)); p.rect_filled(sw, 3.0, Color32::from_rgb(px[0], px[1], px[2])); - let _ = c; } y += 82.0; p.line_segment([pos2(rect.min.x + 8.0, y), pos2(rect.max.x - 8.0, y)], egui::Stroke::new(1.0, t.separator)); diff --git a/crates/ui-egui/src/panels/viewer.rs b/crates/ui-egui/src/panels/viewer.rs index 5664602a7..b1899e1f8 100644 --- a/crates/ui-egui/src/panels/viewer.rs +++ b/crates/ui-egui/src/panels/viewer.rs @@ -352,6 +352,7 @@ fn phase_frame(app: &mut EffectcraftApp, v: &mut Vw<'_>) { let stale = app.viewer_shown.as_ref().is_none_or(|(_, s)| *s != k); // A CPU frame goes up again when the magnification calls for another factor. let minify = minify_factor(app.session.prefs.viewer_zoom_smooth(), scale, (zoom * ppp) as f64); + // (Half-float frames have no such texture: the display pass makes mips.) let refit = !app.viewer_on_gpu() && app.viewer_tex.as_ref().is_some_and(|(_, _, m)| *m != minify); if stale || refit { show_frame(app, &ctx, k, img, minify); diff --git a/crates/ui-egui/src/panels/viewer/frame.rs b/crates/ui-egui/src/panels/viewer/frame.rs index e0d2a3bc8..244285e4d 100644 --- a/crates/ui-egui/src/panels/viewer/frame.rs +++ b/crates/ui-egui/src/panels/viewer/frame.rs @@ -140,7 +140,8 @@ pub(super) fn rect_of(map: &ViewerMap, r: Option<[f64; 4]>) -> Rect { /// Put a rendered frame on screen: CPU pixels go into an egui texture, averaged down by `minify` /// (see [`minify_factor`]); GPU frames are drawn straight from their wgpu texture (registered with -/// egui-wgpu, no readback). +/// egui-wgpu, no readback). Half-float frames (32 bpc) are kept for the display pass, which +/// `viewer_tools::draw_frame` runs (exposure, HDR step, display colour). pub(super) fn show_frame(app: &mut EffectcraftApp, ctx: &egui::Context, key: crate::frames::FrameKey, img: crate::frames::FrameImage, minify: usize) { use crate::frames::FrameImage; // Settings ▸ Previews ▸ Viewer Zoom Quality. @@ -158,8 +159,18 @@ pub(super) fn show_frame(app: &mut EffectcraftApp, ctx: &egui::Context, key: cra } _ => app.viewer_tex = Some((ctx.load_texture("viewer-frame", fitted, opts), key, minify)), } - app.viewer_shown = app.viewer_tex.as_ref().map(|(t, k, _)| (t.id(), *k)); + app.viewer_shown = app.viewer_tex.as_ref().map(|(t, k, _)| (Some(t.id()), *k)); app.viewer_image = Some(img); + app.viewer_f16 = None; + vt::set_texture_roi(ctx, app.session.state.region_of_interest.or(app.ui.viewer.extended)); + } + FrameImage::CpuF16(f) => { + // Drawn through the display pass; the 8-bit texture of an earlier frame goes. + app.viewer_tex = None; + // No texture of its own: the display pass draws it. + app.viewer_shown = Some((None, key)); + app.viewer_image = None; + app.viewer_f16 = Some(f); vt::set_texture_roi(ctx, app.session.state.region_of_interest.or(app.ui.viewer.extended)); } FrameImage::Gpu(f) => { @@ -175,9 +186,10 @@ pub(super) fn show_frame(app: &mut EffectcraftApp, ctx: &egui::Context, key: cra }; drop(ren); app.viewer_native = Some((id, key, f)); - app.viewer_shown = Some((id, key)); + app.viewer_shown = Some((Some(id), key)); // Pixels are read back on demand (viewer_pixels). app.viewer_image = None; + app.viewer_f16 = None; } } } diff --git a/crates/ui-egui/src/panels/viewer_tools.rs b/crates/ui-egui/src/panels/viewer_tools.rs index 316729d46..563657e91 100644 --- a/crates/ui-egui/src/panels/viewer_tools.rs +++ b/crates/ui-egui/src/panels/viewer_tools.rs @@ -343,14 +343,34 @@ pub(crate) fn showing_snapshot(app: &EffectcraftApp, ctx: &egui::Context) -> boo app.session.snapshot.is_some() && (app.session.state.viewer.show_snapshot || f5) } -/// Show Channel and exposure, then the display colour conversion (View ▸ Use Display Color -/// Management / Simulate Output). -fn transformed(img: &egui::ColorImage, ch: Channel, colorized: bool, stops: f32, dc: Option) -> egui::ColorImage { - let mut px: Vec<[u8; 4]> = img.pixels.iter().map(|c| c.to_array()).collect(); - if let Some(dc) = dc.filter(|_| matches!(ch, Channel::Rgb | Channel::RgbStraight)) { - dc.apply(&mut px); +/// The viewer's display pass for frames encoded as `encoding`: Show Channel, exposure, the HDR +/// step such frames still need, then the display colour conversion `dc` (View ▸ Use Display +/// Color Management / Simulate Output). +pub(crate) fn display_pass(app: &EffectcraftApp, encoding: vw::FrameEncoding, dc: Option<&vw::DisplayColor>) -> vw::DisplayPass { + let o = &app.session.state.viewer; + vw::DisplayPass::new(o.channel, o.colorized, o.exposure, encoding, dc) +} + +/// How the project's 8-bit viewer frames are encoded (rendered with the HDR step). +pub(crate) fn rgba8_encoding(app: &EffectcraftApp) -> vw::FrameEncoding { + vw::FrameEncoding::of(&app.session.project.settings, false) +} + +/// A key of a display pass's parameters, for the texture caches. +pub(crate) fn pass_key(pass: &vw::DisplayPass) -> u64 { + use std::hash::{Hash, Hasher}; + let mut h = std::collections::hash_map::DefaultHasher::new(); + crate::display_gpu::uniform(pass).hash(&mut h); + if let Some(vw::PassColor::Lut(l)) = &pass.color { + (std::sync::Arc::as_ptr(l) as usize).hash(&mut h); } - vw::display_transform(&mut px, ch, colorized, stops); + h.finish() +} + +/// The display pass on the CPU over an 8-bit frame. +fn transformed(img: &egui::ColorImage, pass: &vw::DisplayPass) -> egui::ColorImage { + let mut px: Vec<[u8; 4]> = img.pixels.iter().map(|c| c.to_array()).collect(); + pass.apply_rgba8(&mut px); egui::ColorImage::new(img.size, px.into_iter().map(|a| Color32::from_rgba_premultiplied(a[0], a[1], a[2], a[3])).collect()) } @@ -362,9 +382,60 @@ pub(crate) fn display_key(app: &EffectcraftApp, has_conversion: bool) -> String + &format!("{:?}{:?}{}", app.session.project.settings.working_space, app.session.project.settings.output_space, opts.display_color_management) } +/// Draw `src` through `pass` into `rect`: the display pass on egui-wgpu's device (one texture per +/// slot, with mips), else on the CPU into a texture averaged down for the magnification (#417). +fn draw_through_pass( + app: &mut EffectcraftApp, + ctx: &egui::Context, + painter: &egui::Painter, + snapshot: bool, + src: crate::display_gpu::Source, + pass: &vw::DisplayPass, + rect: Rect, +) { + let smooth = app.session.prefs.viewer_zoom_smooth(); + if let Some(gpu) = app.display_gpu() + && let Some(rs) = &app.window_wgpu + { + let bake = if snapshot { &mut app.snapshot_bake } else { &mut app.viewer_bake }; + if let Some(id) = bake.show(rs, &gpu, &src, pass, smooth) { + painter.image(id, rect, bake.uv(), Color32::WHITE); + return; + } + } + let size = src.size(); + let minify = super::viewer::minify_factor(smooth, size[0] as f64, (rect.width() * ctx.pixels_per_point()) as f64); + let key = { + use std::hash::{Hash, Hasher}; + let mut h = std::collections::hash_map::DefaultHasher::new(); + let ptr = match &src { + crate::display_gpu::Source::F16(f) => std::sync::Arc::as_ptr(f) as usize, + crate::display_gpu::Source::Rgba8(c) => std::sync::Arc::as_ptr(c) as usize, + }; + (ptr, pass_key(pass), minify).hash(&mut h); + h.finish() + }; + let id = egui::Id::unique(if snapshot { "viewer-snapshot-tex" } else { "viewer-display-tex" }); + let tex = match ctx.data(|d| d.get_temp::<(u64, egui::TextureHandle)>(id)) { + Some((k, tex)) if k == key => tex, + _ => { + let img = match &src { + crate::display_gpu::Source::F16(f) => f.to_color_image(pass), + crate::display_gpu::Source::Rgba8(c) => transformed(c, pass), + }; + let img = crate::frames::fit_texture_by(img, crate::frames::max_texture_side(ctx), minify); + let tex = ctx.load_texture(if snapshot { "viewer-snapshot" } else { "viewer-display" }, img, super::viewer::zoom_texture_options(smooth)); + ctx.data_mut(|d| d.insert_temp(id, (key, tex.clone()))); + tex + } + }; + painter.image(tex.id(), rect, fitted_uv(ctx, size, minify), Color32::WHITE); +} + /// Draw the frame into `comp_rect`: the rendered frame (covering the region of interest when one -/// is set), or the snapshot while Show Snapshot is on / F5 is held, through Show Channel and the -/// exposure. Fast Previews ▸ Wireframe draws layer outlines instead. +/// is set), or the snapshot while Show Snapshot is on / F5 is held, through the display pass (Show +/// Channel, exposure, the HDR step of 32 bpc frames, display colour). Fast Previews ▸ Wireframe +/// draws layer outlines instead. pub(crate) fn draw_frame(app: &mut EffectcraftApp, ctx: &egui::Context, painter: &egui::Painter, comp_rect: Rect, cid: ItemId, ectx: &EvalCtx) { clear_sample_source(ctx); let opts = app.session.state.viewer.clone(); @@ -391,47 +462,58 @@ pub(crate) fn draw_frame(app: &mut EffectcraftApp, ctx: &egui::Context, painter: } return; } + // The snapshot went (Purge, a new one): its half floats and texture go too. + if app + .snapshot_f16 + .as_ref() + .is_some_and(|(img, _)| app.session.snapshot.as_ref().is_none_or(|s| !std::ptr::eq(img.as_ptr(), std::sync::Arc::as_ptr(&s.image)))) + { + app.snapshot_f16 = None; + app.snapshot_bake.free(app.window_wgpu.as_ref()); + } let dc = vw::DisplayColor::of(&app.session); - let dc_key = display_key(app, dc.is_some()); - let plain = opts.channel == Channel::Rgb && opts.exposure == 0.0 && dc.is_none(); - let display = SampleDisplay { channel: opts.channel, colorized: opts.colorized, exposure: opts.exposure, conversion: dc.clone() }; let region = shown_region(ctx, [ectx.comp.width as f32, ectx.comp.height as f32]); let uv = Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)); if snap { let Some(s) = app.session.snapshot.clone() else { return }; - let key = { - use std::hash::{Hash, Hasher}; - let mut h = std::collections::hash_map::DefaultHasher::new(); - (std::sync::Arc::as_ptr(&s.image) as usize, opts.channel.id(), opts.colorized, opts.exposure.to_bits(), &dc_key).hash(&mut h); - h.finish() - }; - let id = egui::Id::unique("viewer-snapshot-tex"); - let tex = match ctx.data(|d| d.get_temp::<(u64, egui::TextureHandle)>(id)) { - Some((k, tex)) if k == key => tex, - _ => { - let img = crate::frames::to_color_image(&s.image); - let img = if plain { img } else { transformed(&img, opts.channel, opts.colorized, opts.exposure, dc) }; - let opts = super::viewer::zoom_texture_options(app.session.prefs.viewer_zoom_smooth()); - let tex = crate::frames::load_fitted(ctx, "viewer-snapshot", img, opts); - ctx.data_mut(|d| d.insert_temp(id, (key, tex.clone()))); - tex + let pass = display_pass(app, s.encoding, dc.as_ref()); + // The snapshot in half floats, made once (it holds the render's full range). + let f = match &app.snapshot_f16 { + Some((_, f)) => f.clone(), + None => { + let f = std::sync::Arc::new(crate::frames::F16Frame::from_image(&s.image, s.encoding)); + app.snapshot_f16 = Some((std::sync::Arc::downgrade(&s.image), f.clone())); + f } }; - painter.image(tex.id(), comp_rect, uv, Color32::WHITE); - note_sample_source( - ctx, - SampleSource::Snapshot(s.image.clone()), - [0.0, 0.0, ectx.comp.width as f64, ectx.comp.height as f64], - (!plain).then_some(display), - ); + draw_through_pass(app, ctx, painter, true, crate::display_gpu::Source::F16(f), &pass, comp_rect); + note_sample_source(ctx, SampleSource::Snapshot(s.image.clone()), [0.0, 0.0, ectx.comp.width as f64, ectx.comp.height as f64], Some(pass)); painter.text(comp_rect.left_top() + vec2(6.0, 6.0), Align2::LEFT_TOP, crate::i18n::tr("Snapshot"), Tokens::ui(11.0), SNAP_COLOR); return; } + let zoom = comp_rect.width() as f64 / ectx.comp.width.max(1) as f64; + let roi_rect = || match texture_roi(ctx) { + Some([x, y, w, h]) => Rect::from_min_size(comp_rect.min + vec2((x * zoom) as f32, (y * zoom) as f32), vec2((w * zoom) as f32, (h * zoom) as f32)), + None => comp_rect, + }; + // A half-float frame (32 bpc): always through the display pass, which applies the HDR step + // after the exposure. + if let Some(f) = app.viewer_f16.clone() { + if app.viewer_shown.is_none_or(|(_, k)| k.comp != cid.0) { + return; + } + let pass = display_pass(app, f.encoding, dc.as_ref()); + draw_through_pass(app, ctx, painter, false, crate::display_gpu::Source::F16(f.clone()), &pass, roi_rect()); + note_sample_source(ctx, SampleSource::F16(f), region, Some(pass)); + return; + } + let pass = display_pass(app, rgba8_encoding(app), dc.as_ref()); + let plain = pass.is_identity(); // A GPU frame (the web viewer): drawn straight from its texture; Show Channel / exposure // read it back first (asynchronously in the browser: from the next frame on). - let gpu = matches!((&app.viewer_shown, &app.viewer_native), (Some(s), Some(n)) if s.0 == n.0 && s.1 == n.1 && n.1.comp == cid.0); + let gpu = matches!((&app.viewer_shown, &app.viewer_native), (Some(s), Some(n)) if s.0 == Some(n.0) && s.1 == n.1 && n.1.comp == cid.0); let rect = if gpu { - let Some(id) = app.viewer_shown.map(|s| s.0) else { return }; + let Some(id) = app.viewer_shown.and_then(|s| s.0) else { return }; if plain || app.viewer_pixels().is_none() { if !plain { ctx.request_repaint(); @@ -448,11 +530,7 @@ pub(crate) fn draw_frame(app: &mut EffectcraftApp, ctx: &egui::Context, painter: if k.comp != cid.0 { return; } - let zoom = comp_rect.width() as f64 / ectx.comp.width.max(1) as f64; - let rect = match texture_roi(ctx) { - Some([x, y, w, h]) => Rect::from_min_size(comp_rect.min + vec2((x * zoom) as f32, (y * zoom) as f32), vec2((w * zoom) as f32, (h * zoom) as f32)), - None => comp_rect, - }; + let rect = roi_rect(); if plain { let uv = app.viewer_image.as_ref().map_or(uv, |src| fitted_uv(ctx, src.size, minify)); painter.image(tex.id(), rect, uv, Color32::WHITE); @@ -464,51 +542,24 @@ pub(crate) fn draw_frame(app: &mut EffectcraftApp, ctx: &egui::Context, painter: rect }; let Some(src) = app.viewer_image.clone() else { return }; - let smooth = app.session.prefs.viewer_zoom_smooth(); - // Averaged down for the magnification, as the frame's own texture is (#417). - let minify = super::viewer::minify_factor(smooth, src.size[0] as f64, (rect.width() * ctx.pixels_per_point()) as f64); - let key = { - use std::hash::{Hash, Hasher}; - let mut h = std::collections::hash_map::DefaultHasher::new(); - (std::sync::Arc::as_ptr(&src) as usize, opts.channel.id(), opts.colorized, opts.exposure.to_bits(), &dc_key, minify).hash(&mut h); - h.finish() - }; - let id = egui::Id::unique("viewer-display-tex"); - let tex = match ctx.data(|d| d.get_temp::<(u64, egui::TextureHandle)>(id)) { - Some((k, tex)) if k == key => tex, - _ => { - let img = - crate::frames::fit_texture_by(transformed(&src, opts.channel, opts.colorized, opts.exposure, dc), crate::frames::max_texture_side(ctx), minify); - let tex = ctx.load_texture("viewer-display", img, super::viewer::zoom_texture_options(smooth)); - ctx.data_mut(|d| d.insert_temp(id, (key, tex.clone()))); - tex - } - }; - painter.image(tex.id(), rect, fitted_uv(ctx, src.size, minify), Color32::WHITE); - note_sample_source(ctx, SampleSource::Cpu(src), region, Some(display)); + draw_through_pass(app, ctx, painter, false, crate::display_gpu::Source::Rgba8(src.clone()), &pass, rect); + note_sample_source(ctx, SampleSource::Cpu(src), region, Some(pass)); } #[derive(Clone)] enum SampleSource { Cpu(std::sync::Arc), + F16(std::sync::Arc), Gpu(crate::frames::FrameKey), Snapshot(std::sync::Arc), } -#[derive(Clone)] -struct SampleDisplay { - channel: Channel, - colorized: bool, - exposure: f32, - conversion: Option, -} - #[derive(Clone)] struct DisplayedSample { pass: u64, source: SampleSource, region: [f64; 4], - display: Option, + display: Option, } fn sample_source_id() -> egui::Id { @@ -519,36 +570,40 @@ pub(crate) fn clear_sample_source(ctx: &egui::Context) { ctx.data_mut(|data| data.remove::(sample_source_id())); } -fn note_sample_source(ctx: &egui::Context, source: SampleSource, region: [f64; 4], display: Option) { +fn note_sample_source(ctx: &egui::Context, source: SampleSource, region: [f64; 4], display: Option) { let sample = DisplayedSample { pass: ctx.cumulative_pass_nr(), source, region, display }; ctx.data_mut(|data| data.insert_temp(sample_source_id(), sample)); } -/// Sample only a frame actually drawn this pass, with that draw's channel, exposure and -/// display conversion. Hidden and wireframe views deliberately have no sample source. +/// Sample only a frame actually drawn this pass, with that draw's display pass (channel, +/// exposure, HDR step and display conversion). Hidden and wireframe views deliberately have no +/// sample source. pub(crate) fn sample_displayed(app: &EffectcraftApp, ctx: &egui::Context, point: [f64; 2]) -> Option<[f32; 4]> { let sample = ctx.data(|data| data.get_temp::(sample_source_id()))?; if sample.pass != ctx.cumulative_pass_nr() { return None; } - let pixel = match sample.source { - SampleSource::Cpu(image) => super::fx_widgets::frame_pixel(&image, sample.region, point)?, + let rgba8 = |c: Color32| c.to_array().map(|v| v as f32 / 255.0); + // The frame's premultiplied pixel. + let raw: [f32; 4] = match sample.source { + SampleSource::Cpu(image) => rgba8(super::fx_widgets::frame_pixel(&image, sample.region, point)?), + SampleSource::F16(f) => f.pixel(super::fx_widgets::frame_pixel_index(f.size(), sample.region, point)?)?, SampleSource::Gpu(key) => { if app.viewer_shown.map(|(_, shown)| shown) != Some(key) { return None; } - super::fx_widgets::frame_pixel(app.viewer_image.as_ref()?, sample.region, point)? + rgba8(super::fx_widgets::frame_pixel(app.viewer_image.as_ref()?, sample.region, point)?) } SampleSource::Snapshot(image) => { let index = super::fx_widgets::frame_pixel_index([image.width as usize, image.height as usize], sample.region, point)?; - crate::frames::display_pixel(*image.data.get(index)?) + *image.data.get(index)? } }; - let image = egui::ColorImage::new([1, 1], vec![pixel]); - let image = match sample.display { - Some(display) => transformed(&image, display.channel, display.colorized, display.exposure, display.conversion), - None => image, + let shown = match &sample.display { + Some(pass) => pass.pixel(raw), + None => crate::frames::display_pixel(raw).to_array(), }; + let image = egui::ColorImage::new([1, 1], vec![Color32::from_rgba_premultiplied(shown[0], shown[1], shown[2], shown[3])]); super::fx_widgets::sample_frame(&image, [0.0, 0.0, 1.0, 1.0], [0.5, 0.5]) } diff --git a/crates/ui-egui/src/panels/viewers.rs b/crates/ui-egui/src/panels/viewers.rs index e8be901b9..fabfe586f 100644 --- a/crates/ui-egui/src/panels/viewers.rs +++ b/crates/ui-egui/src/panels/viewers.rs @@ -22,13 +22,25 @@ use crate::theme::Tokens; pub enum PassiveTexture { Cpu(egui::TextureHandle), Gpu(egui::TextureId, Arc), + /// A half-float frame (32 bpc) through the display pass on egui-wgpu's device. + Baked(Box), } impl PassiveTexture { - fn id(&self) -> egui::TextureId { + /// The texture to draw (`None`: the display pass has not made one). + fn id(&self) -> Option { match self { - PassiveTexture::Cpu(h) => h.id(), - PassiveTexture::Gpu(id, _) => *id, + PassiveTexture::Cpu(h) => Some(h.id()), + PassiveTexture::Gpu(id, _) => Some(*id), + PassiveTexture::Baked(b) => b.id(), + } + } + + /// Where the frame is in the texture (a frame past the texture limit is averaged down). + fn uv(&self) -> Rect { + match self { + PassiveTexture::Baked(b) => b.uv(), + _ => Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), } } } @@ -106,9 +118,9 @@ pub fn show(app: &mut EffectcraftApp, ui: &mut egui::Ui, id: u32, rect: Rect) { let dkey = display_hash(&super::viewer_tools::display_key(app, dc.is_some())); set_texture(app, &ctx, id, key, img, dc, dkey); } - match app.passive_tex.get(&id).filter(|(k, _)| k.comp == cid.0) { - Some((_, tex)) => { - p.image(tex.id(), r, Rect::from_min_max(pos2(0.0, 0.0), pos2(1.0, 1.0)), Color32::WHITE); + match app.passive_tex.get(&id).filter(|(k, _)| k.comp == cid.0).and_then(|(_, tex)| Some((tex.id()?, tex.uv()))) { + Some((tex, uv)) => { + p.image(tex, r, uv, Color32::WHITE); } None => { let bg = c.background; @@ -128,7 +140,7 @@ pub fn show(app: &mut EffectcraftApp, ui: &mut egui::Ui, id: u32, rect: Rect) { } /// Keep the frame viewer `id` shows in its texture. -fn set_texture( +pub(crate) fn set_texture( app: &mut EffectcraftApp, ctx: &egui::Context, id: u32, @@ -149,16 +161,19 @@ fn set_texture( return; } ctx.data_mut(|d| d.insert_temp(dc_id, dkey)); - let opts = super::viewer::zoom_texture_options(app.session.prefs.viewer_zoom_smooth()); - // The monitor conversion the active viewer applies (Use Display Color Management). - let img = match (img, &dc) { - (FrameImage::Cpu(img), Some(dc)) => { - let mut px: Vec<[u8; 4]> = img.pixels.iter().map(|c| c.to_array()).collect(); - dc.apply(&mut px); - let pixels = px.into_iter().map(|a| Color32::from_rgba_premultiplied(a[0], a[1], a[2], a[3])).collect(); - FrameImage::Cpu(Arc::new(egui::ColorImage::new(img.size, pixels))) + let smooth = app.session.prefs.viewer_zoom_smooth(); + let opts = super::viewer::zoom_texture_options(smooth); + let pass = |encoding| passive_pass(encoding, dc.as_ref()); + let gpu = app.display_gpu(); + let img = match img { + // Frames past the display texture limit take the CPU pass (display conversion first, then + // the averaging down), which the GPU bake leaves out. + FrameImage::CpuF16(f) + if app.window_wgpu.is_some() && gpu.as_ref().is_some_and(|g| !crate::display_gpu::Source::F16(f.clone()).needs_cpu_pass(g.max_side())) => + { + FrameImage::CpuF16(f) } - (img, _) => img, + img => cpu_passive_image(app, img, dc.as_ref()), }; let tex = match (img, app.passive_tex.remove(&id)) { (FrameImage::Cpu(img), Some((_, PassiveTexture::Cpu(mut h)))) @@ -171,10 +186,23 @@ fn set_texture( free(app, old.map(|(_, t)| t)); PassiveTexture::Cpu(crate::frames::load_fitted(ctx, format!("viewer-{id}"), (*img).clone(), opts)) } + (FrameImage::CpuF16(f), old) => { + let Some(gpu) = gpu else { return }; + let mut bake = match old { + Some((_, PassiveTexture::Baked(b))) => b, + old => { + free(app, old.map(|(_, t)| t)); + Box::default() + } + }; + let Some(rs) = &app.window_wgpu else { return }; + bake.show(rs, &gpu, &crate::display_gpu::Source::F16(f.clone()), &pass(f.encoding), smooth); + PassiveTexture::Baked(bake) + } (FrameImage::Gpu(f), old) => { let Some(rs) = &app.wgpu else { return }; let view = f.texture.create_view(&Default::default()); - let sampler = || super::viewer::zoom_sampler(app.session.prefs.viewer_zoom_smooth()); + let sampler = || super::viewer::zoom_sampler(smooth); let mut ren = rs.renderer.write(); let tid = match old { Some((_, PassiveTexture::Gpu(tid, _))) => { @@ -189,6 +217,35 @@ fn set_texture( app.passive_tex.insert(id, (key, tex)); } +/// A passive viewer's display pass: the monitor conversion the active viewer applies (Use +/// Display Color Management), with no channel or exposure of its own; for half-float frames +/// also the HDR step. +fn passive_pass( + encoding: effectcraft_engine::viewer::FrameEncoding, + dc: Option<&effectcraft_engine::viewer::DisplayColor>, +) -> effectcraft_engine::viewer::DisplayPass { + effectcraft_engine::viewer::DisplayPass::new(effectcraft_engine::viewer::Channel::Rgb, false, 0.0, encoding, dc) +} + +/// A frame as a passive viewer shows it without the display pass on the GPU: CPU frames through +/// [`passive_pass`] (GPU frames unchanged). +pub(crate) fn cpu_passive_image(app: &EffectcraftApp, img: FrameImage, dc: Option<&effectcraft_engine::viewer::DisplayColor>) -> FrameImage { + match img { + FrameImage::Cpu(img) => { + let pass = passive_pass(super::viewer_tools::rgba8_encoding(app), dc); + if pass.is_identity() { + return FrameImage::Cpu(img); + } + let mut px: Vec<[u8; 4]> = img.pixels.iter().map(|c| c.to_array()).collect(); + pass.apply_rgba8(&mut px); + let pixels = px.into_iter().map(|a| Color32::from_rgba_premultiplied(a[0], a[1], a[2], a[3])).collect(); + FrameImage::Cpu(Arc::new(egui::ColorImage::new(img.size, pixels))) + } + FrameImage::CpuF16(f) => FrameImage::Cpu(Arc::new(f.to_color_image(&passive_pass(f.encoding, dc)))), + img => img, + } +} + /// A hash of the display conversion's inputs (see `viewer_tools::display_key`). fn display_hash(key: &str) -> u64 { use std::hash::{Hash, Hasher}; @@ -199,8 +256,10 @@ fn display_hash(key: &str) -> u64 { /// Give a passive viewer's texture back (GPU textures are registered with the renderer). fn free(app: &EffectcraftApp, tex: Option) { - if let (Some(PassiveTexture::Gpu(id, _)), Some(rs)) = (tex, &app.wgpu) { - rs.renderer.write().free_texture(&id); + match (tex, &app.wgpu) { + (Some(PassiveTexture::Gpu(id, _)), Some(rs)) => rs.renderer.write().free_texture(&id), + (Some(PassiveTexture::Baked(mut b)), _) => b.free(app.window_wgpu.as_ref()), + _ => {} } } diff --git a/crates/ui-egui/tests/it/ui_gpu.rs b/crates/ui-egui/tests/it/ui_gpu.rs index d5d432db3..28edac27d 100644 --- a/crates/ui-egui/tests/it/ui_gpu.rs +++ b/crates/ui-egui/tests/it/ui_gpu.rs @@ -242,3 +242,42 @@ fn a_lost_device_draws_nothing_autosaves_and_closes() { assert_eq!(told.borrow().len(), 1); let _ = std::fs::remove_dir_all(&dir); } + +/// #2797 in the real viewer on a device: a 32 bpc frame of white +2 stops (about 4.0 linear) +/// goes through the display pass on egui-wgpu's device, so -2 EV still shows it white and -4 EV +/// a quarter of white (the 8-bit viewer showed a sixteenth of white there). +#[test] +fn the_viewer_exposes_32_bpc_frames_on_the_gpu() { + let _device = crate::gpu_test::lock(); + if !any_adapter() { + eprintln!("no wgpu adapter: skipping"); + return; + } + let mut s = Session::default(); + s.execute("comp.new", json!({"name": "Hdr", "width": 64, "height": 64, "duration": 1})).unwrap(); + s.execute("file.projectSettings", json!({"bitDepth": 32, "workingSpace": "srgb"})).unwrap(); + let lid = s.execute("layer.newSolid", json!({"color": "#ffffff"})).unwrap()["layer"].as_u64().unwrap(); + s.execute("effect.apply", json!({"layers": [lid], "effect": "Exposure"})).unwrap(); + s.execute("prop.set", json!({"layer": lid, "path": "effects/#1/master/exposure", "value": 2.0})).unwrap(); + let mut h = Harness::builder().with_size(egui::vec2(1200.0, 800.0)).wgpu().build_eframe(move |_| EffectcraftApp::new(s)); + settle(&mut h); + let p = h.state().viewer_f16().expect("a half-float viewer frame").pixel(0).unwrap(); + let l = effectcraft_engine::color::srgb_to_linear(p[0] / p[3]); + assert!(l > 3.9, "{l}"); + // Screen pixels of the frame's grey (the comp is 64 × 64 points in the viewer at most). + let count = |h: &mut Harness<'_, EffectcraftApp>, stops: f32| { + h.state_mut().session.state.viewer.exposure = stops; + for _ in 0..4 { + h.step(); + } + assert!(h.state().viewer_display_texture().is_some(), "drawn by the display pass on the device"); + let want = (effectcraft_engine::color::linear_to_srgb(l * 2f32.powf(stops)).clamp(0.0, 1.0) * 255.0).round() as i32; + let img = h.render().expect("render"); + let n = img.pixels().filter(|px| px.0[..3].iter().all(|c| (*c as i32 - want).abs() <= 1)).count(); + (want, n) + }; + let (white, n) = count(&mut h, -2.0); + assert!(white == 255 && n >= 1000, "-2 EV: {n} white pixels"); + let (grey, n) = count(&mut h, -4.0); + assert!((130..=145).contains(&grey) && n >= 1000, "-4 EV: {n} pixels of {grey}"); +} diff --git a/crates/ui-egui/tests/it/ui_web_gpu_storage.rs b/crates/ui-egui/tests/it/ui_web_gpu_storage.rs index 024bd1cd0..b015da41b 100644 --- a/crates/ui-egui/tests/it/ui_web_gpu_storage.rs +++ b/crates/ui-egui/tests/it/ui_web_gpu_storage.rs @@ -116,7 +116,7 @@ fn source(s: &Session, gpu: Option) -> RenderSource { fn cpu_pixels(f: &FrameImage) -> Vec { match f { FrameImage::Cpu(c) => c.pixels.clone(), - FrameImage::Gpu(_) => panic!("expected a CPU frame"), + FrameImage::Gpu(_) | FrameImage::CpuF16(_) => panic!("expected an 8-bit CPU frame"), } } diff --git a/crates/ui-egui/tests/it/ui_web_offload.rs b/crates/ui-egui/tests/it/ui_web_offload.rs index 67ec6a9f8..fc0a0cd32 100644 --- a/crates/ui-egui/tests/it/ui_web_offload.rs +++ b/crates/ui-egui/tests/it/ui_web_offload.rs @@ -136,7 +136,7 @@ fn viewer_frames_render_remotely_from_project_diffs() { assert_eq!(frames.remote_busy(), 0); let img = match frames.get(&key(&s, 10)).expect("viewer frame cached") { FrameImage::Cpu(c) => c, - FrameImage::Gpu(_) => unreachable!(), + FrameImage::Gpu(_) | FrameImage::CpuF16(_) => unreachable!(), }; let local = effectcraft_ui_egui::frames::to_color_image(&s.render(comp, c.frame_rate.tick_of(10), opts)); assert_eq!(img.size, local.size); diff --git a/docs/architecture.md b/docs/architecture.md index a5937bc3d..48413d1c1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -220,12 +220,33 @@ instead of skipping pixels (Viewer Zoom Quality More Accurate): GPU frames carry sampled trilinearly, CPU frames go up averaged down by the whole factor that leaves about one texel per screen pixel. +32 bpc projects keep **half-float frames** on the desktop (`frames::F16Frame`, 8 bytes a pixel: +the RAM preview holds half as many as of 8-bit frames). They are rendered without Project +Settings ▸ HDR (`RenderOpts::skip_hdr_step`), so the values above 1.0 stay. The viewer's +**display pass** (`engine::viewer::DisplayPass`; on the GPU `ui-egui/src/display_gpu.rs` + +`display_gpu.wgsl`) then turns a frame into what the panel shows, in this order: decode with the +output space's curve, the viewer's exposure in linear light, the HDR step the frame was rendered +without (Compand / Tone Map), encode, clamp to 0..1, the display colour conversion (Use Display +Color Management, Simulate Output, LUT profiles), Show Channel, premultiply, round to 8 bits. +So exposure reaches the full float range, and tone mapping and the monitor conversion see the +exposed light. The pass runs on egui-wgpu's device (it needs no GPU compositor) into one RGBA8 +texture with mips per viewer slot (main viewer, snapshot, each passive viewer: about 110 MB of +video memory per slot for a 4K frame, input included); without a device the same steps run on +the CPU (`DisplayPass::pixel`, which is also the shader's test reference). 8 and 16 bpc frames +stay 8-bit with the HDR step applied when rendered; they go through the same pass when exposure, +a channel view or a display conversion applies, else they are drawn as they are. Panels that +read the viewer's pixels as 8-bit (Video Preview, keyers, thumbnails) get the half-float frame +converted on demand with the HDR step and without the exposure; the Info panel shows the +frame's float values. The browser keeps 8-bit frames for now (its frame workers send RGBA8). + A persistent **disk cache** (`render::disk_cache`, Settings ▸ Media & Disk Cache) backs both the layer cache and the viewer's RAM preview: layer buffers that were slow to render and every preview frame are written (LZ4, atomic rename, checksummed) under 128-bit content keys — the comp's content hash (the comp, what it uses, footage file size and time) plus frame, scale and view for frames; the layer key salted with the footage fingerprint for layers — so entries -survive restarts and never go stale. LRU eviction keeps it under the size limit; Edit ▸ Purge +survive restarts and never go stale. Frame entries are 8-bit RGBA, or for 32 bpc viewer frames +half floats with their bytes shuffled into planes before LZ4 (the entry header's format byte +tells them apart; the render options in the key keep the two kinds apart). LRU eviction keeps it under the size limit; Edit ▸ Purge empties it, `cache.diskStats` reports it, and the timeline draws disk-only frames in blue. In the browser the frames live in the Origin Private File System: frame workers write them, the page keeps the index (`DiskIndex`) and reads hits ([web.md](web.md#disk-cache)).