diff --git a/crates/video/README.md b/crates/video/README.md index 309b2b8f8..f9db3a6e6 100644 --- a/crates/video/README.md +++ b/crates/video/README.md @@ -21,7 +21,7 @@ No codec feature is on by default. | `dnx` | [`dnx`](src/dnx/README.md) | Avid DNxHD / DNxHR (SMPTE ST 2019-1 VC-3): decoder and encoder | | `apv` | [`apv`](src/apv/README.md) | APV (Advanced Professional Video, RFC 9924): decoder (`apv-decode` alone) and encoder (`apv-encode` alone) | | `vp8` | `vp8` | VP8 (RFC 6386): decoder | -| `mpeg2` | [`mpeg2`](src/mpeg2/README.md) | MPEG-2 video (ITU-T H.262) and MPEG-1 video (ISO/IEC 11172-2): decoder (`mpeg2-decode` alone); MPEG-2 encoder with I, P and B pictures (`mpeg2-encode`, which also turns on the decoder) | +| `mpeg2` | [`mpeg2`](src/mpeg2/README.md) | MPEG-2 video (ITU-T H.262) and MPEG-1 video (ISO/IEC 11172-2): decoder (`mpeg2-decode` alone); MPEG-2 encoder with I, P and B pictures and constant or variable bitrate within the VBV buffer (`mpeg2-encode`, which also turns on the decoder) | | `mpeg4` | [`mpeg4`](src/mpeg4/README.md) | MPEG-4 Part 2 video (ISO/IEC 14496-2, the Xvid/DivX format): Simple profile encoder (`mpeg4-encode`) | | `hap` | `hap` | Vidvox Hap: Hap, Hap Alpha, Hap Q, Hap Q Alpha and Hap Alpha-Only, decoder and encoder, with the Snappy and chunked second stages. DXT1, DXT5, scaled YCoCg-DXT5 and BC4 pictures decode in software; BC7 (Hap R) and BC6 (Hap HDR) frames are read and their blocks returned but not decoded to pictures. | | `mjpeg` | `mjpeg` | Motion JPEG: a decoder (baseline, extended and progressive Huffman JPEG; the Annex K tables for AVI `MJPG` packets that have none) and a baseline encoder, both over `craft-jpeg`. No two-field (MJPEG-A/B) packets and no lossless JPEG. | diff --git a/crates/video/src/lib.rs b/crates/video/src/lib.rs index ec2243913..17b22f320 100644 --- a/crates/video/src/lib.rs +++ b/crates/video/src/lib.rs @@ -8,7 +8,7 @@ //! | `dnx` | [`dnx`] | Avid DNxHD / DNxHR (SMPTE ST 2019-1): decoder and encoder | //! | `apv` | [`apv`] | APV (RFC 9924): decoder (`apv-decode`) and encoder (`apv-encode`) | //! | `vp8` | [`vp8`] | VP8 (RFC 6386): decoder (`vp8-decode`) and key-frame encoder (`vp8-encode`) | -//! | `mpeg2` | [`mpeg2`] | MPEG-2 video (ITU-T H.262) and MPEG-1 video: decoder (`mpeg2-decode`) and MPEG-2 I/P/B encoder (`mpeg2-encode`) | +//! | `mpeg2` | [`mpeg2`] | MPEG-2 video (ITU-T H.262) and MPEG-1 video: decoder (`mpeg2-decode`) and MPEG-2 I/P/B encoder with CBR / VBR (`mpeg2-encode`) | //! | `mpeg4` | [`mpeg4`] | MPEG-4 Part 2 video (ISO/IEC 14496-2, Xvid/DivX): Simple profile encoder (`mpeg4-encode`) | //! | `ffv1` | [`ffv1`] | FFV1 lossless video (RFC 9043): version 3, range coder, decoder and encoder | //! | `hap` | [`hap`] | Vidvox Hap, Hap Alpha, Hap Q, Hap Q Alpha, Hap Alpha-Only: decoder and encoder | diff --git a/crates/video/src/mpeg2/README.md b/crates/video/src/mpeg2/README.md index 069fd67bd..bddcae904 100644 --- a/crates/video/src/mpeg2/README.md +++ b/crates/video/src/mpeg2/README.md @@ -78,14 +78,16 @@ the synthetic-stream tests confirmed against FFmpeg: ## Encoder (feature `mpeg2-encode`) The encoder is built in steps, tracked in storytold/craft-libs#1133: I and P pictures came in -#1136, B pictures in the second step. It writes valid MPEG-2 elementary streams, but not yet DVD or +#1136, B pictures in #1195, and constant and variable bitrate with a VBV guarantee in the third +step. It writes valid MPEG-2 elementary streams that meet their VBV buffer, but not yet DVD or broadcast deliverables: those need the later steps listed below. ```rust use craft_video::mpeg2::{Encoder, EncoderConfig, RateControl, YuvFrame}; let mut cfg = EncoderConfig::new(720, 576, 25, 1); -cfg.rate = RateControl::Abr { kbps: 6000 }; // or RateControl::Quantiser(1..=31) +cfg.rate = RateControl::Vbr { kbps: 6000, max_kbps: 9800 }; // or Cbr, Abr, Quantiser(1..=31) +cfg.vbv_buffer_bits = Some(1_835_008); // default: the level's maximum cfg.gop_size = 12; // pictures from one I picture to the next cfg.b_pictures = 2; // I B B P B B P ...; 0 for I and P only cfg.closed_gop = true; // or open GOPs @@ -101,7 +103,8 @@ es.extend_from_slice(enc.end_of_sequence()); What it does: - Main Profile (4:2:0) at Low, Main, High-1440 and High Level, and 4:2:2 Profile at Main and - High Level. The lowest level that holds the size, frame rate and (average) bitrate is chosen; + High Level. The lowest level that holds the size, frame rate, bitrate (the peak of a variable + bitrate) and VBV buffer size is chosen; sizes beyond High Level are refused. Any even picture size; edges are padded by repetition. - Progressive frame pictures, I, P and B. Every GOP starts with a sequence header, sequence extension and GOP header (with a time code), so every I picture is a random access point. @@ -128,17 +131,32 @@ What it does: previous macroblock's directions with the predictors), whichever costs least, or intra. Interpolated predictions are averaged in the decoder's order. - Adaptive quantisation by macroblock activity (Test Model 5 §5.2.3). -- Rate control: a constant quantiser (B pictures at 1.25 times it), or single-pass average - bitrate with Test Model 5's target bit allocation at picture level (B pictures weighted by - K_b = 1.4). Streams are signalled as variable bitrate - (`vbv_delay` 0xFFFF, the level's maximum bitrate and VBV size in the sequence header); the VBV - buffer is not modelled. +- Rate control: a constant quantiser (B pictures at 1.25 times it), or single-pass average, + constant or variable bitrate with Test Model 5's target bit allocation at picture level (B + pictures weighted by K_b = 1.4). The sequence header carries the constant bitrate, the + variable bitrate's peak, or for a constant quantiser and average bitrate the level's maximum, + rounded up to 400 bit/s, and the VBV buffer size (the level's maximum unless set). +- VBV guarantee (H.262 Annex C): the encoder models the decoder's buffer in decoding order, + exactly (fullness in 1/fps_num bit), and caps every picture at what the buffer holds at its + decoding time. Average bitrate control aims a picture at no more than 85 % of that. A + picture over its cap is coded again with a coarser quantiser (three times, then the coarsest); + an I picture still too large keeps only its DC coefficients, and as a last resort the picture + becomes the smallest there is: a grey I picture, or a P or B picture that repeats its + reference with zero vectors and no residual. That picture's size depends only on the picture + size, so `Encoder::new` codes one of each type and refuses a bitrate whose picture interval + brings fewer bits, or a buffer that cannot hold it; then the buffer cannot underflow. +- Constant bitrate: each picture header carries its `vbv_delay`; a picture that would let the + buffer overflow is padded with zero bytes before the next start code; the first picture is + decoded when the buffer is 7/8 full; the buffer used is at most what 0.728 s of the bitrate + fills (the longest `vbv_delay`), and must hold two picture intervals. Variable bitrate + (`vbv_delay` 0xFFFF): the buffer fills at the peak bitrate until full, before the first + picture too. The model takes the pictures to be decoded one picture interval apart. - The reconstruction uses the decoder's own dequantisation, mismatch control, IDCT and motion compensation, so this crate's decoder outputs exactly the encoder's reconstruction. - Slices (one per macroblock row) are coded in parallel on rayon with `threads`; the stream is identical either way. -Later steps (#1133): CBR and VBR with a VBV guarantee; DVD presets +Later steps (#1133): DVD presets (sizes, aspect, GOP length, 9.8 Mb/s, 23.976 soft telecine); interlaced coding (field/frame DCT, field prediction, alternate scan) and 4:2:2 High Level CBR for XDCAM HD422. MPEG-2 essence in the MXF writer and a program stream muxer belong to `craft-media`. @@ -213,9 +231,24 @@ fell to 77–150 fps). These are lower bounds. decodes from its I picture with its leading B pictures and an open one without them; B pictures are smaller than P pictures; uneven presentation times are refused with B pictures; quality and size follow the quantiser; repeated pictures skip; levels and aspect in the sequence header; - bad configurations and short or inconsistent planes are refused. + bad configurations and short or inconsistent planes are refused. A VBV checker that reads only + the stream (the sequence header's bit rate and buffer size, each picture's size and + `vbv_delay`) replays Annex C: constant bitrate streams (with and without B pictures, open GOPs, + intra only, a 48 kbit buffer, uniform noise, a still picture, 4:2:2 at 29.97 fps) decode one + picture interval apart, never underflow or overflow, and carry their bitrate to within one + buffer; variable bitrate, average bitrate and constant quantiser streams (noise at quantiser 1 + into a 64 kbit buffer, intra only, 4 B pictures) never underflow; all of them decode exactly to + the reconstruction, including the pictures that fell back to DC only or to the smallest + picture. Still pictures at a constant bitrate are padded; the smallest pictures are the same + size for any source; the sequence header carries the bit rate (rounded up to 400 bit/s) and + buffer size; bitrates below the smallest picture, buffers too small for it (or, at a constant + bitrate, for two picture intervals) are refused, and the lowest bitrate accepted for CIF + meets the buffer with noise. The `vbv.rs` model itself: both bounds of a constant bitrate + buffer, a full variable bitrate buffer stops filling, 29.97 fps does not drift. - `tests/mpeg2/encode.rs`: FFmpeg decodes our streams of `testsrc2` (CIF and 576-line 4:2:2; without B pictures, with 2 and 4, closed and open GOPs) without errors and within the decoder suite's tolerance of our reconstruction, ffprobe reports the profile, level, pixel format and picture types in display order, and average bitrate lands within 10 % of 800 and 2500 kbit/s - targets. + targets; constant bitrate CIF streams at 1500 and 3000 kbit/s decode and carry their bitrate + to within one buffer (1540 and 3021 kbit/s over 4 s with a 256 kbit buffer); a DVD-like + 720×576 stream (variable bitrate, 9.8 Mbit/s peak, 1835008-bit buffer) decodes. diff --git a/crates/video/src/mpeg2/encode/bitwriter.rs b/crates/video/src/mpeg2/encode/bitwriter.rs index 75b363d47..f014b4861 100644 --- a/crates/video/src/mpeg2/encode/bitwriter.rs +++ b/crates/video/src/mpeg2/encode/bitwriter.rs @@ -47,7 +47,6 @@ impl BitWriter { } /// Bits written so far. - #[cfg(test)] pub fn bits(&self) -> u64 { self.out.len() as u64 * 8 + u64::from(self.n) } diff --git a/crates/video/src/mpeg2/encode/mod.rs b/crates/video/src/mpeg2/encode/mod.rs index af32b271d..28cc6ec61 100644 --- a/crates/video/src/mpeg2/encode/mod.rs +++ b/crates/video/src/mpeg2/encode/mod.rs @@ -2,8 +2,22 @@ //! //! MPEG-2 encoding in this crate is built in steps (storytold/craft-libs#1133). So far: I, P and B //! pictures, progressive frame pictures, 4:2:0 (Main Profile) and 4:2:2 (4:2:2 Profile), closed or -//! open GOPs, constant quantiser or single-pass average bitrate. A VBV guarantee, interlaced -//! coding and DVD / broadcast presets come later. +//! open GOPs, and constant quantiser, average, constant or variable bitrate. Every stream meets +//! the VBV buffer of its sequence header (H.262 Annex C). Interlaced coding and DVD / broadcast +//! presets come later. +//! +//! ## The VBV buffer +//! +//! The encoder models the decoder's input buffer in decoding order and caps each picture at +//! what the buffer holds at its decoding time. A picture over its cap is coded again with a +//! coarser quantiser; if the coarsest is still too large, an I picture drops its AC +//! coefficients, and as a last resort the picture is replaced by the smallest one there is (a +//! grey I picture, or a P or B picture that repeats its reference). That one's size depends only +//! on the picture size, and [`Encoder::new`] refuses a bitrate that cannot carry it every +//! picture interval, so the buffer can never underflow. With [`RateControl::Cbr`] the encoder +//! also writes each picture's `vbv_delay` and pads pictures with zero bytes before the next +//! start code where the buffer would otherwise overflow. The model takes the pictures to be +//! decoded one picture interval apart, as in an elementary stream without gaps. //! //! ```no_run //! use craft_video::mpeg2::encode::{Encoder, EncoderConfig, RateControl, YuvFrame}; @@ -32,6 +46,7 @@ mod picture; mod rate; mod tables; mod transform; +mod vbv; #[cfg(test)] mod tests; @@ -43,8 +58,9 @@ use crate::mpeg2::headers::{DEFAULT_INTRA, PictureType, SCAN}; use crate::mpeg2::mc::Frame; use bitwriter::BitWriter; -use picture::{PicCtx, RowOut, RowPlanes, encode_row, qcode_for, qscale}; +use picture::{Fallback, PicCtx, RowOut, RowPlanes, encode_row, qcode_for, qscale}; use rate::RateCtl; +use vbv::Vbv; #[derive(Debug, thiserror::Error)] pub enum EncodeError { @@ -86,10 +102,19 @@ impl AspectRatio { pub enum RateControl { /// A constant quantiser_scale_code (1..=31; mapped through H.262 Table 7-6 with /// `nonlinear_quant`). Adaptive quantisation still varies it per macroblock when enabled. + /// Signalled as variable bitrate at the level's maximum; a picture too large for the VBV + /// buffer is coded coarser. Quantiser(u8), - /// Single-pass average bitrate in kbit/s. The stream is signalled as variable bitrate and - /// is not checked against the VBV buffer. + /// Single-pass average bitrate in kbit/s, signalled as variable bitrate at the level's + /// maximum. Abr { kbps: u32 }, + /// Constant bitrate in kbit/s (rounded up to a multiple of 400 bit/s, the unit of the + /// sequence header): every picture carries its `vbv_delay`, and the buffer neither + /// underflows nor overflows. + Cbr { kbps: u32 }, + /// Variable bitrate: an average of `kbps`, with the VBV buffer filled at `max_kbps` (the + /// sequence header's bit rate, rounded up to a multiple of 400 bit/s). + Vbr { kbps: u32, max_kbps: u32 }, } #[derive(Clone, Debug)] @@ -113,6 +138,10 @@ pub struct EncoderConfig { /// GOP decodes on its own. Open GOPs let them predict from the previous GOP too. pub closed_gop: bool, pub rate: RateControl, + /// VBV buffer size in bits (stored in units of 16384, rounded down); `None`: the level's + /// maximum. With a constant bitrate the encoder uses at most what 0.728 s of the bitrate + /// fills, the longest `vbv_delay` there is. + pub vbv_buffer_bits: Option, /// q_scale_type: the non-linear quantiser scale (finer steps at low quantisers). pub nonlinear_quant: bool, /// Vary the quantiser per macroblock with its activity (finer in flat areas). @@ -141,6 +170,7 @@ impl EncoderConfig { b_pictures: 2, closed_gop: true, rate: RateControl::Quantiser(4), + vbv_buffer_bits: None, nonlinear_quant: true, adaptive_quant: true, intra_dc_bits: 9, @@ -167,7 +197,7 @@ pub struct YuvFrame<'a> { #[derive(Clone, Debug)] pub struct Packet { /// Elementary stream bytes: the sequence header and GOP header before every I picture, then - /// the picture. + /// the picture, then (constant bitrate) any zero bytes of stuffing. pub data: Vec, pub pts: i64, /// Decoding timestamp: `pts` without B pictures; with them, the presentation times in @@ -243,9 +273,10 @@ fn choose_level(cfg: &EncoderConfig) -> Result { let fps = f64::from(cfg.fps_num) / f64::from(cfg.fps_den); let luma_rate = f64::from(cfg.width) * f64::from(cfg.height) * fps; let bps = match cfg.rate { - RateControl::Abr { kbps } => u64::from(kbps) * 1000, + RateControl::Abr { kbps } | RateControl::Cbr { kbps } | RateControl::Vbr { max_kbps: kbps, .. } => u64::from(kbps) * 1000, RateControl::Quantiser(_) => 0, }; + let vbv = cfg.vbv_buffer_bits.map_or(0, u64::from); levels .iter() .find(|l| { @@ -254,6 +285,7 @@ fn choose_level(cfg: &EncoderConfig) -> Result { && fps <= f64::from(l.max_fps) + 0.01 && luma_rate <= l.luma_rate as f64 && bps <= l.bit_rate + && vbv <= l.vbv_bits }) .copied() .ok_or_else(|| { @@ -263,6 +295,7 @@ fn choose_level(cfg: &EncoderConfig) -> Result { cfg.height, fps, if bps > 0 { format!(", {bps} bit/s") } else { String::new() } + + &if vbv > 0 { format!(", a {vbv}-bit VBV buffer") } else { String::new() } )) }) } @@ -287,9 +320,13 @@ fn validate(cfg: &EncoderConfig) -> Result<(), EncodeError> { } match cfg.rate { RateControl::Quantiser(q) if !(1..=31).contains(&q) => return bad(format!("quantiser_scale_code {q} (1..=31)")), - RateControl::Abr { kbps: 0 } => return bad("bitrate 0".into()), + RateControl::Abr { kbps: 0 } | RateControl::Cbr { kbps: 0 } | RateControl::Vbr { kbps: 0, .. } => return bad("bitrate 0".into()), + RateControl::Vbr { kbps, max_kbps } if max_kbps < kbps => return bad(format!("peak bitrate {max_kbps} kbit/s below the average {kbps}")), _ => {} } + if cfg.vbv_buffer_bits.is_some_and(|b| b < 16384) { + return bad(format!("VBV buffer of {} bits (at least 16384)", cfg.vbv_buffer_bits.unwrap_or(0))); + } if !(8..=10).contains(&cfg.intra_dc_bits) { return bad(format!("intra DC precision {} bits (8..=10)", cfg.intra_dc_bits)); } @@ -323,8 +360,27 @@ struct Coded { recon: Frame, mvs: Vec<[i32; 2]>, avg_q: f64, + /// Zero bits appended for the VBV buffer (constant bitrate). + stuffing: u64, } +impl Coded { + fn bits(&self) -> u64 { + self.packet.data.len() as u64 * 8 + } +} + +/// Times a picture too large for the VBV buffer is coded again with a coarser quantiser before +/// the fallbacks. +const RECODES: usize = 3; + +/// Share of the VBV buffer's fullness that average bitrate control aims a picture at, at most. +const VBV_TARGET: f64 = 0.85; + +/// Fullness, as a share of the buffer, at which a constant bitrate stream's first picture is +/// decoded. +const CBR_START: (u64, u64) = (7, 8); + /// Quantiser ratio of B to P pictures with a constant quantiser (TM5 uses 1.4 for its rate /// allocation; a constant quantiser keeps B pictures a little closer to the anchors). const B_QUANT_FACTOR: f64 = 1.25; @@ -366,6 +422,11 @@ pub struct Encoder { /// Packets waiting for the presentation time step (the decoding time offset). out: Vec, rc: Option, + /// bit_rate_value and vbv_buffer_size_value of the sequence header (units of 400 bit/s and + /// 16384 bits). + bit_rate_value: u64, + vbv_value: u64, + vbv: Vbv, recon_capture: bool, recon: Vec<(u64, ReconFrame)>, } @@ -377,10 +438,25 @@ impl Encoder { let (shx, shy) = if cfg.chroma == ChromaFormat::Yuv422 { (1, 0) } else { (1, 1) }; let nonlinear = cfg.nonlinear_quant; let q_range = (f64::from(qscale(nonlinear, 1)), f64::from(qscale(nonlinear, 31))); + let fps = f64::from(cfg.fps_num) / f64::from(cfg.fps_den); + let (peak, cbr) = match cfg.rate { + RateControl::Cbr { kbps } => (u64::from(kbps) * 1000, true), + RateControl::Vbr { max_kbps, .. } => (u64::from(max_kbps) * 1000, false), + RateControl::Quantiser(_) | RateControl::Abr { .. } => (level.bit_rate, false), + }; + let bit_rate_value = peak.div_ceil(400); + let vbv_value = cfg.vbv_buffer_bits.map_or(level.vbv_bits, u64::from) / 16384; + let (rate, mut size) = (bit_rate_value * 400, vbv_value * 16384); + if cbr { + // the longest vbv_delay (0xFFFE ticks of 90 kHz) bounds what may wait in the buffer + size = size.min(rate * 0xFFFE / 90_000); + } let rc = match cfg.rate { - RateControl::Abr { kbps } => Some(RateCtl::new(f64::from(kbps) * 1000.0, f64::from(cfg.fps_num) / f64::from(cfg.fps_den), q_range)), + RateControl::Abr { kbps } | RateControl::Vbr { kbps, .. } => Some(RateCtl::new(f64::from(kbps) * 1000.0, fps, q_range)), + RateControl::Cbr { .. } => Some(RateCtl::new(rate as f64, fps, q_range)), RateControl::Quantiser(_) => None, }; + let vbv = Vbv::new(size, rate, (cfg.fps_num, cfg.fps_den), cbr, size / CBR_START.1 * CBR_START.0); let mut enc = Encoder { level, mb_w: (cfg.width as usize).div_ceil(16), @@ -404,14 +480,50 @@ impl Encoder { pts_step: None, out: Vec::new(), rc, + bit_rate_value, + vbv_value, + vbv, recon_capture: false, recon: Vec::new(), cfg, }; enc.seq_header = enc.sequence_header_bytes(); + // every picture can shrink to the smallest one, so the buffer cannot underflow if that + // fits in it and one picture interval brings at least that many bits (the fullness at a + // decoding time is never below one interval's worth, or the whole buffer when it holds + // less). A constant bitrate also needs room to pad pictures within: two intervals. + let fill = enc.vbv.per_picture(); + let smallest = enc.smallest_picture(); + let (w, h) = (enc.cfg.width, enc.cfg.height); + if fill < smallest { + return Err(EncodeError::Config(format!( + "{rate} bit/s at {fps:.3} fps brings {fill} bits a picture; the smallest {w}x{h} picture takes {smallest}" + ))); + } + if size < smallest { + return Err(EncodeError::Config(format!("a VBV buffer of {size} bits; the smallest {w}x{h} picture takes {smallest}"))); + } + if cbr && size < 2 * fill { + return Err(EncodeError::Config(format!("a VBV buffer of {size} bits holds less than two pictures of {fill} bits at {rate} bit/s"))); + } Ok(enc) } + /// Bits of the largest of the minimal I, P and B pictures (with the I picture's sequence + /// and GOP headers). They do not depend on the source, so a grey one tells. + fn smallest_picture(&self) -> u64 { + let mut grey = Frame::new(self.mb_w * 16, self.mb_h * 16, (self.mb_w * 16) >> self.shx, (self.mb_h * 16) >> self.shy); + for p in &mut grey.planes { + p.fill(128); + } + let h = Held { src: grey.clone(), pts: 0, n: self.gop_first }; + [(PictureType::I, None, None), (PictureType::P, Some(&grey), None), (PictureType::B, Some(&grey), Some(&grey))] + .into_iter() + .map(|(t, fwd, bwd)| self.code_picture(&h, t, (fwd, bwd), [&[], &[]], 8.0, Fallback::Minimal).bits()) + .max() + .unwrap_or(0) + } + pub fn config(&self) -> &EncoderConfig { &self.cfg } @@ -467,8 +579,8 @@ impl Encoder { fn sequence_header_bytes(&self) -> Vec { let c = &self.cfg; let mut w = BitWriter::new(); - let bit_rate = self.level.bit_rate.div_ceil(400); - let vbv = self.level.vbv_bits / 16384; + let bit_rate = self.bit_rate_value; + let vbv = self.vbv_value; w.start_code(0xB3); w.put(c.width & 0xFFF, 12); w.put(c.height & 0xFFF, 12); @@ -515,8 +627,11 @@ impl Encoder { w.bit(false); // broken_link } - fn picture_header(&self, w: &mut BitWriter, ptype: PictureType, temporal_reference: u64) { + /// picture_header and picture_coding_extension. `head_bits`: the bits before the picture + /// start code (sequence and GOP headers), for the `vbv_delay`. + fn picture_header(&self, w: &mut BitWriter, ptype: PictureType, temporal_reference: u64, head_bits: u64) { w.start_code(0x00); + let delay = self.vbv.delay(head_bits + w.bits()); w.put((temporal_reference & 0x3FF) as u32, 10); // display order within the GOP w.put( match ptype { @@ -526,7 +641,7 @@ impl Encoder { }, 3, ); - w.put(0xFFFF, 16); // vbv_delay: variable bitrate + w.put(u32::from(delay), 16); // vbv_delay (0xFFFF: variable bitrate) for dir in [PictureType::P, PictureType::B] { if ptype == dir || ptype == PictureType::B { // full_pel_{forward,backward}_vector and {forward,backward}_f_code: MPEG-2 @@ -665,7 +780,7 @@ impl Encoder { } let fwd = if intra { None } else { self.latest.as_ref().map(|r| &r.0) }; let cand: &[[i32; 2]] = if intra { &[] } else { &self.anchor_mvs }; - let c = self.code_picture(&h, ptype, (fwd, None), [cand, &[]]); + let c = self.code_fitting(&h, ptype, (fwd, None), [cand, &[]]); let (recon, mvs) = self.finish(c, h.n, out); self.older = self.latest.take(); self.latest = Some((recon, h.n)); @@ -689,22 +804,74 @@ impl Encoder { }; let cf = fwd.map_or(Vec::new(), |f| scaled(b.n as i64 - f.1 as i64)); let cb = scaled(b.n as i64 - *bn as i64); - let c = self.code_picture(&b, PictureType::B, (fwd.map(|f| &f.0), Some(bwd)), [&cf, &cb]); + let c = self.code_fitting(&b, PictureType::B, (fwd.map(|f| &f.0), Some(bwd)), [&cf, &cb]); self.finish(c, b.n, out); } - /// Code one picture with the given references and search candidates. - fn code_picture(&self, h: &Held, ptype: PictureType, (fwd, bwd): (Option<&Frame>, Option<&Frame>), cand: [&[[i32; 2]]; 2]) -> Coded { - let base = match (&self.rc, self.cfg.rate) { - (Some(rc), _) => rc.qscale(ptype), + /// The picture-level quantiser_scale for the next picture of type `ptype`. + fn base_quant(&self, ptype: PictureType) -> f64 { + match (&self.rc, self.cfg.rate) { + (Some(rc), _) => rc.qscale(ptype, (self.vbv.min_bits() as f64, self.vbv.max_bits() as f64 * VBV_TARGET)), (None, RateControl::Quantiser(q)) => { let qs = f64::from(qscale(self.cfg.nonlinear_quant, q)); if ptype == PictureType::B { qs * B_QUANT_FACTOR } else { qs } } (None, _) => 8.0, - }; + } + } + + /// Code a picture to fit the VBV buffer: coarser while it is too large, then the fallbacks + /// (see the module documentation), then pad it with zero bytes if the buffer would + /// overflow. + fn code_fitting(&self, h: &Held, ptype: PictureType, refs: (Option<&Frame>, Option<&Frame>), cand: [&[[i32; 2]]; 2]) -> Coded { + let max = self.vbv.max_bits(); + let q_max = f64::from(qscale(self.cfg.nonlinear_quant, 31)); + let mut base = self.base_quant(ptype); + let mut c = self.code_picture(h, ptype, refs, cand, base, Fallback::None); + for _ in 0..RECODES { + if c.bits() <= max || base >= q_max { + break; + } + // the size goes roughly with the inverse of the quantiser; aim a little under + base = (base * c.bits() as f64 / max.max(1) as f64 * 1.1).min(q_max); + c = self.code_picture(h, ptype, refs, cand, base, Fallback::None); + } + if c.bits() > max && base < q_max { + c = self.code_picture(h, ptype, refs, cand, q_max, Fallback::None); + } + if c.bits() > max && ptype == PictureType::I { + c = self.code_picture(h, ptype, refs, cand, q_max, Fallback::DcOnly); + } + if c.bits() > max { + // fits: `Encoder::new` made sure one picture interval brings this many bits + c = self.code_picture(h, ptype, refs, cand, q_max, Fallback::Minimal); + } + let min = self.vbv.min_bits(); + if c.bits() < min { + let pad = (min - c.bits()).div_ceil(8); + c.packet.data.resize(c.packet.data.len() + pad as usize, 0); + c.stuffing = pad * 8; + } + c + } + + /// Code one picture with the given references, search candidates and picture quantiser. + fn code_picture( + &self, + h: &Held, + ptype: PictureType, + (fwd, bwd): (Option<&Frame>, Option<&Frame>), + cand: [&[[i32; 2]]; 2], + base: f64, + fallback: Fallback, + ) -> Coded { let src = &h.src; - let mb_qcode = self.mb_quant(src, base); + // the minimal picture keeps one quantiser, so its size does not depend on the source + let mb_qcode = if fallback == Fallback::Minimal { + vec![qcode_for(self.cfg.nonlinear_quant, base); self.mb_w * self.mb_h] + } else { + self.mb_quant(src, base) + }; let mut recon = Frame::new(src.w, src.h, src.cw, src.ch); let ctx = PicCtx { ptype, @@ -722,6 +889,7 @@ impl Encoder { f_code: self.f_code, search: self.cfg.search_range as i32, mb_qcode: &mb_qcode, + fallback, }; let rows = encode_rows(&ctx, &mut recon, self.cfg.threads); let intra = ptype == PictureType::I; @@ -729,7 +897,8 @@ impl Encoder { if intra { self.gop_header(&mut w, self.gop_first, self.gop_closed); } - self.picture_header(&mut w, ptype, h.n - self.gop_first); + let head = if intra { self.seq_header.len() as u64 * 8 } else { 0 }; + self.picture_header(&mut w, ptype, h.n.saturating_sub(self.gop_first), head); let mut data = if intra { self.seq_header.clone() } else { Vec::new() }; data.extend_from_slice(&w.finish()); let mut qsum = 0u64; @@ -741,16 +910,19 @@ impl Encoder { } let avg_q = qsum as f64 / (self.mb_w * self.mb_h).max(1) as f64; let packet = Packet { data, pts: h.pts, dts: h.pts, keyframe: intra, picture_type: ptype, quantiser_scale: avg_q as f32 }; - Coded { packet, recon, mvs, avg_q } + Coded { packet, recon, mvs, avg_q, stuffing: 0 } } /// Account for a coded picture (display index `n`): rate control, its decoding time, the /// reconstruction capture. Returns the reconstruction and the picture's vectors. fn finish(&mut self, c: Coded, n: u64, out: &mut Vec) -> (Frame, Vec<[i32; 2]>) { - let Coded { mut packet, recon, mvs, avg_q } = c; + let Coded { mut packet, recon, mvs, avg_q, stuffing } = c; + let bits = packet.data.len() as u64 * 8; if let Some(rc) = self.rc.as_mut() { - rc.update(packet.picture_type, packet.data.len() as u64 * 8, avg_q); + rc.update(packet.picture_type, bits - stuffing, avg_q); + rc.spend(stuffing); } + self.vbv.remove(bits); // the k-th coded picture decodes at the k-th presentation time (less the offset that // `release` subtracts); one time is queued per input picture packet.dts = self.display_pts.pop_front().unwrap_or(packet.pts); diff --git a/crates/video/src/mpeg2/encode/picture.rs b/crates/video/src/mpeg2/encode/picture.rs index 0be5dbafd..37087edff 100644 --- a/crates/video/src/mpeg2/encode/picture.rs +++ b/crates/video/src/mpeg2/encode/picture.rs @@ -33,6 +33,20 @@ pub(crate) fn qcode_for(nonlinear: bool, qs: f64) -> u8 { best.0 } +/// How far a picture gives up quality to fit the VBV buffer. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Fallback { + /// Coded normally. + None, + /// Intra blocks keep only their DC coefficient (I pictures). + DcOnly, + /// The smallest picture there is, whatever the source: an I picture of mid-grey blocks (every + /// DC equal to its predictor, no AC), or a P or B picture whose macroblocks are skipped or + /// predicted with a zero vector and no residual (backward in a B picture), which repeats the + /// reference. Its size depends only on the picture size and the headers. + Minimal, +} + /// What every slice of one picture shares (read-only). pub(crate) struct PicCtx<'a> { /// I, P or B. @@ -58,6 +72,7 @@ pub(crate) struct PicCtx<'a> { pub search: i32, /// quantiser_scale_code per macroblock. pub mb_qcode: &'a [u8], + pub fallback: Fallback, } impl PicCtx<'_> { @@ -223,6 +238,16 @@ fn macroblock(ctx: &PicCtx, st: &mut SliceSt, w: &mut BitWriter, (mbx, mby): (us let planes = (ctx.src.w, ctx.src.cw); let lambda = ((qs * 3 + 4) / 8).max(1) as u32; let rec = match (ctx.ptype, ctx.fwd) { + (PictureType::P, Some(reference)) if ctx.fallback == Fallback::Minimal => { + let (rec, skipped) = inter_mb(ctx, st, w, &mb, reference, [0, 0]); + ro.stats.skipped += skipped as u32; + Some(rec) + } + (PictureType::B, _) if ctx.bwd.is_some() && ctx.fallback == Fallback::Minimal => { + let (rec, skipped) = b_mb(ctx, st, w, &mb, BPred { dirs: MB_BWD, mvs: [[0; 2]; 2] }); + ro.stats.skipped += skipped as u32; + Some(rec) + } (PictureType::P, Some(reference)) => { let search = Search { reference, src: &src.y, mbx, mby, f_code: ctx.f_code, range: ctx.search, pmv: st.pmv[0], lambda }; let found = search.run(&candidates(ctx, st, addr, 0)); @@ -391,11 +416,19 @@ fn intra_mb(ctx: &PicCtx, st: &mut SliceSt, w: &mut BitWriter, mb: &MbIn) -> MbP let mut rec = mb.src.clone(); for i in 0..ctx.blocks() { let cc = if i < 4 { 0 } else { 1 + (i & 1) }; - let f = fdct(&mb.src.block(i, cbw)); let mut levels = [0i32; 64]; - levels[0] = ((f[0] / dc_mult as f32).round() as i32).clamp(0, dc_max); - for pos in 1..64 { - levels[pos] = quant_intra(f[pos], ctx.intra_matrix[pos], qs); + // The minimal fallback repeats the DC predictor and codes no coefficients, so it needs + // no transform. + if ctx.fallback == Fallback::Minimal { + levels[0] = st.dc[cc]; + } else { + let f = fdct(&mb.src.block(i, cbw)); + levels[0] = ((f[0] / dc_mult as f32).round() as i32).clamp(0, dc_max); + if ctx.fallback == Fallback::None { + for pos in 1..64 { + levels[pos] = quant_intra(f[pos], ctx.intra_matrix[pos], qs); + } + } } let diff = levels[0] - st.dc[cc]; st.dc[cc] = levels[0]; @@ -464,6 +497,9 @@ fn residual(ctx: &PicCtx, mb: &MbIn, pred: &MbPix, qs: i32) -> ([[i32; 64]; 8], let n = ctx.blocks(); let mut levels = [[0i32; 64]; 8]; let mut cbp = 0u32; + if ctx.fallback == Fallback::Minimal { + return (levels, cbp); + } for i in 0..n { let (s, p) = (mb.src.block(i, cbw), pred.block(i, cbw)); let f = fdct(&std::array::from_fn(|k| s[k] - p[k])); @@ -617,6 +653,7 @@ mod tests { f_code: 3, search: 31, mb_qcode: &[4; 3], + fallback: Fallback::None, }; let bad = BPred { dirs: MB_FWD, mvs: [[34, 0], [0, 0]] }; // the bitstream can carry the vector (f_code 3: -64..64): only the position rules it out diff --git a/crates/video/src/mpeg2/encode/rate.rs b/crates/video/src/mpeg2/encode/rate.rs index 96c6f2aa1..7accd5172 100644 --- a/crates/video/src/mpeg2/encode/rate.rs +++ b/crates/video/src/mpeg2/encode/rate.rs @@ -58,8 +58,14 @@ impl RateCtl { self.nb = 0; } - /// quantiser_scale for the next picture of type `ptype`. - pub fn qscale(&self, ptype: PictureType) -> f64 { + /// Bits spent beyond the coded pictures (stuffing). + pub fn spend(&mut self, bits: u64) { + self.remaining -= bits as f64; + } + + /// quantiser_scale for the next picture of type `ptype`, its bit target kept within + /// `(min, max)` (the VBV buffer's bounds). + pub fn qscale(&self, ptype: PictureType, (min, max): (f64, f64)) -> f64 { let floor = self.bps / (8.0 * self.fps); let (np, nb) = (f64::from(self.np), f64::from(self.nb)); let (target, x) = match ptype { @@ -67,7 +73,7 @@ impl RateCtl { PictureType::B => (self.remaining / (nb.max(1.0) + np * KB * self.xp / self.xb), self.xb), _ => (self.remaining / (np.max(1.0) + nb * self.xb / (KB * self.xp)), self.xp), }; - (x / target.max(floor)).clamp(self.q_min, self.q_max) + (x / target.max(floor).min(max).max(min).max(1.0)).clamp(self.q_min, self.q_max) } /// A picture of type `ptype` used `bits` at average quantiser_scale `avg_q`. @@ -109,7 +115,7 @@ mod tests { for _ in 0..gops { rc.start_gop(np, 11 - np); for &t in &types { - let q = rc.qscale(t); + let q = rc.qscale(t, (0.0, f64::MAX)); let bits = c[t as usize] / q; rc.update(t, bits as u64, q); total += bits; @@ -134,7 +140,7 @@ mod tests { // a GOP of 12 that ends after its I picture and two P pictures rc.start_gop(11, 0); for t in [PictureType::I, PictureType::P, PictureType::P] { - let q = rc.qscale(t); + let q = rc.qscale(t, (0.0, f64::MAX)); rc.update(t, (1e7 / q) as u64, q); } rc.end_gop(9); @@ -150,11 +156,11 @@ mod tests { // a picture far larger than its budget pushes the next quantiser to the top rc.update(PictureType::I, 50_000_000, 10.0); rc.start_gop(1, 0); - assert_eq!(rc.qscale(PictureType::I), 62.0); + assert_eq!(rc.qscale(PictureType::I, (0.0, f64::MAX)), 62.0); // and a tiny one to the bottom let mut rc = RateCtl::new(1e6, 25.0, (2.0, 62.0)); rc.start_gop(1, 0); rc.update(PictureType::I, 10, 2.0); - assert_eq!(rc.qscale(PictureType::I), 2.0); + assert_eq!(rc.qscale(PictureType::I, (0.0, f64::MAX)), 2.0); } } diff --git a/crates/video/src/mpeg2/encode/tests.rs b/crates/video/src/mpeg2/encode/tests.rs index 01aac2b32..06b2c20fe 100644 --- a/crates/video/src/mpeg2/encode/tests.rs +++ b/crates/video/src/mpeg2/encode/tests.rs @@ -4,6 +4,7 @@ use super::*; use crate::mpeg2::{Decoder, probe}; +use picture::{Fallback, qscale}; /// A moving, textured test sequence: a gradient with a sharp pattern drifting by a fraction of a /// sample per picture, and a block moving across it. @@ -38,8 +39,34 @@ pub(crate) fn scene(w: usize, h: usize, cw: usize, ch: usize, t: usize) -> (Vec< (y, cb, cr) } +/// Uniform noise, different in every picture: as hard to code as content gets. +fn noise(w: usize, h: usize, cw: usize, ch: usize, t: usize) -> (Vec, Vec, Vec) { + let mut x = 0x9E37_79B9_7F4A_7C15u64 ^ (t as u64).wrapping_mul(0xD6E8_FEB8_6659_FD93); + let mut next = move || { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + (x >> 24) as u8 + }; + let y = (0..w * h).map(|_| next()).collect(); + let cb = (0..cw * ch).map(|_| next()).collect(); + let cr = (0..cw * ch).map(|_| next()).collect(); + (y, cb, cr) +} + +/// The scene's first picture, every time. +fn still(w: usize, h: usize, cw: usize, ch: usize, _t: usize) -> (Vec, Vec, Vec) { + scene(w, h, cw, ch, 0) +} + +type Source = fn(usize, usize, usize, usize, usize) -> (Vec, Vec, Vec); + /// Encode `frames` pictures of the scene; returns the stream and the reconstruction. fn encode(cfg: EncoderConfig, frames: usize) -> (Vec, Vec, Vec) { + encode_from(cfg, frames, scene) +} + +fn encode_from(cfg: EncoderConfig, frames: usize, source: Source) -> (Vec, Vec, Vec) { let (w, h) = (cfg.width as usize, cfg.height as usize); let (cw, ch) = (w / 2, if cfg.chroma == ChromaFormat::Yuv422 { h } else { h / 2 }); let mut enc = Encoder::new(cfg).unwrap(); @@ -47,7 +74,7 @@ fn encode(cfg: EncoderConfig, frames: usize) -> (Vec, Vec, Vec

Vec { } fn assert_decodes_to_recon(name: &str, cfg: EncoderConfig, frames: usize) -> Vec { - let (_, recon, packets) = encode(cfg, frames); + assert_decodes_to_recon_from(name, cfg, frames, scene) +} + +fn assert_decodes_to_recon_from(name: &str, cfg: EncoderConfig, frames: usize, source: Source) -> Vec { + let (_, recon, packets) = encode_from(cfg, frames, source); assert!(packets.windows(2).all(|w| w[0].dts < w[1].dts) && packets.iter().all(|p| p.dts <= p.pts), "{name}: decoding times"); let pics = decode(&packets); assert_eq!(pics.len(), frames, "{name}: picture count"); @@ -91,6 +122,81 @@ fn assert_decodes_to_recon(name: &str, cfg: EncoderConfig, frames: usize) -> Vec packets } +/// What [`check_vbv`] found. +#[derive(Debug)] +struct VbvRun { + cbr: bool, + /// Bit rate and buffer size from the sequence header. + rate: u64, + size: u64, + /// Bits of the whole stream. + bits: u64, +} + +/// Start codes in `data`: (offset of the 00 00 01, code). +fn start_codes(data: &[u8]) -> Vec<(usize, u8)> { + (0..data.len().saturating_sub(3)).filter(|&i| data[i..i + 3] == [0, 0, 1]).map(|i| (i, data[i + 3])).collect() +} + +/// Check a stream (packets in decoding order) against the VBV buffer of H.262 Annex C, using +/// nothing but its bytes: the sequence header's bit rate and buffer size, each picture's size +/// and `vbv_delay`. With a constant bitrate, data arrives at the bit rate from the first byte and +/// picture n is decoded `vbv_delay` after its start code arrives: those times must be one +/// picture interval apart, each picture complete by its time, and the buffer never over full. +/// With a variable bitrate the buffer fills at the bit rate until full and each picture must be +/// in it at its decoding time. +fn check_vbv(name: &str, packets: &[Packet], (fps_num, fps_den): (u32, u32)) -> VbvRun { + use crate::mpeg2::headers::{Extension, PictureHeader, SequenceHeader}; + let first = &packets[0].data; + let codes = start_codes(first); + let at = |code: u8| codes.iter().find(|c| c.1 == code).map(|c| c.0 + 4).unwrap(); + let seq = SequenceHeader::parse(&first[at(0xB3)..]).unwrap(); + let Extension::Sequence(ext) = Extension::parse(&first[at(0xB5)..]).unwrap() else { panic!("{name}: no sequence extension") }; + let rate = (u64::from(seq.bit_rate_value) | u64::from(ext.bit_rate_extension) << 18) * 400; + let size = (u64::from(seq.vbv_buffer_size_value) | u64::from(ext.vbv_buffer_size_extension) << 10) * 16384; + let (num, den) = (u128::from(fps_num), u128::from(fps_den)); + let (r, s) = (u128::from(rate), u128::from(size)); + let pics: Vec<(u64, u64, u16)> = packets + .iter() + .map(|p| { + let pic = start_codes(&p.data).into_iter().find(|c| c.1 == 0).unwrap().0 + 4; + (p.data.len() as u64 * 8, pic as u64 * 8, PictureHeader::parse(&p.data[pic..]).unwrap().vbv_delay) + }) + .collect(); + let cbr = pics[0].2 != 0xFFFF; + assert!(pics.iter().all(|p| (p.2 != 0xFFFF) == cbr), "{name}: some vbv_delay are 0xFFFF and some not"); + if cbr { + // in bit·ticks (bits × 90000): picture n decodes at T = (bits up to its start code) × + // 90000 + vbv_delay × rate; one interval is rate × 90000 × den / num of them + let mut before = 0u128; + let mut last: Option = None; + for (i, &(bits, head, delay)) in pics.iter().enumerate() { + let (bits, head) = (u128::from(bits), u128::from(head)); + let t = (before + head) * 90_000 + u128::from(delay) * r; + if let Some(l) = last { + // each vbv_delay is rounded down to a tick + let step = (t - l) * num; + let want = r * 90_000 * den; + assert!(step.abs_diff(want) <= 2 * r * num, "{name}: picture {i} decodes {step} after the one before, not {want}"); + } + let arrived = t + r; // within the tick the delay was rounded down by + assert!((before + bits) * 90_000 <= arrived, "{name}: picture {i} underflows ({bits} bits)"); + assert!(t <= (before + s) * 90_000, "{name}: the buffer overflows before picture {i}"); + before += bits; + last = Some(t); + } + } else { + // in 1/num bits: the buffer starts full and gains rate × den a picture, up to full + let mut full = s * num; + for (i, &(bits, _, _)) in pics.iter().enumerate() { + let b = u128::from(bits) * num; + assert!(b <= full, "{name}: picture {i} underflows ({bits} bits, {} in the buffer)", full / num); + full = (full - b + r * den).min(s * num); + } + } + VbvRun { cbr, rate, size, bits: pics.iter().map(|p| p.0).sum() } +} + fn psnr(a: &[u8], b: &[u8]) -> f64 { let sse: f64 = a.iter().zip(b).map(|(x, y)| (f64::from(*x) - f64::from(*y)).powi(2)).sum(); 10.0 * (255.0f64.powi(2) * a.len() as f64 / sse.max(1e-9)).log10() @@ -333,6 +439,165 @@ fn flushing_short_gops_keeps_the_average_bitrate() { } } +#[test] +fn constant_bitrate_streams_meet_the_vbv_buffer() { + let qcif = EncoderConfig { gop_size: 12, ..EncoderConfig::new(176, 144, 25, 1) }; + let cases: [(&str, EncoderConfig, Source); 9] = [ + ("scene", EncoderConfig { rate: RateControl::Cbr { kbps: 400 }, ..qcif.clone() }, scene), + ("scene ip", EncoderConfig { rate: RateControl::Cbr { kbps: 400 }, b_pictures: 0, ..qcif.clone() }, scene), + ("scene open b4", EncoderConfig { rate: RateControl::Cbr { kbps: 400 }, b_pictures: 4, closed_gop: false, ..qcif.clone() }, scene), + ("intra", EncoderConfig { rate: RateControl::Cbr { kbps: 1500 }, gop_size: 1, ..qcif.clone() }, scene), + ("small buffer", EncoderConfig { rate: RateControl::Cbr { kbps: 300 }, vbv_buffer_bits: Some(16384 * 3), ..qcif.clone() }, scene), + ("noise", EncoderConfig { rate: RateControl::Cbr { kbps: 600 }, ..qcif.clone() }, noise), + ("noise intra", EncoderConfig { rate: RateControl::Cbr { kbps: 600 }, gop_size: 1, ..qcif.clone() }, noise), + ("still", EncoderConfig { rate: RateControl::Cbr { kbps: 1000 }, ..qcif.clone() }, still), + ( + "422 29.97", + EncoderConfig { rate: RateControl::Cbr { kbps: 900 }, chroma: ChromaFormat::Yuv422, fps_num: 30000, fps_den: 1001, ..qcif.clone() }, + scene, + ), + ]; + for (name, cfg, source) in cases { + let (fps, frames) = ((cfg.fps_num, cfg.fps_den), 40); + // every case is a constant bitrate (0 would fail the rate check below) + let kbps = if let RateControl::Cbr { kbps } = cfg.rate { kbps } else { 0 }; + let packets = assert_decodes_to_recon_from(name, cfg, frames, source); + let run = check_vbv(name, &packets, fps); + assert!(run.cbr && run.rate == u64::from(kbps) * 1000, "{name}: {run:?}"); + // a constant bitrate: the stream is the bit rate's worth of its duration, give or take + // what the buffer held at the start and holds at the end + let want = run.rate as f64 * frames as f64 * f64::from(fps.1) / f64::from(fps.0); + assert!((run.bits as f64 - want).abs() <= run.size as f64, "{name}: {} bits for {want:.0}", run.bits); + } +} + +#[test] +fn constant_bitrate_pads_pictures_that_would_overflow_the_buffer() { + // a still picture costs next to nothing after the first: the rest must be stuffing + let cfg = EncoderConfig { rate: RateControl::Cbr { kbps: 2000 }, gop_size: 12, ..EncoderConfig::new(176, 144, 25, 1) }; + let (_, _, packets) = encode_from(cfg, 24, still); + let padded = packets.iter().filter(|p| p.data.ends_with(&[0; 64])).count(); + assert!(padded >= 12, "{padded} of {} pictures padded", packets.len()); + let run = check_vbv("still", &packets, (25, 1)); + assert!((run.bits as f64 / (2e6 * 24.0 / 25.0) - 1.0).abs() < 0.2, "{run:?}"); +} + +#[test] +fn variable_bitrate_streams_meet_the_vbv_buffer() { + let qcif = EncoderConfig { gop_size: 12, ..EncoderConfig::new(176, 144, 25, 1) }; + let small = Some(16384 * 4); + let cases: [(&str, EncoderConfig, Source); 7] = [ + ("vbr", EncoderConfig { rate: RateControl::Vbr { kbps: 300, max_kbps: 600 }, ..qcif.clone() }, scene), + ("vbr noise", EncoderConfig { rate: RateControl::Vbr { kbps: 300, max_kbps: 600 }, vbv_buffer_bits: small, ..qcif.clone() }, noise), + ("abr small buffer", EncoderConfig { rate: RateControl::Abr { kbps: 2000 }, vbv_buffer_bits: small, ..qcif.clone() }, noise), + ("quantiser 1", EncoderConfig { rate: RateControl::Quantiser(1), vbv_buffer_bits: small, ..qcif.clone() }, noise), + ("quantiser 1 intra", EncoderConfig { rate: RateControl::Quantiser(1), gop_size: 1, vbv_buffer_bits: small, ..qcif.clone() }, noise), + ("quantiser 1 b4", EncoderConfig { rate: RateControl::Quantiser(1), b_pictures: 4, vbv_buffer_bits: small, ..qcif.clone() }, noise), + ("default", qcif.clone(), scene), + ]; + for (name, cfg, source) in cases { + let packets = assert_decodes_to_recon_from(name, cfg, 30, source); + let run = check_vbv(name, &packets, (25, 1)); + assert!(!run.cbr, "{name}: {run:?}"); + } +} + +#[test] +fn pictures_too_large_for_the_buffer_are_coded_coarser() { + // noise at quantiser 1 is far beyond a 64 kbit buffer: every picture must give way, down to + // the coarsest quantiser or the fallbacks, and still decode to the reconstruction + let cfg = EncoderConfig { rate: RateControl::Quantiser(1), vbv_buffer_bits: Some(16384 * 4), ..EncoderConfig::new(176, 144, 25, 1) }; + let q_max = qscale(cfg.nonlinear_quant, 31) as f32; + let packets = assert_decodes_to_recon_from("noise", cfg, 24, noise); + assert!(packets.iter().all(|p| p.quantiser_scale > 8.0), "{:?}", packets.iter().map(|p| p.quantiser_scale).collect::>()); + // adaptive quantisation spreads the macroblocks of the coarsest pictures just under it + assert!(packets.iter().any(|p| p.quantiser_scale > 0.95 * q_max)); + check_vbv("noise", &packets, (25, 1)); +} + +#[test] +fn the_smallest_pictures_do_not_depend_on_the_source() { + let enc = Encoder::new(EncoderConfig { b_pictures: 2, ..EncoderConfig::new(176, 144, 25, 1) }).unwrap(); + let smallest = enc.smallest_picture(); + let frame = |t| { + let (y, cb, cr) = noise(176, 144, 88, 72, t); + enc.load(&YuvFrame { y: &y, cb: &cb, cr: &cr, y_stride: 176, c_stride: 88 }).unwrap() + }; + let (a, b) = (frame(1), frame(2)); + let h = Held { src: frame(3), pts: 0, n: 0 }; + let sizes: Vec = + [(PictureType::I, None, None), (PictureType::P, Some(&a), None), (PictureType::B, Some(&a), Some(&b)), (PictureType::B, None, Some(&b))] + .into_iter() + .map(|(t, fwd, bwd)| { + let c = enc.code_picture(&h, t, (fwd, bwd), [&[], &[]], 30.0, Fallback::Minimal); + // the minimal picture repeats its reference (grey for an I picture) + let want = bwd.or(fwd).map_or(128, |r| r.planes[0][5]); + assert_eq!(c.recon.planes[0][5], want, "{t:?}"); + c.bits() + }) + .collect(); + assert_eq!(sizes.iter().max().copied(), Some(smallest), "{sizes:?}"); + // the same sizes for a grey source and references + let mut grey = frame(0); + for p in &mut grey.planes { + p.fill(128); + } + let g = Held { src: grey.clone(), pts: 0, n: 0 }; + let i = enc.code_picture(&g, PictureType::I, (None, None), [&[], &[]], 30.0, Fallback::Minimal).bits(); + assert_eq!(i, sizes[0]); +} + +#[test] +fn the_sequence_header_carries_the_rate_and_buffer() { + let base = EncoderConfig::new(176, 144, 25, 1); + for (rate, vbv, want_rate, want_vbv) in [ + (RateControl::Cbr { kbps: 1000 }, None, 1_000_000, 475_136 / 16384 * 16384), + // rounded up to 400 bit/s + (RateControl::Cbr { kbps: 1001 }, None, 1_001_200, 475_136 / 16384 * 16384), + (RateControl::Vbr { kbps: 500, max_kbps: 1500 }, Some(200_000), 1_500_000, 196_608), + // the level's maximum + (RateControl::Abr { kbps: 500 }, None, 4_000_000, 475_136 / 16384 * 16384), + (RateControl::Quantiser(4), None, 4_000_000, 475_136 / 16384 * 16384), + ] { + let enc = Encoder::new(EncoderConfig { rate, vbv_buffer_bits: vbv, ..base.clone() }).unwrap(); + let info = probe(enc.sequence_header()).unwrap(); + assert_eq!(info.bit_rate, want_rate, "{rate:?}"); + assert_eq!(enc.vbv_value * 16384, want_vbv, "{rate:?}"); + } + // a larger buffer than the level holds moves to the next level + let enc = Encoder::new(EncoderConfig { vbv_buffer_bits: Some(1_000_000), ..base.clone() }).unwrap(); + assert_eq!(enc.profile_and_level(), 0x48); +} + +#[test] +fn bitrates_the_buffer_cannot_carry_are_refused() { + let base = EncoderConfig::new(352, 288, 25, 1); + let bad = [ + // 2000 bits a picture: less than the smallest CIF picture + EncoderConfig { rate: RateControl::Cbr { kbps: 50 }, ..base.clone() }, + EncoderConfig { rate: RateControl::Vbr { kbps: 40, max_kbps: 50 }, ..base.clone() }, + // a buffer of less than two pictures + EncoderConfig { rate: RateControl::Cbr { kbps: 4000 }, vbv_buffer_bits: Some(16384 * 9), ..base.clone() }, + // a buffer smaller than the smallest CIF picture + EncoderConfig { rate: RateControl::Quantiser(4), vbv_buffer_bits: Some(16384), ..base.clone() }, + EncoderConfig { vbv_buffer_bits: Some(16000), ..base.clone() }, + // beyond every level + EncoderConfig { vbv_buffer_bits: Some(10_000_000), ..base.clone() }, + EncoderConfig { rate: RateControl::Cbr { kbps: 0 }, ..base.clone() }, + EncoderConfig { rate: RateControl::Vbr { kbps: 900, max_kbps: 800 }, ..base.clone() }, + EncoderConfig { rate: RateControl::Cbr { kbps: 90_000 }, ..base.clone() }, + ]; + for cfg in bad { + let desc = format!("{cfg:?}"); + assert!(matches!(Encoder::new(cfg), Err(EncodeError::Config(_))), "{desc}"); + } + // the lowest bitrate that is accepted gives a stream that meets the buffer, noise or not + let lowest = (1..4000).find(|&kbps| Encoder::new(EncoderConfig { rate: RateControl::Cbr { kbps }, ..base.clone() }).is_ok()).unwrap(); + let cfg = EncoderConfig { rate: RateControl::Cbr { kbps: lowest }, ..base.clone() }; + let packets = assert_decodes_to_recon_from("lowest", cfg, 14, noise); + check_vbv("lowest", &packets, (25, 1)); +} + #[cfg(feature = "threads")] #[test] fn threaded_slices_give_the_same_stream() { diff --git a/crates/video/src/mpeg2/encode/vbv.rs b/crates/video/src/mpeg2/encode/vbv.rs new file mode 100644 index 000000000..ceae654d1 --- /dev/null +++ b/crates/video/src/mpeg2/encode/vbv.rs @@ -0,0 +1,118 @@ +//! The video buffering verifier (H.262 Annex C) as the encoder runs it: a model of the decoder's +//! input buffer, in decoding order, that says how large the next picture may be. +//! +//! Bits enter the buffer at `rate`. Each picture is removed whole at its decoding time, one +//! picture interval after the one before. With a constant bitrate (`vbv_delay` in every picture +//! header) the buffer must neither underflow (a picture not yet complete at its decoding time) +//! nor overflow. With a variable bitrate (`vbv_delay` 0xFFFF) the decoder fills the buffer until +//! it is full before the first picture, input stops while it is full, and only underflow is +//! possible. +//! +//! Fullness is counted in units of 1/`fps_num` bit, so one picture interval adds exactly +//! `rate · fps_den` of them and the model never rounds. + +/// `vbv_delay` for a variable bitrate stream. +pub(crate) const VBR_DELAY: u16 = 0xFFFF; + +/// The largest `vbv_delay` a constant bitrate stream can signal (0xFFFF means variable). +const MAX_DELAY: u64 = 0xFFFE; + +pub(crate) struct Vbv { + /// Buffer size in bits. + size: u64, + /// Bits per second: constant, or the peak at which the buffer fills. + rate: u64, + fps_num: u64, + fps_den: u64, + cbr: bool, + /// Fullness just before the next picture is removed, in 1/`fps_num` bits. + full: u64, +} + +impl Vbv { + /// A buffer of `size` bits filled at `rate` bit/s, `fps_num / fps_den` pictures a second. + /// With a constant bitrate the first picture is decoded once `start` bits are in. + pub fn new(size: u64, rate: u64, (fps_num, fps_den): (u32, u32), cbr: bool, start: u64) -> Self { + let (fps_num, fps_den) = (u64::from(fps_num.max(1)), u64::from(fps_den.max(1))); + let start = if cbr { start.min(size) } else { size }; + Self { size, rate, fps_num, fps_den, cbr, full: start.saturating_mul(fps_num) } + } + + /// Bits that enter the buffer in one picture interval, rounded down. + pub fn per_picture(&self) -> u64 { + self.rate.saturating_mul(self.fps_den) / self.fps_num + } + + /// The most bits the next picture may have: what is in the buffer at its decoding time. + pub fn max_bits(&self) -> u64 { + self.full / self.fps_num + } + + /// The fewest bits the next picture may have: with a constant bitrate, removing fewer would + /// let the buffer overflow before the picture after it. + pub fn min_bits(&self) -> u64 { + if !self.cbr { + return 0; + } + let next = self.full.saturating_add(self.rate.saturating_mul(self.fps_den)); + next.saturating_sub(self.size.saturating_mul(self.fps_num)).div_ceil(self.fps_num) + } + + /// `vbv_delay` of the next picture, whose data up to the end of its picture start code is + /// `head_bits` long: the 90 kHz ticks from the arrival of that start code to its decoding. + pub fn delay(&self, head_bits: u64) -> u16 { + if !self.cbr { + return VBR_DELAY; + } + let after = u128::from(self.full.saturating_sub(head_bits.saturating_mul(self.fps_num))); + let ticks = after * 90_000 / (u128::from(self.rate.max(1)) * u128::from(self.fps_num)); + ticks.min(u128::from(MAX_DELAY)) as u16 + } + + /// Remove the next picture (`bits` long, at most [`Vbv::max_bits`]). + pub fn remove(&mut self, bits: u64) { + let left = self.full.saturating_sub(bits.saturating_mul(self.fps_num)); + let next = left.saturating_add(self.rate.saturating_mul(self.fps_den)); + self.full = if self.cbr { next } else { next.min(self.size.saturating_mul(self.fps_num)) }; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_constant_bitrate_buffer_bounds_both_sides() { + // 4000 bits a picture into 10000 bits, starting at 6000 + let mut v = Vbv::new(10_000, 100_000, (25, 1), true, 6000); + assert_eq!((v.per_picture(), v.max_bits(), v.min_bits()), (4000, 6000, 0)); + // 6000 - 1000 + 4000 = 9000: then at least 3000 must go, or 9000 - b + 4000 > 10000 + v.remove(1000); + assert_eq!((v.max_bits(), v.min_bits()), (9000, 3000)); + v.remove(3000); + assert_eq!(v.max_bits(), 10_000); + // the delay of a picture whose start code ends 100 bits in: 9900 bits at 100 kbit/s + assert_eq!(v.delay(100), 8910); + } + + #[test] + fn a_variable_bitrate_buffer_stops_filling_when_full() { + let mut v = Vbv::new(10_000, 100_000, (25, 1), false, 0); + assert_eq!((v.max_bits(), v.min_bits(), v.delay(0)), (10_000, 0, VBR_DELAY)); + v.remove(1000); + assert_eq!(v.max_bits(), 10_000); + v.remove(9000); + assert_eq!(v.max_bits(), 5000); + } + + #[test] + fn fractional_frame_rates_do_not_drift() { + // 30000/1001 pictures a second at 1 Mbit/s: 33366.67 bits a picture + let mut v = Vbv::new(2_000_000, 1_000_000, (30000, 1001), true, 1_000_000); + for _ in 0..30_000 { + v.remove(33_366); + } + // 30000 pictures leave 0.666… bit each in the buffer: exactly 20000 bits + assert_eq!(v.max_bits(), 1_020_000); + } +} diff --git a/crates/video/tests/mpeg2/encode.rs b/crates/video/tests/mpeg2/encode.rs index 122fe2d71..1f13d9c86 100644 --- a/crates/video/tests/mpeg2/encode.rs +++ b/crates/video/tests/mpeg2/encode.rs @@ -228,3 +228,37 @@ fn average_bitrate_reaches_its_target() { assert!(err.abs() < 0.10, "{kbps} kbit/s target: got {got:.0} kbit/s ({:+.1}%)", 100.0 * err); } } + +#[test] +fn constant_bitrate_streams_hold_their_rate() { + let Some(ff) = craft_media_testkit::ffmpeg_or_skip("mpeg2 encoder constant bitrate") else { return }; + // a small buffer, so what it holds at the start and end moves the average little + let vbv = 16384 * 16; + for (kbps, b) in [(1500u32, 2), (3000, 0)] { + let frames = 100; + let c = Case { + name: if b == 0 { "cif_cbr_3000_ip" } else { "cif_cbr_1500" }, + cfg: EncoderConfig { rate: RateControl::Cbr { kbps }, vbv_buffer_bits: Some(vbv), b_pictures: b, ..EncoderConfig::new(352, 288, 25, 1) }, + frames, + probe: "mpeg2video,Main,10,yuv420p", + min_psnr: 28.0, + }; + let (_, got) = check(&ff, &c); + // the stream is the bit rate times its duration, give or take one buffer + let slack = f64::from(vbv) / (frames as f64 / 25.0) / 1000.0; + assert!((got - f64::from(kbps)).abs() <= slack, "{kbps} kbit/s: got {got:.0} kbit/s (±{slack:.0})"); + } +} + +#[test] +fn ffmpeg_decodes_a_dvd_like_variable_bitrate_stream() { + let Some(ff) = craft_media_testkit::ffmpeg_or_skip("mpeg2 encoder variable bitrate") else { return }; + // DVD-Video's limits: 9.8 Mbit/s into a 224 KiB buffer + let cfg = EncoderConfig { + rate: RateControl::Vbr { kbps: 5000, max_kbps: 9800 }, + vbv_buffer_bits: Some(1_835_008), + gop_size: 12, + ..EncoderConfig::new(720, 576, 25, 1) + }; + check(&ff, &Case { name: "sd_vbr_dvd", cfg, frames: 13, probe: "mpeg2video,Main,8,yuv420p", min_psnr: 30.0 }); +}