mirror of
https://github.com/storytold/craft-libs.git
synced 2026-11-14 05:22:10 +00:00
* media: add craft-media with bitstream and isobmff from FilmCraft Moves filmcraft-bitstream and filmcraft-isobmff (storytold/filmcraft@6c3948a) into one crate with a feature per module. The public API matches FilmCraft's so its crates can become re-exports. Tests, oracle tests and fuzzing come along; the ffmpeg discovery helpers replace filmcraft-testkit. Out-of-contract inputs no longer panic: bit widths above 32, a zero NAL length prefix width, and PCM frame sizes that overflow u32. Adds BitReader::read_ue_plus from DeckCraft's copy. * media: public ISO BMFF box reader (isobmff::boxes) Make the demuxer's box iterator and big-endian reader public as isobmff::boxes::{BoxIter, BoxRef, BoxHeader, Reader}, for walkers of boxes the demuxer does not model (CR3 uuid boxes, HEIF meta items, JPEG 2000). BoxIter::within keeps offsets relative to the outer slice, and allow_truncated yields a cut last box. The fuzz test walks every box of each mutated file. * media: matroska feature from FilmCraft's filmcraft-matroska Port storytold/filmcraft@6c3948a crates/matroska as craft_media::matroska behind the opt-in feature matroska, with the same public API so FilmCraft can make filmcraft-matroska a re-export. Its unit and ffprobe oracle tests come with it, on craft-media's test helpers. From DeckCraft's copy: cluster and block timestamps no longer overflow; they saturate at i64::MAX. MuxingApp and the default WritingApp name craft-media. New tests: block additions and mutation fuzzing. The oracle skips the duration of a track's last packet when ffmpeg 8 shortened it. * media: MOV PCM uses lpcm v2 above 16.16 rates and for unsigned 16-bit; signed 8-bit as twos Version 0 sound descriptions store the rate as 16.16 fixed point, so 88.2 kHz and higher were written as 0 Hz. 8-bit signed PCM was labelled raw (unsigned) and 16-bit unsigned as twos/sowt (signed). The reader also took raw under a sound handler for uncompressed video. * media: stop the top-level box scan at a box that runs past the end of the file A 64-bit box size near u64::MAX saturated the scan offset, and the next offset + 8 overflowed. * media: one sample budget for the whole ISO BMFF file The sample cap and the file-length bound on samples without table entries (constant-size stsz, trun without per-sample fields) applied per track or per trun, so repeating a small trak or trun multiplied the allocation. Both are now shared by all tracks and fragments. * media: set_sample_entry rejects entries that change the track's media type or PCM frame size The handler, stsz constant size and PCM timescale check are fixed at add_track; a replacement that changed them produced a corrupt file. A bad track index returns NoSuchTrack like the other writer methods. * media: saturate the sample end in MkvFile::duration_ns Cluster timestamps saturate at i64::MAX, so pts + duration could overflow for files without Info/Duration. * media: checked add for the Matroska cue cluster position CueClusterPosition comes from the file; an add past u64::MAX panicked in debug builds and wrapped in release. Such a cue now falls back to the index. * media: cap the Matroska TimestampScale at 2^32 and saturate tick conversions A scale near 2^64 overflowed u128/i128 in the per-lace split and in Track::ticks_to_ns. A BlockDuration near u64::MAX also overflowed the tick difference at the default scale. Larger scales are ignored like a zero scale, and tick and nanosecond values outside i64 saturate. * media: Matroska fixture generation no longer truncates the shared subtitle inputs Parallel tests generating subs.mkv or webvtt.webm could read in.srt or in.vtt while another thread rewrote it, and cache a fixture with missing packets. * media: rustfmt parse_moof signature * ci: add craft-media workflow covering feature sets, WASM and ffmpeg oracles The lint job runs fmt, clippy for each feature set (none, bitstream, isobmff, matroska, all) and a wasm32 check. The test job runs the all-features tests with Ubuntu's ffmpeg and CRAFT_MEDIA_REQUIRE_ORACLES=1, installing ffmpeg while the tests compile. * media: Matroska lacing keeps timestamp gaps A laced frame's timestamp is implied by the block's and the track's DefaultDuration, so lacing collapsed gaps in the input. A frame now joins a lace only when that implied tick equals its own, and tracks without a DefaultDuration do not lace. The laced remux oracle compares timestamps. * media: required oracles fail on fixture generator and ffprobe failures Every path that gave up on an oracle check (missing tool, failed fixture generator, failed ffprobe run) goes through one helper that prints SKIPPED, or fails the test with CRAFT_MEDIA_REQUIRE_ORACLES=1. Fixture generators of both containers share run_generator, which takes the required flag explicitly so tests can exercise it without changing the process environment. * media: use US spelling * media: a failed ffprobe probe skips the optional oracle check * media: the Matroska oracle compares the last subtitle cue's duration
368 lines
22 KiB
Markdown
368 lines
22 KiB
Markdown
# craft-media
|
||
|
||
Media containers and bit-level I/O for the Craft apps: an MSB-first bit reader and writer, and
|
||
clean-room ISO BMFF (MP4), QuickTime (MOV) and Matroska (MKV, WebM) demuxers and muxers written
|
||
from the public specifications. No `unsafe`, no panics on malformed input, and it builds for
|
||
`wasm32-unknown-unknown`. The only dependency is `thiserror` (for `isobmff`).
|
||
|
||
## Features
|
||
|
||
| Feature | Module | Enables |
|
||
|---|---|---|
|
||
| `bitstream` | `craft_media::bitstream` | none |
|
||
| `isobmff` | `craft_media::isobmff` | `bitstream`, `thiserror` |
|
||
| `matroska` | `craft_media::matroska` | none |
|
||
|
||
`default = ["isobmff"]`, so `bitstream` and `isobmff` build. `matroska` is opt-in: it is about as
|
||
much code as `isobmff`, and most users of MP4/MOV (LightCraft, FigCraft, SoundCraft, EffectCraft)
|
||
do not read Matroska. The features are independent; `matroska` does not need the other two.
|
||
|
||
```toml
|
||
craft-media = { git = "https://github.com/storytold/craft-libs", rev = "<commit>", features = ["matroska"] }
|
||
```
|
||
|
||
A crate that needs only bit I/O (a codec) uses:
|
||
|
||
```toml
|
||
craft-media = { git = "https://github.com/storytold/craft-libs", rev = "<commit>", default-features = false, features = ["bitstream"] }
|
||
```
|
||
|
||
## Boundaries
|
||
|
||
- In scope: bit-level reading and writing, and container demuxing and muxing (boxes, sample
|
||
tables, sample entries and codec configuration records).
|
||
- Out of scope for now: codecs (H.264, HEVC, ProRes, AAC and others). They stay in the apps.
|
||
Ogg and other containers can join later as their own features.
|
||
|
||
## Origin and API
|
||
|
||
The code comes from FilmCraft's `filmcraft-bitstream`, `filmcraft-isobmff` and
|
||
`filmcraft-matroska` crates at
|
||
[storytold/filmcraft@6c3948a](https://github.com/storytold/filmcraft/commit/6c3948a36c80e4996d924766ba7157eec4fca85e).
|
||
Every public item keeps its name, signature and behavior for valid input, so FilmCraft can turn
|
||
all three crates into re-exports (`pub use craft_media::isobmff::*;`). The differences:
|
||
|
||
- `bitstream::BitReader::read_ue_plus` (from DeckCraft's copy) is new.
|
||
- `isobmff::boxes` is new: FilmCraft's private box iterator and byte reader, made public (see
|
||
[Box reader](#box-reader)).
|
||
- `BitReader::peek`, `BitReader::read_bits` and `BitWriter::write_bits` accept widths above 32
|
||
without a debug assertion: `peek` and `read_bits` return the low 32 bits of the field (use
|
||
`read_bits_u64` for wide fields), and `write_bits` writes leading zero bits, at most 64 bits in
|
||
all.
|
||
- `bitstream::length_prefixed_nals` returns `BitError::Eof` for a zero `length_size` on non-empty
|
||
data and cannot overflow on a hostile length prefix.
|
||
- `isobmff::PcmConfig::bytes_per_frame` saturates at `u32::MAX` for a hostile channel count.
|
||
- `matroska`: a cluster timestamp above `i64::MAX` saturates at `i64::MAX` (FilmCraft wraps it to
|
||
a negative value), and adding a block's relative timestamp saturates instead of overflowing (a
|
||
debug-build panic in FilmCraft).
|
||
- `matroska::MkvWriter` writes `MuxingApp` = `craft-media`, and `MuxOptions::default()` has
|
||
`writing_app` = `craft-media` (FilmCraft: `filmcraft-matroska` and `filmcraft`). Apps set
|
||
`writing_app` to their own name.
|
||
- `matroska::ByteSource` stays a separate trait from `isobmff::ByteSource`, with the same shape.
|
||
FilmCraft implements both for one type, which a single shared trait would reject as conflicting
|
||
impls.
|
||
|
||
## `bitstream`
|
||
|
||
- `BitReader`: MSB-first reader with Exp-Golomb (`ue(v)`, `se(v)`, `te(v)`), as used by
|
||
H.264/HEVC/AV1 headers.
|
||
- `BitWriter`: the matching writer, for encoders and header rewriting.
|
||
- `unescape_rbsp` / `escape_rbsp`: remove or insert H.264/HEVC emulation-prevention bytes.
|
||
- `annexb_nals` / `length_prefixed_nals`: split Annex-B byte streams and AVCC/HVCC samples into
|
||
NAL units.
|
||
- `crc32`: CRC-32 (IEEE), as used by Matroska and PNG.
|
||
|
||
## `isobmff`
|
||
|
||
### Demuxing
|
||
|
||
```rust
|
||
use craft_media::isobmff::{open, TrackKind};
|
||
|
||
let file = std::fs::File::open("clip.mov")?;
|
||
let mp4 = open(&file)?; // any ByteSource: &[u8], Vec<u8>, Arc<[u8]>, File, ...
|
||
let v = mp4.track_of_kind(TrackKind::Video).unwrap();
|
||
let track = &mp4.tracks[v];
|
||
let i = track.sample_at_presentation_time(1000).unwrap(); // track timescale, edit-adjusted
|
||
let key = track.sync_sample_before(i); // decode from here
|
||
let bytes = mp4.read_sample(&file, v, key)?; // raw sample (AVCC/HVCC length-prefixed NALs, ...)
|
||
```
|
||
|
||
- `ByteSource` (`len`, `read_at(offset, buf)`) is the input abstraction. Implementations exist for `[u8]`,
|
||
`Vec<u8>`, `&T`, `Arc<T>`, `Box<T>`, and `std::fs::File` on non-wasm targets.
|
||
- `open` reads only top-level box headers, `ftyp`, `moov` and every `moof`. It never reads `mdat`.
|
||
The one exception is the first sample of a `tmcd` track, which it reads (4 bytes) to get the timecode
|
||
start frame.
|
||
- `Mp4File`: brands, `is_quicktime`, movie `timescale`/`duration`, `tracks`, `metadata`, `fragmented`,
|
||
`fragment_count`.
|
||
- `Track`: `id`, `kind` (Video/Audio/Timecode/Subtitle/Other), `handler`, `timescale`, `duration`,
|
||
`language`, `tkhd` fields (size, matrix, volume, enabled), `entries: Vec<SampleEntry>`, `samples`,
|
||
`edits`, `edit_offset`, `references` (`tref`), `sdtp`, `sample_groups` (`sbgp`+`sgpd`, raw), `pcm_chunked`.
|
||
Helpers: `codec()`, `video()`, `audio()`, `presentation_pts(i)`, `sample_at_pts(pts)`,
|
||
`sample_at_presentation_time(t)`, `sync_sample_before(i)`, `sync_samples()`.
|
||
- `Sample`: `offset`, `size`, `dts`, `pts` (dts + ctts), `duration`, `is_sync` (all samples are sync when
|
||
`stss` is absent), `description_index`. All times are raw media time in the track timescale.
|
||
- **Edit lists:** `edits` holds the `elst` entries unchanged. `edit_offset` is
|
||
`leading empty edits (rescaled to media timescale) − first non-empty edit's media_time`, so
|
||
`presentation time = pts + edit_offset`. This matches ffprobe's timestamps for the common cases
|
||
(B-frame delay, AAC priming, empty-edit offsets). Multi-segment edits, dwells and rate changes are exposed
|
||
raw for the caller to interpret.
|
||
- **Uncompressed PCM** (`lpcm`, `sowt`, `twos`, `in24`, `in32`, `fl32`, `fl64`, `raw `, `ipcm`, `fpcm`) is
|
||
merged to one `Sample` per chunk (`pcm_chunked = true`), with `duration` in frames. This avoids one
|
||
table entry per audio frame.
|
||
|
||
#### Parsed boxes
|
||
`ftyp`, `moov/mvhd`, `trak/tkhd`, `edts/elst` (v0/v1), `tref`, `mdia/mdhd` (v0/v1, ISO and Mac language
|
||
codes), `hdlr` (ISO and Pascal names), `minf/stbl`: `stsd`, `stts`, `ctts` (v0/v1, read as signed), `stss`,
|
||
`stsc`, `stsz`/`stz2` (4/8/16-bit), `stco`/`co64`, `sdtp`, `sbgp`/`sgpd`. Also `mvex/trex`, `moof/traf/tfhd/tfdt/trun`
|
||
(every `tfhd` base-offset mode and multiple `moof`s), and `udta` metadata (QuickTime `©xxx` atoms,
|
||
`meta/ilst`, and `meta/keys` (mdta)). `uuid`, `free`, `skip`, `wide`, `sidx` and `mfra` are skipped.
|
||
Boxes with 64-bit sizes and with size 0 (runs to the end of the file) are supported.
|
||
|
||
#### Sample entries → `CodecConfig`
|
||
| Format | Config |
|
||
|---|---|
|
||
| `avc1`/`avc3` + `avcC` | `Avc(AvcConfig)`: profile, level, length size, SPS/PPS, extension bytes |
|
||
| `hvc1`/`hev1` + `hvcC` | `Hevc(HevcConfig)`: all header fields plus NAL arrays (`vps()/sps()/pps()`) |
|
||
| `av01` + `av1C` | `Av1(Av1Config)` |
|
||
| `vp09` + `vpcC` | `Vp9(VpcConfig)` |
|
||
| `apch apcn apcs apco ap4h ap4x` | `ProRes { fourcc }` |
|
||
| `jpeg mjpa mjpb` | `Jpeg { fourcc }` |
|
||
| `AVdn AVdh` | `Dnx { fourcc }` |
|
||
| `mp4a` + `esds` (in `wave` too) | `Aac(AacConfig)`: ASC bytes + decoded object type/rate/channels; `Mp3` for OTI 0x69/0x6B |
|
||
| `.mp3`/`.mp2` | `Mp3`: MPEG-1/2 audio of any layer (each frame header names it) |
|
||
| PCM fourccs | `Pcm(PcmConfig)`: bits, float, endianness (`enda`, lpcm flags, `pcmC`), channels, rate |
|
||
| `alac` | `Alac { cookie }` |
|
||
| `Opus` + `dOps` | `Opus(OpusConfig)` |
|
||
| `fLaC` + `dfLa` | `Flac(FlacConfig)` (STREAMINFO decoded) |
|
||
| `ac-3`/`ec-3` | `Ac3 { dac3 }` / `Eac3 { dec3 }` |
|
||
| `tmcd` | `Timecode(TimecodeConfig)`: flags (drop frame), timescale, frame duration, fps, start frame, reel name; `format_frame()` |
|
||
| anything else | `Unknown { fourcc, raw }` |
|
||
|
||
Video entries also parse `colr` (`nclx`/`nclc`/ICC), `pasp`, `clap`, `fiel`, `gama` and `btrt` into
|
||
`VideoParams`. Audio entries parse QuickTime sound description v0/v1/v2 into `AudioParams`.
|
||
|
||
#### Robustness
|
||
The parser does not panic on malformed input; every error comes back as an `Error`. Table counts are
|
||
checked against the bytes in their box before anything is allocated. A constant-size `stsz` count is
|
||
clamped to what fits in the file. `trun` counts are checked, and each track is capped at 2²⁶ samples.
|
||
`moov`/`moof` are capped at 1 GiB. Compressed movie headers (`cmov`) are rejected with
|
||
`Error::Unsupported`.
|
||
|
||
### Muxing
|
||
|
||
```rust
|
||
use craft_media::isobmff::*;
|
||
let mut w = Mp4Writer::new(std::io::Cursor::new(Vec::new()), WriterOptions::new(Brand::Mp4))?;
|
||
let v = w.add_track(TrackConfig::new(SampleEntry::avc(avcc, 1920, 1080), 90_000))?;
|
||
let mut acfg = TrackConfig::new(SampleEntry::aac(asc, 2, 48_000), 48_000);
|
||
acfg.media_start = Some(1024); // AAC encoder delay → edit list
|
||
let a = w.add_track(acfg)?;
|
||
w.write_sample(v, WriteSample { data: &au, duration: 3000, composition_offset: 6000, is_sync: true })?;
|
||
// ... interleave samples in decode order ...
|
||
let bytes = w.finish_faststart()?.into_inner(); // or finish() for moov-at-end
|
||
```
|
||
|
||
- `Mp4Writer<W: Write + Seek>` streams samples into a single `mdat`. `finish()` appends `moov`.
|
||
`finish_faststart()` (needs `W: Read + Write + Seek`) moves the media data forward in place and writes
|
||
`moov` before it. If `mdat` goes past 4 GiB the header becomes a large-size header, and chunk offsets
|
||
switch to `co64` when needed.
|
||
- Brands: `Brand::Mp4` writes `isom` with `isom iso2 avc1 mp41 mp42`. `Brand::Mov` writes `qt `, uses
|
||
QuickTime `hdlr`/Pascal names, and sets `wide` as the placeholder box.
|
||
- Tables written: `stts`, `ctts` (v1 when an offset is negative), `stss` (omitted when every sample is
|
||
sync), `stsc`, `stsz` (constant or table), `stco`/`co64`. Consecutive samples of one track share a chunk.
|
||
- Sample entries come from `SampleEntry` (`avc`, `hevc`, `prores`, `jpeg`, `aac`, `pcm`, or any demuxed
|
||
entry). Remuxing copies codec config, `colr`, `pasp`, `clap`, `fiel`, `gama` and `btrt`.
|
||
PCM is written as `sowt`/`twos` (16-bit) or `raw ` (8-bit) or sound description v2 `lpcm` in MOV, and as
|
||
`ipcm`/`fpcm` + `pcmC` in MP4. A PCM `write_sample` call can hold any whole number of frames, and the
|
||
track timescale must equal the sample rate.
|
||
- Edit lists: set `TrackConfig::edits` explicitly, or use `media_start` to get a single edit.
|
||
- Timecode: `add_timecode_track(video, TimecodeConfig, start_frame)` adds a `tmcd` track (with `gmhd`)
|
||
and a `tref/tmcd` link from the video track.
|
||
- `WriterOptions::metadata` writes `©xxx` text atoms to `udta`.
|
||
- `FragmentedWriter<W: Write>` writes fMP4. The init segment is `ftyp` + `moov` with `mvex/trex`. Each
|
||
`flush_fragment()` then writes `moof` (`mfhd`, `traf` with default-base-is-moof `tfhd`, `tfdt` v1,
|
||
`trun` with per-sample duration, size, flags and composition offset) followed by `mdat`.
|
||
|
||
### Box reader
|
||
|
||
`isobmff::boxes` gives direct access to the box format, for files and boxes the demuxer does not
|
||
model: CR3 `uuid` boxes, HEIF `meta` items, JPEG 2000 (JP2, JPX, JPM, MJ2) boxes.
|
||
|
||
```rust
|
||
use craft_media::isobmff::boxes::{BoxIter, Reader};
|
||
|
||
for b in BoxIter::new(&bytes) {
|
||
let b = b?;
|
||
if b.kind == b"uuid" && b.user_type == Some(CANON_UUID) {
|
||
for c in BoxIter::within(&bytes, b.payload_range()) { /* CMT1..CMT4, THMB */ }
|
||
}
|
||
}
|
||
```
|
||
|
||
- `BoxIter::new(data)` yields the sibling boxes in `data`; `BoxIter::within(data, range)` yields
|
||
those in `data[range]` with offsets relative to `data`, so a walker keeps file offsets as it
|
||
descends. Each `BoxRef` has `kind`, `user_type` (the 16-byte extended type of a `uuid` box),
|
||
`offset`, `header_len`, `payload`, `payload_range()`, `range()` and `end()`.
|
||
- 64-bit `largesize` and size 0 (runs to the end of the range) are handled. A malformed header or a
|
||
box larger than the range yields one `Err` and ends the iteration; `allow_truncated()` instead
|
||
yields that last box cut at the end of the range, with `truncated` set. A 32-bit zero terminator
|
||
(QuickTime) ends the iteration.
|
||
- `BoxHeader::parse(buf)` reads one header (up to `BoxHeader::MAX_LEN` = 32 bytes), for scanning a
|
||
file through `ByteSource` without loading it.
|
||
- `Reader` is the bounds-checked big-endian cursor the demuxer uses: `u8` to `u64`, `i16` to
|
||
`i64`, `f64`, `u24`, `fourcc`, `full_header` (version and flags), `bytes`, `skip`, `rest` and
|
||
`count` (an entry count checked against the bytes left, before allocating).
|
||
- The iterator does not recurse. The caller bounds nesting depth and box count.
|
||
|
||
### Limitations
|
||
- `presentation time = pts + edit_offset` only applies the leading empty edits and the first media
|
||
edit. Multi-segment edit lists, dwell edits and non-unity rates are exposed raw. ffmpeg handles an edit
|
||
that starts mid-frame differently: it snaps to the frame.
|
||
- Other data references (external `alis`/`url ` files), encrypted entries (`encv`/`enca`, `sinf`,
|
||
`senc`), `cmov`, and QuickTime reference movies are not supported.
|
||
- `sidx`/`mfra` are ignored, because the demuxer scans every top-level box. `stsd` entries with a
|
||
QuickTime color table are not decoded.
|
||
- Subtitle and text tracks (`tx3g`, `c608`, `wvtt`) and `mp4v` show up as `Unknown`.
|
||
- Muxer: one `mdat`, no chunk-size or duration limits (the caller controls interleaving), no `sdtp`,
|
||
`sgpd`/`sbgp` or `cslg` output, ISO-style `meta/ilst` metadata is not written, and the fragmented
|
||
writer has no PCM, `sidx` or `mfra`.
|
||
|
||
## `matroska`
|
||
|
||
A clean-room Matroska (MKV) and WebM demuxer, plus a minimal muxer, written from RFC 8794 (EBML),
|
||
RFC 9559 (Matroska), the Matroska codec mappings and the WebM container guidelines. No GPL or LGPL
|
||
code (FFmpeg, libmatroska, mkvtoolnix) was consulted; FFmpeg is a test oracle only.
|
||
|
||
### Demuxing
|
||
|
||
```rust
|
||
use craft_media::matroska::{Demuxer, TrackKind, open};
|
||
|
||
// Sample tables in the same shape as isobmff (any ByteSource: &[u8], Vec<u8>, Arc<[u8]>, File, ...)
|
||
let file = std::fs::File::open("clip.mkv")?;
|
||
let mkv = open(&file)?;
|
||
let v = mkv.track_of_kind(TrackKind::Video).unwrap();
|
||
let k = mkv.keyframe_before(v, 2_000_000_000).unwrap(); // sample index
|
||
let frame = mkv.read_sample(&file, v, k)?;
|
||
|
||
// Streaming packets and seeking (from a ByteSource, any Read + Seek, or a byte slice)
|
||
let mut d = Demuxer::from_reader(std::io::BufReader::new(std::fs::File::open("clip.webm")?))?;
|
||
let sp = d.seek(v, 2_000_000_000)?; // lands on the keyframe with pts <= 2 s
|
||
while let Some(p) = d.next_packet()? { // file order, all tracks
|
||
// p.track, p.pts (ticks), p.pts_ns, p.duration, p.keyframe, p.data
|
||
}
|
||
```
|
||
|
||
- **Open** (`open`, `open_with(src, &OpenOptions { index, verify_crc })`): EBML header (DocType
|
||
`matroska` or `webm`), the first Segment, and its level-1 elements: SeekHead (followed for
|
||
elements after the clusters), Info, Tracks, Cues, Chapters, Tags, Attachments (listed with offset
|
||
and size; payloads are not read). With `index: true` (the default) every Cluster is scanned,
|
||
element and block headers only, to build per-track `Sample` tables (`offset`, `size`, `pts`,
|
||
`duration`, `keyframe`, cluster, block offset, lace index, `BlockAdditional` location) and
|
||
keyframe lists. With `index: false`, `open` stops at the first Cluster.
|
||
- **Timestamps:** Matroska ticks are `TimestampScale` ns (usually 1 ms); `Track::timebase` is that
|
||
as a reduced fraction of a second. `pts` is in ticks, `pts_ns` in exact nanoseconds. `CodecDelay`
|
||
is subtracted (RFC 9559), as FFmpeg does, so Opus and Vorbis priming gives negative pts. Laced
|
||
frames get `block ts + i × (BlockDuration or DefaultDuration) / n`; when neither is known, later
|
||
laces repeat the block timestamp and have `duration == 0` (`Packet::lace` tells which).
|
||
- **Keyframes:** the SimpleBlock keyframe flag; a BlockGroup is a keyframe when it has no
|
||
ReferenceBlock.
|
||
- **Lacing:** Xiph, fixed-size and EBML. **Unknown sizes:** a Segment (live or streamed output) and
|
||
Clusters end at the next top-level element. **CRC-32:** optional verification of level-1
|
||
elements and clusters; mismatches go to `MkvFile::crc_errors` and are not fatal.
|
||
- **Damage:** an undecodable or oversized element starts a byte scan for the next plausible level-1
|
||
element (a Cluster whose first child is Timestamp, or a sized metadata element); a truncated final
|
||
block is dropped. Problems are listed in `MkvFile::warnings`.
|
||
- **Seeking** (`Demuxer::seek(track, time_ns)`): uses the sample index when present, otherwise Cues
|
||
(refined by scanning the cued cluster for the exact keyframe), otherwise builds the index by
|
||
scanning all clusters. `Demuxer::keyframe_index(track)` returns the keyframe list.
|
||
- **Block additions:** `MkvFile::read_sample_addition` returns the `BlockAdditional` with
|
||
`BlockAddID` 1 of an unlaced block (in WebM, the VP8/VP9 alpha layer).
|
||
- **Tracks:** number, UID, type, flags, `DefaultDuration`, name, language (and BCP 47), CodecID,
|
||
CodecPrivate, CodecName, `CodecDelay`, `SeekPreRoll`; `VideoInfo` (pixel and display size, crop,
|
||
interlacing, stereo, alpha mode, `pixel_aspect()`, `display_rotation()`), `Color` (matrix,
|
||
range, transfer, primaries, subsampling and siting, MaxCLL/MaxFALL, SMPTE 2086 mastering
|
||
metadata); `AudioInfo` (rate, output rate, channels, bit depth); `ContentEncoding`s.
|
||
- **Content encodings:** header stripping is undone in every packet and in `read_sample`. zlib,
|
||
bzlib and lzo compression and encryption are reported (`Track::frames_readable() == false`);
|
||
packets are then returned as stored and `read_sample` fails with `Unsupported`.
|
||
- `matroska::ebml` (public) has the EBML primitives: variable-length integers, element headers,
|
||
typed values, CRC-32 and lacing. `matroska::element_ids` lists the element IDs.
|
||
|
||
#### Codec mapping (`Codec`)
|
||
|
||
| CodecID | `Codec` | Payload |
|
||
|---|---|---|
|
||
| `V_MPEG4/ISO/AVC` | `Avc` | avcC (CodecPrivate) |
|
||
| `V_MPEGH/ISO/HEVC` | `Hevc` | hvcC |
|
||
| `V_VP8`, `V_VP9` | `Vp8`, `Vp9` | VP9 CodecPrivate features |
|
||
| `V_AV1` | `Av1` | av1C |
|
||
| `V_APV` | `Apv` | apvC |
|
||
| `V_PRORES` | `ProRes` | FourCC; frames get the 8-byte `size` + `icpf` header re-added |
|
||
| `V_MJPEG`, `V_MS/VFW/FOURCC` (MJPG) | `Mjpeg` | |
|
||
| `V_MS/VFW/FOURCC` (other) | `VfwFourcc` | BITMAPINFOHEADER |
|
||
| `A_AAC`, legacy `A_AAC/MPEG{2,4}/...` | `Aac` | AudioSpecificConfig (synthesized for legacy IDs, SBR included) |
|
||
| `A_OPUS` | `Opus` | OpusHead |
|
||
| `A_VORBIS` | `Vorbis` | three headers split from the Xiph-laced CodecPrivate |
|
||
| `A_FLAC` | `Flac` | `fLaC` + metadata blocks |
|
||
| `A_PCM/INT/LIT`, `A_PCM/INT/BIG`, `A_PCM/FLOAT/IEEE` | `Pcm` | bit depth from Audio |
|
||
| `A_AC3`, `A_EAC3`, `A_MPEG/L3`, `A_MPEG/L2`, `A_MPEG/L1` | `Ac3`, `Eac3`, `Mp3`, `Mp2`, `Mp1` | |
|
||
| `S_TEXT/UTF8` | `SubRip` | |
|
||
| `S_TEXT/WEBVTT`, `D_WEBVTT/*` (WebM) | `WebVtt` | WebM blocks keep `id\nsettings\npayload` |
|
||
| `S_TEXT/ASS`, `S_TEXT/SSA` | `Ass` | script header |
|
||
| anything else | `Other(id)` | `Track::codec_private` |
|
||
|
||
`Codec::name()` matches FFmpeg's `codec_name` for these.
|
||
|
||
### Muxing
|
||
|
||
`MkvWriter::new(w: Write + Seek, Vec<TrackSpec>, MuxOptions)`, then `write_frame(track, pts_ns,
|
||
keyframe, data, duration_ns)` per frame, then `finish()`. It writes the EBML header, Segment,
|
||
SeekHead, Info (with Duration), Tracks, Clusters of SimpleBlocks (a new cluster at a video keyframe
|
||
after `cluster_duration_ns`), BlockGroups for frames whose duration differs from the track's
|
||
`default_duration_ns`, optional audio lacing (Xiph, EBML, fixed-size), and Cues (video keyframes,
|
||
or the first frame of each track per cluster). `MuxOptions::doc_type` = `"webm"` writes WebM.
|
||
CodecPrivate is fixed when the writer is created, so a codec whose configuration arrives with the
|
||
first frame (AV1 from `rav1e`) creates the writer after that frame.
|
||
|
||
### Limitations
|
||
|
||
- Only the first Segment is read (no linked or chained segments, no ordered-chapter playback).
|
||
- zlib, bzlib and lzo compressed and encrypted tracks are detected, but their frames are not
|
||
decoded. `EncryptedBlock` is ignored.
|
||
- `TrackTimestampScale` (deprecated) is ignored.
|
||
- Lace timestamps without any duration information repeat the block timestamp (FFmpeg derives them
|
||
from codec parsers).
|
||
- Muxer: no CRC-32, Chapters, Tags, Attachments, content encodings or block additions.
|
||
|
||
## Tests
|
||
|
||
- Unit tests cover the bit reader and writer, the box reader (64-bit, size-0, `uuid`, truncated and
|
||
hostile sizes), hand-built `stbl` edge cases, codec configuration round trips and allocation
|
||
guards. For Matroska: EBML integers, typed values, CRC-32, every lacing mode, codec mapping, a
|
||
hand-built file (unknown-size Segment and Clusters, header stripping, BlockGroups, block
|
||
additions, color and mastering metadata, Chapters, Tags, Attachments, CRC-32, Cue-less seek,
|
||
hostile cluster timestamps) and mux-demux round trips.
|
||
- `tests/roundtrip.rs`: property tests that write random multi-track files (progressive and
|
||
fragmented) and read them back.
|
||
- `tests/oracle_demux.rs` and `tests/oracle_mux.rs` compare the demuxer and muxer with ffprobe and
|
||
ffmpeg on generated fixtures in `target/fixtures/isobmff/` (`$CRAFT_MEDIA_FIXTURES_DIR`
|
||
overrides it). Fixtures are never committed.
|
||
- `tests/oracle_matroska.rs` compares every packet (stream, size, keyframe flag, pts, duration),
|
||
codec names, time bases, dimensions and audio parameters with ffprobe on fixtures in
|
||
`target/fixtures/matroska/`: H.264+AAC, VP9+Opus WebM, FLAC, Vorbis, SRT+ASS, WebVTT WebM, Cues
|
||
at the front, a streamed file without Cues, HEVC with HDR color, ProRes+PCM, MJPEG+AC-3, AV1.
|
||
Seeks must land on the latest keyframe at or before the target. The muxer's laced output is read
|
||
back by ffprobe. A damaged file must resync.
|
||
- `tests/fuzz.rs` and `tests/fuzz_matroska.rs`: deterministic mutation fuzzing; nothing may panic.
|
||
`fuzz.rs` also walks every box of a mutated file as a container and checks that each payload
|
||
range stays inside its parent.
|
||
|
||
ffmpeg and ffprobe are test oracles only, never linked. They are looked up in
|
||
`$CRAFT_MEDIA_FFMPEG` / `$CRAFT_MEDIA_FFPROBE`, then `PATH`, then common install directories. When
|
||
a tool is missing, the oracle tests print `SKIPPED` and pass; `CRAFT_MEDIA_REQUIRE_ORACLES=1` turns
|
||
a skip into a failure. CI (`.github/workflows/media.yml`) sets it and installs Ubuntu's ffmpeg.
|
||
Generate every fixture ahead of time with
|
||
`cargo test -p craft-media --all-features -- --ignored generate_fixtures`.
|