From ca99f91a18f40341ea4fa1119bdfacc378304ff5 Mon Sep 17 00:00:00 2001 From: Aleksander Helgaker Date: Wed, 11 Nov 2026 13:21:26 +0100 Subject: [PATCH] craft-audio: AAC encoder (6/10) (#444) * craft-audio: AAC decoder FilmCraft's `filmcraft-aac` decoder as `craft_audio::aac` behind `aac-decode`, kept over symphonia's: the same quality (about 135 dB SNR against ffmpeg), and it decodes 5.1, which symphonia refuses. HE-AAC decodes as its AAC-LC core, upsampled 2x to the stream's rate. - `TrackDecoder` decodes AAC, including ADTS-framed packets without extradata (MPEG-TS, raw `.aac`). - The native read path takes AAC in MP4 and Matroska and raw ADTS streams (frame sync over the file, with resync after damage). - AAC errors convert into `AudioError`. The AAC module's shared parts (Huffman, MDCT, TNS, ICS) arrive whole as ported; their encoder half is unused until the encoder PR. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 1 (#443) Raw ADTS and TrackDecoder: - Channel configuration 0 (a PCE in the stream, as ffmpeg's -aac_pce and AudioToolbox's 7.x write) decodes; it was refused. - Frames with several raw data blocks are refused; only their first block was decoded, silently losing the rest. - A frame that changes the sample rate, profile or channel configuration is concealed; it played at the wrong speed. - A source that fails during the frame scan is AudioError::Io; it looked like the end of the file. - A stream after an ID3 tag longer than the file head is found. - conceal counts all the ADTS frames of a packet, and a packet of more than 1024 frames is refused before it is decoded into memory. AAC decoder: - Indexed channel configurations place elements by arrival order, not tag, as ffmpeg does; tagged differently, channels were swapped. - A PCE in an access unit that fails no longer changes the layout, which left every later packet failing. - Non-finite samples are zeroed before the output and the overlap. MP4: probe and decode agree: the samples' durations end the track when no single edit does, and an edit cannot outlast the media. The routing oracle fails on a fixture ffmpeg cannot write when oracles are required. The README's Limits names HE-AAC v2 as mono, multi-block ADTS, iTunSMPB, Matroska's tick precision and AAC in CAF. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 2 (#443) Corrects round 1: - A multi-block ADTS frame after the stream has started is concealed at its whole length; round 1's filter skipped it, shifting the audio after it. - Sample durations only trim an MP4's last packet: wrong ones (some count a HE-AAC packet's core frames) were cutting audio. Durations of 0 with an edit use the edit. Raw ADTS packets now keep their headers, so TrackDecoder alone handles format changes, multi-block frames and concealment, for apps as for files, and nested ADTS payloads are not split again. Headers come from the frame scan's reads, not one read per frame. The stream's format is the one most of its first frames share, so one damaged header does not silence the file. The ID3 walk and a packet's ADTS split are bounded. AAC: - HE-AAC is delayed by the SBR filterbanks' 481 core samples; it played 19.5 ms early against ffmpeg in every container. - A failed access unit leaves the decoder's state as it was, so the next packet decodes as if it had not been there. - reset restarts the noise generator, so a seek into PNS repeats. - A signalled rate that is not twice the core's plays the core; it reported the signalled rate with the core's samples. The decoder oracle's tolerance tightens from 1e-3 to 1e-5. Tests cover HE-AAC. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 3 (#443) Corrects round 2: - ADTS headers are compared with the stream's format only when the first header gave it. With extradata the format comes from there, so ADTS in MP4 whose headers say something else decodes again; every packet had been concealed. - A stream's only program config element is found in its first frames even when that frame's header is damaged. - The test of wrong sample durations patched the second run's count, not the delta, so "durations of one" changed nothing. It now patches the delta, and checks a file was changed and that seeks land on the same audio. MP4 seeks place a packet by its index when every packet decodes to the same length, as decoding does; they used the durations round 2 had stopped trusting for the length, and returned the wrong audio. A last delta of 0 is overridden too. Docs: what TrackInfo says for raw ADTS, that opening one reads the whole file, and which codecs symphonia reads. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 4 (#443) - TrackDecoder takes an ADTS stream's format from the one most of the lookahead shares, as the file path did, so one damaged first header does not fail every packet of an MPEG-TS stream. open_adts now just asks it. - MP4 samples of several ADTS frames keep the sample table's timing; a packet count of one frame each truncated them silently. - Sample durations past what the packets decode to no longer over-report the length. - split_adts_upto is not behind aac-decode, as split_adts is not: an aac-encode build in the next PR did not compile. Removes an unreachable ID_END check and open_adts's unused core rate. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 5 (#443) No change in decoding. An aac-encode build in the next PR needs the split_adts_upto re-export behind aac-decode, and ID_END only in tests. Docs say the lookahead's headers give an ADTS stream's configuration, that AAC in CAF still goes through symphonia, and that ProgramConfig is not #[non_exhaustive]. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 6 (#443) Two tests from round 1 failed on ffmpeg before 9, including CI's 8.0. Neither was a decoding fault: - ffmpeg 9 ends an MP4 with no edit list at its sample durations and earlier versions present the whole last packet, so the length is checked against the file's known duration, not ffmpeg's decode. - The order ffmpeg gives channels with -aac_pce differs by version, so each decoded channel must match a different one of ffmpeg's, whatever the order. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 7 (#443) Reading: - an element of another type or a tag a program config does not list is refused, not written to another channel; only an LFE coded as an SCE is still allowed, as ffmpeg allows it - an explicit sampling rate of 0 is refused - a packet's later ADTS frames are searched for SBR - a multi-frame ADTS packet that fails leaves the decoder as it was, and one whose frames change the channel count is an error - the whole ADTS frames before a damaged end of a packet decode - a failed read of a raw stream's first frames is an I/O error - an MP4 sample of several ADTS frames past the lookahead keeps its length - an Ogg file with leading bytes keeps its track order and mask comments Writing: - ProgramConfig::write, AudioSpecificConfig::to_bytes and AdtsHeader::write refuse what their fields cannot hold Tests: - the routing test checks one alignment for the ffmpeg version under test - a default decode past an Ogg stream symphonia has no track for Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 8 (#443) From wanikhawar's review: - AudioSpecificConfig::parse reads backward-compatible SBR and PS signalling (a 0x2b7 sync extension after the config), which craft-media's Matroska demuxer writes for legacy HE-AAC IDs. Without lookahead those streams decoded as their core. - An MP4 media edit after one empty edit is applied: its priming skip and its end. The empty edit's delay is not added as silence. - A packet lost whole is concealed as long as the last one decoded, so a stream of multi-frame ADTS packets keeps its timing. - ADTS-framed MP4 samples keep their sample table's timing, instead of every sample being read at open to look for several frames in one. - AdtsHeader::write refuses an object type, sampling frequency index or channel configuration its fields cannot hold; it masked them into other values. - AudioSpecificConfig::lc no longer panics for a sampling frequency index with no rate, and the decoder refuses such a config. - AudioSpecificConfig::to_bytes refuses channel configuration 0 without a program config element, which would not parse back. - Upsample2x starts each channel's history afresh when the channel count changes. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 9 (#443) From CodeRabbit's review of round 8: - AudioSpecificConfig::parse consumes extensionFlag3 after the config when extensionFlag is set, so the 0x2b7 sync extension after it is found. - AudioSpecificConfig::to_bytes refuses a sampling rate of 0, which would not parse back. - ADTS-framed HE-AAC in MP4 whose sample durations count core frames is not cut to the durations' half length: durations short of one frame per sample no longer end an ADTS-framed track, and its length is unknown until decoded. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 10 (#443) ADTS-framed AAC in MP4: durations short of the frames in the samples read ahead no longer end the track. With HE-AAC samples of two ADTS frames each, timed as two core frames, the durations matched one output frame per sample, so the undercount check passed and the track stopped at half its length, which probe reported as known. The frames each read-ahead sample holds are now counted, and durations short of them leave the length unknown until decoded, as durations short of one frame per sample already did. Correctly timed AAC-LC samples of several frames keep their end trim. Reported by FredApps. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 11 (#443) Corrects round 10. A short ADTS-framed MP4 track, whose last sample is in the read-ahead, lost that sample's trim when it cut more than one frame: the trim counted as durations falling short, so the length was left to decoding and its padding was presented. The track's last sample is now left out of the read-ahead comparison. Reported by CodeRabbit. Co-Authored-By: Claude Opus 5.5 * craft-audio: document the ADTS-in-MP4 seek and edit limits (#443) AAC in MP4 stored as ADTS frames, with sample durations that count fewer frames than the samples hold, decodes in full from the start, but seeks place samples by those durations and land later than asked, and an edit list measured in them can end the track early. The README's Limits section now says so, and the seek example points to it. Reported by FredApps. Co-Authored-By: Claude Opus 5.5 * craft-audio: say why the ADTS-in-MP4 seek limit is kept (#443) The Limits entry from 79f5839 said what goes wrong but not why it is left. It now says that such a file is outside the MP4 specification: an MP4 AAC sample is one raw access unit with the configuration in the sample description (ISO/IEC 14496-14, 14496-3), so ADTS headers do not belong in it, and a sample's duration is the time it decodes to (ISO/IEC 14496-12). It says why files within the specification are not affected, what still works, and why seeks and edits in such files are not corrected: finding where each sample's audio starts means counting the ADTS frames in every sample before it, one read per sample, which would slow every ADTS-framed MP4 at open to work around files outside the specification. Co-Authored-By: Claude Opus 5.5 * craft-audio: AAC encoder FilmCraft's AAC-LC encoder behind `aac-encode`: psychoacoustic model, window switching, M/S, TNS, bit reservoir, CBR and VBR, 1-8 channels, optional ADTS framing. `rayon` encodes channels in parallel. `aac::aac_order` maps WAV order to the encoder's input order. Settings the encoder cannot use are `aac::Error::InvalidSettings`, which converts to `AudioError::Encode`. `encode` now turns on `aac-encode`. Tests that need AAC streams written by our own encoder come with it: the encoder oracle and robustness tests, packet concealment, and `TrackDecoder` on ADTS without extradata. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 1 (#444) From CodeRabbit's first review of the AAC encoder: - CBR bitrate limits use the exact bits per frame (bitrate * 1024 against channels * sample_rate) instead of the truncated quotient, so 288001 b/s at 48 kHz mono is rejected. - Rates below 48 bits per channel per frame are rejected. That is the largest silent frame per channel measured across 1-8 channels; below it the encoder overshot the requested rate (96 kHz mono at 1000 b/s came out near 4500 b/s). - Fill elements are capped so the byte-aligned access unit stays within 6144 bits per channel. write_fill can write 8 bits over its request, which gave 769-byte mono units at the maximum rate and when the reservoir was flushed on the last frame. - encode() buffers one frame of input at a time. Appending the whole slice and draining 1024 samples per frame copied the rest of the input once per frame, quadratic in the slice length. Output is byte-identical to feeding the input in chunks. - README: 8 channels are taken in channel configuration 7's element order; a WAV 7.1 buffer has no equivalent there. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 2 (#444) From CodeRabbit's review of round 1: - The CBR minimum adds the program config's bits when ADTS is on. The 7-channel layout repeats it in every ADTS frame, and at full bandwidth a silent frame took 344 bits against the 336 bits that 31,500 b/s gives at 96 kHz. Bandwidth alone stays within 48 bits per channel. - write_fill never writes more than it is asked for: each element's size comes from the bits left, so the total ends 0-6 bits short. The old version could pass the request by 8 bits and could also stop early (asked for 14, it wrote 7). - The encoder fills to the unit's target end, capped at max_frame_bits, instead of keeping round 1's 8-bit margin. Frames at the maximum rate were 767 bytes when there was little to code; they are now exactly 768 bytes per channel. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 3 (#444) Encoder::new rejects a NaN VBR quality, the known issue ported from FilmCraft and listed in the PR description. NaN passes f32::clamp, so the rate-distortion multiplier became NaN. The frame-size loop compares it with `<` and never ran, and no band was skipped, so every frame was coded at full precision: mono units reached 2465 bytes against the 768-byte limit, and 8-channel units of up to 17024 bytes did not fit the ADTS length field, so they went out without headers. Infinities and other out-of-range values are clamped and stay within the limit; the test checks both. Co-Authored-By: Claude Opus 5.5 * craft-audio: review fixes, round 4 (#444) From CodeRabbit's review of round 3: - The CBR fill target is rounded down to a whole byte. Round 2 filled to an unaligned target, so after byte alignment a unit could end up to 7 bits past the bits it had: in a sweep of silent CBR encodes, 1188 of 13200 frames overspent, all of them frames with fill, hidden by the level clamp at 0. The unit now ends exactly at the rounded target. The 0-7 bits rounded off stay in the reservoir and are left unspent after the last frame; the comment says so instead of claiming an exact total. - The NaN-quality test checks that infinities and out-of-range values encode exactly as the nearest end of the 0.5-6.0 range. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- .github/workflows/audio.yml | 2 +- Cargo.lock | 1 + crates/audio/Cargo.toml | 12 +- crates/audio/README.md | 56 +- crates/audio/examples/encode.rs | 10 +- crates/audio/src/aac/config.rs | 1 + crates/audio/src/aac/encoder/mod.rs | 1067 +++++++++++++++++ crates/audio/src/aac/encoder/psy.rs | 176 +++ crates/audio/src/aac/encoder/quant.rs | 383 ++++++ crates/audio/src/aac/mod.rs | 90 +- crates/audio/src/lib.rs | 27 +- crates/audio/tests/codecs/aac/common.rs | 41 +- .../audio/tests/codecs/aac/encoder_oracle.rs | 273 +++++ crates/audio/tests/codecs/aac/mod.rs | 2 + crates/audio/tests/codecs/aac/robustness.rs | 157 +++ crates/audio/tests/codecs/concealment.rs | 56 + crates/audio/tests/codecs/main.rs | 2 + crates/audio/tests/codecs/track_decoder.rs | 34 + 18 files changed, 2354 insertions(+), 36 deletions(-) create mode 100644 crates/audio/src/aac/encoder/mod.rs create mode 100644 crates/audio/src/aac/encoder/psy.rs create mode 100644 crates/audio/src/aac/encoder/quant.rs create mode 100644 crates/audio/tests/codecs/aac/encoder_oracle.rs create mode 100644 crates/audio/tests/codecs/aac/robustness.rs create mode 100644 crates/audio/tests/codecs/concealment.rs create mode 100644 crates/audio/tests/codecs/track_decoder.rs diff --git a/.github/workflows/audio.yml b/.github/workflows/audio.yml index 48572c5b..6093f275 100644 --- a/.github/workflows/audio.yml +++ b/.github/workflows/audio.yml @@ -39,7 +39,7 @@ jobs: # Each feature on its own, so code that needs another feature fails here. - name: Clippy, each feature alone run: | - for f in "" decode decode-native symphonia encode lossless-encode pcm-encode flac-encode peaks aac-decode serde; do + for f in "" decode decode-native symphonia encode lossless-encode pcm-encode flac-encode peaks aac-decode aac-encode "aac-encode,rayon" serde; do echo "::group::features: ${f:-none}" cargo clippy -p craft-audio --all-targets --locked --no-default-features ${f:+--features "$f"} -- -D warnings echo "::endgroup::" diff --git a/Cargo.lock b/Cargo.lock index 54470db0..8bf4c391 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -818,6 +818,7 @@ dependencies = [ "craft-media-testkit", "log", "proptest", + "rayon", "serde", "serde_json", "symphonia", diff --git a/crates/audio/Cargo.toml b/crates/audio/Cargo.toml index 957b4fa5..347e1a26 100644 --- a/crates/audio/Cargo.toml +++ b/crates/audio/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "craft-audio" -description = "Audio file I/O and codecs: WAV/AIFF read and write, FLAC encoder, AAC decoder." +description = "Audio file I/O and codecs: WAV/AIFF read and write, FLAC encoder, AAC decoder and encoder." version.workspace = true edition.workspace = true license.workspace = true @@ -22,8 +22,8 @@ decode = ["decode-native", "symphonia"] decode-native = ["aac-decode", "craft-media/isobmff", "craft-media/matroska", "craft-media/ogg", "dep:log"] # The symphonia fallback for the formats with no native decoder yet. symphonia = ["decode-native", "dep:symphonia"] -# Every encoder: the lossless file writers. -encode = ["lossless-encode"] +# Every encoder: the lossless file writers and the AAC encoder. +encode = ["lossless-encode", "aac-encode"] # Encode WAV/BWF/RF64, AIFF/AIFF-C and FLAC files. lossless-encode = ["pcm-encode", "flac-encode"] # Encode WAV/BWF/RF64 and AIFF/AIFF-C files. @@ -32,8 +32,11 @@ pcm-encode = [] flac-encode = ["craft-media/bitstream"] # Codec modules on their own, for packet-level use. aac-decode = ["craft-media/bitstream"] +aac-encode = ["craft-media/bitstream"] # Multi-resolution waveform overviews (`peaks`). peaks = [] +# Encode AAC channels in parallel (with `aac-encode`). +rayon = ["dep:rayon"] # Serialize and Deserialize for the shared types. serde = ["dep:serde", "craft-media/serde"] @@ -42,6 +45,7 @@ craft-media = { workspace = true } thiserror = { workspace = true } serde = { workspace = true, optional = true } log = { version = "0.4", optional = true } +rayon = { workspace = true, optional = true } # symphonia keeps its AAC decoder for AAC in CAF, a container craft-media does not read. symphonia = { version = "0.6.1", optional = true, default-features = false, features = ["mp1", "mp2", "mp3", "flac", "vorbis", "ogg", "wav", "pcm", "aiff", "isomp4", "aac", "alac", "adpcm", "caf", "mkv"] } @@ -72,4 +76,4 @@ required-features = ["decode"] [[example]] name = "encode" -required-features = ["lossless-encode"] +required-features = ["lossless-encode", "aac-encode"] diff --git a/crates/audio/README.md b/crates/audio/README.md index 6f61aa5a..e95e0f5c 100644 --- a/crates/audio/README.md +++ b/crates/audio/README.md @@ -1,7 +1,7 @@ # craft-audio Audio file I/O and codecs for the Craft apps. Native implementations: WAV and AIFF read/write, -FLAC encoder, AAC decoder. +FLAC encoder, AAC decoder/encoder. Formats without a native decoder are decoded by [symphonia](#symphonia), which is to be replaced by native decoders. @@ -26,12 +26,13 @@ No default features. The shared types (`AudioBuffer`, `AudioInfo`, `ChannelLayou | `decode` | `decode`, `probe`, `Decoder`, `tracks`, `TrackDecoder`; every format in [Format support](#format-support) | `decode-native`, `symphonia` | | `decode-native` | the same, native formats only (others are `Unsupported`) | `aac-decode`, craft-media `isobmff`/`matroska`/`ogg`, `log` | | `symphonia` | the fallback for formats with no native decoder; turning it on is the same as `decode` | `decode-native`, symphonia | -| `encode` | every encoder: the WAV, AIFF and FLAC file writers | `lossless-encode` | +| `encode` | every encoder: the WAV, AIFF and FLAC file writers, and the AAC packet encoder | `lossless-encode`, `aac-encode` | | `lossless-encode` | WAV, AIFF and FLAC writing | `pcm-encode`, `flac-encode` | | `pcm-encode` | `encode`, `FileEncoder` for WAV and AIFF; `wav_float_header` | none | | `flac-encode` | `encode`, `FileEncoder` for FLAC | craft-media `bitstream` | -| `aac-decode` | `aac::Decoder` | craft-media `bitstream` | +| `aac-decode`, `aac-encode` | `aac::Decoder`, `aac::Encoder` | craft-media `bitstream` | | `peaks` | `peaks::Peaks` | none | +| `rayon` | parallel AAC encoding (with `aac-encode`) | `rayon` | | `serde` | `Serialize`/`Deserialize` on the shared types | `serde` | `aac-decode` also gives `TrackDecoder`, for AAC and PCM. @@ -51,10 +52,10 @@ Decode, every encoder and waveform overviews: craft-audio = { git = "https://github.com/storytold/craft-libs", rev = "", features = ["decode", "encode", "peaks"] } ``` -Packets from an app's own demuxer, AAC: +Packets from an app's own demuxer, AAC both ways: ```toml -craft-audio = { git = "https://github.com/storytold/craft-libs", rev = "", features = ["aac-decode"] } +craft-audio = { git = "https://github.com/storytold/craft-libs", rev = "", features = ["aac-decode", "aac-encode"] } ``` craft-audio re-exports the craft-media types its API uses (`ByteSource`, `ReadSeekSource`, @@ -73,7 +74,7 @@ cargo run -p craft-audio --example info --features decode -- take.m4a ``` ```bash -cargo run -p craft-audio --example encode --features lossless-encode -- out +cargo run -p craft-audio --example encode --features lossless-encode,aac-encode -- out ``` ```bash @@ -216,6 +217,31 @@ features can write. `write` refuses a block without one slice per channel, or wi shorter than `frames`, and writes nothing. After a write fails for another reason, every later `write` and `finish` returns an error. +### Encode AAC + +```rust +use craft_audio::AudioError; +use craft_audio::aac::{Encoder, EncoderConfig, aac_order}; + +let mut enc = Encoder::new(EncoderConfig::cbr(48_000, 6, 384_000))?; +let order = aac_order(6).ok_or_else(|| AudioError::Unsupported("no AAC order for 6 channels".into()))?; +let input: Vec<&[f32]> = order.iter().map(|&c| wav_5_1[c].as_slice()).collect(); +let mut units = enc.encode(&input); // zero or more access units +units.extend(enc.flush()); +let asc = enc.audio_specific_config(); // for the MP4 `esds` box +let priming = enc.priming_samples(); // for the edit list (1024) +``` + +The sample rate must be one of the 13 AAC rates (7350–96000 Hz). `aac_order` returns `None` for +mono and stereo, which are already in order, and for 8 channels. Eight channels are written as AAC +channel configuration 7 and taken in its element order: C, Lc Rc, L R, Ls Rs, LFE (front centre, +front left and right of centre, front left and right, surrounds). A WAV 7.1 buffer (L R C LFE Lb Rb +Ls Rs) has no equivalent in that layout; passed unchanged, its channels go to the wrong speakers. + +`EncoderConfig::cbr(..).with_adts(true)` makes every access unit an ADTS frame; concatenated, they +are a playable `.aac` file. To write MP4 or Matroska, pass the access units to craft-media's +muxers ([MP4](../media/README.md#muxing), [Matroska](../media/README.md#muxing-1)). + ### Waveform overviews ```rust @@ -240,7 +266,7 @@ convert into `AudioError` with `?`. | `Unsupported` | valid input using an unsupported codec, feature or option, or a format whose feature is off | | `Malformed` | corrupt or truncated input, or not the claimed format | | `TooLarge` | over a size limit (`DecodeOptions::max_samples`, an allocation) | -| `Encode` | an encoder rejected the input or its settings (a buffer with no channels or a 0 Hz rate, a block shorter than its frame count, a wrong layout) | +| `Encode` | an encoder rejected the input or its settings (a buffer with no channels or a 0 Hz rate, a block shorter than its frame count, a wrong layout, a rate AAC has no index for) | | `Io` | the source failed a read, or the `FileEncoder` writer failed | `AudioError` is `#[non_exhaustive]`, so a `match` needs a `_` arm: @@ -260,7 +286,7 @@ match craft_audio::decode(&bytes, None) { | WAV, BWF, RF64/BW64 | native | native | | AIFF, AIFF-C | native | native | | FLAC | symphonia | native | -| AAC-LC | native | — | +| AAC-LC | native | native | | HE-AAC v1/v2 | native, core only (see [Limits](#limits)) | — | | MP1, MP2, MP3 (MPEG audio Layer I/II/III) | symphonia | — | | ALAC, Vorbis, ADPCM, A-law, µ-law | symphonia | — | @@ -343,7 +369,7 @@ Each codec module's rustdoc lists the coding tools it implements. |---|---|---|---| | WAV, AIFF | Microsoft RIFF, Apple AIFF/AIFF-C | read: integer 8–32-bit, float 32/64-bit, `WAVE_FORMAT_EXTENSIBLE` masks, RF64/BW64, BWF `bext`, truncated files; write: 16/24/32-bit integer and 32-bit float, RF64 above 4 GiB, any layout, TPDF dither | round trips; hostile-input tests | | FLAC encoder | RFC 9639 | 16, 24, 32 bits; 1–8 channels, layouts decoders would not assume for the count as a `WAVEFORMATEXTENSIBLE_CHANNEL_MASK` comment; up to 655,350 Hz; LPC, stereo decorrelation, wasted bits, STREAMINFO MD5 | bit-exact through ffmpeg, STREAMINFO MD5 checked against ffmpeg | -| AAC (`aac`) | ISO/IEC 14496-3 | AAC-LC decode, 1–8 channels; HE-AAC: see [Limits](#limits) | ffmpeg: ~135 dB SNR; PNS noise differs per sample at equal band energy | +| AAC (`aac`) | ISO/IEC 14496-3 | AAC-LC decode and encode (CBR, VBR, ADTS), 1–8 channels; HE-AAC: see [Limits](#limits) | ffmpeg: ~135 dB SNR; PNS noise differs per sample at equal band energy | ## symphonia @@ -414,16 +440,15 @@ For the apps switching from their own code; to be removed once all three have sw The codec modules' public items keep their names, so apps can re-export them (`pub use craft_audio::aac::*;`). Names kept only for that are `#[doc(hidden)]`: -`aac::Upsample2x`, `aac::ProgramConfig::parse`, `aac::config_layout` and `aac::sample_rate_index`. -What else an app adjusts: +`aac::Upsample2x`, `aac::ProgramConfig::parse`, `aac::config_layout` and `aac::sample_rate_index`. What else an app adjusts: - `AudioInfo` has `container`, `codec`, `layout` and `damaged` instead of SoundCraft's `format`. `detect_container` replaces `detect_format`. `probe` reports the layout instead of `wav_channel_mask`. - `EncodeOptions` is built with `EncodeOptions::new(format)` and `with_*` methods. It carries the speaker layout and RF64, which replace `encode_with_layout`, `encode_wav_rf64` and - `FileEncoder::new`'s layout argument. `BwfInfo` is built the same way. `ENCODE_EXTENSIONS` is a - list. + `FileEncoder::new`'s layout argument. `BwfInfo` and `aac::EncoderConfig` are built the same way + (`with_adts`, …). `ENCODE_EXTENSIONS` is a list. - Arguments go sample rate, then channels, and channel counts are `u16`: `FileEncoder::new(out, &opts, sample_rate, channels)`, `wav_float_header(sample_rate, channels, frames)`, `AudioBuffer::new(sample_rate, channels, frames)`, @@ -435,8 +460,9 @@ What else an app adjusts: `Speaker::ALL` is a slice. - `TrackInfo::sample_rate` and `channels` are `Option`s. - Features are per codec. SoundCraft's `encode` (WAV, AIFF, FLAC) is now `lossless-encode`, made - of `pcm-encode` (WAV, AIFF) and `flac-encode`. `encode` turns on every encoder. Writing a format - whose feature is off returns `AudioError::Unsupported`. + of `pcm-encode` (WAV, AIFF) and `flac-encode`. `encode` turns on every encoder, AAC + included. Writing a format whose feature is off returns `AudioError::Unsupported`. +- AAC encoder settings it cannot use are `aac::Error::InvalidSettings` (`AudioError::Encode`). - `FileEncoder::write` takes any per-channel slices, but refuses a block with missing channels or short slices instead of filling them with silence. - `DecodeOptions` is not `Copy`. diff --git a/crates/audio/examples/encode.rs b/crates/audio/examples/encode.rs index 9ed70e7a..28de0f5a 100644 --- a/crates/audio/examples/encode.rs +++ b/crates/audio/examples/encode.rs @@ -1,6 +1,6 @@ -//! Write a tone as WAV and FLAC, whole and streamed. +//! Write a tone as WAV and FLAC, whole and streamed, and as AAC packets. //! -//! `cargo run -p craft-audio --example encode --features lossless-encode -- out` +//! `cargo run -p craft-audio --example encode --features lossless-encode,aac-encode -- out` use craft_audio::{AudioBuffer, BitDepth, ChannelLayout, EncodeFormat, EncodeOptions, FileEncoder, encode}; @@ -27,5 +27,11 @@ fn main() -> Result<(), Box> { enc.write(&[chunk, chunk], chunk.len())?; } enc.finish()?; + + // AAC packets for an MP4 or Matroska muxer, with the configuration its sample entry needs. + let mut aac = craft_audio::aac::Encoder::new(craft_audio::aac::EncoderConfig::cbr(rate as u32, 2, 128_000))?; + let mut units = aac.encode(&[&tone, &tone]); + units.extend(aac.flush()); + println!("{} AAC access units, AudioSpecificConfig {:02x?}, priming {}", units.len(), aac.audio_specific_config(), aac.priming_samples()); Ok(()) } diff --git a/crates/audio/src/aac/config.rs b/crates/audio/src/aac/config.rs index eb7ed7e2..88be1e21 100644 --- a/crates/audio/src/aac/config.rs +++ b/crates/audio/src/aac/config.rs @@ -505,6 +505,7 @@ mod tests { for sf_index in 13..=15 { let asc = AudioSpecificConfig::lc(sf_index, 2); assert_eq!(asc.sample_rate, 0); + #[cfg(feature = "aac-decode")] assert!(crate::aac::Decoder::from_config(asc).is_err()); } } diff --git a/crates/audio/src/aac/encoder/mod.rs b/crates/audio/src/aac/encoder/mod.rs new file mode 100644 index 00000000..5230df5b --- /dev/null +++ b/crates/audio/src/aac/encoder/mod.rs @@ -0,0 +1,1067 @@ +//! AAC-LC encoder. +//! +//! Pipeline per 1024-sample frame and channel element: +//! 1. transient detection on high-passed 128-sample sub-blocks (one frame of look-ahead) → +//! window sequence (`ONLY_LONG` / `LONG_START` / `EIGHT_SHORT` / `LONG_STOP`) and short-window grouping; +//! 2. windowed MDCT (sine or KBD); +//! 3. psychoacoustic model → per-band masking thresholds and perceptual entropy; +//! 4. TNS analysis (long windows; Levinson–Durbin, applied when the prediction gain is high); +//! 5. per-band M/S decision for channel pairs (common window); +//! 6. rate–distortion loop: a single multiplier λ scales every band's allowed noise; each band gets the +//! coarsest scalefactor meeting its noise target, sections/codebooks are chosen by dynamic +//! programming, and λ is bisected until the frame fits its bit budget (CBR) or fixed (VBR); +//! 7. bit reservoir (CBR): budget follows the frame's PE, bounded by the decoder buffer +//! (6144 bits/channel); overflow is padded with fill elements so the rate stays constant. + +mod psy; +mod quant; + +use craft_media::bitstream::BitWriter; + +use crate::aac::config::{AdtsHeader, AudioSpecificConfig, ElementType, ProgramConfig, config_layout}; +use crate::aac::ics::{IcsInfo, WindowSequence}; +use crate::aac::mdct::{WindowShape, mdct_long, mdct_short, window}; +use crate::aac::tables::{sample_rate_index, swb_offsets_long, swb_offsets_short, tns_max_bands}; +use crate::aac::tns::{self, TnsData, TnsFilter}; +use crate::aac::{Error, FRAME_LEN, Result}; +use psy::Psy; +use quant::{ChannelInput, QuantChannel, quantise_channel, write_ics}; + +/// Rate control mode. +#[derive(Debug, Clone, Copy, PartialEq)] +pub enum BitrateMode { + /// Constant bitrate in bits per second (bit reservoir + fill elements). + Cbr(u32), + /// Constant quality, 1.0 (small) ..= 5.0 (transparent-ish). 3.0 targets the masking threshold. + /// Values outside 0.5..=6.0 are clamped to it; NaN is rejected by [`Encoder::new`]. + Vbr(f32), +} + +/// Encoder settings. +#[non_exhaustive] +#[derive(Debug, Clone, PartialEq)] +pub struct EncoderConfig { + /// One of the 13 AAC sampling rates (8–96 kHz). + pub sample_rate: u32, + /// 1–8. Channel order is the AAC element order: C; L R; C L R; C L R Cs; C L R Ls Rs; + /// C L R Ls Rs LFE; (7) C L R Ls Rs Cs LFE; (8) C Lc Rc L R Ls Rs LFE. Audio in WAV order is + /// reordered with [`crate::aac::aac_order`]. + pub channels: usize, + /// Rate control mode. + pub mode: BitrateMode, + /// Audio bandwidth in Hz (`None` = chosen from the bitrate / quality). + pub bandwidth: Option, + /// Temporal noise shaping on long windows. + pub tns: bool, + /// Per-band mid/side stereo for channel pairs. + pub ms_stereo: bool, + /// Window shape for every frame (`window_shape` bit). + pub window_shape: WindowShape, + /// Prefix every access unit with an ADTS header. + pub adts: bool, +} + +impl EncoderConfig { + /// Constant-bitrate settings at `bitrate_bps` bits per second, with TNS and M/S stereo on, sine + /// window, automatic bandwidth and no ADTS. + pub fn cbr(sample_rate: u32, channels: usize, bitrate_bps: u32) -> EncoderConfig { + EncoderConfig { + sample_rate, + channels, + mode: BitrateMode::Cbr(bitrate_bps), + bandwidth: None, + tns: true, + ms_stereo: true, + window_shape: WindowShape::Sine, + adts: false, + } + } + /// Constant-quality settings (`quality` as in [`BitrateMode::Vbr`]); otherwise as [`Self::cbr`]. + pub fn vbr(sample_rate: u32, channels: usize, quality: f32) -> EncoderConfig { + EncoderConfig { mode: BitrateMode::Vbr(quality), ..Self::cbr(sample_rate, channels, 0) } + } + + /// Sets `bandwidth` (`None` = chosen from the bitrate or quality). + pub fn with_bandwidth(mut self, bandwidth: Option) -> EncoderConfig { + self.bandwidth = bandwidth; + self + } + + /// Sets `tns`. + pub fn with_tns(mut self, tns: bool) -> EncoderConfig { + self.tns = tns; + self + } + + /// Sets `ms_stereo`. + pub fn with_ms_stereo(mut self, ms_stereo: bool) -> EncoderConfig { + self.ms_stereo = ms_stereo; + self + } + + /// Sets `window_shape`. + pub fn with_window_shape(mut self, window_shape: WindowShape) -> EncoderConfig { + self.window_shape = window_shape; + self + } + + /// Sets `adts`: prefix every access unit with an ADTS header. + pub fn with_adts(mut self, adts: bool) -> EncoderConfig { + self.adts = adts; + self + } +} + +/// Running statistics. +#[non_exhaustive] +#[derive(Debug, Clone, Copy, Default, PartialEq)] +pub struct EncoderStats { + /// Access units produced. + pub frames: u64, + /// Raw access-unit bytes (without ADTS headers). + pub bytes: u64, + /// Element-frames coded with eight short windows. + pub short_blocks: u64, + /// Scalefactor bands coded as mid/side. + pub ms_bands: u64, + /// Scalefactor bands of channel pairs considered for mid/side coding. + pub stereo_bands: u64, + /// TNS filters applied. + pub tns_filters: u64, + /// Bits spent on fill elements (CBR padding). + pub fill_bits: u64, +} + +impl EncoderStats { + /// Average bitrate of the raw access units. + pub fn bitrate(&self, sample_rate: u32) -> f64 { + if self.frames == 0 { 0.0 } else { self.bytes as f64 * 8.0 * sample_rate as f64 / (self.frames as f64 * FRAME_LEN as f64) } + } +} + +struct ElementState { + ty: ElementType, + tag: u8, + first_ch: usize, + prev_seq: WindowSequence, + short_next: Option, + /// Previous long-window thresholds per channel (pre-echo control). + prev_thr: Vec>, +} + +/// Per-frame analysis of one element. +struct ElementFrame { + info: IcsInfo, + chans: Vec, + ms_mask: Option<(u32, Vec)>, +} + +const HISTORY: usize = 2048; +/// Bits per channel in a silent frame, rounded up: the smallest frame the encoder writes. +const MIN_FRAME_BITS_PER_CHANNEL: u64 = 48; + +/// AAC-LC encoder. +pub struct Encoder { + cfg: EncoderConfig, + sf_index: u8, + asc: AudioSpecificConfig, + psy: Psy, + elements: Vec, + /// Input history per channel; `hist[c][0]` is absolute sample `origin`. + hist: Vec>, + origin: i64, + total_in: u64, + frames_done: u64, + flushed: bool, + bandwidth: u32, + // rate control + mean_acc: u64, + level: i64, + res_max: i64, + max_frame_bits: i64, + pe_avg: f32, + lambda: f32, + stats: EncoderStats, +} + +impl Encoder { + /// Create an encoder. Fails if the sampling rate is not an AAC rate, `channels` is outside + /// 1..=8, the CBR bitrate is out of range for the channel count and rate, or the VBR quality is + /// NaN. + pub fn new(cfg: EncoderConfig) -> Result { + let sf_index = + sample_rate_index(cfg.sample_rate).ok_or(Error::InvalidSettings("sample rate must be one of the 13 AAC rates (7350–96000 Hz)"))?; + if !(1..=8).contains(&cfg.channels) { + return Err(Error::InvalidSettings("channels must be 1..=8")); + } + // A NaN quality passes `clamp` and makes the rate-distortion multiplier NaN, which turns off + // the frame-size limit: every band is coded at full precision, and 8-channel frames outgrow + // the ADTS length field. + if let BitrateMode::Vbr(q) = cfg.mode + && q.is_nan() + { + return Err(Error::InvalidSettings("VBR quality must be a number")); + } + use ElementType::*; + let (channel_config, layout) = match cfg.channels { + 7 => (0u8, vec![(Sce, 0u8), (Cpe, 0), (Cpe, 1), (Sce, 1), (Lfe, 0)]), + 8 => (7, config_layout(7).unwrap_or_default()), + n => (n as u8, config_layout(n as u8).unwrap_or_default()), + }; + let mut asc = AudioSpecificConfig::lc(sf_index, channel_config); + if channel_config == 0 { + asc.pce = Some(ProgramConfig { + element_instance_tag: 0, + object_type: 1, + sf_index, + front: vec![(false, 0), (true, 0)], + side: vec![(true, 1)], + back: vec![(false, 1)], + lfe: vec![0], + ..Default::default() + }); + } + let mut elements = Vec::new(); + let mut ch = 0; + for (ty, tag) in layout { + elements.push(ElementState { + ty, + tag, + first_ch: ch, + prev_seq: WindowSequence::OnlyLong, + short_next: None, + prev_thr: vec![Vec::new(); ty.channels()], + }); + ch += ty.channels(); + } + let nfull = cfg.channels - elements.iter().filter(|e| e.ty == Lfe).count(); + let nyq = cfg.sample_rate / 2; + let auto_bw = match cfg.mode { + BitrateMode::Cbr(b) => { + let per = b as f32 / nfull.max(1) as f32 / 1000.0; + let pts = [ + (12.0f32, 4000.0f32), + (16.0, 5500.0), + (24.0, 8000.0), + (32.0, 11000.0), + (48.0, 14000.0), + (64.0, 16000.0), + (80.0, 17500.0), + (96.0, 19000.0), + (112.0, 20000.0), + (144.0, 22000.0), + ]; + interp(&pts, per) as u32 + } + BitrateMode::Vbr(q) => interp(&[(1.0, 13000.0), (2.0, 15000.0), (3.0, 17000.0), (4.0, 19000.0), (5.0, 20000.0)], q) as u32, + }; + let bandwidth = cfg.bandwidth.unwrap_or(auto_bw).clamp(1000, nyq.saturating_sub(nyq / 50).max(1000)); + let max_frame_bits = 6144 * cfg.channels as i64; + let mean = match cfg.mode { + BitrateMode::Cbr(b) => b as i64 * FRAME_LEN as i64 / cfg.sample_rate as i64, + BitrateMode::Vbr(_) => 0, + }; + // Compare the exact bits per frame, `b * 1024 / rate`, against what a frame can hold: at + // most 6144 bits per channel (the decoder buffer), and at least what a silent frame takes: + // 48 bits per channel, plus the program config that ADTS repeats in every 7-channel frame. + // Outside that range the encoder cannot keep the requested rate. + if let BitrateMode::Cbr(b) = cfg.mode { + let per_frame = b as u64 * FRAME_LEN as u64; + let rate = cfg.sample_rate as u64; + let pce_bits = match &asc.pce { + Some(p) if cfg.adts => { + let mut bw = BitWriter::new(); + let _ = p.write(&mut bw); // its own program config, which fits + 3 + bw.bit_len() as u64 + } + _ => 0, + }; + let min_frame_bits = MIN_FRAME_BITS_PER_CHANNEL * cfg.channels as u64 + pce_bits; + if b < 1000 * cfg.channels as u32 || per_frame > max_frame_bits as u64 * rate || per_frame < min_frame_bits * rate { + return Err(Error::InvalidSettings("bitrate out of range for the channel count and sample rate")); + } + } + let res_max = (max_frame_bits - mean).max(0); + let hist = vec![vec![0f32; HISTORY]; cfg.channels]; + Ok(Encoder { + psy: Psy::new(cfg.sample_rate, sf_index), + sf_index, + asc, + elements, + hist, + origin: -(HISTORY as i64), + total_in: 0, + frames_done: 0, + flushed: false, + bandwidth, + mean_acc: 0, + level: 0, + res_max, + max_frame_bits, + pe_avg: 0.0, + lambda: 1.0, + stats: EncoderStats::default(), + cfg, + }) + } + + /// The settings the encoder was created with. + pub fn config(&self) -> &EncoderConfig { + &self.cfg + } + + /// `AudioSpecificConfig` bytes for the MP4 `esds` decoder-specific info. + pub fn audio_specific_config(&self) -> Vec { + // The only program config the encoder writes is its own, for 7 channels, which fits. + self.asc.to_bytes().unwrap_or_default() + } + + /// Encoder delay: the first 1024 decoded samples are priming and should be skipped (MP4 edit + /// list `media_time`, or iTunSMPB). + pub fn priming_samples(&self) -> u32 { + FRAME_LEN as u32 + } + + /// Audio bandwidth in Hz. + pub fn bandwidth(&self) -> u32 { + self.bandwidth + } + + /// Statistics so far. + pub fn stats(&self) -> EncoderStats { + self.stats + } + + /// Feed planar samples (±1.0 full scale); returns any complete access units. Missing channels are + /// treated as silence; all channels are truncated to the shortest slice. + pub fn encode(&mut self, planar: &[&[f32]]) -> Vec> { + if self.flushed { + return Vec::new(); + } + let n = planar.iter().take(self.cfg.channels).map(|p| p.len()).min().unwrap_or(0); + let mut out = Vec::new(); + // Buffer one frame of input at a time: each encoded frame shifts the history, so buffering a + // long slice whole would copy the rest of it once per frame. + let mut start = 0; + while start < n { + let end = (start + FRAME_LEN).min(n); + for c in 0..self.cfg.channels { + match planar.get(c) { + Some(p) => self.hist[c].extend(p[start..end].iter().map(|v| if v.is_finite() { v.clamp(-8.0, 8.0) } else { 0.0 })), + None => self.hist[c].extend(std::iter::repeat_n(0.0, end - start)), + } + } + self.total_in += (end - start) as u64; + while self.available() >= self.needed_for(self.frames_done) { + out.push(self.encode_frame(false)); + } + start = end; + } + out + } + + /// Encode the remaining input (zero-padded). Afterwards the decoded stream, minus + /// [`priming_samples`](Self::priming_samples), covers every input sample. + pub fn flush(&mut self) -> Vec> { + if self.flushed { + return Vec::new(); + } + let frames = if self.total_in == 0 { 0 } else { self.total_in.div_ceil(FRAME_LEN as u64) + 1 }; + let mut out = Vec::new(); + while self.frames_done < frames { + let need = self.needed_for(self.frames_done); + let have = self.available(); + if have < need { + for h in &mut self.hist { + h.extend(std::iter::repeat_n(0.0, (need - have) as usize)); + } + } + let last = self.frames_done + 1 == frames; + out.push(self.encode_frame(last)); + } + self.flushed = true; + out + } + + /// Absolute sample index one past the buffered input. + fn available(&self) -> i64 { + self.origin + self.hist[0].len() as i64 + } + /// Input needed (absolute end) before frame `t` can be encoded (window + transient look-ahead). + fn needed_for(&self, t: u64) -> i64 { + (t as i64 + 1) * FRAME_LEN as i64 + 640 + } + + fn sample(&self, ch: usize, abs: i64) -> f32 { + let i = abs - self.origin; + if i < 0 { 0.0 } else { self.hist[ch].get(i as usize).copied().unwrap_or(0.0) } + } + + fn subblock_energy(&self, ch: usize, j: i64) -> f32 { + let s = j * 128; + let mut e = 0f32; + let mut prev = self.sample(ch, s - 1); + for n in s..s + 128 { + let x = self.sample(ch, n); + let d = x - prev; + e += d * d; + prev = x; + } + e + } + + /// Transient in the region covered by frame `t`'s short windows. + fn transient(&self, first_ch: usize, nch: usize, t: i64) -> bool { + for ch in first_ch..first_ch + nch { + let base = 8 * t; + let mut hist: Vec = (base - 13..base - 5).map(|j| self.subblock_energy(ch, j)).collect(); + for j in base - 5..=base + 4 { + let e = self.subblock_energy(ch, j); + let mean = hist.iter().sum::() / hist.len() as f32; + if e > 1.5e-4 && e > 10.0 * mean { + return true; + } + hist.remove(0); + hist.push(e); + } + } + false + } + + fn max_sfb(&self, short: bool, lfe: bool) -> usize { + let bw = if lfe { 240.min(self.bandwidth) } else { self.bandwidth }; + let (swb, m) = if short { (swb_offsets_short(self.sf_index), 128f64) } else { (swb_offsets_long(self.sf_index), 1024f64) }; + let bin = (bw as f64 / (self.cfg.sample_rate as f64 / 2.0) * m).ceil() as u16; + swb.iter().take(swb.len() - 1).filter(|&&o| o < bin).count().max(1) + } + + fn mdct(&self, ch: usize, seq: WindowSequence) -> Vec { + let shape = self.cfg.window_shape; + let seg = &self.hist[ch][HISTORY - FRAME_LEN..HISTORY + FRAME_LEN]; + let mut spec = vec![0f32; 1024]; + let scale = 32768.0 * 2.0; + if seq == WindowSequence::EightShort { + let ws = window(shape, false); + let mut buf = vec![0f32; 256]; + for w in 0..8 { + let off = 448 + 128 * w; + for n in 0..128 { + buf[n] = seg[off + n] * ws[n] * scale; + buf[128 + n] = seg[off + 128 + n] * ws[127 - n] * scale; + } + mdct_short().forward(&buf, &mut spec[w * 128..(w + 1) * 128]); + } + return spec; + } + let wl = window(shape, true); + let ws = window(shape, false); + let mut buf = vec![0f32; 2048]; + for n in 0..1024 { + let w = match seq { + WindowSequence::LongStop => { + if n < 448 { + 0.0 + } else if n < 576 { + ws[n - 448] + } else { + 1.0 + } + } + _ => wl[n], + }; + buf[n] = seg[n] * w * scale; + } + for n in 1024..2048 { + let w = match seq { + WindowSequence::LongStart => { + if n < 1472 { + 1.0 + } else if n < 1600 { + ws[127 - (n - 1472)] + } else { + 0.0 + } + } + _ => wl[2047 - n], + }; + buf[n] = seg[n] * w * scale; + } + mdct_long().forward(&buf, &mut spec); + spec + } + + fn tns_analyse(&self, spec: &mut [f32], info: &IcsInfo) -> Option { + let swb = swb_offsets_long(self.sf_index); + let lim = tns_max_bands(self.sf_index, false).min(info.max_sfb); + let hz = self.cfg.sample_rate as f64 / 2048.0; + let start = swb.iter().position(|&o| o as f64 * hz >= 1400.0).unwrap_or(swb.len() - 1); + if lim < start + 4 { + return None; + } + let (lo, hi) = (swb[start] as usize, swb[lim] as usize); + let x = &spec[lo..hi]; + const ORDER: usize = 8; + let mut r = [0f64; ORDER + 1]; + for (lag, rv) in r.iter_mut().enumerate() { + let mut acc = 0f64; + for n in lag..x.len() { + acc += x[n] as f64 * x[n - lag] as f64; + } + // Gaussian lag window for robustness + *rv = acc * (-0.5 * (0.02 * lag as f64).powi(2)).exp(); + } + if r[0] <= 1e-3 { + return None; + } + let (ks, gain) = tns::levinson(&r, ORDER); + if gain < 1.4 { + return None; + } + let mut coef = [0i8; 12]; + let mut order = 0; + for (i, &k) in ks.iter().enumerate() { + coef[i] = tns::quant_coef(k, true); + if coef[i] != 0 { + order = i + 1; + } + } + if order == 0 { + return None; + } + let mut t = TnsData::default(); + t.windows[0].n_filt = 1; + t.windows[0].coef_res = true; + t.windows[0].filt[0] = TnsFilter { length: (swb.len() - 1 - start) as u8, order: order as u8, direction: false, coef_compress: false, coef }; + tns::apply_encoder(spec, info, self.sf_index, &t); + Some(t) + } + + fn analyse_element(&mut self, ei: usize, t: i64) -> (ElementFrame, f32) { + let (ty, first, nch, prev_seq, cached) = { + let e = &self.elements[ei]; + (e.ty, e.first_ch, e.ty.channels(), e.prev_seq, e.short_next) + }; + let lfe = ty == ElementType::Lfe; + let (short_now, short_next) = + if lfe { (false, false) } else { (cached.unwrap_or_else(|| self.transient(first, nch, t)), self.transient(first, nch, t + 1)) }; + use WindowSequence::*; + let seq = if short_now { + EightShort + } else if prev_seq == EightShort || prev_seq == LongStart { + if short_next { EightShort } else { LongStop } + } else if short_next { + LongStart + } else { + OnlyLong + }; + let specs: Vec> = (first..first + nch).map(|c| self.mdct(c, seq)).collect(); + let short = seq == EightShort; + let mut info = IcsInfo { window_sequence: seq, window_shape: self.cfg.window_shape, max_sfb: self.max_sfb(short, lfe), ..Default::default() }; + if short { + // group windows of similar energy; an attack starts a new group + let en: Vec = + (0..8).map(|w| specs.iter().map(|s| s[w * 128..(w + 1) * 128].iter().map(|v| v * v).sum::()).sum::() + 1.0).collect(); + info.num_groups = 1; + info.group_len = [1, 0, 0, 0, 0, 0, 0, 0]; + for w in 1..8 { + let r = en[w] / en[w - 1]; + if !(1.0 / 6.0..=3.0).contains(&r) { + info.num_groups += 1; + info.group_len[info.num_groups - 1] = 1; + } else { + info.group_len[info.num_groups - 1] += 1; + } + } + } + let mut pe = 0f32; + let mut specs = specs; + let mut thrs: Vec>> = Vec::with_capacity(nch); + let mut tnss = Vec::with_capacity(nch); + for (i, spec) in specs.iter_mut().enumerate() { + let mut r = self.psy.analyse(spec, short); + pe += r.pe; + let e = &mut self.elements[ei]; + if !short { + if prev_seq != EightShort && e.prev_thr[i].len() == r.thr[0].len() { + for (t, p) in r.thr[0].iter_mut().zip(&e.prev_thr[i]) { + *t = t.min(2.0 * p.max(1e-9)); + } + } + e.prev_thr[i] = r.thr[0].clone(); + } else { + e.prev_thr[i].clear(); + } + // group thresholds: the most sensitive window times the group length + let mut thr_g = vec![vec![0f32; 64]; info.num_groups]; + for g in 0..info.num_groups { + let w0 = info.group_start(g); + let gl = info.group_len[g] as usize; + for sfb in 0..info.max_sfb { + let m = (w0..w0 + gl).map(|w| r.thr[w][sfb]).fold(f32::MAX, f32::min); + thr_g[g][sfb] = m * gl as f32; + } + } + thrs.push(thr_g); + let tns = if self.cfg.tns && !short && !lfe { self.tns_analyse(spec, &info) } else { None }; + if let Some(t) = &tns { + self.stats.tns_filters += t.windows[0].n_filt as u64; + } + tnss.push(tns); + } + let swb = info.swb_offsets(self.sf_index); + let mut ms_mask = None; + if ty == ElementType::Cpe { + let mut used = Vec::with_capacity(info.num_groups * info.max_sfb); + let (l, r) = specs.split_at_mut(1); + let (l, r) = (&mut l[0], &mut r[0]); + for g in 0..info.num_groups { + let w0 = info.group_start(g); + let gl = info.group_len[g] as usize; + for sfb in 0..info.max_sfb { + let (mut el, mut er, mut em, mut es) = (0f32, 0f32, 0f32, 0f32); + for w in w0..w0 + gl { + for k in w * 128 + swb[sfb] as usize..w * 128 + swb[sfb + 1] as usize { + let (a, b) = (l[k], r[k]); + el += a * a; + er += b * b; + em += 0.25 * (a + b) * (a + b); + es += 0.25 * (a - b) * (a - b); + } + } + let (tl, tr) = (thrs[0][g][sfb].max(1e-9), thrs[1][g][sfb].max(1e-9)); + let tm = 0.5 * tl.min(tr); + let c = |e: f32, t: f32| (e / t).max(1.0).log2(); + let use_ms = self.cfg.ms_stereo && c(em, tm) + c(es, tm) < c(el, tl) + c(er, tr); + used.push(use_ms); + self.stats.stereo_bands += 1; + if use_ms { + self.stats.ms_bands += 1; + thrs[0][g][sfb] = tm; + thrs[1][g][sfb] = tm; + for w in w0..w0 + gl { + for k in w * 128 + swb[sfb] as usize..w * 128 + swb[sfb + 1] as usize { + let (a, b) = (l[k], r[k]); + l[k] = 0.5 * (a + b); + r[k] = 0.5 * (a - b); + } + } + } + } + } + let n_ms = used.iter().filter(|&&u| u).count(); + let present = if n_ms == 0 { + 0 + } else if n_ms == used.len() { + 2 + } else { + 1 + }; + ms_mask = Some((present, used)); + } + let chans = specs.iter().zip(thrs.iter()).zip(tnss).map(|((s, th), tn)| ChannelInput::new(info, swb, s, th, tn)).collect(); + let e = &mut self.elements[ei]; + e.prev_seq = seq; + e.short_next = Some(short_next); + if short { + self.stats.short_blocks += 1; + } + (ElementFrame { info, chans, ms_mask }, pe) + } + + fn element_overhead(&self, ei: usize, f: &ElementFrame) -> u32 { + let info_bits = if f.info.is_short() { 15 } else { 11 }; + match self.elements[ei].ty { + ElementType::Cpe => { + let ms = match &f.ms_mask { + Some((1, used)) => used.len() as u32, + _ => 0, + }; + 3 + 4 + 1 + info_bits + 2 + ms + } + _ => 3 + 4 + info_bits, + } + } + + fn quantise_all(frames: &[ElementFrame], lambda: f32) -> Vec> { + #[cfg(feature = "rayon")] + { + use rayon::prelude::*; + frames.par_iter().map(|f| f.chans.par_iter().map(|c| quantise_channel(c, lambda)).collect()).collect() + } + #[cfg(not(feature = "rayon"))] + { + frames.iter().map(|f| f.chans.iter().map(|c| quantise_channel(c, lambda)).collect()).collect() + } + } + + /// `last`: final frame of the stream — a CBR stream spends (or pads away) the whole reservoir so + /// the total size is exactly `frames × bitrate × 1024 / rate`. + fn encode_frame(&mut self, last: bool) -> Vec { + let t = self.frames_done as i64; + let mut frames = Vec::with_capacity(self.elements.len()); + let mut pe = 0f32; + for ei in 0..self.elements.len() { + let (f, p) = self.analyse_element(ei, t); + pe += p; + frames.push(f); + } + let pce_bits = if self.cfg.adts && self.asc.channel_config == 0 { + let mut bw = BitWriter::new(); + if let Some(p) = &self.asc.pce { + let _ = p.write(&mut bw); // its own program config, which fits + } + 3 + bw.bit_len() as u32 + 8 + } else { + 0 + }; + let overhead: u32 = (0..frames.len()).map(|i| self.element_overhead(i, &frames[i])).sum::() + 3 + 7 + pce_bits; + let total_bits = |q: &Vec>| overhead + q.iter().flatten().map(|c| c.bits).sum::(); + + let any_short = frames.iter().any(|f| f.info.is_short()); + let (mean, target) = match self.cfg.mode { + BitrateMode::Cbr(b) => { + self.mean_acc += b as u64 * FRAME_LEN as u64; + let mean = (self.mean_acc / self.cfg.sample_rate as u64) as i64; + self.mean_acc %= self.cfg.sample_rate as u64; + self.pe_avg = if self.pe_avg == 0.0 { pe.max(1.0) } else { 0.95 * self.pe_avg + 0.05 * pe.max(1.0) }; + let mut ratio = (pe.max(1.0) / self.pe_avg).clamp(0.7, 1.6); + if any_short { + ratio = (ratio * 1.3).min(2.0); + } + let mut desired = mean as f32 * ratio + (self.level as f32 - 0.5 * self.res_max as f32) * 0.15; + desired = desired.min((mean + self.level) as f32).max((mean - (self.res_max - self.level)) as f32); + desired = desired.min(self.max_frame_bits as f32); + if last { + desired = (mean + self.level) as f32; + } + (mean, desired.max(0.0) as u32) + } + BitrateMode::Vbr(_) => (0, self.max_frame_bits as u32), + }; + + let quant = match self.cfg.mode { + BitrateMode::Vbr(q) => { + let d = q.clamp(0.5, 6.0) - 1.0; + let mut lambda = 10f32.powf(-0.2 - 1.2 * d + 0.09 * d * d); + let mut res = Self::quantise_all(&frames, lambda); + while total_bits(&res) > target && lambda < 1e9 { + lambda *= 2.0; + res = Self::quantise_all(&frames, lambda); + } + res + } + BitrateMode::Cbr(_) => { + let mut lam = self.lambda.clamp(1e-6, 1e9); + let mut cur = Self::quantise_all(&frames, lam); + let (mut lo, mut hi, mut best); + if total_bits(&cur) <= target { + hi = lam; + best = cur; + lo = lam; + loop { + lo /= 4.0; + if lo < 1e-6 { + break; + } + cur = Self::quantise_all(&frames, lo); + if total_bits(&cur) > target { + break; + } + hi = lo; + best = cur; + } + } else { + lo = lam; + loop { + lam *= 4.0; + cur = Self::quantise_all(&frames, lam); + if total_bits(&cur) <= target || lam > 1e9 { + break; + } + lo = lam; + } + hi = lam; + best = cur; + } + for _ in 0..6 { + if hi / lo < 1.03 { + break; + } + let mid = (lo * hi).sqrt(); + let r = Self::quantise_all(&frames, mid); + if total_bits(&r) <= target { + hi = mid; + best = r; + } else { + lo = mid; + } + } + self.lambda = hi; + best + } + }; + + // assemble the raw_data_block + let mut bw = BitWriter::new(); + if pce_bits > 0 + && let Some(p) = &self.asc.pce + { + bw.write_bits(5, 3); + let _ = p.write(&mut bw); // its own program config, which fits + } + for (ei, (f, q)) in frames.iter().zip(&quant).enumerate() { + let e = &self.elements[ei]; + match e.ty { + ElementType::Cpe => { + bw.write_bits(1, 3); + bw.write_bits(e.tag as u32, 4); + bw.write_bits(1, 1); // common_window + f.info.write(&mut bw); + let (present, used) = f.ms_mask.clone().unwrap_or((0, Vec::new())); + bw.write_bits(present, 2); + if present == 1 { + for &u in &used { + bw.write_bits(u as u32, 1); + } + } + write_ics(&mut bw, &f.chans[0], &q[0], false); + write_ics(&mut bw, &f.chans[1], &q[1], false); + } + ty => { + bw.write_bits(if ty == ElementType::Lfe { 3 } else { 0 }, 3); + bw.write_bits(e.tag as u32, 4); + write_ics(&mut bw, &f.chans[0], &q[0], true); + } + } + } + if let BitrateMode::Cbr(_) = self.cfg.mode { + // Fill up to the bits this unit should end at, never past `max_frame_bits`, rounded down + // to a whole byte since units are whole bytes. `write_fill` ends at most 6 bits short and + // never over, so the byte-aligned unit ends exactly there. The 0-7 bits rounded off stay + // in the reservoir; after the last frame they are left unspent. + let target_end = (self.level + mean - if last { 0 } else { self.res_max }).min(self.max_frame_bits) / 8 * 8; + let excess = target_end - (bw.bit_len() as i64 + 3); + if excess > 0 { + self.stats.fill_bits += write_fill(&mut bw, excess as u64); + } + } + bw.write_bits(7, 3); // ID_END + let au = bw.finish(); + let used = au.len() as i64 * 8; + if let BitrateMode::Cbr(_) = self.cfg.mode { + self.level = (self.level + mean - used).clamp(0, self.res_max); + } + self.stats.frames += 1; + self.stats.bytes += au.len() as u64; + + // slide the input window + for h in &mut self.hist { + h.drain(..FRAME_LEN); + } + self.origin += FRAME_LEN as i64; + self.frames_done += 1; + + if self.cfg.adts { + let fullness = match self.cfg.mode { + BitrateMode::Cbr(_) => ((self.level / (32 * self.cfg.channels as i64)) as u16).min(0x7FE), + BitrateMode::Vbr(_) => 0x7FF, + }; + // The header holds everything the encoder writes: channel configurations 0-7, rates from + // the table, and frames of at most 6144 bits per channel, far below its length limit. + let Ok(header) = AdtsHeader::write(2, self.sf_index, self.asc.channel_config, au.len(), fullness) else { return au }; + [&header[..], &au].concat() + } else { + au + } + } +} + +/// Write fill elements totalling between `bits - 6` and `bits` bits; returns bits written. +fn write_fill(bw: &mut BitWriter, bits: u64) -> u64 { + let start = bw.bit_len() as u64; + let mut left = bits as i64; + while left >= 7 { + // An element of n payload bytes is 7 + 8n bits for n < 15, and 15 + 8n with the escape + // count. Each element is at most `left` bits, so the total never passes `bits`. + let n = if left >= 15 + 8 * 15 { (((left - 15) / 8) as usize).min(269) } else { (((left - 7) / 8) as usize).min(14) }; + bw.write_bits(6, 3); + if n >= 15 { + bw.write_bits(15, 4); + bw.write_bits((n - 14) as u32, 8); + } else { + bw.write_bits(n as u32, 4); + } + if n > 0 { + bw.write_bits(0, 8); // extension_type EXT_FILL + fill_nibble + for _ in 1..n { + bw.write_bits(0xA5, 8); + } + } + let el = 7 + if n >= 15 { 8 } else { 0 } + 8 * n as i64; + left -= el; + } + bw.bit_len() as u64 - start +} + +fn interp(pts: &[(f32, f32)], x: f32) -> f32 { + if x <= pts[0].0 { + return pts[0].1; + } + for w in pts.windows(2) { + if x <= w[1].0 { + let t = (x - w[0].0) / (w[1].0 - w[0].0); + return w[0].1 + t * (w[1].1 - w[0].1); + } + } + pts[pts.len() - 1].1 +} + +#[cfg(test)] +mod tests { + use super::*; + + fn noise(n: usize) -> Vec { + let mut s = 7u32; + (0..n) + .map(|_| { + s = s.wrapping_mul(1664525).wrapping_add(1013904223); + ((s >> 8) as f32 / (1 << 24) as f32 - 0.5) * 1.6 + }) + .collect() + } + + fn encode_all(cfg: EncoderConfig, input: &[f32]) -> Vec> { + let ch = cfg.channels; + let Ok(mut e) = Encoder::new(cfg) else { panic!("encoder settings rejected") }; + let chans: Vec<&[f32]> = (0..ch).map(|_| input).collect(); + let mut units = e.encode(&chans); + units.extend(e.flush()); + units + } + + #[test] + fn cbr_bitrate_limits_use_the_exact_bits_per_frame() { + // 288000 b/s at 48 kHz is exactly 6144 bits per frame; one more cannot fit a mono frame. + assert!(Encoder::new(EncoderConfig::cbr(48_000, 1, 288_000)).is_ok()); + assert!(Encoder::new(EncoderConfig::cbr(48_000, 1, 288_001)).is_err()); + // 4500 b/s at 96 kHz is exactly 48 bits per frame, the size of a silent mono frame. + assert!(Encoder::new(EncoderConfig::cbr(96_000, 1, 4_500)).is_ok()); + assert!(Encoder::new(EncoderConfig::cbr(96_000, 1, 4_499)).is_err()); + assert!(Encoder::new(EncoderConfig::cbr(96_000, 2, 2_000)).is_err()); + } + + #[test] + fn cbr_access_units_fit_the_decoder_buffer() { + let sil = vec![0f32; 48_000]; + let nz = noise(48_000); + for (rate, ch, bps) in [(48_000, 1, 288_000), (48_000, 2, 576_000), (44_100, 1, 264_600), (8_000, 1, 1_000)] { + for input in [&sil, &nz] { + let units = encode_all(EncoderConfig::cbr(rate, ch, bps), input); + let max = units.iter().map(Vec::len).max().unwrap_or(0); + assert!(max <= 768 * ch, "{rate} Hz, {ch} ch, {bps} b/s: {max}-byte access unit"); + } + } + } + + #[test] + fn the_lowest_accepted_cbr_rate_holds_a_silent_frame() { + // The full bandwidth makes the largest silent frames; ADTS adds the program config to every + // 7-channel frame. + let sil = vec![0f32; 96_000]; + for ch in 1..=8 { + for adts in [false, true] { + let cfg = |b| EncoderConfig::cbr(96_000, ch, b).with_adts(adts).with_bandwidth(Some(47_040)); + let Some(b) = (1000..20_000).map(|k| k * 10).find(|&b| Encoder::new(cfg(b)).is_ok()) else { panic!("no rate accepted") }; + let budget = b as usize * FRAME_LEN / 96_000; + let header = if adts { 56 } else { 0 }; + let units = encode_all(cfg(b), &sil); + let max = units[1..units.len() - 1].iter().map(|u| u.len() * 8 - header).max().unwrap_or(0); + assert!(max <= budget, "{ch} ch, ADTS {adts}: {b} b/s accepted, {budget} bits per frame, silent frames take {max}"); + } + } + } + + #[test] + fn maximum_rate_frames_are_filled_to_the_limit() { + let sil = vec![0f32; 48_000]; + for bandwidth in [None, Some(1000)] { + let units = encode_all(EncoderConfig::cbr(48_000, 1, 288_000).with_bandwidth(bandwidth), &sil); + for u in &units[..units.len() - 1] { + assert_eq!(u.len(), 768, "bandwidth {bandwidth:?}"); + } + } + } + + #[test] + fn cbr_fill_never_spends_more_than_the_reservoir_holds() { + // Encode frame by frame, as `flush` does, so the bits each frame may spend are known: the + // reservoir level plus this frame's share of the rate. + for (rate, ch, bps) in [(48_000, 1, 5_907), (48_000, 2, 96_011), (8_000, 1, 7_919), (44_100, 2, 128_000)] { + for n in [1, 1500, 10_000] { + let Ok(mut e) = Encoder::new(EncoderConfig::cbr(rate, ch, bps)) else { panic!("settings rejected") }; + // Silent input, buffered without encoding: `encode` would code the frames it can. + for h in &mut e.hist { + h.extend(std::iter::repeat_n(0.0, n)); + } + e.total_in = n as u64; + let frames = (n as u64).div_ceil(FRAME_LEN as u64) + 1; + while e.frames_done < frames { + let (need, have) = (e.needed_for(e.frames_done), e.available()); + for h in &mut e.hist { + h.extend(std::iter::repeat_n(0.0, (need - have).max(0) as usize)); + } + let last = e.frames_done + 1 == frames; + let mean = ((e.mean_acc + bps as u64 * FRAME_LEN as u64) / rate as u64) as i64; + let available = e.level + mean; + let used = e.encode_frame(last).len() as i64 * 8; + let at = format!("{rate} Hz, {ch} ch, {bps} b/s, {n} samples, frame {}", e.frames_done); + assert!(used <= available, "{at}: {used} bits used, {available} available"); + if last { + assert!(used >= available.min(e.max_frame_bits) / 8 * 8, "{at}: last frame {used} bits, {available} available"); + } + } + } + } + } + + #[test] + fn fill_never_passes_the_request() { + for bits in 0..3000u64 { + let mut bw = BitWriter::new(); + let n = write_fill(&mut bw, bits); + assert_eq!(n, bw.bit_len() as u64); + assert!(n <= bits && n + 6 >= bits, "asked {bits}, wrote {n}"); + } + } + + #[test] + fn vbr_quality_must_be_a_number() { + assert!(Encoder::new(EncoderConfig::vbr(48_000, 2, f32::NAN)).is_err()); + // Infinities and other out-of-range values are clamped: they encode exactly as the nearest + // end of the range, within the frame limit. + let nz = noise(48_000); + for (q, end) in [(f32::INFINITY, 6.0), (f32::NEG_INFINITY, 0.5), (100.0, 6.0), (-3.0, 0.5)] { + let units = encode_all(EncoderConfig::vbr(48_000, 8, q).with_adts(true), &nz); + assert!(units == encode_all(EncoderConfig::vbr(48_000, 8, end).with_adts(true), &nz), "quality {q}"); + for u in &units { + assert!(u.len() <= 6144 + 7 && u.starts_with(&[0xFF]), "quality {q}: {}-byte unit", u.len()); + } + } + } + + #[test] + fn one_long_encode_call_buffers_one_frame_at_a_time() { + let input = noise(44_100 * 2); + let Ok(mut whole) = Encoder::new(EncoderConfig::cbr(44_100, 1, 128_000)) else { panic!("settings rejected") }; + let mut a = whole.encode(&[&input]); + assert!(whole.hist[0].capacity() < 16 * FRAME_LEN, "history grew to {}", whole.hist[0].capacity()); + a.extend(whole.flush()); + let Ok(mut parts) = Encoder::new(EncoderConfig::cbr(44_100, 1, 128_000)) else { panic!("settings rejected") }; + let mut b = Vec::new(); + for chunk in input.chunks(1000) { + b.extend(parts.encode(&[chunk])); + } + b.extend(parts.flush()); + assert_eq!(a, b); + } +} diff --git a/crates/audio/src/aac/encoder/psy.rs b/crates/audio/src/aac/encoder/psy.rs new file mode 100644 index 00000000..04432a17 --- /dev/null +++ b/crates/audio/src/aac/encoder/psy.rs @@ -0,0 +1,176 @@ +//! Psychoacoustic model operating on MDCT coefficients. +//! +//! Per scalefactor band: energy; tonality from the spectral flatness measure (SFM) of a +//! neighbourhood of ≥ 16 lines; masking-threshold PSD by spreading the band PSDs with Schroeder's +//! spreading function in the Bark domain; a tonality-dependent masking offset (≈18 dB for tonal, +//! ≈6 dB for noise-like maskers); the absolute threshold of hearing (Terhardt) as a floor; and +//! perceptual entropy (PE) as an estimate of the bits the frame needs. + +use crate::aac::tables::{swb_offsets_long, swb_offsets_short}; + +/// Precomputed per-band data for one window length. +pub struct BandModel { + pub offsets: &'static [u16], + /// Linear spreading factor `[maskee][masker]` (power). + spread: Vec>, + /// ATH as band energy in encoder spectrum units. + ath: Vec, + /// Neighbourhood `[lo, hi)` in lines used for the tonality estimate. + tonal_span: Vec<(usize, usize)>, +} + +fn bark(f: f64) -> f64 { + 13.0 * (0.00076 * f).atan() + 3.5 * (f / 7500.0).powi(2).atan() +} + +/// Terhardt's threshold in quiet (dB SPL), capped at 40 dB so content near the band edge is not +/// discarded wholesale (the bandwidth setting limits the coded range instead). +fn ath_db(f: f64) -> f64 { + let f = (f / 1000.0).max(0.02); + (3.64 * f.powf(-0.8) - 6.5 * (-0.6 * (f - 3.3).powi(2)).exp() + 1e-3 * f.powi(4)).min(40.0) +} + +impl BandModel { + fn new(sample_rate: u32, offsets: &'static [u16], m: usize) -> BandModel { + let nb = offsets.len() - 1; + let hz_per_line = sample_rate as f64 / (2.0 * m as f64); + let centre: Vec = (0..nb).map(|b| bark((offsets[b] as f64 + offsets[b + 1] as f64) * 0.5 * hz_per_line)).collect(); + let spread = (0..nb) + .map(|i| { + (0..nb) + .map(|j| { + let dz = centre[i] - centre[j] + 0.474; + let db = 15.81 + 7.5 * dz - 17.5 * (1.0 + dz * dz).sqrt(); + if db < -60.0 { 0.0 } else { 10f64.powf(db / 10.0) as f32 } + }) + .collect() + }) + .collect(); + // A full-scale (16-bit) sine of power P_ref = 32768²/2 has MDCT energy ≈ 2·M²·P per frame + // with the encoder's scaling; it is taken as 96 dB SPL. 3 dB of safety margin. + let p_ref = 32768f64 * 32768.0 / 2.0; + let ath = (0..nb) + .map(|b| { + let mut lo = f64::MAX; + for k in offsets[b]..offsets[b + 1] { + lo = lo.min(ath_db((k as f64 + 0.5) * hz_per_line)); + } + (2.0 * (m * m) as f64 * p_ref * 10f64.powf((lo - 96.0 - 3.0) / 10.0)) as f32 + }) + .collect(); + let tonal_span = (0..nb) + .map(|b| { + let (lo, hi) = (offsets[b] as usize, offsets[b + 1] as usize); + if hi - lo >= 16 { + (lo, hi) + } else { + let c = (lo + hi) / 2; + let lo = c.saturating_sub(8); + let hi = (lo + 16).min(m); + (hi.saturating_sub(16), hi) + } + }) + .collect(); + BandModel { offsets, spread, ath, tonal_span } + } + + pub fn num_bands(&self) -> usize { + self.offsets.len() - 1 + } +} + +pub struct Psy { + pub long: BandModel, + pub short: BandModel, +} + +/// Per-window, per-band analysis of one channel. +#[derive(Clone, Default)] +pub struct PsyResult { + /// `[window][band]` energy. + pub en: Vec>, + /// `[window][band]` masking threshold (allowed noise energy). + pub thr: Vec>, + pub pe: f32, +} + +impl Psy { + pub fn new(sample_rate: u32, sf_index: u8) -> Psy { + Psy { + long: BandModel::new(sample_rate, swb_offsets_long(sf_index), 1024), + short: BandModel::new(sample_rate, swb_offsets_short(sf_index), 128), + } + } + + /// Analyse a spectrum: 1024 lines (long) or 8×128 (short). + pub fn analyse(&self, spec: &[f32], short: bool) -> PsyResult { + let (model, nwin, wlen) = if short { (&self.short, 8, 128) } else { (&self.long, 1, 1024) }; + let nb = model.num_bands(); + let mut res = PsyResult { en: Vec::with_capacity(nwin), thr: Vec::with_capacity(nwin), pe: 0.0 }; + let mut psd = vec![0f32; nb]; + let mut tonality = vec![0f32; nb]; + for w in 0..nwin { + let s = &spec[w * wlen..(w + 1) * wlen]; + let mut en = vec![0f32; nb]; + for b in 0..nb { + let (lo, hi) = (model.offsets[b] as usize, model.offsets[b + 1] as usize); + en[b] = s[lo..hi].iter().map(|v| v * v).sum(); + psd[b] = en[b] / (hi - lo) as f32; + // spectral flatness over the neighbourhood + let (tlo, thi) = model.tonal_span[b]; + let n = (thi - tlo) as f64; + let mut sum = 0f64; + let mut log_sum = 0f64; + for &v in &s[tlo..thi] { + let p = v as f64 * v as f64 + 1e-3; + sum += p; + log_sum += p.ln(); + } + let sfm_db = 10.0 * ((log_sum / n).exp() / (sum / n)).log10(); + // real-valued Gaussian noise gives ≈ -5.5 dB; a pure tone ≤ -25 dB + tonality[b] = ((-sfm_db - 6.0) / 19.0).clamp(0.0, 1.0) as f32; + } + let mut thr = vec![0f32; nb]; + for b in 0..nb { + let row = &model.spread[b]; + let mut sp = 0f32; + for j in 0..nb { + sp += psd[j] * row[j]; + } + let width = (model.offsets[b + 1] - model.offsets[b]) as f32; + let offset_db = tonality[b] * 30.0 + (1.0 - tonality[b]) * 6.0; + let t = sp * width * 10f32.powf(-offset_db / 10.0); + thr[b] = t.max(model.ath[b]); + if en[b] > thr[b] { + res.pe += width * 0.5 * (en[b] / thr[b]).log2(); + } + } + res.en.push(en); + res.thr.push(thr); + } + res + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn tone_is_masked_less_than_noise() { + let psy = Psy::new(44100, 4); + let tone: Vec = (0..1024).map(|k| if k == 100 { 1e6 } else { 0.0 }).collect(); + let mut seed = 1u32; + let noise: Vec = (0..1024) + .map(|_| { + seed = seed.wrapping_mul(1664525).wrapping_add(1013904223); + ((seed >> 8) as f32 / (1 << 24) as f32 - 0.5) * 1e4 + }) + .collect(); + let rt = psy.analyse(&tone, false); + let rn = psy.analyse(&noise, false); + let b = psy.long.offsets.iter().position(|&o| o > 100).unwrap() - 1; + assert!(rt.thr[0][b] < rt.en[0][b] / 30.0); + assert!(rn.thr[0][b] > rn.en[0][b] / 10.0); + } +} diff --git a/crates/audio/src/aac/encoder/quant.rs b/crates/audio/src/aac/encoder/quant.rs new file mode 100644 index 00000000..5ca742d2 --- /dev/null +++ b/crates/audio/src/aac/encoder/quant.rs @@ -0,0 +1,383 @@ +//! Quantisation, scalefactor selection, codebook/section selection, bit counting and writing of one +//! `individual_channel_stream`. +//! +//! Coefficients are kept in bitstream order ("interleaved": group → band → window → line), so every +//! (group, band) is a contiguous slice and short-window grouping needs no special casing. + +use craft_media::bitstream::BitWriter; + +use crate::aac::huffman::{self, ZERO_HCB}; +use crate::aac::ics::IcsInfo; +use crate::aac::tables::pow43; +use crate::aac::tns::TnsData; + +/// Magic rounding offset for AAC quantisation (`0.5 - 0.0946`), minimising the error for the +/// `x^(3/4)` companding. +const ROUND: f32 = 0.4054; +const MAX_Q: i32 = 8191; + +/// One (group, band) of a channel, in bitstream order. +#[derive(Clone, Copy, Debug)] +pub struct Band { + /// Range in the interleaved coefficient array. + pub start: usize, + pub end: usize, + /// Allowed noise energy at λ = 1. + pub thr: f32, + /// Coefficient energy. + pub en: f32, + /// Largest `|x|^(3/4)`. + pub max34: f32, +} + +/// Everything the rate loop needs for one channel. +pub struct ChannelInput { + pub info: IcsInfo, + /// Interleaved (bitstream-order) coefficients. + pub x: Vec, + pub xabs: Vec, + pub x34: Vec, + pub bands: Vec, + pub tns: Option, +} + +impl ChannelInput { + /// `spec` is window-major (short windows at stride 128); `thr[g][sfb]` and the grouping come from + /// `info`. + pub fn new(info: IcsInfo, swb: &[u16], spec: &[f32], thr: &[Vec], tns: Option) -> ChannelInput { + let mut x = Vec::with_capacity(1024); + let mut bands = Vec::new(); + for g in 0..info.num_groups { + let w0 = info.group_start(g); + for sfb in 0..info.max_sfb { + let start = x.len(); + for w in w0..w0 + info.group_len[g] as usize { + x.extend_from_slice(&spec[w * 128 + swb[sfb] as usize..w * 128 + swb[sfb + 1] as usize]); + } + bands.push(Band { start, end: x.len(), thr: thr[g][sfb], en: 0.0, max34: 0.0 }); + } + } + let xabs: Vec = x.iter().map(|v| v.abs()).collect(); + let x34: Vec = xabs.iter().map(|v| v.powf(0.75)).collect(); + for b in &mut bands { + b.en = x[b.start..b.end].iter().map(|v| v * v).sum(); + b.max34 = x34[b.start..b.end].iter().cloned().fold(0.0, f32::max); + } + ChannelInput { info, x, xabs, x34, bands, tns } + } +} + +/// A quantised channel ready to be written. +#[derive(Clone, Default)] +pub struct QuantChannel { + pub q: Vec, + /// Scalefactor per band (only meaningful where `cb != 0`). + pub sf: Vec, + pub cb: Vec, + /// Per group: (codebook, number of bands). + pub sections: Vec>, + pub global_gain: i32, + /// Bits of the ICS excluding `ics_info`. + pub bits: u32, +} + +#[inline] +fn gain34(sf: i32) -> f32 { + (-0.1875 * (sf as f32 - 100.0)).exp2() +} +#[inline] +fn gain(sf: i32) -> f32 { + (0.25 * (sf as f32 - 100.0)).exp2() +} + +/// Quantise a slice at `sf`, returning the distortion; writes the (signed) values to `out` if given. +fn quantise(x: &[f32], xabs: &[f32], x34: &[f32], sf: i32, out: Option<&mut [i32]>) -> f32 { + let g34 = gain34(sf); + let g = gain(sf); + let mut d = 0f32; + match out { + Some(out) => { + for i in 0..x.len() { + let q = ((x34[i] * g34 + ROUND) as i32).min(MAX_Q); + let e = xabs[i] - pow43(q as u32) * g; + d += e * e; + out[i] = if x[i] < 0.0 { -q } else { q }; + } + } + None => { + for i in 0..x.len() { + let q = ((x34[i] * g34 + ROUND) as i32).min(MAX_Q); + let e = xabs[i] - pow43(q as u32) * g; + d += e * e; + } + } + } + d +} + +/// Largest scalefactor whose distortion stays within `allowed`. +fn search_sf(ch: &ChannelInput, b: &Band, allowed: f32) -> i32 { + let r = b.start..b.end; + let (x, xa, x34) = (&ch.x[r.clone()], &ch.xabs[r.clone()], &ch.x34[r]); + let l2 = b.max34.log2(); + let lo = ((100.0 - ((MAX_Q as f32 + 0.5).log2() - l2) / 0.1875).ceil() as i32).clamp(0, 255); + let zero = ((100.0 - ((1.0 - ROUND).log2() - l2) / 0.1875).floor() as i32 + 1).clamp(0, 256); + let mut a = lo; + let mut hi = (zero - 1).clamp(lo, 255); + if quantise(x, xa, x34, a, None) > allowed { + return a; + } + while a < hi { + let mid = (a + hi + 1) / 2; + if quantise(x, xa, x34, mid, None) <= allowed { + a = mid; + } else { + hi = mid - 1; + } + } + a +} + +/// Candidate codebooks for a band whose largest magnitude is `m` (0 for all-zero bands). +fn candidates(m: i32) -> &'static [u8] { + match m { + 0 => &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], + 1 => &[1, 2, 3, 4, 5, 6], + 2 => &[3, 4, 5, 6, 7, 8], + 3..=4 => &[5, 6, 7, 8, 9, 10], + 5..=7 => &[7, 8, 9, 10, 11], + 8..=12 => &[9, 10, 11], + _ => &[11], + } +} + +/// Quantise one channel at rate-distortion multiplier `lambda` and select sections. +#[allow(clippy::cognitive_complexity, reason = "the AAC rate/distortion loop, ported as-is from FilmCraft")] +pub fn quantise_channel(ch: &ChannelInput, lambda: f32) -> QuantChannel { + let nb = ch.bands.len(); + let mut sf = vec![-1i32; nb]; + for (i, b) in ch.bands.iter().enumerate() { + let allowed = lambda * b.thr; + if b.max34 <= 0.0 || b.en <= allowed { + continue; + } + sf[i] = search_sf(ch, b, allowed); + } + // Scalefactor differences between coded bands must stay within ±60: lower (refine) offenders, + // then quantise; repeat if a band unexpectedly quantised to all zeros (it leaves the chain). + let mut q = vec![0i32; ch.x.len()]; + let mut maxq = vec![0i32; nb]; + loop { + for _ in 0..8 { + let mut changed = false; + let coded: Vec = (0..nb).filter(|&i| sf[i] >= 0).collect(); + for w in coded.windows(2) { + if sf[w[1]] > sf[w[0]] + 60 { + sf[w[1]] = sf[w[0]] + 60; + changed = true; + } + } + for w in coded.windows(2).rev() { + if sf[w[0]] > sf[w[1]] + 60 { + sf[w[0]] = sf[w[1]] + 60; + changed = true; + } + } + if !changed { + break; + } + } + let mut removed = false; + for (i, b) in ch.bands.iter().enumerate() { + let r = b.start..b.end; + if sf[i] < 0 { + q[r].fill(0); + maxq[i] = 0; + continue; + } + quantise(&ch.x[r.clone()], &ch.xabs[r.clone()], &ch.x34[r.clone()], sf[i], Some(&mut q[r.clone()])); + maxq[i] = q[r].iter().map(|v| v.abs()).max().unwrap_or(0); + if maxq[i] == 0 { + sf[i] = -1; + removed = true; + } + } + if !removed { + break; + } + } + + // Section selection: dynamic programming per group over codebooks. + let info = &ch.info; + let (sect_bits, esc) = if info.is_short() { (3u32, 7usize) } else { (5u32, 31usize) }; + let mut cb = vec![ZERO_HCB; nb]; + let mut sections = Vec::with_capacity(info.num_groups); + let mut spectral_bits = 0u32; + let mut section_bits = 0u32; + let per_group = info.max_sfb; + for g in 0..info.num_groups { + let base = g * per_group; + let n = per_group; + if n == 0 { + sections.push(Vec::new()); + continue; + } + // cost[i][c] + const INF: u32 = u32::MAX / 4; + let mut cost = vec![[INF; 12]; n]; + for i in 0..n { + let bi = base + i; + let b = &ch.bands[bi]; + let vals = &q[b.start..b.end]; + for &c in candidates(maxq[bi]) { + cost[i][c as usize] = if c == 0 { + 0 + } else { + // all-zero band inside a coded section also needs a (zero) scalefactor difference + huffman::count_bits(c, vals) + if maxq[bi] == 0 { 1 } else { 0 } + }; + } + } + let new_sec = 4 + sect_bits; + let mut best = [INF; 12]; + let mut back = vec![[0u8; 12]; n]; + for c in 0..12 { + if cost[0][c] < INF { + best[c] = new_sec + cost[0][c]; + } + } + for i in 1..n { + let (mut min_c, mut min_v) = (0usize, INF); + for c in 0..12 { + if best[c] < min_v { + min_v = best[c]; + min_c = c; + } + } + let mut nb_ = [INF; 12]; + for c in 0..12 { + if cost[i][c] >= INF { + continue; + } + let stay = best[c]; + let switch = min_v + new_sec; + if stay <= switch { + nb_[c] = stay + cost[i][c]; + back[i][c] = c as u8; + } else { + nb_[c] = switch + cost[i][c]; + back[i][c] = min_c as u8; + } + } + best = nb_; + } + let mut c = (0..12).min_by_key(|&c| best[c]).unwrap_or(0); + for i in (0..n).rev() { + cb[base + i] = c as u8; + c = back[i][c] as usize; + } + // run-length sections and exact costs + let mut secs: Vec<(u8, usize)> = Vec::new(); + for i in 0..n { + let c = cb[base + i]; + match secs.last_mut() { + Some((lc, len)) if *lc == c => *len += 1, + _ => secs.push((c, 1)), + } + if c != ZERO_HCB { + let b = &ch.bands[base + i]; + spectral_bits += huffman::count_bits(c, &q[b.start..b.end]); + } + } + for &(_, len) in &secs { + section_bits += 4 + sect_bits * (len / esc + 1) as u32; + } + sections.push(secs); + } + + // Scalefactors: global gain is the first coded band's value; bands coded with a non-zero + // codebook but all-zero values reuse the previous value (difference 0). + let first = (0..nb).find(|&i| sf[i] >= 0).map(|i| sf[i]).unwrap_or(100); + let mut prev = first; + let mut sf_bits = 0u32; + for i in 0..nb { + if cb[i] == ZERO_HCB { + continue; + } + if sf[i] < 0 { + sf[i] = prev; + } + sf_bits += huffman::sf_bits(sf[i] - prev); + prev = sf[i]; + } + let tns_bits = 1 + ch.tns.as_ref().map(|t| t.bit_count(info)).unwrap_or(0); + let bits = 8 + section_bits + sf_bits + 1 + tns_bits + 1 + spectral_bits; + QuantChannel { q, sf, cb, sections, global_gain: first, bits } +} + +/// Write an `individual_channel_stream`; `write_info` is false inside a common-window CPE. +pub fn write_ics(bw: &mut BitWriter, ch: &ChannelInput, qc: &QuantChannel, write_info: bool) { + let info = &ch.info; + bw.write_bits(qc.global_gain as u32, 8); + if write_info { + info.write(bw); + } + let (sect_bits, esc) = if info.is_short() { (3u32, 7usize) } else { (5u32, 31usize) }; + for secs in &qc.sections { + for &(c, len) in secs { + bw.write_bits(c as u32, 4); + let mut l = len; + while l >= esc { + bw.write_bits(esc as u32, sect_bits); + l -= esc; + } + bw.write_bits(l as u32, sect_bits); + } + } + let mut prev = qc.global_gain; + for i in 0..ch.bands.len() { + if qc.cb[i] == ZERO_HCB { + continue; + } + huffman::write_sf(bw, qc.sf[i] - prev); + prev = qc.sf[i]; + } + bw.write_bits(0, 1); // pulse_data_present + match &ch.tns { + Some(t) => { + bw.write_bits(1, 1); + t.write(bw, info); + } + None => bw.write_bits(0, 1), + } + bw.write_bits(0, 1); // gain_control_data_present + for (i, b) in ch.bands.iter().enumerate() { + if qc.cb[i] != ZERO_HCB { + huffman::write_spectral(bw, qc.cb[i], &qc.q[b.start..b.end]); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::aac::ics::WindowSequence; + use crate::aac::mdct::WindowShape; + use crate::aac::tables::swb_offsets_long; + use craft_media::bitstream::BitWriter; + + #[test] + fn bit_count_matches_written_bits() { + let swb = swb_offsets_long(4); + let info = IcsInfo { window_sequence: WindowSequence::OnlyLong, window_shape: WindowShape::Sine, max_sfb: 40, ..Default::default() }; + let spec: Vec = (0..1024).map(|i| ((i * 7919 % 1000) as f32 - 500.0) * (1000.0 / (i as f32 + 10.0))).collect(); + let thr = vec![vec![1000.0f32; 64]]; + let ch = ChannelInput::new(info, swb, &spec, &thr, None); + for lambda in [0.01f32, 1.0, 100.0, 1e5] { + let qc = quantise_channel(&ch, lambda); + let mut bw = BitWriter::new(); + write_ics(&mut bw, &ch, &qc, false); + assert_eq!(bw.bit_len() as u32, qc.bits, "lambda {lambda}"); + } + } +} diff --git a/crates/audio/src/aac/mod.rs b/crates/audio/src/aac/mod.rs index 23aef936..eb1c7d40 100644 --- a/crates/audio/src/aac/mod.rs +++ b/crates/audio/src/aac/mod.rs @@ -1,8 +1,11 @@ -//! Clean-room AAC-LC decoder (ISO/IEC 14496-3, MPEG-4 Audio, Low Complexity profile). +//! Clean-room AAC-LC encoder and decoder (ISO/IEC 14496-3, MPEG-4 Audio, Low Complexity profile). //! -//! [`Decoder`] (feature `aac-decode`): `AudioSpecificConfig` + raw access units → planar f32. -//! Supports M/S, intensity stereo, PNS, TNS, pulse data, both window shapes and every window -//! sequence. +//! - [`Encoder`] (feature `aac-encode`): planar f32 → raw access units (+ +//! [`Encoder::audio_specific_config`] for MP4 `esds`, optional ADTS framing). Psychoacoustic +//! model, window switching, M/S, TNS, bit reservoir, CBR/VBR, 1–8 channels. +//! - [`Decoder`] (feature `aac-decode`): `AudioSpecificConfig` + raw access units → planar f32. +//! Supports M/S, intensity stereo, PNS, TNS, pulse data, both window shapes and every window +//! sequence. //! //! HE-AAC (SBR/PS), Main/SSR/LTP profiles, coupling channels and 960-sample frames are not //! supported; for HE-AAC only the AAC-LC core is decoded (at the core rate; [`Decoder::sbr`] says @@ -10,15 +13,38 @@ //! //! From FilmCraft's `filmcraft-aac` (storytold/filmcraft@8fda2f2). Public items keep their names, //! so FilmCraft can re-export this module. +//! +//! ``` +//! # #[cfg(all(feature = "aac-encode", feature = "aac-decode"))] { +//! use craft_audio::aac::{Decoder, Encoder, EncoderConfig}; +//! +//! let mut enc = Encoder::new(EncoderConfig::cbr(48_000, 2, 128_000)).unwrap(); +//! let left: Vec = (0..4800).map(|i| (i as f32 * 0.05).sin() * 0.5).collect(); +//! let right = left.clone(); +//! let mut aus = enc.encode(&[&left, &right]); // 0..n access units +//! aus.extend(enc.flush()); +//! let asc = enc.audio_specific_config(); // for the MP4 `esds` +//! assert_eq!(enc.priming_samples(), 1024); // MP4 edit list media_time +//! +//! let mut dec = Decoder::new(&asc).unwrap(); +//! for au in &aus { +//! let pcm = dec.decode(au).unwrap(); // planar, 1024 samples per channel +//! assert_eq!(pcm.len(), 2); +//! } +//! # } +//! ``` // Index loops follow the specification's pseudo-code (FilmCraft allows this lint workspace-wide). #![allow(clippy::needless_range_loop)] -// Tables, Huffman coding, MDCT, window and TNS code shared by the decoder and the encoder, as -// ported; the encoder's half is unused until the encoder is added. -#![allow(dead_code)] +// Tables, Huffman coding, MDCT, window and TNS code shared by the decoder and the encoder; each +// side uses only part of it. +#![cfg_attr(not(all(feature = "aac-decode", feature = "aac-encode")), allow(dead_code))] mod config; +#[cfg(feature = "aac-decode")] mod decoder; +#[cfg(feature = "aac-encode")] +mod encoder; mod huffman; mod huffman_tables; mod ics; @@ -29,20 +55,27 @@ mod tns; #[cfg(feature = "aac-decode")] pub(crate) use config::split_adts_upto; pub use config::{AdtsHeader, AudioSpecificConfig, ElementType, ProgramConfig, config_layout, split_adts}; +#[cfg(feature = "aac-decode")] pub use decoder::Decoder; +#[cfg(feature = "aac-encode")] +pub use encoder::{BitrateMode, Encoder, EncoderConfig, EncoderStats}; pub use mdct::WindowShape; pub use tables::{SAMPLE_RATES, sample_rate_index}; /// Samples per AAC-LC frame (per channel). pub const FRAME_LEN: usize = 1024; -/// Errors from configuration parsing and decoding. +/// Errors from configuration parsing, encoding and decoding. #[non_exhaustive] #[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] pub enum Error { /// The configuration is malformed or outside the supported range. #[error("invalid configuration: {0}")] InvalidConfig(&'static str), + /// [`EncoderConfig`] asks for something the encoder cannot do (a rate that is not an AAC + /// rate, too many channels, a bitrate out of range). + #[error("invalid encoder settings: {0}")] + InvalidSettings(&'static str), /// A valid stream uses a feature this crate does not implement. #[error("unsupported feature: {0}")] Unsupported(&'static str), @@ -80,6 +113,28 @@ pub fn wav_order(channel_config: u8) -> Option<&'static [usize]> { }) } +/// The order the [`Encoder`] takes its input in, from WAV order: for each encoder input channel +/// (AAC element order, see [`EncoderConfig::channels`]), the WAV-order channel to feed it. The +/// inverse of [`wav_order`] for 3 to 7 channels; `None` for mono and stereo (already in order) and +/// for 8 channels, whose AAC layout (configuration 7) has no WAV equivalent. +/// +/// ``` +/// let wav: Vec> = vec![vec![0.0; 1024]; 6]; // L R C LFE Ls Rs +/// let order = craft_audio::aac::aac_order(6).unwrap(); +/// let input: Vec<&[f32]> = order.iter().map(|&c| wav[c].as_slice()).collect(); // C L R Ls Rs LFE +/// assert_eq!(order, &[2, 0, 1, 4, 5, 3]); +/// ``` +pub fn aac_order(channels: usize) -> Option<&'static [usize]> { + Some(match channels { + 3 => &[2, 0, 1], + 4 => &[2, 0, 1, 3], + 5 => &[2, 0, 1, 3, 4], + 6 => &[2, 0, 1, 4, 5, 3], + 7 => &[2, 0, 1, 5, 6, 4, 3], + _ => return None, + }) +} + /// The output rate of an AAC stream: twice the core rate for HE-AAC (ISO/IEC 14496-3 §1.6.5). /// Explicit signalling names the rate in the `AudioSpecificConfig`; with implicit signalling (an /// AAC-LC config at 24 kHz or less) SBR is only visible as fill-element data in the access units, @@ -219,3 +274,22 @@ mod tests { assert!(up.process(vec![vec![0.0; 600]]).remove(0).iter().all(|&s| s == 0.0)); } } + +#[cfg(test)] +mod order_tests { + use super::*; + + /// `aac_order` undoes `wav_order` for each channel count the encoder maps to a configuration + /// (3–6 channels: configurations 3–6; 7: configuration 11). + #[test] + fn aac_order_inverts_wav_order() { + for (channels, config) in [(3, 3), (4, 4), (5, 5), (6, 6), (7, 11)] { + let (to_wav, to_aac) = (wav_order(config).unwrap(), aac_order(channels).unwrap()); + for (wav_channel, &aac_channel) in to_wav.iter().enumerate() { + assert_eq!(to_aac[aac_channel], wav_channel, "{channels} channels"); + } + } + assert_eq!(aac_order(2), None); + assert_eq!(aac_order(8), None); + } +} diff --git a/crates/audio/src/lib.rs b/crates/audio/src/lib.rs index 18ff0606..3df3da3e 100644 --- a/crates/audio/src/lib.rs +++ b/crates/audio/src/lib.rs @@ -17,14 +17,15 @@ //! `f32` in WAV channel order. //! - [`encode`], [`FileEncoder`]: WAV/BWF/RF64 and AIFF/AIFF-C (feature `pcm-encode`, which also //! gives [`wav_float_header`]) and FLAC (feature `flac-encode`), whole-buffer or incremental into -//! any `Write + Seek`. Feature `lossless-encode` turns on both; feature `encode` is -//! the same as `lossless-encode`. +//! any `Write + Seek`. Feature `lossless-encode` turns on both; feature `encode` turns on +//! `lossless-encode` and `aac-encode`. //! - [`peaks`] (feature `peaks`): min/max/RMS pyramid for waveform drawing. -//! - Codec modules for packet-level use, each with its own features: [`aac`] (`aac-decode`). +//! - Codec modules for packet-level use, each with its own features: [`aac`] (`aac-decode`, +//! `aac-encode`). //! //! The shared types below build with no features. -#[cfg(feature = "aac-decode")] +#[cfg(any(feature = "aac-decode", feature = "aac-encode"))] pub mod aac; mod buffer; mod layout; @@ -87,11 +88,12 @@ pub type Result = std::result::Result; // use `?` throughout: a codec's "unsupported" stays `Unsupported`, anything else in the input is // `Malformed`, and an encoder's refusal is `Encode`. -#[cfg(feature = "aac-decode")] +#[cfg(any(feature = "aac-decode", feature = "aac-encode"))] impl From for AudioError { fn from(e: aac::Error) -> Self { match e { aac::Error::Unsupported(_) => AudioError::Unsupported(e.to_string()), + aac::Error::InvalidSettings(_) => AudioError::Encode(e.to_string()), _ => AudioError::Malformed(e.to_string()), } } @@ -345,3 +347,18 @@ impl Default for EncodeOptions { Self::new(EncodeFormat::Wav) } } + +// An encoder refusing its settings is an encode error, not damaged input. +#[cfg(all(test, feature = "aac-encode"))] +mod tests { + use super::*; + + #[test] + fn aac_encoder_settings_are_encode_errors() { + let e = match aac::Encoder::new(aac::EncoderConfig::cbr(44_000, 2, 128_000)) { + Err(e) => AudioError::from(e), + Ok(_) => panic!("44 kHz is not an AAC rate"), + }; + assert!(matches!(e, AudioError::Encode(_)), "{e}"); + } +} diff --git a/crates/audio/tests/codecs/aac/common.rs b/crates/audio/tests/codecs/aac/common.rs index d63fc074..1206c70b 100644 --- a/crates/audio/tests/codecs/aac/common.rs +++ b/crates/audio/tests/codecs/aac/common.rs @@ -3,7 +3,7 @@ use std::path::{Path, PathBuf}; use std::process::Command; -use craft_audio::aac::{Decoder, split_adts}; +use craft_audio::aac::{Decoder, Encoder, EncoderConfig, EncoderStats, split_adts}; /// ffmpeg (see `crate::testkit::oracle`); `None` after printing `SKIPPED`. pub fn ffmpeg() -> Option { @@ -119,6 +119,45 @@ pub fn clicks(rate: u32, n: usize) -> Vec { .collect() } +pub struct Encoded { + pub adts: Vec, + pub raw_bytes: usize, + pub frames: usize, + pub stats: EncoderStats, + pub asc: Vec, + pub bandwidth: u32, +} + +pub fn encode(cfg: EncoderConfig, planar: &[Vec]) -> Encoded { + let mut enc = Encoder::new(cfg.with_adts(true)).unwrap(); + let mut adts = Vec::new(); + // feed in uneven chunks + let n = planar[0].len(); + let mut pos = 0; + let mut step = 700; + while pos < n { + let end = (pos + step).min(n); + let chunk: Vec<&[f32]> = planar.iter().map(|c| &c[pos..end]).collect(); + for au in enc.encode(&chunk) { + adts.extend(au); + } + pos = end; + step = step * 7 % 3001 + 100; + } + for au in enc.flush() { + adts.extend(au); + } + let stats = enc.stats(); + Encoded { + raw_bytes: stats.bytes as usize, + frames: stats.frames as usize, + stats, + asc: enc.audio_specific_config(), + bandwidth: enc.bandwidth(), + adts, + } +} + /// Decode with ffmpeg → planar f32 (ffmpeg's own channel order), plus stderr. pub fn ffmpeg_decode(ff: &Path, file: &Path, channels: usize) -> (Vec>, String) { let out = Command::new(ff).args(["-v", "error", "-i"]).arg(file).args(["-f", "f32le", "-"]).output().unwrap(); diff --git a/crates/audio/tests/codecs/aac/encoder_oracle.rs b/crates/audio/tests/codecs/aac/encoder_oracle.rs new file mode 100644 index 00000000..99f14203 --- /dev/null +++ b/crates/audio/tests/codecs/aac/encoder_oracle.rs @@ -0,0 +1,273 @@ +//! Encoder quality/bitrate tests using ffmpeg as an external decoding oracle (skipped if absent). + +use super::common; + +use common::*; +use craft_audio::aac::{BitrateMode, EncoderConfig, WindowShape}; + +struct Case { + name: &'static str, + rate: u32, + signal: fn(u32, usize, usize) -> Vec>, + channels: usize, +} + +fn mono(v: Vec, ch: usize) -> Vec> { + (0..ch).map(|c| if c == 0 { v.clone() } else { v.iter().map(|x| x * (0.8 - 0.1 * c as f32)).collect() }).collect() +} + +fn sig_sweep(r: u32, n: usize, ch: usize) -> Vec> { + mono(sweep(r, n, 0.5), ch) +} +fn sig_noise(_r: u32, n: usize, ch: usize) -> Vec> { + (0..ch).map(|c| noise(n, 0.3, 7 + c as u64)).collect() +} +fn sig_speech(r: u32, n: usize, ch: usize) -> Vec> { + mono(speech(r, n, 3), ch) +} +fn sig_chords(r: u32, n: usize, ch: usize) -> Vec> { + (0..ch).map(|c| chords(r, n, 0.002 * c as f32)).collect() +} +fn sig_clicks(r: u32, n: usize, ch: usize) -> Vec> { + mono(clicks(r, n), ch) +} + +struct Row { + label: String, + target: u32, + actual: f64, + snr: f64, + bw_snr: f64, + bw: u32, + seg: f64, + peak: f32, + shorts: u64, + ours_vs_ff: f32, +} + +fn run(case: &Case, bitrate: u32, cfg_mod: impl Fn(&mut EncoderConfig)) -> Option { + let ff = ffmpeg()?; + let secs = 2.0; + let n = (case.rate as f32 * secs) as usize + 333; + let input = (case.signal)(case.rate, n, case.channels); + let mut cfg = EncoderConfig::cbr(case.rate, case.channels, bitrate); + cfg_mod(&mut cfg); + let enc = encode(cfg.clone(), &input); + let tag = match cfg.mode { + BitrateMode::Cbr(b) => format!("{b}"), + BitrateMode::Vbr(q) => format!("q{q}"), + }; + let path = fixtures().join(format!("enc_{}_{}ch_{}_{}.aac", case.name, case.channels, case.rate, tag)); + std::fs::write(&path, &enc.adts).unwrap(); + let (dec, err) = ffmpeg_decode(&ff, &path, case.channels); + assert!(err.trim().is_empty(), "ffmpeg reported errors for {}: {err}", path.display()); + assert_eq!(dec[0].len(), enc.frames * 1024); + assert!(dec[0].len() >= n + 1024, "decoded stream too short"); + // ffmpeg reorders channels to its native layouts: map each of our (AAC element order) decoded + // channels to the matching ffmpeg channel. + let ours = our_decode(&enc.adts); + let mut taken = vec![false; case.channels]; + let mut snr_sum = 0.0; + let mut seg_sum = 0.0; + let mut bw_sum = 0.0; + let mut peak = 0f32; + let mut ours_vs_ff = 0f32; + let mut counted = 0; + for c in 0..case.channels { + let (j, err) = + (0..case.channels).filter(|&j| !taken[j]).map(|j| (j, max_abs_diff(&ours[c], &dec[j]))).min_by(|a, b| a.1.total_cmp(&b.1)).unwrap(); + taken[j] = true; + ours_vs_ff = ours_vs_ff.max(err); + let d = &dec[j][1024..1024 + n]; + peak = peak.max(d.iter().fold(0f32, |m, v| m.max(v.abs()))); + if case.channels >= 6 && c == case.channels - 1 { + continue; // LFE is band-limited to 240 Hz by design + } + snr_sum += snr(&input[c], d); + let fc = enc.bandwidth as f32 * 0.97; + bw_sum += snr(&lowpass(&input[c], fc, case.rate as f32), &lowpass(d, fc, case.rate as f32)); + seg_sum += seg_snr(&input[c], d); + counted += 1; + } + let actual = enc.stats.bitrate(case.rate); + Some(Row { + label: format!("{:<7} {}ch {:>5}", case.name, case.channels, case.rate), + target: if matches!(cfg.mode, BitrateMode::Vbr(_)) { 0 } else { bitrate }, + actual, + snr: snr_sum / counted as f64, + bw_snr: bw_sum / counted as f64, + bw: enc.bandwidth, + seg: seg_sum / counted as f64, + peak, + shorts: enc.stats.short_blocks, + ours_vs_ff, + }) +} + +fn print(rows: &[Row]) { + println!( + "{:<22} {:>8} {:>8} {:>7} {:>6} {:>7} {:>7} {:>7} {:>6} {:>6} {:>9}", + "signal", "target", "actual", "dev%", "bw", "SNR", "bwSNR", "segSNR", "peak", "short", "us-vs-ff" + ); + for r in rows { + let (target, dev) = if r.target == 0 { + ("vbr".to_string(), "-".to_string()) + } else { + (r.target.to_string(), format!("{:+.2}", (r.actual / r.target as f64 - 1.0) * 100.0)) + }; + println!( + "{:<22} {:>8} {:>8.0} {:>7} {:>6} {:>7.2} {:>7.2} {:>7.2} {:>6.3} {:>6} {:>9.2e}", + r.label, target, r.actual, dev, r.bw, r.snr, r.bw_snr, r.seg, r.peak, r.shorts, r.ours_vs_ff + ); + } +} + +fn check(r: &Row, min_seg: f64) { + let dev = (r.actual / r.target as f64 - 1.0).abs(); + assert!(dev < 0.05, "{}: bitrate {} vs target {}", r.label, r.actual, r.target); + assert!(r.peak < 1.0, "{}: clipping (peak {})", r.label, r.peak); + assert!(r.seg > min_seg, "{}: segmental SNR {} too low", r.label, r.seg); + assert!(r.ours_vs_ff < 1e-3, "{}: our decoder differs from ffmpeg by {}", r.label, r.ours_vs_ff); +} + +#[test] +fn stereo_matrix_44k_48k() { + if ffmpeg().is_none() { + eprintln!("ffmpeg not found; skipping"); + return; + } + let cases = [ + Case { name: "sweep", rate: 44100, signal: sig_sweep, channels: 2 }, + Case { name: "noise", rate: 44100, signal: sig_noise, channels: 2 }, + Case { name: "speech", rate: 48000, signal: sig_speech, channels: 2 }, + Case { name: "chords", rate: 44100, signal: sig_chords, channels: 2 }, + Case { name: "clicks", rate: 48000, signal: sig_clicks, channels: 2 }, + ]; + let mut rows = Vec::new(); + for c in &cases { + for br in [64_000, 128_000, 192_000, 320_000] { + let r = run(c, br, |_| {}).unwrap(); + rows.push(r); + } + } + print(&rows); + for r in &rows { + check( + r, + if r.label.starts_with("noise") { + 0.5 + } else if r.target >= 128_000 { + 8.0 + } else { + 3.0 + }, + ); + } +} + +#[test] +fn mono_and_surround() { + if ffmpeg().is_none() { + eprintln!("ffmpeg not found; skipping"); + return; + } + let mut rows = Vec::new(); + for (c, br) in [ + (Case { name: "speech", rate: 44100, signal: sig_speech, channels: 1 }, 64_000), + (Case { name: "chords", rate: 48000, signal: sig_chords, channels: 1 }, 96_000), + (Case { name: "clicks", rate: 44100, signal: sig_clicks, channels: 1 }, 128_000), + (Case { name: "chords", rate: 48000, signal: sig_chords, channels: 6 }, 320_000), + (Case { name: "noise", rate: 48000, signal: sig_noise, channels: 6 }, 384_000), + ] { + rows.push(run(&c, br, |_| {}).unwrap()); + } + print(&rows); + for r in &rows { + check(r, 3.0); + } +} + +#[test] +fn kbd_windows_no_tns_no_ms() { + if ffmpeg().is_none() { + return; + } + let c = Case { name: "kbd", rate: 44100, signal: sig_clicks, channels: 2 }; + let r = run(&c, 128_000, |cfg| { + cfg.window_shape = WindowShape::Kbd; + cfg.tns = false; + cfg.ms_stereo = false; + }) + .unwrap(); + print(std::slice::from_ref(&r)); + check(&r, 5.0); +} + +#[test] +fn sample_rates_8k_to_96k() { + if ffmpeg().is_none() { + return; + } + let mut rows = Vec::new(); + for (rate, br) in [ + (8000u32, 24_000u32), + (11025, 32_000), + (12000, 32_000), + (16000, 48_000), + (22050, 64_000), + (24000, 64_000), + (32000, 96_000), + (64000, 192_000), + (88200, 256_000), + (96000, 256_000), + ] { + let c = Case { name: "chords", rate, signal: sig_chords, channels: 2 }; + rows.push(run(&c, br, |_| {}).unwrap()); + } + print(&rows); + for r in &rows { + check(r, 5.0); + } +} + +#[test] +fn channel_layouts_3_to_8() { + if ffmpeg().is_none() { + return; + } + let mut rows = Vec::new(); + for (ch, br) in [(3usize, 192_000u32), (4, 256_000), (5, 320_000), (7, 384_000), (8, 448_000)] { + let c = Case { name: "chords", rate: 48000, signal: sig_chords, channels: ch }; + rows.push(run(&c, br, |_| {}).unwrap()); + } + print(&rows); + for r in &rows { + check(r, 5.0); + } +} + +#[test] +fn vbr_quality_scale() { + if ffmpeg().is_none() { + return; + } + let mut rows = Vec::new(); + for sig in [sig_chords as fn(u32, usize, usize) -> Vec>, sig_speech] { + for q in [1.0f32, 2.0, 3.0, 4.0, 5.0] { + let c = Case { name: "vbr", rate: 44100, signal: sig, channels: 2 }; + let mut r = run(&c, 1, |cfg| cfg.mode = BitrateMode::Vbr(q)).unwrap(); + r.label = format!("vbr q{q} 2ch 44100"); + rows.push(r); + } + } + print(&rows); + for pair in rows.chunks(5) { + for w in pair.windows(2) { + assert!(w[1].actual > w[0].actual, "VBR bitrate must grow with quality"); + assert!(w[1].seg > w[0].seg - 0.5, "VBR quality must grow with quality"); + } + for r in pair { + assert!(r.peak < 1.0 && r.ours_vs_ff < 1e-3); + } + } +} diff --git a/crates/audio/tests/codecs/aac/mod.rs b/crates/audio/tests/codecs/aac/mod.rs index 1a992c77..45e62724 100644 --- a/crates/audio/tests/codecs/aac/mod.rs +++ b/crates/audio/tests/codecs/aac/mod.rs @@ -2,3 +2,5 @@ mod common; mod decoder_oracle; +mod encoder_oracle; +mod robustness; diff --git a/crates/audio/tests/codecs/aac/robustness.rs b/crates/audio/tests/codecs/aac/robustness.rs new file mode 100644 index 00000000..2f97f2ae --- /dev/null +++ b/crates/audio/tests/codecs/aac/robustness.rs @@ -0,0 +1,157 @@ +//! Fuzz-style robustness: mutated access units, configs and ADTS streams must never panic, and the +//! encoder must accept hostile input (NaN, ±inf, > full scale, silence, odd chunk sizes). + +use super::common; + +use common::*; +use craft_audio::aac::{AudioSpecificConfig, Decoder, Encoder, EncoderConfig, split_adts}; + +fn corpus() -> Vec<(Vec, Vec>)> { + let mut out = Vec::new(); + for (ch, rate, br) in [(1usize, 44100u32, 96_000u32), (2, 48000, 128_000), (6, 48000, 256_000), (7, 48000, 256_000)] { + let n = rate as usize / 2; + let sig: Vec> = (0..ch).map(|c| if c % 2 == 0 { clicks(rate, n) } else { chords(rate, n, 0.01) }).collect(); + let mut enc = Encoder::new(EncoderConfig::cbr(rate, ch, br)).unwrap(); + let refs: Vec<&[f32]> = sig.iter().map(|v| v.as_slice()).collect(); + let mut aus = enc.encode(&refs); + aus.extend(enc.flush()); + out.push((enc.audio_specific_config(), aus)); + } + out +} + +#[test] +fn mutated_access_units_never_panic() { + let mut rng = Rng(0x9E37_79B9_7F4A_7C15); + let mut rand = move |n: usize| ((rng.next() * 0.5 + 0.5) * n as f32) as usize % n.max(1); + let mut decoded = 0usize; + let mut errors = 0usize; + for (asc, aus) in corpus() { + let mut dec = Decoder::new(&asc).unwrap(); + for round in 0..3000 { + let mut au = aus[rand(aus.len())].clone(); + match round % 6 { + 0 => { + for _ in 0..1 + rand(8) { + let i = rand(au.len()); + au[i] ^= 1 << rand(8); + } + } + 1 => { + let i = rand(au.len()); + au[i] = rand(256) as u8; + } + 2 => au.truncate(rand(au.len() + 1)), + 3 => au.extend((0..rand(64)).map(|_| rand(256) as u8)), + 4 => au = (0..rand(2000)).map(|_| rand(256) as u8).collect(), + _ => { + // splice two access units + let other = &aus[rand(aus.len())]; + let cut = rand(au.len()); + au.truncate(cut); + au.extend_from_slice(&other[rand(other.len())..]); + } + } + match dec.decode(&au) { + Ok(pcm) => { + decoded += 1; + assert!(pcm.iter().all(|c| c.len() == 1024)); + } + Err(_) => errors += 1, + } + } + } + println!("mutated AUs: {decoded} decoded, {errors} rejected, 0 panics"); +} + +#[test] +fn random_configs_and_adts_never_panic() { + let mut rng = Rng(12345); + let mut byte = move || ((rng.next() * 0.5 + 0.5) * 256.0) as u8; + for _ in 0..20000 { + let len = (byte() % 12) as usize; + let asc: Vec = (0..len).map(|_| byte()).collect(); + if let Ok(cfg) = AudioSpecificConfig::parse(&asc) { + let _ = cfg.channels(); + if let Ok(mut d) = Decoder::from_config(cfg) { + let au: Vec = (0..(byte() as usize * 3)).map(|_| byte()).collect(); + let _ = d.decode(&au); + } + } + } + for _ in 0..2000 { + let mut s = vec![0xFF, 0xF1]; + s.extend((0..(byte() as usize * 8)).map(|_| byte())); + if let Ok(frames) = split_adts(&s) + && let Ok(mut d) = Decoder::from_adts(&frames[0].0) + { + for (_, au) in frames { + let _ = d.decode(au); + } + } + } +} + +#[test] +fn encoder_handles_hostile_input() { + for ch in [1usize, 2, 6] { + let mut enc = Encoder::new(EncoderConfig::cbr(44100, ch, 64_000 * ch as u32)).unwrap(); + let n = 5000; + let evil: Vec = (0..n) + .map(|i| match i % 7 { + 0 => f32::NAN, + 1 => f32::INFINITY, + 2 => -f32::INFINITY, + 3 => 40.0, + 4 => -1e30, + _ => ((i as f32) * 0.37).sin(), + }) + .collect(); + let chans: Vec<&[f32]> = (0..ch).map(|_| evil.as_slice()).collect(); + let mut aus = Vec::new(); + for chunk in [0usize, 1, 7, 1023, 1024, 1025, 3000] { + let c: Vec<&[f32]> = chans.iter().map(|s| &s[..chunk.min(s.len())]).collect(); + aus.extend(enc.encode(&c)); + } + // fewer channels than configured: treated as silence + aus.extend(enc.encode(&[&evil[..100]])); + aus.extend(enc.flush()); + let mut dec = Decoder::new(&enc.audio_specific_config()).unwrap(); + for au in &aus { + let pcm = dec.decode(au).unwrap(); + assert!(pcm.iter().flatten().all(|v| v.is_finite())); + } + let expected = (1 + 7 + 1023 + 1024 + 1025 + 3000 + 100usize).div_ceil(1024) + 1; + assert_eq!(aus.len(), expected); + } + // empty stream: flush yields nothing + let mut enc = Encoder::new(EncoderConfig::cbr(48000, 2, 128_000)).unwrap(); + assert!(enc.flush().is_empty()); + // bad configs are rejected + assert!(Encoder::new(EncoderConfig::cbr(44000, 2, 128_000)).is_err()); + assert!(Encoder::new(EncoderConfig::cbr(44100, 9, 128_000)).is_err()); + assert!(Encoder::new(EncoderConfig::cbr(8000, 2, 2_000_000)).is_err()); +} + +#[test] +fn roundtrip_and_delay() { + // Own encoder → own decoder: priming is exactly 1024 samples and the stream covers the input. + let rate = 48000; + let n = 10_000; + let sig = chords(rate, n, 0.0); + let mut enc = Encoder::new(EncoderConfig::cbr(rate, 1, 128_000)).unwrap(); + assert_eq!(enc.priming_samples(), 1024); + let mut aus = enc.encode(&[&sig]); + aus.extend(enc.flush()); + let mut dec = Decoder::new(&enc.audio_specific_config()).unwrap(); + let mut out = Vec::new(); + for au in &aus { + out.extend(dec.decode(au).unwrap().remove(0)); + } + assert!(out.len() >= n + 1024); + let s = snr(&sig, &out[1024..1024 + n]); + assert!(s > 15.0, "roundtrip SNR {s}"); + // no energy in the priming region beyond pre-echo of the first attack + let e: f32 = out[..512].iter().map(|v| v * v).sum(); + assert!(e < 1e-3); +} diff --git a/crates/audio/tests/codecs/concealment.rs b/crates/audio/tests/codecs/concealment.rs new file mode 100644 index 00000000..299a7ffb --- /dev/null +++ b/crates/audio/tests/codecs/concealment.rs @@ -0,0 +1,56 @@ +//! A packet that fails to decode is replaced by silence of its length, so the audio after it keeps +//! its timing; a stream where nothing decodes is still an error. The streams are made with +//! craft-audio's own AAC encoder (no ffmpeg needed). + +use craft_audio::aac::{AdtsHeader, Encoder, EncoderConfig}; +use craft_audio::{AudioError, decode}; + +/// One second of a 440 Hz stereo tone as ADTS AAC, and the offset and length of each frame. +fn adts_stream() -> (Vec, Vec<(usize, usize)>) { + let rate = 48_000; + let tone: Vec = (0..rate).map(|i| (i as f32 * 440.0 * std::f32::consts::TAU / rate as f32).sin() * 0.5).collect(); + let mut enc = Encoder::new(EncoderConfig::cbr(rate as u32, 2, 128_000).with_adts(true)).unwrap(); + let mut units = enc.encode(&[&tone, &tone]); + units.extend(enc.flush()); + let mut frames = Vec::new(); + let mut bytes = Vec::new(); + for u in units { + frames.push((bytes.len(), u.len())); + bytes.extend(u); + } + (bytes, frames) +} + +#[test] +fn garbled_frames_keep_the_timeline() { + let (clean, frames) = adts_stream(); + let (_, reference) = decode(&clean, Some("aac")).unwrap(); + let mut garbled = clean.clone(); + // Overwrite the payload (not the 7-byte header) of ten frames in the middle. + for &(at, len) in &frames[10..20] { + let header = AdtsHeader::parse(&garbled[at..]).unwrap().header_length; + for b in &mut garbled[at + header..at + len] { + *b = 0xFF; + } + } + let (_, damaged) = decode(&garbled, Some("aac")).unwrap(); + assert_eq!(damaged.frames(), reference.frames(), "a garbled frame must not shorten the stream"); + // Audio after the damage stays where it was. + let at = reference.frames() - 4800; + assert_eq!(&damaged.channels[0][at..], &reference.channels[0][at..]); +} + +#[test] +fn a_stream_where_nothing_decodes_is_an_error() { + let (mut bytes, frames) = adts_stream(); + for &(at, len) in &frames { + let header = AdtsHeader::parse(&bytes[at..]).unwrap().header_length; + for b in &mut bytes[at + header..at + len] { + *b = 0xFF; + } + } + match decode(&bytes, Some("aac")) { + Err(AudioError::Malformed(_) | AudioError::Unsupported(_)) => {} + other => panic!("expected an error, not silence: {:?}", other.map(|(i, b)| (i.frames, b.frames()))), + } +} diff --git a/crates/audio/tests/codecs/main.rs b/crates/audio/tests/codecs/main.rs index 49c50580..21bc9bd3 100644 --- a/crates/audio/tests/codecs/main.rs +++ b/crates/audio/tests/codecs/main.rs @@ -8,4 +8,6 @@ use craft_media_testkit as common; mod testkit; mod aac; +mod concealment; mod routing_oracle; +mod track_decoder; diff --git a/crates/audio/tests/codecs/track_decoder.rs b/crates/audio/tests/codecs/track_decoder.rs new file mode 100644 index 00000000..0607db4a --- /dev/null +++ b/crates/audio/tests/codecs/track_decoder.rs @@ -0,0 +1,34 @@ +//! `TrackDecoder` on streams an app's own demuxer hands over without codec configuration: +//! ADTS-framed AAC (MPEG-TS). + +use craft_audio::aac::{Encoder as AacEncoder, EncoderConfig}; +use craft_audio::{CodecId, CodecParams, TrackDecoder}; + +fn tone(n: usize) -> Vec { + (0..n).map(|i| (i as f32 * 0.03).sin() * 0.4).collect() +} + +#[test] +fn adts_aac_without_extradata() { + let left = tone(48_000); + let right = tone(48_000); + let mut enc = AacEncoder::new(EncoderConfig::cbr(48_000, 2, 128_000).with_adts(true)).unwrap(); + let mut units = enc.encode(&[left.as_slice(), right.as_slice()]); + units.extend(enc.flush()); + + // No AudioSpecificConfig: the first ADTS header gives it, and every header is stripped. + let lookahead: Vec<&[u8]> = units.iter().take(craft_audio::LOOKAHEAD_PACKETS).map(Vec::as_slice).collect(); + let mut dec = TrackDecoder::new(&CodecParams::new(CodecId::Aac), &lookahead).unwrap(); + assert_eq!((dec.codec(), dec.sample_rate(), dec.channels()), (CodecId::Aac, 48_000, 2)); + let frames: usize = units.iter().map(|u| dec.decode(u).unwrap().frames()).sum(); + assert_eq!(frames, units.len() * 1024); + + // Two ADTS frames in one packet, as a PES payload often holds, decode as both. + let mut dec = TrackDecoder::new(&CodecParams::new(CodecId::Aac), &lookahead).unwrap(); + let pair = [units[0].as_slice(), units[1].as_slice()].concat(); + assert_eq!(dec.decode(&pair).unwrap().frames(), 2048); + + // Raw access units still need their configuration. + let raw = &units[0][7..]; + assert!(TrackDecoder::new(&CodecParams::new(CodecId::Aac), &[raw]).is_err()); +}