.aep: import twice as fast, faster export (#3302)

* aep: a layer's markers and Source Text are read where they are stored, not searched for (#3276)

A layer's marker list is a direct child of its root tdgp, and its Source Text record
btds a direct child of the ADBE Text Properties group. Markers walked the whole layer
subtree twice per layer (most layers have no marker list, so the walk always ran to the
end), and the three text-document accessors walked it once each.

* aep export: build the prototype template once, sort names with cached keys (#3277)

The template every export clones from is immutable; it is now built by the
first export of the process and kept (about 0.93 MB). The name sort builds
each natural key once instead of on every comparison. Root solids move into
the Solids folder instead of being copied. The two key-list rewrites share one
emit-local helper and match names compare as bytes instead of a decoded String.
Output is byte-identical.

* aep container: size sibling lists, hex and write output once; borrow plain .aepx attributes (#3275)

- read.rs: count the chunk headers of a sequence first and allocate the
  children Vec once (bounded by the input and by the chunk limit).
- aepx.rs: decode_hex sizes its output once; attribute values without
  an entity are borrowed instead of copied through decode_entities.
- typed.rs: PropGroup::get compares match names without building a
  String (same result as the lossy compare, invalid UTF-8 included).
- write.rs: reserve the exact output size up front (try_reserve, so a
  hand-built tree cannot abort on the allocation).
- read.rs, settings.rs, values.rs: use the shared bytes.rs readers.

Output is identical on the corpus (tree, written bytes, import).

* aep container: review fixes (#3275)

- read.rs: reserve with try_reserve_exact, so a raised chunk limit on a
  small target cannot abort before parsing; the capped reservation and a
  deep, retry-heavy nest of broken lists are pinned by tests.
- write.rs: a tree past the 32-bit container size fails with TooLarge
  before anything is serialised.
- aepx.rs: drop an unneeded local and lifetime annotation.

* aep export: note what the concurrency test covers and why a failed template build is not kept (#3277)

* aep: text documents are read without a String per token, defaults built once (#3276)

TextDocument::parse reads a borrowed tree: names, bare words and strings without
escapes point into the body, keys are borrowed when they are /names, numbers parse from
the borrowed bytes, and dictionaries and arrays are moved into vectors of their exact size
when they close. The style and paragraph defaults are built once instead of per document.
parse_text_dict keeps its public TextValue result (the borrowed tree turned owned).

The old owning parser stays as a test-only oracle: a differential test compares both on
40 000 generated bodies (every key form the owned keys canonicalise, non-UTF-8 names,
escapes, unclosed strings, non-finite numbers), every cut of a realistic document and
nesting around the depth limit.

* aep: text-dict test builds its prefix without repeat(1) (#3276)

* aep: an escaped text string drops its byte order mark without a range drain (#3276)

* aep export: one key-time loop, key-list rewrite, tdmn decoder and byte writer for the listed modules (#3278)

In masks, shapes, styles, synth, fxsynth, base, camlight, skeleton,
textanim and tree (effects.rs, plan.rs and emit.rs keep their copies for
the follow-up work):
- New export/keys.rs: the key-time loop (timed_keys), the key-list rewrite
  (fill_key_list, set_key_list, replace_cdat) and the spatial key record,
  shared by masks, shapes and styles.
- One tdmn decoder (synth::tdmn_name, lossy UTF-8 up to the first NUL, as the
  importer reads it) and one encoder (synth::tdmn) instead of the copies in
  masks, styles, shapes, textanim, camlight and skeleton.
- One byte writer: tree::put, which errors. base::bytes, synth::tdb4,
  synth::key_list, fxsynth::pard, masks::shph and the keyed path records write
  through it; their fixed layouts keep the infallible signatures behind
  tree::fixed, which logs, and a test builds every layout and requires no error.
- camlight::put_group uses the shared group split (tree::split_group), which
  errors on a match name without a node instead of stopping quietly.

Export output is byte-identical on the corpus and the synthetic fixtures.

* aep export: pin the key-time loop's report, name the layout in the base::bytes log (#3278)

* aep export: review fixes for #3278

- tree::Fixed replaces tree::fixed: a fixed layout writes every field that
  fits (as the old code did) and keeps the first error; finish() logs it
  once with the layout's description and counts it per thread, so the
  every-layout test cannot race with other tests.
- base::bytes logs one line naming its length and fields.
- masks::build_atom errors when the prototype has no Mask Shape instead of
  dropping the path of every mask quietly.
- log sorted into the dependency list.

* aep: review fixes for #3276 text documents and markers

text_document reads the first btdk in the Source Text record, the documents
text_document_count counts. The old slot checks (same, is_neutral, undecoded_fields)
join the test oracle and are compared on 20 000 generated slot dictionaries; mixed
dictionary/array chains and a document left open at the depth limit join the parser
differential. An escaped string's byte order mark is drained without a copy.

* aep: remove test-only container accessors, share report and time-base code (#3281)

- Remove 35 public container accessors that only tests called (header,
  camera/light, keys, footage, masks, shapes, version) and the dead
  StyleMap::switch_name. Tests now check the importer's own reading of the
  same bytes, or the decoder it uses; three import tests gain the values
  the removed checks covered.
- Import and export reports share one loss entry (Loss<C>) with record,
  total, count and summary; the JSON shape is pinned for both.
- Collector::lose_with builds a detail only below the detail cap (callers
  switch later).
- The export uses the container's TimeBase; one NTSC table serves import
  and export; TimeBase::frames saturates like units instead of returning 0.

* aep: review fixes for #3281: NTSC table beside TimeBase, real-path NTSC round trip, light and camera defaults on import, doc wording

* aep: drop a local check tool committed by mistake (#3281)

* aep export: sort keys as one compact byte string, bounded by the name (#3277)

The cached natural keys were Vecs of (bool, String, u128) segments: a 255-byte
name of alternating letters and digits took about 14 KB, held for every item at
once. Each key is now one byte string whose byte order is the natural order,
at most 5 bytes per name byte. Order and output are unchanged.

* aep container: check bdata digits and size before reserving (#3275)

decode_hex reserved the decoded size before it knew the value decodes,
so a large bdata with a bad digit reserved half its length first. It now
refuses a value larger than a chunk's 32-bit size field (TooLarge), checks
every digit in one vectorisable pass, and only then reserves, with
try_reserve_exact so a value that does not fit in memory is an error
instead of an abort.

* aep export: the every-layout test counts from a baseline, not from zero (#3278)

* aep export: correct the order the natural sort key doc gives for equal numbers (#3277)

* aep: the format notes name the accessors that remain (#3281)

* aep: the format notes name the 1 x 1 mask scale of text and shape layers (#3281)

* aep: read imports through a flat tree index (#3300)

Import reads an .aep through TreeIndex, one depth-first array of chunks that
borrows the file's bytes, instead of building and freeing the owned Chunk tree
(one Vec per list). The parser mirrors read::Parser step for step: the same
chunk count order against max_chunks, the same depth limit and opaque
fallbacks, the same errors. The typed view reads through Node, a handle to a
chunk of either tree, so the accessors work unchanged on both; .aepx keeps its
owned tree. Chunk, parse and write stay for export and editing.

The array starts small and doubles, never past what the input can hold or the
chunk limit allows; a failed allocation is TooLarge, a tree past u32 indices
TooManyChunks.

* aep export: the time remap and Transform keys use the shared key-list and match-name helpers; doc comments for the audit's functions (#3300)

* #3300: import .aepx through the flat index

The .aepx reader no longer builds the owned chunk tree element by
element. It decodes every body (hex, text, alias JSON, numbers) into
one buffer and records the chunks as depth-first entries whose bodies
are ranges of that buffer; TreeIndex::from_drafts turns them into the
index the importer reads, which borrows the buffer. read_aepx converts
that index into the owned tree in one pass.

The tree and the errors are those of the previous reader, which the
tests keep as the reference: generated and damaged documents under
seven limit sets, invalid UTF-8, hostile input, the fixture directory,
and imports through both. Bound tests pin the chunk list to the input
and the limit, and check that dropped content leaves nothing behind.

* #3300: read_aepx keeps its own owned-tree reader

read_aepx, read_aepx_with_limits and parse_any build the owned chunk
tree element by element again, as before the .aepx index, instead of
converting the index (which held the decoded buffer and the owned tree
at the same time). Import keeps the index reader. The tests now
compare the two production readers.

* #3300: test keys that round to the same time

* aep: compare the destination projects in the index-equivalence tests

both() and both_aepx() kept only the ImportResult of each path; they now keep
both destination Projects and assert structural and serialized equality, so a
layer or property value regression cannot pass on equal ids and reports.

---------

Co-authored-by: FredApps <153173854+FredApps@users.noreply.github.com>
This commit is contained in:
Valentin Klein
2026-11-13 00:54:29 +01:00
committed by GitHub
co-authored by FredApps
parent 10a34b854f
commit a4bde72bca
70 changed files with 5248 additions and 1428 deletions
+20 -16
View File
@@ -119,6 +119,11 @@ counterpart (a name that is not 1 to 4 characters) are skipped with their conten
with the byte offset. It runs without recursion; the nesting depth (`Limits::max_depth`) and the
element count (`Limits::max_chunks`) are bounded, and everything it allocates is bounded by the
size of the input.
- There are two readers. `read_aepx` builds the owned tree element by element. The importer, which
only reads, uses a second reader that decodes every body (hex, text, the alias JSON, the
numbers) into one buffer and records the chunks as a flat depth-first list whose bodies are
ranges of that buffer; it indexes that buffer (`TreeIndex`) as it indexes an `.aep`. The tests
hold both readers to the same tree and the same errors.
- Tests: `tests/aepx.rs` compares the five fixture pairs chunk by chunk and byte by byte, compares
the typed view of both forms, and runs truncated, bit-flipped, deeply nested and oversized
inputs; `crates/aep/tests/it/import/aepx.rs` imports both forms of each pair and expects the same
@@ -138,7 +143,7 @@ write on their own.
| `svap` (4 bytes, the first top-level chunk) is the version of the After Effects that **wrote** the file: `0f 11 0e 02` in all 36 (a down-saved copy keeps the 2026 word) and in all 1 192 other `.aep` files of the local corpus folders. After Effects opens a file without `svap` (`t-nosvap`). | All 36; corpus scan. |
| `head` bytes 4 to 7 are the version the file is **written for**: `0f 11 0e 02` (= `svap`) in the 2026 saves, `0f 0b 06 61` in the 25.x copies, `0f 03 06 41` in the 24.x copies, the same word in every family. After Effects refuses a file whose word is zero ("created with an older version [0.0x0]", `u-zhead`). | All 36; `u-zhead`. |
| The word is packed (bit 31 first): bits 25-24 and 21-19 the major, 18-15 the minor, 14-11 the bug-fix, 10-8 a stage (6 in all), 7-0 the build; bits 31-26 (`000011`) and 23-22 (`00`) are the same in all samples and not decoded. `0f110e02` = **26.2.1x2**, which is what the saving After Effects reports for itself (`app.version` recorded by the corpus scripts) - this one sample fixes minor, bug-fix and build. The copies read as **25.6.0x97** and **24.6.0x65** (major **V** from three consecutive majors; minor, bug-fix and build of the copies **E**: one other word decides their positions). `effectcraft_aep::AeVersion`, `AepFile::file_version`, `AepFile::saved_by_version`. | Tests `decodes_the_sampled_words`, `the_container_of_a_copy_states_its_version_in_head_and_the_writer_in_svap`. |
| `head` bytes 0 to 1 (a `u16`) step with the major: 97 (`0x61`) in the 2026 saves, 96 in 25.x, 95 in 24.x; bytes 2 to 3 are 5, 9 and 6 (**?**). Bytes 8 to 15 differ between families, not between versions; bytes 16 to 19 are the save counter (one higher in each copy). Nothing is derived from bytes 0 to 3 (`AepFile::head_revision` exposes the `u16`). | All 36. |
| `head` bytes 0 to 1 (a `u16`) step with the major: 97 (`0x61`) in the 2026 saves, 96 in 25.x, 95 in 24.x; bytes 2 to 3 are 5, 9 and 6 (**?**). Bytes 8 to 15 differ between families, not between versions; bytes 16 to 19 are the save counter (one higher in each copy). Nothing is derived from bytes 0 to 3, and the reader does not expose them. | All 36. |
| The XMP packet after the container is no guide: its `CreatorTool` is "Adobe After Effects 2026" in the copies too. | All 36. |
| **26 against 25.x** (per family, chunk tree aligned by item and layer id): besides `head`, `nhed` / `nnhd` differ in 6 bytes (`nhed` 20-23 and 26-27: `827fa930` / `0015` in the 2026 save, `00000000` / `011b` in both copies; session state, see section 4). Every **solid's** `Pin /CLRS` changes: the 2026 save has `dcui` (1 byte, 1), `apid` all `0xff` and an empty `Utf8`; the copies drop `dcui`, set `apid` to a GUID (`601e9973...`, the same in all) and embed a 935-byte JSON with an ICC profile (`baseColorProfile/colorProfileData`). A **footage** item that the 2026 save stores with an assigned profile (`mcsp` = 1 and a profile JSON) loses `mcsp` and that JSON in the copies, and its `epid` becomes all `0xff`. The rest is save noise: the footage alias `ascendcount_base` (the copies were saved in another folder), `sspc` byte 112 of a footage item (a modification time), the `oEXR` `opti` of an EXR item (the copies hold the display device name `\.\DISPLAY1` as UTF-16 at byte 126 and, in the 24.x copy, a few non-zero bytes from byte 2 102 that look like uninitialised memory; the 2026 save has zeros), the plug-ins' sequence data `sdat` of Lumetri Color (16 instances and the project's `EfdG`), the mask counter `mkif` byte 8. No chunk is added or reordered, and no `pard`, `cdat`, `ldta`, `cdta`, `tdb4`, keyframe, text or shape chunk differs. | All 12 families. |
| **25.x against 24.x**: only `head` (bytes 1, 3, 5 and 7), and the session noise in `opti` and `sdat` above. | All 12 families. |
@@ -216,12 +221,12 @@ the value from the corpus manifest (test file in parentheses at the end of each
API: `LayerHeader` (`is_video_enabled`/`is_enabled`, `is_audio_enabled`, `is_effects_enabled`,
`is_auto_oriented`, `light_type`, `blend_mode`, `stretch` / `stretch_den`, `stretch_ratio`,
`stretch_percent`, `in_point`, `out_point`); the same switches on `Layer`; `Layer::track_matte`
`in_point`, `out_point`); the same switches on `Layer`; `Layer::track_matte`
(`TrackMatte`, `MatteMode`; test `tests/matte.rs`). `stretch` is now the
numerator of the ratio (it was the whole multiplier before 50 % and 150 % were sampled). Tests:
`tests/header.rs`, `tests/layer_switches.rs`. The layer name itself is the `Utf8` child of the
`Layr` (empty while the name is the source's); the layer then shows the name of its source item
(`Project::layer_display_name`).
(`Project::layer_source(...)` then `Item::display_name`, the rule the importer uses).
### 3.2 Composition header (`Item/cdta`, 204 bytes; the decoder needs the first 176) and renderer
@@ -246,9 +251,9 @@ numerator of the ratio (it was the whole multiplier before 50 % and 150 % were s
The inert `ADBE Time Remapping` record of every AV layer has `tduM` = the source's duration in seconds: 5 for 5 s compositions (124, 257, 258), 3 and 2 for a 3 s and a 2 s composition (600, 601), 1 for the 1 s WAV (351), 0.1 for the three-frame sequence (353), 0 on solids and stills (000, 126). Every property's keyframe time scale (`tdb4` bytes 12 to 15) follows the composition (23 976 in every
property of `L` after 229 and 230). The corpus manifest calls the classic renderer
`ADBE Advanced 3d`; the file stores `ADBE Escher` / `Classic 3D`. API: `CompHeader::frame_rate_exact`
(exact fraction), `duration_frames`, `shutter_phase`, `display_start`, `display_start_seconds`,
`display_start_frame`. Tests: `tests/header.rs`, `tests/layer_switches.rs`.
`ADBE Advanced 3d`; the file stores `ADBE Escher` / `Classic 3D`. API: `CompHeader::{time_base,
frame_rate_fixed, duration_units, shutter_phase, display_start}` (`display_start` is the exact
fraction). Tests: `tests/header.rs`, `tests/layer_switches.rs`.
### 3.3 Items, footage and precomps
@@ -293,7 +298,8 @@ name. A footage layer shows the item's name while it has no name of its own (351
Test: `tests/header.rs` (precomp), `tests/text_markers_footage.rs` and `tests/batch3_objects.rs`
(footage), `tests/layer_switches.rs` (solid colour). API: `Footage::{duration, native_frame_rate,
bits_per_pixel, file_size, audio_sample_rate, is_sequence, sequence_frame_count}`.
bits_per_pixel, file_size, audio_sample_rate, is_sequence}`; `Item::sequence_info` (with
`SequenceInfo::frame_count`).
### 3.4 Properties: sparse storage, static values and units
@@ -402,8 +408,8 @@ out speed, at 40 out influence**). Scale key 1 of 305 (`n = 3`, in 0 / 33 %, out
15 / 66) is `0,0, .33,.33, 15,15, .66,.66`. Component `i` uses slot `i` of each block (**V** for the
block layout and the units, Scale speed in fractions per second and Mask Feather speed in pixels
per second; **E** that different values per dimension sit in the slot of their dimension, because
the tool that made the corpus applies one ease to all dimensions; **V** since the exporter wrote three different eases for the three Scale dimensions and After Effects read the same three back, aep-export.md R13). `Keyframe::plain_ease_slots`
returns the doubles as stored, `Keyframe::ease_component(i)` one component's `KeyEase`. Layout of
the tool that made the corpus applies one ease to all dimensions; **V** since the exporter wrote three different eases for the three Scale dimensions and After Effects read the same three back, aep-export.md R13). `Keyframe::ease_component(i)`
returns one component's `KeyEase`, `Keyframe::ease_components` all of them. Layout of
the spatial template: `u32` at 8 (1 normally, 0 after the spatial auto-bezier and continuous
switches were turned off together, 054), `f64` arc length of the segment to the next key at 16
(100 in 204 key 1, 0 on the last key and right after creation; safe to ignore), **ease at 24 in
@@ -428,7 +434,7 @@ in tangent (`n` x `f64`, relative offset from the value) at `56 + 8 n`, out tang
| **ease, colour switched to bezier** | spatial n=4 @24..55 | keyframe.set_interpolation bezier on two colour keys stores speed 0 and influence 0 on both sides, and After Effects interpolates them linearly (598 Start Color red to blue: 0.75 / 0.5 / 0.25 red at 0.5 / 1 / 1.5 s); a custom colour ease cannot be set by script | 598 | V |
| **ease, mask path** | 64-byte record @24..55 | in speed, in influence, out speed, out influence as in the spatial template: 599 out / in influence 0.75, 700 0.2 / 0.5, speeds 0 (the path at 0.5 s is 5.42 % / 28.0 % of the way). The `f64` at 16 is 1 (093) or 2 (599, 700) on the first key and 0 on the last (not decoded) | 093, 599, 700 | V (influence), ? (speed unit) |
API: `Keyframe::{interpolation_in, interpolation_out, kind, value, ease, ease_component, ease_components, plain_ease_slots, color_distance, spatial}`;
API: `Keyframe::{interpolation_in, interpolation_out, kind, value, ease, ease_component, ease_components, spatial}` (the `f64` at byte 16 of a colour key is `SpatialKey::segment_length`);
`Property::keys()` hands the property's template to its records. Marker record (16 bytes): time,
`03 03` hold bytes, 8 stale bytes. Mask path record (64 bytes): the common fields and the ease at byte 24 (599, 700). Test:
`tests/values.rs`, `tests/keyframes.rs`.
@@ -765,7 +771,7 @@ pins are placed by hand or by a preset.
328), so mode and inverted are reported per mask. `mkif` byte 8 is the running mask id counter of the
**layer** in the session (1, 2 for the two masks of 328; 329 then gets 3; undo does not reset it; a new
layer starts at 1 again): exposed as `Mask::counter`, a session counter and not a stable id. API:
`Mask::{mode, inverted, counter}`, `Project::mask_vertices_px` (vertices in layer pixels), `Layer::mask_atoms` (`MaskAtom`: the mask with its atom group, its path property and its shapes by position, for the keys and expressions of the mask properties). Tests:
`Mask::{mode, inverted, counter}`, `ShapePath::vertices(size)` (vertices in layer pixels: `size` is the size of the layer's source, `Project::layer_source_size`; a text or shape layer has no source size and its mask boxes are already pixels, so the importer passes 1 x 1), `Layer::mask_atoms` (`MaskAtom`: the mask with its atom group, its path property and its shapes by position, for the keys and expressions of the mask properties). Tests:
`tests/masks.rs` (with a synthetic check of the handle order), `tests/batch3_objects.rs`.
### 3.9 Shape layers (`tdgp[ADBE Root Vectors Group]`)
@@ -893,7 +899,7 @@ groups as on other layers, with these conventions:
| What | Path | Stored | Evidence | Conf. |
|---|---|---|---|---|
| camera Position, default (auto state) | `ADBE Transform Group/ADBE Position`, `tdb4` 76 and 77 = 1 although no expression exists | `[0, 0, -2666.67]` (x and y are 0 whatever the composition size; z = -zoom). A **one-node** camera reports `[0, 0, -2666.67]` (251). **Correction (S16): a reopened two-node camera reports `[0, 0, -2666.67]` too (256 read back), and After Effects renders it there**: the default two-node camera of 537 above the 3D solid of 486 renders from `[0, 0, -2666.67]` towards the point of interest, not like the composition's default view (537 differs from 486 in 24 % of the grid points, and EffectCraft's import with the stored position matches 537 but for 4 edge points). The `[960, 540, ...]` of the batch-3 notes was not reproduced (535 and 537, created by script, read back the stored value too); `CameraPosition::reported` keeps the old rule and is marked as contradicted | 256, 251, 537 | V |
| camera Position, default (auto state) | `ADBE Transform Group/ADBE Position`, `tdb4` 76 and 77 = 1 although no expression exists | `[0, 0, -2666.67]` (x and y are 0 whatever the composition size; z = -zoom). A **one-node** camera reports `[0, 0, -2666.67]` (251). **Correction (S16): a reopened two-node camera reports `[0, 0, -2666.67]` too (256 read back), and After Effects renders it there**: the default two-node camera of 537 above the 3D solid of 486 renders from `[0, 0, -2666.67]` towards the point of interest, not like the composition's default view (537 differs from 486 in 24 % of the grid points, and EffectCraft's import with the stored position matches 537 but for 4 edge points). The `[960, 540, ...]` of the batch-3 notes was not reproduced (535 and 537, created by script, read back the stored value too); the reader keeps only the stored value (`CameraPosition::stored`) | 256, 251, 537 | V |
| camera Position, set by the user | same, `tdb4` 76 and 77 = 0 | the absolute value `[100, 200, -1000]` | 253 | V |
| camera point of interest | `ADBE Anchor Point` | `[960, 540, 0]` in **pixels** (not a fraction); the record's `tdsb` is 1 on a two-node and 3 on the one-node camera (not asserted) | 251, 253, 256 | V (value), E (`tdsb`) |
| light Position, default | same as the camera | `[0, 0, -1333.33]`, as After Effects reports it (255 read back); the point of interest `[960, 540, 0]` | 254, 255 | V |
@@ -909,10 +915,8 @@ groups as on other layers, with these conventions:
A point light made by converting a spot light has a different property set (`Orientation`, `Rotate
X/Y/Z`, `Cone Angle`, `Cone Feather 2` in the default state) that a spot light lacks (252); whether that
is an artefact of the conversion is untested. API: `Layer::camera_position`,
`CameraPosition::reported`, `Layer::camera_point_of_interest`, `Layer::light_intensity`,
`Layer::{camera_zoom, camera_depth_of_field, camera_focus_distance, camera_aperture, light_color,
light_cone_angle, light_cone_feather}`, `DEFAULT_CAMERA_ZOOM` (2 666.67 is verified for the 1920 px
is an artefact of the conversion is untested. API: `Layer::camera_position`; the camera and light
options are read by the importer's table (`import/camera.rs`); `DEFAULT_CAMERA_ZOOM` (2 666.67 is verified for the 1920 px
composition only), `Layer::orientation` (`Orientation`; test `orientation_is_otst_otda` in
`tests/batch5.rs`). Render checks against After Effects (20 px grid, 5 184 points): 486 1 point
off, 535 2, 536 0, 537 4 (anti-aliased edges); **488 (Orientation `[10, 20, 30]`): 272 points