Three-way merge of the changes EffectCraft made to its av1enc and hevcenc
crates after craft-video took them:
- hevc: EncoderConfig carries colour_primaries, transfer_characteristics and
matrix_coeffs (H.273 code points, BT.709 by default) and the VUI writes them.
- av1: EncoderConfig carries color_primaries, transfer_characteristics and
matrix_coefficients and color_config writes them.
- av1: motion search keeps a visited set and stops a SAD once it cannot win,
SATD has fixed-size kernels, and a decision snapshot saves only the block's
BlockDecoded flags. Output is unchanged.
ffprobe tests check the colour description of both streams.
The encodings every Lottie layer, mask and shape reuses, on serde_json
values: the document header, animatable properties and their keyframes
(legacy end values, hold keys, spatial tangents), ease tangents to and
from speed and influence, bezier paths, gradient stop lists, the tolerant
number and colour readers, and embedded images as data URIs.
Taken from effectcraft crates/lottie at e18f48fb (anim.rs, shapes.rs,
import.rs, export.rs, base64.rs) with EffectCraft's project types replaced
by plain values. The layer, shape, effect and text mapping stays in
EffectCraft, and dotLottie is ZIP, which is not this crate's format.
Carved from FilmCraft's filmcraft-captions (crates/captions at 4bec22bd): the
file readers and writers, CEA-608/708 coding, format detection and the cue
document. Rendering and the mapping onto FilmCraft's caption tracks stay there.
- time: the crate has its own exact Tick (1/254016000000 s) and rational
FrameRate, the subset of filmcraft-time the formats use.
- TTML is read through craft-xml instead of roxmltree: no entity expansion,
nesting bounded by the reader (128 levels, as before).
- parse refuses input over 64 MiB and keeps at most 1,000,000 cues.
- MCC and EBU STL name the caller's program (WriteOptions::program).
- plain_text (inline tag stripping) moves with the SCC/MCC writers that need it.
The format half of FigCraft's crates/vectordrawable (at 24cf7032): the
model of paths, groups, clips and gradients, the XML writer, and the
mapping of a gradient on an affine frame to what a drawable can say.
The walk over FigCraft's document, its compatibility modes and warnings
stay in FigCraft.
Part of #778.
* craft-raw: the file-format layer of camera raw files, with DNG and CRW
A new crate for #587 and #778: bytes in, the sensor data out as a neutral
RawImage (samples, CFA, levels, active area, crop, orientation, colour tags,
opcode lists, profile tables), and back for DNG. One Cargo feature per format:
dng (read and write), jxl (JPEG XL DNG), crw (through craft-crw).
The base is LightCraft's crates/raw at 5ff567a4: dng, dngwrite, tiffraw,
unpack, ljpeg, jxl, semantic, preview, the parsing half of opcodes, profile
and gaintable, the model and probe. Demosaicing and everything after it stay
with the develop pipeline.
Ported from PhotoCraft's and FilmCraft's DNG readers, each with a test: the
default crop's per-tag defaults, an unusable crop falling back to the active
area, non-finite white levels, the white default of a linearized image, the
AsShotWhiteXY range, 8-bit-only lossy JPEG, lossy tiles that must cover their
part of the image, and the CinemaDNG frame tags.
* craft-raw: Hasselblad 3FR and FFF (feature hasselblad)
RawFormat::CfaTiff decodes: the DNG-layout reader plus what these files add.
Uncompressed data or Hasselblad's lossless JPEG (selection value 8), RGGB when
no CFAPattern is stated, an FFF's colour matrix from the 3FR header it embeds,
and the 8-bit RGB preview handed on as a baseline TIFF.
Moved from LightCraft's vendor/hasselblad.rs at 5ff567a4 with its tests.
Ported from PhotoCraft's reader: four BlackLevel values without a
BlackLevelRepeatDim are one per CFA site in a file that is not a DNG.
* pdf inspection: count pending text clips and drop page content after reading
- hayro-interpret: state_bytes counts the pending text-clip outlines a
saved state copies, so MAX_SAVED_STATE_BYTES bounds them too.
- plate-test structure: read each page through a ContentReader into a
buffer dropped after the page, instead of page.operations(), which keeps
every page's decoded content for as long as the document.
* address review: cap page content read at the 256 MiB decode limit
* fix the saved-state test: start the clip outline with a MoveTo
A new crate for #587 and #778: bytes in, the sensor data out as a neutral
RawImage (samples, CFA, levels, active area, crop, orientation, colour tags,
opcode lists, profile tables), and back for DNG. One Cargo feature per format:
dng (read and write), jxl (JPEG XL DNG), crw (through craft-crw).
The base is LightCraft's crates/raw at 5ff567a4: dng, dngwrite, tiffraw,
unpack, ljpeg, jxl, semantic, preview, the parsing half of opcodes, profile
and gaintable, the model and probe. Demosaicing and everything after it stay
with the develop pipeline.
Ported from PhotoCraft's and FilmCraft's DNG readers, each with a test: the
default crop's per-tag defaults, an unusable crop falling back to the active
area, non-finite white levels, the white default of a linearized image, the
AsShotWhiteXY range, 8-bit-only lossy JPEG, lossy tiles that must cover their
part of the image, and the CinemaDNG frame tags.
Moves filmcraft-avi (storytold/filmcraft@4bec22b) into craft_media::avi.
ByteSource is the crate's trait, Error::Io holds the io::Error, and
Stream::codec_params() maps the FourCC and the WAVEFORMATEX tag to a CodecId.
Adds Container::Avi and tests for the stream, header-size, nesting and
OpenDML index limits.
* metafile: hold a placed pen width to what covers the frame
Picture::place scaled a pen width by the fit with no upper bound, so a
pen of 1e30 units made a renderer flatten round caps of that size. A
placed pen is now at most twice the diagonal of the frame and the path
together, which already covers the whole frame.
Ported from storytold/wordcraft#5427.
* metafile: one byte budget for items, EMF+ objects and clips; count clip regions when placing
- Canvas::shade_shape returns when the canvas is full or the clip is
empty, instead of sampling a tile that is then dropped.
- EMF+ objects, the pieces of a continued object and the current and
saved EMF+ clips count against the canvas's 256 MiB budget, in both
directions: an object is not stored past what the canvas holds, and
the canvas stops at what the objects hold.
- A nested metafile held by an EMF+ image object is charged for what its
items own (segments, pixels, text, clip rectangles; a shared tile or
clip path once) rather than 128 bytes an item.
- Picture::place charges each clip region it expands to the 500,000
element budget, for every kind of item, and expands a region once for
the consecutive items that share it.
Ported from storytold/deckcraft#1777, with its review comments addressed
(the canvas checks include the held bytes; text and bitmap items are
checked too; regions are matched by content, not by address).
* metafile: a negative crop side pads the picture
crop_window took each side as 0..1, so a negative side (a negative
a:srcRect side in an Office file, or a picture fitted whole into a frame
of another shape) was dropped and the picture was stretched over its
frame. A side is now taken down to -100: the window starts before the
source or ends past it, and Picture::fit places the source inside the
frame at its own proportions.
Ported from storytold/wordcraft#2027 (the -100 bound and the invariant
test) and storytold/wordcraft#5206 (the padding test).
* metafile: decode 8-bit text in the font's code page
8-bit text (WMF TEXTOUT and EXTTEXTOUT, EMF EXTTEXTOUTA and SMALLTEXTOUT)
was decoded in Windows-1250, 1251 or 1252 only, and a double-byte
character became a blank.
- Greek, Turkish, Vietnamese, Hebrew, Arabic, Baltic and Thai character
sets decode in their code pages (encoding_rs, already a workspace
dependency).
- Shift_JIS, UHC, GBK and Big5 pairs decode to their character, with the
lead and trail ranges of each set; the character keeps the advances of
both bytes. A lead byte without a trail byte is one undefined byte.
- A WMF face name is read in the font's code page.
- The Symbol font selected with DEFAULT_CHARSET keeps its bytes as symbol
codes (U+F000 + code), as SYMBOL_CHARSET fonts do.
Ported from storytold/wordcraft#5187 (the code pages, the double-byte
ranges, the face name and the Symbol font) and storytold/deckcraft#792
(the Hebrew and Arabic tests).
* metafile: dual EMF+ files draw their EMF+ records, as GDI+ does
For a dual EMF+ file the reader played the plain EMF records and ignored
the EMF+ ones. Office draws metafiles with GDI+, which does the
opposite, and a file's GDI fallback can be incomplete (the EMF+ part
draws a nested metafile the fallback lacks).
EMF+ records now always play. In a dual file the plain EMF records are
skipped, except between an EmfPlusGetDC record and the next EMF+ record.
EMF+ only files and plain EMF files are read as before.
Ported from storytold/deckcraft#1464.
---------
Co-authored-by: Dmitrii Khlapov <19991457+dkhlapov@users.noreply.github.com>
Co-authored-by: David Lin <woei66@gmail.com>
Co-authored-by: Harrison Booth <73434116+Harrison-Booth@users.noreply.github.com>
* craft-audio: AAC Huffman lookahead uses BitReader::peek
`BitReader::peek` reads from one 8-byte load since the bit reader was made
faster, which is what the AAC decoder's private `peek32` did. The private copy
goes; the lookahead is the same zero-padded 32 bits.
Ported from storytold/filmcraft#4232.
* craft-audio: a damaged FLAC keeps its length and timing
symphonia's FLAC demuxer drops a frame whose checksum fails and resyncs on the
next one, so the decoder never saw an error: the file decoded shorter and all
later audio came out one block early, with `damaged` at 0.
The jump in the packet timestamps is the audio that was lost. It is returned
as silence, in blocks of at most 65 536 frames, and counted once in
`AudioInfo::damaged`. It never passes the length the file declares; with no
declared length a jump of more than 2^20 frames is not believed. A seek into
the lost frames lands on silence at the right position.
Ported from storytold/soundcraft#556, onto the streaming decoder.
---------
Co-authored-by: vangel76 <vangel1976@googlemail.com>
Co-authored-by: Gert Pellin <gert@pellin.be>
* craft-video: H.264 decoder from FilmCraft (`h264-decode` feature)
FilmCraft's crates/h264 at 4c0d2ad7 becomes craft_video::h264::decode. The encoder moves to
h264/encode and is still exported from craft_video::h264. Feature h264 now turns on h264-decode
and h264-encode, so either half can be built without the other.
Without the threads feature a reference row that was never published read through expect();
it now reads as a gray row. craft_video::budget (the picture memory budget from DeckCraft's
bitstream crate) is added for the next commit. craft-video is built at opt-level 3 in the dev
profile, as FilmCraft built its codecs.
* craft-video: H.264 decoder hardening from DeckCraft's copy
- The picture store (DPB, current picture, pictures in flight or queued for output) is checked
against a memory budget before a picture is allocated; Decoder::set_memory_budget.
- configure_avcc checks every entry before handling any: SPS NAL units in the first list, PPS in
the second. A picture slice in the record starts no picture.
- The inverse transforms clamp coefficients to the spec's r range instead of overflowing.
- Temporal direct motion vectors are scaled in i32 and clipped to the spec range.
- slice_qp_delta is added with a checked add.
- A picture job sets up its picture state inside the panic guard.
* craft-heif: stills decode at the size the container declares
A coded HEVC picture is a whole number of coding blocks and can be larger
than the image the container's ispe declares (Sony codes 1920x1080 as
1920x1088 without a conformance window). decode returned the coded size,
so probe and decode disagreed and the padding rows showed as a dark band.
decode_item now crops to ispe before the alpha merge and the transforms,
measures the clap parity on the declared size, and compares the alpha
item's declared size with the picture's instead of the coded planes.
A declared size larger than the coded picture is Malformed.
Ported from storytold/lightcraft#3977.
* craft-heif: read_metadata reads the primary image's EXIF and XMP without decoding
Cataloguing a photo needs its EXIF and XMP, not its pixels. read_metadata
opens the container under the same panic guard and box census as probe
and returns what decode would attach.
Ported from storytold/lightcraft#2087.
* craft-heif testdata: Spec::stereo builds a spatial photo (grpl/ster entity group)
A second image of the primary's size and a ster entity group naming both,
so callers can test stereo detection on a file this crate decodes.
Ported from storytold/lightcraft#1684.
* craft-heif: put_alpha uses is_multiple_of
---------
Co-authored-by: Mason <mason276752@gmail.com>
Co-authored-by: jkalkstein <76634771+jkalkstein@users.noreply.github.com>
Co-authored-by: Mitch Lindgren <mitch@lindg.ren>
* craft-mesh OBJ: map_* options are passed over by their argument counts
A map line with options named its image by its final token, so
'map_Kd -o 0.5 0.5 my checker.png' named 'checker.png'. Options are now
skipped by the arguments they take (-o, -s and -t take one to three
numbers), so a name with spaces after options is one name. After an option
the reader does not know, the name is still the final token.
Ported from storytold/effectcraft#1324.
* craft-mesh OBJ: a material library named again is read once
Exporters repeat the mtllib line for every object. Each repeat asked the
resolver again, read the library again, added its materials and images a
second time, and counted its bytes against max_data_bytes once per line, so
a legitimate file with many objects could reach the limit.
Ported from storytold/effectcraft#2965 (its OBJ part: 'a library already
loaded is skipped').
---------
Co-authored-by: Lloyd Nicholson <nichlloyd@gmail.com>
* craft-xml: a limit on the attributes a tree holds over all its elements
max_attrs bounds one start tag and max_nodes counts elements and text, so
attributes were the one thing nothing counted across the tree: 1,025
elements of 1,024 attributes each are 1,025 nodes and over a million
(String, String) pairs, ten times the input in headers alone.
Limits::max_attrs_total (default 4,000,000) bounds the attributes kept at
once; elements handed to a streaming callback take theirs out of the
count. Error::TooManyAttrsTotal.
Ported from storytold/wordcraft#4320, which added the same budget to
WordCraft's DOCX reader.
* craft-xml tests: an internal DTD subset is refused wherever it hides, in linear time
The cases of storytold/filmcraft#4139 (a marker inside a comment or a
processing instruction, a '>' inside the system literal, 100,000 markers
that never close) against Doctype::RejectInternalSubset. The reader already
handles them; the tests keep it so.
* craft-xml: El::node_count, for one node budget over the parts of a package
max_nodes bounds one parse. A package of many parts, each inside the limit,
still costs a tree per part the reader keeps. node_count is what max_nodes
counted for an element, so a reader charges each tree against one budget
and reads the next part with what is left as its max_nodes: a part over
the budget stops before its tree is built.
Ported from storytold/wordcraft#5665, which adds a package-wide element
budget to WordCraft's DOCX reader (parse_counted with a max).
* craft-xml: parse_streaming_partial returns what was read when a streaming read fails
A worksheet whose rows cannot all be read (one row over the node limit, a
callback error, markup the tokenizer cannot read) lost everything else in
the part with it. parse_streaming_partial returns the error together with
the root as far as it was read: the children complete before the failure
and the elements open at that point. parse_streaming is unchanged.
Ported from storytold/gridcraft#3317 (xml::parse_streaming_partial).
* craft-xml: a strict pull reader refuses a second root and text outside the root
Options::events builds no tree, so the checks the tree reader makes for a
strict read (a second root element, character data outside the root) were
not made at all: a strict pull over '<a/><b/>' or '<a/>text' ended without
an error. A reader that walks events would take a damaged file's first
element for the whole document. The reader now makes both checks itself,
so the pull reader and the tree agree.
Ported from storytold/wordcraft#1862, which made WordCraft's ODT event
reader report malformed XML instead of stopping silently.
---------
Co-authored-by: Oliwer Urbaniak <oliwerx12@gmail.com>
Co-authored-by: Bra1nZ <95569966+Bra1nZ@users.noreply.github.com>
Co-authored-by: Neil Cawse <neilcawse@geotab.com>
Co-authored-by: midasmatterhorn <248507632+midasmatterhorn@users.noreply.github.com>
* craft-dxf: the ASCII reader unescapes only text that holds a backslash
Only a string with a backslash can hold a \U+ or \M+ escape; every other
string moves into the tag as it is instead of being copied. The EOF check
reads the value before it moves.
Ported from storytold/cadcraft#1833.
* craft-dxf: write_ascii_in_page writes the code page without the UTF-8 fallback
write_ascii_in falls back to pure ASCII when the code page bytes happen to
be valid UTF-8, for readers that guess UTF-8. A reader that decodes a
pre-2007 DXF strictly by its $DWGCODEPAGE needs the code page bytes.
Ported from storytold/cadcraft#2851.
* craft-dxf: behaviour-preserving cleanups
Exact group-code mappings in one place, documented errors on the fallible
public parsers, and simpler option, result and formatting idioms. No change
to parsing, writing, numeric conversion, encoding or the public API.
Ported from storytold/cadcraft#2571.
* craft-dxf: read R12 and earlier binary DXF (1-byte group codes)
Binary DXF written before R13 stores each group code in one byte, with 255
escaping a 2-byte code. The reader only knew the R13+ layout, where codes
are two bytes, and also treated 255 as an escape there. The layout is now
told from the first tag; 255 is an escape only in the 1-byte layout and an
ordinary code in R13+.
Ported from storytold/cadcraft#1499.
* craft-dxf: section_slices, record_slices and Tag::str_ref borrow instead of copying
sections() and records() copy every tag out of the parsed list, so a reader
that keeps the list holds a large file's tags two or three times. The new
functions return slices of the list. sections() and records() now build
their copies from them, so there is one implementation of each split.
Ported from storytold/cadcraft#1708. storytold/cadcraft#1588 solves the same
problem by changing Section itself; this keeps the existing API.
* craft-dxf: reals beyond 1e16 or below 1e-10 are written in exponent form; string_groups
format_real wrote f64::MAX as 309 digits. Values outside 1e-10..1e16 are now
written as AutoCAD writes them (1.0E+20) and read back exactly.
string_groups splits text into the 250-character groups long strings are
stored in (MAX_STRING_CHARS).
Ported from storytold/cadcraft#1790.
* craft-dxf: DOS 720/737/775/855/857/860/861/863/864/865/869 and ISO 8859-15 code pages; cheaper unescape and hex
$DWGCODEPAGE values DOS720, 737, 775, 855, 857, 860, 861, 863, 864, 865, 869
and ISO8859_15 had no decoder and fell back to Windows-1252. ISO 8859-15
decodes through encoding_rs, the DOS pages through tables checked against
the Unicode Consortium mapping files (dos_tables.rs). DOS 864's byte 0x25 is
U+066A, so a writer escapes '%' there. U+FFFD is never written as a table
byte. A string without an escape moves into its tag without a copy, cstr
finds its terminator with one scan, and binary chunks are hex-encoded from a
table.
Ported from storytold/cadcraft#1271. Its group-code-255 change is carried by
the R12 binary port: the 1-byte layout is detected only when the escaped
code is one that needs the escape, so an R13+ file starting with code 255
still reads.
* craft-dxf: Korean Johab (ANSI_1361) text and \M+4 escapes decode
Johab had no decoder: files declaring ANSI_1361 were read as Windows-1252
and \M+4 escapes stayed literal. Hangul syllables are composed from the
three 5-bit jamo indices (KS X 1001 annex 3); symbols and Hanja map onto
KS X 1001 rows decoded as EUC-KR. Read only: a writer escapes instead.
Ported from storytold/cadcraft#1911 (its DOS code pages are covered by the
tables from storytold/cadcraft#1271).
---------
Co-authored-by: mo <131036780+mbolling@users.noreply.github.com>
Co-authored-by: ffishman <20211677+ffishman@users.noreply.github.com>
Co-authored-by: Cédric Legrand <cedric@legrand-tech.fr>
Co-authored-by: valentinevg812-eng <valentinevg812@gmail.com>
Co-authored-by: Mads Holten Rasmussen <mads.h.rasmussen@gmail.com>
Co-authored-by: LastSkywalker <maksim@sky-tehnol.uk>
Co-authored-by: vangel76 <vangel1976@googlemail.com>
Co-authored-by: Francesco BV <francesco.esposito@bvitalia.net>
* craft-tiff: keep shared child IFDs and every string of a multi-string ASCII value
Two pointers to the same child IFD (two SubIFDs entries, or an Exif IFD
shared by two directories) were treated as a loop and the second was
dropped. Loops are now detected on the current parent path only; a shared
IFD is parsed once and copied for later pointers, and each copy is charged
its IFDs, entries and value bytes so max_ifds, max_entries_total and the
value budget still hold.
ASCII values keep every NUL-separated string (Value::as_strings); as_str
and as_bytes return the first.
Ported from storytold/lightcraft#3002.
* craft-tiff: parse_with_status reports whether the whole IFD structure was read
The reader is best effort: truncated tables, over-budget values, dangling
pointers and unfollowed children are skipped silently. A caller that must
tell a clean absence from something it could not inspect (a privacy scan
for GPS data) gets the same tree plus a completeness flag.
Also adds the PHOTOSHOP_IMAGE_RESOURCES and GPS destination tag constants.
Ported from storytold/lightcraft#3766. On top of the shared-child change,
a second pointer to an IFD already read counts as read, and a next pointer
that loops back to the chain is incomplete.
---------
Co-authored-by: Andrew Frahn <andrew@emmertex.com>
Co-authored-by: ronibalthazar <61232335+ronibalthazar@users.noreply.github.com>
color: parses the CSS colours SVG paint uses (hex, rgb()/hsl(), the 148 named
colours) into an opaque 8-bit sRGB Color, and writes the canonical #rrggbb form.
Shape::stroke is still returned as written; parse_color reads it.
write: SvgWriter (XML declaration, root element with the SVG namespace, elements
and attributes escaped through craft-xml, caller-controlled line breaks), path
data in shortest round-trip form or with fixed decimals, and fixed-decimal
numbers that are never NaN or infinite.
A granule outside 0..=MAX_OPUS_GRANULE (negative other than -1, or absurdly
large) no longer moves the packet positions or the total: the page's packets
continue from the previous position. Ported from storytold/filmcraft#4049.
Co-authored-by: vangel76 <vangel1976@googlemail.com>
Each third-party action that StepSecurity publishes a secure drop-in
replacement for now uses that replacement, pinned to a commit with its
version as a comment. Versions follow what we used: the newest
StepSecurity release in the same major, or the closest major they
publish. dtolnay/rust-toolchain refs that named a toolchain (stable,
nightly, 1.76) become the replacement's `toolchain` input.
* metafile: hold a placed pen width to what covers the frame
Picture::place scaled a pen width by the fit with no upper bound, so a
pen of 1e30 units made a renderer flatten round caps of that size. A
placed pen is now at most twice the diagonal of the frame and the path
together, which already covers the whole frame.
Ported from storytold/wordcraft#5427.
* metafile: one byte budget for items, EMF+ objects and clips; count clip regions when placing
- Canvas::shade_shape returns when the canvas is full or the clip is
empty, instead of sampling a tile that is then dropped.
- EMF+ objects, the pieces of a continued object and the current and
saved EMF+ clips count against the canvas's 256 MiB budget, in both
directions: an object is not stored past what the canvas holds, and
the canvas stops at what the objects hold.
- A nested metafile held by an EMF+ image object is charged for what its
items own (segments, pixels, text, clip rectangles; a shared tile or
clip path once) rather than 128 bytes an item.
- Picture::place charges each clip region it expands to the 500,000
element budget, for every kind of item, and expands a region once for
the consecutive items that share it.
Ported from storytold/deckcraft#1777, with its review comments addressed
(the canvas checks include the held bytes; text and bitmap items are
checked too; regions are matched by content, not by address).
* metafile: a negative crop side pads the picture
crop_window took each side as 0..1, so a negative side (a negative
a:srcRect side in an Office file, or a picture fitted whole into a frame
of another shape) was dropped and the picture was stretched over its
frame. A side is now taken down to -100: the window starts before the
source or ends past it, and Picture::fit places the source inside the
frame at its own proportions.
Ported from storytold/wordcraft#2027 (the -100 bound and the invariant
test) and storytold/wordcraft#5206 (the padding test).
* metafile: decode 8-bit text in the font's code page
8-bit text (WMF TEXTOUT and EXTTEXTOUT, EMF EXTTEXTOUTA and SMALLTEXTOUT)
was decoded in Windows-1250, 1251 or 1252 only, and a double-byte
character became a blank.
- Greek, Turkish, Vietnamese, Hebrew, Arabic, Baltic and Thai character
sets decode in their code pages (encoding_rs, already a workspace
dependency).
- Shift_JIS, UHC, GBK and Big5 pairs decode to their character, with the
lead and trail ranges of each set; the character keeps the advances of
both bytes. A lead byte without a trail byte is one undefined byte.
- A WMF face name is read in the font's code page.
- The Symbol font selected with DEFAULT_CHARSET keeps its bytes as symbol
codes (U+F000 + code), as SYMBOL_CHARSET fonts do.
Ported from storytold/wordcraft#5187 (the code pages, the double-byte
ranges, the face name and the Symbol font) and storytold/deckcraft#792
(the Hebrew and Arabic tests).
---------
Co-authored-by: Dmitrii Khlapov <19991457+dkhlapov@users.noreply.github.com>
Co-authored-by: David Lin <woei66@gmail.com>
* craft-media: read and write the QuickTime logs box
A video sample entry's `logs` child holds the CoreMedia log transfer-function
identifier (`com.apple.rec2020.apple-log` for Apple Log). It is kept as
`VideoParams::log_transfer_function`, whatever curve it names.
The reader takes non-empty UTF-8 without NUL of at most 255 bytes; anything
else, or a second `logs` box, reads as unknown. The writer refuses an
identifier the reader would not accept.
Ported from storytold/filmcraft#1764.
* craft-media: the Matroska muxer writes a video track's Colour element
`TrackSpec::color` takes the reader's `Color` and the muxer writes the fields
that are set: matrix, range, transfer, primaries, chroma layout, light levels
and mastering metadata (RFC 9559). VP9's own colour syntax carries only the
matrix and range, so without the element a Rec. 2020, P3, PQ or HLG WebM reads
as BT.709.
Ported from storytold/effectcraft#2926, which wrote the element by hand in
EffectCraft's WebM export.
* craft-media: the laced remux test laces ffmpeg's Vorbis fixture again
ffmpeg's Vorbis files declare a short DefaultDuration and store longer
durations per block, so with the track's own default nothing laced and the
test failed ("vorbis.mkv Xiph: nothing was laced", ffmpeg 9.0.2). Audio tracks
now take their median packet duration as the default, and frames are written
with their explicit durations, so a short last packet ends its lace.
Ported from storytold/filmcraft#3622 (test-only part).
---------
Co-authored-by: 20Coder <51251284+BitL8-ByteShort@users.noreply.github.com>
Co-authored-by: JonRC <jonarc06@gmail.com>
Co-authored-by: Michael Butler <michaelbutler75@gmail.com>
DECODE_EXTENSIONS lists the file extensions of the formats decode reads.
EncodeFormat gains ALL, extension() and mime_type(). SoundCraft kept these
tables in its own FileFormat type; they are format knowledge, so they live
here.
A WAV with many empty data chunks parsed in quadratic time: whether an empty
data chunk is followed by chunks alone was decided for every one of them, by
walking the rest of the file. Only the first data chunk is used, so it is
decided once.
Tests SoundCraft added to audio-io after craft-audio was cut from it: Matroska
element sizes beyond the file, an AIFF channel count with the sign bit set,
probe and decode agreeing on the length of AAC in MP4, and one small file per
decodable extension (tests/fixtures).
* craft-video: HEVC decoder from FilmCraft (`hevc-decode` feature)
FilmCraft's crates/hevc at 4c0d2ad7 becomes craft_video::hevc::decode. The encoder moves to
hevc/encode and is still exported from craft_video::hevc. Feature hevc now turns on hevc-decode
and hevc-encode, so either half can be built without the other.
craft_video::budget (the picture memory budget from DeckCraft's bitstream crate) is added for the
next commit. craft-video is built at opt-level 3 in the dev profile, as FilmCraft built its
codecs.
* craft-video: HEVC decoder hardening from DeckCraft's copy
- The picture store (DPB, current picture, pictures in flight) is checked against a memory
budget before a picture is allocated; Decoder::set_memory_budget.
- configure_hvcc checks every array before handling any NAL: only VPS, SPS, PPS and SEI arrays,
each NAL header agreeing with its array type. A picture slice in the record starts no picture.
- init_qp_minus26, slice_qp_delta and delta_chroma_log2_weight_denom are range checked before
the additions that overflowed on Exp-Golomb values near the i32 limits.
- A picture job sets up its picture state inside the panic guard.
* craft-heif: cap hvcC parameter sets and tiles, count alpha-grid tiles in the visit budget
* craft-heif: charge every tile reference of a thumbnail candidate to the visit budget
* craft-video: VP9 decoder from FilmCraft (`vp9-decode` feature)
FilmCraft's crates/vp9 at 4c0d2ad7 becomes craft_video::vp9::decode. The encoder moves to
vp9/encode and is still exported from craft_video::vp9. Feature vp9 now turns on vp9-decode and
vp9-encode, so either half can be built without the other.
The decoder's tests find ffmpeg and the fixture directory through craft-media-testkit; a fixture
that cannot be generated fails under CRAFT_MEDIA_REQUIRE_ORACLES=1. craft-video is built at
opt-level 3 in the dev profile, as FilmCraft built its codecs.
* craft-video: VP9 decoder checks its picture store against a memory budget
DeckCraft's copy of the decoder refuses a frame size whose pictures (eight reference slots, the
new frame and those in flight) would not fit a byte budget. The budget type comes with it as
craft_video::budget; FilmCraft's 2^26 luma sample cap stays.
* craft-metafile: DeckCraft's WMF, EMF and EMF+ reader with WordCraft's changes merged
The base is DeckCraft's crates/metafile at cf0d7fbb. WordCraft's changes since
the copy was taken (WordCraft 05679497, the ancestor) are merged three ways from
WordCraft 639b8133: the cropped, depth-once DIB decoder, the bitmap side cap,
the header-only size(), crop_window, the free-slot object table, capacity-based
budget charging and the exact reservation in subpaths. WordCraft's own text
records lose to DeckCraft's richer text model. All 120 DeckCraft tests and all
69 WordCraft tests are kept (132 distinct).
* craft-metafile: line styles, clip paths and the records behind them, from VectorCraft's reader
The model gains what VectorCraft's reader kept and the other two dropped:
LineStyle (cap, join, miter limit, dashes) on paths, and clip paths
(SELECTCLIPPATH) shared between the items they clip. The reader now plays
EXTCREATEPEN style entries and the pen's cap and join bits, SETMITERLIMIT,
SETARCDIRECTION, SCALEVIEWPORTEXTEX, SCALEWINDOWEXTEX, SELECTCLIPPATH and
ABORTPATH; STROKEANDFILLPATH closes open figures and CLOSEFIGURE moves the
current position; the header's micrometre device size is used when present;
BITMAPCOREHEADER, BI_JPEG and BI_PNG bitmaps decode; records of unknown types
are counted in Picture::skipped_records. Each has a test.
* craft-metafile: EMF and WMF writers behind the write feature, bitmap transforms, a decode budget
The writers are VectorCraft's, taking a list of drawing operations (fills,
strokes with pens, images, clips) instead of its document; the document walk
stays in the application. Bitmap items gain a transform, so an image drawn
under a turning world transform keeps its turn (the writer places images that
way). JPEG and PNG bitmaps share one per-file pixel budget. The tests cover the
writers record by record, round-trip both formats through the reader, and play
VectorCraft's hand-built files into the picture model.
* craft-metafile: a test per limit, README, crate docs
Adds tests for the limits no test held to their value (string lengths, clip
region rectangles, placed elements, tile side, palette size, nesting, the EMF+
object table, save stack, point lists, gradient stops, region depth, continued
objects, bitmap masking work) and a truncation and mutation sweep over the new
records. Records that change what is drawn but are not played now count as
skipped instead of ignored. The README has the coverage tables, the limits,
what stays in the apps, the origin of each part and a migration table per app.
* craft-metafile: a region node cap test, the box of a turned bitmap
The region node cap gets a test of its own (a balanced tree of intersections),
and a turned bitmap's rectangle is the box around its four corners, so a caller
that draws rectangles still has an area to draw into.
* Add craft-mesh: neutral mesh model with STL, OBJ/MTL and PLY
* craft-mesh: glTF 2.0 and GLB
* craft-mesh: README, crate list entry, a test for the UV set bound
The base is FilmCraft's filmcraft-cfb (reader, writer, tests) at
filmcraft 6d5269ed. Ported on top of it, each with a test:
From GridCraft's and DeckCraft's readers: only sector shifts 9 and 12;
a file that ends inside its last sector still reads; a chain loop
shorter than the stream is an error instead of repeated data; a damaged
mini stream no longer fails the open; links to missing directory
entries are ignored; a stream size cap (Limits, default 1 GiB, and a
per-call cap); a lenient directory scan (directory_names,
has_entry_named) for files too damaged to open.
From WordCraft's guard: a FAT sector listed twice is refused; spare FAT
sectors and zero-padded DIFAT sectors are accepted.
New: Limits::max_dir_entries, walk() and path() to list entries,
has_stream, Writer::stream_at and write_streams, and only storages the
root reaches are walked.
Almost every Craft app carries its own small XML helper: a bounded
element tree on a pull parser and an escaping string writer. This is
the shared one, so each app can drop its copy.
- An owned tree (El, Node) read from bytes with limits on input size,
depth, node count and attributes per element. A limit exceeded is an
error. No DTD is processed and no entity is expanded beyond the five
predefined ones and numeric character references. UTF-8 and UTF-16.
- Names as written, by local name, or rewritten to the caller's
canonical prefixes by namespace URI, with hooks for a prefix that
depends on the parent and for attribute values that hold prefixes.
- Lenient and strict reading; streaming of repeated elements to a
callback; first_start; a pull reader with byte ranges.
- Accessors, typed attribute getters, mutation and a builder. Where
the apps disagreed (whole or local names, own or deep text, exact or
lossy numbers) there are two spellings.
- esc, esc_attr, compact and pretty serialising, and a Writer.
A name that is not an XML name is never written.
Built on quick-xml 0.41's tokenizer (MIT), the version the apps use.
The apps' own tests for their helpers are ported in tests/apps.rs.
No app is switched here.
MailCraft and WordCraft each parse MIME: MailCraft for mail (crates/mime,
crates/charset), WordCraft for MHT web archives (formats/src/mht.rs). This
adds one shared crate for the format, taken from MailCraft's code where it
is format code, with WordCraft's multipart/related lookups and its fix for
comments in Content-Type.
Reader: parse() gives a tree of parts (header fields, content type and
parameters, decoded body, charset-decoded text). It never fails or panics,
is linear in the input, and is bounded by Limits (input size, header count
and sizes, depth, part count, decoded size); Message::truncated reports a
limit that left something out.
Writer: Entity::text / file / multipart and write_message, with folded
headers, RFC 2047 and RFC 2231 encoding, quoted-printable and base64 with
line limits, and boundaries checked against the content. Every header
value loses CR, LF and NUL, and a name that is no field name is not
written.
Also: address lists, mbox splitting and writing on raw message bytes, and
Date parsing and formatting on a plain struct (chrono is a private
dependency; no date type is in the public API).
No app is switched here. The README has the migration tables and says what
stays in the apps.
Packet byte ranges are checked against the source before a buffer is
sized for them (deckcraft 51d19baa), and packets after the last known
granule count towards the Opus duration (deckcraft 9d36b593). Each test
failed before its fix.
Adds step-security/harden-runner (audit, pinned to v2.22.4 like
craft-common-actions) as the first step of each job that did not have it,
so every workflow in the org's public repos reports its egress to
StepSecurity. Audit, not block: block-mode allow lists come from each
job's observed traffic.
* craft-audio: FilmCraft's FLAC encoder and MP1/MP2 decoder speed-ups
FLAC: per-order predictor kernels, merged Rice partition sums, the LPC
window kept per block length, and batches consumed by offset
(filmcraft 5eaff794, d9dad4a5). The output is unchanged: 64 golden cases
hash the same before and after.
MP1/MP2: the faster synthesis filterbank (filmcraft 51c755e5), with its
bit-exact test.
* craft-audio: FilmCraft's AAC and Opus decoder speed-ups and refactors
AAC: table-driven Huffman decoding, per-stage FFT twiddles and fused
stages, reused per-channel work buffers, and the encoder's memoised
per-band work (filmcraft f9627280, d587ceed, 6121635c, dc124ab7,
7a532eca).
Opus: radix-2/3/4 FFT kernels in the CELT IMDCT, the PVQ table built
once, one de-emphasis loop for both channels, the SILK resampler
without per-sample divisions and with eight partial sums, concealment
and FEC capped at 120 ms (filmcraft bf7ac420, 12e32026, c4dc64d1,
eb352ec8, 87b8e22d, 014b5e56, 78d77b72).
The per-frame mid-only flag this crate fixed in the SILK decoder is
kept in the refactored decode.
* craft-media: Ogg CRC-32 by slice-by-16
FilmCraft's 6763f9bf: sixteen independent lookups per 16 input bytes, so
opening a file is several times faster. Checked against a bit-at-a-time
reference on every short length and alignment, and across every split.
* craft-audio: SoundCraft's CAF cookie-size and FLAC vendor-length checks
A CAF magic cookie size above the signed range is refused instead of
passed to symphonia (soundcraft 3bd3965), and a Vorbis comment block
whose vendor string claims more than the block holds is refused
(soundcraft ddfe991). Each with its regression test.
* craft-audio: AC-3 reads bits through the shared BitReader
FilmCraft's 1c9c2e82: the AC-3 decoder's private bit-at-a-time reader is
replaced by craft-media's BitReader, with its two tests: the decoded
output of 2,000 synthetic and 12 real syncframes (and their damaged
variants) hashes the same as before, and the reader agrees with the old
one on random reads, skips and overruns. The three AC-3 features now
turn on craft-media's bitstream module.
* craft-audio: the quantiser rounds without a libm call
SoundCraft's d293336, the audio-io part: round-half-away by integer
arithmetic in the bounce's inner loop, with its test against f64::round
over the quantiser's range.
* craft-audio: flac::Encoder, the FLAC packet encoder, is public
FilmCraft muxes FLAC into its own outputs from integer samples, so it
needs the encoder below FileEncoder: planar samples in, frames out, and
the STREAMINFO to patch in at the end. The module is behind flac-encode,
its errors convert to AudioError::Encode, and pending_samples says how
long the final frame will be.
* craft-audio: FilmCraft's FLAC encoder and MP1/MP2 decoder speed-ups
FLAC: per-order predictor kernels, merged Rice partition sums, the LPC
window kept per block length, and batches consumed by offset
(filmcraft 5eaff794, d9dad4a5). The output is unchanged: 64 golden cases
hash the same before and after.
MP1/MP2: the faster synthesis filterbank (filmcraft 51c755e5), with its
bit-exact test.
* craft-audio: FilmCraft's AAC and Opus decoder speed-ups and refactors
AAC: table-driven Huffman decoding, per-stage FFT twiddles and fused
stages, reused per-channel work buffers, and the encoder's memoised
per-band work (filmcraft f9627280, d587ceed, 6121635c, dc124ab7,
7a532eca).
Opus: radix-2/3/4 FFT kernels in the CELT IMDCT, the PVQ table built
once, one de-emphasis loop for both channels, the SILK resampler
without per-sample divisions and with eight partial sums, concealment
and FEC capped at 120 ms (filmcraft bf7ac420, 12e32026, c4dc64d1,
eb352ec8, 87b8e22d, 014b5e56, 78d77b72).
The per-frame mid-only flag this crate fixed in the SILK decoder is
kept in the refactored decode.
* craft-media: Ogg CRC-32 by slice-by-16
FilmCraft's 6763f9bf: sixteen independent lookups per 16 input bytes, so
opening a file is several times faster. Checked against a bit-at-a-time
reference on every short length and alignment, and across every split.
* craft-audio: SoundCraft's CAF cookie-size and FLAC vendor-length checks
A CAF magic cookie size above the signed range is refused instead of
passed to symphonia (soundcraft 3bd3965), and a Vorbis comment block
whose vendor string claims more than the block holds is refused
(soundcraft ddfe991). Each with its regression test.
* craft-audio: AC-3 reads bits through the shared BitReader
FilmCraft's 1c9c2e82: the AC-3 decoder's private bit-at-a-time reader is
replaced by craft-media's BitReader, with its two tests: the decoded
output of 2,000 synthetic and 12 real syncframes (and their damaged
variants) hashes the same as before, and the reader agrees with the old
one on random reads, skips and overruns. The three AC-3 features now
turn on craft-media's bitstream module.
* craft-audio: the quantiser rounds without a libm call
SoundCraft's d293336, the audio-io part: round-half-away by integer
arithmetic in the bounce's inner loop, with its test against f64::round
over the quantiser's range.
Dylan M. Taylor