mirror of
https://github.com/storytold/craft-libs.git
synced 2026-11-12 08:52:32 +00:00
Dylan M. Taylor
f806e149bb
* Add craft-kiwi: the kiwi binary schema format, decoded and encoded The generic kiwi decoder and encoder from FigCraft's .fig import and export (crates/format/src/fig/kiwi.rs at 3b9bf8c9), moved as it is: binary schema decode and encode, the compact text form of a schema, message decode into a tree keyed by field name, and message encode checked against the schema. The value budget that follows a .fig file's compressed size stays with the .fig reader; here decode_message_within takes the budget and DEFAULT_MAX_VALUES is the floor FigCraft used. * craft-kiwi: no scan per field in schema decode; the text form keeps the limits Schema::decode compared each message field id with every earlier one: 20 definitions of 10 000 fields, within every cap, took a billion comparisons. A set per definition now. Schema::parse applies the definition and field caps of the binary form, looks type names up in an index instead of scanning the lines per field, accepts a struct or message without fields, and refuses two fields of one name and a message field id that is 0 (it ends a message) or repeated.
65 lines
3.1 KiB
Markdown
65 lines
3.1 KiB
Markdown
# craft-kiwi
|
|
|
|
A clean-room decoder and encoder for the kiwi binary schema format: an open, MIT-licensed message
|
|
format with a public specification. No dependencies beyond `std`, no `unsafe`.
|
|
|
|
A binary schema lists definitions (enums, structs and messages, each with typed fields). A message
|
|
is decoded with a schema into a `Kv` tree keyed by field name, and encoded back from one. The
|
|
schema usually travels with the data (`.fig` files carry theirs), so nothing here is generated
|
|
from a schema at build time: a reader looks fields up by name and takes what it understands.
|
|
|
|
## Provenance
|
|
|
|
Written by FigCraft's contributors for its `.fig` import and export, from the format's public
|
|
description only: the published specification of the encoding (variable-length integers, zigzag
|
|
signed integers, floats with the exponent byte first, NUL-terminated strings, messages as
|
|
`field id, value, …, 0`, structs as every field in order). No code of the reference
|
|
implementation or of any other product was copied. Moved here from
|
|
`figcraft/crates/format/src/fig/kiwi.rs`.
|
|
|
|
## Reading
|
|
|
|
```rust
|
|
let schema = craft_kiwi::Schema::decode(&schema_bytes)?;
|
|
let root = schema.find("Message").ok_or("no Message definition")?;
|
|
let message = schema.decode_message(root, &message_bytes)?;
|
|
for node in message.get("nodeChanges").map(craft_kiwi::Kv::list).unwrap_or_default() {
|
|
let name = node.get("name").and_then(craft_kiwi::Kv::str);
|
|
}
|
|
```
|
|
|
|
`Kv` is `Bool`, `Int`, `Uint`, `Float`, `Str`, `Bytes` (a `byte[]` field), `Enum` (the value's
|
|
name; an unknown value stays a `Uint`), `Array` and `Object` (a struct or message: its fields in
|
|
the order read). Field and enum names borrow from the schema.
|
|
|
|
## Writing
|
|
|
|
```rust
|
|
let schema = craft_kiwi::Schema::parse("struct Point float x, float y\nmessage Message Point[] points, string name")?;
|
|
let bytes = craft_kiwi::encode::message(&schema, root, &value)?;
|
|
let schema_bytes = craft_kiwi::encode::schema(&schema);
|
|
```
|
|
|
|
`Schema::parse` reads a compact text form, one definition per line: `enum Name A B …` (values
|
|
from 0), `struct Name type field, type field[] …` or `message Name …` (field ids from 1); `A=3`
|
|
or `type field=7` gives a value or a field its own number. The schema limits below apply to the
|
|
text form too, and it refuses two fields of one name and a message field id that is 0 or repeated. The encoder checks values against the
|
|
schema and fails on what doesn't fit: unknown fields or enum names, missing struct fields, wrong
|
|
types, numbers out of range, NaN.
|
|
|
|
## Limits
|
|
|
|
Input is hostile. Every read is bounds-checked and nothing panics.
|
|
|
|
| Limit | Value |
|
|
|---|---|
|
|
| Nesting of structs and messages (`MAX_DEPTH`) | 64 |
|
|
| Definitions in a schema (`MAX_DEFS`) | 10 000 |
|
|
| Fields in one definition (`MAX_FIELDS`) | 10 000 |
|
|
| Fields in a schema (`MAX_SCHEMA_FIELDS`) | 200 000 |
|
|
| Values in one message (`DEFAULT_MAX_VALUES`; `decode_message_within` takes another budget) | 20 000 000 |
|
|
|
|
A value is every scalar, array element, array field and nested definition. An array count larger
|
|
than the bytes left is refused before anything is allocated, a `u32` varint that overflows is
|
|
refused, and a message definition with two fields of one id is refused as ambiguous.
|