WAI Extension: Packaging

Mirrored from the canonical text at commit 0c6a8897 (2026-10-06).

Status: Draft. A packager turns sources into WAI objects under a declared policy; this extension fixes what a verifier recomputes. This revision defines the policy, its digest and what it may ask a packager to emit (§3), how a source is cut into pieces (§4), and the integer video payloads a packager writes (§5.5). Reference impl: wai-pack — policy, capcheck, source, scenecut, classify; the encoder wai-rs int_video_enc; the corpus pack-conformance/. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

1. Scope

A packager turns one or more sources into WAI objects, stage indexes, catalogs and a package receipt, under a declared policy. This extension fixes what a package contains and what a verifier recomputes. It fixes an encoder’s choices only where it names them.

2. Terms

• Component: one media stream of a package (for example video, or audio).
• Piece: a span of a component that decodes with no reference outside itself.
• Stage: one object of a piece in staged order: the base is layer 0, refinement k is layer k (staged-delivery).
• Rung: one entry of the policy’s ladder: a capability a component’s pieces are coded in, with its role (base, refinement or floor) and parameters.
• Class: a delivery profile the policy names.
• Moment: a declared span of presentation time with tags.
• Floor: a rendition in a standard codec, for sinks that lack the primary.
• Timebase of a component: num / den units per second. For video one unit is one frame; for audio, one sample.

3. Policy

3.1 A package is made under exactly one policy, a TOML 1.0 document of schema wai.pack.policy/1, in UTF-8 with no byte-order mark.

3.2 The effective policy is the policy with every default written in (§3.3) and every override the packager applies (such as a command line forcing royalty_free_only). Its JSON form is §3.3’s members, with an optional member that has no default omitted when absent, and arrays of tables in document order. Its policy digest is

BLAKE3("wai:pack-policy\x01" ‖ JCS(effective policy as JSON))

where JCS is RFC 8785. Two spellings of one policy (member order, inline or standard tables, dotted keys, string or integer notation, comments, an omitted default or the default written out) therefore have one digest; the order of an array of tables is part of the policy.

A packager MUST refuse with policy-toml, before anything else, a policy whose bytes are not UTF-8, that begins with a byte-order mark, that is not TOML 1.0 (an integer outside TOML’s 64-bit range included), or that nests arrays and tables more than 32 deep: the document’s own table is at depth 1, and an array or a table directly inside one at depth k is at depth k + 1. Then, before it reads any key, it MUST refuse one holding anywhere a float (policy-float), a date or time (policy-datetime), a negative integer (policy-negative), or an integer above 2^53 − 1 (policy-integer-range), visiting members in byte order of their keys and arrays in order.

3.3 Schema. Within each table, a key this section does not define is refused first (policy-unknown-key, the first in byte order), then each member in the order listed: a required member that is absent (policy-missing-key), a member of another type (policy-type), a value outside its range or vocabulary (policy-value). An absent table reads as empty.

A rung’s params are those of its capability; every other capability takes none.

wai.audio.int_layered is reserved for the layered integer audio base: it is not registered yet, and its parameters are defined when it is.

3.4 Constraints. A policy that passes §3.3 is checked in this order, and the first that fails is the refusal:

1. 1 to 8 components (policy-too-many-components); at most 64 rungs (policy-too-many-rungs), 32 classes (policy-too-many-classes), 32 rules (policy-too-many-rules) and 32 moments (policy-too-many-moments).
2. Ids unique among components, rungs, classes, moments and rules taken together, so a reference never names two things (policy-duplicate-id).
3. lead names a component (policy-lead).
4. segmentation.closed is true: every piece is closed, §4.1 (policy-open-gop).
5. 1 ≤ min_ms ≤ target_ms ≤ max_ms (policy-piece-bounds).
6. Every rung names a component (policy-ladder-component).
7. A rung carries refines exactly when its role is refinement (policy-refines-role); refines names a rung (policy-dangling-refines); no chain of refines returns to a rung it passed (policy-refines-cycle); every chain stays in its component and ends at a base (policy-refines-base); no rung is refined twice (policy-refines-branch).
8. Every component has a base rung (policy-no-base).
9. A rung’s layer is 0 for a base or a floor, and one more than the rung it refines for a refinement; the policy has one more layer than its highest rung layer. No class’s and no rule’s max_layer exceeds the highest (policy-class-layer).
10. A rule’s then.ladder names known rungs, all of one component, at least one of them a base (policy-rule-ladder).
11. live.layer_lag_ms has one entry per layer, the first 0, the entries non-decreasing (policy-layer-lag).
12. The deadline values (defaults.deadline_ms and every rule’s then.deadline_ms) number at most 4 (policy-too-many-deadlines), and for every refinement layer k and every deadline value d, d − layer_lag_ms[k] ≥ 1 (policy-deadline-unreachable).

Rules fold in document order: every matching rule applies, and a later then member overrides an earlier one.

3.5 A package receipt carries the policy digest. A verifier holding the effective policy recomputes it; it cannot check that the packager followed the policy, except by packing again.

3.6 What a packager may emit. A packager MUST refuse a policy with a rung it may not emit, reading SPEC §5’s registry, never the policy. For each rung in ladder order the first rule that holds is its refusal, and a policy’s refusal is its first rung’s:

1. SPEC §5 does not register the capability: capability-unknown:<cap>.
2. SPEC §5 registers it as a record or a derivation operation, which names no payload format: capability-not-payload:<cap>.
3. It is a candidate, whose payload format no revision has pinned (SPEC §5, Candidate capabilities): capability-candidate:<cap>.
4. A base or a floor whose capability is not standalone media, or a refinement whose capability is not refinement-only: capability-role:<cap>.
5. Under royalty_free_only, a capability the registry gives no licence class: license-unclassified:<cap>; one of any class other than royalty-free: license-refused:<cap>:<label>, with the class’s SPEC §5 label. The class is the registry’s, “licensed royalty-free by its standards body”; it is not an opinion on freedom to operate, and a package states nothing about patents.

4. Pieces

4.1 Every piece MUST be closed: its first frame, sample or tick is coded with no reference outside the piece, and no later object of the package references into it.

4.2 Piece boundaries are a pure function of the source frames and the effective policy. Where the policy names wai.pack.scenecut/1, the boundaries MUST be those §4.3 computes. The list of piece starts, in timebase units, is the plan, and its digest is

BLAKE3("wai:pack-plan\x01" ‖ u64 LE n ‖ n × u64 LE start)

4.3 wai.pack.scenecut/1. All arithmetic is integer; >> is a shift and every division rounds down. For each RGB8 frame f of w × h pixels:

1. Luma Y = (77R + 150G + 29B + 128) >> 8, and the histogram H_f of 64 bins, Y >> 2.
2. D(a, b) = (Σ_i |H_a[i] − H_b[i]|) · 1000 / (2wh): the share of pixels, in thousandths, whose bin differs. The score of frame f ≥ 1 is s_f = D(f, f − 1).
3. The current piece starts at frame p, and len = f − p. Its window W_f is the scores s_g for p < g < f, the last cut_window of them; m_f is their lower median (sorted ascending, the element at index (|W_f| − 1) / 2).
4. Hard cut at f: len ≥ min, s_f ≥ cut_abs, and either |W_f| < 3 or s_f · 1000 ≥ cut_rel · m_f.
5. Soft cut at f: len ≥ target and 2 · s_f ≥ cut_abs.
6. Return clause. A cut at f is rejected when D(f + k, f − 1) < cut_abs for some k in 1..=min(L, last − f): the content came back within L frames, so f starts a flash.
7. Revisit clause. A cut at f is rejected when D(f, f − 1 − j) < cut_abs for some j in 1..=min(L, f − 1): f returns to content seen within L frames, so it ends a flash. Frames before p count.
8. A hard or soft cut that clauses 6 and 7 do not reject starts a piece at f. Otherwise, when len ≥ max, a piece starts at f.

Frame 0 starts the first piece. L is lookahead_frames, last the index of the source’s last frame, cut_abs and cut_rel the policy’s permille values, and min, target and max the policy’s milliseconds on the component’s timebase (§4.4). A decision about frame f reads no frame after f + L, so a live segmenter decides it once frame f + L has arrived, and a live and an on-demand segmenter with equal L produce equal boundaries. A piece that follows a cut is flagged a scene cut; one started at max, forced.

4.4 A piece’s duration is an integer number of timebase units, and lies between min and max except the last piece of a source, which may be shorter than min. A duration in milliseconds converts to timebase units as floor(ms · num / (1000 · den)); a result of 0 becomes 1, and one above 2^64 − 1 becomes 2^64 − 1.

4.5 Components. Every component of a package shares one presentation timeline that starts at 0. The policy names one lead component; every other component cuts its pieces at the lead’s boundary instants, each converted to its own timebase as floor(start_lead · num_c · den_lead / (den_c · num_lead)) (above 2^64 − 1, 2^64 − 1). A sink presents piece p of every component from that start.

4.6 Classification. Each video piece has a kind, which rules match (rule.when.piece_kind). Over the piece’s consecutive frame pairs, with Y the luma of §4.3:

• mad_milli = Σ |Y_f − Y_{f−1}| · 1000 / (wh · (n − 1)), and 0 when the piece has one frame (n = 1);
• changed_permille = the pixels whose luma differs, · 1000 / (wh · (n − 1)), and 0 when n = 1;
• occupancy = the non-empty histogram bins of the piece’s first frame.

The kind is the first that holds of: static (mad_milli ≤ static_mad_milli and changed_permille ≤ static_permille); synthetic (occupancy ≤ synthetic_bins); motion_high (mad_milli ≥ motion_high_milli); natural.

4.7 Sources.

• Y4M: 8-bit 4:2:0 (C420jpeg, C420paldv, C420mpeg2, C420, or no C parameter) or 4:4:4 (C444), progressive. A header that is not YUV4MPEG2 with a width and a height is refused y4m-header; an interlaced stream or frame, y4m-interlaced; another colour space, y4m-colorspace; a FRAME line that sets the width, height or colour space, y4m-size-change; a frame that ends early or a line that is not FRAME, y4m-truncated. Each pixel converts to RGB8 by BT.709 limited range, with 4:2:0 chroma taken at (y >> 1, x >> 1):

  • R = clip((298(Y−16) + 459(Cr−128) + 128) >> 8)
  • G = clip((298(Y−16) − 55(Cb−128) − 136(Cr−128) + 128) >> 8)
  • B = clip((298(Y−16) + 541(Cb−128) + 128) >> 8)

  where >> is the arithmetic shift and clip clamps to 0..=255.

• WAV: RIFF/WAVE with a PCM fmt chunk (format tag 1) of 16-bit samples, block alignment 2 · channels. Anything else is refused wav-format; a chunk that ends early, wav-truncated.

5. Objects

5.1–5.4 Reserved: an object’s identity, its independence of its position in a package, its name, and refines on a refinement object are specified with the packager stage that writes objects.

5.5 Integer video. An object of wai.video.int_motion carries a WIV1 payload with keyframe 0, as SPEC §5 pins it (integer-payloads §3); this extension defines no other layout for it, and its decode is that section’s. The reference packager does not write objects yet; when it does, it will write these with one encoder (wai-rs int_video_enc) at the step §5.6 chooses. The encoder’s choices make a payload a function of the source frames, the rung’s block, search and mv_precision (§3.3), and the step q:

• Motion search (frames 1 on): blocks in raster order; for each, every vector (dy, dx) with both components in −R..=R, R = search · mv_precision (at most 127), dy outer and dx inner, each ascending. Its cost is the sum, over the block’s pixels inside the frame and the three channels, of |source − prediction|, where the prediction is the decode’s (integer-payloads §3) from the previous frame as decoded. The least cost wins; a tie goes to the smaller |dy| + |dx|, then to the earlier vector.
• Residual: each frame, each pixel in raster order, each of R, G, B: the prediction (128 for frame 0) is subtracted from the source and divided by q, rounding halves away from zero; the frame as decoded is clamp(prediction + r·q, 0, 255), and the next frame predicts from it.
• Table: one residual table for the clip, with offset the least residual, top the greatest, and h_i the count of residual offset + i, for n = min(top − offset + 1, 4095) symbols. With R = 2^16 − n − 1 and H = Σ h_i, symbol i gets frequency 1 + ⌊R·h_i / H⌋; the R − Σ⌊R·h_i / H⌋ left over go one each to the symbols with the largest R·h_i mod H, the lower symbol first on a tie; the escape gets 1. A residual past the table’s span is coded through the bypass escape (integer-payloads §2).
• A clip of one frame is written with block 1 and mv_precision 1, as the format requires.

The encoder also states the SSE of its reconstruction against the source: the sum of squared differences over every sample. It codes in two passes (the first finds the vectors, the reconstruction and the residuals’ counts; the second derives each residual again and codes it last first), so beyond its input it holds the reconstruction, the vectors, the stream and the residual table: at most n·h·w·3 + 24n + 12·v + 3·(2·n·h·w·3 + 16) + T bytes for n frames and v motion vectors (int_video_enc::working_bytes). T is the widest table’s, of S = 2·255 + 1 symbols: its fit, 16·(S + 1) + 2·8·S + 4·(S + 2) bytes (a frequency for each symbol and the escape, an index for each symbol and the sort’s scratch, the table), and the table’s three copies and the payload’s header, 3·4·(S + 2) + 64. For a clip past the format’s caps, which the encoder refuses before it allocates, the figure is 2^64 − 1.

5.6 Choosing the step. For each piece, the base encode of a wai.video.int_motion rung chooses its step from q_candidates, read as a set (a value listed twice is one candidate). It encodes the piece at each candidate, from the largest down, and chooses the first whose floor(SSE · 1000 / (3·w·h·n)) ≤ max_sse_px_milli, where SSE is the encoder’s (§5.5) and n the piece’s frames: the largest step that meets the bound, so candidates that meet it alike are decided by their step. Where no candidate meets it, it chooses the smallest. The function the reference packager’s base encode calls is wai_pack::encode::choose_q.

6–11. Stage index, floor, classes and delivery, receipt

Reserved. This revision does not yet define a package’s stage index, its floor, its class plans and catalogs, its receipt, royalty-free packaging of a whole package or re-pack equivalence; they are specified with the packager stages that write them.

12. Reasons