WAI Extension: Live Composition

Mirrored from the canonical text at commit f871af55 (2026-10-20).

Status: Draft. Registers the record wai.live.compose: the deterministic composition of a live room’s participants. A signed composition descriptor fixes a canvas, a background, a layout (a grid, with a featured tile, or a grid under a picture-in-picture window), the grant attributes it reads and its limits. The participant set at a room-clock instant is derived from the room’s lifecycle replay (live-rooms §9): who publishes, each one’s video and audio source and its dimensions, the attributes of its grant in effect, and its join order. The layout is then a pure integer function of the descriptor and the set, and the mix a pure integer function of decoded PCM, so every sink composes the same rectangles locally, with no server encode, and a server compositor that publishes a composed track signs a composition receipt that anyone can check against the same function. No new transport: a composed track is a track of the room, and the receipt names the objects it was made from. The reference implementation is the live_compose module of wai-rs (feature live_compose); the corpus is live-conformance/compose/, and its second implementation is live-conformance/live_compose_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: draft-ietf-moq-transport-22 (MOQT), an Internet-Draft not past Working Group Last Call, for the Full Track Name (§2.4.1) and the Location (§8.2); RFC 6716 §6; RFC 8032; FIPS 180-4; and IEC 61966-2-1 (sRGB). A section number of that draft in this text is that revision’s.

1. Scope

In scope: the composition descriptor, its digest and signature, its limits and who may sign it (§3); the participant set at an instant, the attributes the descriptor reads, the order, and the set’s digest (§4); the layout function (§5); the audio mix (§6); where composition runs (§7); the composition receipt and its verification (§8, §9).

Out of scope: a transport; how a renderer scales and filters a source into its rectangle, and how it converts the background’s colour into its output’s (the layout fixes rectangles, not pixels); decoding, resampling and encoding; and the browser compositor that renders a layout in a page, which a later slice builds from this function.

Relationship to the core spec: wai.live.compose is a record (SPEC §5): a signed record a WAI tool emits and verifies. It names no payload format and is not dispatchable.

2. Terms

• Room, room clock, principal, event, chain, grant, grant in effect, room key, merged order, slot, publish profile: as in live-events §2 and live-rooms §2, §8, §9.2.
• Room authority: the key that signs the room’s descriptor (live-rooms §2).
• Compose power: bit 2 of a capability set’s powers (live-rooms §3.3).
• Canvas: the picture a composition makes, width × height pixels.
• Source: the track a participant is composed from (§4.2).
• Compositor: the principal that publishes a composed track (§7).
• Rectangle: (x, y, w, h): its left edge and top edge, in pixels from the top left of the canvas (or of a source), and its width and height.
• A short field is a u8 length and that many bytes. Integers are little-endian. A digest is SHA-256 (FIPS 180-4), 32 bytes.

3. The composition descriptor

3.1 Layout

A grid’s pip_attr is empty and its corner, pip_width, pip_height, offset and behaviour are 0 (§3.3).

3.2 Digest and signature

The digest of a descriptor is SHA-256 over its record from magic through max_voices: every byte but sig. The signature is Ed25519 (RFC 8032) by signer over the 17 bytes wai:live-compose (ASCII) and 0x01, followed by the bytes the digest covers. It is verified strictly, as SPEC §7 Signatures requires: a key that fails step 1 or 2 is bad_key, and a signature that fails a later step is bad_signature.

3.3 Reading and field rules

A reader reads exactly the bytes it is given as one descriptor, in the order of §3.1, and refuses at the first of these that holds:

1. Fewer than 4 bytes: truncated. A magic other than WCD1: malformed. No byte after the magic: truncated. A version other than 1: unknown_version.
2. The bytes end before a field the layout requires: truncated.
3. Bytes after sig: malformed.

It then holds the canvas to §3.4, and refuses one beyond its limits as canvas_out_of_range. It then refuses, as compose_invalid, a descriptor that breaks any of these rules:

• layout is 1 or 2, aspect at most 3, fill at most 3, and omit_stopped 0 or 1.
• Each attribute key is empty or a slot name of live-rooms §2 (1 to 32 bytes of a–z, 0–9 and _), which is what a grant attribute’s key is (live-rooms §4.4).
• A grid has an empty pip_attr, and corner, pip_width, pip_height, offset and behaviour 0. A picture in picture has a corner of 1 to 4 and a behaviour of 1 or 2, an offset below both width and height, pip_width + offset at most width, and pip_height + offset at most height.
• channels is 1 or 2, and sample_rate 8 000 to 192 000.
• max_tiles and max_voices are each 1 to 64.
• With k the least integer whose square is at least max_tiles: gap is below height; width is at least k + (k − 1) × gap; and the band under a featured tile (§5.4), height − gap − ⌊(height − gap) × 3 / 4⌋, is at least k + (k − 1) × gap. Every grid the layout makes then has a candidate whose cells are at least a pixel each way (§5.3).

Then the signature (§3.2): bad_key or bad_signature.

3.4 The canvas

width and height are each even, and each 16 to 1920; width × height is at most 2 073 600, the pixels of 1920 × 1080; and fps is 1 to 60. The edges are even so that a picture subsampled 4:2:0 has whole chroma samples. A portrait 1080 × 1920 and a square 1440 × 1440 canvas are within these limits.

3.5 Who may sign

Against a room, a descriptor that reads (§3.3) is refused as wrong_room when its room is not the room id; and as compose_not_permitted unless its signer is the room authority, or a principal present at effective whose grant in effect then (§4.1) has the compose power.

The power is checked at effective, when the descriptor takes effect: a descriptor does not lapse when its signer leaves or later loses the power. A composition has one descriptor; a room that changes its layout signs another, and each composed track and each receipt names the one it follows (§8).

4. The participant set

4.1 The room at an instant

A composition reads a room that verifies (live-rooms §9): one refused is refused here with the room’s reason, at room. A room whose outcome is a gap is composed only when no gap is in a room key’s chain: the lifecycle is then whole, and a gap in another principal’s chain (a chat not delivered) changes no room state. A gap in a room key’s chain is lifecycle_incomplete, at room: the state at no instant after it is known.

The room at t is the room state (live-rooms §9.3) replayed from the lifecycle events whose t is at most t, in the merged order: equally, the room each chain gives when it is cut before its first record whose t is above t. Events at t itself are applied, so a principal that joins at t is in the set at t.

The join order of a principal present at t is the number of the join that made it present (its last), where the room’s accepted join events up to t are numbered from 0 in the merged order, a duplicate (live-events §6.2 step 1) counted once.

4.2 Participants and their sources

The participants at t are the principals present at t, other than the descriptor’s compositor, that publish at t: each has an own track of any media kind published, or is the live publisher of a slot (live-rooms §7.1). A standby is not composed for its place in a slot, since its media is not delivered.

Each participant has:

• its video source: the first of its own video track; its own screen track; and its live place in a slot of media kind video or screen, the first such slot in the room descriptor’s order. With none, its video is stopped.
• its audio source: its own audio track; or else its live place in the first slot of media kind audio. With none, it has no voice.
• the attributes of its grant in effect at t (live-rooms §4.4).

A source’s profile is the profile its track was published under (the publish or takeover that made it, live-rooms §7.2), and its layers the number of layers published. Its composed layer L is the lower of layers − 1 and the descriptor’s layer, and its width and height are those of layer L of the profile (live-rooms §8.1); an audio source’s are 0. A descriptor whose layer is below a publisher’s top layer composes from a layer the compositor’s grant can subscribe to, and every sink lays the room out in that layer’s pixels.

4.3 Attributes

For each participant, each attribute key the descriptor names (not empty) is looked up among the attributes of its grant in effect, and read:

A decimal integer is 0, or an optional - followed by 1 to 10 of the digits 0–9 of which the first is not 0: no +, no space, no -0. Every attribute the descriptor names is read for every participant, whether or not it has the source the attribute is about; one whose value is not of its row is attr_malformed, and the composition at t is refused, at set. The keys featured, order and pip that live-rooms §4.4 names are these attributes under the keys a descriptor gives them; gain and pan are the mix’s.

4.4 Order

The participants are ordered: those with an order first, by order ascending, then those with none; between two of one order (or none), by join order ascending; and between two of one join order, by key, as unsigned bytes. This order is S.

4.5 The set’s digest

The digest of the participant set at t is SHA-256 over the 21 bytes wai:live-compose-set (ASCII) and 0x01, the room id, t (u64), the number of participants (u32), and then, for each participant in ascending order of key:

The grant’s digest fixes its attributes, so the set’s digest fixes everything §5 and §6 read of the room at t.

5. The layout

5.1 Integers

Every quantity of this section is an integer, and none is negative. ⌊a / b⌋, for b above 0, rounds down. fit(bw, bh, a, b), the largest rectangle of aspect a:b within bw × bh, is (bw, max(1, ⌊bw × b / a⌋)) when bw × b ≤ bh × a, and (max(1, ⌊bh × a / b⌋), bh) otherwise. A rectangle of w × h centred in (x, y, W, H) is (x + ⌊(W − w) / 2⌋, y + ⌊(H − h) / 2⌋, w, h). No product here exceeds 2^63.

5.2 Which participants have tiles

From S (§4.4):

1. V is S, less every participant whose video is stopped when omit_stopped is 1.
2. For a picture in picture, the PiP participant P is the first of V whose pip_attr attribute is true; with none there is no PiP window. For a grid there is none.
3. G is V without P, in order. The featured participant F is the first of G whose featured_attr attribute is true; it is moved to the front of G.
4. G is cut to its first max_tiles members, less one when there is a P. F, being first, is kept; when G is then empty there is no F.

5.3 The grid

A grid of n ≥ 1 tiles over a region (X, Y, W, H) is chosen by its reference aspect a:b: 16:9 for auto and video, 1:1 for square, 3:4 for portrait.

• Candidates. For each number of columns c from 1 to n, with r = ⌈n / c⌉ rows: c is a candidate when (c − 1) × gap is below W, (r − 1) × gap below H, and the cells, cw = ⌊(W − (c − 1) × gap) / c⌋ wide and ch = ⌊(H − (r − 1) × gap) / r⌋ high, are each at least 1. Its area is tw × th, where (tw, th) = fit(cw, ch, a, b).
• The choice. The candidate of the largest area; of several, the one with the most columns. §3.3’s rule on gap makes c = ⌈√n⌉ a candidate for every n a descriptor allows.
• Placement. The rows are centred: the first row’s top is y0 = Y + ⌊(H − (r × ch + (r − 1) × gap)) / 2⌋. Tile k, from 0, is in row i = ⌊k / c⌋ and column j = k mod c. Every row holds c tiles but the last, which holds n − (r − 1) × c; a row of m tiles is centred, beginning at x0 = X + ⌊(W − (m × cw + (m −
  1. × gap)) / 2⌋. Cell k is (x0 + j × (cw + gap), y0 + i × (ch + gap), cw, ch).
• The tile. Under auto, tile k is its cell; otherwise it is fit(cw, ch, a, b) centred in its cell.

With no tile the canvas is its background.

5.4 The featured tile

With F and n ≥ 2 members of G, the featured tile is (0, 0, width, mh), with mh = ⌊(height − gap) × 3 / 4⌋, and the other n − 1 members are a grid (§5.3) over the band below it, (0, mh + gap, width, height − mh − gap). The featured tile is its whole region: aspect is the grid tiles’. With F alone the featured tile is the whole canvas. With no F, the members of G are a grid over the whole canvas.

5.5 Picture in picture

With P and a G that is not empty, G is laid out over the canvas as §5.3 and §5.4 lay it out, and P’s window is drawn over it. With P and an empty G:

• static: P’s window, and the background;
• dynamic: P is laid out as though it had no pip_attr attribute: a grid of P alone, or, when P is featured, the featured tile, the whole canvas.

The window is pip_width × pip_height when both are above 0. Otherwise, with a:b the width and height of P’s video source, or the canvas’s when P’s video is stopped, it is fit(bw, bh, a, b), where bw is pip_width when that is above 0, width − offset when only pip_height is, and the lower of ⌊width / 4⌋ and width − offset when neither is; and bh is pip_height when that is above 0 and height − offset otherwise. With o the offset and pw × ph the window, it is at (o, o) in the top left corner, (width − o − pw, o) in the top right, (o, height − o − ph) in the bottom left, and (width − o − pw, height − o − ph) in the bottom right.

5.6 Fill

A tile (tx, ty, tw, th) shows a source of sw × sh by its fill mode. The source is wider when sw × th ≥ sh × tw.

• Fill: the whole source, (0, 0, sw, sh), stretched over the whole tile.
• Cover: the source cropped to the tile’s aspect, over the whole tile. A wider source’s crop is cw = max(1, ⌊sh × tw / th⌋) wide, at (⌊(sw − cw) / 2⌋, 0, cw, sh); any other’s is ch = max(1, ⌊sw × th / tw⌋) high, at (0, ⌊(sh − ch) / 2⌋, sw, ch).
• Contain: the whole source, letterboxed in the tile. A wider source is drawn into (tx, ty + ⌊(th − dh) / 2⌋, tw, dh) with dh = max(1, ⌊tw × sh / sw⌋); any other into (tx + ⌊(tw − dw) / 2⌋, ty, dw, th) with dw = max(1, ⌊th × sw / sh⌋).
• Default: cover for a grid tile and the PiP window, contain for the featured tile.

Each mode gives a source rectangle and a destination rectangle: the part of the source drawn, and where on the canvas it is drawn. A tile of stopped video has neither, and shows the background, as does the part of a tile outside its destination. How the source rectangle is scaled into the destination is the renderer’s; a sink that holds a source at a layer below its composed one scales it to the composed layer’s width and height first.

5.7 What the layout gives

The layout of a descriptor and a participant set gives:

• the order: S, every participant, by key;
• the tiles, in drawing order: the featured tile, then the grid’s in G’s order, then the PiP window. Each has its participant, its place (grid, featured or pip), its z (0, or 1 for the PiP window, drawn over the others), the tile, and, unless its video is stopped, its source and destination rectangles;
• the voices (§6).

A renderer paints the background over the canvas, then the tiles in order. No two tiles of z 0 overlap.

5.8 Re-layout

The participant set changes only at the t of a lifecycle event: a join, leave, publish, unpublish, grant_exchange, takeover or failover (an exchange always changes it, since the set names each participant’s grant). At every such instant at or after effective the layout is computed again, and it changes when the sources, the attributes or the order do; between two such instants it does not change. The layout at t is the function of this section over the descriptor and the set at t, and of nothing a sink sees otherwise, so a sink that joins late lays the room out as one that was there throughout, and a grant exchange that changes an attribute re-lays the room out at the exchange’s tick, with no participant leaving.

6. The audio mix

The voices are the members of S (before any is omitted for its video) that have an audio source, in S’s order, the first max_voices of them. Each has its gain g, from 0 to 16 384 in units of 1/4096 (4096 is unity), and its pan p, from −4096 (left) to 4096 (right) (§4.3).

The mixer mixes one block of N frames. Each voice’s input is its decoded audio for the block: N frames of signed 16-bit samples, of 1 or 2 channels (interleaved, left first), at the descriptor’s sample_rate. Each output sample is a sum over the voices in signed 64-bit integers, one term per voice:

• mono output (channels 1): x × 4096 × g for a mono voice’s sample x; (xL + xR) × 2048 × g for a stereo voice’s samples xL and xR.
• stereo output (channels 2), with aL = 4096 − max(p, 0) and aR = 4096 + min(p, 0): to the left channel x × g × aL and to the right x × g × aR for a mono voice; xL × g × aL and xR × g × aR for a stereo one.

The output sample is ⌊(s + 2^23) / 2^24⌋ of the sum s, the arithmetic shift right by 24 bits of s + 2^23, which rounds half up; then saturated once to −32 768 … 32 767. A term is at most 2^41 in magnitude and a mix has at most 64 voices, so the sum is exact, the order of its terms changes nothing, and saturation applies to the whole once. The pan is a balance: at the centre both channels are at the voice’s gain, and panning attenuates one channel only. A single voice at unity gain and the centre is passed through unchanged. A mixer that holds no decoded block of a voice for the output mixes it as silence.

A mixer is refused as mix_input_malformed when an output has other than 1 or 2 channels, or an input has other than 1 or 2 channels, has other than N frames, or a gain or pan out of range.

The mix is byte-exact over the decoded PCM: two mixers given the same blocks give the same samples. It is not byte-exact over what precedes and follows it. A decoder of a lossy codec need not reproduce another decoder’s PCM bit for bit (RFC 6716 §6 holds a decoder to a quality measure, not to equality), and a lossy encode of the mix is not byte-reproducible: two encoders, or one encoder under two settings, give different bytes. The receipt (§8) therefore binds the input objects and the output object, not samples.

7. Where composition runs

On each sink, by default. A sink composes the room locally from the tracks it subscribes to, by §5 and §6, with no server decode or encode. It needs no power: the descriptor states how the room is laid out, and laying it out is reading it.

On a server, under the compose power. A principal that publishes a composed track (its own video track for a layout, its own audio track for a mix, live-rooms §6.1) MUST be the descriptor’s compositor, MUST hold the compose power in its grant in effect, and MUST sign a receipt (§8) for each object of a composed track it publishes. It is not a participant of its own composition (§4.2), so a composed track is never composed again. A subscriber subscribes to a composed track as to any other; the descriptor and the receipts let it check that the track is the room’s composition.

8. The composition receipt

8.1 Layout

An object is 80 bytes: track (32), the digest live-events §3.1 gives an anchor’s track, over the track’s namespace as live-rooms §6.2 renders it, one zero byte, and its name; group and object (u64 each), its MOQT Location (MOQT §8.2); and digest (32), the SHA-256 of the object’s payload. An input whose digest is 32 zero bytes states that the compositor held no object of that track for the output: it drew the tile with no video, or mixed the voice as silence.

The digest of a receipt is SHA-256 over its bytes before sig; its signature is Ed25519 by compositor over the 25 bytes wai:live-compose-receipt (ASCII) and 0x01, followed by the bytes the digest covers, verified strictly (SPEC §7 Signatures). A reader reads exactly the bytes it is given, as §3.3 steps 1 to 3 read a descriptor with WCR1 for WCD1 (truncated, malformed, unknown_version); refuses as receipt_invalid a receipt whose media is not 1 or 2, or that names two inputs of one participant and one track; and then verifies the signature (bad_key, bad_signature).

8.2 The inputs

The inputs of an output at t are, in this order:

• video (media 2): for each tile of the layout at t that shows video, in drawing order (§5.7), the tracks of its participant’s video source;
• audio (media 1): for each voice, in order (§6), the tracks of its audio source.

The tracks of a source are named as live-rooms §6.1 names them: the namespace is the room, the owner (the participant’s key, or _ and the slot’s name for a slot) and the media kind; the names follow the profile’s layering (live-rooms §6.3) at the composed layer L: track 0 for single and svc (an svc track carries every layer), track L for simulcast, and tracks 0 to L for staged, since a refinement is applied to the layers below it (staged-delivery). The receipt names, for each, the object it read.

8.3 Verifying a receipt

A receipt verifier is given a room (as live-rooms §9.1), a descriptor and a receipt. In this order, the first that fails names the refusal and where:

1. The room (§4.1): its refusal, at room.
2. The descriptor (§3.3, §3.5): at descriptor.
3. The receipt reads (§8.1), at receipt; and its room is the room id, otherwise wrong_room.
4. Its descriptor is the descriptor’s digest: otherwise descriptor_mismatch.
5. Its t is at or after the descriptor’s effective: otherwise not_yet_effective.
6. The descriptor names a compositor, the receipt’s compositor is it, and that principal is present at t with a grant in effect that has the compose power: otherwise compositor_not_permitted.
7. Its output’s track is a track of the compositor’s own track of the receipt’s media kind, published at t, at a layer from 0 to its layers − 1: otherwise output_not_published.
8. The set and the layout at t (§4, §5): their refusal, at set.
9. Its state is the set’s digest: otherwise state_mismatch.
10. Its inputs, each its principal and its object’s track, are §8.2’s, in order: otherwise inputs_mismatch.

A receipt that passes is accepted: its output object was composed under this descriptor, from this set, out of objects of exactly the tracks the layout reads. Anyone holding the room’s events and the descriptor recomputes the layout and checks that the inputs were the right ones; a party holding the objects checks their digests. The receipt does not fix the output’s pixels or samples (§6).

9. Verification

A composition verifier is given a room, a descriptor and a list of instants. It runs these steps and stops at the first refusal, which it reports with the reason and where:

1. The room (§4.1): at room, with the room’s own place (live-rooms §9.1) when the room is refused.
2. The descriptor (§3.3, §3.5): at descriptor.
3. Each instant, in the list’s order: the instant is at or after effective (otherwise not_yet_effective), and the participants’ attributes read (§4.3, otherwise attr_malformed): at set, with the instant.

Its outcome is refuse, with the reason and the place; or the composition: the descriptor’s digest and, for each instant, the participants with their sources and tracks, the set’s digest, the order, the tiles and the voices. A receipt verifier (§8.3) reports its refusals at the same places, and at receipt.

10. Reasons

The codes this extension adds, each with where it is raised:

A verifier of this extension also raises truncated, malformed and unknown_version for a descriptor or a receipt that does not read; bad_key and bad_signature for their signatures; wrong_room for one of another room (§3.5, §8.3); and every code a room verifier raises (live-rooms §11 and the live-events codes it lists), at room.

11. Conformance

live-conformance/compose/ holds the corpus, in one file, vectors.json, written from fixed seeds by wai_live_compose_vectors gen. It has:

• layouts: the layout function alone (§5, §6’s voices), each case its parameters and its participants with their join orders, video sizes, audio and attributes, and the order, tiles and voices it gives, or its refusal. Grids of 0 to 12 tiles on a 16:9 canvas, and on portrait, square and auto ones; every fill mode over sources of four shapes and of extreme aspects; a featured participant present, absent, kept past the tile limit, and first in order of two; ties in order broken by join order and by key; stopped video shown and omitted; picture in picture in every corner under both behaviours, alone static and alone dynamic, with derived sizes and stopped video; and attributes that do not read.
• mixes: blocks of PCM and the mix of them (§6): passing through, both widths into both, saturation high and low and only at the end, pans and gains at their limits, rounding at ties, silence, and inputs refused.
• descriptors: records read (§3), with their digest and fields, or refused with each reason a descriptor alone can have.
• rooms: a studio room under the live-rooms descriptor, grants and chains that its vectors name, and four variants of it: a grant whose order does not read, a lifecycle record not delivered, a viewer’s chat not delivered, and a viewer’s record refused.
• compositions: a room, a descriptor and instants (§9), with the composition or the refusal. The grid over the whole session re-lays the room out at each lifecycle instant, among them a grant exchange that reorders it; the picture in picture composes a lower layer.
• receipts: a room, a descriptor and a receipt (§8.3), accepted with its digest and the set’s, or refused with each reason a receipt can have.

A conforming verifier gives every case the outcome, the reason, the place and the data the corpus states. wai_live_compose_vectors verify and the test tests/live_compose.rs check that the file regenerates byte for byte, that wai-rs gives every case its outcome, that every reason of §10 refuses a case, and that §10 lists exactly the reference’s codes. live-conformance/live_compose_verify.py is the second implementation, written from this text: it shares no code with wai-rs, verifies signatures with WAI’s strict rule written from RFC 8032 (claim-conformance/_ed25519.py), verifies each room with the live-rooms corpus’s own second implementation (live_rooms_verify.py) and replays its lifecycle to each instant itself, and gives every case the same outcome, rectangle for rectangle and sample for sample; it also mutates copies of the corpus and requires every copy to fail.

12. Security and privacy considerations

• The layout is computed from what the room authority signed: the descriptor, or a principal’s under the power the authority granted, and the attributes of grants the authority signed. A participant cannot place itself; a relay that drops or reorders lifecycle events is seen as a gap in a room key’s chain (§4.1), not as another layout.
• A server compositor’s receipt fixes which objects it composed and under which set. A compositor that left a participant out, read a layer other than the composed one, or composed a stale set is refused by §8.3. The receipt does not fix the output’s pixels or samples, which a lossy encode does not reproduce (§6); a party that needs them composes locally from the inputs.
• Composition on each sink needs no server to decode the room’s media, so a room whose tracks are end-to-end encrypted (confidentiality) can still be laid out: the layout reads the room’s events and the grants, never media.
• The participant set names every publisher, its grant and its join order. Composing a room reveals nothing a member of it does not already hold, but a set digest published outside the room links the participants it names; a room that needs participation unlinkable across rooms gives principals a key per room (live-events §11).
• Attributes are read as the descriptor’s keys name them and nothing else is read of a grant’s attributes, so an attribute a descriptor does not name changes nothing.