WAI Extension: Live Ingest and Egress
Mirrored from the canonical text at commit f871af55 ().
Status: Draft. Binds the contribution and playback protocols of standard encoders and players to the tracks of a room (live-rooms): WHIP (RFC 9725) into a room’s tracks, WHEP (
draft-ietf-wish-whep-05) out of them, and, as mappings only in this revision, Enhanced RTMP v2 multitrack and SRT. An ingest gateway admits a publisher by its room grant, takes its media frame by frame, packages each frame as one WAI envelope on a MoQ track (moq-streaming-format), maps its media time onto the room clock with a stated bound, and signs every group’s receipts as the first hop (jwp-receipts). An egress gateway verifies every object against its receipt before it hands the frame to a player. No new transport and no new token format: WAI defines none by charter. The reference implementation of WHIP and WHEP isjoule-moq-ingestinjoule-code/joule-code-rs/crates/, on thejoule-moqcodec and thejoule-moq-relayrelay. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.Texts read for this revision: RFC 9725 (WHIP);
draft-ietf-wish-whep-05(WHEP), an Internet-Draft not past Working Group Last Call; RFC 6750 §2.1 and §3; RFC 8840 (the trickle ICE fragment); RFC 9110 §8.8.3 and §13.1.1; RFC 6585 §3; RFC 9457; RFC 6184 (H.264 over RTP); RFC 7741 (VP8); RFC 9628 (VP9); the AV1 RTP payload format v1.0.0; RFC 7587 and RFC 6716 (Opus);draft-ietf-moq-transport-22(MOQT) anddraft-ietf-moq-msf-01(MSF); Enhanced RTMP v2, document versionv2-2026-01-31-r2(E-RTMP); and SRT 1.5.7, its stream ID guidelines and its rejection codes. A section number of one of those texts here is that revision’s.
1. Scope
In scope: how a WHIP request is admitted under a room grant (§3); the tracks an ingest
publishes and its catalog (§4); one frame as one object, the payload framing of each
codec, and how frames form groups (§5); the live member of an object’s manifest (§6);
the mapping from RTP time to the room clock and its bound (§7); the receipts the gateway
signs (§8); WHEP egress and what it verifies (§9); the bounds a gateway holds a session to
(§10); trickle ICE and ICE restarts (§11); the mappings of E-RTMP v2 multitrack (§12) and
SRT (§13); the refusals and their statuses (§14); and how an ingest relates to capture
binding and to metering (§15, §16).
Out of scope: a transport (WebRTC, MoQ, RTMP and SRT are the transports); how a principal obtains a grant (live-rooms §1); transcoding: a gateway carries the frames the publisher sent, as the publisher encoded them; slots and their takeover and failover (live-rooms §7), which an ingest into a slot follows and this revision does not map; and simulcast or SVC within one WHIP session (RFC 9725 §4.8).
Why an extension of its own. moq-streaming-format says
what a WAI track is on MoQ. This text binds four other protocols onto those tracks, under
the rooms and grants of the wai.live family; written into that extension it would make
a MoQ format describe RTMP and SRT. It sits beside live-rooms,
live-events and capture-binding, and changes none
of them.
Relationship to the core spec: no record and no capability is registered here. The
payload capabilities a live track carries are SPEC §5’s, among them wai.video.vp8 and
wai.audio.opus.packet, registered for this binding. The manifest member live (§6) is
defined here; a reader that does not know it ignores it (SPEC §3).
2. Terms
- Room, room clock, principal, grant, room authority: as in live-events §2 and live-rooms §2.
- Gateway: an endpoint that speaks one of the protocols of this text on one side and MOQT to a relay on the other. An ingest gateway takes media from a publisher; an egress gateway hands media to a player. One process may be both.
- First hop: the gateway an ingest’s media reaches first. It is the MoQ publisher of the tracks (MOQT §7.6) and the signer of their receipts (§8).
- Owner: the principal whose tracks they are: the grant’s
principal(live-rooms §6.1). - Frame: one access unit of video, or one packet of audio, as the RTP payload format of its codec delivers it whole (§5.1).
- µs: microseconds. Integers are little-endian where bytes are written.
3. Admission
3.1 The bearer token
A WHIP or WHEP request is authenticated by an HTTP Authorization header with a bearer
token (RFC 9725 §4.7.1; WHEP §4.9.1; RFC 6750 §2.1). The token is a room grant: the
WRG1 record of live-rooms §4.1, whole, in the base64url alphabet without padding (RFC
4648 §5). The room it names is the room the request is for. WAI defines no second token
format; a deployment that issues tokens of another format maps them onto a grant as
live-rooms §10 says, and admits by the grant.
A grant is a public record, so presenting it proves nothing about who presents it. Every
request that carries a grant also carries a proof of possession of the grant’s key
(live-rooms §10.5): a WRP1 record, in base64url, in the WAI-Room-Proof field, over a
challenge the gateway issued and the request’s context (§3.2 step 7). The gateway issues
single-use challenges in the WAI-Room-Challenge field: on a 401 that refuses for want of
a good proof, and on every response to a request that carried a grant, the one a 201
carries being scoped to the session it opened (live-rooms §10.5.3).
3.2 The checks
A gateway holds the signed descriptor of every room it serves. It admits a token by these checks, in order, and refuses at the first that fails with the reason it names:
- The token decodes, and the room its grant names (bytes 5 to 36) is a room the gateway
holds: otherwise
room_unknown, or the reading refusal of live-rooms §4.3 step 1 (truncated,malformed,unknown_version). - live-rooms §4.3 steps 1, 2 and 4 to 8, against that room’s descriptor: the record
reads; its signature verifies under the room’s authority; it names the room; its
fields keep §4.3’s rules; it is at most fourteen days long; it is within its role and
within the room’s ceiling. Each refusal is §4.3’s reason. Step 3, an exchanged grant’s
link to the grant it replaces, needs the earlier grant, which a token does not bring:
the gateway checks instead that the room clock has reached the exchanged grant’s
effectivetick. - The grant is good at the gateway’s clock (live-rooms §4.3): otherwise
outside_grant_window. - Neither its digest nor its viewer’s tag is among the revocations the gateway holds for
the room’s authority’s key log (live-rooms §5): otherwise
grant_revoked. - Where the grant lists
origins, the gateway’s own origin is one of them: otherwiseorigin_not_granted. - Where the grant’s
onceis not zero, no session of this gateway was opened under it: otherwisegrant_reused. The nonce is spent when a session opens. - The request’s proof passes live-rooms §10.5.4 for the grant, as this gateway reads it
(its digest and its principal), with the challenge it holds for the request’s scope (the
endpoint, or the session its URL names), the challenges it consumed before, and the
context
METHOD //authority/pathof the request it received: otherwise that section’s reason (pop_missing,pop_malformed,pop_wrong_grant,pop_wrong_key,pop_bad_signature,pop_wrong_challenge,pop_replayed,pop_wrong_context). A proof that passes consumes its challenge: it is good for one request.
A token missing or not a bearer token is answered 401 with WWW-Authenticate: Bearer realm=… and no error (RFC 6750 §3.1). A token refused by these checks is answered 401
with error="invalid_token" and the reason as error_description.
3.3 The powers
The grant must give what the request asks for (live-rooms §3.3). A grant that passes §3.2
and lacks it is answered 403 with WWW-Authenticate: Bearer error="insufficient_scope" (RFC 6750 §3.1):
- WHIP: the grant’s
publishbit for each media kind the offer carries, audio (1) and video (2): otherwisepublish_not_permitted. A screen is published as media kindscreen(3) only through a mapping that names it, which this revision does not. - WHEP:
subscribe, and amax_layerat or above the layer asked for: otherwisesubscribe_not_permittedorlayer_not_permitted. The room in the resource’s path is the grant’s: otherwisewrong_room.
Where the grant names a publish profile for a media kind (live-rooms §4.1 profiles),
the session accepts only the codecs of that profile’s codecs that the gateway carries,
by RFC 6381 family (avc1, vp8, vp09, av01, opus), and states the profile’s
layer-0 max_kbps as the media section’s bandwidth (§10).
3.4 The session’s resource
Every request to a session’s resource (PATCH, DELETE, GET, HEAD) carries the token
the session was opened with, and a proof of possession over a challenge scoped to that
session (§3.2 step 7); another token, however good, is answered 403
(not_the_session_token). A session’s URL carries at least 128 random bits (RFC 9725
§5).
4. Tracks and the catalog
4.1 Names
A WHIP session publishes, for each media kind its offer carries, one media track and its receipts track, and, for the owner, one catalog track and its receipts track:
| track | Track Namespace (live-rooms §6.1) | Track Name |
|---|---|---|
| media | (room id, owner, audio or video) | 0 |
| its receipts | the media track’s | 0_receipts |
| catalog | (room id, owner) | catalog |
| its receipts | the catalog’s | catalog_receipts |
A WHIP session sends one encoding of each kind, so its media tracks are layer 0
(live-rooms §6.3, single). A receipts track is named as live-rooms §6.1 names
one, the media track’s name and _receipts, and is of packaging wai-receipts. The
catalog is the owner’s MSF catalog (MSF §5): the namespace prefix (room id, owner) reaches
it and every track it declares.
4.2 Publication
The gateway publishes every track at the relay with PUBLISH (MOQT §9.8) before it answers the WHIP request, so a subscriber can subscribe from the first group. The tracks’ default publisher priority is 0 for audio and the catalog and 1 for video (MOQT §5.1.1): a subscriber that falls behind keeps the sound. A receipts track’s priority is its media track’s (moq-streaming-format §4.1). No track carries a delivery timeout.
When the session ends the gateway ends each open group, publishes its receipts, waits for
every stream to be acknowledged, and sends PUBLISH_DONE with status TRACK_ENDED on each
track (MOQT §9.9). The relay then ends its subscriptions: a subscriber sees the track end.
4.3 The catalog
The catalog is RFC 8785 canonical JSON, {"tracks": [...], "version": 1}, one object a
group. Each media track’s entry carries MSF’s namespace (the MOQT §8.8 rendering of its
namespace, live-rooms §6.2), name, packaging "wai", isLive true, role,
renderGroup 1, and, where the bitstream states them, codec (RFC 6381), width and
height, or samplerate and channelConfig; and moq-streaming-format §4’s
wai_capabilities (the one capability its objects dispatch to), wai_receipts true,
wai_receipt_alias (§8.2) and wai_receipts_track. Each is followed by its receipts
track’s entry.
The gateway publishes the catalog once every media track has carried its first frame, or a
second after the first did, whichever comes first, and publishes it again, whole, in a new
group, when a track’s codec or picture size changes. Its group IDs follow §5.3’s rule. The
catalog is signed as the media is (§8): each catalog group is one object, receipted on
catalog_receipts. A relay that checks every track of a room at its door forwards it, and
a player verifies what the catalog declares before it relies on it (§9.1).
5. Objects and groups
5.1 One frame, one object
Each frame is one MOQT object, and each object is exactly one WAI1 envelope
(moq-streaming-format §3). The frame is what the codec’s RTP payload format reassembles,
whole:
| codec | RTP payload format | capability (SPEC §5) | the object’s payload |
|---|---|---|---|
| H.264 | RFC 6184 | wai.video.avc | one access unit, Annex B byte stream: each NAL unit after a four-byte start code |
| VP8 | RFC 7741 | wai.video.vp8 | one frame in the IVF frame container, FourCC VP80: the 32-byte file header (the frame’s size, a time base of 1/90 000, one frame), the 12-byte frame header (its length, its time in 90 kHz ticks), the frame |
| VP9 | RFC 9628 | wai.video.vp9 | as VP8, FourCC VP90 |
| AV1 | AV1 RTP payload format v1.0.0 | wai.video.av1 | one temporal unit in SPEC §5’s AV1 framing: n = 1, the largest frame’s height and width from the sequence header, a frame rate of 90 000 over the ticks since the track’s previous frame, then the temporal unit’s length and the unit: a temporal delimiter OBU (which RTP does not carry, and which the gateway restores), then every OBU with its size field |
| Opus | RFC 7587 | wai.audio.opus.packet | one Opus packet (RFC 6716 §3) |
A frame of a codec this table does not list is not carried. A frame larger than the gateway’s bound is dropped and counted (§10).
A live track’s objects are one stream. A frame is a unit of its codec’s stream, not a picture or a sound decodable alone. The objects of a media track, in object order from a group’s first object and in group order, are consecutive units of one stream of the codec: a sink decodes them in that order with one decoder state, from a group’s first object, and resets that state where a group follows one it did not receive. The capability of every object says how its payload is read; this rule says how consecutive payloads relate. A sink that joins a track joins at a group’s first object.
5.2 Groups
- Video: a group begins at each keyframe (an IDR access unit for H.264; a key frame
for VP8 and VP9; a temporal unit with a sequence header for AV1) and holds it and every
frame up to the next. A frame that arrives while no group is open (before the first
keyframe) is dropped, counted, and answered with a keyframe request (a PLI, RFC 4585
§6.3.1). A group that runs longer than
max_video_group_ms(2 000 by default) on the room clock makes the gateway ask for a keyframe, at most once akeyframe_request_interval_ms; the group ends when the keyframe comes. - Audio: a group holds the frames whose room-clock time (§7) falls in one window of
audio_group_ms(200 by default). The windows are laid end to end from the track’s first frame; after a pause longer than a window the next window starts at the next frame.
A group’s objects travel on one subgroup stream (MOQT §10.3, Subgroup ID 0), written as the frames arrive, and the stream ends with the group: its END_OF_GROUP bit is set and its FIN follows the group’s last object.
5.3 Group IDs
A track’s group IDs ascend by one. The first group’s ID is the room-clock time, in µs, of its first frame (§7). A publisher that reconnects, or a new session of the same owner, therefore continues above every ID the track used before, so a relay never takes a new group for a copy of an old one (MOQT §7.3). A catalog group’s ID follows the same rule, from the room-clock time it is published.
6. The live manifest member
An object’s manifest carries SPEC §3’s members and the member live, an object:
| member | type | meaning |
|---|---|---|
t | integer | the frame’s room-clock time, µs since the room’s t0 (§7) |
rtp | integer | the frame’s RTP timestamp, extended past its 32 bits by the WebRTC stack |
rate | integer | its RTP clock rate: 90 000 for video, 48 000 for Opus |
key | boolean | video only: whether the frame is a keyframe, the first of its group |
grant | object | on the first object of each group only: the grant reference of live-rooms §4.5, grant_hash, capability (wai.live.room), joule_ceiling_micro and funds_ceiling (null for none) |
The receipt’s content hash covers the envelope’s bytes (jwp-receipts §2.1), so the group receipt that covers a group’s first object signs the grant reference for the whole group: this is how every object of an ingest names the grant it was carried under (live-rooms §4.5), at the cost of one reference a group. The manifest’s members are written in a fixed order with no insignificant whitespace (SPEC §2).
A sink that reads t reads the gateway’s statement of the frame’s time, bounded as §7
says; it is not the publisher’s.
7. Time
7.1 The mapping
The gateway’s clock gives Unix time; the room clock is that time less the room’s t0
(live-rooms §3.1). For each media track, the first frame anchors the mapping: its
room-clock time is its arrival at the gateway, A, and its extended RTP timestamp is
R0. A later frame whose timestamp is R maps to
t = A + ⌊(R − R0) · 10^6 / rate⌋ µs.
A frame whose mapped time lies more than max_skew_ms (250 by default) from its own
arrival, or whose timestamp is below R0, re-anchors the mapping at itself: its time
is its arrival, and the anchor is (its arrival, its timestamp). Mapped times never
decrease on a track: a frame that would map before the last one takes the last one’s time.
7.2 The bound
So every frame’s t lies within max_skew_ms of its arrival at the gateway. A frame
arrives after it was captured, by the path’s one-way delay, so t is no earlier than its
capture less max_skew_ms, and no later than its capture plus its one-way delay plus
max_skew_ms, both on the gateway’s clock. Two frames of one session, of either track, are
placed relative to each other within their arrivals’ difference plus twice max_skew_ms.
Between re-anchors, the publisher’s media clock is followed exactly: t advances as the
RTP timestamps do.
The bound is to the gateway’s clock. Its error against the room clock, as the room’s clock key keeps it, is the gateway host’s clock error, which this revision neither measures nor states. A gateway counts its re-anchors in its session record (§16).
7.3 Across the room
A marker or a command set on the room clock lands on an ingest’s media within §7.2’s bound.
An event anchored to an object of an ingest track (live-events §3.1) names the object, and
the object’s live.t gives that object’s time; a sink that maps an anchored event takes
t from the object, not from the event.
8. Receipts
8.1 What the gateway signs
When a group ends, the gateway signs its group receipt (jwp-receipts §2.4.1) over every
object of the group and publishes it with the object receipts as group g of the track’s
receipts track (moq-streaming-format §4.1). Every object receipt is labelled
(jwp-receipts §2.4.2): its figure is the unmetered marker, 0 with class tag 0
(energy-measurement §2), since no meter backs the gateway; it is
never a figure of zero joules. Its origin label is the gateway’s. The group receipt is
signed in the labelled form, its class composed from its objects’, and chains to the
track’s previous group by parent_receipt_hash. Its signer_id is the gateway’s
(jwp-receipts §7.1): origin-bound where the gateway’s origin publishes its key log, its
key’s did:key otherwise.
What the receipts state is the first hop’s: that the gateway received these bytes, under this grant, at these room-clock times. A WebRTC publisher signs nothing, so they are not the publisher’s statement. A publisher that speaks WAI publishes its tracks over MoQ itself and signs its own receipts.
8.2 The chain
A track’s wai_receipt_alias names its receipt chain (moq-streaming-format §4): the first
eight bytes, little-endian, of BLAKE3 over wai:ingest-receipt-alias (ASCII), 0x01, the
session’s 16-byte id, the rendered namespace, one zero byte, and the Track Name, reduced
below 2^53. Each session’s tracks so sign chains of their own: a publisher that reconnects
starts a new chain, and never forks the old one. The catalog that follows the reconnection
names the new alias.
8.3 When the receipts are published
A group’s receipts go out after its last object, since the group receipt’s root covers
every object of the group. This differs from moq-streaming-format §4.1’s rule for a stage
set, where a live group holds one object and its receipt precedes it: an ingest’s groups
hold many frames, and a frame is published as it arrives. A subscriber that does not verify
receives each frame at once; one that verifies holds a group until its receipts arrive, at
the group’s end. A relay with a door on the room’s namespace (jwp-receipts §4.1) holds each
object until its group’s receipts arrive, so its hold must exceed the longest group,
max_video_group_ms plus the time a keyframe takes to come.
9. WHEP egress
9.1 A session
A player POSTs its offer to the resource of one owner’s tracks,
…/whep/<room id>/<owner>[?layer=<n>], the room id and the owner as 64 lowercase
hexadecimal digits each, and n the layer, 0 by default (WHEP §4.3). Admission is §3’s.
The gateway:
- subscribes at the relay to the owner’s catalog and its receipts track, and reads the
latest catalog whose receipt verifies (§9.2) under a key it trusts as a first hop.
Where none arrives within its wait, the request is answered
409withRetry-After(WHEP §4.3.9): the owner is not live. Where the catalog declares no track of the layer,404(layer_unknown); - answers the offer with the codecs of the declared tracks only, so a section of a kind
the owner does not publish, or of no common codec, is rejected (WHEP §4.3.4). An offer
none of whose sections is accepted is
422(no_common_codec); - subscribes to each accepted kind’s media track and its receipts track from the start of the latest group: the live part from the Next Object, and a fill of the latest group from its first object (MOQT §3.5, §9.20.9). A player so starts at a group’s first object, a keyframe for video.
A track’s end at the relay (PUBLISH_DONE) ends the session, after every frame that verified is handed on.
9.2 Verification
For each group of each track, before any of its frames reaches the player:
- its receipts object reads as a bundle: otherwise
refused_bundle; - its receipts name this MOQT group, each object receipt its Object ID, and the track’s
wai_receipt_alias(moq-streaming-format §4.1,moqt_mismatch): otherwiserefused_moqt. The catalog’s own chain is not declared anywhere, so a catalog receipt is held to its group and Object ID alone; - its group receipt verifies under a trusted first hop’s key (jwp-receipts §3):
otherwise
refused_signature; - where the session holds the receipt of the group before it, its
parent_receipt_hashis that receipt’s link hash: otherwiserefused_chain; - each object matches its object receipt: its content hash, and its Merkle proof to the
signed root (jwp-receipts §3): otherwise
refused_inclusion; - each object opens as a
WAI1envelope of the track’s capability with alivemember, and its payload as one frame of its framing (§5.1): otherwiseunopenable.
What fails is dropped and counted, and never reaches the player. A group whose receipts do
not arrive within hold_ms (5 000 by default) of its last object is dropped and counted
(receipt_missing); an object a verified group’s receipts name that did not arrive is
counted as receipted and not delivered (object_missing). Groups are handed on in group
order, their frames in object order. A frame’s RTP time is its live.t at the codec’s
clock rate, and its wallclock is placed from the session’s first frame by t, so the
tracks of a session keep the timing §7 gave them.
9.3 What a WHEP player cannot see
WebRTC carries media, not receipts: a WHEP player receives frames and nothing that proves where they came from or that they are whole. The egress gateway is the verifier on its behalf, and its session record (§16) is its account of what it verified, dropped and why. A WAI-aware player gets the receipts itself: it subscribes to the same media tracks and their receipts tracks over MoQ, through the relay’s WebTransport or WebSocket paths, and verifies each object as §9.2 does, with no gateway between. Both read the same tracks; only the second holds the proof.
9.4 Latency
A frame reaches a WHEP player when its group’s receipts have verified, at the group’s end: up to a video group’s length after it reached the relay, plus the relay’s and the network’s delay. A deployment that wants a shorter path shortens its groups (§5.2), at the cost of keyframes, or plays over MoQ unverified and verifies afterwards.
10. Bounds
A gateway states each bound in its configuration, and refuses what passes it:
| bound | refused with |
|---|---|
| WHIP sessions at once; WHEP sessions at once | 503 with Retry-After (RFC 9725 §4.5), session_limit |
media sections in one offer (max_tracks_per_session, 2 by default); more than one of a kind | 422 (RFC 9725 §4.4.2), too_many_tracks, one_track_per_kind |
| the media a WHIP session sends, in kilobits per second over a window | stated per media section in the answer as b=AS (RFC 8866 §5.8); a session that passes it over bitrate_window_ms is ended, bitrate_exceeded |
| the largest frame | dropped and counted |
| the bytes waiting for the relay | the session is ended, relay_backpressure |
proof-of-possession challenges held at once, issued and consumed each (max_challenges), and how long one stays good (challenge_ttl_s) | the oldest is dropped; a proof over a challenge past its time is refused pop_wrong_challenge |
11. ICE
A session supports trickle ICE and ICE restarts (RFC 9725 §4.3; WHEP §4.5), with
entity-tags: the 201 carries the ICE session’s ETag, and a PATCH names the ICE
session it is for in If-Match. A PATCH without If-Match is 428; one whose tag is not
the current one’s, or *, is 412 (RFC 9110 §13.1.1; RFC 6585 §3).
- Trickle: a fragment whose credentials are the current ones, or absent, adds its
candidates; one the gateway cannot use is discarded.
204, noETag. - Restart: a fragment with new
ice-ufragandice-pwdrestarts ICE. The gateway applies to its WebRTC stack the client’s offer with those credentials and the fragment’s candidates in place of the old ones, which is the offer an ICE restart would have sent (RFC 8839 §4.4.1.1.1), and answers200with anapplication/trickle-ice-sdpfragbody: itsice-liteandice-optionsattributes, the BUNDLE group, the first media section’sm=line andmid, its new credentials, its candidates,a=end-of-candidates; and a newETag. A restart it cannot apply keeps the ICE session it has and is422(ice_restart_refused, RFC 9725 §4.3.3).
The gateway is an ICE lite agent where every client reaches its address (RFC 9725 §4.4.5),
and a full agent otherwise. A session whose ICE stays disconnected past a grace period is
ended (ice_disconnected).
12. E-RTMP v2 multitrack (mapping)
This section maps Enhanced RTMP v2 (v2-2026-01-31-r2) onto room tracks. This revision
implements none of it.
-
Connection. The gateway is the RTMP server. In its
connect_resultit declarescapsExwithReconnect(0x01) andMultitrack(0x02), andModEx(0x04) andTimestampNanoOffset(0x08) where it reads them, and the codecs of the table below invideoFourCcInfoMapandaudioFourCcInfoMap(E-RTMP Enhancing NetConnection connect Command). -
Admission. The
publishcommand’s stream name is the bearer token of §3.1, and is checked as §3.2 and §3.3 check it. A refusal is anonStatuswithlevelerror, codeNetStream.Publish.BadName, and the reason of §14 asdescription; then the stream is closed. -
Tracks. Each (media type,
trackId) of a multitrack stream is a candidate room track.trackIds are identifiers that imply no order (E-RTMP Multitrack Media Message Guidelines), so a video track’s layer is the layer of the grant’s publish profile whose width, height and frame rate the track’sonMetaDataentry states, matched exactly, and layer 0 where the profile has one layer or the grant names none. A track that matches no layer, a second track of one layer (another codec of the same rendition, asManyTracksManyCodecssends), and an audio track other thantrackId0, are not carried in this revision, and are counted. A layer is a room track named as §4.1 names layer 0, with the layer as its Track Name (live-rooms §6.1, §6.3). -
Codec FourCCs to the catalog.
FourCC capability a frame the object’s payload avc1wai.video.avca CodedFramesbody: length-prefixed NAL unitsAnnex B: each NAL unit after a start code, the SPS and PPS of the AVCDecoderConfigurationRecordbefore each IDR access unit, so a group decodes from its first objecthvc1wai.video.hevcas avc1Annex B, the HEVCDecoderConfigurationRecord’s parameter sets before each IRAP access unitvvc1wai.video.vvcas avc1Annex B, the VVCDecoderConfigurationRecord’s parameter sets before each IRAP access unitav01wai.video.av1one temporal unit of OBUs §5.1’s AV1 framing, the AV1CodecConfigurationRecord’s sequence header before each key temporal unit that lacks onevp09wai.video.vp9one frame §5.1’s IVF framing vp08wai.video.vp8one frame §5.1’s IVF framing Opuswai.audio.opus.packetone Opus packet, for one stream ( N= 1 in the ID header)the packet; a multistream body ( N> 1) is not carriedfLaCwai.audio.flacone or more FLAC frames a FLAC stream: fLaC, the sequence start’sSTREAMINFO, the framesmp4awai.audio.aacone raw AAC frame one ADTS frame, its header built from the AudioSpecificConfigA FourCC this table does not list (
ac-3,ec-3,.mp3) is not carried. The catalog’scodecis written from the configuration record (avc1.PPCCLLfrom theAVCDecoderConfigurationRecord’s profile, compatibility and level;av01from theAV1CodecConfigurationRecord;vp09from theVPCodecConfigurationRecord), and itswidth,height,samplerateandchannelConfigfrom the configuration record or the track’sonMetaData. -
Time. A message’s timestamp is milliseconds on the publisher’s clock; a video frame’s presentation time adds its
compositionTimeOffset, and aTimestampOffsetNanoModEx adds its nanoseconds. §7’s mapping applies with these as the media time and a rate of 10^6 per µs, anchored at the first message’s arrival; a timestamp that wraps its 32 bits is extended past them. -
Groups. A video group begins at a
KeyFrameof its track (§5.2); audio windows as §5.2. -
The reconnect request. Before a gateway shuts down, or to move a publisher to another gateway, it sends
onStatuswith codeNetConnection.Connect.ReconnectRequest,levelstatus, and the new gateway’stcUrl, to a client that declaredReconnectincapsEx(E-RTMP Reconnect Request). The client reconnects at its next keyframe. The new session’s first group ID is its first frame’s room-clock time (§5.3), above every ID the old session used; its receipt chain is new (§8.2), and its catalog names it. A gateway that receives a MOQT GOAWAY from its relay (MOQT §9.2) sends the reconnect request to its publishers. -
Latency. RTMP runs over TCP, so no frame is lost and none reordered; a stall shows as frames that arrive late together, which §7 bounds by re-anchoring.
13. SRT (mapping)
This section maps SRT 1.5.7 caller-listener connections onto room tracks. This revision implements none of it.
- The stream ID and the grant. A stream ID holds at most 512 characters (SRT stream ID
guidelines). A grant’s fixed fields alone take 315 bytes, 420 characters in base64url,
and with the room, the owner and the mode beside them a stream ID would pass 512 before
the grant names a profile, an origin or an attribute. A caller therefore first obtains a
session ID: it POSTs to the gateway’s
…/srtresource with the bearer token of §3.1, admitted as §3 admits WHIP and WHEP, and receives a one-time session ID, valid for 60 seconds and invalidated by its first use, and the passphrase for the connection. Its stream ID is then#!::r=<room id>/<owner>,m=publish,s=<session ID>to publish andm=requestto play, in the guidelines’ key-value syntax with their standard keys (rthe resource,mthe mode,sthe session ID, which the guidelines define for exactly this case: details negotiated over a separate connection first). The gateway’s listener callback finds the session, checks that its grant gives the mode (m=publishthe publish power,m=requestthe subscribe power) for that room and owner, and sets the connection’s passphrase, so the SRT handshake fails for a caller without it. - Refusals. The listener callback rejects with the SRT rejection codes of the HTTP
statuses §14 gives:
1401(SRT_REJX_UNAUTHORIZED) for a session or grant that is not good,1403(SRT_REJX_FORBIDDEN) for a power the grant does not give,1404(SRT_REJX_NOTFOUND) for a room the gateway does not hold,1405(SRT_REJX_BAD_MODE) for a mode other thanpublishandrequest,1406(SRT_REJX_UNACCEPTABLE) for a payload it cannot read,1409(SRT_REJX_CONFLICT) for an owner already publishing that kind through another session, and1402(SRT_REJX_OVERLOAD) past a bound of §10. - Payload. SRT carries any payload; in live mode a contribution encoder sends an MPEG
transport stream (ISO/IEC 13818-1), seven 188-byte packets a datagram. The gateway
demultiplexes it by its program map table: stream type
0x1Biswai.video.avc(each PES packet’s access unit, already Annex B),0x24wai.video.hevc,0x0Fwai.audio.aac(one ADTS frame an object). Other stream types are not carried in this revision. Each carried elementary stream of a kind is a room track as §4.1 names it; a second of one kind is not carried. - Time. A PES packet’s presentation time stamp, 33 bits at 90 kHz, extended past its 33 bits, is the media time of §7’s mapping, with a rate of 90 000. SRT delivers each packet at its sender’s time plus the connection’s latency (its timestamp-based packet delivery), so frames arrive with that latency added and evenly; §7’s bound applies to the variation.
- Latency. The connection’s latency is the larger of the caller’s and the listener’s
(
SRTO_LATENCY; 120 ms unless set). It bounds how long SRT retransmits a lost packet, and a retransmission takes at least one round trip, so a gateway sets its listener’s latency to several round-trip times of the paths it serves. The gateway states its latency in its configuration; a grant’s profile states none. - Groups and receipts are §5 and §8’s.
14. Reasons
The refusals of this extension, each with its HTTP status and where it is raised. A reader
of the WHIP and WHEP statuses keeps RFC 9725 §4.1’s rule: a client falls back to a
status’s generic class for a reason it does not know. Each refusal’s body is a problem
details object (RFC 9457) whose reason member is the code.
| reason | status | raised at |
|---|---|---|
token_missing | 401 | §3.2 |
token_malformed | 401 | §3.2 step 1 |
room_unknown | 401 | §3.2 step 1 |
truncated, malformed, unknown_version, bad_key, bad_signature, wrong_room, grant_invalid, grant_too_long, caps_exceed_role, grant_exceeds_room | 401 | §3.2 step 2 (live-rooms §4.3, §11) |
outside_grant_window | 401 | §3.2 steps 2 and 3 |
grant_revoked | 401 | §3.2 step 4 |
origin_not_granted | 401 | §3.2 step 5 |
grant_reused | 401 | §3.2 step 6 |
pop_missing, pop_malformed, pop_wrong_grant, pop_wrong_key, pop_bad_signature, pop_wrong_challenge, pop_replayed, pop_wrong_context | 401 | §3.2 step 7 (live-rooms §10.5.4, §11) |
publish_not_permitted | 403 | §3.3 |
subscribe_not_permitted, layer_not_permitted | 403 | §3.3 |
wrong_room | 403 | §3.3, a WHEP resource of another room |
not_the_session_token | 403 | §3.4 |
not_sdp, not_sdpfrag | 415 | an offer that is not application/sdp; a PATCH that is not a fragment |
offer_malformed, sdpfrag_malformed | 400 | the body does not read |
section_unsupported, direction, no_media, too_many_tracks, one_track_per_kind, one_media_stream | 422 | §10; RFC 9725 §4.2, §4.4.2; WHEP §4.3.6 |
no_common_codec, offer_refused | 422 | §3.3, §9.1; RFC 9725 §4.4.3 |
if_match_missing | 428 | §11 |
if_match_stale | 412 | §11 |
ice_restart_refused | 422 | §11 |
not_live | 409, with Retry-After | §9.1 |
layer_unknown, owner_unknown, session_unknown | 404 | §9.1; a session that has ended |
body_too_large | 413 | a request body past the gateway’s bound |
body_timeout | 408 | a request body that does not arrive in time |
not_found | 404 | a path that is no resource of the gateway |
session_limit | 503, with Retry-After | §10 |
relay_unreachable, relay_refused, media_socket | 503 | the gateway cannot reach its relay, or bind a socket |
A session that ends states why in its record: deleted, connect_timeout, idle,
ice_disconnected, bitrate_exceeded, relay_backpressure, relay_lost, or, for WHEP,
track_ended.
15. Ingest and capture binding
An ingest is not a capture binding. A WHIP session’s tracks are the live tier: what the publisher’s encoder sent, receipted by the first hop on arrival. A take (capture-binding) is the master tier: the device’s own recording, committed by the device’s key into the room’s events while it is captured. The two relate through the room clock (capture-binding §10): the ingest’s objects carry their room-clock times (§6, §7), and the take’s commitments carry theirs.
A WHIP publisher’s take can still be committed by the device, if the device speaks
capture binding: it joins the room under a grant that binds it to the principal it captures
for (capture-binding §3.1), records its master, and sends its take_commit events on its
own events track over MoQ, beside its WHIP session. Its WHIP media and its master then
share the room clock: a marker set on the live call at room time t lands at t in the
master (capture-binding §10), within the master’s bound there and this extension’s §7.2
here. A device that speaks WHIP alone has a live tier and no take; nothing in this
extension makes its live media a take, and a single-take claim is never made of an
ingest’s tracks.
16. Metering
A gateway is a hop of every room session it carries, in the role of an ingest gateway.
When each of its sessions ends it signs a record of what the session carried: per track
and direction the objects and bytes, and the drops by reason; the subscriber sessions
served and their time by layer class; the grant the session ran under, with its ceilings
and the bytes in and out under it; and its own energy, the unmetered marker. These are the
counts of the per-hop session record that WAI’s live metering defines, kept in that
record’s shape. Until that record is defined, the reference gateway signs them as JSON
(joule-moq-ingest/session/1); a later revision has the gateway sign the per-hop record
itself, under the key-log authority live metering gives an ingest gateway’s key.
17. Status and conformance
| mapping | status | where |
|---|---|---|
| WHIP (§3 to §11) | implemented | joule-moq-ingest: admission, the tracks and catalog, framing for H.264, VP8, VP9, AV1 and Opus, groups, time, receipts, ICE, bounds |
| WHEP (§9) | implemented | joule-moq-ingest: the catalog read and verified, joining at a group, per-object verification, egress |
| E-RTMP v2 multitrack (§12) | specified | — |
| SRT (§13) | specified | — |
The reference implementation’s tests run on loopback: a WHIP publisher of H.264 and Opus
whose frames a relay subscriber receives whole, every receipt verified (by the subscriber
and by the relay’s door); a WHEP player that gets the same frames from the group it
joined; VP8, VP9 and AV1 each through WHIP; every refusal of §3 with its status; trickle
ICE and an ICE restart with media across it; and DELETE ending the tracks at the relay. Its
reading of room descriptors and grants is held to the corpus of live-rooms
(live-conformance/rooms/): every room id and grant digest there, the registered
profiles’ records, and every refusal of a descriptor or a grant but those of an exchange’s
link (§3.2 step 2) and of the key log (§3.2 step 4), which admission does not make.
Not implemented in this revision, beside §12 and §13: a WHEP counter-offer (WHEP §4.3.2); a
WHEP player’s keyframe request carried upstream (MOQT’s NEW_GROUP_REQUEST); publishing
into a slot (live-rooms §7); simulcast within one WHIP session; STUN and TURN servers in
Link headers (RFC 9725 §4.6); and reading the key log itself (§3.2 step 4 holds the
revocations a deployment gives it).
18. Security and privacy considerations
- A token is a bearer capability (live-rooms §13): whoever holds it may publish as its principal through a gateway. Its window is fourteen days at most, a single-use grant opens one session, and a gateway’s TLS front keeps it off the wire in clear (RFC 9725 §5).
- The first hop’s receipts attest what the gateway received, not what the publisher’s camera saw: a gateway that lies signs its lie with its own key, and a player that trusts its receipts trusts that key. A deployment lists the gateways it trusts as first hops, and a player that needs more verifies a take (§15).
- The receipts are signed after a group ends: a relay that withholds a group’s receipts holds back its frames from every verifying sink, which notices the gap by the chain (§9.2 step 4), not by a missing frame alone.
- An egress gateway that verifies hands a player only frames that verified, so a tampered relay cannot reach a WHEP player with altered media; it can still withhold media, which the session record counts.
- A session’s URL is unguessable (§3.4), and its resource answers only its own token, so a third party cannot end or restart another’s session.
- Every bound of §10 limits what one session, and one grant, can make a gateway hold.