Skip to main content

WAI Extension: Ensemble timing

Mirrored from the canonical text at commit 498e6b88 ().

Status: Draft. Registers the record wai.live.ensemble: performers in different places playing together in a room, with the delay between each two of them measured, signed and judged, and a whole-bar delay for performers too far apart to play in time. The room authority signs an ensemble policy, WJP1: the slots and windows of measurement, the ensemble threshold, the percentile a verdict is taken at, and the bound a whole-bar delay holds to. Each performer’s page signs, periodically, a presentation statement, WJS1: at an instant of the room clock, the capture instant, on the room clock, of the sound of each other performer it was presenting. From those statements the room clock signs a latency statement, WJL1, per window: for each ordered pair of performers, the one-way delay from the source’s capture to the listener’s presentation, as percentiles, with a verdict against the threshold, which anyone holding the room recomputes, so that anyone can say who could play with whom at each instant. A pair one of whose performers’ grants carries the delay attribute plays in whole-bar delay: the listener presents the source’s sound exactly k bars after its capture on the room’s beat clock (co-performance §6.8), and the latency statement judges the pair by how close each sample came to that instant. A pair whose samples are missing is unknown, never in time. Statements ride in the room’s events, as meta payloads their authors sign. No new transport. The reference implementation is the ensemble module of wai-rs (feature ensemble); the corpus is live-conformance/ensemble/, and its second implementation is live-conformance/ensemble_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: the live family this extension builds on, as merged (live-events, live-rooms with its publish profiles of version 2, co-watch §7, §8 and §15 and co-performance with its beat clock); RFC 8032 and FIPS 180-4. No outside text is pinned. The numbers this extension is built for are the measured ones of networked ensemble playing: performers keep together best with 8 to 25 ms of one-way delay and lose accuracy past 55 to 66 ms, slow tempi tolerate about 110 ms, music built of repeated patterns accepts a delay of whole patterns, and light in fibre takes about 5 ms per 1,000 km (decision D124).

1. Scope

In scope: the ensemble policy, its signature and the policy in effect at each instant (§3); how this extension’s statements ride in a room’s events, who may sign each, and the grant attribute it reads (§4); presentation statements (§5); whole-bar delay: a pair’s delay, the instant a sample is due at and what a page does (§6); windows, due slots, samples, percentiles, verdicts, the latency statement, its completeness, and who plays with whom (§7); where each rule is enforced (§8); and verification (§9).

Out of scope: a transport; how a page captures, encodes, sends and plays sound, which live-rooms §8 profiles bound (the registered jam profile states a short frame and a jitter target, live-rooms §8.2); how a page estimates the room clock, which co-watch §7.3 states for a player and §12 here for a wrong estimate; the tempo, which the room’s beat clock states (co-performance §6.8); and how a sink renders a delay, a verdict or a bar.

Relationship to the core spec: wai.live.ensemble is a record (SPEC §5): signed records a WAI tool emits and verifies. It names no payload format and is not dispatchable.

Relationship to live-events: this extension changes nothing in it. Its statements are payloads of the meta kind (live-events §4.2), so every statement is signed by its author, chained, stamped with the room clock and rate-bounded by its author’s policy (live-events §5, §6).

Relationship to live-rooms: the descriptor’s and the grant’s layouts are unchanged. A room states its ensemble policy in a record of its own, WJP1 (§3), chained as co-watch’s policy is. The delay attribute is a grant attribute (§4.4), which live-rooms §4.4 signs with the grant and leaves to the extensions that read it.

Relationship to co-watch: the presentation statement and the latency statement are made as co-watch’s presentation and skew statements are (co-watch §7, §8): a page’s signed claim of what it presented at an instant, and the room clock’s percentiles over a window’s slots, recomputed by anyone holding the room. Co-watch measures how far each viewer is from a source’s controlled position; this extension measures how long each performer’s sound takes to reach each other performer, and has no controlled position.

Relationship to co-performance: a jam room is a co-performance room. Its performers state their media’s timestamps on the room clock (co-performance §9.1), which is how a page knows each sample’s capture instant; its beat clock (co-performance §6.8) places the bars a whole-bar delay counts; and its turns and intents are co-performance’s. This extension adds no statement to it and reads its beat clocks as it defines them.

2. Terms

Integers are little-endian. A digest is SHA-256 (FIPS 180-4), 32 bytes. Ed25519 is RFC 8032’s, verified strictly (SPEC §7 Signatures): a key that fails step 1 or 2 is bad_key, and a signature that fails a later step is bad_signature.

3. The ensemble policy

3.1 Layout

fieldsizemeaning
magic4WJP1 (ASCII)
version11 in this revision
room32the room id (live-rooms §3.2)
prev32the digest of the policy this one replaces (§3.4); 32 zero bytes for the first
effective8u64: the room-clock instant from which this policy applies; 0 for the first
period_ms4u32: the slot length: a page states its presentation at least once in each (§5.3, §7.1), above 0
window_ms4u32: the length of a window (§7.1), a multiple of period_ms of 1 to 1000 slots
settle_ms4u32: how long after a listener joins, and after a source publishes its sound, a pair’s slots are not yet due (§7.2)
threshold_us4u32: the ensemble threshold: the one-way delay at or below which a pair plays together (§7.5), above 0
quantile1the percentile a verdict is taken at (§7.5), 1 to 100
align_us4u32: the bound within which a pair in whole-bar delay presents each sample at its due instant (§6.2, §7.5), above 0
sig64§3.2

3.2 Digest and signature

The digest of a policy is SHA-256 over its record from magic through align_us. The signature is Ed25519 by the room descriptor’s authority over the 20 bytes wai:ensemble-policy (ASCII) and 0x01, followed by the bytes the digest covers.

3.3 Reading and rules

A reader reads exactly the bytes it is given as one policy, in the order of §3.1, and refuses at the first of these that holds:

  1. Fewer than 4 bytes: truncated. A magic other than WJP1: malformed. No byte after the magic: truncated. A version other than 1: unknown_version.
  2. The bytes end before a field the layout requires: truncated.
  3. Bytes after sig: malformed.

It then holds the policy to the rules of §3.1’s table, and refuses one that breaks any of them as ens_policy_malformed: period_ms, threshold_us and align_us are above 0; window_ms is period_ms times a whole number from 1 to 1000; and quantile is 1 to 100. Then its room is the room id (wrong_room), its signature verifies (bad_key, bad_signature), and its link holds (§3.4, ens_policy_unlinked).

3.4 The policy in effect

A verifier is given a room’s policies in a list. The first has prev zero and effective 0. Each later one has as prev the digest of the one before it in the list, and an effective above that one’s. A policy that breaks this is refused ens_policy_unlinked. The policy in effect at room-clock instant t is the last of the list whose effective is at or below t. A policy is replaced, never amended.

4. Statements

4.1 Carriage

magicstatementsectionwho signs it
WJS1presentation statement§5.1a performer’s page, with the performer’s key
WJL1latency statement§7.4a room key

Each statement is the payload of a meta event: its magic, a version byte of 1, and the fields its section lists. A meta payload whose first four bytes are neither magic is not this extension’s, and a sink that does not implement it shows neither: they are timed metadata for its application (live-events §4.2). Every statement is bounded by its author’s policy (live-events §5, max_payload), 1024 bytes for a room key, which bounds what a latency statement lists.

4.2 Reading

A statement that ends before a field, that holds bytes after its last field, whose version is not 1, or whose field holds a value its section does not allow, is ens_statement_malformed.

4.3 Who signs

A presentation statement by a room key, and a latency statement by any other principal, are ens_not_permitted. Since a statement is a meta event, the room verifier has already held it to its author’s grant: a principal whose policy does not permit meta cannot send one (live-rooms §9.2 step 4, kind_not_permitted).

4.4 The delay attribute

This extension reads one grant attribute (live-rooms §4.4):

keyvaluemeaning
delay_barsa decimal integer from 1 to 64the performer plays in whole-bar delay of this many bars with every other performer (§6.1)

A decimal integer here is 1 or 2 of the digits 0–9, the first not 0. A grant whose delay_bars is not of that form, or is above 64, is refused ens_attr_malformed. A grant without it plays live. The authority gives the attribute to a performer whose path to the others is longer than the threshold allows (§7.5), which a latency statement shows, and takes it away by a grant exchange (live-rooms §4.6).

4.5 The policy a statement is read under

A statement is read under the policy in effect at its event’s t. A statement at an instant with no policy in effect is refused ens_no_policy.

5. Presentation statements

5.1 Layout

fieldsizemeaning
magic, version5WJS1, 1
room32the room id
listener32the key that signs it: the performer’s whose page presented
at8u64: the room-clock instant it describes
entries1 + 40 × na u8 count (1 to 16), then each entry: source (32, a performer’s key) and capture (u64, the capture instant of the sample of that source’s sound the page was presenting at at)
sig64Ed25519 by listener over the 26 bytes wai:ensemble-presentation (ASCII) and 0x01, followed by the record from magic through entries

A statement whose entries are not in strictly ascending order of source, or one of whose entries names the listener, is ens_statement_malformed. Its digest, where a party names it apart from its event, is SHA-256 over the record from magic through entries.

The statement carries its own signature beside its event’s, so that it stands on its own, as a co-watch presentation statement does (co-watch §7.1): a performer that shows a third party how it heard the others presents its statements and its grant, with no other event of the room.

5.2 Rules

A presentation statement is checked, in this order:

  1. its room is the room id: otherwise wrong_room;
  2. its listener is its event’s principal: otherwise ens_wrong_key. A statement lifted from another performer’s chain, or made by a key the performer does not hold, is so refused;
  3. its signature verifies under listener (§5.1): bad_key, bad_signature;
  4. its at is at most its event’s t: a statement describes what was presented, never what will be. Otherwise ens_presentation_ahead.

5.3 The page

A performer’s page in a room with an ensemble policy MUST, while it is present and presents another performer’s sound:

The capture instant is the source’s statement, carried in its media, and the presentation instant the listener’s: their difference is the whole path from one performer’s microphone to another’s ears, capture and encoding, the network and every relay, the jitter buffer, decoding and output, each counted once. A page that presents a source in whole-bar delay presents it as §6.3 says.

6. Whole-bar delay

6.1 A pair’s delay

A pair’s delay at instant x, in bars, is the greater of the delay_bars of its two performers’ grants in effect at x, a grant without the attribute counting 0. A pair whose delay is 0 plays live: each hears the other as soon as the path allows. A pair whose delay k is above 0 plays in whole-bar delay: each hears the other k bars late. The delay is the same in both directions of a pair, so that two performers in whole-bar delay hear each other alike.

Music whose bars repeat in patterns of k bars is played over any distance this way: what the listener plays along to is the source’s bar b, presented in bar b + k, and when the pattern repeats every k bars the two fit. The delay is chosen whole for that reason, and a fraction of a bar is not offered: a delay that is not whole bars puts the sound off the beat, which is what the threshold already measures.

6.2 The instant a sample is due

A sample of a pair in whole-bar delay of k bars, with capture instant c, presented at x, is due at

c + I(b + k) − I(b)

where I(n) is the instant bar n begins and b the bar holding c, both on the beat clock held at x (co-performance §6.8): the same place in the bar, k bars on, as the bars that began from b last. At a steady tempo that is c plus k bars’ length. A sample whose capture instant no bar holds (before the beat clock’s origin), presented when no beat clock is held, or whose due instant does not fit a u64, has none: it is unplaced. A placed sample’s alignment error is |x − its due instant|.

The beat clock read is the one held at the presentation, not at the capture: a change of tempo held by then is in it, and every bar from b to b + k that began by x is final in it (co-performance §6.8), so the page that presented at x and a verifier reading the whole room compute one instant. A change of tempo stated less than k bars ahead moves the due instants of samples already captured; a beat clock that states its changes at least k bars ahead keeps every due instant where the page first scheduled it.

6.3 What a page does

A page that presents a source in whole-bar delay MUST present each sample of it at its due instant (§6.2) on its estimate of the room clock: it holds what it has received until then, and a sample that reaches it later than its due instant plus align_us it does not present, since it is no longer in its bar. It states its presentation as §5.3 says, so that the alignment of each sample it states is checked (§7.5). A page SHOULD show its performer the bar and beat on the beat clock, and which others it hears live and which k bars late.

7. Latency

7.1 Windows and slots

A window is a stretch of the room clock, from from up to, but not including, to, over which one policy is in effect and whose length is a whole number of that policy’s slots, from 1 to 1000. Its slots are from + j × P to from + (j + 1) × P, j from 0, with P the policy’s period_ms × 1000. A room clock that states windows regularly states them window_ms long, starting at multiples of window_ms × 1000; §7.4 checks any window a statement names.

7.2 Due slots and the pairs of a window

A principal is present from the t of a join of it to the t of the following leave of it, if any. A source publishes its sound from the t of a publish of its own audio track (live-rooms §7.2: no slot, media 1) to the t of the following unpublish of that track or leave of the source, if any. A slot is due for pair (l, s) when:

The pairs of a window are the pairs with a slot of the window due.

7.3 Samples

A latency statement names, for each performer it lists, held: how many of the performer’s events the room clock held when it made the statement (its next seq). A pair’s sample in a slot is, of the listener’s accepted presentation statements whose event’s seq is below its held, whose at is within the slot and which have an entry for the source, the first in the merged order, with that entry’s capture instant. A statement the clock had not received when it made its statement is not a sample of that statement, whenever it arrived.

A pair’s delay for a window is its delay (§6.1) at the start of its first due slot of the window, and each of its samples in the window is measured by it:

A sample with a measure is counted.

7.4 The latency statement

fieldsizemeaning
magic, version5WJL1, 1
from, to8 eachu64: the window (§7.1)
threshold_us4u32: the threshold it judges by, the policy’s
quantile1the percentile it judges at, the policy’s
align_us4u32: the bound it judges a whole-bar delay by, the policy’s
performers1 + 40 × va u8 count (1 to 16), then each performer: principal (32) and held (u64)
pairs1 + 20 × na u8 count (1 to 240), then each pair: l, s (u8 each, indexes into performers), bars (u8), due, n (u16 each), p50, pq, max (u32 each), verdict (u8)

For a pair: bars is its delay for the window (§7.3), 0 to 64; due is its due slots and n those in which it has a counted sample; p50, pq and max are the 50th and the quantile-th percentile and the greatest of the measures of its counted samples, and all three 0 when it has none; verdict is §7.5’s. A value above 2^32 − 1 is written 2^32 − 1.

A latency statement is ens_statement_malformed when: its performers are not in strictly ascending order of principal; its pairs are not in strictly ascending order of (l, s), or a pair’s l is its s or either is not an index of performers; a performer is in none of its pairs; a bars is above 64 or a verdict outside 1 to 3; from is not below to; to is above its event’s t; or the policy in effect at from is not in effect up to to, or to − from is not a whole number of its slots from 1 to 1000. Then, under the policy in effect at from:

  1. threshold_us, quantile and align_us are the policy’s: otherwise latency_mismatch;
  2. each pair it lists is a pair of the window, each performer’s held is at most the number of that performer’s events the room holds, and each pair’s entry is the one recomputed from the room: otherwise latency_mismatch.

A statement may list some of a window’s pairs, since one holds at most what the room key’s max_payload allows; it lists the performers its pairs name. §7.6 requires every pair of the window to be listed by one.

7.5 Percentiles and verdicts

The q-th percentile of n values, n above 0, is the value at index ⌈q × n / 100⌉ − 1 of the values in ascending order: the nearest rank, as co-watch §8.5 takes it. A pair’s verdict is 1, 2 or 3:

“Together” is so a claim with a stated threshold, checked against the statements, and “aligned” one with a stated bound, checked against every sample. A pair whose listener’s statements are missing for a slot that was due is unknown, never together: a page that stops stating is not counted as keeping time.

7.6 Completeness

A room clock that states a window states all of it. Once every record is read, for each window that an accepted latency statement names, every pair of the window is listed by a latency statement of that window: otherwise latency_incomplete, at the window’s first latency statement in the merged order.

7.7 Who plays with whom

For a window that accepted latency statements name, two performers a and b with a pair of the window between them are judged by the pairs of the window between them, (a, b), (b, a) or both: they play together when each is together; play in whole-bar delay when each is aligned; are unknown when any is unknown; and are apart otherwise. A performer that does not listen (one whose grant may not subscribe, as a backing source’s) is so judged by the one direction it has: whether the others play to it in time. Each pair is read from the first latency statement of the window, in the merged order, that lists it. Every instant of the window has that window’s answer, so anyone holding the room recomputes, for each instant a latency statement covers, who could play with whom, and how.

8. Where each rule is enforced

After the fact, a verifier (§9) checks every one of these against the room: a statement signed by a key that is not its performer’s, a latency statement that does not recompute, that calls a pair together whose samples are missing, or that leaves a pair out, is found out.

9. Verification

9.1 Inputs

An ensemble verifier is given what a co-performance verifier is given (co-performance §9.1: a descriptor, grants, a key-log head or none, the chains, the room’s co-performance policies, the objects and the payloads), and:

9.2 Steps

It runs these steps and stops at the first refusal, which it reports with the reason and where:

  1. The room and its co-performance (co-performance §9.2, every step, which begins with the room, live-rooms §9): a refusal, at co with that verifier’s own place and reason. A gap is not a refusal; the steps go on, and the checks of a latency statement that recompute from the room (§7.4 step 2, and §7.6) are not made: each depends on events the verifier may not hold. The beat clocks the room holds are those co-performance’s verification accepted.
  2. The policies, in the list’s order (§3.3, §3.4): at the policy’s index.
  3. The grants’ attributes (§4.4), in the grants’ order: at the grant’s index.
  4. The statements, in the merged order: each meta event the room accepted whose payload begins with a magic of §4.1: §4.2, then §4.5 (ens_no_policy), then §4.3, then the rules of its section, in the order each section gives. A latency statement’s checks that recompute (§7.4 step 2) are made after every statement has been read, in the merged order, since its samples may follow it there. At the record.
  5. Completeness (§7.6): at the record.
  6. The outcome (§9.3).

9.3 Outcomes and what a verification states

With accept or gap, the verifier states:

10. Reasons

The codes this extension adds, each with where it is raised:

reasonraised at
ens_policy_malformed§3.3
ens_policy_unlinked§3.4
ens_attr_malformed§4.4, §9.2 step 3
ens_statement_malformed§4.2, §5.1, §7.4
ens_no_policy§4.5
ens_not_permitted§4.3
ens_wrong_key§5.2 step 2
ens_presentation_ahead§5.2 step 4
latency_mismatch§7.4
latency_incomplete§7.6

An ensemble verifier also raises, at co, every code a co-performance verifier raises (co-performance §10, and the room’s codes it lists); truncated, malformed and unknown_version for a policy that does not read; wrong_room, bad_key and bad_signature for a policy of another room or whose signature fails, and for a presentation statement (§5.2); and delivery_gap for a gap, which is not a refusal.

11. Conformance

live-conformance/ensemble/ holds the corpus, in one file, vectors.json, written from fixed seeds by wai_ensemble_vectors gen. It has rooms, each a room (descriptor, grants, key-log head or none, chains), its co-performance policies, objects and payloads, its ensemble policies, and the outcome §9.3 gives them with everything it states:

A conforming verifier gives every room the outcome, the reason, the place, and everything else the corpus states. wai_ensemble_vectors verify and the test tests/ensemble.rs check that the file regenerates byte for byte, that wai-rs gives every room its outcome, that every reason of §10 appears, and that §10 lists exactly the reference’s codes. live-conformance/ensemble_verify.py is the second implementation, written from this text: it shares no code with wai-rs; it verifies each room’s co-performance with that corpus’s second implementation (co_performance_verify.py, which verifies the room with live_rooms_verify.py) and every signature with WAI’s strict rule (claim-conformance/_ed25519.py); and it mutates copies of the corpus and requires every copy to fail, and drops each of its own checks in turn and requires the corpus to catch the weakened verifier.

12. Security and privacy considerations