Skip to main content

WAI Extension: Fleet

Mirrored from the canonical text at commit 260cbd09 ().

Status: Draft. Registers the record wai.delivery.fleet: the nodes an organisation runs, enrolled by their signing keys, and a report of what they did that anyone holding their records recomputes. The organisation signs an enrolment, WFE1, for each node: the node’s Ed25519 key, its role (a relay, a compositor, an ingest gateway, a room service, a ledger or an audit collector), a label and the span the enrolment holds. A node contributes the records it already signs, as it signs them: a relay’s session summaries, and every hop’s per-hop session records (live-metering §3) with the room descriptors that place them in time. For a period, the organisation signs a fleet report, WFR1: one line per node and one per room, with the sessions, bytes and energy their records state, each line citing the records it came from, and the fleet’s totals. Energy keeps its acquisition class and its declared uncertainty, one row per class and per scope, never summed across either. Every unmetered figure is counted as unmetered, never as zero joules, and a total over measured and unmetered figures states that it is partial. A verifier given a report with its enrolments and records, a bundle, recomputes every line and refuses a report that does not recompute. No accounts, no billing and no new transport. The reference implementation is the fleet module of wai-rs (feature fleet) and the service wai/wai-fleet; the corpus is live-conformance/fleet/, and its second implementation is live-conformance/fleet_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: the WAI texts this one cites; RFC 3629; RFC 8032; RFC 8259; FIPS 180-4.

1. Scope

In scope: the organisation key and the enrolment of a node (§3); the records a node contributes, how each is read, verified and placed in time, and how one thing is counted once (§4); the figures a report states for each node, each room and the fleet, with their energy and citations (§5); the report, its signature and its rules (§6); the bundle and its verification (§7); how the fleet composes with the receipt ledger (§8); and the reference service and its operations (§9).

Out of scope: accounts, members, single sign-on, quotas, billing and any control plane shared by several organisations; a transport; how a node measures its energy, which energy-measurement governs; the records a node signs, which their own texts and implementations define and this one reads unchanged; and prices, which the receipt ledger states.

Relationship to the core spec: wai.delivery.fleet is a record (SPEC §5): a signed record a WAI tool emits and verifies. It names no payload format and is not dispatchable.

2. Terms

3. Enrolment

3.1 Layout

fieldsizemeaning
magic4WFE1 (ASCII)
version11 in this revision
organisation32the organisation key, which signs the enrolment
node32the node key
role1the node’s role (§2)
from, until8 eachu64: the span the enrolment holds, from from (included) to until (excluded); until is 2^64 − 1 for a span with no end
label1 + na u8 n, then n bytes of UTF-8 (RFC 3629): the node’s name in reports, empty for none
sig64§3.2

3.2 Digest and signature

The digest of an enrolment is SHA-256 over its bytes before sig. Its signature is Ed25519 by organisation over the 16 bytes wai:fleet-enrol (ASCII) and 0x01, followed by the bytes the digest covers, verified strictly, as SPEC §7 Signatures requires: a key that fails step 1 or 2 is bad_key, and a signature that fails a later step is bad_signature.

3.3 Reading and rules

A reader reads exactly the bytes it is given as one enrolment, as live-events §3.3 reads a record, with WFE1 for WEV1 (truncated, malformed, unknown_version). It then refuses as enrolment_invalid an enrolment that breaks any of these:

Then the signature (§3.2).

3.4 What each role contributes

rolecontributes in this revision
1 relayits session summaries (§4.2) and its per-hop records of role 1 (§4.3)
2 compositorits per-hop records of role 2
3 ingest gatewayits per-hop records of role 3
4 room service, 5 ledger, 6 audit collectornone: the node is enrolled and listed, and reported silent

A record of a kind its signer’s role does not contribute, and a per-hop record whose role is not its signer’s, is refused role_mismatch. A node holds one enrolment in a fleet: a second enrolment of the same node key is refused enrolment_duplicate.

4. Records

4.1 Kinds and their carriage

A record is carried as a JSON object (RFC 8259) whose kind names its kind, with exactly the members of its kind:

kindmembersrecord kindidentitysigner
hoprecord and ledger, hexadecimal: a per-hop record and its ledger (live-metering §3)2its digest (live-metering §3.5)its hop
summaryline, a string: one line of a relay’s summary file, without its line end (§4.2)10SHA-256 over the summary’s text, Sits relay_key
descriptordescriptor, hexadecimal: a room descriptor (live-rooms §3)11its digest, which is the room id (live-rooms §3.2)none: its digest binds it (§4.4)

Kind 2 is the kind and identity receipt-ledger §5.2 gives a per-hop record, so a fleet report and an invoice name such a record alike (§8); kinds 10 and 11 are kinds that table does not use. A carriage that is not such an object, whose kind is none of these, or whose members are not exactly its kind’s, each of its type, is refused malformed.

4.2 The session summary

The reference relay keeps a summary of every MOQT session it serves, signs it when the session ends, and appends it to its summary file, one line each (live-metering §8). Its format is named joule-moq-relay/session-summary/2.

The line. A line is exactly the 11 bytes {"summary":, the summary’s text S, the 14 bytes ,"signature":", 128 hexadecimal digits (the signature, 64 bytes), and the 2 bytes "}. That is the form the reference relay writes; a line of any other form is malformed.

The text. S is read as JSON text (RFC 8259) of one object: valid UTF-8, no member named twice in one object, no value inside more than 63 objects and arrays, no escaped surrogate that is not one of a pair, and nothing after the value; otherwise it is malformed. An integer here is a number spelled as a plain decimal integer, with no sign, fraction or exponent and no leading zero, below 2^64. These members are read, in the order of the table, and the first that is missing or of another type is malformed:

membertypemeaning
formatstringjoule-moq-relay/session-summary/2; any other string is unknown_version
relay_keystring of 64 hexadecimal digitsthe relay’s node key, the signer
sessionintegerthe relay’s number for the session
opened_unix_ms, closed_unix_msintegerwhen the session opened and closed, in milliseconds since 1970-01-01T00:00:00Z
relay_energyobjectthe session’s own energy figure: acquisition (string) and joules_micro (integer)
tallyobjectwhat the session carried
tally.peerstring of 64 hexadecimal digits, or absentthe relay at the other end, when the session is a hop link (live-metering §3.10)
tally.publisherobjectholding object_bytes_received (integer): the payload bytes the session published
tally.subscriberobjectholding object_bytes_forwarded (integer), the payload bytes forwarded to the session live, and tracks, an object whose every member is an object holding, or not, fetch_bytes (integer; 0 when absent): the payload bytes it was served by fetch and fill on that track

Every other member is signed and is not read: the reference relay adds members as it counts more, and the signature covers them. Of those, tally.publisher_signed states the figures publishers signed for producing the objects the session received, which are the publishers’ energy and not the relay’s: a report states none of them.

A summary that reads is then checked in this order: its energy, its rules, its signature.

Energy. relay_energy holds exactly its two members. With acquisition unmetered and joules_micro 0, the figure is the unmetered marker; with ModelBased or Estimator and joules_micro above 0, it is a figure of tag 2 or 3. Anything else is refused energy_invalid: another label, a figure beside unmetered, a class beside zero, and the classes whose evidence (energy-measurement §2.7) a summary has no member to carry. The reference relay states the marker always: sessions run together in one process, and its counter cannot tell their shares apart (live-metering §8).

Rules. A summary is refused record_invalid when opened_unix_ms is above closed_unix_ms, when closed_unix_ms × 1000 is 2^64 or more, or when the bytes out it states, its object_bytes_forwarded and the sum of its tracks’ fetch_bytes, add to 2^64 or more.

Signature and identity. The signature is Ed25519 by relay_key over the 32 bytes joule-moq-relay:session-summary (ASCII) and 0x01, followed by S, verified strictly (SPEC §7 Signatures: bad_key, bad_signature). The identity is SHA-256 over S.

4.3 The per-hop record

A per-hop record is read as live-metering §3.6 reads one, of the room its own room field names, with that section’s codes (truncated, malformed, unknown_version, energy_invalid, record_invalid), and its signature is verified as §3.5 there requires (bad_key, bad_signature). Its ledger is read as §3.6 there reads a ledger (ledger_invalid), and has grants entries and the root ledger: otherwise ledger_mismatch. The size bound of live-metering §3.7 is not applied: it depends on the media tracks the room published, which a fleet does not hold, and the reference service bounds what one request may carry instead (§9).

Its figure is its joules with its class, in scope 2: the hop’s own energy over its span, which the reference relay states as OnChipCounter with its evidence where its counter was live across the span and it served that room alone, and as the unmetered marker otherwise (live-metering §8).

4.4 The room descriptor

A per-hop record states its span on its room’s clock. The fleet reads, of the room’s descriptor (live-rooms §3.1), only what places that span in time: magic, which is WRD1 (fewer than 4 bytes: truncated; another: malformed); version, which is 1 (unknown_version); a length of at least 141 bytes (truncated); authority, the 32 bytes after the version; and t0, the u64 at byte 69. Its signature is verified as live-rooms §3.2 requires: Ed25519 by authority over the 14 bytes wai:live-room (ASCII) and 0x01, followed by every byte but the last 64, strictly (bad_key, bad_signature). Its digest is SHA-256 over every byte but the last 64, and is the room id.

A descriptor needs no enrolment. Its digest binds it to the room id a per-hop record names, and the record’s own signature binds the room id to its hop, so the t0 that places a record is the t0 of the room its hop signed for. The descriptor’s other fields are the room verifier’s to read (live-rooms §9).

4.5 Placement in time

Each summary and each per-hop record has an instant:

A record is in a period when its instant is at or after the period’s start and before its end; otherwise it is refused outside_period. It is within its enrolment when its instant is at or after its signer’s from and before its until: otherwise outside_enrolment. So of consecutive periods, each record is in exactly one.

4.6 One thing once

Of two records, the later is refused:

5. The figures

A report’s figures are computed from its records: every summary and every per-hop record, each in its period and within its enrolment, and the descriptors that place them.

5.1 Node lines

There is one node line for each node that signed at least one summary or per-hop record, in ascending order of node key:

figurewhat it is
node, rolethe node key, and its enrolment’s role
sessionsits summaries with no tally.peer: the sessions it served for clients
linksits summaries with a tally.peer: its hop links to other relays
bytes_in, bytes_outover its summaries with no tally.peer: the sum of object_bytes_received; the sum of object_bytes_forwarded and the tracks’ fetch_bytes
link_bytes_in, link_bytes_outthe same, over its summaries with a tally.peer
hopsits per-hop records (u32)
hop_bytes_in, hop_bytes_outover its per-hop records, the sum of bytes of their track entries in (dir 1) and out (dir 2)
energyits summaries’ figures in scope 1, its per-hop records’ in scope 2 (§5.4)
citationsits summaries and per-hop records (§5.5)

The bytes of the summaries and the bytes of the per-hop records are two accounts of a relay’s traffic and are never added: a relay’s summaries count every object of every session, and its per-hop records count the objects of each room’s tracks, a hop link’s at both of its ends (live-metering §3.10). A compositor or an ingest gateway signs no summaries, and its traffic is in its per-hop records alone.

5.2 Room lines

There is one room line for each room a per-hop record names, in ascending order of room id:

figurewhat it is
room, t0the room id, and its descriptor’s t0
hopsits per-hop records (u32)
sessionsthe sum of their sessions: subscriber sessions, each counted at the hop that served it (live-metering §3.3)
subscriber_usthe sum of their subscriber_us over the nine layer classes
bytes_in, bytes_outthe sum of bytes of their track entries in and out
energytheir figures, in scope 2
citationsits per-hop records and its descriptor

A room’s bytes are the bytes its hops handled: an object carried over a hop link is counted out at one relay and in at the next.

5.3 Totals

The totals are the sums over the node lines of sessions, links, bytes_in, bytes_out, link_bytes_in, link_bytes_out, hops, hop_bytes_in and hop_bytes_out; and their energy is every summary’s figure in scope 1 and every per-hop record’s in scope 2.

5.4 Energy

Each summary states one figure, in scope 1, and each per-hop record one, in scope 2. Over the records of a line, or of the totals, for each scope:

A row never sums figures of two classes, and no row sums two scopes, as receipt-ledger §6.6 requires of an invoice: a counter reading and a model’s output added are neither. A scope’s coverage is whole when it has a row and no unmetered figure, partial when it has both, unmetered when it has unmetered figures alone, and none when it has no figure. The rows of a partial scope are the energy of its metered records alone, beside a count of records whose energy nobody measured, and MUST NOT be presented as the energy of the line or of the fleet. A report whose unmetered for a scope is below the unmetered markers of its records has counted a figure nobody measured as something else, and is refused unmetered_summed.

A sum that a u64 cannot hold cannot be stated, and an organisation does not sign a report that would state one.

5.5 Citations

Each line cites the records it came from (§5.1, §5.2): cited is their number, and cites is SHA-256 over the 16 bytes wai:fleet-cites (ASCII) and 0x01, followed by each cited record’s kind (u8) and identity (32 bytes), in ascending order of (kind, identity). Every cited record is in the report’s records, and a line that cites none is refused line_unsourced. A reader shows one line to an auditor with the records it cites and nothing else, and the auditor recomputes it.

6. The report

6.1 Layout

fieldsizemeaning
magic4WFR1 (ASCII)
version11 in this revision
organisation32the organisation key, which signs the report
start, end8 eachu64: the period it reports
records4 + 33 × ra u32 count, then each record: kind (u8) and identity (32), in strictly ascending order of (kind, identity)
enrolments2 + 64 × ka u16 count, then each enrolment: node (32) and its digest (32), in strictly ascending order of node
nodes2 + …a u16 count, then each node line (§6.2), in strictly ascending order of node
rooms2 + …a u16 count, then each room line (§6.2), in strictly ascending order of room
totals68 + …§6.2
sig64§6.3

6.2 Lines, totals and the energy block

A node line is node (32), role (u8), sessions, links, bytes_in, bytes_out, link_bytes_in and link_bytes_out (u64 each), hops (u32), hop_bytes_in and hop_bytes_out (u64 each), an energy block, cited (u32) and cites (32).

A room line is room (32), t0 (u64), hops (u32), sessions, subscriber_us, bytes_in and bytes_out (u64 each), an energy block, cited (u32) and cites (32).

The totals are sessions, links, bytes_in, bytes_out, link_bytes_in and link_bytes_out (u64 each), hops (u32), hop_bytes_in and hop_bytes_out (u64 each), and an energy block.

An energy block is a u8 count e, then e rows, each scope (u8), tag (u8), figures (u32), joules (u64, µJ) and max_ppm (u32), in strictly ascending order of (scope, tag); then unmetered, two u32: the unmetered figures of scope 1 and of scope 2.

6.3 Digest and signature

The digest of a report is SHA-256 over its bytes before sig. Its signature is Ed25519 by organisation over the 17 bytes wai:fleet-report (ASCII) and 0x01, followed by the bytes the digest covers, verified strictly (SPEC §7 Signatures: bad_key, bad_signature).

6.4 Reading and rules

A reader reads exactly the bytes it is given as one report, as §3.3 reads an enrolment with WFR1 for WFE1 (truncated, malformed, unknown_version), and refuses as classes_summed a report with a row of tag 0: a sum over figures of no one class. It refuses as report_invalid one that breaks any of these:

Then the signature (§6.3).

7. The bundle and its verification

7.1 The bundle

A bundle is one JSON object with exactly these members: kind, the string wai-fleet-bundle; report, the report in hexadecimal; enrolments, an array of enrolments in hexadecimal; and records, an array of records in their carriage (§4.1). A bundle that is not such an object is refused malformed, at bundle. A file that holds one is named with the suffix .wai-fleet.json.

The report’s records are every record of the bundle, and its enrolments are every enrolment of the bundle: a bundle is a report with exactly what it rests on.

7.2 Steps

A verifier given a bundle runs these steps and stops at the first refusal, which it reports with the reason and where:

  1. The report (§6.4), at report: truncated, malformed, unknown_version, classes_summed, report_invalid, bad_key, bad_signature.

  2. Each enrolment, in order, at enrolment and its index: it reads and its signature verifies (§3.3: truncated, malformed, unknown_version, enrolment_invalid, bad_key, bad_signature); its organisation is the report’s (wrong_organisation); its node is no earlier enrolment’s (enrolment_duplicate); and its span meets the report’s period, from below the period’s end and the period’s start below until (outside_period).

  3. Each record, in order, at record and its index: it reads and its signature verifies, by its kind (§4.1 to §4.4, with their codes).

  4. Each summary and per-hop record, in order, at record and its index: its signer is the node of an enrolment (signer_not_enrolled); its kind and its role are its signer’s role’s (§3.4, role_mismatch); it is placed (§4.5, descriptor_missing); it is in the report’s period (outside_period); and it is within its enrolment (outside_enrolment).

  5. Across records, at record and the index of the later (§4.6): record_duplicate, record_overlap.

  6. The figures, recomputed from the bundle as §5 computes them, with the report’s records and enrolments (§7.1), and compared in this order:

    1. a node line whose node signed no summary or per-hop record of the bundle, a room line whose room no per-hop record of the bundle names, or a line whose cited is 0: line_unsourced, at report, nodes or rooms, and the line’s index;
    2. a row of a line or of the totals whose joules is not the sum of the figures of its scope and class, and is the sum of the figures of its scope over its class and at least one other class of that scope: classes_summed, at the line’s place, or at report and totals;
    3. a line or the totals whose unmetered for a scope is below the unmetered markers of its records (§5.4): unmetered_summed, at the same places;
    4. a line whose cited or cites is not the records it came from: citation_mismatch, at the line’s place;
    5. any other difference: the records, the enrolments, a line missing or of another subject or role, a figure, a row, unmetered, or the totals: report_mismatch, at report.

    Steps 6.2 to 6.4 compare each line of the report with the recomputed line of the same node or room where there is one. A figure that cannot be held in its field is a difference.

7.3 Outcome

A silent node is not an unmetered one. An unmetered node signed records whose energy nobody measured; a silent node signed no record in the period, and the report states nothing of what it did.

8. With the receipt ledger

The fleet and the receipt ledger read the same records and compute different things from them, and neither recomputes the other’s:

9. The reference service and its operations

wai/wai-fleet runs a fleet as an HTTP service over a directory it owns, under the organisation key:

requestdoes
GET /v1/fleetthe service, its organisation key, its limits
GET /v1/fleet/healthok
POST /v1/fleet/enrolments{"enrolment": hex}: an enrolment the organisation key signed, read and verified (§3), refused for a node already enrolled (enrolment_duplicate), kept
GET /v1/fleet/enrolmentsevery enrolment held, with its node, role, label, span, digest and the records held of it
POST /v1/fleet/records{"records": [...]}: each record (§4.1), in order, verified as §7.2 steps 3 to 5 verify it against the enrolments and the records held, with no period, a descriptor held before it or earlier in the request; each held record is kept, and the answer gives each record’s kind and identity or its refusal
GET /v1/fleet/rooms/{room}/hopsthe per-hop records held of a room, each record and ledger (§8)
POST /v1/fleet/reports{"start", "end"}: the report of the period over the enrolments whose span meets it and the records held in it, with the descriptors those name, signed, verified as §7 verifies it, and kept with its bundle
GET /v1/fleet/reportsevery report kept, with its period
GET /v1/fleet/reports/{digest}the report’s bundle and its outcome
POST /v1/fleet/verifya bundle: its outcome (§7.3)

Each refusal names its reason and place. The service writes every file whole, appends records and never rewrites one, holds a lock on its directory so that one process writes it, and on start reads every enrolment, record and report back and verifies each again. It has no accounts: an enrolment is accepted only when the organisation key signed it, and a record only when it verifies under an enrolled node’s key, whoever submits either. The service listens on a loopback address unless it is told otherwise; who may reach it is decided in front of it.

wai-fleet collect reads a node’s published summary file, a local file or an http or https address answering with its lines, and submits each line to a running service as a summary record.

The operations, named by the toolchain’s rule (an operation noun.verb, its command line the words in order):

operationcommand linerouteeffect
enrolment.createwai enrolment create <node> --role <role> --key <file> [--label L] [--from µs] [--until µs] [--out F] [--target URL]POST /v1/fleet/enrolments with --targetwrites a file; with --target, the service
enrolment.listwai enrolment list --target URLGET /v1/fleet/enrolmentsreads
report.createwai report create --target URL --start µs --end µsPOST /v1/fleet/reportsthe service
report.exportwai report export <digest> --target URL [--out F]GET /v1/fleet/reports/{digest}writes a file
receipt.verifywai verify <bundle>POST /v1/fleet/verifyreads

Each keeps the toolchain’s agent rules: --json writes its JSON envelope; each that writes takes --dry-run, which writes nothing, sends nothing and says what it would; one that changes the service, or would replace a file, waits for --yes (--confirm, deprecated, is the same), and without it exits 4 with the proposed change and the command that confirms it.

10. Reasons

The codes this extension adds, each with where it is raised:

reasonraised at
enrolment_invalid§3.3, §7.2 step 2
enrolment_duplicate§3.4, §7.2 step 2
wrong_organisation§7.2 step 2
report_invalid§6.4, §7.2 step 1
signer_not_enrolled§7.2 step 4
role_mismatch§3.4, §7.2 step 4
descriptor_missing§4.5, §7.2 step 4
outside_enrolment§4.5, §7.2 step 4
line_unsourced§5.5, §7.2 step 6
unmetered_summed§5.4, §7.2 step 6
report_mismatch§7.2 step 6

A verifier of this extension also raises, with the meaning this text gives them: outside_period (§4.5, §7.2 steps 2 and 4), record_duplicate and record_overlap (§4.6, §7.2 step 5) and citation_mismatch (§5.5, §7.2 step 6), which receipt-ledger §10 adds; classes_summed (§5.4, §6.4, §7.2 steps 1 and 6), energy_invalid and record_invalid (§4.2, §4.3), ledger_invalid and ledger_mismatch (§4.3), which live-metering §9 adds; and truncated, malformed and unknown_version for what does not read, and bad_key and bad_signature for its signature.

11. Conformance

live-conformance/fleet/ holds the corpus, in one file, vectors.json, written by wai_fleet_vectors gen from fixed seeds and from records taken from live-conformance/metering/: the lesson room’s descriptor and its per-hop records, of three classes, with their ledgers; and the relay room’s descriptor with the per-hop record the reference relay signed, which states the unmetered marker. The summaries are written by the generator in the reference relay’s form, and wai-fleet’s tests read each one with the relay’s own types and require its bytes, re-serialised, to be the line’s. Each vector is a bundle with the outcome §7.3 gives it:

wai_fleet_vectors verify and the test tests/fleet.rs check that the file regenerates byte for byte, that wai-rs gives every vector its outcome, that every reason of §10 and every reused code refuses a vector, and that §10 lists exactly the reference’s codes. live-conformance/fleet_verify.py is the second implementation, written from this text: it shares no code with wai-rs, verifies signatures with WAI’s strict rule written from RFC 8032 (claim-conformance/_ed25519.py), reads per-hop records and their ledgers with the second implementation of live metering (live_metering_verify.py), recomputes every line, row and citation itself, gives every vector the same outcome, requires every mutated copy of the corpus to fail, and drops each of its own checks in turn and requires the corpus to catch the weakened verifier.

12. Security and privacy considerations