Skip to main content

WAI Extension: Impression Aggregates

Mirrored from the canonical text at commit 260cbd09 ().

Status: Draft. Registers the record wai.delivery.impressions and measure 5, impressions, of telemetry-aggregation §3 (its §3.1): ad presentations counted over a bearer with no return path. A receiver judges each ad of a shared pod-input record it presented by stitched-presentation’s rules, and turns each presentation into one Prio3Histogram report of a task of that measure, whose scope is the digest of a catalogue (WIC1, §4) that fixes which index of the vector is which creative, region and quartile. It generates the report once, keeps its bytes, and sends them over whatever path it later has, in an upload file (WIU1, §6.4) that relays forward unchanged. It accepts a task only from a task file (WIT1, §5.4) whose signer its pinned key log lists with the authority aggregate/task, and with attestation each report carries a Privacy Pass token bound to its task and its report id, in a private report extension (§7.2). The leader holds each collection for the catalogue’s hold_s, and the aggregators keep the task until its retention instant, so late reports still count (§7.1). The task layout, the task id, the aggregators, the statements (WQS1) and the receipt (WQA1) are telemetry-aggregation’s; a reader verifies a receipt against its catalogue and recomputes a table of presentations and quartile counts per creative and region (§8): a lower bound of what receivers reported, never an audience, beside what each aggregator states it rejected (§7.3). The reference implementation is the impression_aggregate module of wai-rs (feature impression_aggregate); the corpus is live-conformance/impressions/, and its second implementation is live-conformance/impression_aggregate_verify.py. Keywords MUST, MUST NOT, SHOULD, MAY are RFC 2119/8174.

Texts read for this revision: draft-ietf-ppm-dap-20 (DAP) §2.3, §3.1, §3.6, §4.1, §4.1.1, §4.2, §4.2.2, §4.2.3, §4.4.1, §4.4.2, §4.4.2.1, §4.4.2.2, §4.4.3, §4.5.3.3, §4.5.3.4, §4.5.4.3, §4.6.1, §5.1, §6.1.1, §6.3, §6.4.1, §8.1 to §8.4, §8.6.4, §9.2, §9.2.4 and Appendix B.4; draft-irtf-cfrg-vdaf-22 (VDAF) §7.4.3.1, §7.4.4, §7.4.5, §9.2, Table 4, Table 13 and §10 Table 19; RFC 9576 §3.3; RFC 9577 §2.1.1, §2.1.1.2 and §2.2.1; RFC 9578 §6.4 and §6.5; and, informatively, draft-ietf-privacypass-batched-tokens-10 §2, §3 and §7. ISO 3166-1 and ISO 3166-2 were not read; the code form is the one rung-price §2 states. A clause number of one of them in this text is that text’s.

1. Scope

In scope: the catalogue (§4); measure 5’s task, its rules and the task file (§5); the receiver’s contribution, report, upload file, upload plan and settlement (§6); what the aggregators add to DAP: retention, the hold, the token check and what each aggregator rejected (§7); the verification of a receipt of an impression task, and the impression table (§8).

Out of scope: issuing tokens (RFC 9578, and the batched issuance of draft-ietf-privacypass-batched-tokens-10); how a receiver learns its region or sets its clock; running aggregators; choosing a helper’s operator (DAP §8.7); differential privacy; prices and invoices; an audience or a reach; and two-way paths, which keep stream-audit and stitched-presentation.

Relationship to the core spec: wai.delivery.impressions is a record (SPEC §5). It names no payload format and is not dispatchable.

Relationship to telemetry-aggregation: a task of measure 5 is one of its tasks. This text supplies the measure’s configuration and rules (§5.3), its reports (§6.3), what its aggregators add (§7) and the layer a reader puts over its accepted receipts (§8). Every other rule of that text stands.

2. Terms

3. What one-way delivery changes

Telemetry-aggregation §5 (its §5.1) has a sink report when its session ends, over a path it has then, and DAP §6.1.1 says that “Clients are not expected to store any state across any upload flows, nor are they required to implement any sort of report upload retry mechanism”. On a bearer with no return path:

4. The catalogue

4.1 Layout

A catalogue, WIC1, fixes the meaning of a task’s vector. Integers are big-endian, except inside terms, which is copied byte for byte from a pod-input record and is little-endian there (stitched-presentation §4). A short field is a u8 length and that many bytes.

sizefieldmeaning
4magicWIC1 (ASCII)
1version1
4 + longtermsthe measurement terms exactly as a pod-input record carries them (stitched-presentation §4.1): presentation (1), viewability (1), token_type (u16), issuer (a u16 length, then its bytes); ad-presentation §8’s fields
4hold_su32: the seconds after a batch interval ends before the leader completes a collection of it (§7.1)
1sthe signers’ count
32 × ssignersEd25519 keys whose shared pod-input records count, strictly ascending
1cthe creatives’ count
32 × ccreativescreative digests (ad-presentation §4), strictly ascending
1rthe named regions’ count
short × rregionsregion codes, strictly ascending by bytes
1kthe issuer keys’ count
(2 + n) × kkeyseach a u16 length, then the DER SubjectPublicKeyInfo of a type-0x0002 issuer key (ad-presentation §7.3), strictly ascending by bytes

The catalogue’s digest is SHA-256 over all its bytes. It carries no signature: a task binds it as its scope (§5.1), which enters every input share, every aggregate share and the collector-signed receipt, and a task file signs it (§5.4).

4.2 Regions and cells

With creative index i (the catalogue’s order), column j and quartile q (1 to 5, stitched-presentation §7.4), the cell is (i · (r + 1) + j) · 5 + (q − 1), and the vector’s length is L = c · (r + 1) · 5. The chunk length is ⌈√L⌉, the least integer whose square is at least L: VDAF §7.4.4 recommends a chunk length “near the square root of length”, and §7.4.3.1 says the square root gives proofs “up to 50% larger than the optimal”. It fixes the configuration as one value.

4.3 Reading and rules

A reader reads exactly the given bytes, at most 1,048,576 of them. It refuses them catalogue_malformed when they end before a field, go on past the last key, or when:

  1. magic is not WIC1, or version is not 1;
  2. terms breaks ad-presentation §8, does not set bit 0 (claim), or sets attest with a token_type other than 2;
  3. hold_s is 0: a collection at a bucket’s end loses every report of a receiver that had no path at that moment;
  4. s is 0 or above 16, or signers is not strictly ascending;
  5. c is 0, or creatives is not strictly ascending;
  6. a region is not a code; regions is not strictly ascending; or a listed two-byte code x has a listed code beginning x- (rung-price §4.5 rule 3’s rule, for one footprint);
  7. L is above 4,096. L is a multiple of 5, so the largest is 4,095, and a leader’s measurement share is then at most 65,520 bytes (Field128: VDAF Table 4 and Table 13);
  8. k is not 0 when attest is clear, or not 1 to 4 when it is set; a key is not an id-RSASSA-PSS SubjectPublicKeyInfo with a 2,048-bit modulus (ad-presentation §7.3); or keys is not strictly ascending by bytes.

Each catalogue has exactly one encoding. The size bound decides no outcome: a catalogue that meets these rules is under 340,000 bytes, its only long fields being the issuer and at most four keys, each of a u16 length, so a longer buffer is refused whatever the bound. It lets a reader refuse one without reading it.

5. The task

5.1 Measure 5 and its scope

A task of telemetry-aggregation’s measure 5, impressions, has that text’s task_info (telemetry-aggregation §4.1: 153 bytes, label wai-qoe1) with measure 5 and scope the digest of its catalogue. Its VDAF is Prio3Histogram (VDAF §7.4.4, identifier 0x00000004 in §10 Table 19) of the catalogue’s L and chunk length, two aggregators. Its measurement is one presentation’s cell (§6.2).

5.2 Task id and verification key

Its task id is telemetry-aggregation §4.2’s, and its verification key §4.4’s. Two catalogues give two scopes, two TaskConfigurations and two ids.

5.3 Rules

Given a catalogue C that reads (§4.3), a TaskConfiguration is an impression task of C unless, in order:

  1. task_invalid: telemetry-aggregation §4.3 rule 1 refuses it;
  2. catalogue_mismatch: its measure is not 5, or its scope is not C’s digest;
  3. task_invalid when its time_precision is below 3,600; its extensions are not exactly one task_interval (DAP §4.2.3) holding an Interval of a duration of at least 1; or its retention instant, in seconds, (task_interval.start + task_interval.duration) · time_precision + hold_s, overflows a u64 or is above the last instant in seconds, 253,402,300,799 (carousel §4.2; timeliness §2);
  4. measure_mismatch: vdaf_type is not 0x00000004, or vdaf_configuration is not Prio3HistogramConfig {length L, chunk_length ⌈√L⌉} (DAP Appendix B.4);
  5. floor_too_low: min_batch_size is below 50;
  6. aggregator_alone: leader is helper.

These are telemetry-aggregation §4.3 for a task of measure 5: its rule 2 cannot fail, since 5 is a code of its §3, and its rule 3 is steps 2 to 4 here. The codes of steps 1 and 3 to 6 are that text’s, with its meaning.

5.4 The task file

A task file, WIT1, is how a task reaches receivers. Integers are big-endian.

sizefieldmeaning
4magicWIT1 (ASCII)
1version1
2 + ntaska u16 length, then the TaskConfiguration (DAP §4.2)
4 + ncataloguea u32 length, then the catalogue
nleader_hpkethe leader’s HpkeConfigList as DAP §4.4.1 encodes it, its own u16 length first
nhelper_hpkethe helper’s
32signeran Ed25519 key
2 + nsigner_ida u16 length, then the signer’s identity string (jwp-receipts §7.1), ASCII
64sigEd25519 by signer over wai:impression-task (ASCII), 0x01, then every byte before sig, verified strictly (SPEC §7 Signatures)

The task file’s digest is SHA-256 over the bytes before sig. Its signed bytes include signer and signer_id, so it is a receipt of another WAI family that jwp-receipts §7.4 confirms.

A receiver holds key-log revisions and anchor records as bearer-receipts §8 (its §8.3) says: provisioned logs pinned, and logs that arrive on the bearer held. It accepts a task file unless, in order:

  1. task_file_malformed: the file exceeds 1,048,576 bytes, has another magic or version, ends early or has bytes after sig, or its signer_id is empty or not ASCII; or an HpkeConfigList does not read (DAP §4.4.1: HpkeConfig HpkeConfigList<10..2^16-1>), holds two configurations of one id (DAP §4.4.1: “Aggregators MUST allocate distinct id values”), or holds none of KEM 0x0020, KDF 0x0001 and AEAD 0x0001 with a 32-byte key (DAP §7; telemetry-aggregation §5);
  2. catalogue_malformed: the catalogue does not read (§4.3);
  3. the task fails §5.3 against that catalogue, with that step’s reason;
  4. bad_key, bad_signature: the signature does not verify (SPEC §7);
  5. the signer is not confirmed under jwp-receipts §7.4 with the authority aggregate/task, which is pinned-only (jwp-receipts §7.5), the item untimed: the reason of the step that fails, one of jwp-receipts §7.7’s (signer_id_not_origin_bound, signer_id_names_another_key, origin_not_anchored, a key-log reason of step 3, key_not_listed, key_mismatch, key_revoked, authority_not_granted, anchor_not_pinned, origin_not_expected). Unconfirmed and refused are both not accepted; the reason says which.

A file that reaches step 5 has been read, checked and signature-verified; only the trust decision remains. A revocation of the task key in a revision that arrived on the bearer reaches a receiver whose anchor that revision descends from (bearer-receipts §8.2), and refuses the file key_revoked. An untimed item passes a retired key (jwp-receipts §7.4 step 5): an origin stops a task key by revoking it.

6. The receiver

6.1 What a receiver holds

Evaluating anything here MUST NOT cause a request (unidirectional-delivery §2). A receiver sends only §6.4’s upload files, and token requests, on a path it already has.

6.2 Contribution

Inputs: an accepted task file; the pod-input record, by its bytes; the manifests the player read; the creatives it holds; claims, the claims of this playback session already judged and then the one judged now, each signed by the session key (stitched-presentation §5.2); region, bytes or none; at_ms, the instant the presentation ended by the receiver’s clock, or none when the clock is not set; and pool, the unused pairs of an attested catalogue.

Output: report, with the cell, the creative index, the column, the quartile, the time ⌊at_ms / (1000 · time_precision)⌋, the report id (the chosen pair’s, or a fresh one without attestation) and the pool’s ids discarded; or no report, with the reason of the first step that fails, and the ids discarded:

  1. The record, under stitched-presentation §7.2 steps 1 to 3 with the catalogue’s signers as the keys it accepts. Its response is not checked, as that section’s step 5 allows, and no price is used. Reasons: pod_input_malformed, bad_key, bad_signature, pod_signer_unknown.
  2. record_not_shared: the record’s session is not 16 zero bytes. A record of one session needs the session key to reach its signer (stitched-presentation §4.6), which a receiver with no path cannot do.
  3. record_terms_differ: the record’s measurement-term bytes are not the catalogue’s terms. Every count of a table means one rule.
  4. The claim judged now, under stitched-presentation §7.3 steps 1 to 10, over claims in order, against that record: its reason (stitched_claim_malformed, bad_key, bad_signature, claim_key_unbound, claim_no_decision, claim_manifest_unknown, claim_marker_absent, claim_duplicate, claim_asset_mismatch, claim_frame_mismatch, claim_not_viewable). Step 11 is not run: with attest, the report’s token takes its place (§7.2). claim_duplicate here is one report for a record, an ad and a session.
  5. creative_not_listed: the ad’s creative digest is not among creatives.
  6. instant_unknown: at_ms is none. The upload time is never used in its place.
  7. outside_task: the time is not within task_interval, which is half-open (DAP §4.1.1).
  8. token_exhausted: the catalogue sets attest and no pair of pool passes §7.2 for this task. Pairs are tried in ascending order of report id; a pair whose token fails is discarded and listed.

Otherwise the quartile q is stitched-presentation §7.4’s for the claim, the column is §4.2’s for region, and the cell is §4.2’s.

6.3 The report

For a contribution that gives a report, the receiver generates one report as soon as the presentation ends:

This departs from DAP §6.1.1 in what it expects of a client, not in anything DAP forbids. DAP §4.4.2.2’s “When the Client receives an hpke_unknown_config_id error, it SHOULD invalidate any cached HpkeConfigList and retry with a freshly generated Report” is not followed: that is the regeneration this section forbids (§6.6).

6.4 The upload file

sizefieldmeaning
4magicWIU1 (ASCII)
1version1
32task_idthe task the reports were made for
4 + nuploada u32 length, then a DAP UploadRequest: one or more Reports (DAP §4.4.2), in strictly ascending order of report id

A reader reads exactly the given bytes, at most 16,777,216 of them, and refuses them upload_malformed when they end early or go on past upload, the magic or the version is wrong, upload holds no report, a report does not read as a DAP Report, or the ids are not strictly ascending.

6.5 The upload plan

Inputs: an accepted task file, the stored reports (bytes), an instant t (ms) and a budget b (bytes, or none for no limit).

For a report of time τ, in this order: it is dropped when τ is outside task_interval, or when t is at or past the retention instant · 1000, since the aggregators may then have deleted the task (DAP §6.4.1). Otherwise its deadline is ((τ + 1) · time_precision + hold_s) · 1000, which cannot overflow since τ is within the flight: the earliest instant the hold allows any batch holding τ to be collected. It is timely when t is before its deadline, and late otherwise; a late report may still count.

Order: timely before late; within each, ascending deadline, then ascending id. Budget: reports are taken in that order while 41 plus the sum of their lengths stays within b; the first that does not fit ends the taking. One task’s reports are all of one length (one L, one set of extensions), so nothing smaller follows.

Outputs: send (ids in order, each with its class), defer (ids), drop (ids), and file: the WIU1 of send in ascending order of id, or none when send is empty.

6.6 Settlement

Inputs: a WIU1 the receiver sent, and the response body.

An upload with no response keeps every report. A pending report is dropped by the plan at the retention instant. Outputs: delete and keep, each a list of ids.

7. The aggregators

The aggregators and the collector run telemetry-aggregation §5, §6 and §7 as written for a task of measure 5, with these additions.

7.1 Retention and the hold

Retention. An aggregator keeps the task, its batch buckets, its replay state and the HPKE configurations the task file names until the task’s retention instant (§5.3), and accepts reports sealed to those configurations until then. DAP §6.4.1 lets it delete a task’s data after its task_interval, and §4.4.1 asks it to “continue to accept reports with old keys”; this sets both bounds for a client that holds no newer configuration.

The hold. The leader MUST NOT finish a collection job until its clock is at or past its batch interval’s end in seconds, (start + duration) · time_precision, plus hold_s; until then the job is processing (DAP §3.1, §4.6.1).

7.2 Report extensions and the token

At DAP §4.5.3.4, after its input share decrypts, an aggregator refuses a report:

Each aggregator checks its own input share, so a token in one share alone is refused by the other. attestation_absent and attestation_invalid are ad-presentation §11’s codes. Double spending is the report-id replay check (DAP §2.3): RFC 9577 §2.1.1.2 says that for a non-empty redemption context the double-spend state “only needs to be stored across the set of Origin servers that will accept tokens with that redemption context”, and R names one task and one report.

The codepoint. DAP §9.2.4 sets aside no range for unregistered use, and its registries are Specification Required (DAP §9.2). wai_impression_token, 0x5749, is proposed, with this text as its specification, and is not registered. An aggregator that does not know it refuses every attested report (DAP §4.5.3.4 step 4): no count, never a false one.

7.3 What each aggregator rejected

An aggregator of a task of measure 5 signs its statement in version 2, stating rejected (telemetry-aggregation §6.1): the reports of the batch it rejected at aggregation, so a reader sees beside the count what did not count. Reports with no token, with one of another report’s id, with one that does not verify, and with a measurement that does not check are among them. A report the leader refuses never reaches the helper, and a token in one input share alone is refused by the aggregator that lacks it (§7.2), so the two counts can differ, each aggregator stating its own. Neither is held to the other, and neither is a count of presentations: a rejected report says what an aggregator refused, never what a receiver saw.

A reader accepts a statement of version 1 of such a task, an earlier aggregator’s, beside a statement of either version. It states no count, which a reader reports as null for that aggregator alone, never as 0: the other aggregator’s count stands beside it (§8.1, §8.2).

8. Verification

8.1 A receipt of an impression task

Inputs: the receipt, the catalogue, and optionally the batch’s reports (each report’s id, time and task).

Steps, stopping at the first refusal, which is stated with its reason and place: the catalogue reads (§4.3; catalogue_malformed, at catalogue). Then telemetry-aggregation §8.1 steps 1 to 11 as written, its step 2 being §5.3 here against the catalogue. Then:

  1. at leader: the statements’ batch falls within the task_interval (DAP §4.1.1), otherwise batch_invalid (telemetry-aggregation’s code: a batch the task does not admit);
  2. at result: each element of the result is at most the report count, and the elements’ sum, as integers, is the report count, otherwise count_mismatch. With L at most 4,095 and each element at most a u64 count, the sum fits in a u128.

Outcome: accepted, stating telemetry-aggregation §8.2’s fields (measure impressions, scope the catalogue’s digest), rejected, each aggregator’s count or null for a statement of version 1, which states none, and reports_checked, whether step 11 ran; or the first refusal.

8.2 The table

Inputs: the catalogue, the TaskConfiguration, and receipts in an order, each with its reports where they are given.

  1. The catalogue reads (catalogue_malformed) and the task meets §5.3 against it; otherwise the table is refused with that reason.
  2. Each receipt, in order, is verified by §8.1, and its task must then be the given TaskConfiguration byte for byte (task_mismatch at task, telemetry-aggregation’s code). A refused receipt is stated with its reason and place and takes no further part.
  3. Of the accepted receipts, those with equal digests are one. Two whose batch intervals share a time unit are both set aside, receipts_overlap: DAP collects a bucket once (DAP §2.3; batchOverlap), so an honest pair cannot exist, and a reader holds no ground to choose between them (carousel §6.2 step 4, holdings §5.3). With the receipts sorted by batch start, a receipt overlaps an earlier one when the greatest end before it is past its start, and a later one when the next start is before its end.
  4. Each receipt left stands.

Outcome:

A row is a (creative index, column) pair, for every creative and column in the catalogue’s order, other last. It states presentations, the sum of its five cells; quartiles, five counts, those at quartile at least 1, 2, 3, 4 and 5, as stitched-presentation §7.4 states them; and floor_met, whether presentations is at least 50. Every row is stated: a 0 is a counted zero of a collected batch, never an unknown.

8.3 What a table states

9. Class, money and composition

10. Reasons

The codes this extension adds, each with where it is raised:

reasonraised at
catalogue_malformed§4.3, §5.4 step 2, §8.1, §8.2 step 1
catalogue_mismatch§5.3 step 2
task_file_malformed§5.4 step 1
record_not_shared§6.2 step 2
record_terms_differ§6.2 step 3
creative_not_listed§6.2 step 5
instant_unknown§6.2 step 6
outside_task§6.2 step 7
token_exhausted§6.2 step 8
upload_malformed§6.4
receipts_overlap§8.2 step 3

This extension also raises: telemetry-aggregation §11’s codes, task_invalid, measure_mismatch, floor_too_low and aggregator_alone at §5.3, every code of its §8.1 at §8.1, batch_invalid also at §8.1 step 12, and task_mismatch also at §8.2 step 2; age-assurance §7.3’s count_mismatch (a receipt’s counts are not the counts they must be) at §8.1 step 13; truncated, malformed, unknown_version, bad_key and bad_signature; jwp-receipts §7.7’s confirmation reasons at §5.4 step 5; stitched-presentation §9’s codes and the ad-presentation codes it reuses, at §6.2 steps 1 and 4; ad-presentation’s attestation_absent and attestation_invalid at §7.2; and DAP’s report errors (DAP §4.1) and problem types (DAP §3.6), by DAP’s names, at §6.6 and §7.

11. Conformance

live-conformance/impressions/ holds the corpus, written from fixed seeds by wai_impression_aggregate_vectors gen as vectors.json. Its regions are ISO 3166-1 user-assigned codes and their subdivisions, its origins reserved example domains, and its issuer key RFC 9578’s published test key. It has:

A conforming verifier gives every case the outcome, the reason and everything else the corpus states. wai_impression_aggregate_vectors verify and the test tests/impression_aggregate.rs check that the file regenerates byte for byte, that wai-rs gives every case its outcome, that every reason of §10 appears, and that §10 lists exactly the reference’s codes. The test also checks, without storing it, that each standing receipt’s result is the plaintext tally of the cells of the contributions its reports came from, and that stitched-presentation’s own verification of the flight’s claims, step 11 left out, gives per creative the presentations and quartile counts the flight’s table gives summed over its columns. The size bounds of §5.4 and §6.4 are checked in code, since no vector carries them: a file of exactly the bound that reads in every other respect is read, a task file then reaching step 2, and the same file one byte longer is refused at step 1. §4.3’s bound decides no outcome (§4.3), so no check can observe it. live-conformance/impression_aggregate_verify.py is the second implementation, written from this text: it shares no code with wai-rs, evaluates Prio3 and opens every report with the QoE corpus’s second implementation, judges records and claims with stitched-presentation’s, verifies tokens with ad-presentation’s, confirms task-file signers with the signer corpus’s, writes the catalogue, the task file, the upload file, the contribution, the plan, the settlement and the table itself, requires every mutated copy of the corpus to fail, and drops each of its own checks in turn.

12. Security and privacy considerations