diff --git a/draft-lcurley-moq-compression.md b/draft-lcurley-moq-compression.md index 13f7302..d9ccc4d 100644 --- a/draft-lcurley-moq-compression.md +++ b/draft-lcurley-moq-compression.md @@ -19,15 +19,17 @@ author: normative: moqt: I-D.ietf-moq-transport RFC1951: + RFC8878: informative: + RFC7692: --- abstract This document defines a payload compression extension for MoQ Transport {{moqt}}. -A track-level Compression property lets the original publisher signal that a track's object payloads are worth compressing, and with which algorithm. -Compression is then applied independently on each hop: a payload is compressed only on a hop that has negotiated the extension and whose receiver supports the algorithm, and is sent verbatim otherwise. -Each object is compressed independently so objects remain individually decodable, and the decompressed bytes — the actual object — are unchanged end to end. +A track-level Compression property names the algorithm the original publisher used for a track's object payloads. +Endpoints advertise the algorithms they can decode during SETUP, and a payload is compressed on a hop only when the receiver supports the named algorithm; otherwise it is sent verbatim. +Compression is scoped to a subgroup: the object payloads of a subgroup form one compressed stream, sliced back into the individual object payloads so the object framing stays in the clear and the decompressed bytes — the actual objects — are unchanged end to end. --- middle @@ -43,10 +45,9 @@ But MoQ also carries non-media tracks — JSON, text, telemetry, captions, uncom For these tracks there is no standard, transport-visible way to compress payloads, so each application reinvents it, and relays cannot help. Like HTTP Transfer-Encoding, the on-wire compression is a hop-by-hop optimization: it does not conceptually change the object payload — the decompressed bytes *are* the object — it only changes how those bytes are carried over a single hop. -What this extension adds on top is an end-to-end *signal*: a track property by which the original publisher marks the content as worth compressing and names the algorithm. The signal travels end-to-end; the compression happens per hop. -- **Publisher signals, hops apply**: the COMPRESSION track property is set by the original publisher and carried end-to-end, but a payload is only compressed on a hop that negotiated the extension and whose receiver supports the algorithm. Where the extension is not negotiated, the same payload travels verbatim. -- **Per object, independently**: each object payload is an independent compressed stream with no shared dictionary or state between objects. This keeps every object individually decodable and avoids head-of-line decoding within a group. +- **Publisher names, hops apply**: the COMPRESSION track property names the algorithm the original publisher used; it is carried end to end and forwarded unchanged. A payload is compressed on a hop only when the receiver advertised that algorithm; otherwise it travels verbatim. Each hop's behavior is fixed by the publisher's algorithm and that hop's negotiation, with no per-object signal. +- **Per subgroup, sliced into objects**: within a subgroup the object payloads form one compressed stream, flushed at each object boundary so every object still carries its own payload slice, while the object headers and framing stay in the clear. This keeps the subgroup — already one ordered, reliable stream — as the unit of compression, and lets relays and caches store payloads compressed and re-frame them without recompressing. # Setup Negotiation @@ -64,15 +65,17 @@ COMPRESSION Setup Option { ~~~ **Algorithm**: -One or more Algorithm identifiers (see [Compression Algorithms](#compression-algorithms)) that the sender can decompress, each a varint, filling the Option Value. +One or more Algorithm identifiers (see [Compression Algorithms](#compression-algorithms)) that the sender can decompress, each a varint, filling the Option Value — its decode capability. An endpoint does not advertise which algorithms it can *produce*; when sending, it compresses with one the receiver advertised. The identifier `none` (0) MUST NOT be listed (it requires no negotiation). +An endpoint that does not support the extension omits the option. -A sender MUST NOT compress with an algorithm the receiver did not advertise in its SETUP. -This makes the on-wire state unambiguous on every hop without any per-object signaling: a receiver decompresses a track's payloads **if and only if** the COMPRESSION track property is present and the receiver advertised that algorithm in its own SETUP. In every other case — the property absent, the extension not negotiated, or the algorithm not advertised by the receiver — the sender was not permitted to compress, so the receiver treats the payloads as verbatim. +A sender MUST NOT compress with an algorithm the receiver did not advertise, and MUST NOT compress before it has received the receiver's COMPRESSION option. +This makes the on-wire state unambiguous with no per-object signaling: a receiver decompresses a track's object payloads **if and only if** the COMPRESSION property names a non-`none` algorithm and the receiver advertised that algorithm in its own SETUP. +In every other case — the property absent or `none`, the extension not negotiated, or the algorithm not advertised by the receiver — the sender was not permitted to compress, so the receiver treats the payloads as verbatim. # COMPRESSION Track Property -The COMPRESSION property is the original publisher's signal that a track's object payloads are worth compressing, and which algorithm to use. +The COMPRESSION property names the algorithm the original publisher applied to a track's object payloads. It is a track-level Key-Value-Pair carried with the track's properties (see {{moqt}} Section 2.5 and Section 12), set by the original publisher and forwarded unchanged by relays. Because the value is a single integer, COMPRESSION uses an even Type so the value is a bare varint: @@ -84,59 +87,73 @@ COMPRESSION Track Property { ~~~ **Value**: -The Algorithm identifier the publisher recommends for this track's payloads. -The absence of the property, or a value of `none` (0), means the track is not marked for compression and its payloads are always transmitted verbatim. +The Algorithm identifier the publisher used for this track's payloads (see [Compression Algorithms](#compression-algorithms)). +The absence of the property, or a value of `none` (0), means the track is uncompressed and its payloads are always transmitted verbatim. +The publisher MUST choose an algorithm that its peer advertised in the [COMPRESSION Setup Option](#setup-negotiation), or `none` if the peer advertised none the publisher can produce. The property is fixed for the lifetime of the track and MUST NOT change. -A relay MUST forward it unchanged on every hop, including a hop that has not negotiated the extension: there it is simply an ignored unknown Key-Value-Pair, but forwarding it lets a further-downstream hop that does negotiate the extension still act on the publisher's signal. +A relay MUST forward it unchanged on every hop, including a hop that has not negotiated the extension: there it is simply an ignored unknown Key-Value-Pair, but forwarding it lets a further-downstream hop that does negotiate the extension still act on the publisher's algorithm. -Compression is enabled only by the combination of this track property and the extension being negotiated on a hop. -A publisher MUST NOT compress object payloads on a track that does not carry the COMPRESSION property, and there is no way to enable compression on a per-object basis: the property governs the whole track, and on a compressing hop every non-empty payload is compressed. +Whether a payload is actually compressed is decided per hop: -Whether payloads are actually compressed is decided per hop: +- On a hop where the receiver advertised the property's algorithm, each non-empty object payload is compressed with that algorithm, and the receiver decompresses it. +- On any other hop — the extension not negotiated, or the receiver did not advertise that algorithm — payloads are sent verbatim, and the receiver treats them as such. -- On a hop where the extension is negotiated and the receiver advertised the property's algorithm, every non-empty object payload MUST be compressed with that algorithm, and the receiver decompresses it. -- On any other hop — the extension not negotiated, or the receiver did not advertise that algorithm — payloads are sent verbatim. The receiver either never sees the property (an ignored unknown Key-Value-Pair) or sees it but knows the sender was not permitted to compress for it, so it treats the payloads as verbatim either way. - -Compression applies to the object payload only; object properties and message framing are never compressed. +Compression applies to the object payload only; object headers, properties, and message framing are never compressed. An empty payload (size 0) MUST NOT be compressed and remains empty on the wire. A publisher SHOULD set COMPRESSION only for payload types that benefit from it. Already-compressed media SHOULD omit it (or use `none`). -# Compression Algorithms {#compression-algorithms} +# Compression {#compression} +Compression is **scoped to a subgroup**. +Within a subgroup the object payloads form a single compressed stream in the algorithm named by the [COMPRESSION property](#compression-track-property), reset at each subgroup boundary. +The stream's output is partitioned at object boundaries: the compressor flushes at the end of each object so that object's slice is exactly the bytes carried as its payload, and the payload length in the object header gives the on-wire (compressed) slice size. +Both algorithms provide a window-retaining flush (DEFLATE's sync flush; Zstandard's `ZSTD_e_flush`), so later objects in a subgroup reuse the compression context and retain cross-object redundancy. + +A receiver maintains a single decoder per subgroup, reset at each subgroup boundary, and feeds each object's payload through it in order: the first object of a subgroup starts the decoder fresh — so a receiver joining at a group boundary needs nothing earlier — while later objects build on it. +There is no shared state between subgroups; an empty payload contributes nothing to the stream. +An object delivered as a datagram is a single-object stream, compressed on its own. + +Because the object framing already delimits each slice, an algorithm's own redundant boundary and container bytes are omitted: for `deflate`, the trailing four `00 00 FF FF` bytes a sync flush emits are removed from each payload and the decoder re-inserts them (as in {{RFC7692}}); for `zstd`, the per-subgroup stream uses the magicless frame format and omits the content checksum. + +Leaving the framing uncompressed is deliberate. +A relay or cache can hold the object payloads compressed in memory and forward them without inflating, and can re-frame a subgroup — for example to bridge a transport version that changes the subgroup or object headers — without touching the compressed payloads. + +## Compression Algorithms {#compression-algorithms} This document defines the following algorithms. -Further algorithms MAY be registered (see [IANA Considerations](#iana-considerations)). -| ID | Name | Description | +| ID | Name | Description | |---:|:--------|:--------------------------------------------------------| -| 0 | none | Payloads are transmitted verbatim. The default. | -| 1 | deflate | Raw DEFLATE {{RFC1951}}, with no zlib or gzip framing. | +| 0 | none | Verbatim; the absence of compression. Never advertised. | +| 1 | deflate | Raw DEFLATE {{RFC1951}}, with no zlib or gzip framing. | +| 2 | zstd | Zstandard {{RFC8878}}. | -For `deflate`, each object payload is an independent raw DEFLATE stream. -There is no shared dictionary or state between objects, so each object decompresses on its own. +Endpoints advertise whichever of these algorithms they support; none is mandatory, and a publisher uses one its peer advertised (or `none` if they share none). +Further algorithms MAY be registered (see [IANA Considerations](#iana-considerations)). # Relay Behavior -A relay forwards the COMPRESSION track property unchanged — it is the publisher's end-to-end signal — and applies compression independently on each hop. - -On its upstream subscription, the relay receives payloads compressed if and only if that hop compressed them (the extension negotiated and the relay advertised the algorithm); it decompresses them as needed. -On each downstream subscription the relay serves, it compresses payloads with the track's algorithm when that downstream negotiated the extension and advertised the algorithm, and sends them verbatim otherwise. +A relay forwards the COMPRESSION track property unchanged — it is the publisher's end-to-end signal — and applies compression independently on each hop, driven by each hop's negotiation rather than by its own initiative; a relay does not compress a track the publisher did not mark. -Compression is thus driven by the publisher's track property, not by the relay: a relay does not compress a track the publisher did not mark. -In every case the decompressed bytes delivered to the application MUST be identical to what the origin published. +On its upstream subscription the relay receives each subgroup compressed with the property's algorithm (if it advertised that algorithm) or verbatim, and decompresses as needed. +On a downstream subscription that advertised the property's algorithm, it sends each subgroup compressed with that algorithm (recompressing as needed); on one that did not, it sends the subgroup verbatim. +A relay MUST NOT recompress with an algorithm other than the one the property names, because the property tells the receiver how to decode and a relay MUST NOT rewrite it. +In every case the decompressed bytes delivered to the application MUST be identical to what the origin published, and a relay or generic library MUST NOT inspect or modify the decompressed contents unless otherwise negotiated. -A relay or generic library MUST NOT inspect or modify the decompressed contents unless otherwise negotiated; only recompression that preserves the decompressed bytes exactly is permitted. +Open issue: +because the COMPRESSION property both names the algorithm and is immutable, a downstream that supports a *different* algorithm than the publisher chose (for example only `deflate` when the publisher used `zstd`) receives the payloads verbatim rather than transcoded — a relay cannot offer it the algorithm it does support without rewriting the property, which {{moqt}} forbids. +Whether to relax this — by permitting a relay to rewrite this property for a downstream subscription, or by carrying the per-hop algorithm as transport metadata rather than an end-to-end track property — is an open question for the working group. # Security Considerations -Compressing data that mixes attacker-controlled and secret content in the same object can leak the secret through compressed size, as in the CRIME and BREACH attacks. +Compressing data that mixes attacker-controlled and secret content can leak the secret through compressed size, as in the CRIME and BREACH attacks. A publisher MUST NOT set COMPRESSION on a track whose object payloads combine secret material with attacker-influenced material. -Because compression here is per-object with no cross-object dictionary, the exposure is bounded to within a single object, but it is not eliminated. +Because compression is scoped to a subgroup, the exposure is bounded to within a single subgroup — which may combine several objects, a wider window than a single object — but it is not eliminated. A malicious sender could emit a small compressed payload that decompresses to a very large buffer (a "decompression bomb"). -A receiver MUST bound the size of a decompressed object payload. If the bound is exceeded it MUST reset the affected Subscribe/Fetch stream (rather than allocate unbounded memory) and MAY close the session with a PROTOCOL_VIOLATION if it considers the peer abusive; the reset is stream-scoped so a single bad object does not tear down unrelated subscriptions. +Because compression is subgroup-scoped, a receiver MUST bound the cumulative decompressed size of a subgroup — not merely each object's payload, since many small payloads can otherwise accumulate without limit. If the bound is exceeded it MUST reset the affected stream (rather than allocate unbounded memory) and MAY close the session with a PROTOCOL_VIOLATION if it considers the peer abusive; the reset is stream-scoped so a single bad subgroup does not tear down unrelated subscriptions. Compression is orthogonal to {{moqt}} end-to-end encryption: an encrypted payload is effectively incompressible, so a publisher using end-to-end encryption SHOULD omit COMPRESSION (or use `none`). @@ -145,7 +162,7 @@ Compression is orthogonal to {{moqt}} end-to-end encryption: an encrypted payloa This document requests the following registrations. High, distinctive values are requested to avoid the low ranges reserved by {{moqt}} and to minimize collisions with provisional registrations by other extensions; they also avoid the greasing pattern (`0x7f * N + 0x9D`). -The parameter Type is even so that its value is a bare varint with no length prefix (see {{moqt}} Section 2.5). +Each Type is even so that its value is a bare varint with no length prefix (see {{moqt}} Section 2.5). ## MOQT Setup Options @@ -172,6 +189,7 @@ The initial contents are: |---:|:--------|:--------------| | 0 | none | This Document | | 1 | deflate | This Document | +| 2 | zstd | This Document | --- back diff --git a/draft-lcurley-moq-lite.md b/draft-lcurley-moq-lite.md index a6b71bf..1fc468d 100644 --- a/draft-lcurley-moq-lite.md +++ b/draft-lcurley-moq-lite.md @@ -309,7 +309,7 @@ When the accepted track has already ended with no matching groups there is no st A rejection is a stream reset: if the publisher cannot serve the subscription — the track does not exist, or it otherwise refuses — it MUST reset the stream rather than leave it pending, and SHOULD do so promptly (within roughly a round trip) so the subscriber is not left waiting. A subscription the publisher accepts but has no groups for yet is not a rejection: for a live track the publisher MAY withhold SUBSCRIBE_OK until the first matching group resolves the start. A subscriber therefore distinguishes "pending" from "refused" by the stream reset, not by a timeout. The Subscribe Stream does not carry the track's publisher properties — those are immutable and fetched once via a [Track Stream](#track-stream) (see [TRACK_INFO](#track-info)). -The subscriber MUST have the track's TRACK_INFO before it can parse the FRAME messages that arrive on Group Streams, since the timescale and compression determine the frame wire format; it MAY open the Track and Subscribe streams concurrently and buffer frames until TRACK_INFO arrives. +The subscriber MUST have the track's TRACK_INFO before it can parse the FRAME messages that arrive on Group Streams, since the timescale determines the frame wire format and the `Publisher Compression` value determines whether and how frame payloads are compressed; it MAY open the Track and Subscribe streams concurrently and buffer frames until TRACK_INFO arrives. The publisher sends SUBSCRIBE_OK once the absolute start group is resolved, and SUBSCRIBE_END once no further groups will be produced (see [SUBSCRIBE_OK](#subscribe-ok) and [SUBSCRIBE_END](#subscribe-end)). The publisher closes the stream (FIN) only once every group from start to end has been accounted for, either via a GROUP stream (completed or reset) or a SUBSCRIBE_DROP message. @@ -321,7 +321,7 @@ Either endpoint MAY reset/cancel the stream at any time. A subscriber opens a Fetch Stream (0x3) to request a single Group from a Track. The subscriber sends a FETCH message containing the broadcast path, track name, priority, and group sequence. -The publisher responds with FRAME messages directly on the same bidirectional stream — there is no response header. +The publisher responds with FRAME messages directly on the same bidirectional stream — there is no response header. When the Track is compressed on this hop (see [Compression](#compression)), each frame's payload is a slice of the group's compressed stream, as on a Group Stream. The Subscribe ID and Group Sequence for the returned FRAME messages are implicit, taken from the original FETCH request. As with a subscription, the subscriber MUST already have the track's [TRACK_INFO](#track-info) to parse the returned frames; because the properties are immutable, a single Track Stream lookup is reused across every FETCH of that track (group-by-group fetches do not re-fetch it). The publisher FINs the stream after the last frame, or resets the stream on error. @@ -473,6 +473,7 @@ See the [Session](#session) section for how an endpoint avoids waiting on the pe A publisher creates Group Streams in response to a Subscribe Stream. A Group Stream MUST start with a GROUP message and MAY be followed by any number of FRAME messages. +When the Track is compressed on this hop (see [Compression](#compression)), each frame's payload is a slice of the group's single compressed stream, while the FRAME framing stays uncompressed. A Group MAY contain zero FRAME messages, potentially indicating a gap in the track. A frame MAY contain an empty payload, potentially indicating a gap in the group. @@ -498,12 +499,13 @@ DATAGRAM Body { Subscribe ID (i) Group Sequence (i) [Timestamp (i)] + [Decompressed Length (i)] Payload (b) } ~~~ -`Timestamp` is present only when the Track's `Publisher Timescale` (see [TRACK_INFO](#track-info)) is non-zero. -When `Publisher Timescale` is 0, the field is omitted from the wire and the datagram body consists of just `Subscribe ID`, `Group Sequence`, and `Payload`. +`Timestamp` is present only when the Track's `Publisher Timescale` (see [TRACK_INFO](#track-info)) is non-zero, and `Decompressed Length` only when the Track is compressed on this hop (see [Compression](#compression)). +Each is independently omitted when its condition does not hold; with both absent the datagram body is just `Subscribe ID`, `Group Sequence`, and `Payload`. **Subscribe ID**: The Subscribe ID of an active subscription on the same session. @@ -519,7 +521,7 @@ Any varint value (including 0) is a valid absolute timestamp. **Payload**: The frame payload, extending to the end of the datagram. -If the Track's `Publisher Compression` is non-zero, the payload is compressed using the negotiated algorithm (see [TRACK_INFO](#track-info)). +A datagram is a single-frame group: when the Track is compressed on this hop (see [Compression](#compression)) the payload is a single compressed stream in the Track's algorithm — with `Decompressed Length` giving its post-decompression size for the same allocation and bound role as in [FRAME](#frame) — otherwise it is verbatim. The total datagram body (including all header fields above and the compressed payload if applicable) MUST NOT exceed 1200 bytes. This limit ensures the datagram fits within the minimum QUIC path MTU without IP-layer fragmentation. Payloads that would not fit MUST be sent as a Group Stream instead. @@ -586,13 +588,15 @@ The parameter-specific value, interpreted according to Parameter ID. A capability is available for the session only if the relevant endpoint advertises it; an absent parameter means the sender does not support that capability. The following Setup Parameters are defined: -|------|----------|-------------| -| ID | Name | Value | -|-----:|:---------|:------------| -| 0x1 | Probe | Level (i) | -|------|----------|-------------| -| 0x2 | Path | Path (s) | -|------|----------|-------------| +|------|-------------|-------------------| +| ID | Name | Value | +|-----:|:------------|:------------------| +| 0x1 | Probe | Level (i) | +|------|-------------|-------------------| +| 0x2 | Path | Path (s) | +|------|-------------|-------------------| +| 0x3 | Compression | Algorithm (i) ... | +|------|-------------|-------------------| ### Probe Parameter {#probe-parameter} The Probe Parameter advertises the sender's capability level when acting as a publisher on a [Probe Stream](#probe). @@ -627,6 +631,15 @@ The remaining bindings convey the path in their own handshake. A relay MUST NOT forward the Path Parameter; like other per-hop setup metadata it applies only to this hop (see [Session](#session)). +### Compression Parameter {#compression-parameter} +The Compression Parameter advertises the payload compression [algorithms](#compression) the sender can *decompress* on this hop — its decode capability. An endpoint does not advertise which algorithms it can *produce*; when sending, it compresses with one the receiver advertised. +The Parameter Value is a sequence of algorithm identifiers, each a variable-length integer, packed back-to-back to fill the Parameter Length. +The identifier `none` (0) MUST NOT be listed. An endpoint that does not support compression omits the parameter. + +A relay MUST NOT forward the parameter (see [Session](#session)); it is negotiated independently on each hop. +The list constrains what a publisher may use: a publisher MUST set `Publisher Compression` to an algorithm its peer advertised here (see [TRACK_INFO](#track-info)), and more generally a sender MUST NOT compress with an algorithm the receiver did not advertise, nor compress at all before it has received the receiver's Compression Parameter. +A receiver decompresses a Track's payloads if and only if `Publisher Compression` names a non-`none` algorithm and the receiver advertised that algorithm; a receiver that advertised no list, or not the Track's algorithm, gets every Group Stream, Fetch Stream, and datagram verbatim. + ## ANNOUNCE_REQUEST A subscriber sends an ANNOUNCE_REQUEST message to indicate it wants to receive an ANNOUNCE_BROADCAST message for any broadcasts with a path that starts with the requested prefix. @@ -852,22 +865,44 @@ When `Publisher Timescale` is 0, the per-frame `Timestamp Delta` field is omitte Common values include `1000` (milliseconds), `1000000` (microseconds), `48000` (audio sample rate), and `90000` (RTP video clock). **Publisher Compression**: -The compression algorithm applied to every Frame `Payload` on this Track. +The compression algorithm the original publisher applied to this Track's payloads (see [Compression](#compression)). + +- `0` (`none`): payloads are uncompressed (the default). +- `1` (`deflate`) or `2` (`zstd`): payloads are compressed with that algorithm. + +The publisher MUST choose an algorithm its peer advertised in the [Compression Parameter](#compression-parameter), or `0` if the peer advertised none the publisher can produce. +The value is fixed for the lifetime of the Track and forwarded unchanged by relays, so even a hop that does not compress passes it along for a further-downstream hop to act on. +It does not by itself cause compression: a receiver decompresses if and only if the value names a non-`none` algorithm and the receiver advertised that algorithm in its own [Compression Parameter](#compression-parameter); otherwise the payloads are verbatim (see [Compression](#compression)). A subscriber that does not recognize the value treats the payloads as verbatim, so an unknown future algorithm degrades to uncompressed rather than blocking the Track. + +The publisher SHOULD set it only for payload types that benefit from compression (e.g. JSON, text, uncompressed binary structures); already-compressed media (e.g. H.264, Opus, AV1) gains nothing and SHOULD leave it `0`. See [Security Considerations](#security-considerations) for content that MUST NOT be marked compressible. + +## Compression {#compression} +moq-lite can compress payloads hop by hop, like HTTP Transfer-Encoding: the decompressed bytes *are* the payload and are identical end to end, but how they are carried MAY differ on each hop — a group compressed on one hop is decompressed and MAY be re-sent verbatim on the next. Compression is governed by two pieces, with no per-group flag on the wire: + +- the Track's `Publisher Compression` value (see [TRACK_INFO](#track-info)) — the algorithm the original publisher used, carried end to end; and +- the per-hop [Compression Parameter](#compression-parameter) — each endpoint advertises the algorithms it can decompress. + +A hop compresses a Track's payloads when, and only when, `Publisher Compression` names a non-`none` algorithm **and** the receiver advertised that algorithm; otherwise they are verbatim. A receiver needs no per-group signal — it takes the algorithm from `Publisher Compression`, and whether the payloads are compressed from its own advertisement. Because the value is carried end to end and forwarded unchanged, a hop that has not negotiated compression still passes it along, so a further-downstream hop that has can act on it. + +The following algorithms are defined: -- `none` (0): payloads are transmitted verbatim (default). -- `deflate` (1): payloads are compressed with raw DEFLATE as defined in {{!RFC1951}}, with no zlib or gzip framing. +| ID | Name | Description | +|---:|:--------|:--------------------------------------------------------| +| 0 | none | Verbatim; the absence of compression. Never advertised. | +| 1 | deflate | Raw DEFLATE {{!RFC1951}}, with no zlib or gzip framing. | +| 2 | zstd | Zstandard {{!RFC8878}}. | -Compression is applied per-frame: each Frame `Payload` is an independent compressed stream with no shared dictionary or state between frames. -This keeps frames independently decodable and avoids head-of-line decoding within a group. -The Frame `Message Length` describes the compressed (on-wire) size. -An empty payload (size 0) MUST NOT be compressed and remains empty on the wire. +Endpoints advertise whichever of these algorithms they support; none is mandatory, and a publisher uses one its peer advertised, or `none` if they share none. Further algorithms MAY be defined by future extensions. -The publisher SHOULD only enable compression for payload types that benefit from it (e.g. JSON, text, uncompressed binary structures). -Already-compressed media (e.g. H.264, Opus, AV1) gains nothing and SHOULD use `none`. -A subscriber that does not recognize the value MUST NOT open a Subscribe or Fetch stream for the track; if it already opened one before receiving TRACK_INFO, it MUST reset that stream with a protocol violation. The Track Stream itself needs no reset — the publisher FINs it after TRACK_INFO. +Compression is **group-scoped** and applied only to frame payloads, never to the FRAME framing. Within a group the payloads form a single compressed stream in the Track's algorithm, reset at each group boundary, whose output is partitioned at frame boundaries: the compressor flushes at the end of each frame so that frame's slice is exactly the bytes stored in its `Payload` (delimited by `Payload Length`, with the decompressed size in `Decompressed Length`). Both algorithms provide such a flush that retains the window (DEFLATE's sync flush; Zstandard's `ZSTD_e_flush`), so later frames in a group reuse the compression context. A receiver maintains a single decoder per group, reset at each group boundary, and feeds each frame's `Payload` through it in order: the first frame starts the decoder fresh — so a subscriber joining at a group boundary needs nothing earlier — while later frames retain cross-frame redundancy across the group. There is no shared state between groups; a frame with an empty payload contributes nothing to the stream. -A relay MAY transcode payloads between compression algorithms (including bridging different protocol versions, e.g. a moq-lite-05 publisher to a moq-lite-04 subscriber) provided the decompressed bytes are identical to what the publisher produced. -A relay SHOULD NOT compress an originally-uncompressed payload unless there is a strong content signal that compression is beneficial (e.g. the track name ends in `.json`), because the relay cannot otherwise predict whether compression will help or hurt. +Because moq-lite delimits the slices itself, each algorithm's own redundant boundary and container bytes are omitted: for `deflate`, the four `00 00 FF FF` bytes a sync flush emits are removed from each `Payload` and the decoder re-inserts them (as in {{?RFC7692}}); for `zstd`, the per-group stream uses the magicless frame format and omits the content checksum. + +Compressing only the payloads, and leaving the framing in the clear, is deliberate. A relay or cache can hold the payloads compressed in memory and forward them without inflating, and can re-frame a group — for example to bridge a future transport version that changes the GROUP or FRAME headers — without touching the compressed payloads. Neither is possible if the framing is buried inside the compressed stream. + +A relay applies compression independently on each hop, driven by each downstream's negotiation — not by the relay's own initiative, and never on a Track the publisher did not mark. On its upstream subscription it decompresses each compressed group as needed; on a downstream subscription that advertised the Track's algorithm it (re)compresses with that algorithm, and on one that did not it sends the group verbatim. A relay MUST NOT recompress with an algorithm other than the one `Publisher Compression` names — that value tells the receiver how to decode and is immutable — though it MAY recompress with the same algorithm, and MAY bridge protocol versions (e.g. a moq-lite-05 publisher to a moq-lite-04 subscriber) provided the decompressed bytes are identical to what the publisher produced. A relay or generic library MUST NOT inspect or modify the decompressed contents unless otherwise negotiated. + +Open issue: because `Publisher Compression` both names the algorithm and is immutable, a downstream that supports only a *different* algorithm than the publisher chose (e.g. only `deflate` when the publisher used `zstd`) receives the payloads verbatim rather than transcoded — a relay cannot offer it an algorithm it does support without rewriting the immutable value. Whether to allow a per-hop algorithm distinct from the publisher's is left as an open question. ## SUBSCRIBE_OK {#subscribe-ok} A SUBSCRIBE_OK message confirms a subscription and resolves its absolute start group. @@ -964,7 +999,7 @@ See the [Prioritization](#prioritization) section for more information. **Group Sequence**: The sequence number of the group to fetch. -The publisher responds with FRAME messages directly on the same stream — there is no response header. +The publisher responds with FRAME messages directly on the same stream — there is no response header; when the Track is compressed on this hop each frame's payload is a slice of the group's compressed stream (see [Compression](#compression)). The subscriber parses them using the track's [TRACK_INFO](#track-info), which it MUST already have (see the [Track Stream](#track-stream)); the group sequence is implicit from the FETCH request. The publisher FINs the stream after the last frame, or resets on error. There is no FETCH_ERROR message — the publisher signals failure by resetting the stream. @@ -1035,13 +1070,15 @@ The FRAME message is a payload within a group. ~~~ FRAME Message { [Timestamp Delta (i)] - Message Length (i) + [Decompressed Length (i)] + Payload Length (i) Payload (b) } ~~~ -`Timestamp Delta` is present only when the Track's `Publisher Timescale` (see [TRACK_INFO](#track-info)) is non-zero. -When `Publisher Timescale` is 0, the field is omitted from the wire and the FRAME consists of just `Message Length` and `Payload`. +`Timestamp Delta` is present only when the Track's `Publisher Timescale` (see [TRACK_INFO](#track-info)) is non-zero, and `Decompressed Length` only when the group is compressed (see [Compression](#compression)). +Each is independently omitted when its condition does not hold; with both absent the FRAME is just `Payload Length` and `Payload`. +The framing is never compressed; only the `Payload` is, so `Payload Length` is always the on-wire `Payload` size. **Timestamp Delta**: A signed delta from the previous frame's timestamp, in the Track's negotiated `Timescale`. @@ -1053,10 +1090,14 @@ Encoded as a zigzag-mapped variable-length integer: Zigzag interleaves non-negative and negative values (`0 → 0, -1 → 1, 1 → 2, -2 → 3, 2 → 4, ...`) so small magnitudes of either sign fit in a 1-byte varint and there is exactly one wire encoding for zero. The first frame of a group is delta-encoded from `0`, so its `Timestamp Delta` is the zigzag encoding of the absolute timestamp. +**Decompressed Length**: +The size in bytes of this frame's `Payload` after decompression, present only when the group is compressed. +A receiver uses it to size the output buffer and as a per-frame bound: it MUST reject a value larger than its configured limit, and MUST reset the stream if decompressing the `Payload` yields a different number of bytes than this declares (see [Security Considerations](#security-considerations)). + **Payload**: An application-specific payload. -If the Track's `Publisher Compression` is non-zero, the payload is compressed using the negotiated algorithm (see [TRACK_INFO](#track-info)) and the `Message Length` describes the compressed size. -A generic library or relay MUST NOT inspect or modify the decompressed contents unless otherwise negotiated; recompression that preserves the decompressed bytes exactly is allowed (see [TRACK_INFO](#track-info)). +When the group is compressed (see [Compression](#compression)) this holds this frame's slice of the group's compressed stream and `Payload Length` is its compressed (on-wire) size; otherwise it is verbatim. The framing around it is never compressed. +A generic library or relay MUST NOT inspect or modify the decompressed contents unless otherwise negotiated; recompression that preserves the decompressed bytes exactly is allowed (see [Compression](#compression)). # Appendix A: Changelog @@ -1079,7 +1120,7 @@ A generic library or relay MUST NOT inspect or modify the decompressed contents - Removed `Publisher Max Latency`. The publisher's retention guarantee is no longer part of the wire format; retention for FETCH and future subscriptions is best-effort and left to the publisher. - Timestamp-based expiration replaces wall-clock arrival time when a Track timescale is negotiated. - Added QUIC datagram delivery for groups, sharing Subscribe IDs with existing subscriptions (no separate control stream). -- Added `Publisher Compression` to TRACK_INFO for per-frame payload compression (`none` or `deflate`). +- Added payload compression, scoped per group. `Publisher Compression` in TRACK_INFO now names the algorithm the publisher used (`none`/`deflate`/`zstd`), and a new SETUP `Compression` parameter carries the algorithms each endpoint can decompress on a hop. The publisher MUST pick an algorithm its peer advertised (or `none` if they share none); `deflate` and `zstd` are defined, neither mandatory. There is no per-group or per-frame flag on the wire — a receiver decompresses iff `Publisher Compression` names a non-`none` algorithm it advertised, else treats payloads as verbatim. When compressed, a group's frame payloads (only the payloads, never the framing) form a single stream in that algorithm, reset at each group boundary and sliced per frame into each frame's opaque `Payload`, with the algorithm's redundant container bytes omitted (the RFC 7692 trim for deflate; magicless, checksum-less frames for zstd) since moq-lite frames the slices itself; the decoder keeps one context per group. Keeping the framing uncompressed lets a relay store payloads compressed and re-frame across transport versions without recompressing. A compressed frame also carries a `Decompressed Length` (post-decompression size, for buffer allocation and as a per-frame decompression bound), and FRAME's `Message Length` is renamed `Payload Length` (it has only ever delimited the payload). - Added Qmux [qmux] transport bindings for TCP/TLS and WebSocket, for environments where UDP is unavailable. The WebSocket binding uses the WebSocket message framing in place of the Qmux Record `Size` field. ## moq-lite-04 @@ -1176,7 +1217,17 @@ Some of these fields occur in multiple messages. # Security Considerations -TODO Security +TODO: general security considerations. + +## Payload Compression +Compressing data that mixes attacker-controlled and secret material in the same payload can leak the secret through the compressed size, as in the CRIME and BREACH attacks. +A publisher MUST NOT compress a Track (set a non-`none` `Publisher Compression`) whose payloads combine secret material with attacker-influenced material. +Because compression is group-scoped, the exposure is bounded to within a single group — which may combine several frames, a wider window than a single frame — but it is not eliminated. + +A malicious sender could emit a small compressed payload that decompresses to a very large buffer (a "decompression bomb"). +Each compressed frame declares its post-decompression size in `Decompressed Length` (see [FRAME](#frame)): a receiver MUST reject a frame whose `Decompressed Length` exceeds its configured limit, and MUST reset the stream if decompressing the `Payload` yields more bytes than declared, so it never allocates or emits more than that bounded size. A receiver SHOULD also bound the cumulative decompressed size across a group, since many in-bound frames can still accumulate. On any breach it MUST reset the affected stream rather than allocate unbounded memory, and MAY close the session with a PROTOCOL_VIOLATION if it considers the peer abusive. + +Compression is orthogonal to end-to-end encryption: an encrypted payload is effectively incompressible, so a publisher using end-to-end encryption SHOULD leave `Publisher Compression` at `0`. # IANA Considerations