From 816db1f25f8fc2d996084a2631f455ad4ea009a3 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 09:58:47 -0500 Subject: [PATCH 01/35] docs: plan WebSub hub support roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lay out the full WebSub hub implementation plan in TODO.md as ordered TDD vertical slices spanning @rsscloud/core, @rsscloud/express, and apps/server. Settled decisions: async-202 intent verification behind an in-process best-effort VerificationScheduler seam (persisted queue + retry deferred); both thin-publish (re-fetch) and fat-ping content sourcing; honor the requested lease clamped to a configurable range; HMAC-SHA256 signatures. Headline use case: an rssCloud publisher adds and keeps pinging via rssCloud, while WebSub subscribers to the same topic receive full content distribution — which falls out of core's existing resource-keyed fan-out. Each flow gets an e2e acceptance test as its TDD outer loop, and server integration (plugin registration, route mount, config) is spelled out per file. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 283 +++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 250 insertions(+), 33 deletions(-) diff --git a/TODO.md b/TODO.md index d178f71..7e6e40c 100644 --- a/TODO.md +++ b/TODO.md @@ -14,36 +14,253 @@ domain vocabulary in `CONTEXT.md`. ## WebSub hub support (bigger — spans core + express) Make the server act as a [WebSub](https://www.w3.org/TR/websub/) **hub** (the W3C -successor to PubSubHubbub, rssCloud's cousin). Needs new protocol logic in -`@rsscloud/core` **and** a new `@rsscloud/express` middleware, plus a delivery model -the notification plugins don't cover. Sketch, not a spec. - -*What it adds over rssCloud's notify-only model:* -- **Subscribe request:** form-encoded POST — `hub.callback`, `hub.mode` - (subscribe|unsubscribe), `hub.topic`, optional `hub.lease_seconds` + `hub.secret`. - Hub replies `202` (async verify) or 4xx. -- **Intent verification:** hub GETs the callback with `hub.challenge`; the subscriber - echoes it. Same shape as the rssCloud REST challenge core already does — reuse it. -- **Content distribution (the big new piece):** on update the hub POSTs the *actual - feed content* to each callback — topic `Content-Type`, `Link` rel=hub/self, and - `X-Hub-Signature: sha256=HMAC(secret, body)`. The REST/XML-RPC plugins send a - notification, not content, so this needs a new delivery plugin. -- **Leases:** `hub.lease_seconds` + renewal (distinct from `ctSecsResourceExpire`). - -*Pieces:* a core subscribe/unsubscribe dispatcher + content-delivery plugin (new -`Subscription` fields `secret` / `leaseSeconds` / `callback`+`topic` / mode; likely a -`websub` protocol value); a `websub({ core })` express factory branching on -`hub.mode`; mount the hub at a stable URL (publishers reference it via -`` in their own feeds — the hub doesn't host the source). The -REST/XML-RPC subscribe parsing now shares `buildSubscribeRequest(SubscribeParams)` in -core (one callback-assembly seam); a WebSub `hub.*` parser can build a `SubscribeRequest` -through it rather than re-deriving callback/scheme/`diffDomain` logic. - -*Open questions:* sync vs async intent verification (spec prefers async `202`); which -HMAC algos to require; content source on publish (fetch vs publisher-pushed). The new -subscription fields now persist directly — the domain-model v2 disk format is in place, -so new `Subscription` fields ride along with no extra mapping. - -*First slice:* core `subscribe` happy path (parse, verify intent, persist) + the -express `websub` factory + an e2e callback handshake. Defer content distribution, -HMAC, and leases. +successor to PubSubHubbub, rssCloud's cousin). Hub only — `apps/client` already owns the +subscriber/publisher side; the hub never hosts source feeds (publishers point at it via +`` in their own feeds). Needs new protocol logic in `@rsscloud/core` +**and** a new `@rsscloud/express` middleware, plus a content-delivery model the +notify-only REST/XML-RPC plugins don't cover. + +The engine is already primed: `protocol.ts` lists `'websub'`; `Subscription` / +`SubscribeRequest` / `UnsubscribeRequest` carry a `details` bag for protocol extras; +`whenExpires` + `removeExpired()` are where a lease maps; `ProtocolPlugin` has +`verify`/`deliver`; and `DeliveryContext` already carries `payload: ResourcePayload` +(the feed body + content-type captured by `detectChange` on every ping). So the +fan-out machinery is waiting for a plugin — most new code is the WebSub plugin, a +`hub.*` parser/dispatcher, an express factory, the async-accept seam, and wiring. + +### Primary use case — free WebSub for rssCloud publishers + +A publisher already on this server for rssCloud adds `` +to their feed and **keeps pinging exactly as today** (`/ping` / `rssCloud.ping`). Anyone +who subscribes to that feed *via WebSub* then gets full WebSub content distribution — +**the publisher never speaks WebSub and changes nothing but the feed header.** + +This falls out of core's existing design, which is why WebSub belongs *in core*, not the +HTTP edge: `ping()` → `detectChange()` already fetches the feed body and builds `payload` +on **every** ping; `fanOut(resourceUrl, …)` loads **all** subscriptions for the resource +and selects the plugin **per subscription** (`deliverTo`). So one rssCloud ping already +iterates every subscriber of that topic and dispatches each through its own plugin — +an rssCloud sub gets a notify, a `protocol:'websub'` sub gets content distribution, from +the *same* ping and the *same* already-fetched body. The only missing piece is the +WebSub `deliver()` plugin. Consequences: + +- **No new publish path is required for the headline case** — the trigger is the + existing rssCloud ping. The WebSub `hub.mode=publish` front door and fat pings serve + *pure-WebSub* publishers (no rssCloud) and are therefore **secondary** (later phases). +- **Topic identity is the one hard requirement:** a subscriber's `hub.topic` must be the + same URL string the publisher pings (the store keys feed entries by exact resource + URL). Same exactness rssCloud already requires between subscribe-URL and ping-URL — + WebSub just inherits it. URL normalization is out of scope (matches today's behavior). +- **WebSub adds no fetch overhead on ping** — it reuses the body `detectChange` already + fetched; it only adds an extra outbound POST per WebSub subscriber. + +### Decisions (settled — 2026-06-14) + +1. **Intent verification = async `202`.** The hub validates the request synchronously + (→ `4xx` on malformed), returns `202 Accepted`, then performs the `hub.challenge` + GET out of band and records the subscription only on success. +2. **Best-effort now, queue later — behind one seam.** Async ≠ a queue. A single + **verification-dispatch seam** runs the verify+persist task in-process (one attempt; + failures logged; a restart mid-flight drops the pending request — the subscriber + re-subscribes). A persisted-queue + retry implementation later satisfies the *same* + seam (draining on the existing maintenance interval, persisting via the store) with + **no change** to the `hub.*` parser, the plugin's `verify()`, or the express factory. + Captured as an ADR. **The scheduler is additive and WebSub-only:** rssCloud + `pleaseNotify`/`subscribe` stays synchronous (its callers expect an immediate yes/no) + and `ping`/`fanOut`/`deliver` are untouched — it's a brand-new caller of an unchanged + `core.subscribe`, so no existing rssCloud behavior changes. It lives in core (not + express) only so the future persisted queue can reach the store; the in-process + default would work anywhere. +3. **Publish = both.** Accept a thin WebSub publish (`hub.mode=publish`, `hub.url`/ + `hub.topic`) — and keep rssCloud `/ping` — re-fetching the topic and reusing + `core.ping`'s existing fetch→`payload`→`fanOut`. *Also* accept fat pings (publisher + POSTs the body), distributed verbatim without a re-fetch; this adds an optional + pushed-content path to `PingRequest`/`detectChange`. (Fat-ping wire format is + non-standard — see open questions — so it lands last.) +4. **Lease = honor requested, clamped.** Use `hub.lease_seconds` clamped to a + configurable `[min, max]` (default when omitted); store the chosen value in + `details.leaseSeconds`, set `whenExpires = now + chosen`, and echo the chosen value + in the verification GET. `removeExpired()` drops it on lapse, unchanged. +5. **Signature = HMAC-SHA256, configurable.** When a subscriber supplied `hub.secret`, + sign each delivery with `X-Hub-Signature: sha256=…` (algorithm a config knob, default + `sha256`). No `hub.secret` → no signature header. + +### Architecture notes / corrections to the original sketch + +- **WebSub builds `SubscribeRequest` directly — it does *not* reuse + `buildSubscribeRequest`.** That builder exists to assemble a callback from + port/path/domain (`glueUrlParts`, scheme, `diffDomain`) for REST/XML-RPC. WebSub + already arrives with a complete `hub.callback` URL, so the dispatcher sets + `callbackUrl = hub.callback`, `resourceUrls = [hub.topic]`, `protocol = 'websub'`, + `details = { secret?, leaseSeconds }` and skips the builder. (The sketch's hope to + share that seam doesn't pan out.) `buildSubscribeRequest` also gates on + `VALID_PROTOCOLS` (rssCloud only) — leave it as-is. +- **WebSub always verifies intent** (spec mandate), so the plugin's `verify()` ignores + `diffDomain` and always does the challenge GET — never the same-domain test-notify. +- **`core.unsubscribe()` has no verify hook today.** WebSub unsubscribe must *also* be + intent-verified (`hub.mode=unsubscribe` challenge GET) before removal — the scheduled + task verifies, then calls `core.unsubscribe`. +- **`VerifyContext` likely needs the WebSub `mode` and the chosen lease** (to send + `hub.mode` / `hub.lease_seconds` / `hub.topic` on the challenge GET). Thread these + through `VerifyContext` or read them from `subscription.details` — decide in the + verify slice. +- **Public hub URL is a host concern** (per `config.ts`: host concerns excluded from + `RssCloudConfig`). Only the plugin's `deliver()` needs it (for `Link rel="hub"`) — so + inject `hubUrl` (plus signature algo, timeout, challenge generator) as **plugin** + construction options in `apps/server/core.js`. The express factory **and** the + dispatcher take only `{ core }`, exactly like `ping`/`pleaseNotify`/`rpc2`; the + scheduler is a `createRssCloudCore` option (default in-process, injectable for tests), + not an arg of either. Lease bounds *are* protocol-relevant → add them to + `RssCloudConfig` alongside `ctSecsResourceExpire`. + +### Files this will touch + +- **core (new):** `protocols/websub-plugin.ts` (verify + deliver), `protocols/websub-dispatcher.ts` (`hub.*` parse/validate, branch on `hub.mode`, drive the accept seam). +- **core (changed):** the verification-dispatch seam + async-accept entry on the engine; `PingRequest`/`detectChange` optional pushed content (fat ping); verified-unsubscribe path; `RssCloudConfig` lease bounds; `VerifyContext` WebSub fields. +- **express (new):** `websub-middleware.ts` — `websub({ core })` factory (same `{ core }` shape as `ping`/`pleaseNotify`/`rpc2`) delegating to core's `websub-dispatcher`; export from `index.ts`. +- **apps/server (the integration that makes e2e runnable):** `core.js` — add `createWebSubProtocolPlugin({ hubUrl, requestTimeoutMs, signatureAlgo, createChallenge })` to the `plugins` array (registers the `'websub'` protocol; without it `core.subscribe` → `UNSUPPORTED_PROTOCOL`) and feed lease bounds into `resolveConfig`; `controllers/index.js` — `router.post('/websub', websub({ core }))`; `config.js` — new env vars (hub URL, mount path, lease bounds, signature algo). Scheduler defaults inside `createRssCloudCore`, so no extra server wiring. +- **apps/e2e:** mock subscriber callback that echoes `hub.challenge`; handshake/publish/signature suites (copy any new helper into `helpers/`, don't cross the workspace boundary). +- **docs:** ADR for the async/best-effort+seam decision; `CONTEXT.md` vocabulary (Hub, Topic, Callback, Intent verification, Lease, Content distribution, Fat ping, `X-Hub-Signature`). + +### e2e strategy (the TDD outer loop) + +Every new endpoint/flow gets an `apps/e2e` acceptance test **written as the outer red of +its slice** — the HTTP-level test fails first, the core/express units make it green; the +slice isn't done until its e2e passes. e2e drives the running server over `APP_URL`; per +CLAUDE.md, anything new a test needs goes in `apps/e2e/test/helpers/` (copied, **not** +imported across the workspace boundary). + +A reusable **mock WebSub subscriber** (alongside the existing rssCloud mock servers on +8002/8003) is grown incrementally as phases need it: +- **challenge-echo** (Phase 1): answers the intent-verification GET by echoing + `hub.challenge` with `2xx`; a toggle to *refuse* (wrong/absent echo) drives the negatives. +- **content-capture** (Phase 2): records each distribution POST — body, `Content-Type`, + `Link` rels — for assertions. +- **signature-verify** (Phase 3): recomputes `HMAC-SHA256(secret, body)` and checks + `X-Hub-Signature`. + +Flows that must have an e2e (happy path + the ★ negatives): +- **subscribe** → `202`, callback verified, sub recorded; ★ no-echo → **not** recorded; + ★ malformed `hub.*` → `4xx`. +- **cross-protocol fan-out** — one rssCloud `/ping` fires BOTH an rssCloud sub and a + WebSub sub on the same topic (the headline proof; see S2.2). +- **authenticated delivery** — subscriber validates the signature; ★ no `hub.secret` → + no header. +- **unsubscribe** → verified removal; ★ no-echo → **not** removed. +- **leases** — requested value clamped + echoed in the verification GET; expiry via + `removeExpired()`. +- **WebSub-native publish** (`hub.mode=publish`) and **fat ping** each deliver content. + +### Slices (TDD vertical slices, red→green, in order) + +**Phase 0 — Foundations** +- [ ] **S0.1** ADR: WebSub hub = async-`202` intent verification via an in-process + best-effort `VerificationScheduler` seam; persisted queue + retry is a future refactor + behind the same seam. Record the lease + signature decisions too. +- [ ] **S0.2** `CONTEXT.md`: add the WebSub vocabulary above (tie "Hub" to the existing + Hub-end note; distinguish **Topic** from **Resource**, **Callback** from + **Subscription.url**). + +**Phase 1 — Subscribe happy path (async handshake; no secret/lease/content yet)** +- [ ] **S1.1** `websub-dispatcher` param parse/validate: `hub.mode`, `hub.callback` + (valid absolute URL), `hub.topic` (present) → malformed returns `{status:400}`; a valid + subscribe builds a `websub` `SubscribeRequest` **directly** (`callbackUrl=hub.callback`, + `resourceUrls=[hub.topic]`, not via `buildSubscribeRequest`). Pure unit tests, no network. +- [ ] **S1.2** `websub-plugin.verify()`: challenge GET to the callback with `hub.mode`, + `hub.topic`, `hub.challenge`; require `2xx` and an exact `hub.challenge` echo, else + throw (always verifies — ignores `diffDomain`). Injected `fetch` + challenge generator. + `protocols: ['websub']`. +- [ ] **S1.3** `VerificationScheduler` as a `createRssCloudCore` option (default + in-process: run task next tick, catch+log; injectable for tests) + an engine + async-accept method `acceptSubscription(req)` that returns immediately and schedules + verify→persist via the scheduler: success persists a `protocol:'websub'` subscription + (with `details`), failure records nothing. `core.subscribe` is unchanged — the accept + method is a new caller of it. Unit test drains a capturing scheduler. +- [ ] **S1.4** core `websub-dispatcher` ↔ express `websub({ core })` factory (same shape + as `ping`/`pleaseNotify`): parse the form body, `hub.mode=subscribe` → `core.accept…` + → `202`, malformed → `4xx`. Mirror `rest-middleware` (thin; dispatcher owns logic). + Export from `index.ts`. (No `scheduler`/`hubUrl` args — see architecture notes.) +- [ ] **S1.5** Server integration (prerequisite for the S1.6 e2e): + **(a)** `apps/server/core.js` — add `createWebSubProtocolPlugin({ hubUrl, + requestTimeoutMs })` to the `plugins` array (registers `'websub'`; otherwise + `core.subscribe` rejects it). + **(b)** `apps/server/controllers/index.js` — `router.post('/websub', websub({ core }))`. + **(c)** `apps/server/config.js` — env for the hub's public base URL (`HUB_URL`, + default derived from `DOMAIN`/`PORT`) and mount path (`WEBSUB_PATH`, default `/websub`). + (Lease bounds + signature algo are added in Phases 5/3 when their slices need them.) +- [ ] **S1.6** e2e (**establishes the reusable mock subscriber harness** — challenge-echo): + POST subscribe → `202`, callback receives the verification GET, then **poll** + `/subscriptions.json` (already lists every sub incl. `protocol:'websub'`) until the + record appears — verification is async, so the test waits rather than asserting inline; + ★ callback refuses to echo → record never appears (bounded timeout); ★ malformed + `hub.*` (missing callback/topic, bad mode) → `4xx`. + +**Phase 2 — Content distribution via the existing rssCloud ping (THE PAYOFF)** +> Proves the primary use case: an rssCloud-only publisher's `/ping` fans content out to +> WebSub subscribers. No WebSub publish path — relies on core's resource-keyed fan-out. +- [ ] **S2.1** `websub-plugin.deliver()`: POST `payload.body` to the callback, relaying + the topic's `Content-Type = payload.contentType` **verbatim** (xml/atom/json/etc. — the + hub is content-type-agnostic; `payload.contentType` is `string | null`, so pick a + fallback like `application/octet-stream` when the origin sent none), plus + `Link: ; rel="hub", ; rel="self"`. No signature yet. Inject `hubUrl`. + Unit tests with injected `fetch` (cover the present-and-null content-type branches). +- [ ] **S2.2** e2e (**the killer test** — extends the harness with content-capture): + put an rssCloud subscriber **and** a WebSub subscriber on the same topic `T`, then hit + the *existing* rssCloud `/ping` for `T` with changed content; assert **both** fire from + that single ping — the rssCloud sub gets its notify, the WebSub callback gets a POST + carrying the feed body + relayed `Content-Type` + `Link` rels. No `hub.mode=publish` + involved — this is the headline "free WebSub for rssCloud publishers" cross-protocol + proof. + +**Phase 3 — Authenticated distribution (HMAC-SHA256)** +- [ ] **S3.1** parse + store `hub.secret` in `details` at subscribe. **S3.2** when + `details.secret` present, add `X-Hub-Signature: sha256=HMAC(secret, body)`; algorithm a + configurable plugin option (default `sha256`); no secret → no header. **S3.3** e2e: + subscriber verifies the signature over the rssCloud-ping-delivered body. + +**Phase 4 — Unsubscribe (intent-verified)** +- [ ] **S4.1** plugin verify for `hub.mode=unsubscribe` (shared verify keyed by mode). + **S4.2** verified-unsubscribe path: scheduled task verifies intent then + `core.unsubscribe` (which has no verify hook today). **S4.3** dispatcher/express branch + `hub.mode=unsubscribe` → `202`. **S4.4** e2e unsubscribe handshake. + +**Phase 5 — Leases (honor requested, clamped)** +- [ ] **S5.1** `RssCloudConfig` lease bounds (default/min/max secs) + resolve defaults. + **S5.2** parse `hub.lease_seconds`, clamp, store `details.leaseSeconds`, + `whenExpires = now + chosen`; echo the chosen lease in the verification GET (thread the + chosen value into `verify`). **S5.3** e2e: requested lease clamped + echoed; expiry via + `removeExpired()`. + +**Phase 6 — WebSub-native publish front door (secondary — pure-WebSub publishers)** +- [ ] **S6.1** dispatcher/express `hub.mode=publish` (thin: `hub.url`/`hub.topic`) → + `core.ping(topic)` → `2xx`/`204`. Lets a publisher with *no* rssCloud ping trigger the + same fan-out. Reuses everything from Phase 2. **S6.2** e2e: WebSub publish → WebSub + subscriber receives content. + +**Phase 7 — Fat pings (secondary — publisher pushes the body)** +- [ ] **S7.1** decide + document the (non-standard) fat-ping wire format — topic via + param/header, raw body, and how to tell it from a thin publish (see open questions). + **S7.2** `PingRequest` optional pushed content; `detectChange` uses it instead of + fetching (still hashes for change detection). **S7.3** express publish detects a fat + ping → `core.ping` with pushed content → distributed verbatim. **S7.4** e2e fat ping. + +**Phase 8 — Hardening / spec niceties (deferred, optional)** +- [ ] `hub.mode=denied` callback notification on verification/validation failure. +- [ ] Persisted verification queue + retry (the seam refactor) — its own ADR/project. +- [ ] Publisher-facing docs: advertising the hub via ``. +- [ ] [websub.rocks](https://websub.rocks/) hub-conformance pass. + +*Coverage:* `packages/` stays at **100%** — every branch in the plugin, dispatcher, and +seam needs a test (or an explicit, justified ignore). e2e covers the integration. + +### Open questions (carry into the relevant slice) + +- **Fat-ping wire format (S7.1):** WebSub has no standard fat ping (it was a + PubSubHubbub 0.4 extension). Decide how a publisher indicates the topic when pushing a + body — a query/`hub.topic` param alongside a raw body, a `Content-Location`/`Link` + header, etc. — and how to distinguish it from a thin `hub.mode=publish`. +- **Resource pre-read on subscribe:** `core.subscribe` pre-pings the resource; WebSub + subscribe may skip that (the spec only requires intent verification). Decide when + wiring the accept path. +- **Seam ownership:** confirm the `VerificationScheduler` is core-owned (so a future + persisted queue lives next to the store) vs. injected from the composition root. From 7b1e708f96d0a5bac33661e410c9a58c4b8f3d03 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 10:57:15 -0500 Subject: [PATCH 02/35] docs(websub): record async-202 ADR and WebSub vocabulary ADR-0002 captures the settled WebSub hub design: async-202 intent verification behind an in-process best-effort VerificationScheduler seam (a persisted queue + retry is a future refactor behind the same seam), plus the lease (honor-requested-clamped) and HMAC-SHA256 signature decisions. CONTEXT.md gains a WebSub vocabulary cluster (Topic vs Resource, Callback vs Subscription.url, Intent verification, VerificationScheduler, Lease, Content distribution, Fat ping, X-Hub-Signature), ties the Hub and Notification entries to their WebSub roles, and adds a dialogue exchange. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTEXT.md | 91 ++++++++++++++++++- ...2-websub-async-intent-verification-seam.md | 75 +++++++++++++++ 2 files changed, 163 insertions(+), 3 deletions(-) create mode 100644 docs/adr/0002-websub-async-intent-verification-seam.md diff --git a/CONTEXT.md b/CONTEXT.md index 69da97a..c9d5ff9 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -73,7 +73,9 @@ _Avoid_: cross-origin, external, remote. **Hub**: The server end of the protocol: it answers **pleaseNotify** and **Ping**, owns the **Resource**/**Subscription** state, and fans **Notification**s out. `@rsscloud/core` is the -protocol-neutral hub engine; `apps/server` is one deployment of it. +protocol-neutral hub engine; `apps/server` is one deployment of it. The same engine also +plays the [WebSub](https://www.w3.org/TR/websub/) **Hub** role (the W3C term is literally +"hub") — see the **WebSub** terms below; it never hosts source feeds in either protocol. _Avoid_: server (that's a deployment of the hub, not the role), broker. **Client**: @@ -98,8 +100,9 @@ _Avoid_: feed (that's the parsed metadata on a **Resource**), source, producer. **Notification**: The outbound delivery from the **Hub** to a **Subscriber**'s callback when a **Resource** -changes — an `http-post` `url=` form or an XML-RPC `rssCloud.notify` call. What a -**Protocol plugin** sends and the **Client** receives and acknowledges. +changes — an `http-post` `url=` form, an XML-RPC `rssCloud.notify` call, or (for WebSub) a +**Content distribution** POST carrying the body itself. What a **Protocol plugin** sends +and the **Client** receives and acknowledges. _Avoid_: ping (that's the inbound publisher signal), pleaseNotify (the inbound subscribe), message, event. @@ -116,6 +119,81 @@ shared by the **Hub** and the **Client**. It speaks typed `XmlRpcValue`s and car `rssCloud.*` semantics — each end maps its own method shapes onto it. _Avoid_: parser (that's one half), serializer, XML library. +### WebSub + +The **Hub** also speaks [WebSub](https://www.w3.org/TR/websub/), the W3C successor to +PubSubHubbub. Hub-only: `apps/client` still owns the subscriber/publisher side, and the +hub never hosts source feeds (publishers point at it via `` in their own +feeds). These terms name what's WebSub-specific; they reuse the core terms above wherever +the concept is the same. + +**Topic**: +The feed URL a WebSub **Subscriber** names in `hub.topic` — the WebSub-wire name for the +same URL core stores change-detection state about as a **Resource**. A subscriber's +`hub.topic` must be the *exact* URL string the publisher **Ping**s, because the store keys +feed entries by exact URL (the same exactness rssCloud already requires between the +subscribe-URL and the ping-URL; URL normalization is out of scope). +_Avoid_: Resource (that's core's stored state for the URL; "Topic" is the WebSub-wire name +for the URL the subscriber names), feed. + +**Callback**: +The complete URL a WebSub **Subscriber** supplies in `hub.callback` — where **Content +distribution** POSTs and the **Intent verification** GET are sent. It becomes the +**Subscription**'s `url` directly: unlike rssCloud (where **buildSubscribeRequest** glues +the callback from port/path/domain), WebSub arrives with a finished URL, so the dispatcher +sets `callbackUrl = hub.callback` and skips the builder. +_Avoid_: Subscription.url (that's the stored field the callback becomes), notify endpoint, +apiurl. + +**Intent verification**: +The WebSub handshake confirming a **Subscriber** actually requested a (un)subscribe: the +**Hub** GETs the **Callback** with `hub.mode` / `hub.topic` / `hub.challenge` (plus the +chosen **Lease**) and requires an exact `hub.challenge` echo with a `2xx`. WebSub *always* +verifies (spec mandate), so its **Protocol plugin** ignores **diffDomain** and never does +the rssCloud same-domain test-notify. Verification is async: the Hub answers the inbound +request `202` first, then runs the GET out of band via the **VerificationScheduler**. +_Avoid_: challenge handshake (rssCloud's term — related, but WebSub always verifies, echoes +a challenge, and runs async), diffDomain (WebSub ignores it). + +**VerificationScheduler**: +The core-owned seam that runs the verify-then-persist task behind the async `202`. The +default runs it in-process, best-effort (one attempt, failures logged, a restart drops the +pending request). A future persisted-queue + retry implementation satisfies the same seam +with no change to the dispatcher, the plugin's verify, or the express factory. WebSub-only +and additive — rssCloud subscribe stays synchronous. See ADR-0002. +_Avoid_: queue (the default isn't durable yet), job runner, worker. + +**Lease**: +The bounded lifetime of a WebSub **Subscription**. The **Hub** honors the subscriber's +requested `hub.lease_seconds` clamped to a configurable `[min, max]` (a default applies +when omitted), stores the chosen value in `details.leaseSeconds`, sets +`whenExpires = now + chosen`, and echoes the chosen value in the **Intent verification** +GET. `removeExpired()` drops the subscription on lapse, unchanged. +_Avoid_: expiry (that's the resulting `whenExpires`; the Lease is the requested-then-clamped +duration), TTL. + +**Content distribution**: +The WebSub form of **Notification**: the **Hub** POSTs the changed **Topic**'s body +*verbatim* to the **Callback**, relaying the origin `Content-Type` and adding +`Link: ; rel="hub", ; rel="self"`. Where an rssCloud **Notification** sends +only the changed URL, Content distribution sends the content itself — so one rssCloud +**Ping** can drive both, from the same already-fetched body. +_Avoid_: notify (rssCloud's content-free signal), push, broadcast. + +**Fat ping**: +A publish in which the **Publisher** POSTs the changed body itself, so the **Hub** +distributes it verbatim *without* re-fetching the **Topic**. Non-standard (a PubSubHubbub +0.4 extension), so its wire format is a project decision. Contrast a thin publish +(`hub.mode=publish`), which names only the URL and triggers a re-fetch through `core.ping`. +_Avoid_: publish (a thin publish re-fetches; a Fat ping carries the body), push. + +**X-Hub-Signature**: +The HMAC the **Hub** adds over a **Content distribution** body (`X-Hub-Signature: sha256=…`) +when the **Subscriber** supplied a `hub.secret`, letting the subscriber authenticate the +delivery. The algorithm is a config knob (default `sha256`); no `hub.secret` → no header. +_Avoid_: HMAC (that's the algorithm; the header is the wire artifact), auth token, signature +(ambiguous — name the header). + ## Example dialogue > **Dev:** When a `pleaseNotify` comes in over XML-RPC, who decides the callback is `diffDomain`? @@ -127,3 +205,10 @@ _Avoid_: parser (that's one half), serializer, XML library. > **Domain expert:** They share the **XML-RPC codec** (`@rsscloud/xml-rpc`), not each other's calls. The Client builds `rssCloud.pleaseNotify`/`rssCloud.ping`; the Hub parses those and sends a **Notification**. Each maps its own `rssCloud.*` shapes onto the codec's typed values. > **Dev:** And how does a **Publisher** point a **Subscriber** at us? > **Domain expert:** Via the **Cloud element** in the publisher's own feed — the Client's `renderCloudFeed` writes it. The Hub never hosts the feed; it just answers the **pleaseNotify** the subscriber sends after reading that ``. + +> **Dev:** A WebSub subscriber names a **Topic** and core stores a **Resource** — are those two different things? +> **Domain expert:** Same URL, different vantage point. **Topic** is the WebSub-wire name for the feed URL the subscriber asks about; **Resource** is core's stored change-detection state for that URL. They have to be the *exact* same string — the store keys by exact URL, just like rssCloud already requires the subscribe-URL to match the ping-URL. +> **Dev:** So when an rssCloud **Publisher** **Ping**s, does a WebSub subscriber on that Topic hear about it? +> **Domain expert:** Yes — that's the headline. One **Ping** fetches the body once and fans out per **Subscription**: the rssCloud sub gets a **Notification**, the WebSub sub gets a **Content distribution** POST of that same body. The publisher never speaks WebSub; it only added `` to its feed. +> **Dev:** And the subscriber's `202`? +> **Domain expert:** That's just "accepted". **Intent verification** runs async behind the **VerificationScheduler** — the Hub GETs the **Callback**, checks the `hub.challenge` echo, and only then records the **Subscription**. So a test polls `/subscriptions.json`; it doesn't assert the record exists the instant the `202` lands. diff --git a/docs/adr/0002-websub-async-intent-verification-seam.md b/docs/adr/0002-websub-async-intent-verification-seam.md new file mode 100644 index 0000000..f3bcb97 --- /dev/null +++ b/docs/adr/0002-websub-async-intent-verification-seam.md @@ -0,0 +1,75 @@ +# WebSub intent verification is async-202 behind a best-effort VerificationScheduler seam + +The [WebSub](https://www.w3.org/TR/websub/) spec mandates that a hub confirm a +subscriber's intent before recording a (un)subscribe: the hub GETs the subscriber's +`hub.callback` carrying `hub.mode` / `hub.topic` / `hub.challenge` and requires an exact +`hub.challenge` echo with a `2xx`. The spec lets the hub do this synchronously (`2xx`) +*or* asynchronously (`202 Accepted`, verify out of band). This ADR records that we take +the async-`202` route, the seam it hides behind, and two adjacent settled decisions +(lease handling and delivery signatures) made the same day. + +## Status + +accepted + +## Decision + +1. **Async `202`.** The `hub.*` dispatcher validates the request *synchronously* + (malformed → `4xx`), returns `202 Accepted`, and only then performs the + `hub.challenge` GET out of band, recording the `protocol:'websub'` subscription on a + successful echo and recording nothing on failure. This keeps the inbound request fast + and decouples the subscriber's HTTP round-trip from our outbound verification. + +2. **One verification-dispatch seam; in-process best-effort now, persisted queue later.** + Async ≠ a durable queue. A single seam — a `VerificationScheduler` — runs the + verify-then-persist task. The default implementation runs it **in-process, one + attempt, failures logged**; a restart mid-flight simply drops the pending request and + the subscriber re-subscribes (WebSub subscribers are expected to renew). A future + persisted-queue + retry implementation satisfies the *same* seam — draining on the + existing maintenance interval and persisting through the store — with **no change** to + the `hub.*` dispatcher, the plugin's `verify()`, or the express factory. + +3. **The scheduler is core-owned, additive, and WebSub-only.** It is a + `createRssCloudCore` option (default in-process, injectable for tests), not an argument + of the dispatcher or the express factory. It lives in core — not express — only so the + future persisted queue can reach the store; the in-process default would work + anywhere. rssCloud `pleaseNotify` / `subscribe` stays **synchronous** (its callers + expect an immediate yes/no), and `ping` / `fanOut` / `deliver` are untouched. The + async-accept path is a brand-new caller of an unchanged `core.subscribe`, so no + existing rssCloud behaviour changes. + +4. **Lease = honor requested, clamped.** The hub uses the subscriber's + `hub.lease_seconds` clamped to a configurable `[min, max]` (a default applies when the + subscriber omits it), stores the chosen value in `details.leaseSeconds`, sets + `whenExpires = now + chosen`, and echoes the chosen value in the verification GET. The + existing `removeExpired()` drops the subscription on lapse, unchanged. Lease bounds are + protocol-relevant, so they belong in `RssCloudConfig` alongside `ctSecsResourceExpire`. + +5. **Signature = HMAC-SHA256, configurable.** When a subscriber supplies a `hub.secret`, + each content delivery is signed with `X-Hub-Signature: sha256=…` (the algorithm is a + plugin config knob, default `sha256`). No `hub.secret` → no signature header. + +## Why a seam rather than committing to a queue now + +The headline use case — free WebSub content distribution for publishers already pinging +this server over rssCloud — needs none of the durability a persisted queue buys: it rides +the existing resource-keyed fan-out, where the WebSub `deliver()` is just another plugin. +Only the *subscribe/unsubscribe handshake* is async, and a dropped handshake is +self-healing (the subscriber retries). Building the persisted queue up front would be +speculative complexity; refusing to leave room for it would be a trap. A seam is the +cheap middle: it lets the best-effort default ship now and the durable implementation +land later as a pure substitution, captured here so the substitution isn't mistaken for +a behavioural change. + +## Consequences + +- A subscriber's `202` does **not** mean "subscribed" — only "request accepted; intent + verification pending". The e2e suite therefore **polls** `/subscriptions.json` until the + record appears (or a bounded timeout proves it never will), rather than asserting inline. +- A process restart between `202` and a successful challenge GET loses that pending + request with no record anywhere. Acceptable under best-effort; the subscriber + re-subscribes. The future persisted-queue implementation removes this window. +- `core.unsubscribe()` has no verify hook today. A verified WebSub unsubscribe must run + the `hub.mode=unsubscribe` challenge GET through the **same** scheduler before calling + `core.unsubscribe` — the verification belongs to the scheduled task, not to + `core.unsubscribe` itself. From c4dd16cbf3a166fb7f3b482c7e6ed98853edf316 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 10:57:23 -0500 Subject: [PATCH 03/35] feat(core): parse and validate WebSub hub.* subscribe requests Add parseSubscribe in protocols/websub-dispatcher.ts: validates hub.mode (subscribe), hub.callback (a valid absolute URL), and hub.topic (present), returning {status:400} for anything malformed. A valid request builds a 'websub' SubscribeRequest directly (callbackUrl=hub.callback, resourceUrls=[hub.topic]) without buildSubscribeRequest, which gates on rssCloud-only protocols and assembles callbacks from port/path/domain. Internal for now; createWebSubDispatcher and the index export land with the express factory (S1.4). 100% coverage maintained. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 8 ++- .../src/protocols/websub-dispatcher.test.ts | 68 +++++++++++++++++++ .../core/src/protocols/websub-dispatcher.ts | 53 +++++++++++++++ 3 files changed, 126 insertions(+), 3 deletions(-) create mode 100644 packages/core/src/protocols/websub-dispatcher.test.ts create mode 100644 packages/core/src/protocols/websub-dispatcher.ts diff --git a/TODO.md b/TODO.md index 7e6e40c..6fbb4b6 100644 --- a/TODO.md +++ b/TODO.md @@ -154,18 +154,20 @@ Flows that must have an e2e (happy path + the ★ negatives): ### Slices (TDD vertical slices, red→green, in order) **Phase 0 — Foundations** -- [ ] **S0.1** ADR: WebSub hub = async-`202` intent verification via an in-process +- [x] **S0.1** ADR: WebSub hub = async-`202` intent verification via an in-process best-effort `VerificationScheduler` seam; persisted queue + retry is a future refactor behind the same seam. Record the lease + signature decisions too. -- [ ] **S0.2** `CONTEXT.md`: add the WebSub vocabulary above (tie "Hub" to the existing + (→ `docs/adr/0002-websub-async-intent-verification-seam.md`) +- [x] **S0.2** `CONTEXT.md`: add the WebSub vocabulary above (tie "Hub" to the existing Hub-end note; distinguish **Topic** from **Resource**, **Callback** from **Subscription.url**). **Phase 1 — Subscribe happy path (async handshake; no secret/lease/content yet)** -- [ ] **S1.1** `websub-dispatcher` param parse/validate: `hub.mode`, `hub.callback` +- [x] **S1.1** `websub-dispatcher` param parse/validate: `hub.mode`, `hub.callback` (valid absolute URL), `hub.topic` (present) → malformed returns `{status:400}`; a valid subscribe builds a `websub` `SubscribeRequest` **directly** (`callbackUrl=hub.callback`, `resourceUrls=[hub.topic]`, not via `buildSubscribeRequest`). Pure unit tests, no network. + (→ `packages/core/src/protocols/websub-dispatcher.ts`: `parseSubscribe`) - [ ] **S1.2** `websub-plugin.verify()`: challenge GET to the callback with `hub.mode`, `hub.topic`, `hub.challenge`; require `2xx` and an exact `hub.challenge` echo, else throw (always verifies — ignores `diffDomain`). Injected `fetch` + challenge generator. diff --git a/packages/core/src/protocols/websub-dispatcher.test.ts b/packages/core/src/protocols/websub-dispatcher.test.ts new file mode 100644 index 0000000..0902861 --- /dev/null +++ b/packages/core/src/protocols/websub-dispatcher.test.ts @@ -0,0 +1,68 @@ +import { describe, expect, it } from 'vitest'; +import { parseSubscribe } from './websub-dispatcher.js'; + +describe('parseSubscribe', () => { + it('builds a websub SubscribeRequest directly from hub.callback and hub.topic', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ + ok: true, + request: { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example.com/listener', + protocol: 'websub' + } + }); + }); + + it('rejects a body with no hub.mode as a 400', () => { + const result = parseSubscribe({ + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); + + it('rejects a missing hub.callback as a 400', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); + + it('rejects a hub.callback that is not a valid absolute URL as a 400', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'not a url', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); + + it('rejects a missing hub.topic as a 400', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); + + it('rejects an empty hub.topic as a 400', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': '' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); +}); diff --git a/packages/core/src/protocols/websub-dispatcher.ts b/packages/core/src/protocols/websub-dispatcher.ts new file mode 100644 index 0000000..df4e624 --- /dev/null +++ b/packages/core/src/protocols/websub-dispatcher.ts @@ -0,0 +1,53 @@ +import type { SubscribeRequest } from '../engine/dto.js'; + +/** + * Outcome of parsing a WebSub `hub.*` subscribe request: either a ready-to-drive + * {@link SubscribeRequest}, or a malformed-request status the front door renders. + */ +export type WebSubParseResult = + | { ok: true; request: SubscribeRequest } + | { ok: false; status: number }; + +/** Any `hub.*` shape the hub can't act on is a malformed request. */ +const MALFORMED: WebSubParseResult = { ok: false, status: 400 }; + +/** True when `value` parses as an absolute URL (a relative URL throws sans base). */ +function isAbsoluteUrl(value: string): boolean { + try { + new URL(value); + return true; + } catch { + return false; + } +} + +/** + * Parse and validate a WebSub subscribe form body (`hub.mode` / `hub.callback` / + * `hub.topic`). On success builds a `websub` {@link SubscribeRequest} *directly* + * — the complete `hub.callback` is the callback URL and `hub.topic` the sole + * resource, so this skips `buildSubscribeRequest` (which assembles a callback + * from port/path/domain and gates on rssCloud-only protocols). + */ +export function parseSubscribe( + body: Record +): WebSubParseResult { + if (body['hub.mode'] !== 'subscribe') { + return MALFORMED; + } + const callback = body['hub.callback']; + if (typeof callback !== 'string' || !isAbsoluteUrl(callback)) { + return MALFORMED; + } + const topic = body['hub.topic']; + if (typeof topic !== 'string' || topic === '') { + return MALFORMED; + } + return { + ok: true, + request: { + resourceUrls: [topic], + callbackUrl: callback, + protocol: 'websub' + } + }; +} From 9f0853f8840b3b0b558e72770ae52ab0cf3f7b64 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 11:03:25 -0500 Subject: [PATCH 04/35] feat(core): verify WebSub subscriber intent with a challenge GET MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add createWebSubProtocolPlugin (protocols: ['websub']). verify() always performs the WebSub intent-verification GET — never the rssCloud same-domain test-notify, so it ignores diffDomain — appending hub.mode=subscribe / hub.topic / hub.challenge to the callback (preserving any existing query) and requiring a 2xx with an exact challenge echo, else throwing. fetch and the challenge generator are injectable. deliver() is an interim stub reporting failure (it must not throw; the engine's deliverTo does not catch); real content distribution is S2.1. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 5 +- .../core/src/protocols/websub-plugin.test.ts | 243 ++++++++++++++++++ packages/core/src/protocols/websub-plugin.ts | 77 ++++++ 3 files changed, 323 insertions(+), 2 deletions(-) create mode 100644 packages/core/src/protocols/websub-plugin.test.ts create mode 100644 packages/core/src/protocols/websub-plugin.ts diff --git a/TODO.md b/TODO.md index 6fbb4b6..9735be8 100644 --- a/TODO.md +++ b/TODO.md @@ -168,10 +168,11 @@ Flows that must have an e2e (happy path + the ★ negatives): subscribe builds a `websub` `SubscribeRequest` **directly** (`callbackUrl=hub.callback`, `resourceUrls=[hub.topic]`, not via `buildSubscribeRequest`). Pure unit tests, no network. (→ `packages/core/src/protocols/websub-dispatcher.ts`: `parseSubscribe`) -- [ ] **S1.2** `websub-plugin.verify()`: challenge GET to the callback with `hub.mode`, +- [x] **S1.2** `websub-plugin.verify()`: challenge GET to the callback with `hub.mode`, `hub.topic`, `hub.challenge`; require `2xx` and an exact `hub.challenge` echo, else throw (always verifies — ignores `diffDomain`). Injected `fetch` + challenge generator. - `protocols: ['websub']`. + `protocols: ['websub']`. (→ `packages/core/src/protocols/websub-plugin.ts`; + `deliver()` is an interim failing stub until S2.1.) - [ ] **S1.3** `VerificationScheduler` as a `createRssCloudCore` option (default in-process: run task next tick, catch+log; injectable for tests) + an engine async-accept method `acceptSubscription(req)` that returns immediately and schedules diff --git a/packages/core/src/protocols/websub-plugin.test.ts b/packages/core/src/protocols/websub-plugin.test.ts new file mode 100644 index 0000000..6d3a318 --- /dev/null +++ b/packages/core/src/protocols/websub-plugin.test.ts @@ -0,0 +1,243 @@ +import { describe, expect, it } from 'vitest'; +import type { DeliveryContext, VerifyContext } from '../engine/plugin.js'; +import type { Resource } from '../engine/resource.js'; +import type { Subscription } from '../engine/subscription.js'; +import { createWebSubProtocolPlugin } from './websub-plugin.js'; + +const epoch = new Date(0); + +function subscription(url: string): Subscription { + return { + url, + protocol: 'websub', + ctUpdates: 0, + ctErrors: 0, + ctConsecutiveErrors: 0, + whenCreated: epoch, + whenLastUpdate: null, + whenLastError: null, + whenExpires: new Date('2099-01-01T00:00:00Z') + }; +} + +function verifyContext( + callbackUrl: string, + resourceUrl: string, + diffDomain: boolean +): VerifyContext { + return { + subscription: subscription(callbackUrl), + resourceUrl, + diffDomain + }; +} + +function resource(url: string): Resource { + return { + url, + lastHash: '', + lastSize: 0, + ctChecks: 0, + whenLastCheck: epoch, + ctUpdates: 0, + whenLastUpdate: epoch + }; +} + +function deliveryContext( + callbackUrl: string, + resourceUrl: string +): DeliveryContext { + return { + subscription: subscription(callbackUrl), + resource: resource(resourceUrl), + payload: { body: '', contentType: null } + }; +} + +describe('createWebSubProtocolPlugin verify', () => { + it('GETs the callback with hub.mode/topic/challenge and resolves on an exact 2xx echo', async () => { + const calls: string[] = []; + const fakeFetch = (async (url: string | URL) => { + calls.push(String(url)); + const challenge = new URL(String(url)).searchParams.get( + 'hub.challenge' + ); + return new Response(challenge, { status: 200 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + createChallenge: () => 'chal-123' + }); + + await expect( + plugin.verify( + verifyContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + true + ) + ) + ).resolves.toBeUndefined(); + + const url = new URL(calls[0] as string); + expect(url.origin + url.pathname).toBe('https://sub.example/listener'); + expect(url.searchParams.get('hub.mode')).toBe('subscribe'); + expect(url.searchParams.get('hub.topic')).toBe( + 'http://feed.example/rss' + ); + expect(url.searchParams.get('hub.challenge')).toBe('chal-123'); + }); + + it('rejects when the 2xx response does not echo the exact challenge', async () => { + const fakeFetch = (async () => + new Response('not-the-challenge', { status: 200 })) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + createChallenge: () => 'chal-123' + }); + + await expect( + plugin.verify( + verifyContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + true + ) + ) + ).rejects.toThrow(); + }); + + it('rejects when the challenge response is non-2xx even if it echoes', async () => { + const fakeFetch = (async (url: string | URL) => { + const challenge = new URL(String(url)).searchParams.get( + 'hub.challenge' + ); + return new Response(challenge, { status: 404 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + createChallenge: () => 'chal-123' + }); + + await expect( + plugin.verify( + verifyContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + true + ) + ) + ).rejects.toThrow(); + }); + + it('always verifies via the challenge GET, ignoring diffDomain=false', async () => { + const calls: { url: string; method: string | undefined }[] = []; + const fakeFetch = (async (url: string | URL, init?: RequestInit) => { + calls.push({ url: String(url), method: init?.method }); + const challenge = new URL(String(url)).searchParams.get( + 'hub.challenge' + ); + return new Response(challenge, { status: 200 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + createChallenge: () => 'chal-123' + }); + + await expect( + plugin.verify( + verifyContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + false + ) + ) + ).resolves.toBeUndefined(); + + expect(calls).toHaveLength(1); + expect(calls[0]?.method).toBe('GET'); + expect( + new URL(calls[0]?.url as string).searchParams.get('hub.challenge') + ).toBe('chal-123'); + }); + + it('preserves existing query params on the callback when appending hub.*', async () => { + const calls: string[] = []; + const fakeFetch = (async (url: string | URL) => { + calls.push(String(url)); + const challenge = new URL(String(url)).searchParams.get( + 'hub.challenge' + ); + return new Response(challenge, { status: 200 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + createChallenge: () => 'chal-123' + }); + + await plugin.verify( + verifyContext( + 'https://sub.example/listener?id=42', + 'http://feed.example/rss', + true + ) + ); + + const url = new URL(calls[0] as string); + expect(url.searchParams.get('id')).toBe('42'); + expect(url.searchParams.get('hub.mode')).toBe('subscribe'); + }); + + it('generates its own challenge token when none is injected', async () => { + let sentChallenge: string | null = null; + const fakeFetch = (async (url: string | URL) => { + sentChallenge = new URL(String(url)).searchParams.get( + 'hub.challenge' + ); + return new Response(sentChallenge, { status: 200 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ fetch: fakeFetch }); + + await expect( + plugin.verify( + verifyContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + true + ) + ) + ).resolves.toBeUndefined(); + + expect(sentChallenge).toMatch(/^[0-9a-f]+$/); + }); +}); + +describe('createWebSubProtocolPlugin protocols', () => { + it('owns the websub protocol value', () => { + const plugin = createWebSubProtocolPlugin(); + expect(plugin.protocols).toEqual(['websub']); + }); +}); + +describe('createWebSubProtocolPlugin deliver', () => { + it('reports failure since content distribution is not implemented yet (S2.1)', async () => { + const plugin = createWebSubProtocolPlugin(); + + const result = await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss' + ) + ); + + expect(result.ok).toBe(false); + expect(result.error).toBeInstanceOf(Error); + }); +}); diff --git a/packages/core/src/protocols/websub-plugin.ts b/packages/core/src/protocols/websub-plugin.ts new file mode 100644 index 0000000..c25aad4 --- /dev/null +++ b/packages/core/src/protocols/websub-plugin.ts @@ -0,0 +1,77 @@ +import type { + DeliveryResult, + ProtocolPlugin, + VerifyContext +} from '../engine/plugin.js'; +import type { Protocol } from '../engine/protocol.js'; +import { fetchWithTimeout } from '../fetch-with-timeout.js'; + +/** Construction-time dependencies for the WebSub protocol plugin. */ +export interface WebSubProtocolPluginOptions { + /** Injectable fetch (tests, edge runtimes); defaults to global fetch. */ + fetch?: typeof fetch; + /** Per-request timeout (ms) for outbound calls. */ + requestTimeoutMs?: number; + /** Challenge generator for the intent-verification GET (injectable for tests). */ + createChallenge?: () => string; +} + +const WEBSUB_PROTOCOLS: Protocol[] = ['websub']; + +/** Fallback request timeout when none is supplied (mirrors the server default). */ +const DEFAULT_REQUEST_TIMEOUT_MS = 4000; + +/** Portable, hard-to-guess token for the intent-verification challenge. */ +function defaultCreateChallenge(): string { + const bytes = new Uint8Array(16); + globalThis.crypto.getRandomValues(bytes); + return Array.from(bytes, byte => byte.toString(16).padStart(2, '0')).join( + '' + ); +} + +/** + * The WebSub delivery protocol (`websub`). A new subscription's intent is always + * confirmed with the WebSub verification GET (the spec mandate) — never the + * rssCloud same-domain test-notify — so `verify` ignores `diffDomain`. + */ +export function createWebSubProtocolPlugin( + options: WebSubProtocolPluginOptions = {} +): ProtocolPlugin { + const doFetch = options.fetch ?? fetch; + const requestTimeoutMs = + options.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS; + const createChallenge = options.createChallenge ?? defaultCreateChallenge; + + async function verify(ctx: VerifyContext): Promise { + const challenge = createChallenge(); + const verifyUrl = new URL(ctx.subscription.url); + verifyUrl.searchParams.set('hub.mode', 'subscribe'); + verifyUrl.searchParams.set('hub.topic', ctx.resourceUrl); + verifyUrl.searchParams.set('hub.challenge', challenge); + + const res = await fetchWithTimeout( + doFetch, + requestTimeoutMs, + verifyUrl.toString(), + { method: 'GET' } + ); + const body = await res.text(); + + if (!res.ok || body !== challenge) { + throw new Error('WebSub intent verification failed'); + } + } + + // Content distribution lands in S2.1; until then delivery reports failure + // rather than throwing (the engine's deliverTo does not catch). The context + // parameter is omitted until the real implementation consumes it. + function deliver(): Promise { + return Promise.resolve({ + ok: false, + error: new Error('WebSub content distribution not implemented') + }); + } + + return { protocols: WEBSUB_PROTOCOLS, verify, deliver }; +} From 264aa394d1f59c8e780878a0087790e73fab6d75 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 11:14:46 -0500 Subject: [PATCH 05/35] feat(core): add async-202 accept seam for WebSub subscriptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce VerificationScheduler — the seam behind WebSub's async-202 intent verification. The default in-process scheduler runs each verify→persist task on the microtask queue (best-effort, one attempt) and routes a rejection to onError; a future persisted queue can satisfy the same interface (ADR-0002). core.acceptSubscription(req) returns immediately and schedules the work via the scheduler. It is a new caller of the unchanged subscribe, so a successful verify persists the subscription and a refusal persists nothing — the synchronous rssCloud subscribe path is untouched. The default scheduler surfaces a thrown task through the existing error event (scope: websub-verification), coercing any non-Error throwable. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 5 +- packages/core/src/engine/core.ts | 15 ++ packages/core/src/engine/create-core.test.ts | 166 ++++++++++++++++++ packages/core/src/engine/create-core.ts | 25 +++ .../src/engine/verification-scheduler.test.ts | 40 +++++ .../core/src/engine/verification-scheduler.ts | 38 ++++ packages/core/src/index.ts | 5 + 7 files changed, 293 insertions(+), 1 deletion(-) create mode 100644 packages/core/src/engine/verification-scheduler.test.ts create mode 100644 packages/core/src/engine/verification-scheduler.ts diff --git a/TODO.md b/TODO.md index 9735be8..c7351df 100644 --- a/TODO.md +++ b/TODO.md @@ -173,12 +173,15 @@ Flows that must have an e2e (happy path + the ★ negatives): throw (always verifies — ignores `diffDomain`). Injected `fetch` + challenge generator. `protocols: ['websub']`. (→ `packages/core/src/protocols/websub-plugin.ts`; `deliver()` is an interim failing stub until S2.1.) -- [ ] **S1.3** `VerificationScheduler` as a `createRssCloudCore` option (default +- [x] **S1.3** `VerificationScheduler` as a `createRssCloudCore` option (default in-process: run task next tick, catch+log; injectable for tests) + an engine async-accept method `acceptSubscription(req)` that returns immediately and schedules verify→persist via the scheduler: success persists a `protocol:'websub'` subscription (with `details`), failure records nothing. `core.subscribe` is unchanged — the accept method is a new caller of it. Unit test drains a capturing scheduler. + (→ `engine/verification-scheduler.ts`; default scheduler routes a thrown task to the + `error` event, scope `websub-verification`. Pre-ping-on-subscribe kept for now — see + open question.) - [ ] **S1.4** core `websub-dispatcher` ↔ express `websub({ core })` factory (same shape as `ping`/`pleaseNotify`): parse the form body, `hub.mode=subscribe` → `core.accept…` → `202`, malformed → `4xx`. Mirror `rest-middleware` (thin; dispatcher owns logic). diff --git a/packages/core/src/engine/core.ts b/packages/core/src/engine/core.ts index d89a149..0f11ca7 100644 --- a/packages/core/src/engine/core.ts +++ b/packages/core/src/engine/core.ts @@ -13,6 +13,7 @@ import type { ProtocolPlugin } from './plugin.js'; import type { MaintenanceResult, Stats } from './stats.js'; import type { Resource } from './resource.js'; import type { Subscription } from './subscription.js'; +import type { VerificationScheduler } from './verification-scheduler.js'; import type { FeedEntry, Store } from '../store/store.js'; /** @@ -40,6 +41,12 @@ export interface RssCloudCoreOptions { now?: () => Date; /** Feed metadata parser; defaults to core's built-in. */ feedParser?: FeedParser; + /** + * Runs WebSub's out-of-band verify→persist work after an async-`202` accept. + * Defaults to an in-process best-effort scheduler (see ADR-0002); a host may + * inject a persisted-queue implementation. + */ + scheduler?: VerificationScheduler; } /** @@ -50,6 +57,14 @@ export interface RssCloudCoreOptions { export interface RssCloudCore { /** Establish or renew subscriptions. */ subscribe(req: SubscribeRequest): Promise; + /** + * Accept a subscription for async (WebSub-style) intent verification: returns + * immediately and schedules the verify→persist work via the + * {@link RssCloudCoreOptions.scheduler}. A successful verify persists the + * subscription; a failed one persists nothing. A new caller of + * {@link subscribe} — the synchronous rssCloud path is unchanged. + */ + acceptSubscription(req: SubscribeRequest): void; /** Cancel subscriptions. */ unsubscribe(req: UnsubscribeRequest): Promise; /** diff --git a/packages/core/src/engine/create-core.test.ts b/packages/core/src/engine/create-core.test.ts index 25a60c8..ada3293 100644 --- a/packages/core/src/engine/create-core.test.ts +++ b/packages/core/src/engine/create-core.test.ts @@ -890,3 +890,169 @@ describe('createRssCloudCore feed seeding', () => { expect(await core.listFeeds()).toEqual([]); }); }); + +describe('createRssCloudCore acceptSubscription', () => { + function captureScheduler(): { + tasks: (() => Promise)[]; + schedule: (task: () => Promise) => void; + } { + const tasks: (() => Promise)[] = []; + return { tasks, schedule: task => void tasks.push(task) }; + } + + it('schedules verify→persist and persists a websub subscription on success', async () => { + const store = createInMemoryStore(); + const scheduler = captureScheduler(); + + const core = createRssCloudCore({ + store, + plugins: [makePlugin({ protocols: ['websub'] })], + config: resolveConfig(), + fetch: fetchReturning(RSS), + scheduler + }); + + core.acceptSubscription({ + resourceUrls: [FEED], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub', + details: { leaseSeconds: 600 } + }); + + // Returns immediately: the task is queued, not run — nothing persisted. + expect(scheduler.tasks).toHaveLength(1); + expect(await store.getSubscriptions(FEED)).toEqual([]); + + await scheduler.tasks[0]?.(); + + const subs = await store.getSubscriptions(FEED); + expect(subs).toHaveLength(1); + expect(subs[0]).toMatchObject({ + url: 'https://sub.example/listener', + protocol: 'websub', + details: { leaseSeconds: 600 } + }); + }); + + it('persists nothing when the scheduled verification fails', async () => { + const store = createInMemoryStore(); + const scheduler = captureScheduler(); + + const core = createRssCloudCore({ + store, + plugins: [ + makePlugin({ + protocols: ['websub'], + verify: vi.fn(async () => { + throw new Error('callback did not echo the challenge'); + }) + }) + ], + config: resolveConfig(), + fetch: fetchReturning(RSS), + scheduler + }); + + core.acceptSubscription({ + resourceUrls: [FEED], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub' + }); + + await scheduler.tasks[0]?.(); + + expect(await store.getSubscriptions(FEED)).toEqual([]); + }); + + it('runs the verify→persist on the default in-process scheduler when none is injected', async () => { + const store = createInMemoryStore(); + + const core = createRssCloudCore({ + store, + plugins: [makePlugin({ protocols: ['websub'] })], + config: resolveConfig(), + fetch: fetchReturning(RSS) + }); + + core.acceptSubscription({ + resourceUrls: [FEED], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub' + }); + + // The default scheduler runs out of band; let the microtask drain. + await new Promise(resolve => setTimeout(resolve, 0)); + + const subs = await store.getSubscriptions(FEED); + expect(subs).toHaveLength(1); + expect(subs[0]).toMatchObject({ + url: 'https://sub.example/listener', + protocol: 'websub' + }); + }); + + it('surfaces a thrown verify→persist task via the error event', async () => { + const events = createEventBus(); + const errors: RssCloudEventMap['error'][] = []; + events.on('error', payload => void errors.push(payload)); + + // No 'websub' plugin registered → subscribe throws UNSUPPORTED_PROTOCOL. + const core = createRssCloudCore({ + store: createInMemoryStore(), + plugins: [], + config: resolveConfig(), + fetch: fetchReturning(RSS), + events + }); + + core.acceptSubscription({ + resourceUrls: [FEED], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub' + }); + + await new Promise(resolve => setTimeout(resolve, 0)); + + expect(errors).toHaveLength(1); + expect(errors[0]?.scope).toBe('websub-verification'); + expect(errors[0]?.error).toBeInstanceOf(Error); + }); + + it('coerces a non-Error rejection into an Error on the error event', async () => { + const base = createInMemoryStore(); + // A misbehaving store that rejects the success-path write (1 sub) with a + // non-Error value; the empty pre-ping write (0 subs) still succeeds. + const store: Store = { + ...base, + putSubscriptions: async (feedUrl, subscriptions) => { + if (subscriptions.length > 0) { + throw 'store exploded'; + } + await base.putSubscriptions(feedUrl, subscriptions); + } + }; + const events = createEventBus(); + const errors: RssCloudEventMap['error'][] = []; + events.on('error', payload => void errors.push(payload)); + + const core = createRssCloudCore({ + store, + plugins: [makePlugin({ protocols: ['websub'] })], + config: resolveConfig(), + fetch: fetchReturning(RSS), + events + }); + + core.acceptSubscription({ + resourceUrls: [FEED], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub' + }); + + await new Promise(resolve => setTimeout(resolve, 0)); + + expect(errors).toHaveLength(1); + expect(errors[0]?.error).toBeInstanceOf(Error); + expect(errors[0]?.error.message).toBe('store exploded'); + }); +}); diff --git a/packages/core/src/engine/create-core.ts b/packages/core/src/engine/create-core.ts index 1cf995e..92927e2 100644 --- a/packages/core/src/engine/create-core.ts +++ b/packages/core/src/engine/create-core.ts @@ -20,6 +20,7 @@ import type { ResourcePayload, ProtocolPlugin } from './plugin.js'; import type { Protocol } from './protocol.js'; import type { Resource } from './resource.js'; import type { Subscription } from './subscription.js'; +import { createInProcessVerificationScheduler } from './verification-scheduler.js'; import type { FeedEntry, Store } from '../store/store.js'; import type { RssCloudCore, @@ -69,6 +70,18 @@ export function createRssCloudCore( const events = options.events ?? createEventBus(); const doFetch = options.fetch ?? fetch; const now = options.now ?? (() => new Date()); + const scheduler = + options.scheduler ?? + createInProcessVerificationScheduler({ + onError: error => + events.emit('error', { + scope: 'websub-verification', + error: + error instanceof Error + ? error + : new Error(String(error)) + }) + }); const feedParser = options.feedParser ?? createDefaultFeedParser({ maxResourceSize: config.maxResourceSize }); @@ -398,6 +411,17 @@ export function createRssCloudCore( }; } + function acceptSubscription(req: SubscribeRequest): void { + // A new caller of the unchanged `subscribe`: the scheduler runs the + // verify→persist out of band so the front door can answer `202` first. + // `subscribe` persists only after `verify` succeeds and records nothing + // on a refusal, so no extra failure handling is needed here — only a + // genuine exception reaches the scheduler's onError. + scheduler.schedule(async () => { + await subscribe(req); + }); + } + async function unsubscribe( req: UnsubscribeRequest ): Promise { @@ -451,6 +475,7 @@ export function createRssCloudCore( return { subscribe, + acceptSubscription, unsubscribe, ping, events, diff --git a/packages/core/src/engine/verification-scheduler.test.ts b/packages/core/src/engine/verification-scheduler.test.ts new file mode 100644 index 0000000..3dc631b --- /dev/null +++ b/packages/core/src/engine/verification-scheduler.test.ts @@ -0,0 +1,40 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { createInProcessVerificationScheduler } from './verification-scheduler.js'; + +afterEach(() => { + vi.useRealTimers(); +}); + +describe('createInProcessVerificationScheduler', () => { + it('runs the scheduled task out of band, after schedule() returns', async () => { + const order: string[] = []; + const scheduler = createInProcessVerificationScheduler({ + onError: () => undefined + }); + + scheduler.schedule(async () => { + order.push('task'); + }); + order.push('after-schedule'); + + await new Promise(resolve => setTimeout(resolve, 0)); + + expect(order).toEqual(['after-schedule', 'task']); + }); + + it('routes a rejected task to onError instead of letting it go unhandled', async () => { + const seen: unknown[] = []; + const scheduler = createInProcessVerificationScheduler({ + onError: error => seen.push(error) + }); + + const boom = new Error('boom'); + scheduler.schedule(async () => { + throw boom; + }); + + await new Promise(resolve => setTimeout(resolve, 0)); + + expect(seen).toEqual([boom]); + }); +}); diff --git a/packages/core/src/engine/verification-scheduler.ts b/packages/core/src/engine/verification-scheduler.ts new file mode 100644 index 0000000..fd467a6 --- /dev/null +++ b/packages/core/src/engine/verification-scheduler.ts @@ -0,0 +1,38 @@ +/** + * The seam behind WebSub's async-`202` accept. After the hub validates a request + * and answers `202`, the verify→persist work runs out of band through a + * scheduler. The default is in-process and best-effort (one attempt; a rejected + * task is surfaced, not retried; a restart drops anything in flight). A future + * persisted queue + retry can satisfy this same interface without touching the + * dispatcher, the plugin's `verify`, or the express factory. See ADR-0002. + */ +export interface VerificationScheduler { + /** + * Enqueue a verify→persist task. Must return immediately without awaiting the + * task, and must not throw — a rejected task is the scheduler's to absorb. + */ + schedule(task: () => Promise): void; +} + +/** Construction-time dependencies for the in-process scheduler. */ +export interface InProcessVerificationSchedulerOptions { + /** Surfaces a task that rejected (the composition root logs/emits it). */ + onError: (error: unknown) => void; +} + +/** + * The default {@link VerificationScheduler}: runs each task on the microtask + * queue so the caller's `202` is sent first, and routes a rejection to + * `onError` so it never becomes an unhandled rejection. + */ +export function createInProcessVerificationScheduler( + options: InProcessVerificationSchedulerOptions +): VerificationScheduler { + return { + schedule(task) { + queueMicrotask(() => { + void task().catch(options.onError); + }); + } + }; +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 5962985..47c4e12 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -2,6 +2,11 @@ export const version = '0.0.0'; // Implementations export { createRssCloudCore } from './engine/create-core.js'; +export { + createInProcessVerificationScheduler, + type VerificationScheduler, + type InProcessVerificationSchedulerOptions +} from './engine/verification-scheduler.js'; export { DEFAULT_CONFIG, resolveConfig } from './config.js'; export { createEventBus } from './events.js'; export { RssCloudError } from './errors.js'; From 693e25e4829825376d16d89acd6264b3e6d0f148 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 11:18:48 -0500 Subject: [PATCH 06/35] feat: wire the WebSub subscribe front door (core dispatcher + express) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add createWebSubDispatcher in core: parse the hub.* body and, on a valid subscribe, hand the built request to core.acceptSubscription and answer 202; a malformed body is 400. Add the thin express websub({ core }) factory mirroring ping/pleaseNotify — it parses the urlencoded body and copies the dispatcher's status onto the reply, with the hub.* logic owned by core. Export both, plus createWebSubProtocolPlugin, from their package indexes. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 3 +- packages/core/src/index.ts | 10 ++++ .../src/protocols/websub-dispatcher.test.ts | 43 +++++++++++++- .../core/src/protocols/websub-dispatcher.ts | 39 +++++++++++++ packages/express/src/index.ts | 4 ++ .../express/src/websub-middleware.test.ts | 58 +++++++++++++++++++ packages/express/src/websub-middleware.ts | 25 ++++++++ 7 files changed, 180 insertions(+), 2 deletions(-) create mode 100644 packages/express/src/websub-middleware.test.ts create mode 100644 packages/express/src/websub-middleware.ts diff --git a/TODO.md b/TODO.md index c7351df..75554ab 100644 --- a/TODO.md +++ b/TODO.md @@ -182,10 +182,11 @@ Flows that must have an e2e (happy path + the ★ negatives): (→ `engine/verification-scheduler.ts`; default scheduler routes a thrown task to the `error` event, scope `websub-verification`. Pre-ping-on-subscribe kept for now — see open question.) -- [ ] **S1.4** core `websub-dispatcher` ↔ express `websub({ core })` factory (same shape +- [x] **S1.4** core `websub-dispatcher` ↔ express `websub({ core })` factory (same shape as `ping`/`pleaseNotify`): parse the form body, `hub.mode=subscribe` → `core.accept…` → `202`, malformed → `4xx`. Mirror `rest-middleware` (thin; dispatcher owns logic). Export from `index.ts`. (No `scheduler`/`hubUrl` args — see architecture notes.) + (→ core `createWebSubDispatcher`; express `websub-middleware.ts`; both exported.) - [ ] **S1.5** Server integration (prerequisite for the S1.6 e2e): **(a)** `apps/server/core.js` — add `createWebSubProtocolPlugin({ hubUrl, requestTimeoutMs })` to the `plugins` array (registers `'websub'`; otherwise diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 47c4e12..ff21eb3 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -32,6 +32,16 @@ export { type RestResponse, type RestResponseFormat } from './protocols/rest-dispatcher.js'; +export { + createWebSubDispatcher, + type WebSubDispatcher, + type WebSubDispatcherOptions, + type WebSubResponse +} from './protocols/websub-dispatcher.js'; +export { + createWebSubProtocolPlugin, + type WebSubProtocolPluginOptions +} from './protocols/websub-plugin.js'; export { createDefaultFeedParser, type DefaultFeedParserOptions diff --git a/packages/core/src/protocols/websub-dispatcher.test.ts b/packages/core/src/protocols/websub-dispatcher.test.ts index 0902861..d23819a 100644 --- a/packages/core/src/protocols/websub-dispatcher.test.ts +++ b/packages/core/src/protocols/websub-dispatcher.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest'; -import { parseSubscribe } from './websub-dispatcher.js'; +import type { SubscribeRequest } from '../engine/dto.js'; +import { createWebSubDispatcher, parseSubscribe } from './websub-dispatcher.js'; describe('parseSubscribe', () => { it('builds a websub SubscribeRequest directly from hub.callback and hub.topic', () => { @@ -66,3 +67,43 @@ describe('parseSubscribe', () => { expect(result).toEqual({ ok: false, status: 400 }); }); }); + +describe('createWebSubDispatcher', () => { + function fakeCore(): { + calls: SubscribeRequest[]; + acceptSubscription: (req: SubscribeRequest) => void; + } { + const calls: SubscribeRequest[] = []; + return { calls, acceptSubscription: req => void calls.push(req) }; + } + + it('accepts a valid subscribe with 202 and hands core the built request', () => { + const core = fakeCore(); + const dispatcher = createWebSubDispatcher({ core }); + + const result = dispatcher.dispatch({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ status: 202 }); + expect(core.calls).toEqual([ + { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub' + } + ]); + }); + + it('returns 400 for a malformed request without accepting anything', () => { + const core = fakeCore(); + const dispatcher = createWebSubDispatcher({ core }); + + const result = dispatcher.dispatch({ 'hub.mode': 'subscribe' }); + + expect(result).toEqual({ status: 400 }); + expect(core.calls).toEqual([]); + }); +}); diff --git a/packages/core/src/protocols/websub-dispatcher.ts b/packages/core/src/protocols/websub-dispatcher.ts index df4e624..da4429c 100644 --- a/packages/core/src/protocols/websub-dispatcher.ts +++ b/packages/core/src/protocols/websub-dispatcher.ts @@ -1,3 +1,4 @@ +import type { RssCloudCore } from '../engine/core.js'; import type { SubscribeRequest } from '../engine/dto.js'; /** @@ -51,3 +52,41 @@ export function parseSubscribe( } }; } + +/** A fully-resolved WebSub HTTP status the front door copies onto its reply. */ +export interface WebSubResponse { + status: number; +} + +/** Construction-time dependencies for the WebSub front door. */ +export interface WebSubDispatcherOptions { + core: Pick; +} + +/** Parsed-body-in, status-out WebSub `hub.*` front door. */ +export interface WebSubDispatcher { + dispatch(body: Record): WebSubResponse; +} + +/** + * Build the WebSub front door. A malformed `hub.*` body is rejected synchronously + * (`400`); a valid subscribe is accepted for async intent verification + * (`202` — see ADR-0002) by handing the built request to + * {@link RssCloudCore.acceptSubscription}. + */ +export function createWebSubDispatcher( + options: WebSubDispatcherOptions +): WebSubDispatcher { + const { core } = options; + + function dispatch(body: Record): WebSubResponse { + const parsed = parseSubscribe(body); + if (!parsed.ok) { + return { status: parsed.status }; + } + core.acceptSubscription(parsed.request); + return { status: 202 }; + } + + return { dispatch }; +} diff --git a/packages/express/src/index.ts b/packages/express/src/index.ts index 3428d8a..2e7b859 100644 --- a/packages/express/src/index.ts +++ b/packages/express/src/index.ts @@ -9,3 +9,7 @@ export { rpc2, type XmlRpcMiddlewareOptions } from './xml-rpc-middleware.js'; +export { + websub, + type WebSubMiddlewareOptions +} from './websub-middleware.js'; diff --git a/packages/express/src/websub-middleware.test.ts b/packages/express/src/websub-middleware.test.ts new file mode 100644 index 0000000..5e1b08c --- /dev/null +++ b/packages/express/src/websub-middleware.test.ts @@ -0,0 +1,58 @@ +import { describe, it, expect } from 'vitest'; +import express from 'express'; +import request from 'supertest'; +import type { RssCloudCore, SubscribeRequest } from '@rsscloud/core'; +import { websub } from './websub-middleware.js'; + +function fakeCore(): { + core: Pick; + accepted: SubscribeRequest[]; +} { + const accepted: SubscribeRequest[] = []; + const core: Pick = { + acceptSubscription(req) { + accepted.push(req); + } + }; + return { core, accepted }; +} + +describe('websub middleware', () => { + it('accepts a valid subscribe with 202 and hands core the built request', async () => { + const fake = fakeCore(); + const app = express(); + app.post('/websub', websub({ core: fake.core })); + + const res = await request(app) + .post('/websub') + .type('form') + .send({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(res.status).toBe(202); + expect(fake.accepted).toEqual([ + { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub' + } + ]); + }); + + it('responds 400 to a malformed hub.* body without accepting anything', async () => { + const fake = fakeCore(); + const app = express(); + app.post('/websub', websub({ core: fake.core })); + + const res = await request(app) + .post('/websub') + .type('form') + .send({ 'hub.mode': 'subscribe' }); + + expect(res.status).toBe(400); + expect(fake.accepted).toEqual([]); + }); +}); diff --git a/packages/express/src/websub-middleware.ts b/packages/express/src/websub-middleware.ts new file mode 100644 index 0000000..71083b8 --- /dev/null +++ b/packages/express/src/websub-middleware.ts @@ -0,0 +1,25 @@ +import express, { type RequestHandler } from 'express'; +import { createWebSubDispatcher, type RssCloudCore } from '@rsscloud/core'; + +/** Construction-time dependencies for the WebSub front-door middleware. */ +export interface WebSubMiddlewareOptions { + core: Pick; +} + +/** Parses the `application/x-www-form-urlencoded` `hub.*` body. */ +const urlencodedParser = express.urlencoded({ extended: false }); + +/** + * Express handler stack for the WebSub hub front door. Thin by design — it + * parses the form body and copies the dispatcher's status onto the reply; the + * `hub.*` parsing and the accept/`202` decision live in core's + * {@link createWebSubDispatcher}. + */ +export function websub(options: WebSubMiddlewareOptions): RequestHandler[] { + const dispatcher = createWebSubDispatcher({ core: options.core }); + const handler: RequestHandler = (req, res) => { + const result = dispatcher.dispatch(req.body as Record); + res.status(result.status).end(); + }; + return [urlencodedParser, handler]; +} From f37332f79c4a03ff0c98b6b41c990c430d4ad457 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 11:24:00 -0500 Subject: [PATCH 07/35] feat(server): mount the WebSub hub front door Register createWebSubProtocolPlugin in the core composition root so core.subscribe accepts the 'websub' protocol, and mount websub({ core }) at config.webSubPath (default /websub). Add WEBSUB_PATH and HUB_URL config (hubUrl defaults to domain/port/path; consumed once content distribution lands). The plugin gets requestTimeoutMs for now; hubUrl wiring follows with deliver(). Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 4 +++- apps/server/config.js | 15 ++++++++++++--- apps/server/controllers/index.js | 7 ++++++- apps/server/core.js | 8 +++++++- 4 files changed, 28 insertions(+), 6 deletions(-) diff --git a/TODO.md b/TODO.md index 75554ab..75b7149 100644 --- a/TODO.md +++ b/TODO.md @@ -187,7 +187,7 @@ Flows that must have an e2e (happy path + the ★ negatives): → `202`, malformed → `4xx`. Mirror `rest-middleware` (thin; dispatcher owns logic). Export from `index.ts`. (No `scheduler`/`hubUrl` args — see architecture notes.) (→ core `createWebSubDispatcher`; express `websub-middleware.ts`; both exported.) -- [ ] **S1.5** Server integration (prerequisite for the S1.6 e2e): +- [x] **S1.5** Server integration (prerequisite for the S1.6 e2e): **(a)** `apps/server/core.js` — add `createWebSubProtocolPlugin({ hubUrl, requestTimeoutMs })` to the `plugins` array (registers `'websub'`; otherwise `core.subscribe` rejects it). @@ -195,6 +195,8 @@ Flows that must have an e2e (happy path + the ★ negatives): **(c)** `apps/server/config.js` — env for the hub's public base URL (`HUB_URL`, default derived from `DOMAIN`/`PORT`) and mount path (`WEBSUB_PATH`, default `/websub`). (Lease bounds + signature algo are added in Phases 5/3 when their slices need them.) + (Done; `hubUrl` is config-only until S2.1's deliver consumes it — plugin gets + `requestTimeoutMs` for now. Route mounts at `config.webSubPath`.) - [ ] **S1.6** e2e (**establishes the reusable mock subscriber harness** — challenge-echo): POST subscribe → `202`, callback receives the verification GET, then **poll** `/subscriptions.json` (already lists every sub incl. `protocol:'websub'`) until the diff --git a/apps/server/config.js b/apps/server/config.js index 022bade..a7dcf80 100644 --- a/apps/server/config.js +++ b/apps/server/config.js @@ -11,11 +11,18 @@ function getNumericConfig(key, defaultValue) { return value ? parseInt(value, 10) : defaultValue; } +// The hub's public base URL and mount path. The WebSub endpoint mounts at +// webSubPath; hubUrl is the externally-reachable URL advertised to subscribers +// (consumed when content distribution lands), defaulting to domain/port/path. +const domain = getConfig('DOMAIN', 'localhost'); +const port = getNumericConfig('PORT', 5337); +const webSubPath = getConfig('WEBSUB_PATH', '/websub'); + module.exports = { appName: 'rssCloudServer', appVersion: packageJson.version, - domain: getConfig('DOMAIN', 'localhost'), - port: getNumericConfig('PORT', 5337), + domain, + port, maxConsecutiveErrors: getNumericConfig('MAX_CONSECUTIVE_ERRORS', 3), maxResourceSize: getNumericConfig('MAX_RESOURCE_SIZE', 256000), ctSecsResourceExpire: getNumericConfig('CT_SECS_RESOURCE_EXPIRE', 90000), @@ -24,5 +31,7 @@ module.exports = { dataFilePath: getConfig('DATA_FILE_PATH', './data/subscriptions.json'), statsFilePath: getConfig('STATS_FILE_PATH', './data/stats.json'), statsIntervalMs: getNumericConfig('STATS_INTERVAL_MS', 3600000), - feedsChangedWindowDays: getNumericConfig('FEEDS_CHANGED_WINDOW_DAYS', 7) + feedsChangedWindowDays: getNumericConfig('FEEDS_CHANGED_WINDOW_DAYS', 7), + webSubPath, + hubUrl: getConfig('HUB_URL', `http://${domain}:${port}${webSubPath}`) }; diff --git a/apps/server/controllers/index.js b/apps/server/controllers/index.js index d42b04b..34f9e16 100644 --- a/apps/server/controllers/index.js +++ b/apps/server/controllers/index.js @@ -1,9 +1,10 @@ const express = require('express'), + config = require('../config'), { createFeedsOpml } = require('../services/feeds-opml'), { createStats } = require('../services/stats'), { toFeedsJson } = require('../services/feeds-json'), { renderMarkdownDoc } = require('../services/markdown-doc'), - { ping, pleaseNotify, rpc2 } = require('@rsscloud/express'), + { ping, pleaseNotify, rpc2, websub } = require('@rsscloud/express'), { createTestController } = require('./test'); // Render-only pages — identical Accept→render/406 shells, mounted from a table @@ -46,6 +47,10 @@ function createControllers({ core }) { router.post('/pleaseNotify', pleaseNotify({ core })); router.post('/RPC2', rpc2({ core })); + // WebSub hub front door: validates hub.* synchronously and answers 202, + // then verifies subscriber intent out of band (ADR-0002). + router.post(config.webSubPath, websub({ core })); + for (const { path, view } of NEGOTIATED_VIEWS) { router.get(path, (req, res) => { if (req.accepts('html') === 'html') { diff --git a/apps/server/core.js b/apps/server/core.js index f6c4afb..8c20b1e 100644 --- a/apps/server/core.js +++ b/apps/server/core.js @@ -6,6 +6,7 @@ const { createRssCloudCore, createRestProtocolPlugin, createXmlRpcProtocolPlugin, + createWebSubProtocolPlugin, createFileStore, resolveConfig } = require('@rsscloud/core'); @@ -20,9 +21,14 @@ const coreConfig = resolveConfig({ feedsChangedWindowDays: config.feedsChangedWindowDays }); +// Registers the 'websub' protocol so core.subscribe accepts WebSub subscriptions +// (without it, core.subscribe → UNSUPPORTED_PROTOCOL). Content distribution +// (and the hubUrl it needs) lands in a later phase; for now the plugin verifies +// subscriber intent. const plugins = [ createRestProtocolPlugin({ requestTimeoutMs: config.requestTimeout }), - createXmlRpcProtocolPlugin({ requestTimeoutMs: config.requestTimeout }) + createXmlRpcProtocolPlugin({ requestTimeoutMs: config.requestTimeout }), + createWebSubProtocolPlugin({ requestTimeoutMs: config.requestTimeout }) ]; // createFileStore is async, but core.js is required synchronously — the From 0f40e52a7d5afab5c5b75089b22e86e89f716f3f Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 11:28:37 -0500 Subject: [PATCH 08/35] test(e2e): cover the WebSub subscribe handshake end-to-end Add the WebSub subscribe acceptance suite against the running server: a challenge-echoing callback (the existing mock's function responseBody returning req.query['hub.challenge']) is recorded after the async 202, polled via the test API; a refusing callback is never recorded within a bounded timeout; and a malformed hub.* body (missing callback/topic, or an unsupported mode) returns 400. Full suite: 138 e2e passing. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 5 +- apps/e2e/test/websub.js | 139 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 143 insertions(+), 1 deletion(-) create mode 100644 apps/e2e/test/websub.js diff --git a/TODO.md b/TODO.md index 75b7149..01d8742 100644 --- a/TODO.md +++ b/TODO.md @@ -197,12 +197,15 @@ Flows that must have an e2e (happy path + the ★ negatives): (Lease bounds + signature algo are added in Phases 5/3 when their slices need them.) (Done; `hubUrl` is config-only until S2.1's deliver consumes it — plugin gets `requestTimeoutMs` for now. Route mounts at `config.webSubPath`.) -- [ ] **S1.6** e2e (**establishes the reusable mock subscriber harness** — challenge-echo): +- [x] **S1.6** e2e (**establishes the reusable mock subscriber harness** — challenge-echo): POST subscribe → `202`, callback receives the verification GET, then **poll** `/subscriptions.json` (already lists every sub incl. `protocol:'websub'`) until the record appears — verification is async, so the test waits rather than asserting inline; ★ callback refuses to echo → record never appears (bounded timeout); ★ malformed `hub.*` (missing callback/topic, bad mode) → `4xx`. + (→ `apps/e2e/test/websub.js`; challenge-echo via the existing mock's function + `responseBody` (`req.query['hub.challenge']`); polls `storeApi.findSubscription`. + 138 e2e passing.) **Phase 2 — Content distribution via the existing rssCloud ping (THE PAYOFF)** > Proves the primary use case: an rssCloud-only publisher's `/ping` fans content out to diff --git a/apps/e2e/test/websub.js b/apps/e2e/test/websub.js new file mode 100644 index 0000000..a1059d5 --- /dev/null +++ b/apps/e2e/test/websub.js @@ -0,0 +1,139 @@ +const chai = require('chai'), + chaiHttp = require('chai-http'), + expect = chai.expect, + SERVER_URL = process.env.APP_URL || 'http://localhost:5337', + mock = require('./mock'), + storeApi = require('./store-api'); + +chai.use(chaiHttp); + +// Send a WebSub hub request as a urlencoded form body. URLSearchParams keeps the +// dotted hub.* keys literal so the server's body parser sees hub.mode etc. +function hubRequest(params) { + return chai + .request(SERVER_URL) + .post('/websub') + .set('content-type', 'application/x-www-form-urlencoded') + .send(new URLSearchParams(params).toString()); +} + +// Intent verification is async (the hub answers 202, then verifies out of band), +// so the test polls the store until the websub subscription appears or times out. +async function waitForWebSubSubscription( + topicUrl, + { timeoutMs = 5000, intervalMs = 100 } = {} +) { + const deadline = Date.now() + timeoutMs; + for (;;) { + const subscriptions = (await storeApi.findSubscription(topicUrl)) || []; + const websub = subscriptions.find( + subscription => subscription.protocol === 'websub' + ); + if (websub) { + return websub; + } + if (Date.now() >= deadline) { + return null; + } + await new Promise(resolve => setTimeout(resolve, intervalMs)); + } +} + +describe('WebSub subscribe', function() { + before(async function() { + await storeApi.before(); + await mock.before(); + }); + + after(async function() { + await storeApi.after(); + await mock.after(); + }); + + beforeEach(async function() { + await storeApi.beforeEach(); + await mock.beforeEach(); + }); + + afterEach(async function() { + await storeApi.afterEach(); + await mock.afterEach(); + }); + + it('accepts a subscribe, verifies the callback, and records the subscription', async function() { + const feedPath = '/websub-feed.xml', + topicUrl = mock.serverUrl + feedPath, + callbackPath = '/websub-callback', + callbackUrl = mock.serverUrl + callbackPath; + + mock.route('GET', feedPath, 200, ''); + // Challenge-echo: answer the intent-verification GET by echoing hub.challenge. + mock.route('GET', callbackPath, 200, req => req.query['hub.challenge']); + + const res = await hubRequest({ + 'hub.mode': 'subscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl + }); + + expect(res).status(202); + + const subscription = await waitForWebSubSubscription(topicUrl); + expect(subscription, 'websub subscription should be recorded').to.not.be + .null; + expect(subscription.url).to.equal(callbackUrl); + expect(subscription.protocol).to.equal('websub'); + + // The hub performed the intent-verification GET on the callback. + expect(mock.requests.GET) + .property(callbackPath) + .lengthOf(1, `Missing verification GET ${callbackPath}`); + }); + + it('does not record the subscription when the callback refuses to echo', async function() { + const feedPath = '/websub-feed.xml', + topicUrl = mock.serverUrl + feedPath, + callbackPath = '/websub-refuse', + callbackUrl = mock.serverUrl + callbackPath; + + mock.route('GET', feedPath, 200, ''); + // Refuse: answer the verification GET without echoing the challenge. + mock.route('GET', callbackPath, 200, 'not-the-challenge'); + + const res = await hubRequest({ + 'hub.mode': 'subscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl + }); + + // Still 202 — validation is synchronous, verification is not. + expect(res).status(202); + + const subscription = await waitForWebSubSubscription(topicUrl, { + timeoutMs: 2000 + }); + expect( + subscription, + 'subscription must not be recorded without a valid echo' + ).to.be.null; + + // The hub still attempted verification. + expect(mock.requests.GET) + .property(callbackPath) + .lengthOf(1, `Missing verification GET ${callbackPath}`); + }); + + it('rejects a hub.* body missing callback and topic with 400', async function() { + const res = await hubRequest({ 'hub.mode': 'subscribe' }); + expect(res).status(400); + }); + + it('rejects an unsupported hub.mode with 400', async function() { + const res = await hubRequest({ + 'hub.mode': 'publish', + 'hub.callback': mock.serverUrl + '/websub-callback', + 'hub.topic': mock.serverUrl + '/websub-feed.xml' + }); + expect(res).status(400); + }); +}); From 47f143043faa5b98b4a7420e02bc7c1080baf125 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 15:20:12 -0500 Subject: [PATCH 09/35] feat: distribute feed content to WebSub subscribers on fan-out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement the WebSub plugin's deliver(): POST the changed feed body to each subscriber's callback, relaying the origin Content-Type verbatim (falling back to application/octet-stream when absent) and advertising the hub/self Link rels. Delivery follows 3xx redirects like the rssCloud REST notify path; any non-2xx is a failed delivery. The hub's public URL is injected as a createWebSubProtocolPlugin option and wired from config.hubUrl in apps/server. With this in place a single rssCloud ping already fans content out to WebSub subscribers through the engine's existing resource-keyed fan-out — no new publish path needed. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/server/core.js | 11 +- .../core/src/protocols/websub-plugin.test.ts | 133 +++++++++++++++++- packages/core/src/protocols/websub-plugin.ts | 57 ++++++-- 3 files changed, 184 insertions(+), 17 deletions(-) diff --git a/apps/server/core.js b/apps/server/core.js index 8c20b1e..91672cb 100644 --- a/apps/server/core.js +++ b/apps/server/core.js @@ -22,13 +22,16 @@ const coreConfig = resolveConfig({ }); // Registers the 'websub' protocol so core.subscribe accepts WebSub subscriptions -// (without it, core.subscribe → UNSUPPORTED_PROTOCOL). Content distribution -// (and the hubUrl it needs) lands in a later phase; for now the plugin verifies -// subscriber intent. +// (without it, core.subscribe → UNSUPPORTED_PROTOCOL). The plugin verifies +// subscriber intent and, on fan-out, distributes the feed body to WebSub +// callbacks — advertising this hub's public URL in the Link rel="hub" header. const plugins = [ createRestProtocolPlugin({ requestTimeoutMs: config.requestTimeout }), createXmlRpcProtocolPlugin({ requestTimeoutMs: config.requestTimeout }), - createWebSubProtocolPlugin({ requestTimeoutMs: config.requestTimeout }) + createWebSubProtocolPlugin({ + requestTimeoutMs: config.requestTimeout, + hubUrl: config.hubUrl + }) ]; // createFileStore is async, but core.js is required synchronously — the diff --git a/packages/core/src/protocols/websub-plugin.test.ts b/packages/core/src/protocols/websub-plugin.test.ts index 6d3a318..b66d673 100644 --- a/packages/core/src/protocols/websub-plugin.test.ts +++ b/packages/core/src/protocols/websub-plugin.test.ts @@ -1,5 +1,9 @@ import { describe, expect, it } from 'vitest'; -import type { DeliveryContext, VerifyContext } from '../engine/plugin.js'; +import type { + DeliveryContext, + ResourcePayload, + VerifyContext +} from '../engine/plugin.js'; import type { Resource } from '../engine/resource.js'; import type { Subscription } from '../engine/subscription.js'; import { createWebSubProtocolPlugin } from './websub-plugin.js'; @@ -46,12 +50,13 @@ function resource(url: string): Resource { function deliveryContext( callbackUrl: string, - resourceUrl: string + resourceUrl: string, + payload: ResourcePayload = { body: '', contentType: null } ): DeliveryContext { return { subscription: subscription(callbackUrl), resource: resource(resourceUrl), - payload: { body: '', contentType: null } + payload }; } @@ -227,8 +232,126 @@ describe('createWebSubProtocolPlugin protocols', () => { }); describe('createWebSubProtocolPlugin deliver', () => { - it('reports failure since content distribution is not implemented yet (S2.1)', async () => { - const plugin = createWebSubProtocolPlugin(); + it('POSTs the feed body to the callback with the relayed Content-Type and Link rels', async () => { + const calls: { url: string; init: RequestInit | undefined }[] = []; + const fakeFetch = (async (url: string | URL, init?: RequestInit) => { + calls.push({ url: String(url), init }); + return new Response(null, { status: 204 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub' + }); + + const result = await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + { body: 'updated', contentType: 'application/rss+xml' } + ) + ); + + expect(result.ok).toBe(true); + expect(calls).toHaveLength(1); + expect(calls[0]?.url).toBe('https://sub.example/listener'); + expect(calls[0]?.init?.method).toBe('POST'); + expect(calls[0]?.init?.body).toBe('updated'); + + const headers = new Headers(calls[0]?.init?.headers); + expect(headers.get('content-type')).toBe('application/rss+xml'); + expect(headers.get('link')).toBe( + '; rel="hub", ; rel="self"' + ); + }); + + it('falls back to application/octet-stream when the origin sent no Content-Type', async () => { + const calls: { init: RequestInit | undefined }[] = []; + const fakeFetch = (async (_url: string | URL, init?: RequestInit) => { + calls.push({ init }); + return new Response(null, { status: 204 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub' + }); + + const result = await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + { body: 'raw bytes', contentType: null } + ) + ); + + expect(result.ok).toBe(true); + const headers = new Headers(calls[0]?.init?.headers); + expect(headers.get('content-type')).toBe('application/octet-stream'); + }); + + it('follows a 3xx redirect and re-POSTs the body to the new location', async () => { + const calls: { url: string; init: RequestInit | undefined }[] = []; + const fakeFetch = (async (url: string | URL, init?: RequestInit) => { + calls.push({ url: String(url), init }); + if (calls.length === 1) { + return new Response(null, { + status: 302, + headers: { location: 'https://sub.example/moved' } + }); + } + return new Response(null, { status: 204 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub' + }); + + const result = await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + { body: 'updated', contentType: 'application/rss+xml' } + ) + ); + + expect(result.ok).toBe(true); + expect(calls.map(c => c.url)).toEqual([ + 'https://sub.example/listener', + 'https://sub.example/moved' + ]); + expect(calls[1]?.init?.body).toBe('updated'); + }); + + it('reports failure when the callback responds non-2xx', async () => { + const fakeFetch = (async () => + new Response('nope', { status: 404 })) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub' + }); + + const result = await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss' + ) + ); + + expect(result.ok).toBe(false); + expect(result.error).toBeInstanceOf(Error); + }); + + it('reports failure on a 3xx redirect with no Location to follow', async () => { + const fakeFetch = (async () => + new Response(null, { status: 302 })) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub' + }); const result = await plugin.deliver( deliveryContext( diff --git a/packages/core/src/protocols/websub-plugin.ts b/packages/core/src/protocols/websub-plugin.ts index c25aad4..136ed1a 100644 --- a/packages/core/src/protocols/websub-plugin.ts +++ b/packages/core/src/protocols/websub-plugin.ts @@ -1,4 +1,5 @@ import type { + DeliveryContext, DeliveryResult, ProtocolPlugin, VerifyContext @@ -14,6 +15,12 @@ export interface WebSubProtocolPluginOptions { requestTimeoutMs?: number; /** Challenge generator for the intent-verification GET (injectable for tests). */ createChallenge?: () => string; + /** + * The hub's externally-reachable URL, advertised to subscribers in the + * `Link rel="hub"` header on every content distribution. Required for + * `deliver`; a host always injects it (see `apps/server`). + */ + hubUrl?: string; } const WEBSUB_PROTOCOLS: Protocol[] = ['websub']; @@ -42,6 +49,7 @@ export function createWebSubProtocolPlugin( const requestTimeoutMs = options.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS; const createChallenge = options.createChallenge ?? defaultCreateChallenge; + const hubUrl = options.hubUrl; async function verify(ctx: VerifyContext): Promise { const challenge = createChallenge(); @@ -63,14 +71,47 @@ export function createWebSubProtocolPlugin( } } - // Content distribution lands in S2.1; until then delivery reports failure - // rather than throwing (the engine's deliverTo does not catch). The context - // parameter is omitted until the real implementation consumes it. - function deliver(): Promise { - return Promise.resolve({ - ok: false, - error: new Error('WebSub content distribution not implemented') - }); + /** POST the feed body to one callback, following redirects like rssCloud notify. */ + async function distribute( + targetUrl: string, + ctx: DeliveryContext + ): Promise { + const res = await fetchWithTimeout( + doFetch, + requestTimeoutMs, + targetUrl, + { + method: 'POST', + headers: { + 'Content-Type': + ctx.payload.contentType ?? 'application/octet-stream', + Link: `<${hubUrl}>; rel="hub", <${ctx.resource.url}>; rel="self"` + }, + body: ctx.payload.body, + redirect: 'manual' + } + ); + + if (res.status >= 300 && res.status < 400) { + const location = res.headers.get('location'); + if (location) { + await distribute(new URL(location, targetUrl).toString(), ctx); + return; + } + } + + if (!res.ok) { + throw new Error('WebSub content distribution failed'); + } + } + + async function deliver(ctx: DeliveryContext): Promise { + try { + await distribute(ctx.subscription.url, ctx); + return { ok: true }; + } catch (err) { + return { ok: false, error: err as Error }; + } } return { protocols: WEBSUB_PROTOCOLS, verify, deliver }; From 880fd23a11b0da7770113283845046ded8658415 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 15:20:22 -0500 Subject: [PATCH 10/35] test(e2e): prove an rssCloud ping fans out to both protocols MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a cross-protocol fan-out acceptance test: an rssCloud subscriber and a WebSub subscriber share one topic, and a single ordinary rssCloud /ping fires both — the rssCloud sub gets its notify, the WebSub callback gets a POST carrying the changed feed body, relayed Content-Type, and hub/self Link rels. This is the headline "free WebSub for rssCloud publishers" proof; no hub.mode=publish is involved. Extend the shared mock subscriber with content-capture: a catch-all bodyParser.text records raw, non-urlencoded POST bodies (the WebSub delivery) while leaving rssCloud notify bodies parsed as objects. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 10 +++- apps/e2e/test/mock.js | 13 ++++- apps/e2e/test/websub.js | 104 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+), 3 deletions(-) diff --git a/TODO.md b/TODO.md index 01d8742..5f49625 100644 --- a/TODO.md +++ b/TODO.md @@ -210,19 +210,25 @@ Flows that must have an e2e (happy path + the ★ negatives): **Phase 2 — Content distribution via the existing rssCloud ping (THE PAYOFF)** > Proves the primary use case: an rssCloud-only publisher's `/ping` fans content out to > WebSub subscribers. No WebSub publish path — relies on core's resource-keyed fan-out. -- [ ] **S2.1** `websub-plugin.deliver()`: POST `payload.body` to the callback, relaying +- [x] **S2.1** `websub-plugin.deliver()`: POST `payload.body` to the callback, relaying the topic's `Content-Type = payload.contentType` **verbatim** (xml/atom/json/etc. — the hub is content-type-agnostic; `payload.contentType` is `string | null`, so pick a fallback like `application/octet-stream` when the origin sent none), plus `Link: ; rel="hub", ; rel="self"`. No signature yet. Inject `hubUrl`. Unit tests with injected `fetch` (cover the present-and-null content-type branches). -- [ ] **S2.2** e2e (**the killer test** — extends the harness with content-capture): + (Done; `hubUrl` is an optional `createWebSubProtocolPlugin` option, wired from + `config.hubUrl` in `apps/server/core.js`; delivery follows 3xx redirects like + `rest-plugin`. 100% core coverage; 217 core tests passing.) +- [x] **S2.2** e2e (**the killer test** — extends the harness with content-capture): put an rssCloud subscriber **and** a WebSub subscriber on the same topic `T`, then hit the *existing* rssCloud `/ping` for `T` with changed content; assert **both** fire from that single ping — the rssCloud sub gets its notify, the WebSub callback gets a POST carrying the feed body + relayed `Content-Type` + `Link` rels. No `hub.mode=publish` involved — this is the headline "free WebSub for rssCloud publishers" cross-protocol proof. + (Done; `WebSub cross-protocol fan-out` in `apps/e2e/test/websub.js`. Content-capture + added to the shared `mock` via a catch-all `bodyParser.text` that records raw, + non-urlencoded POST bodies without disturbing rssCloud notify parsing. 139 e2e passing.) **Phase 3 — Authenticated distribution (HMAC-SHA256)** - [ ] **S3.1** parse + store `hub.secret` in `details` at subscribe. **S3.2** when diff --git a/apps/e2e/test/mock.js b/apps/e2e/test/mock.js index 083806a..519dcee 100644 --- a/apps/e2e/test/mock.js +++ b/apps/e2e/test/mock.js @@ -4,6 +4,12 @@ const https = require('https'), bodyParser = require('body-parser'), textParser = bodyParser.text({ type: '*/xml' }), urlencodedParser = bodyParser.urlencoded({ extended: false }), + // Content-capture: record the raw body of any POST the urlencoded parser + // skipped (e.g. a WebSub content distribution carrying the feed verbatim). + // body-parser bails out when an earlier parser already set `req._body`, so + // this only fires for non-urlencoded POSTs and leaves rssCloud notify + // bodies (parsed into objects) untouched. + rawBodyParser = bodyParser.text({ type: () => true }), parseRpcRequest = require('./helpers/parse-rpc-request'), querystring = require('querystring'), MOCK_SERVER_DOMAIN = process.env.MOCK_SERVER_DOMAIN, @@ -107,7 +113,12 @@ module.exports = { before: async function() { this.app.post('/RPC2', textParser, rpcController.bind(this)); this.app.get('*', restController.bind(this)); - this.app.post('*', urlencodedParser, restController.bind(this)); + this.app.post( + '*', + urlencodedParser, + rawBodyParser, + restController.bind(this) + ); this.server = await this.app.listen(MOCK_SERVER_PORT); console.log(` → Mock server started on port: ${MOCK_SERVER_PORT}`); diff --git a/apps/e2e/test/websub.js b/apps/e2e/test/websub.js index a1059d5..46494ba 100644 --- a/apps/e2e/test/websub.js +++ b/apps/e2e/test/websub.js @@ -137,3 +137,107 @@ describe('WebSub subscribe', function() { expect(res).status(400); }); }); + +describe('WebSub cross-protocol fan-out', function() { + before(async function() { + await storeApi.before(); + await mock.before(); + }); + + after(async function() { + await storeApi.after(); + await mock.after(); + }); + + beforeEach(async function() { + await storeApi.beforeEach(); + await mock.beforeEach(); + }); + + afterEach(async function() { + await storeApi.afterEach(); + await mock.afterEach(); + }); + + // The headline use case: a publisher who only speaks rssCloud keeps pinging + // as today, and a single /ping fans the changed feed out to BOTH an rssCloud + // subscriber (a notify) and a WebSub subscriber (the feed body) — no + // hub.mode=publish involved. + it('fans one rssCloud ping out to both an rssCloud and a WebSub subscriber', async function() { + const feedPath = '/cross-feed.xml', + topicUrl = mock.serverUrl + feedPath, + websubCallbackPath = '/cross-websub-callback', + websubCallbackUrl = mock.serverUrl + websubCallbackPath, + restNotifyPath = '/cross-rest-notify', + restNotifyUrl = mock.serverUrl + restNotifyPath, + initialFeed = 'version-1', + changedFeed = 'version-2-changed'; + + // The topic feed starts at version 1. + mock.route('GET', feedPath, 200, initialFeed); + // WebSub callback: echo the challenge on the verification GET, and + // accept the content distribution on the POST. + mock.route('GET', websubCallbackPath, 200, req => { + return req.query['hub.challenge']; + }); + mock.route('POST', websubCallbackPath, 200, 'ok'); + // The rssCloud subscriber's notify endpoint. + mock.route('POST', restNotifyPath, 200, 'Thanks for the update! :-)'); + + // Subscribe via WebSub and wait for the async handshake to record it. + // (core pre-pings the topic here, recording version 1's hash; no + // subscribers exist yet, so that pre-ping fans out to no one.) + const subRes = await hubRequest({ + 'hub.mode': 'subscribe', + 'hub.callback': websubCallbackUrl, + 'hub.topic': topicUrl + }); + expect(subRes).status(202); + + const websubSub = await waitForWebSubSubscription(topicUrl); + expect(websubSub, 'websub subscription should be recorded').to.not.be + .null; + + // Add an rssCloud REST subscriber on the SAME topic. + await storeApi.addSubscription( + topicUrl, + false, + restNotifyUrl, + 'http-post' + ); + + // The feed changes to version 2. + mock.route('GET', feedPath, 200, changedFeed); + + // A single, ordinary rssCloud ping for the topic. + const pingRes = await chai + .request(SERVER_URL) + .post('/ping') + .set('content-type', 'application/x-www-form-urlencoded') + .send({ url: topicUrl }); + expect(pingRes).status(200); + + // The rssCloud subscriber received its form-encoded notify. + expect(mock.requests.POST) + .property(restNotifyPath) + .lengthOf(1, `Missing rssCloud notify POST ${restNotifyPath}`); + expect(mock.requests.POST[restNotifyPath][0].body).property( + 'url', + topicUrl + ); + + // The WebSub subscriber received the changed feed body as content + // distribution, with the origin's Content-Type relayed and the hub/self + // Link rels advertised. + expect(mock.requests.POST) + .property(websubCallbackPath) + .lengthOf(1, `Missing WebSub content POST ${websubCallbackPath}`); + const delivery = mock.requests.POST[websubCallbackPath][0]; + expect(delivery.body).to.equal(changedFeed); + expect(delivery.headers['content-type']).to.match(/text\/html/); + const link = delivery.headers['link']; + expect(link, 'Link header').to.be.a('string'); + expect(link).to.include('rel="hub"'); + expect(link).to.include(`<${topicUrl}>; rel="self"`); + }); +}); From 53f3db04fd7fce0b42edb229e3deb6866e179af9 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 15:27:57 -0500 Subject: [PATCH 11/35] feat: sign WebSub deliveries with X-Hub-Signature Authenticate content distribution for subscribers that supply a hub.secret. parseSubscribe carries the secret through as details.secret; the plugin then signs each delivery body with HMAC and adds X-Hub-Signature: =. No secret means no header. The HMAC algorithm is a plugin option (default sha256, names both the digest and the header method prefix), wired from a new WEBSUB_SIGNATURE_ALGO env knob in apps/server. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/server/config.js | 5 +- apps/server/core.js | 3 +- .../src/protocols/websub-dispatcher.test.ts | 19 ++++ .../core/src/protocols/websub-dispatcher.ts | 16 ++-- .../core/src/protocols/websub-plugin.test.ts | 94 ++++++++++++++++++- packages/core/src/protocols/websub-plugin.ts | 28 +++++- 6 files changed, 149 insertions(+), 16 deletions(-) diff --git a/apps/server/config.js b/apps/server/config.js index a7dcf80..05e3409 100644 --- a/apps/server/config.js +++ b/apps/server/config.js @@ -33,5 +33,8 @@ module.exports = { statsIntervalMs: getNumericConfig('STATS_INTERVAL_MS', 3600000), feedsChangedWindowDays: getNumericConfig('FEEDS_CHANGED_WINDOW_DAYS', 7), webSubPath, - hubUrl: getConfig('HUB_URL', `http://${domain}:${port}${webSubPath}`) + hubUrl: getConfig('HUB_URL', `http://${domain}:${port}${webSubPath}`), + // HMAC algorithm for the X-Hub-Signature header on authenticated WebSub + // deliveries (subscribers that supplied a hub.secret). Default sha256. + webSubSignatureAlgo: getConfig('WEBSUB_SIGNATURE_ALGO', 'sha256') }; diff --git a/apps/server/core.js b/apps/server/core.js index 91672cb..9401d5f 100644 --- a/apps/server/core.js +++ b/apps/server/core.js @@ -30,7 +30,8 @@ const plugins = [ createXmlRpcProtocolPlugin({ requestTimeoutMs: config.requestTimeout }), createWebSubProtocolPlugin({ requestTimeoutMs: config.requestTimeout, - hubUrl: config.hubUrl + hubUrl: config.hubUrl, + signatureAlgo: config.webSubSignatureAlgo }) ]; diff --git a/packages/core/src/protocols/websub-dispatcher.test.ts b/packages/core/src/protocols/websub-dispatcher.test.ts index d23819a..89ed812 100644 --- a/packages/core/src/protocols/websub-dispatcher.test.ts +++ b/packages/core/src/protocols/websub-dispatcher.test.ts @@ -66,6 +66,25 @@ describe('parseSubscribe', () => { expect(result).toEqual({ ok: false, status: 400 }); }); + + it('carries a supplied hub.secret through as details.secret', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'http://feed.example/rss', + 'hub.secret': 's3cr3t' + }); + + expect(result).toEqual({ + ok: true, + request: { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example.com/listener', + protocol: 'websub', + details: { secret: 's3cr3t' } + } + }); + }); }); describe('createWebSubDispatcher', () => { diff --git a/packages/core/src/protocols/websub-dispatcher.ts b/packages/core/src/protocols/websub-dispatcher.ts index da4429c..80d091a 100644 --- a/packages/core/src/protocols/websub-dispatcher.ts +++ b/packages/core/src/protocols/websub-dispatcher.ts @@ -43,14 +43,16 @@ export function parseSubscribe( if (typeof topic !== 'string' || topic === '') { return MALFORMED; } - return { - ok: true, - request: { - resourceUrls: [topic], - callbackUrl: callback, - protocol: 'websub' - } + const request: SubscribeRequest = { + resourceUrls: [topic], + callbackUrl: callback, + protocol: 'websub' }; + const secret = body['hub.secret']; + if (typeof secret === 'string') { + request.details = { secret }; + } + return { ok: true, request }; } /** A fully-resolved WebSub HTTP status the front door copies onto its reply. */ diff --git a/packages/core/src/protocols/websub-plugin.test.ts b/packages/core/src/protocols/websub-plugin.test.ts index b66d673..d14e83a 100644 --- a/packages/core/src/protocols/websub-plugin.test.ts +++ b/packages/core/src/protocols/websub-plugin.test.ts @@ -1,3 +1,4 @@ +import { createHmac } from 'node:crypto'; import { describe, expect, it } from 'vitest'; import type { DeliveryContext, @@ -51,10 +52,15 @@ function resource(url: string): Resource { function deliveryContext( callbackUrl: string, resourceUrl: string, - payload: ResourcePayload = { body: '', contentType: null } + payload: ResourcePayload = { body: '', contentType: null }, + details?: Record ): DeliveryContext { + const sub = subscription(callbackUrl); + if (details !== undefined) { + sub.details = details; + } return { - subscription: subscription(callbackUrl), + subscription: sub, resource: resource(resourceUrl), payload }; @@ -344,6 +350,90 @@ describe('createWebSubProtocolPlugin deliver', () => { expect(result.error).toBeInstanceOf(Error); }); + it('signs the delivery with X-Hub-Signature when the subscription has a secret', async () => { + const calls: { init: RequestInit | undefined }[] = []; + const fakeFetch = (async (_url: string | URL, init?: RequestInit) => { + calls.push({ init }); + return new Response(null, { status: 204 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub' + }); + + const body = 'signed'; + const result = await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + { body, contentType: 'application/rss+xml' }, + { secret: 'top-secret' } + ) + ); + + expect(result.ok).toBe(true); + const headers = new Headers(calls[0]?.init?.headers); + const expected = + 'sha256=' + + createHmac('sha256', 'top-secret').update(body).digest('hex'); + expect(headers.get('x-hub-signature')).toBe(expected); + }); + + it('signs with the configured signatureAlgo when one is supplied', async () => { + const calls: { init: RequestInit | undefined }[] = []; + const fakeFetch = (async (_url: string | URL, init?: RequestInit) => { + calls.push({ init }); + return new Response(null, { status: 204 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub', + signatureAlgo: 'sha512' + }); + + const body = 'signed'; + await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + { body, contentType: 'application/rss+xml' }, + { secret: 'top-secret' } + ) + ); + + const headers = new Headers(calls[0]?.init?.headers); + const expected = + 'sha512=' + + createHmac('sha512', 'top-secret').update(body).digest('hex'); + expect(headers.get('x-hub-signature')).toBe(expected); + }); + + it('omits X-Hub-Signature when the subscription has no secret', async () => { + const calls: { init: RequestInit | undefined }[] = []; + const fakeFetch = (async (_url: string | URL, init?: RequestInit) => { + calls.push({ init }); + return new Response(null, { status: 204 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub' + }); + + await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + { body: 'unsigned', contentType: 'application/rss+xml' } + ) + ); + + const headers = new Headers(calls[0]?.init?.headers); + expect(headers.get('x-hub-signature')).toBeNull(); + }); + it('reports failure on a 3xx redirect with no Location to follow', async () => { const fakeFetch = (async () => new Response(null, { status: 302 })) as typeof fetch; diff --git a/packages/core/src/protocols/websub-plugin.ts b/packages/core/src/protocols/websub-plugin.ts index 136ed1a..cb0e2e9 100644 --- a/packages/core/src/protocols/websub-plugin.ts +++ b/packages/core/src/protocols/websub-plugin.ts @@ -1,3 +1,4 @@ +import { createHmac } from 'node:crypto'; import type { DeliveryContext, DeliveryResult, @@ -21,6 +22,12 @@ export interface WebSubProtocolPluginOptions { * `deliver`; a host always injects it (see `apps/server`). */ hubUrl?: string; + /** + * HMAC algorithm for the `X-Hub-Signature` header when a subscriber + * supplied a `hub.secret`. Names the digest and the header method prefix + * (`=`). Defaults to `sha256`. + */ + signatureAlgo?: string; } const WEBSUB_PROTOCOLS: Protocol[] = ['websub']; @@ -50,6 +57,7 @@ export function createWebSubProtocolPlugin( options.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS; const createChallenge = options.createChallenge ?? defaultCreateChallenge; const hubUrl = options.hubUrl; + const signatureAlgo = options.signatureAlgo ?? 'sha256'; async function verify(ctx: VerifyContext): Promise { const challenge = createChallenge(); @@ -76,17 +84,27 @@ export function createWebSubProtocolPlugin( targetUrl: string, ctx: DeliveryContext ): Promise { + const headers: Record = { + 'Content-Type': + ctx.payload.contentType ?? 'application/octet-stream', + Link: `<${hubUrl}>; rel="hub", <${ctx.resource.url}>; rel="self"` + }; + + const secret = ctx.subscription.details?.['secret']; + if (typeof secret === 'string') { + const digest = createHmac(signatureAlgo, secret) + .update(ctx.payload.body) + .digest('hex'); + headers['X-Hub-Signature'] = `${signatureAlgo}=${digest}`; + } + const res = await fetchWithTimeout( doFetch, requestTimeoutMs, targetUrl, { method: 'POST', - headers: { - 'Content-Type': - ctx.payload.contentType ?? 'application/octet-stream', - Link: `<${hubUrl}>; rel="hub", <${ctx.resource.url}>; rel="self"` - }, + headers, body: ctx.payload.body, redirect: 'manual' } From 33035aa274a86195fdf7f6f36f5340e5acc2e99f Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 15:28:04 -0500 Subject: [PATCH 12/35] test(e2e): verify X-Hub-Signature over the delivered body MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add an authenticated-distribution suite: subscribe with a hub.secret, fire one rssCloud ping, and recompute HMAC-SHA256(secret, body) over the body the WebSub callback received to confirm it matches X-Hub-Signature. Cover the negative too — no hub.secret means no signature header. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 7 +++- apps/e2e/test/websub.js | 87 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 5f49625..90826cb 100644 --- a/TODO.md +++ b/TODO.md @@ -231,10 +231,15 @@ Flows that must have an e2e (happy path + the ★ negatives): non-urlencoded POST bodies without disturbing rssCloud notify parsing. 139 e2e passing.) **Phase 3 — Authenticated distribution (HMAC-SHA256)** -- [ ] **S3.1** parse + store `hub.secret` in `details` at subscribe. **S3.2** when +- [x] **S3.1** parse + store `hub.secret` in `details` at subscribe. **S3.2** when `details.secret` present, add `X-Hub-Signature: sha256=HMAC(secret, body)`; algorithm a configurable plugin option (default `sha256`); no secret → no header. **S3.3** e2e: subscriber verifies the signature over the rssCloud-ping-delivered body. + (Done; `parseSubscribe` stores `details.secret`; plugin signs via `node:crypto` + `createHmac` keyed by the `signatureAlgo` option (default `sha256`), wired from + `WEBSUB_SIGNATURE_ALGO`/`config.webSubSignatureAlgo`. e2e `WebSub authenticated + distribution` recomputes the HMAC over the received body. 221 core tests, 100% + coverage; 141 e2e passing.) **Phase 4 — Unsubscribe (intent-verified)** - [ ] **S4.1** plugin verify for `hub.mode=unsubscribe` (shared verify keyed by mode). diff --git a/apps/e2e/test/websub.js b/apps/e2e/test/websub.js index 46494ba..6634daf 100644 --- a/apps/e2e/test/websub.js +++ b/apps/e2e/test/websub.js @@ -1,5 +1,6 @@ const chai = require('chai'), chaiHttp = require('chai-http'), + crypto = require('node:crypto'), expect = chai.expect, SERVER_URL = process.env.APP_URL || 'http://localhost:5337', mock = require('./mock'), @@ -241,3 +242,89 @@ describe('WebSub cross-protocol fan-out', function() { expect(link).to.include(`<${topicUrl}>; rel="self"`); }); }); + +describe('WebSub authenticated distribution', function() { + before(async function() { + await storeApi.before(); + await mock.before(); + }); + + after(async function() { + await storeApi.after(); + await mock.after(); + }); + + beforeEach(async function() { + await storeApi.beforeEach(); + await mock.beforeEach(); + }); + + afterEach(async function() { + await storeApi.afterEach(); + await mock.afterEach(); + }); + + // Subscribe via WebSub (optionally with a secret), wait for the async + // handshake, change the feed, then fire one rssCloud ping. Returns the + // captured content-distribution POST so a test can verify its signature. + async function deliverViaPing({ secret }) { + const feedPath = '/auth-feed.xml', + topicUrl = mock.serverUrl + feedPath, + callbackPath = '/auth-websub-callback', + callbackUrl = mock.serverUrl + callbackPath, + changedFeed = 'authenticated-payload'; + + mock.route('GET', feedPath, 200, 'version-1'); + mock.route('GET', callbackPath, 200, req => { + return req.query['hub.challenge']; + }); + mock.route('POST', callbackPath, 200, 'ok'); + + const subRes = await hubRequest({ + 'hub.mode': 'subscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl, + ...(secret ? { 'hub.secret': secret } : {}) + }); + expect(subRes).status(202); + + const websubSub = await waitForWebSubSubscription(topicUrl); + expect(websubSub, 'websub subscription should be recorded').to.not.be + .null; + + mock.route('GET', feedPath, 200, changedFeed); + + const pingRes = await chai + .request(SERVER_URL) + .post('/ping') + .set('content-type', 'application/x-www-form-urlencoded') + .send({ url: topicUrl }); + expect(pingRes).status(200); + + expect(mock.requests.POST) + .property(callbackPath) + .lengthOf(1, `Missing WebSub content POST ${callbackPath}`); + return { delivery: mock.requests.POST[callbackPath][0], changedFeed }; + } + + it('signs the delivered body with X-Hub-Signature when the subscriber supplied a secret', async function() { + const secret = 'shared-websub-secret'; + const { delivery, changedFeed } = await deliverViaPing({ secret }); + + // The subscriber recomputes the HMAC over the body it received. + expect(delivery.body).to.equal(changedFeed); + const expected = + 'sha256=' + + crypto + .createHmac('sha256', secret) + .update(delivery.body) + .digest('hex'); + expect(delivery.headers['x-hub-signature']).to.equal(expected); + }); + + it('sends no X-Hub-Signature when the subscriber supplied no secret', async function() { + const { delivery } = await deliverViaPing({ secret: null }); + + expect(delivery.headers).to.not.have.property('x-hub-signature'); + }); +}); From 31244abffea3f554979eba6390284611be5c7d79 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 16:43:09 -0500 Subject: [PATCH 13/35] feat: intent-verify WebSub unsubscribe before removal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WebSub unsubscribe, like subscribe, must confirm the subscriber's intent before the hub acts — but core.unsubscribe has no verify hook. Add the verified path: - VerifyContext gains an optional `mode`, threaded onto the plugin's challenge GET as hub.mode (defaults to subscribe; rssCloud ignores it). - core.acceptUnsubscription schedules a challenge GET in unsubscribe mode and calls unsubscribe only once confirmed — a no-op when the sub is absent or the callback refuses to echo. - The websub dispatcher branches on hub.mode (subscribe/unsubscribe → 202, anything else → 400) via a shared hub.callback/hub.topic parser; the express factory's core Pick widens to acceptUnsubscription. Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/core/src/engine/core.ts | 7 + packages/core/src/engine/create-core.test.ts | 135 +++++++++++++++++- packages/core/src/engine/create-core.ts | 43 ++++++ packages/core/src/engine/plugin.ts | 6 + .../src/protocols/websub-dispatcher.test.ts | 102 ++++++++++++- .../core/src/protocols/websub-dispatcher.ts | 97 ++++++++++--- .../core/src/protocols/websub-plugin.test.ts | 41 +++++- packages/core/src/protocols/websub-plugin.ts | 2 +- .../express/src/websub-middleware.test.ts | 47 +++++- packages/express/src/websub-middleware.ts | 2 +- 10 files changed, 450 insertions(+), 32 deletions(-) diff --git a/packages/core/src/engine/core.ts b/packages/core/src/engine/core.ts index 0f11ca7..de5759e 100644 --- a/packages/core/src/engine/core.ts +++ b/packages/core/src/engine/core.ts @@ -65,6 +65,13 @@ export interface RssCloudCore { * {@link subscribe} — the synchronous rssCloud path is unchanged. */ acceptSubscription(req: SubscribeRequest): void; + /** + * Accept an unsubscribe for async intent verification (WebSub + * `hub.mode=unsubscribe`): returns immediately and schedules a challenge + * GET, removing the subscription only once the callback confirms intent. + * The verified counterpart to {@link unsubscribe}, which has no verify hook. + */ + acceptUnsubscription(req: UnsubscribeRequest): void; /** Cancel subscriptions. */ unsubscribe(req: UnsubscribeRequest): Promise; /** diff --git a/packages/core/src/engine/create-core.test.ts b/packages/core/src/engine/create-core.test.ts index ada3293..01973f2 100644 --- a/packages/core/src/engine/create-core.test.ts +++ b/packages/core/src/engine/create-core.test.ts @@ -4,7 +4,7 @@ import { resolveConfig } from '../config.js'; import { createEventBus } from '../events.js'; import type { RssCloudEventMap } from '../events.js'; import { createInMemoryStore } from '../store/memory-store.js'; -import type { ProtocolPlugin } from './plugin.js'; +import type { ProtocolPlugin, VerifyContext } from './plugin.js'; import type { Resource } from './resource.js'; import type { Store } from '../store/store.js'; import type { Subscription } from './subscription.js'; @@ -740,6 +740,139 @@ describe('createRssCloudCore unsubscribe', () => { }); }); +describe('createRssCloudCore acceptUnsubscription', () => { + function captureScheduler(): { + tasks: (() => Promise)[]; + schedule: (task: () => Promise) => void; + } { + const tasks: (() => Promise)[] = []; + return { tasks, schedule: task => void tasks.push(task) }; + } + + const CALLBACK = 'https://sub.example/listener'; + + it('schedules a verified unsubscribe that removes the sub on a confirmed intent', async () => { + const store = createInMemoryStore(); + await store.putSubscriptions(FEED, [ + subscription({ url: CALLBACK, protocol: 'websub' }) + ]); + const scheduler = captureScheduler(); + const verify = vi.fn<(ctx: VerifyContext) => Promise>( + async () => undefined + ); + + const core = createRssCloudCore({ + store, + plugins: [makePlugin({ protocols: ['websub'], verify })], + config: resolveConfig(), + fetch: fetchReturning(RSS), + scheduler + }); + + core.acceptUnsubscription({ + resourceUrls: [FEED], + callbackUrl: CALLBACK, + protocol: 'websub' + }); + + // Returns immediately: queued, not run — still subscribed. + expect(scheduler.tasks).toHaveLength(1); + expect(await store.getSubscriptions(FEED)).toHaveLength(1); + + await scheduler.tasks[0]?.(); + + expect(await store.getSubscriptions(FEED)).toEqual([]); + expect(verify).toHaveBeenCalledTimes(1); + expect(verify.mock.calls[0]?.[0]).toMatchObject({ + mode: 'unsubscribe', + resourceUrl: FEED + }); + }); + + it('keeps the subscription when the unsubscribe intent is not confirmed', async () => { + const store = createInMemoryStore(); + await store.putSubscriptions(FEED, [ + subscription({ url: CALLBACK, protocol: 'websub' }) + ]); + const scheduler = captureScheduler(); + + const core = createRssCloudCore({ + store, + plugins: [ + makePlugin({ + protocols: ['websub'], + verify: vi.fn(async () => { + throw new Error('callback did not echo the challenge'); + }) + }) + ], + config: resolveConfig(), + fetch: fetchReturning(RSS), + scheduler + }); + + core.acceptUnsubscription({ + resourceUrls: [FEED], + callbackUrl: CALLBACK, + protocol: 'websub' + }); + + // A refusal is expected, not an error: the task resolves cleanly. + await scheduler.tasks[0]?.(); + + expect(await store.getSubscriptions(FEED)).toHaveLength(1); + }); + + it('does not verify or remove anything when no matching subscription exists', async () => { + const store = createInMemoryStore(); + await store.putSubscriptions(FEED, [ + subscription({ url: 'https://other.example/listener', protocol: 'websub' }) + ]); + const scheduler = captureScheduler(); + const verify = vi.fn(async () => undefined); + + const core = createRssCloudCore({ + store, + plugins: [makePlugin({ protocols: ['websub'], verify })], + config: resolveConfig(), + fetch: fetchReturning(RSS), + scheduler + }); + + core.acceptUnsubscription({ + resourceUrls: [FEED], + callbackUrl: CALLBACK, + protocol: 'websub' + }); + + await scheduler.tasks[0]?.(); + + expect(verify).not.toHaveBeenCalled(); + expect(await store.getSubscriptions(FEED)).toHaveLength(1); + }); + + it('surfaces an error when no plugin is registered for the protocol', async () => { + const store = createInMemoryStore(); + const scheduler = captureScheduler(); + + const core = createRssCloudCore({ + store, + plugins: [makePlugin()], + config: resolveConfig(), + fetch: fetchReturning(RSS), + scheduler + }); + + core.acceptUnsubscription({ + resourceUrls: [FEED], + callbackUrl: CALLBACK, + protocol: 'websub' + }); + + await expect(scheduler.tasks[0]?.()).rejects.toThrow(); + }); +}); + describe('createRssCloudCore initialization', () => { it('runs each plugin init hook once', () => { const init = vi.fn(); diff --git a/packages/core/src/engine/create-core.ts b/packages/core/src/engine/create-core.ts index 92927e2..64fc4d1 100644 --- a/packages/core/src/engine/create-core.ts +++ b/packages/core/src/engine/create-core.ts @@ -422,6 +422,48 @@ export function createRssCloudCore( }); } + function acceptUnsubscription(req: UnsubscribeRequest): void { + // The unsubscribe counterpart to acceptSubscription: the scheduler runs + // the intent-verification challenge GET out of band so the front door + // answers 202 first, removing the subscription only once confirmed. + scheduler.schedule(async () => { + await verifiedUnsubscribe(req); + }); + } + + async function verifiedUnsubscribe(req: UnsubscribeRequest): Promise { + const plugin = pluginByProtocol.get(req.protocol); + if (plugin === undefined) { + throw new RssCloudError( + 'UNSUPPORTED_PROTOCOL', + `No plugin is registered for protocol "${req.protocol}".` + ); + } + + for (const resourceUrl of req.resourceUrls) { + const subscriptions = await store.getSubscriptions(resourceUrl); + const existing = subscriptions.find( + s => s.url === req.callbackUrl && s.protocol === req.protocol + ); + if (existing === undefined) { + continue; + } + try { + await plugin.verify({ + subscription: existing, + resourceUrl, + diffDomain: false, + mode: 'unsubscribe' + }); + } catch { + // Intent not confirmed — leave the subscription in place. + return; + } + } + + await unsubscribe(req); + } + async function unsubscribe( req: UnsubscribeRequest ): Promise { @@ -476,6 +518,7 @@ export function createRssCloudCore( return { subscribe, acceptSubscription, + acceptUnsubscription, unsubscribe, ping, events, diff --git a/packages/core/src/engine/plugin.ts b/packages/core/src/engine/plugin.ts index a6d495c..99eacb1 100644 --- a/packages/core/src/engine/plugin.ts +++ b/packages/core/src/engine/plugin.ts @@ -23,6 +23,12 @@ export interface VerifyContext { subscription: Subscription; resourceUrl: string; diffDomain: boolean; + /** + * Which WebSub intent is being confirmed — sent as `hub.mode` on the + * challenge GET. Absent for the rssCloud handshake (which ignores it) and + * defaults to subscribe semantics. + */ + mode?: 'subscribe' | 'unsubscribe'; } /** Passed to `ProtocolPlugin.deliver` for each fan-out notification. */ diff --git a/packages/core/src/protocols/websub-dispatcher.test.ts b/packages/core/src/protocols/websub-dispatcher.test.ts index 89ed812..67d5979 100644 --- a/packages/core/src/protocols/websub-dispatcher.test.ts +++ b/packages/core/src/protocols/websub-dispatcher.test.ts @@ -1,6 +1,10 @@ import { describe, expect, it } from 'vitest'; -import type { SubscribeRequest } from '../engine/dto.js'; -import { createWebSubDispatcher, parseSubscribe } from './websub-dispatcher.js'; +import type { SubscribeRequest, UnsubscribeRequest } from '../engine/dto.js'; +import { + createWebSubDispatcher, + parseSubscribe, + parseUnsubscribe +} from './websub-dispatcher.js'; describe('parseSubscribe', () => { it('builds a websub SubscribeRequest directly from hub.callback and hub.topic', () => { @@ -87,13 +91,59 @@ describe('parseSubscribe', () => { }); }); +describe('parseUnsubscribe', () => { + it('builds a websub UnsubscribeRequest directly from hub.callback and hub.topic', () => { + const result = parseUnsubscribe({ + 'hub.mode': 'unsubscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ + ok: true, + request: { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example.com/listener', + protocol: 'websub' + } + }); + }); + + it('rejects a body whose mode is not unsubscribe as a 400', () => { + const result = parseUnsubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); + + it('rejects a missing hub.topic as a 400', () => { + const result = parseUnsubscribe({ + 'hub.mode': 'unsubscribe', + 'hub.callback': 'https://sub.example.com/listener' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); +}); + describe('createWebSubDispatcher', () => { function fakeCore(): { calls: SubscribeRequest[]; + unsubscribeCalls: UnsubscribeRequest[]; acceptSubscription: (req: SubscribeRequest) => void; + acceptUnsubscription: (req: UnsubscribeRequest) => void; } { const calls: SubscribeRequest[] = []; - return { calls, acceptSubscription: req => void calls.push(req) }; + const unsubscribeCalls: UnsubscribeRequest[] = []; + return { + calls, + unsubscribeCalls, + acceptSubscription: req => void calls.push(req), + acceptUnsubscription: req => void unsubscribeCalls.push(req) + }; } it('accepts a valid subscribe with 202 and hands core the built request', () => { @@ -125,4 +175,50 @@ describe('createWebSubDispatcher', () => { expect(result).toEqual({ status: 400 }); expect(core.calls).toEqual([]); }); + + it('accepts a valid unsubscribe with 202 and hands core the built request', () => { + const core = fakeCore(); + const dispatcher = createWebSubDispatcher({ core }); + + const result = dispatcher.dispatch({ + 'hub.mode': 'unsubscribe', + 'hub.callback': 'https://sub.example/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ status: 202 }); + expect(core.unsubscribeCalls).toEqual([ + { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub' + } + ]); + expect(core.calls).toEqual([]); + }); + + it('returns 400 for a malformed unsubscribe without accepting anything', () => { + const core = fakeCore(); + const dispatcher = createWebSubDispatcher({ core }); + + const result = dispatcher.dispatch({ 'hub.mode': 'unsubscribe' }); + + expect(result).toEqual({ status: 400 }); + expect(core.unsubscribeCalls).toEqual([]); + }); + + it('returns 400 for an unsupported hub.mode without accepting anything', () => { + const core = fakeCore(); + const dispatcher = createWebSubDispatcher({ core }); + + const result = dispatcher.dispatch({ + 'hub.mode': 'publish', + 'hub.callback': 'https://sub.example/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ status: 400 }); + expect(core.calls).toEqual([]); + expect(core.unsubscribeCalls).toEqual([]); + }); }); diff --git a/packages/core/src/protocols/websub-dispatcher.ts b/packages/core/src/protocols/websub-dispatcher.ts index 80d091a..a9440a8 100644 --- a/packages/core/src/protocols/websub-dispatcher.ts +++ b/packages/core/src/protocols/websub-dispatcher.ts @@ -1,5 +1,5 @@ import type { RssCloudCore } from '../engine/core.js'; -import type { SubscribeRequest } from '../engine/dto.js'; +import type { SubscribeRequest, UnsubscribeRequest } from '../engine/dto.js'; /** * Outcome of parsing a WebSub `hub.*` subscribe request: either a ready-to-drive @@ -9,8 +9,13 @@ export type WebSubParseResult = | { ok: true; request: SubscribeRequest } | { ok: false; status: number }; +/** Outcome of parsing a WebSub `hub.*` unsubscribe request (see {@link WebSubParseResult}). */ +export type WebSubUnsubscribeParseResult = + | { ok: true; request: UnsubscribeRequest } + | { ok: false; status: number }; + /** Any `hub.*` shape the hub can't act on is a malformed request. */ -const MALFORMED: WebSubParseResult = { ok: false, status: 400 }; +const MALFORMED = { ok: false as const, status: 400 }; /** True when `value` parses as an absolute URL (a relative URL throws sans base). */ function isAbsoluteUrl(value: string): boolean { @@ -22,6 +27,25 @@ function isAbsoluteUrl(value: string): boolean { } } +/** + * The two fields every actionable `hub.*` request shares: a valid absolute + * `hub.callback` and a non-empty `hub.topic`. Returns `null` when either is + * malformed. + */ +function parseHubCallbackTopic( + body: Record +): { callback: string; topic: string } | null { + const callback = body['hub.callback']; + if (typeof callback !== 'string' || !isAbsoluteUrl(callback)) { + return null; + } + const topic = body['hub.topic']; + if (typeof topic !== 'string' || topic === '') { + return null; + } + return { callback, topic }; +} + /** * Parse and validate a WebSub subscribe form body (`hub.mode` / `hub.callback` / * `hub.topic`). On success builds a `websub` {@link SubscribeRequest} *directly* @@ -35,17 +59,13 @@ export function parseSubscribe( if (body['hub.mode'] !== 'subscribe') { return MALFORMED; } - const callback = body['hub.callback']; - if (typeof callback !== 'string' || !isAbsoluteUrl(callback)) { - return MALFORMED; - } - const topic = body['hub.topic']; - if (typeof topic !== 'string' || topic === '') { + const parsed = parseHubCallbackTopic(body); + if (parsed === null) { return MALFORMED; } const request: SubscribeRequest = { - resourceUrls: [topic], - callbackUrl: callback, + resourceUrls: [parsed.topic], + callbackUrl: parsed.callback, protocol: 'websub' }; const secret = body['hub.secret']; @@ -55,6 +75,31 @@ export function parseSubscribe( return { ok: true, request }; } +/** + * Parse and validate a WebSub unsubscribe form body. Like {@link parseSubscribe} + * it builds the request directly from `hub.callback`/`hub.topic`; an unsubscribe + * carries no `details` (no secret/lease to renew). + */ +export function parseUnsubscribe( + body: Record +): WebSubUnsubscribeParseResult { + if (body['hub.mode'] !== 'unsubscribe') { + return MALFORMED; + } + const parsed = parseHubCallbackTopic(body); + if (parsed === null) { + return MALFORMED; + } + return { + ok: true, + request: { + resourceUrls: [parsed.topic], + callbackUrl: parsed.callback, + protocol: 'websub' + } + }; +} + /** A fully-resolved WebSub HTTP status the front door copies onto its reply. */ export interface WebSubResponse { status: number; @@ -62,7 +107,7 @@ export interface WebSubResponse { /** Construction-time dependencies for the WebSub front door. */ export interface WebSubDispatcherOptions { - core: Pick; + core: Pick; } /** Parsed-body-in, status-out WebSub `hub.*` front door. */ @@ -71,10 +116,11 @@ export interface WebSubDispatcher { } /** - * Build the WebSub front door. A malformed `hub.*` body is rejected synchronously - * (`400`); a valid subscribe is accepted for async intent verification - * (`202` — see ADR-0002) by handing the built request to - * {@link RssCloudCore.acceptSubscription}. + * Build the WebSub front door. A malformed `hub.*` body (or an unsupported + * `hub.mode`) is rejected synchronously (`400`); a valid subscribe/unsubscribe + * is accepted for async intent verification (`202` — see ADR-0002) by handing + * the built request to {@link RssCloudCore.acceptSubscription} / + * {@link RssCloudCore.acceptUnsubscription}. */ export function createWebSubDispatcher( options: WebSubDispatcherOptions @@ -82,12 +128,23 @@ export function createWebSubDispatcher( const { core } = options; function dispatch(body: Record): WebSubResponse { - const parsed = parseSubscribe(body); - if (!parsed.ok) { - return { status: parsed.status }; + if (body['hub.mode'] === 'subscribe') { + const parsed = parseSubscribe(body); + if (!parsed.ok) { + return { status: parsed.status }; + } + core.acceptSubscription(parsed.request); + return { status: 202 }; + } + if (body['hub.mode'] === 'unsubscribe') { + const parsed = parseUnsubscribe(body); + if (!parsed.ok) { + return { status: parsed.status }; + } + core.acceptUnsubscription(parsed.request); + return { status: 202 }; } - core.acceptSubscription(parsed.request); - return { status: 202 }; + return { status: 400 }; } return { dispatch }; diff --git a/packages/core/src/protocols/websub-plugin.test.ts b/packages/core/src/protocols/websub-plugin.test.ts index d14e83a..5a2c4a9 100644 --- a/packages/core/src/protocols/websub-plugin.test.ts +++ b/packages/core/src/protocols/websub-plugin.test.ts @@ -28,13 +28,18 @@ function subscription(url: string): Subscription { function verifyContext( callbackUrl: string, resourceUrl: string, - diffDomain: boolean + diffDomain: boolean, + mode?: 'subscribe' | 'unsubscribe' ): VerifyContext { - return { + const ctx: VerifyContext = { subscription: subscription(callbackUrl), resourceUrl, diffDomain }; + if (mode !== undefined) { + ctx.mode = mode; + } + return ctx; } function resource(url: string): Resource { @@ -101,6 +106,38 @@ describe('createWebSubProtocolPlugin verify', () => { expect(url.searchParams.get('hub.challenge')).toBe('chal-123'); }); + it('sends hub.mode=unsubscribe when the verify context is for an unsubscribe', async () => { + const calls: string[] = []; + const fakeFetch = (async (url: string | URL) => { + calls.push(String(url)); + const challenge = new URL(String(url)).searchParams.get( + 'hub.challenge' + ); + return new Response(challenge, { status: 200 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + createChallenge: () => 'chal-123' + }); + + await plugin.verify( + verifyContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + true, + 'unsubscribe' + ) + ); + + const url = new URL(calls[0] as string); + expect(url.searchParams.get('hub.mode')).toBe('unsubscribe'); + expect(url.searchParams.get('hub.topic')).toBe( + 'http://feed.example/rss' + ); + expect(url.searchParams.get('hub.challenge')).toBe('chal-123'); + }); + it('rejects when the 2xx response does not echo the exact challenge', async () => { const fakeFetch = (async () => new Response('not-the-challenge', { status: 200 })) as typeof fetch; diff --git a/packages/core/src/protocols/websub-plugin.ts b/packages/core/src/protocols/websub-plugin.ts index cb0e2e9..99a9db1 100644 --- a/packages/core/src/protocols/websub-plugin.ts +++ b/packages/core/src/protocols/websub-plugin.ts @@ -62,7 +62,7 @@ export function createWebSubProtocolPlugin( async function verify(ctx: VerifyContext): Promise { const challenge = createChallenge(); const verifyUrl = new URL(ctx.subscription.url); - verifyUrl.searchParams.set('hub.mode', 'subscribe'); + verifyUrl.searchParams.set('hub.mode', ctx.mode ?? 'subscribe'); verifyUrl.searchParams.set('hub.topic', ctx.resourceUrl); verifyUrl.searchParams.set('hub.challenge', challenge); diff --git a/packages/express/src/websub-middleware.test.ts b/packages/express/src/websub-middleware.test.ts index 5e1b08c..ddb357a 100644 --- a/packages/express/src/websub-middleware.test.ts +++ b/packages/express/src/websub-middleware.test.ts @@ -1,20 +1,34 @@ import { describe, it, expect } from 'vitest'; import express from 'express'; import request from 'supertest'; -import type { RssCloudCore, SubscribeRequest } from '@rsscloud/core'; +import type { + RssCloudCore, + SubscribeRequest, + UnsubscribeRequest +} from '@rsscloud/core'; import { websub } from './websub-middleware.js'; +type WebSubCore = Pick< + RssCloudCore, + 'acceptSubscription' | 'acceptUnsubscription' +>; + function fakeCore(): { - core: Pick; + core: WebSubCore; accepted: SubscribeRequest[]; + unsubscribed: UnsubscribeRequest[]; } { const accepted: SubscribeRequest[] = []; - const core: Pick = { + const unsubscribed: UnsubscribeRequest[] = []; + const core: WebSubCore = { acceptSubscription(req) { accepted.push(req); + }, + acceptUnsubscription(req) { + unsubscribed.push(req); } }; - return { core, accepted }; + return { core, accepted, unsubscribed }; } describe('websub middleware', () => { @@ -42,6 +56,31 @@ describe('websub middleware', () => { ]); }); + it('accepts a valid unsubscribe with 202 and hands core the built request', async () => { + const fake = fakeCore(); + const app = express(); + app.post('/websub', websub({ core: fake.core })); + + const res = await request(app) + .post('/websub') + .type('form') + .send({ + 'hub.mode': 'unsubscribe', + 'hub.callback': 'https://sub.example/listener', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(res.status).toBe(202); + expect(fake.unsubscribed).toEqual([ + { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example/listener', + protocol: 'websub' + } + ]); + expect(fake.accepted).toEqual([]); + }); + it('responds 400 to a malformed hub.* body without accepting anything', async () => { const fake = fakeCore(); const app = express(); diff --git a/packages/express/src/websub-middleware.ts b/packages/express/src/websub-middleware.ts index 71083b8..c0e0c27 100644 --- a/packages/express/src/websub-middleware.ts +++ b/packages/express/src/websub-middleware.ts @@ -3,7 +3,7 @@ import { createWebSubDispatcher, type RssCloudCore } from '@rsscloud/core'; /** Construction-time dependencies for the WebSub front-door middleware. */ export interface WebSubMiddlewareOptions { - core: Pick; + core: Pick; } /** Parses the `application/x-www-form-urlencoded` `hub.*` body. */ From 03af1df76a091057dd078daad905a01e98c98523 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 16:43:16 -0500 Subject: [PATCH 14/35] test(e2e): cover the WebSub unsubscribe handshake Subscribe, then drive hub.mode=unsubscribe: when the callback echoes the unsubscribe-mode challenge the subscription is removed; when it refuses, the subscription survives. Polls the store for removal since verification is async. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 8 ++- apps/e2e/test/websub.js | 119 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index 90826cb..a72e392 100644 --- a/TODO.md +++ b/TODO.md @@ -242,10 +242,16 @@ Flows that must have an e2e (happy path + the ★ negatives): coverage; 141 e2e passing.) **Phase 4 — Unsubscribe (intent-verified)** -- [ ] **S4.1** plugin verify for `hub.mode=unsubscribe` (shared verify keyed by mode). +- [x] **S4.1** plugin verify for `hub.mode=unsubscribe` (shared verify keyed by mode). **S4.2** verified-unsubscribe path: scheduled task verifies intent then `core.unsubscribe` (which has no verify hook today). **S4.3** dispatcher/express branch `hub.mode=unsubscribe` → `202`. **S4.4** e2e unsubscribe handshake. + (Done; `VerifyContext.mode` threads `subscribe`/`unsubscribe` into the plugin's challenge + GET; new `core.acceptUnsubscription` schedules a verified removal (no-op if the sub is + absent or intent is refused); `websub-dispatcher` branches on `hub.mode` via a shared + `parseHubCallbackTopic`, express `core` Pick widened. e2e `WebSub unsubscribe` covers + verified removal + the refuse-echo negative. 232 core + 19 express tests, 100% coverage; + 143 e2e passing.) **Phase 5 — Leases (honor requested, clamped)** - [ ] **S5.1** `RssCloudConfig` lease bounds (default/min/max secs) + resolve defaults. diff --git a/apps/e2e/test/websub.js b/apps/e2e/test/websub.js index 6634daf..16b789f 100644 --- a/apps/e2e/test/websub.js +++ b/apps/e2e/test/websub.js @@ -40,6 +40,28 @@ async function waitForWebSubSubscription( } } +// Unsubscribe is async too (202, then a verification GET, then removal), so the +// test polls until the websub subscription is gone or the timeout lapses. +async function waitForWebSubUnsubscription( + topicUrl, + { timeoutMs = 5000, intervalMs = 100 } = {} +) { + const deadline = Date.now() + timeoutMs; + for (;;) { + const subscriptions = (await storeApi.findSubscription(topicUrl)) || []; + const websub = subscriptions.find( + subscription => subscription.protocol === 'websub' + ); + if (!websub) { + return true; + } + if (Date.now() >= deadline) { + return false; + } + await new Promise(resolve => setTimeout(resolve, intervalMs)); + } +} + describe('WebSub subscribe', function() { before(async function() { await storeApi.before(); @@ -328,3 +350,100 @@ describe('WebSub authenticated distribution', function() { expect(delivery.headers).to.not.have.property('x-hub-signature'); }); }); + +describe('WebSub unsubscribe', function() { + before(async function() { + await storeApi.before(); + await mock.before(); + }); + + after(async function() { + await storeApi.after(); + await mock.after(); + }); + + beforeEach(async function() { + await storeApi.beforeEach(); + await mock.beforeEach(); + }); + + afterEach(async function() { + await storeApi.afterEach(); + await mock.afterEach(); + }); + + // Establish a recorded websub subscription, then return its topic/callback so + // a test can drive the unsubscribe handshake. `echoOnUnsubscribe` toggles + // whether the callback confirms the unsubscribe intent. + async function subscribed({ echoOnUnsubscribe }) { + const feedPath = '/unsub-feed.xml', + topicUrl = mock.serverUrl + feedPath, + callbackPath = '/unsub-callback', + callbackUrl = mock.serverUrl + callbackPath; + + mock.route('GET', feedPath, 200, ''); + // Always echo the subscribe challenge; echo the unsubscribe challenge + // only when the scenario wants the intent confirmed. + mock.route('GET', callbackPath, 200, req => { + if (req.query['hub.mode'] === 'unsubscribe' && !echoOnUnsubscribe) { + return 'refused'; + } + return req.query['hub.challenge']; + }); + + const subRes = await hubRequest({ + 'hub.mode': 'subscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl + }); + expect(subRes).status(202); + + const sub = await waitForWebSubSubscription(topicUrl); + expect(sub, 'websub subscription should be recorded').to.not.be.null; + + return { topicUrl, callbackUrl, callbackPath }; + } + + it('accepts an unsubscribe, verifies intent, and removes the subscription', async function() { + const { topicUrl, callbackUrl, callbackPath } = await subscribed({ + echoOnUnsubscribe: true + }); + + const res = await hubRequest({ + 'hub.mode': 'unsubscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl + }); + expect(res).status(202); + + const removed = await waitForWebSubUnsubscription(topicUrl); + expect(removed, 'subscription should be removed after verified unsubscribe') + .to.be.true; + + // The hub performed an unsubscribe-mode verification GET on the callback. + const unsubscribeVerifications = mock.requests.GET[callbackPath].filter( + req => req.query['hub.mode'] === 'unsubscribe' + ); + expect(unsubscribeVerifications).to.have.lengthOf(1); + }); + + it('does not remove the subscription when the callback refuses to echo', async function() { + const { topicUrl, callbackUrl } = await subscribed({ + echoOnUnsubscribe: false + }); + + const res = await hubRequest({ + 'hub.mode': 'unsubscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl + }); + // Still 202 — validation is synchronous, verification is not. + expect(res).status(202); + + const removed = await waitForWebSubUnsubscription(topicUrl, { + timeoutMs: 2000 + }); + expect(removed, 'subscription must survive an unconfirmed unsubscribe').to + .be.false; + }); +}); From 9b650c204f378095ba7812be6edfc4c6c4ece46c Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 18:42:17 -0500 Subject: [PATCH 15/35] feat: honor WebSub lease requests, clamped to configured bounds Add WebSub lease handling. RssCloudConfig gains webSubLease{Default,Min,Max}Secs; the dispatcher parses hub.lease_seconds into details, and subscribeOne clamps it to [min, max] (or grants the default when omitted), records the chosen value in details.leaseSeconds, and maps it to whenExpires = now + chosen. The chosen lease is threaded through VerifyContext so the plugin echoes hub.lease_seconds on the subscribe challenge GET. removeExpired drops a lapsed lease unchanged. Lease bounds are wired from WEBSUB_LEASE_* env in apps/server. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/server/config.js | 7 +- apps/server/core.js | 5 +- packages/core/src/config.test.ts | 5 +- packages/core/src/config.ts | 11 ++- packages/core/src/engine/create-core.test.ts | 71 +++++++++++++++++++ packages/core/src/engine/create-core.ts | 47 +++++++++++- packages/core/src/engine/plugin.ts | 5 ++ .../src/protocols/websub-dispatcher.test.ts | 57 +++++++++++++++ .../core/src/protocols/websub-dispatcher.ts | 26 ++++++- .../core/src/protocols/websub-plugin.test.ts | 60 +++++++++++++++- packages/core/src/protocols/websub-plugin.ts | 6 ++ 11 files changed, 291 insertions(+), 9 deletions(-) diff --git a/apps/server/config.js b/apps/server/config.js index 05e3409..9ec9d5b 100644 --- a/apps/server/config.js +++ b/apps/server/config.js @@ -36,5 +36,10 @@ module.exports = { hubUrl: getConfig('HUB_URL', `http://${domain}:${port}${webSubPath}`), // HMAC algorithm for the X-Hub-Signature header on authenticated WebSub // deliveries (subscribers that supplied a hub.secret). Default sha256. - webSubSignatureAlgo: getConfig('WEBSUB_SIGNATURE_ALGO', 'sha256') + webSubSignatureAlgo: getConfig('WEBSUB_SIGNATURE_ALGO', 'sha256'), + // WebSub lease bounds (secs): the lease granted when hub.lease_seconds is + // omitted, and the [min, max] a requested lease is clamped to. + webSubLeaseDefaultSecs: getNumericConfig('WEBSUB_LEASE_DEFAULT_SECS', 86400), + webSubLeaseMinSecs: getNumericConfig('WEBSUB_LEASE_MIN_SECS', 300), + webSubLeaseMaxSecs: getNumericConfig('WEBSUB_LEASE_MAX_SECS', 864000) }; diff --git a/apps/server/core.js b/apps/server/core.js index 9401d5f..38489fb 100644 --- a/apps/server/core.js +++ b/apps/server/core.js @@ -18,7 +18,10 @@ const coreConfig = resolveConfig({ maxConsecutiveErrors: config.maxConsecutiveErrors, maxResourceSize: config.maxResourceSize, requestTimeoutMs: config.requestTimeout, - feedsChangedWindowDays: config.feedsChangedWindowDays + feedsChangedWindowDays: config.feedsChangedWindowDays, + webSubLeaseDefaultSecs: config.webSubLeaseDefaultSecs, + webSubLeaseMinSecs: config.webSubLeaseMinSecs, + webSubLeaseMaxSecs: config.webSubLeaseMaxSecs }); // Registers the 'websub' protocol so core.subscribe accepts WebSub subscriptions diff --git a/packages/core/src/config.test.ts b/packages/core/src/config.test.ts index e3d60fe..be20316 100644 --- a/packages/core/src/config.test.ts +++ b/packages/core/src/config.test.ts @@ -9,7 +9,10 @@ describe('resolveConfig', () => { maxConsecutiveErrors: 3, maxResourceSize: 256000, requestTimeoutMs: 4000, - feedsChangedWindowDays: 7 + feedsChangedWindowDays: 7, + webSubLeaseDefaultSecs: 86400, + webSubLeaseMinSecs: 300, + webSubLeaseMaxSecs: 864000 }); }); diff --git a/packages/core/src/config.ts b/packages/core/src/config.ts index acead1f..c08b695 100644 --- a/packages/core/src/config.ts +++ b/packages/core/src/config.ts @@ -16,6 +16,12 @@ export interface RssCloudConfig { requestTimeoutMs: number; /** Window (days) used by stats and expiry housekeeping. */ feedsChangedWindowDays: number; + /** WebSub lease (secs) granted when a subscriber omits `hub.lease_seconds`. */ + webSubLeaseDefaultSecs: number; + /** Lower bound (secs) a requested WebSub lease is clamped up to. */ + webSubLeaseMinSecs: number; + /** Upper bound (secs) a requested WebSub lease is clamped down to. */ + webSubLeaseMaxSecs: number; } /** @@ -34,7 +40,10 @@ export const DEFAULT_CONFIG: RssCloudConfig = { maxConsecutiveErrors: 3, maxResourceSize: 256000, requestTimeoutMs: 4000, - feedsChangedWindowDays: 7 + feedsChangedWindowDays: 7, + webSubLeaseDefaultSecs: 86400, + webSubLeaseMinSecs: 300, + webSubLeaseMaxSecs: 864000 }; /** Fill a partial config with {@link DEFAULT_CONFIG} values. */ diff --git a/packages/core/src/engine/create-core.test.ts b/packages/core/src/engine/create-core.test.ts index 01973f2..9bb907a 100644 --- a/packages/core/src/engine/create-core.test.ts +++ b/packages/core/src/engine/create-core.test.ts @@ -873,6 +873,77 @@ describe('createRssCloudCore acceptUnsubscription', () => { }); }); +describe('createRssCloudCore websub leases', () => { + const NOW = new Date('2026-01-01T00:00:00.000Z'); + const CALLBACK = 'https://sub.example/listener'; + + function leaseCore(verify: ProtocolPlugin['verify']) { + const store = createInMemoryStore(); + const core = createRssCloudCore({ + store, + plugins: [makePlugin({ protocols: ['websub'], verify })], + config: resolveConfig({ + webSubLeaseDefaultSecs: 86400, + webSubLeaseMinSecs: 300, + webSubLeaseMaxSecs: 864000 + }), + fetch: fetchReturning(RSS), + now: () => NOW + }); + return { store, core }; + } + + async function subscribeWebSub(details?: Record) { + const verify = vi.fn<(ctx: VerifyContext) => Promise>( + async () => undefined + ); + const { store, core } = leaseCore(verify); + await core.subscribe({ + resourceUrls: [FEED], + callbackUrl: CALLBACK, + protocol: 'websub', + ...(details ? { details } : {}) + }); + const sub = (await store.getSubscriptions(FEED))[0]; + return { sub, verify }; + } + + it('clamps a too-small requested lease up to the minimum and records it', async () => { + const { sub, verify } = await subscribeWebSub({ leaseSeconds: 5 }); + + expect(sub?.details).toEqual({ leaseSeconds: 300 }); + expect(sub?.whenExpires).toEqual(new Date(NOW.getTime() + 300 * 1000)); + expect(verify.mock.calls[0]?.[0]).toMatchObject({ leaseSeconds: 300 }); + }); + + it('clamps a too-large requested lease down to the maximum', async () => { + const { sub } = await subscribeWebSub({ leaseSeconds: 99999999 }); + + expect(sub?.details).toEqual({ leaseSeconds: 864000 }); + expect(sub?.whenExpires).toEqual( + new Date(NOW.getTime() + 864000 * 1000) + ); + }); + + it('grants the default lease when none is requested', async () => { + const { sub } = await subscribeWebSub(); + + expect(sub?.details).toEqual({ leaseSeconds: 86400 }); + expect(sub?.whenExpires).toEqual( + new Date(NOW.getTime() + 86400 * 1000) + ); + }); + + it('preserves a supplied secret alongside the chosen lease', async () => { + const { sub } = await subscribeWebSub({ + secret: 's3cr3t', + leaseSeconds: 3600 + }); + + expect(sub?.details).toEqual({ secret: 's3cr3t', leaseSeconds: 3600 }); + }); +}); + describe('createRssCloudCore initialization', () => { it('runs each plugin init hook once', () => { const init = vi.fn(); diff --git a/packages/core/src/engine/create-core.ts b/packages/core/src/engine/create-core.ts index 64fc4d1..787625b 100644 --- a/packages/core/src/engine/create-core.ts +++ b/packages/core/src/engine/create-core.ts @@ -16,7 +16,11 @@ import { generateStats as runGenerateStats, removeExpired as runRemoveExpired } from './maintenance.js'; -import type { ResourcePayload, ProtocolPlugin } from './plugin.js'; +import type { + ResourcePayload, + ProtocolPlugin, + VerifyContext +} from './plugin.js'; import type { Protocol } from './protocol.js'; import type { Resource } from './resource.js'; import type { Subscription } from './subscription.js'; @@ -98,6 +102,20 @@ export function createRssCloudCore( return new Date(base.getTime() + config.ctSecsResourceExpire * 1000); } + /** + * Resolve a WebSub lease: the requested `hub.lease_seconds` clamped to the + * configured `[min, max]` bounds, or the default when none was requested. + */ + function clampLease(requested: unknown): number { + if (typeof requested !== 'number') { + return config.webSubLeaseDefaultSecs; + } + return Math.min( + config.webSubLeaseMaxSecs, + Math.max(config.webSubLeaseMinSecs, requested) + ); + } + function newResource(url: string): Resource { return { url, @@ -350,8 +368,28 @@ export function createRssCloudCore( ).slice(); const subscription = upsertSubscription(subscriptions, req); + // WebSub subscriptions carry a lease: the chosen value is recorded in + // details, echoed on the verification GET, and maps to whenExpires. + const leaseSeconds = + req.protocol === 'websub' + ? clampLease(req.details?.['leaseSeconds']) + : undefined; + + const verifyContext: VerifyContext = { + subscription, + resourceUrl, + diffDomain + }; + if (leaseSeconds !== undefined) { + subscription.details = { + ...(subscription.details ?? {}), + leaseSeconds + }; + verifyContext.leaseSeconds = leaseSeconds; + } + try { - await plugin.verify({ subscription, resourceUrl, diffDomain }); + await plugin.verify(verifyContext); } catch { return { resourceUrl, @@ -363,7 +401,10 @@ export function createRssCloudCore( subscription.ctUpdates += 1; subscription.ctConsecutiveErrors = 0; subscription.whenLastUpdate = now(); - subscription.whenExpires = expiryFrom(now()); + subscription.whenExpires = + leaseSeconds !== undefined + ? new Date(now().getTime() + leaseSeconds * 1000) + : expiryFrom(now()); await store.putSubscriptions(resourceUrl, subscriptions); events.emit('subscribe', { diff --git a/packages/core/src/engine/plugin.ts b/packages/core/src/engine/plugin.ts index 99eacb1..4b92a5c 100644 --- a/packages/core/src/engine/plugin.ts +++ b/packages/core/src/engine/plugin.ts @@ -29,6 +29,11 @@ export interface VerifyContext { * defaults to subscribe semantics. */ mode?: 'subscribe' | 'unsubscribe'; + /** + * The chosen WebSub lease (secs) to echo as `hub.lease_seconds` on the + * subscribe challenge GET. Absent for rssCloud and for unsubscribe. + */ + leaseSeconds?: number; } /** Passed to `ProtocolPlugin.deliver` for each fan-out notification. */ diff --git a/packages/core/src/protocols/websub-dispatcher.test.ts b/packages/core/src/protocols/websub-dispatcher.test.ts index 67d5979..f88306d 100644 --- a/packages/core/src/protocols/websub-dispatcher.test.ts +++ b/packages/core/src/protocols/websub-dispatcher.test.ts @@ -89,6 +89,63 @@ describe('parseSubscribe', () => { } }); }); + + it('parses hub.lease_seconds into details.leaseSeconds', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'http://feed.example/rss', + 'hub.lease_seconds': '600' + }); + + expect(result).toEqual({ + ok: true, + request: { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example.com/listener', + protocol: 'websub', + details: { leaseSeconds: 600 } + } + }); + }); + + it('carries both hub.secret and hub.lease_seconds in details', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'http://feed.example/rss', + 'hub.secret': 's3cr3t', + 'hub.lease_seconds': '3600' + }); + + expect(result).toEqual({ + ok: true, + request: { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example.com/listener', + protocol: 'websub', + details: { secret: 's3cr3t', leaseSeconds: 3600 } + } + }); + }); + + it('ignores a non-numeric hub.lease_seconds', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'http://feed.example/rss', + 'hub.lease_seconds': 'soon' + }); + + expect(result).toEqual({ + ok: true, + request: { + resourceUrls: ['http://feed.example/rss'], + callbackUrl: 'https://sub.example.com/listener', + protocol: 'websub' + } + }); + }); }); describe('parseUnsubscribe', () => { diff --git a/packages/core/src/protocols/websub-dispatcher.ts b/packages/core/src/protocols/websub-dispatcher.ts index a9440a8..ff77982 100644 --- a/packages/core/src/protocols/websub-dispatcher.ts +++ b/packages/core/src/protocols/websub-dispatcher.ts @@ -46,6 +46,22 @@ function parseHubCallbackTopic( return { callback, topic }; } +/** + * Parse a `hub.lease_seconds` form value to a positive integer, or `undefined` + * when absent/malformed (the hub then applies its default). Core clamps the + * requested value to the configured bounds. + */ +function parseLeaseSeconds(value: unknown): number | undefined { + if (typeof value !== 'string') { + return undefined; + } + const seconds = Number(value); + if (!Number.isInteger(seconds) || seconds <= 0) { + return undefined; + } + return seconds; +} + /** * Parse and validate a WebSub subscribe form body (`hub.mode` / `hub.callback` / * `hub.topic`). On success builds a `websub` {@link SubscribeRequest} *directly* @@ -68,9 +84,17 @@ export function parseSubscribe( callbackUrl: parsed.callback, protocol: 'websub' }; + const details: Record = {}; const secret = body['hub.secret']; if (typeof secret === 'string') { - request.details = { secret }; + details['secret'] = secret; + } + const leaseSeconds = parseLeaseSeconds(body['hub.lease_seconds']); + if (leaseSeconds !== undefined) { + details['leaseSeconds'] = leaseSeconds; + } + if (Object.keys(details).length > 0) { + request.details = details; } return { ok: true, request }; } diff --git a/packages/core/src/protocols/websub-plugin.test.ts b/packages/core/src/protocols/websub-plugin.test.ts index 5a2c4a9..f427fc4 100644 --- a/packages/core/src/protocols/websub-plugin.test.ts +++ b/packages/core/src/protocols/websub-plugin.test.ts @@ -29,7 +29,8 @@ function verifyContext( callbackUrl: string, resourceUrl: string, diffDomain: boolean, - mode?: 'subscribe' | 'unsubscribe' + mode?: 'subscribe' | 'unsubscribe', + leaseSeconds?: number ): VerifyContext { const ctx: VerifyContext = { subscription: subscription(callbackUrl), @@ -39,6 +40,9 @@ function verifyContext( if (mode !== undefined) { ctx.mode = mode; } + if (leaseSeconds !== undefined) { + ctx.leaseSeconds = leaseSeconds; + } return ctx; } @@ -138,6 +142,60 @@ describe('createWebSubProtocolPlugin verify', () => { expect(url.searchParams.get('hub.challenge')).toBe('chal-123'); }); + it('echoes hub.lease_seconds on the challenge GET when a lease is set', async () => { + const calls: string[] = []; + const fakeFetch = (async (url: string | URL) => { + calls.push(String(url)); + const challenge = new URL(String(url)).searchParams.get( + 'hub.challenge' + ); + return new Response(challenge, { status: 200 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + createChallenge: () => 'chal-123' + }); + + await plugin.verify( + verifyContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + true, + 'subscribe', + 600 + ) + ); + + const url = new URL(calls[0] as string); + expect(url.searchParams.get('hub.lease_seconds')).toBe('600'); + }); + + it('omits hub.lease_seconds when no lease is set', async () => { + const calls: string[] = []; + const fakeFetch = (async (url: string | URL) => { + calls.push(String(url)); + const challenge = new URL(String(url)).searchParams.get( + 'hub.challenge' + ); + return new Response(challenge, { status: 200 }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ fetch: fakeFetch }); + + await plugin.verify( + verifyContext( + 'https://sub.example/listener', + 'http://feed.example/rss', + true + ) + ); + + expect( + new URL(calls[0] as string).searchParams.has('hub.lease_seconds') + ).toBe(false); + }); + it('rejects when the 2xx response does not echo the exact challenge', async () => { const fakeFetch = (async () => new Response('not-the-challenge', { status: 200 })) as typeof fetch; diff --git a/packages/core/src/protocols/websub-plugin.ts b/packages/core/src/protocols/websub-plugin.ts index 99a9db1..92023de 100644 --- a/packages/core/src/protocols/websub-plugin.ts +++ b/packages/core/src/protocols/websub-plugin.ts @@ -65,6 +65,12 @@ export function createWebSubProtocolPlugin( verifyUrl.searchParams.set('hub.mode', ctx.mode ?? 'subscribe'); verifyUrl.searchParams.set('hub.topic', ctx.resourceUrl); verifyUrl.searchParams.set('hub.challenge', challenge); + if (ctx.leaseSeconds !== undefined) { + verifyUrl.searchParams.set( + 'hub.lease_seconds', + String(ctx.leaseSeconds) + ); + } const res = await fetchWithTimeout( doFetch, From 949e4788fff966bbc86399b9bcc0368a0df2199a Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 18:42:23 -0500 Subject: [PATCH 16/35] test(e2e): cover WebSub lease clamping and expiry Subscribe with a below-minimum hub.lease_seconds and assert the chosen lease is clamped up to the bound, recorded in details, and echoed on the verification GET. Separately, force a recorded lease to lapse and confirm removeExpired drops it. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 8 +++- apps/e2e/test/websub.js | 84 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/TODO.md b/TODO.md index a72e392..adab54f 100644 --- a/TODO.md +++ b/TODO.md @@ -254,11 +254,17 @@ Flows that must have an e2e (happy path + the ★ negatives): 143 e2e passing.) **Phase 5 — Leases (honor requested, clamped)** -- [ ] **S5.1** `RssCloudConfig` lease bounds (default/min/max secs) + resolve defaults. +- [x] **S5.1** `RssCloudConfig` lease bounds (default/min/max secs) + resolve defaults. **S5.2** parse `hub.lease_seconds`, clamp, store `details.leaseSeconds`, `whenExpires = now + chosen`; echo the chosen lease in the verification GET (thread the chosen value into `verify`). **S5.3** e2e: requested lease clamped + echoed; expiry via `removeExpired()`. + (Done; `RssCloudConfig` gains `webSubLease{Default,Min,Max}Secs` (default 86400/300/864000, + env `WEBSUB_LEASE_*`); the dispatcher parses `hub.lease_seconds` into `details`, core + clamps it in `subscribeOne`, stores the chosen value, sets `whenExpires = now + chosen`, + and threads it through `VerifyContext.leaseSeconds` so the plugin echoes `hub.lease_seconds`. + e2e `WebSub leases` covers clamp+echo and expiry via `removeExpired`. 241 core tests, + 100% coverage; 145 e2e passing.) **Phase 6 — WebSub-native publish front door (secondary — pure-WebSub publishers)** - [ ] **S6.1** dispatcher/express `hub.mode=publish` (thin: `hub.url`/`hub.topic`) → diff --git a/apps/e2e/test/websub.js b/apps/e2e/test/websub.js index 16b789f..b821a09 100644 --- a/apps/e2e/test/websub.js +++ b/apps/e2e/test/websub.js @@ -2,6 +2,7 @@ const chai = require('chai'), chaiHttp = require('chai-http'), crypto = require('node:crypto'), expect = chai.expect, + getDayjs = require('./helpers/dayjs-wrapper'), SERVER_URL = process.env.APP_URL || 'http://localhost:5337', mock = require('./mock'), storeApi = require('./store-api'); @@ -447,3 +448,86 @@ describe('WebSub unsubscribe', function() { .be.false; }); }); + +describe('WebSub leases', function() { + before(async function() { + await storeApi.before(); + await mock.before(); + }); + + after(async function() { + await storeApi.after(); + await mock.after(); + }); + + beforeEach(async function() { + await storeApi.beforeEach(); + await mock.beforeEach(); + }); + + afterEach(async function() { + await storeApi.afterEach(); + await mock.afterEach(); + }); + + it('clamps the requested lease to the configured bounds and echoes it in the verification GET', async function() { + const feedPath = '/lease-feed.xml', + topicUrl = mock.serverUrl + feedPath, + callbackPath = '/lease-callback', + callbackUrl = mock.serverUrl + callbackPath; + + mock.route('GET', feedPath, 200, ''); + mock.route('GET', callbackPath, 200, req => req.query['hub.challenge']); + + // 5 seconds is below the 300s minimum and is clamped up to it. + const res = await hubRequest({ + 'hub.mode': 'subscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl, + 'hub.lease_seconds': '5' + }); + expect(res).status(202); + + const sub = await waitForWebSubSubscription(topicUrl); + expect(sub, 'websub subscription should be recorded').to.not.be.null; + expect(sub.details).to.have.property('leaseSeconds', 300); + + // The verification GET echoed the chosen (clamped) lease. + const verification = mock.requests.GET[callbackPath][0]; + expect(verification.query['hub.lease_seconds']).to.equal('300'); + }); + + it('drops a lapsed lease on removeExpired', async function() { + const feedPath = '/lease-expire-feed.xml', + topicUrl = mock.serverUrl + feedPath, + callbackPath = '/lease-expire-callback', + callbackUrl = mock.serverUrl + callbackPath; + + mock.route('GET', feedPath, 200, ''); + mock.route('GET', callbackPath, 200, req => req.query['hub.challenge']); + + const res = await hubRequest({ + 'hub.mode': 'subscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl + }); + expect(res).status(202); + + const sub = await waitForWebSubSubscription(topicUrl); + expect(sub, 'websub subscription should be recorded').to.not.be.null; + + // Force the lease to have lapsed, then run expiry housekeeping. + const dayjs = await getDayjs(); + sub.whenExpires = dayjs() + .utc() + .subtract(1, 'hour') + .format(); + await storeApi.updateSubscription(topicUrl, sub); + + await storeApi.removeExpired(); + + const remaining = (await storeApi.findSubscription(topicUrl)) || []; + const stillThere = remaining.find(s => s.protocol === 'websub'); + expect(stillThere, 'lapsed lease should be removed').to.be.undefined; + }); +}); From e15806f15e7180a8a6bcbea64c7d68f5fa671e48 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 19:00:22 -0500 Subject: [PATCH 17/35] feat: accept WebSub-native publish to trigger fan-out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Let a pure-WebSub publisher (no rssCloud ping) notify the hub that a topic changed via hub.mode=publish. The dispatcher parses the topic from hub.url (falling back to hub.topic) and calls a new core.acceptPublish, which — per WebSub §7 — acknowledges immediately (202) and re-fetches the topic out of band, reusing ping's existing fetch→payload→fanOut. A failed fetch is surfaced on the error event (scope websub-publish) rather than thrown. The dispatcher and express factory core Picks widen accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/core/src/engine/core.ts | 6 ++ packages/core/src/engine/create-core.test.ts | 83 ++++++++++++++++++ packages/core/src/engine/create-core.ts | 15 ++++ .../src/protocols/websub-dispatcher.test.ts | 86 ++++++++++++++++++- .../core/src/protocols/websub-dispatcher.ts | 65 ++++++++++++-- .../express/src/websub-middleware.test.ts | 29 ++++++- packages/express/src/websub-middleware.ts | 5 +- 7 files changed, 277 insertions(+), 12 deletions(-) diff --git a/packages/core/src/engine/core.ts b/packages/core/src/engine/core.ts index de5759e..aff5b01 100644 --- a/packages/core/src/engine/core.ts +++ b/packages/core/src/engine/core.ts @@ -74,6 +74,12 @@ export interface RssCloudCore { acceptUnsubscription(req: UnsubscribeRequest): void; /** Cancel subscriptions. */ unsubscribe(req: UnsubscribeRequest): Promise; + /** + * Accept a WebSub-native publish: acknowledge immediately and re-fetch the + * topic out of band, reusing {@link ping}'s fetch→fan-out. The publisher is + * not told the fetch outcome (a failure is surfaced on the error event). + */ + acceptPublish(req: PingRequest): void; /** * Handle a change signal: re-fetch the resource, detect a change, and on a * change fan out to every subscriber via its protocol's plugin. diff --git a/packages/core/src/engine/create-core.test.ts b/packages/core/src/engine/create-core.test.ts index 9bb907a..f8e7fab 100644 --- a/packages/core/src/engine/create-core.test.ts +++ b/packages/core/src/engine/create-core.test.ts @@ -873,6 +873,89 @@ describe('createRssCloudCore acceptUnsubscription', () => { }); }); +describe('createRssCloudCore acceptPublish', () => { + it('re-fetches the topic and fans out to subscribers', async () => { + const store = createInMemoryStore(); + await store.putSubscriptions(FEED, [ + subscription({ protocol: 'websub' }) + ]); + const deliver = vi.fn(async () => ({ ok: true })); + + const core = createRssCloudCore({ + store, + plugins: [deliverPlugin(deliver, ['websub'])], + config: resolveConfig(), + fetch: fetchReturning(RSS) + }); + + const result = core.acceptPublish({ resourceUrl: FEED }); + expect(result).toBeUndefined(); + + // The publish is acknowledged immediately; the fetch runs out of band. + await new Promise(resolve => setTimeout(resolve, 0)); + + expect(deliver).toHaveBeenCalledTimes(1); + }); + + it('routes a failed publish fetch to the error event', async () => { + const events = createEventBus(); + const errors: RssCloudEventMap['error'][] = []; + events.on('error', payload => void errors.push(payload)); + + const core = createRssCloudCore({ + store: createInMemoryStore(), + plugins: [ + deliverPlugin(vi.fn(async () => ({ ok: true })), ['websub']) + ], + config: resolveConfig(), + fetch: fetchReturning('Not Found', 404), + events + }); + + core.acceptPublish({ resourceUrl: FEED }); + await new Promise(resolve => setTimeout(resolve, 0)); + + expect(errors).toHaveLength(1); + expect(errors[0]?.scope).toBe('websub-publish'); + expect(errors[0]?.error).toBeInstanceOf(Error); + }); + + it('coerces a non-Error publish rejection into an Error on the error event', async () => { + const base = createInMemoryStore(); + await base.putSubscriptions(FEED, [subscription({ protocol: 'websub' })]); + // A misbehaving store that rejects the fan-out write with a non-Error. + const store: Store = { + ...base, + putSubscriptions: async (feedUrl, subscriptions) => { + if (subscriptions.length > 0) { + throw 'store exploded'; + } + await base.putSubscriptions(feedUrl, subscriptions); + } + }; + const events = createEventBus(); + const errors: RssCloudEventMap['error'][] = []; + events.on('error', payload => void errors.push(payload)); + + const core = createRssCloudCore({ + store, + plugins: [ + deliverPlugin(vi.fn(async () => ({ ok: true })), ['websub']) + ], + config: resolveConfig(), + fetch: fetchReturning(RSS), + events + }); + + core.acceptPublish({ resourceUrl: FEED }); + await new Promise(resolve => setTimeout(resolve, 0)); + + expect(errors).toHaveLength(1); + expect(errors[0]?.error).toBeInstanceOf(Error); + expect(errors[0]?.error.message).toBe('store exploded'); + }); +}); + describe('createRssCloudCore websub leases', () => { const NOW = new Date('2026-01-01T00:00:00.000Z'); const CALLBACK = 'https://sub.example/listener'; diff --git a/packages/core/src/engine/create-core.ts b/packages/core/src/engine/create-core.ts index 787625b..b869a5c 100644 --- a/packages/core/src/engine/create-core.ts +++ b/packages/core/src/engine/create-core.ts @@ -505,6 +505,20 @@ export function createRssCloudCore( await unsubscribe(req); } + function acceptPublish(req: PingRequest): void { + // A well-formed WebSub publish is acknowledged immediately (202) and the + // topic re-fetched out of band, reusing ping's fetch→payload→fanOut. Per + // the spec the publisher isn't told the fetch outcome, so a failure is + // surfaced on the error event rather than thrown. + void ping(req).catch(error => + events.emit('error', { + scope: 'websub-publish', + error: + error instanceof Error ? error : new Error(String(error)) + }) + ); + } + async function unsubscribe( req: UnsubscribeRequest ): Promise { @@ -560,6 +574,7 @@ export function createRssCloudCore( subscribe, acceptSubscription, acceptUnsubscription, + acceptPublish, unsubscribe, ping, events, diff --git a/packages/core/src/protocols/websub-dispatcher.test.ts b/packages/core/src/protocols/websub-dispatcher.test.ts index f88306d..e803297 100644 --- a/packages/core/src/protocols/websub-dispatcher.test.ts +++ b/packages/core/src/protocols/websub-dispatcher.test.ts @@ -1,9 +1,14 @@ import { describe, expect, it } from 'vitest'; -import type { SubscribeRequest, UnsubscribeRequest } from '../engine/dto.js'; +import type { + PingRequest, + SubscribeRequest, + UnsubscribeRequest +} from '../engine/dto.js'; import { createWebSubDispatcher, parseSubscribe, - parseUnsubscribe + parseUnsubscribe, + parsePublish } from './websub-dispatcher.js'; describe('parseSubscribe', () => { @@ -186,20 +191,66 @@ describe('parseUnsubscribe', () => { }); }); +describe('parsePublish', () => { + it('builds a PingRequest from hub.url', () => { + const result = parsePublish({ + 'hub.mode': 'publish', + 'hub.url': 'http://feed.example/rss' + }); + + expect(result).toEqual({ + ok: true, + request: { resourceUrl: 'http://feed.example/rss' } + }); + }); + + it('falls back to hub.topic when hub.url is absent', () => { + const result = parsePublish({ + 'hub.mode': 'publish', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ + ok: true, + request: { resourceUrl: 'http://feed.example/rss' } + }); + }); + + it('rejects a body whose mode is not publish as a 400', () => { + const result = parsePublish({ + 'hub.mode': 'subscribe', + 'hub.url': 'http://feed.example/rss' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); + + it('rejects a publish missing both hub.url and hub.topic as a 400', () => { + const result = parsePublish({ 'hub.mode': 'publish' }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); +}); + describe('createWebSubDispatcher', () => { function fakeCore(): { calls: SubscribeRequest[]; unsubscribeCalls: UnsubscribeRequest[]; + publishCalls: PingRequest[]; acceptSubscription: (req: SubscribeRequest) => void; acceptUnsubscription: (req: UnsubscribeRequest) => void; + acceptPublish: (req: PingRequest) => void; } { const calls: SubscribeRequest[] = []; const unsubscribeCalls: UnsubscribeRequest[] = []; + const publishCalls: PingRequest[] = []; return { calls, unsubscribeCalls, + publishCalls, acceptSubscription: req => void calls.push(req), - acceptUnsubscription: req => void unsubscribeCalls.push(req) + acceptUnsubscription: req => void unsubscribeCalls.push(req), + acceptPublish: req => void publishCalls.push(req) }; } @@ -264,12 +315,38 @@ describe('createWebSubDispatcher', () => { expect(core.unsubscribeCalls).toEqual([]); }); - it('returns 400 for an unsupported hub.mode without accepting anything', () => { + it('accepts a valid publish with 202 and pings the topic', () => { const core = fakeCore(); const dispatcher = createWebSubDispatcher({ core }); const result = dispatcher.dispatch({ 'hub.mode': 'publish', + 'hub.url': 'http://feed.example/rss' + }); + + expect(result).toEqual({ status: 202 }); + expect(core.publishCalls).toEqual([ + { resourceUrl: 'http://feed.example/rss' } + ]); + expect(core.calls).toEqual([]); + }); + + it('returns 400 for a malformed publish without pinging anything', () => { + const core = fakeCore(); + const dispatcher = createWebSubDispatcher({ core }); + + const result = dispatcher.dispatch({ 'hub.mode': 'publish' }); + + expect(result).toEqual({ status: 400 }); + expect(core.publishCalls).toEqual([]); + }); + + it('returns 400 for an unsupported hub.mode without accepting anything', () => { + const core = fakeCore(); + const dispatcher = createWebSubDispatcher({ core }); + + const result = dispatcher.dispatch({ + 'hub.mode': 'bogus', 'hub.callback': 'https://sub.example/listener', 'hub.topic': 'http://feed.example/rss' }); @@ -277,5 +354,6 @@ describe('createWebSubDispatcher', () => { expect(result).toEqual({ status: 400 }); expect(core.calls).toEqual([]); expect(core.unsubscribeCalls).toEqual([]); + expect(core.publishCalls).toEqual([]); }); }); diff --git a/packages/core/src/protocols/websub-dispatcher.ts b/packages/core/src/protocols/websub-dispatcher.ts index ff77982..b09c613 100644 --- a/packages/core/src/protocols/websub-dispatcher.ts +++ b/packages/core/src/protocols/websub-dispatcher.ts @@ -1,5 +1,9 @@ import type { RssCloudCore } from '../engine/core.js'; -import type { SubscribeRequest, UnsubscribeRequest } from '../engine/dto.js'; +import type { + PingRequest, + SubscribeRequest, + UnsubscribeRequest +} from '../engine/dto.js'; /** * Outcome of parsing a WebSub `hub.*` subscribe request: either a ready-to-drive @@ -14,6 +18,11 @@ export type WebSubUnsubscribeParseResult = | { ok: true; request: UnsubscribeRequest } | { ok: false; status: number }; +/** Outcome of parsing a WebSub `hub.mode=publish` request (see {@link WebSubParseResult}). */ +export type WebSubPublishParseResult = + | { ok: true; request: PingRequest } + | { ok: false; status: number }; + /** Any `hub.*` shape the hub can't act on is a malformed request. */ const MALFORMED = { ok: false as const, status: 400 }; @@ -46,6 +55,23 @@ function parseHubCallbackTopic( return { callback, topic }; } +/** + * The updated topic a publish names: `hub.url` preferred, falling back to + * `hub.topic` for compatibility. Returns `null` when neither is a non-empty + * string. + */ +function publishTopic(body: Record): string | null { + const url = body['hub.url']; + if (typeof url === 'string' && url !== '') { + return url; + } + const topic = body['hub.topic']; + if (typeof topic === 'string' && topic !== '') { + return topic; + } + return null; +} + /** * Parse a `hub.lease_seconds` form value to a positive integer, or `undefined` * when absent/malformed (the hub then applies its default). Core clamps the @@ -124,6 +150,23 @@ export function parseUnsubscribe( }; } +/** + * Parse and validate a WebSub publish form body. The updated topic is named by + * `hub.url` (or `hub.topic` for compatibility); the hub re-fetches it via ping. + */ +export function parsePublish( + body: Record +): WebSubPublishParseResult { + if (body['hub.mode'] !== 'publish') { + return MALFORMED; + } + const resourceUrl = publishTopic(body); + if (resourceUrl === null) { + return MALFORMED; + } + return { ok: true, request: { resourceUrl } }; +} + /** A fully-resolved WebSub HTTP status the front door copies onto its reply. */ export interface WebSubResponse { status: number; @@ -131,7 +174,10 @@ export interface WebSubResponse { /** Construction-time dependencies for the WebSub front door. */ export interface WebSubDispatcherOptions { - core: Pick; + core: Pick< + RssCloudCore, + 'acceptSubscription' | 'acceptUnsubscription' | 'acceptPublish' + >; } /** Parsed-body-in, status-out WebSub `hub.*` front door. */ @@ -142,9 +188,10 @@ export interface WebSubDispatcher { /** * Build the WebSub front door. A malformed `hub.*` body (or an unsupported * `hub.mode`) is rejected synchronously (`400`); a valid subscribe/unsubscribe - * is accepted for async intent verification (`202` — see ADR-0002) by handing - * the built request to {@link RssCloudCore.acceptSubscription} / - * {@link RssCloudCore.acceptUnsubscription}. + * is accepted for async intent verification and a publish for an async topic + * re-fetch (`202` — see ADR-0002) by handing the built request to + * {@link RssCloudCore.acceptSubscription} / {@link RssCloudCore.acceptUnsubscription} + * / {@link RssCloudCore.acceptPublish}. */ export function createWebSubDispatcher( options: WebSubDispatcherOptions @@ -168,6 +215,14 @@ export function createWebSubDispatcher( core.acceptUnsubscription(parsed.request); return { status: 202 }; } + if (body['hub.mode'] === 'publish') { + const parsed = parsePublish(body); + if (!parsed.ok) { + return { status: parsed.status }; + } + core.acceptPublish(parsed.request); + return { status: 202 }; + } return { status: 400 }; } diff --git a/packages/express/src/websub-middleware.test.ts b/packages/express/src/websub-middleware.test.ts index ddb357a..db7a93c 100644 --- a/packages/express/src/websub-middleware.test.ts +++ b/packages/express/src/websub-middleware.test.ts @@ -2,6 +2,7 @@ import { describe, it, expect } from 'vitest'; import express from 'express'; import request from 'supertest'; import type { + PingRequest, RssCloudCore, SubscribeRequest, UnsubscribeRequest @@ -10,25 +11,30 @@ import { websub } from './websub-middleware.js'; type WebSubCore = Pick< RssCloudCore, - 'acceptSubscription' | 'acceptUnsubscription' + 'acceptSubscription' | 'acceptUnsubscription' | 'acceptPublish' >; function fakeCore(): { core: WebSubCore; accepted: SubscribeRequest[]; unsubscribed: UnsubscribeRequest[]; + published: PingRequest[]; } { const accepted: SubscribeRequest[] = []; const unsubscribed: UnsubscribeRequest[] = []; + const published: PingRequest[] = []; const core: WebSubCore = { acceptSubscription(req) { accepted.push(req); }, acceptUnsubscription(req) { unsubscribed.push(req); + }, + acceptPublish(req) { + published.push(req); } }; - return { core, accepted, unsubscribed }; + return { core, accepted, unsubscribed, published }; } describe('websub middleware', () => { @@ -81,6 +87,25 @@ describe('websub middleware', () => { expect(fake.accepted).toEqual([]); }); + it('accepts a valid publish with 202 and hands core the topic', async () => { + const fake = fakeCore(); + const app = express(); + app.post('/websub', websub({ core: fake.core })); + + const res = await request(app) + .post('/websub') + .type('form') + .send({ + 'hub.mode': 'publish', + 'hub.url': 'http://feed.example/rss' + }); + + expect(res.status).toBe(202); + expect(fake.published).toEqual([ + { resourceUrl: 'http://feed.example/rss' } + ]); + }); + it('responds 400 to a malformed hub.* body without accepting anything', async () => { const fake = fakeCore(); const app = express(); diff --git a/packages/express/src/websub-middleware.ts b/packages/express/src/websub-middleware.ts index c0e0c27..bfaab4c 100644 --- a/packages/express/src/websub-middleware.ts +++ b/packages/express/src/websub-middleware.ts @@ -3,7 +3,10 @@ import { createWebSubDispatcher, type RssCloudCore } from '@rsscloud/core'; /** Construction-time dependencies for the WebSub front-door middleware. */ export interface WebSubMiddlewareOptions { - core: Pick; + core: Pick< + RssCloudCore, + 'acceptSubscription' | 'acceptUnsubscription' | 'acceptPublish' + >; } /** Parses the `application/x-www-form-urlencoded` `hub.*` body. */ From a329ad64bbc296ab7802a8de4cb31007e5c1d2da Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 19:00:29 -0500 Subject: [PATCH 18/35] test(e2e): cover WebSub-native publish content distribution A WebSub subscriber subscribes to a topic, then a pure-WebSub publisher POSTs hub.mode=publish for it; poll for the out-of-band delivery and assert the subscriber receives the changed feed body. Also retarget the "unsupported hub.mode" rejection at a bogus mode now that publish is a supported mode. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 8 +++- apps/e2e/test/websub.js | 81 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 87 insertions(+), 2 deletions(-) diff --git a/TODO.md b/TODO.md index adab54f..c0727f8 100644 --- a/TODO.md +++ b/TODO.md @@ -267,10 +267,16 @@ Flows that must have an e2e (happy path + the ★ negatives): 100% coverage; 145 e2e passing.) **Phase 6 — WebSub-native publish front door (secondary — pure-WebSub publishers)** -- [ ] **S6.1** dispatcher/express `hub.mode=publish` (thin: `hub.url`/`hub.topic`) → +- [x] **S6.1** dispatcher/express `hub.mode=publish` (thin: `hub.url`/`hub.topic`) → `core.ping(topic)` → `2xx`/`204`. Lets a publisher with *no* rssCloud ping trigger the same fan-out. Reuses everything from Phase 2. **S6.2** e2e: WebSub publish → WebSub subscriber receives content. + (Done; new `core.acceptPublish` fire-and-forgets `ping` out of band — per WebSub §7 a + well-formed publish is acknowledged `202` and the topic re-fetched async (failures → + `error` event, scope `websub-publish`). Dispatcher branches `hub.mode=publish` via + `parsePublish` (`hub.url`, falling back to `hub.topic`); express/core Picks widened. + e2e `WebSub native publish` polls for the out-of-band delivery. 250 core + 20 express + tests, 100% coverage; 146 e2e passing.) **Phase 7 — Fat pings (secondary — publisher pushes the body)** - [ ] **S7.1** decide + document the (non-standard) fat-ping wire format — topic via diff --git a/apps/e2e/test/websub.js b/apps/e2e/test/websub.js index b821a09..78a83d7 100644 --- a/apps/e2e/test/websub.js +++ b/apps/e2e/test/websub.js @@ -41,6 +41,25 @@ async function waitForWebSubSubscription( } } +// A WebSub publish is acknowledged with 202 and the topic re-fetched out of +// band, so the test polls the mock for the content-distribution POST. +async function waitForDeliveryPost( + callbackPath, + { timeoutMs = 5000, intervalMs = 100 } = {} +) { + const deadline = Date.now() + timeoutMs; + for (;;) { + const posts = mock.requests.POST[callbackPath] || []; + if (posts.length > 0) { + return posts[0]; + } + if (Date.now() >= deadline) { + return null; + } + await new Promise(resolve => setTimeout(resolve, intervalMs)); + } +} + // Unsubscribe is async too (202, then a verification GET, then removal), so the // test polls until the websub subscription is gone or the timeout lapses. async function waitForWebSubUnsubscription( @@ -154,7 +173,7 @@ describe('WebSub subscribe', function() { it('rejects an unsupported hub.mode with 400', async function() { const res = await hubRequest({ - 'hub.mode': 'publish', + 'hub.mode': 'bogus', 'hub.callback': mock.serverUrl + '/websub-callback', 'hub.topic': mock.serverUrl + '/websub-feed.xml' }); @@ -531,3 +550,63 @@ describe('WebSub leases', function() { expect(stillThere, 'lapsed lease should be removed').to.be.undefined; }); }); + +describe('WebSub native publish', function() { + before(async function() { + await storeApi.before(); + await mock.before(); + }); + + after(async function() { + await storeApi.after(); + await mock.after(); + }); + + beforeEach(async function() { + await storeApi.beforeEach(); + await mock.beforeEach(); + }); + + afterEach(async function() { + await storeApi.afterEach(); + await mock.afterEach(); + }); + + // A pure-WebSub publisher (no rssCloud ping) triggers the same fan-out by + // POSTing hub.mode=publish; the hub re-fetches the topic and distributes. + it('distributes content to a WebSub subscriber from a hub.mode=publish', async function() { + const feedPath = '/publish-feed.xml', + topicUrl = mock.serverUrl + feedPath, + callbackPath = '/publish-callback', + callbackUrl = mock.serverUrl + callbackPath, + changedFeed = 'published-update'; + + mock.route('GET', feedPath, 200, 'version-1'); + mock.route('GET', callbackPath, 200, req => req.query['hub.challenge']); + mock.route('POST', callbackPath, 200, 'ok'); + + // Subscribe via WebSub (the pre-ping records version 1's hash). + const subRes = await hubRequest({ + 'hub.mode': 'subscribe', + 'hub.callback': callbackUrl, + 'hub.topic': topicUrl + }); + expect(subRes).status(202); + const sub = await waitForWebSubSubscription(topicUrl); + expect(sub, 'websub subscription should be recorded').to.not.be.null; + + // The feed changes, then a pure-WebSub publisher notifies the hub. + mock.route('GET', feedPath, 200, changedFeed); + const pubRes = await hubRequest({ + 'hub.mode': 'publish', + 'hub.url': topicUrl + }); + expect(pubRes).status(202); + + // The re-fetch + fan-out run out of band, so poll for the delivery. + const delivery = await waitForDeliveryPost(callbackPath); + expect(delivery, 'WebSub subscriber should receive content').to.not.be + .null; + expect(delivery.body).to.equal(changedFeed); + }); +}); From cd0acf6e87d09c23aa36fb4bec3ae645da06f9b9 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Sun, 14 Jun 2026 21:59:44 -0500 Subject: [PATCH 19/35] docs: retire the WebSub TODO; mark fat pings out of scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The WebSub hub is functionally complete (subscribe, content distribution, HMAC signatures, unsubscribe, leases, native publish). Remove TODO.md now that the roadmap is done — durable decisions live in docs/adr, CONTEXT.md, and git history per CLAUDE.md. Annotate CONTEXT.md's Fat ping entry as out of scope: it is non-standard (a PubSubHubbub-era extension with no WebSub wire format), so the hub only ever does thin publishes. The term is kept solely to explain the naming. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTEXT.md | 10 +- TODO.md | 307 ----------------------------------------------------- 2 files changed, 6 insertions(+), 311 deletions(-) delete mode 100644 TODO.md diff --git a/CONTEXT.md b/CONTEXT.md index c9d5ff9..749e2f4 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -180,12 +180,14 @@ only the changed URL, Content distribution sends the content itself — so one r **Ping** can drive both, from the same already-fetched body. _Avoid_: notify (rssCloud's content-free signal), push, broadcast. -**Fat ping**: +**Fat ping** (out of scope — not implemented): A publish in which the **Publisher** POSTs the changed body itself, so the **Hub** distributes it verbatim *without* re-fetching the **Topic**. Non-standard (a PubSubHubbub -0.4 extension), so its wire format is a project decision. Contrast a thin publish -(`hub.mode=publish`), which names only the URL and triggers a re-fetch through `core.ping`. -_Avoid_: publish (a thin publish re-fetches; a Fat ping carries the body), push. +0.4 extension) with no WebSub wire format, so we **deliberately don't implement it** +(decided 2026-06-15): the hub only ever does thin publishes — it names a **Topic** and +re-fetches through `core.ping`, exactly as rssCloud's **Ping** already works. The term is +kept here only to explain why our publish is called "thin." +_Avoid_: using "publish" to mean Fat ping (our publish is always thin); push. **X-Hub-Signature**: The HMAC the **Hub** adds over a **Content distribution** body (`X-Hub-Signature: sha256=…`) diff --git a/TODO.md b/TODO.md deleted file mode 100644 index c0727f8..0000000 --- a/TODO.md +++ /dev/null @@ -1,307 +0,0 @@ -# TODO — rsscloud-server: open work - -Outstanding + future work only. Completed work lives in git history, not here — -that includes the `apps/server` → `@rsscloud/core` migration, the on-disk **v2 -format unification** (disk == domain model; `legacy-store-shape.js` deleted; one-way -legacy importer in `file-store.ts`), the 2026-06 architecture-cleanup passes -across `@rsscloud/core` and `apps/server`, and the shared **`@rsscloud/xml-rpc`** codec -(core builds its `/RPC2` dispatcher on it). The subscriber/publisher client logic lives -in `apps/client` (its `lib/`), not a published package — a real subscriber must host a -notify endpoint, so it's app logic for now. Per CLAUDE.md: build with the `tdd` skill (red-green vertical slices); -Conventional Commits enforced. Architecture decisions are recorded in `docs/adr/`; -domain vocabulary in `CONTEXT.md`. - -## WebSub hub support (bigger — spans core + express) - -Make the server act as a [WebSub](https://www.w3.org/TR/websub/) **hub** (the W3C -successor to PubSubHubbub, rssCloud's cousin). Hub only — `apps/client` already owns the -subscriber/publisher side; the hub never hosts source feeds (publishers point at it via -`` in their own feeds). Needs new protocol logic in `@rsscloud/core` -**and** a new `@rsscloud/express` middleware, plus a content-delivery model the -notify-only REST/XML-RPC plugins don't cover. - -The engine is already primed: `protocol.ts` lists `'websub'`; `Subscription` / -`SubscribeRequest` / `UnsubscribeRequest` carry a `details` bag for protocol extras; -`whenExpires` + `removeExpired()` are where a lease maps; `ProtocolPlugin` has -`verify`/`deliver`; and `DeliveryContext` already carries `payload: ResourcePayload` -(the feed body + content-type captured by `detectChange` on every ping). So the -fan-out machinery is waiting for a plugin — most new code is the WebSub plugin, a -`hub.*` parser/dispatcher, an express factory, the async-accept seam, and wiring. - -### Primary use case — free WebSub for rssCloud publishers - -A publisher already on this server for rssCloud adds `` -to their feed and **keeps pinging exactly as today** (`/ping` / `rssCloud.ping`). Anyone -who subscribes to that feed *via WebSub* then gets full WebSub content distribution — -**the publisher never speaks WebSub and changes nothing but the feed header.** - -This falls out of core's existing design, which is why WebSub belongs *in core*, not the -HTTP edge: `ping()` → `detectChange()` already fetches the feed body and builds `payload` -on **every** ping; `fanOut(resourceUrl, …)` loads **all** subscriptions for the resource -and selects the plugin **per subscription** (`deliverTo`). So one rssCloud ping already -iterates every subscriber of that topic and dispatches each through its own plugin — -an rssCloud sub gets a notify, a `protocol:'websub'` sub gets content distribution, from -the *same* ping and the *same* already-fetched body. The only missing piece is the -WebSub `deliver()` plugin. Consequences: - -- **No new publish path is required for the headline case** — the trigger is the - existing rssCloud ping. The WebSub `hub.mode=publish` front door and fat pings serve - *pure-WebSub* publishers (no rssCloud) and are therefore **secondary** (later phases). -- **Topic identity is the one hard requirement:** a subscriber's `hub.topic` must be the - same URL string the publisher pings (the store keys feed entries by exact resource - URL). Same exactness rssCloud already requires between subscribe-URL and ping-URL — - WebSub just inherits it. URL normalization is out of scope (matches today's behavior). -- **WebSub adds no fetch overhead on ping** — it reuses the body `detectChange` already - fetched; it only adds an extra outbound POST per WebSub subscriber. - -### Decisions (settled — 2026-06-14) - -1. **Intent verification = async `202`.** The hub validates the request synchronously - (→ `4xx` on malformed), returns `202 Accepted`, then performs the `hub.challenge` - GET out of band and records the subscription only on success. -2. **Best-effort now, queue later — behind one seam.** Async ≠ a queue. A single - **verification-dispatch seam** runs the verify+persist task in-process (one attempt; - failures logged; a restart mid-flight drops the pending request — the subscriber - re-subscribes). A persisted-queue + retry implementation later satisfies the *same* - seam (draining on the existing maintenance interval, persisting via the store) with - **no change** to the `hub.*` parser, the plugin's `verify()`, or the express factory. - Captured as an ADR. **The scheduler is additive and WebSub-only:** rssCloud - `pleaseNotify`/`subscribe` stays synchronous (its callers expect an immediate yes/no) - and `ping`/`fanOut`/`deliver` are untouched — it's a brand-new caller of an unchanged - `core.subscribe`, so no existing rssCloud behavior changes. It lives in core (not - express) only so the future persisted queue can reach the store; the in-process - default would work anywhere. -3. **Publish = both.** Accept a thin WebSub publish (`hub.mode=publish`, `hub.url`/ - `hub.topic`) — and keep rssCloud `/ping` — re-fetching the topic and reusing - `core.ping`'s existing fetch→`payload`→`fanOut`. *Also* accept fat pings (publisher - POSTs the body), distributed verbatim without a re-fetch; this adds an optional - pushed-content path to `PingRequest`/`detectChange`. (Fat-ping wire format is - non-standard — see open questions — so it lands last.) -4. **Lease = honor requested, clamped.** Use `hub.lease_seconds` clamped to a - configurable `[min, max]` (default when omitted); store the chosen value in - `details.leaseSeconds`, set `whenExpires = now + chosen`, and echo the chosen value - in the verification GET. `removeExpired()` drops it on lapse, unchanged. -5. **Signature = HMAC-SHA256, configurable.** When a subscriber supplied `hub.secret`, - sign each delivery with `X-Hub-Signature: sha256=…` (algorithm a config knob, default - `sha256`). No `hub.secret` → no signature header. - -### Architecture notes / corrections to the original sketch - -- **WebSub builds `SubscribeRequest` directly — it does *not* reuse - `buildSubscribeRequest`.** That builder exists to assemble a callback from - port/path/domain (`glueUrlParts`, scheme, `diffDomain`) for REST/XML-RPC. WebSub - already arrives with a complete `hub.callback` URL, so the dispatcher sets - `callbackUrl = hub.callback`, `resourceUrls = [hub.topic]`, `protocol = 'websub'`, - `details = { secret?, leaseSeconds }` and skips the builder. (The sketch's hope to - share that seam doesn't pan out.) `buildSubscribeRequest` also gates on - `VALID_PROTOCOLS` (rssCloud only) — leave it as-is. -- **WebSub always verifies intent** (spec mandate), so the plugin's `verify()` ignores - `diffDomain` and always does the challenge GET — never the same-domain test-notify. -- **`core.unsubscribe()` has no verify hook today.** WebSub unsubscribe must *also* be - intent-verified (`hub.mode=unsubscribe` challenge GET) before removal — the scheduled - task verifies, then calls `core.unsubscribe`. -- **`VerifyContext` likely needs the WebSub `mode` and the chosen lease** (to send - `hub.mode` / `hub.lease_seconds` / `hub.topic` on the challenge GET). Thread these - through `VerifyContext` or read them from `subscription.details` — decide in the - verify slice. -- **Public hub URL is a host concern** (per `config.ts`: host concerns excluded from - `RssCloudConfig`). Only the plugin's `deliver()` needs it (for `Link rel="hub"`) — so - inject `hubUrl` (plus signature algo, timeout, challenge generator) as **plugin** - construction options in `apps/server/core.js`. The express factory **and** the - dispatcher take only `{ core }`, exactly like `ping`/`pleaseNotify`/`rpc2`; the - scheduler is a `createRssCloudCore` option (default in-process, injectable for tests), - not an arg of either. Lease bounds *are* protocol-relevant → add them to - `RssCloudConfig` alongside `ctSecsResourceExpire`. - -### Files this will touch - -- **core (new):** `protocols/websub-plugin.ts` (verify + deliver), `protocols/websub-dispatcher.ts` (`hub.*` parse/validate, branch on `hub.mode`, drive the accept seam). -- **core (changed):** the verification-dispatch seam + async-accept entry on the engine; `PingRequest`/`detectChange` optional pushed content (fat ping); verified-unsubscribe path; `RssCloudConfig` lease bounds; `VerifyContext` WebSub fields. -- **express (new):** `websub-middleware.ts` — `websub({ core })` factory (same `{ core }` shape as `ping`/`pleaseNotify`/`rpc2`) delegating to core's `websub-dispatcher`; export from `index.ts`. -- **apps/server (the integration that makes e2e runnable):** `core.js` — add `createWebSubProtocolPlugin({ hubUrl, requestTimeoutMs, signatureAlgo, createChallenge })` to the `plugins` array (registers the `'websub'` protocol; without it `core.subscribe` → `UNSUPPORTED_PROTOCOL`) and feed lease bounds into `resolveConfig`; `controllers/index.js` — `router.post('/websub', websub({ core }))`; `config.js` — new env vars (hub URL, mount path, lease bounds, signature algo). Scheduler defaults inside `createRssCloudCore`, so no extra server wiring. -- **apps/e2e:** mock subscriber callback that echoes `hub.challenge`; handshake/publish/signature suites (copy any new helper into `helpers/`, don't cross the workspace boundary). -- **docs:** ADR for the async/best-effort+seam decision; `CONTEXT.md` vocabulary (Hub, Topic, Callback, Intent verification, Lease, Content distribution, Fat ping, `X-Hub-Signature`). - -### e2e strategy (the TDD outer loop) - -Every new endpoint/flow gets an `apps/e2e` acceptance test **written as the outer red of -its slice** — the HTTP-level test fails first, the core/express units make it green; the -slice isn't done until its e2e passes. e2e drives the running server over `APP_URL`; per -CLAUDE.md, anything new a test needs goes in `apps/e2e/test/helpers/` (copied, **not** -imported across the workspace boundary). - -A reusable **mock WebSub subscriber** (alongside the existing rssCloud mock servers on -8002/8003) is grown incrementally as phases need it: -- **challenge-echo** (Phase 1): answers the intent-verification GET by echoing - `hub.challenge` with `2xx`; a toggle to *refuse* (wrong/absent echo) drives the negatives. -- **content-capture** (Phase 2): records each distribution POST — body, `Content-Type`, - `Link` rels — for assertions. -- **signature-verify** (Phase 3): recomputes `HMAC-SHA256(secret, body)` and checks - `X-Hub-Signature`. - -Flows that must have an e2e (happy path + the ★ negatives): -- **subscribe** → `202`, callback verified, sub recorded; ★ no-echo → **not** recorded; - ★ malformed `hub.*` → `4xx`. -- **cross-protocol fan-out** — one rssCloud `/ping` fires BOTH an rssCloud sub and a - WebSub sub on the same topic (the headline proof; see S2.2). -- **authenticated delivery** — subscriber validates the signature; ★ no `hub.secret` → - no header. -- **unsubscribe** → verified removal; ★ no-echo → **not** removed. -- **leases** — requested value clamped + echoed in the verification GET; expiry via - `removeExpired()`. -- **WebSub-native publish** (`hub.mode=publish`) and **fat ping** each deliver content. - -### Slices (TDD vertical slices, red→green, in order) - -**Phase 0 — Foundations** -- [x] **S0.1** ADR: WebSub hub = async-`202` intent verification via an in-process - best-effort `VerificationScheduler` seam; persisted queue + retry is a future refactor - behind the same seam. Record the lease + signature decisions too. - (→ `docs/adr/0002-websub-async-intent-verification-seam.md`) -- [x] **S0.2** `CONTEXT.md`: add the WebSub vocabulary above (tie "Hub" to the existing - Hub-end note; distinguish **Topic** from **Resource**, **Callback** from - **Subscription.url**). - -**Phase 1 — Subscribe happy path (async handshake; no secret/lease/content yet)** -- [x] **S1.1** `websub-dispatcher` param parse/validate: `hub.mode`, `hub.callback` - (valid absolute URL), `hub.topic` (present) → malformed returns `{status:400}`; a valid - subscribe builds a `websub` `SubscribeRequest` **directly** (`callbackUrl=hub.callback`, - `resourceUrls=[hub.topic]`, not via `buildSubscribeRequest`). Pure unit tests, no network. - (→ `packages/core/src/protocols/websub-dispatcher.ts`: `parseSubscribe`) -- [x] **S1.2** `websub-plugin.verify()`: challenge GET to the callback with `hub.mode`, - `hub.topic`, `hub.challenge`; require `2xx` and an exact `hub.challenge` echo, else - throw (always verifies — ignores `diffDomain`). Injected `fetch` + challenge generator. - `protocols: ['websub']`. (→ `packages/core/src/protocols/websub-plugin.ts`; - `deliver()` is an interim failing stub until S2.1.) -- [x] **S1.3** `VerificationScheduler` as a `createRssCloudCore` option (default - in-process: run task next tick, catch+log; injectable for tests) + an engine - async-accept method `acceptSubscription(req)` that returns immediately and schedules - verify→persist via the scheduler: success persists a `protocol:'websub'` subscription - (with `details`), failure records nothing. `core.subscribe` is unchanged — the accept - method is a new caller of it. Unit test drains a capturing scheduler. - (→ `engine/verification-scheduler.ts`; default scheduler routes a thrown task to the - `error` event, scope `websub-verification`. Pre-ping-on-subscribe kept for now — see - open question.) -- [x] **S1.4** core `websub-dispatcher` ↔ express `websub({ core })` factory (same shape - as `ping`/`pleaseNotify`): parse the form body, `hub.mode=subscribe` → `core.accept…` - → `202`, malformed → `4xx`. Mirror `rest-middleware` (thin; dispatcher owns logic). - Export from `index.ts`. (No `scheduler`/`hubUrl` args — see architecture notes.) - (→ core `createWebSubDispatcher`; express `websub-middleware.ts`; both exported.) -- [x] **S1.5** Server integration (prerequisite for the S1.6 e2e): - **(a)** `apps/server/core.js` — add `createWebSubProtocolPlugin({ hubUrl, - requestTimeoutMs })` to the `plugins` array (registers `'websub'`; otherwise - `core.subscribe` rejects it). - **(b)** `apps/server/controllers/index.js` — `router.post('/websub', websub({ core }))`. - **(c)** `apps/server/config.js` — env for the hub's public base URL (`HUB_URL`, - default derived from `DOMAIN`/`PORT`) and mount path (`WEBSUB_PATH`, default `/websub`). - (Lease bounds + signature algo are added in Phases 5/3 when their slices need them.) - (Done; `hubUrl` is config-only until S2.1's deliver consumes it — plugin gets - `requestTimeoutMs` for now. Route mounts at `config.webSubPath`.) -- [x] **S1.6** e2e (**establishes the reusable mock subscriber harness** — challenge-echo): - POST subscribe → `202`, callback receives the verification GET, then **poll** - `/subscriptions.json` (already lists every sub incl. `protocol:'websub'`) until the - record appears — verification is async, so the test waits rather than asserting inline; - ★ callback refuses to echo → record never appears (bounded timeout); ★ malformed - `hub.*` (missing callback/topic, bad mode) → `4xx`. - (→ `apps/e2e/test/websub.js`; challenge-echo via the existing mock's function - `responseBody` (`req.query['hub.challenge']`); polls `storeApi.findSubscription`. - 138 e2e passing.) - -**Phase 2 — Content distribution via the existing rssCloud ping (THE PAYOFF)** -> Proves the primary use case: an rssCloud-only publisher's `/ping` fans content out to -> WebSub subscribers. No WebSub publish path — relies on core's resource-keyed fan-out. -- [x] **S2.1** `websub-plugin.deliver()`: POST `payload.body` to the callback, relaying - the topic's `Content-Type = payload.contentType` **verbatim** (xml/atom/json/etc. — the - hub is content-type-agnostic; `payload.contentType` is `string | null`, so pick a - fallback like `application/octet-stream` when the origin sent none), plus - `Link: ; rel="hub", ; rel="self"`. No signature yet. Inject `hubUrl`. - Unit tests with injected `fetch` (cover the present-and-null content-type branches). - (Done; `hubUrl` is an optional `createWebSubProtocolPlugin` option, wired from - `config.hubUrl` in `apps/server/core.js`; delivery follows 3xx redirects like - `rest-plugin`. 100% core coverage; 217 core tests passing.) -- [x] **S2.2** e2e (**the killer test** — extends the harness with content-capture): - put an rssCloud subscriber **and** a WebSub subscriber on the same topic `T`, then hit - the *existing* rssCloud `/ping` for `T` with changed content; assert **both** fire from - that single ping — the rssCloud sub gets its notify, the WebSub callback gets a POST - carrying the feed body + relayed `Content-Type` + `Link` rels. No `hub.mode=publish` - involved — this is the headline "free WebSub for rssCloud publishers" cross-protocol - proof. - (Done; `WebSub cross-protocol fan-out` in `apps/e2e/test/websub.js`. Content-capture - added to the shared `mock` via a catch-all `bodyParser.text` that records raw, - non-urlencoded POST bodies without disturbing rssCloud notify parsing. 139 e2e passing.) - -**Phase 3 — Authenticated distribution (HMAC-SHA256)** -- [x] **S3.1** parse + store `hub.secret` in `details` at subscribe. **S3.2** when - `details.secret` present, add `X-Hub-Signature: sha256=HMAC(secret, body)`; algorithm a - configurable plugin option (default `sha256`); no secret → no header. **S3.3** e2e: - subscriber verifies the signature over the rssCloud-ping-delivered body. - (Done; `parseSubscribe` stores `details.secret`; plugin signs via `node:crypto` - `createHmac` keyed by the `signatureAlgo` option (default `sha256`), wired from - `WEBSUB_SIGNATURE_ALGO`/`config.webSubSignatureAlgo`. e2e `WebSub authenticated - distribution` recomputes the HMAC over the received body. 221 core tests, 100% - coverage; 141 e2e passing.) - -**Phase 4 — Unsubscribe (intent-verified)** -- [x] **S4.1** plugin verify for `hub.mode=unsubscribe` (shared verify keyed by mode). - **S4.2** verified-unsubscribe path: scheduled task verifies intent then - `core.unsubscribe` (which has no verify hook today). **S4.3** dispatcher/express branch - `hub.mode=unsubscribe` → `202`. **S4.4** e2e unsubscribe handshake. - (Done; `VerifyContext.mode` threads `subscribe`/`unsubscribe` into the plugin's challenge - GET; new `core.acceptUnsubscription` schedules a verified removal (no-op if the sub is - absent or intent is refused); `websub-dispatcher` branches on `hub.mode` via a shared - `parseHubCallbackTopic`, express `core` Pick widened. e2e `WebSub unsubscribe` covers - verified removal + the refuse-echo negative. 232 core + 19 express tests, 100% coverage; - 143 e2e passing.) - -**Phase 5 — Leases (honor requested, clamped)** -- [x] **S5.1** `RssCloudConfig` lease bounds (default/min/max secs) + resolve defaults. - **S5.2** parse `hub.lease_seconds`, clamp, store `details.leaseSeconds`, - `whenExpires = now + chosen`; echo the chosen lease in the verification GET (thread the - chosen value into `verify`). **S5.3** e2e: requested lease clamped + echoed; expiry via - `removeExpired()`. - (Done; `RssCloudConfig` gains `webSubLease{Default,Min,Max}Secs` (default 86400/300/864000, - env `WEBSUB_LEASE_*`); the dispatcher parses `hub.lease_seconds` into `details`, core - clamps it in `subscribeOne`, stores the chosen value, sets `whenExpires = now + chosen`, - and threads it through `VerifyContext.leaseSeconds` so the plugin echoes `hub.lease_seconds`. - e2e `WebSub leases` covers clamp+echo and expiry via `removeExpired`. 241 core tests, - 100% coverage; 145 e2e passing.) - -**Phase 6 — WebSub-native publish front door (secondary — pure-WebSub publishers)** -- [x] **S6.1** dispatcher/express `hub.mode=publish` (thin: `hub.url`/`hub.topic`) → - `core.ping(topic)` → `2xx`/`204`. Lets a publisher with *no* rssCloud ping trigger the - same fan-out. Reuses everything from Phase 2. **S6.2** e2e: WebSub publish → WebSub - subscriber receives content. - (Done; new `core.acceptPublish` fire-and-forgets `ping` out of band — per WebSub §7 a - well-formed publish is acknowledged `202` and the topic re-fetched async (failures → - `error` event, scope `websub-publish`). Dispatcher branches `hub.mode=publish` via - `parsePublish` (`hub.url`, falling back to `hub.topic`); express/core Picks widened. - e2e `WebSub native publish` polls for the out-of-band delivery. 250 core + 20 express - tests, 100% coverage; 146 e2e passing.) - -**Phase 7 — Fat pings (secondary — publisher pushes the body)** -- [ ] **S7.1** decide + document the (non-standard) fat-ping wire format — topic via - param/header, raw body, and how to tell it from a thin publish (see open questions). - **S7.2** `PingRequest` optional pushed content; `detectChange` uses it instead of - fetching (still hashes for change detection). **S7.3** express publish detects a fat - ping → `core.ping` with pushed content → distributed verbatim. **S7.4** e2e fat ping. - -**Phase 8 — Hardening / spec niceties (deferred, optional)** -- [ ] `hub.mode=denied` callback notification on verification/validation failure. -- [ ] Persisted verification queue + retry (the seam refactor) — its own ADR/project. -- [ ] Publisher-facing docs: advertising the hub via ``. -- [ ] [websub.rocks](https://websub.rocks/) hub-conformance pass. - -*Coverage:* `packages/` stays at **100%** — every branch in the plugin, dispatcher, and -seam needs a test (or an explicit, justified ignore). e2e covers the integration. - -### Open questions (carry into the relevant slice) - -- **Fat-ping wire format (S7.1):** WebSub has no standard fat ping (it was a - PubSubHubbub 0.4 extension). Decide how a publisher indicates the topic when pushing a - body — a query/`hub.topic` param alongside a raw body, a `Content-Location`/`Link` - header, etc. — and how to distinguish it from a thin `hub.mode=publish`. -- **Resource pre-read on subscribe:** `core.subscribe` pre-pings the resource; WebSub - subscribe may skip that (the spec only requires intent verification). Decide when - wiring the accept path. -- **Seam ownership:** confirm the `VerificationScheduler` is core-owned (so a future - persisted queue lives next to the store) vs. injected from the composition root. From 4f684ecc475adad2b2a0180d8e25d2d04e717f1c Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Mon, 15 Jun 2026 09:54:48 -0500 Subject: [PATCH 20/35] feat(client): add WebSub subscribe/publish to the dev harness Teach the client harness to speak WebSub alongside rssCloud so the hub's WebSub paths can be exercised by hand. - lib/websub.js: createWebSubClient (subscribe/unsubscribe/publish over hub.* forms) and readVerification for the intent-verification GET. - feed.js: renderCloudFeed optionally advertises the hub via + rel="self"; output is unchanged when omitted. - client.js: a WebSub control set (with lease_seconds/secret inputs), a /websub-callback that echoes the challenge and reports the delivery's X-Hub-Signature with a valid/invalid verdict, and a hub-advertising feed. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/client/README.md | 40 +++++- apps/client/client.js | 244 ++++++++++++++++++++++++++++++++- apps/client/lib/feed.js | 59 ++++---- apps/client/lib/feed.test.js | 51 +++++++ apps/client/lib/index.js | 9 +- apps/client/lib/websub.js | 80 +++++++++++ apps/client/lib/websub.test.js | 188 +++++++++++++++++++++++++ 7 files changed, 639 insertions(+), 32 deletions(-) create mode 100644 apps/client/lib/websub.js create mode 100644 apps/client/lib/websub.test.js diff --git a/apps/client/README.md b/apps/client/README.md index 31b88f2..54d926a 100644 --- a/apps/client/README.md +++ b/apps/client/README.md @@ -6,8 +6,14 @@ notification protocol — the subscriber + publisher end, the mirror of `@rssclo by hand. The Express app ([`client.js`](client.js)) serves a **Subscribe/Ping UI** with a live -**request log**, and hosts the callback endpoint a hub notifies. All the protocol wire -work lives in [`lib/`](lib/) and is reusable on its own. +**request log**, and hosts the callback endpoint a hub notifies. It speaks both the +classic rssCloud protocol **and** [WebSub](https://www.w3.org/TR/websub/): the UI has a +separate WebSub control set (subscribe/unsubscribe/publish, with optional +`lease_seconds` and `secret`), the served feed advertises the hub via +``, and the WebSub callback echoes the intent-verification +challenge and reports the hub's `X-Hub-Signature` (with a valid/invalid verdict) on +content distribution. All the protocol wire work lives in [`lib/`](lib/) and is +reusable on its own. ## Running @@ -34,15 +40,41 @@ It listens on `PORT`, advertises itself as `DOMAIN`, and targets a hub at ## The `lib/` API -`require('./lib')` exposes three helpers (CommonJS): +`require('./lib')` exposes these helpers (CommonJS): - **`createRssCloudClient({ serverUrl, fetch? })`** — send `pleaseNotify` (subscribe) and `ping` (publish) to a hub over an injectable `fetch`. Returns `{ pleaseNotify, ping }`. +- **`createWebSubClient({ serverUrl, path?, fetch? })`** — send WebSub `hub.*` requests + to a hub's front door (`path` defaults to `/websub`). Returns + `{ subscribe, unsubscribe, publish }`; each resolves to the hub's raw reply + (`{ status, body }`) and does **not** throw on a non-2xx. +- **`readVerification(query)`** — given a callback GET's query, return + `{ mode, topic, challenge, leaseSeconds }` when it's a WebSub intent-verification + request (the subscriber must echo `challenge` verbatim), else `null`. - **`renderCloudFeed(feed)`** — emit an RSS 2.0 document carrying the `` element - that advertises a hub. + that advertises a hub. Pass `hub` (a URL) to also advertise a WebSub hub via + `` plus a `rel="self"` link. - **`buildNotifyResponse(success)`** — build the XML-RPC notify acknowledgement a subscriber returns to the hub. +### WebSub + +```js +const { createWebSubClient } = require('./lib'); + +const hub = createWebSubClient({ serverUrl: 'http://localhost:5337' }); + +await hub.subscribe({ + callbackUrl: 'http://localhost:9000/websub-callback', + topicUrl: 'http://localhost:9000/rss-01.xml', + leaseSeconds: 3600, // optional; the hub clamps to its configured bounds + secret: 's3cr3t' // optional; opts into a signed X-Hub-Signature delivery +}); + +await hub.publish({ topicUrl: 'http://localhost:9000/rss-01.xml' }); // hub.mode=publish +await hub.unsubscribe({ callbackUrl: '…', topicUrl: '…' }); +``` + ### Subscribe ```js diff --git a/apps/client/client.js b/apps/client/client.js index d4662c9..6aa7b80 100644 --- a/apps/client/client.js +++ b/apps/client/client.js @@ -1,13 +1,19 @@ const bodyParser = require('body-parser'), + crypto = require('crypto'), express = require('express'), morgan = require('morgan'), packageJson = require('./package.json'), { createRssCloudClient, + createWebSubClient, + readVerification, buildNotifyResponse, renderCloudFeed } = require('./lib'), textParser = bodyParser.text({ type: '*/xml' }), + // Content distribution arrives with the origin feed's Content-Type relayed + // verbatim, so the callback parses any media type as a raw string to log it. + rawTextParser = bodyParser.text({ type: () => true }), urlencodedParser = bodyParser.urlencoded({ extended: false }); // Simple config utility @@ -28,15 +34,58 @@ const clientConfig = { rsscloudServer: 'http://localhost:5337' }; -// All protocol wire work (pleaseNotify/ping calls, the XML-RPC notify ack, and -// feed rendering) lives in ./lib; this file is just the UI. +// The hub's WebSub front door, advertised in feeds via . +clientConfig.hubUrl = `${clientConfig.rsscloudServer}/websub`; +// The path the hub verifies and delivers WebSub content to on this harness. +const WEBSUB_CALLBACK_PATH = '/websub-callback'; + +// All protocol wire work (pleaseNotify/ping calls, WebSub hub.* calls, the +// XML-RPC notify ack, and /atom feed rendering) lives in ./lib; this file +// is just the UI. const client = createRssCloudClient({ serverUrl: clientConfig.rsscloudServer }); +const webSubClient = createWebSubClient({ + serverUrl: clientConfig.rsscloudServer +}); // In-memory data stores (reset on restart) const requestLog = []; const feedItems = {}; +// Secrets supplied on WebSub subscribe, keyed by topic URL, so the callback can +// check the hub's X-Hub-Signature on delivery. +const webSubSecrets = {}; const MAX_LOG_ENTRIES = 100; +// The callback URL this harness registers with the hub for a feed. +function webSubCallbackUrl() { + return `http://${clientConfig.domain}:${clientConfig.port}${WEBSUB_CALLBACK_PATH}`; +} + +// Pull the topic URL out of a delivery's Link header (`; rel="self"`). +function selfLink(link) { + const match = /<([^>]+)>\s*;\s*rel="self"/.exec(link || ''); + return match ? match[1] : undefined; +} + +// Verify a relayed X-Hub-Signature (`=`) against the body using the +// secret we subscribed with. Returns a human-readable verdict for the log. +function checkSignature(topicUrl, signature, body) { + const secret = webSubSecrets[topicUrl]; + if (!secret) { + return 'no stored secret — not verified'; + } + const [algo, digest] = String(signature).split('='); + if (!algo || !digest) { + return 'malformed header'; + } + let expected; + try { + expected = crypto.createHmac(algo, secret).update(body).digest('hex'); + } catch { + return `unsupported algorithm: ${algo}`; + } + return expected === digest ? 'valid ✓' : 'INVALID ✗'; +} + let app, server; console.log(`${clientConfig.appName} ${clientConfig.appVersion}`); @@ -77,14 +126,39 @@ app.use((req, res, next) => { if (req.path === '/subscribe' || req.path === '/ping-feed') { return; } + // WebSub UI actions are outbound; only hub -> harness traffic is logged. + if ( + req.path === '/websub-subscribe' || + req.path === '/websub-unsubscribe' || + req.path === '/websub-publish' + ) { + return; + } if (req.path.startsWith('/.well-known/')) { return; } + // Surface the WebSub delivery headers so the hub/self links and the + // signature (with our verdict) are visible in the log. + const headers = {}; + if (req.headers.link) { + headers.Link = req.headers.link; + } + if (req.headers['x-hub-signature']) { + const topic = selfLink(req.headers.link); + headers['X-Hub-Signature'] = + `${req.headers['x-hub-signature']} (${checkSignature( + topic, + req.headers['x-hub-signature'], + req.body + )})`; + } + const logEntry = { timestamp: new Date().toISOString(), method: req.method, url: req.originalUrl, + headers: Object.keys(headers).length ? headers : null, body: req.body || null }; @@ -125,9 +199,20 @@ function generateHtmlPage() { const logHtml = requestLog .map(entry => { const bodyDisplay = formatBody(entry.body); + const headersDisplay = entry.headers + ? Object.entries(entry.headers) + .map( + ([key, value]) => + `
${escapeHtml( + key + )}: ${escapeHtml(String(value))}
` + ) + .join('') + : ''; return `
${entry.method} ${escapeHtml(entry.url)} + ${headersDisplay ? `
${headersDisplay}
` : ''} ${bodyDisplay ? `
${bodyDisplay}
` : ''} ${entry.timestamp}
`; @@ -218,12 +303,25 @@ function generateHtmlPage() { } .result.success { background: #d4edda; } .result.error { background: #f8d7da; } + .headers { + margin: 5px 0 5px 55px; + font-size: 11px; + color: #555; + } + .header { word-break: break-all; } + .controls h3 { + margin: 0 0 10px; + font-size: 13px; + color: #666; + } + .controls + .controls { margin-top: -10px; }

rssCloud Test Client

+

rssCloud

+
+

WebSub

+
+ + + + + + +
+
+

Incoming Requests

${logHtml || '
No requests logged yet. Subscribe to a feed and ping it to see activity.
'} @@ -352,6 +462,135 @@ app.post('/ping-feed', urlencodedParser, async(req, res) => { } }); +// Render a simple result/error page for a WebSub UI action. +function webSubResultPage(action, feedUrl, status, body) { + return ` + + + WebSub ${action} Result + +

WebSub ${action} Result

+

Topic: ${escapeHtml(feedUrl)}

+

Status: ${status} ${status === 202 ? '(accepted — verification/fan-out is async)' : ''}

+ ${body ? `
${escapeHtml(body)}
` : ''} +

Back to client

+ + + `; +} + +function webSubErrorPage(action, error) { + return ` + + + WebSub ${action} Error + +

WebSub ${action} Error

+

${escapeHtml(error.message)}

+

Back to client

+ + + `; +} + +// Parse an optional positive-integer lease from the form, else undefined. +function parseLease(value) { + const seconds = parseInt(value, 10); + return Number.isInteger(seconds) && seconds > 0 ? seconds : undefined; +} + +// Route: WebSub subscribe (hub.mode=subscribe) +app.post('/websub-subscribe', urlencodedParser, async(req, res) => { + const feedName = req.body.feedName || 'rss-01.xml'; + const feedUrl = `http://${clientConfig.domain}:${clientConfig.port}/${feedName}`; + const secret = req.body.secret || undefined; + + // Remember the secret so the callback can verify the delivery signature. + if (secret) { + webSubSecrets[feedUrl] = secret; + } else { + delete webSubSecrets[feedUrl]; + } + + try { + const { status, body } = await webSubClient.subscribe({ + callbackUrl: webSubCallbackUrl(), + topicUrl: feedUrl, + leaseSeconds: parseLease(req.body.leaseSeconds), + secret + }); + res.type('html').send( + webSubResultPage('Subscribe', feedUrl, status, body) + ); + } catch (error) { + res.type('html').send(webSubErrorPage('Subscribe', error)); + } +}); + +// Route: WebSub unsubscribe (hub.mode=unsubscribe) +app.post('/websub-unsubscribe', urlencodedParser, async(req, res) => { + const feedName = req.body.feedName || 'rss-01.xml'; + const feedUrl = `http://${clientConfig.domain}:${clientConfig.port}/${feedName}`; + delete webSubSecrets[feedUrl]; + + try { + const { status, body } = await webSubClient.unsubscribe({ + callbackUrl: webSubCallbackUrl(), + topicUrl: feedUrl + }); + res.type('html').send( + webSubResultPage('Unsubscribe', feedUrl, status, body) + ); + } catch (error) { + res.type('html').send(webSubErrorPage('Unsubscribe', error)); + } +}); + +// Route: WebSub publish (hub.mode=publish) — mutate the feed, then notify the +// hub so it re-fetches and fans the change out to subscribers. +app.post('/websub-publish', urlencodedParser, async(req, res) => { + const feedName = req.body.feedName || 'rss-01.xml'; + const feedUrl = `http://${clientConfig.domain}:${clientConfig.port}/${feedName}`; + + if (!feedItems[feedName]) { + feedItems[feedName] = [{ title: 'initialized', timestamp: new Date() }]; + } + const now = new Date(); + feedItems[feedName].unshift({ + title: `Update at ${now.toISOString()}`, + timestamp: now + }); + + try { + const { status, body } = await webSubClient.publish({ + topicUrl: feedUrl + }); + res.type('html').send( + webSubResultPage('Publish', feedUrl, status, body) + ); + } catch (error) { + res.type('html').send(webSubErrorPage('Publish', error)); + } +}); + +// Route: WebSub intent verification — the hub GETs the callback with a +// hub.challenge the subscriber must echo verbatim to confirm the subscription. +app.get(WEBSUB_CALLBACK_PATH, (req, res) => { + const verification = readVerification(req.query); + if (verification) { + res.send(verification.challenge); + return; + } + res.status(404).send('Not a WebSub verification'); +}); + +// Route: WebSub content distribution — the hub POSTs the full feed body here. +// The request-logging middleware records the body, the hub/self Link header, and +// the signature verdict; we just acknowledge with a 2xx. +app.post(WEBSUB_CALLBACK_PATH, rawTextParser, (req, res) => { + res.status(204).end(); +}); + // Route: Handle challenge verification for http-post subscriptions app.get('/notify', (req, res) => { const challenge = req.query.challenge || ''; @@ -396,6 +635,7 @@ app.get('/:feedName', (req, res) => { registerProcedure: 'rssCloud.pleaseNotify', protocol: 'xml-rpc' }, + hub: clientConfig.hubUrl, items: items.map((item, index) => ({ title: item.title, description: `Feed item: ${item.title}`, diff --git a/apps/client/lib/feed.js b/apps/client/lib/feed.js index ea1d50a..42f95b0 100644 --- a/apps/client/lib/feed.js +++ b/apps/client/lib/feed.js @@ -2,33 +2,42 @@ const { Builder } = require('xml2js'); // Render an RSS 2.0 feed carrying a element — the document a publisher // serves so a hub knows where to register for change notifications. Item -// pubDates are emitted in RFC 822 form. +// pubDates are emitted in RFC 822 form. When `opts.hub` is given, the feed also +// advertises a WebSub hub via (with a rel="self" pointing +// at the feed's own URL), so the same document is discoverable over both +// protocols. function renderCloudFeed(opts) { - return new Builder().buildObject({ - rss: { - $: { version: '2.0' }, - channel: { - title: opts.title, - link: opts.link, - description: opts.description, - cloud: { - $: { - domain: opts.cloud.domain, - port: String(opts.cloud.port), - path: opts.cloud.path, - registerProcedure: opts.cloud.registerProcedure, - protocol: opts.cloud.protocol - } - }, - item: opts.items.map(item => ({ - title: item.title, - description: item.description, - pubDate: item.pubDate.toUTCString(), - guid: item.guid - })) + const rssAttrs = { version: '2.0' }; + const channel = { + title: opts.title, + link: opts.link, + description: opts.description, + cloud: { + $: { + domain: opts.cloud.domain, + port: String(opts.cloud.port), + path: opts.cloud.path, + registerProcedure: opts.cloud.registerProcedure, + protocol: opts.cloud.protocol } - } - }); + }, + item: opts.items.map(item => ({ + title: item.title, + description: item.description, + pubDate: item.pubDate.toUTCString(), + guid: item.guid + })) + }; + + if (opts.hub) { + rssAttrs['xmlns:atom'] = 'http://www.w3.org/2005/Atom'; + channel['atom:link'] = [ + { $: { rel: 'hub', href: opts.hub } }, + { $: { rel: 'self', href: opts.link } } + ]; + } + + return new Builder().buildObject({ rss: { $: rssAttrs, channel } }); } module.exports = { renderCloudFeed }; diff --git a/apps/client/lib/feed.test.js b/apps/client/lib/feed.test.js index d3e692a..2e3be7c 100644 --- a/apps/client/lib/feed.test.js +++ b/apps/client/lib/feed.test.js @@ -47,6 +47,57 @@ test('renders a channel with the cloud element and an item', async() => { assert.equal(rss.channel.item.pubDate, 'Fri, 02 Jan 2026 03:04:05 GMT'); }); +test('advertises a WebSub hub via atom:link rel=hub and rel=self', async() => { + const xml = renderCloudFeed({ + title: 'Test Feed', + link: 'http://sub.example:9000/rss-01.xml', + description: 'Test feed for rssCloud', + cloud: CLOUD, + hub: 'http://localhost:5337/websub', + items: [ + { + title: 'Update one', + description: 'first', + pubDate: new Date('2026-01-02T03:04:05Z'), + guid: 'rss-01-0' + } + ] + }); + + const { rss } = await reparse(xml); + assert.equal(rss.$['xmlns:atom'], 'http://www.w3.org/2005/Atom'); + assert.deepEqual( + rss.channel['atom:link'].map(link => link.$), + [ + { rel: 'hub', href: 'http://localhost:5337/websub' }, + { rel: 'self', href: 'http://sub.example:9000/rss-01.xml' } + ] + ); + // the rssCloud element is still emitted alongside the hub links + assert.equal(rss.channel.cloud.$.protocol, 'xml-rpc'); +}); + +test('omits the atom namespace and links when no hub is given', async() => { + const xml = renderCloudFeed({ + title: 'Test Feed', + link: 'http://sub.example:9000/rss-01.xml', + description: 'Test feed for rssCloud', + cloud: CLOUD, + items: [ + { + title: 'Update one', + description: 'first', + pubDate: new Date('2026-01-02T03:04:05Z'), + guid: 'rss-01-0' + } + ] + }); + + const { rss } = await reparse(xml); + assert.equal(rss.$['xmlns:atom'], undefined); + assert.equal(rss.channel['atom:link'], undefined); +}); + test('renders multiple items in order', async() => { const xml = renderCloudFeed({ title: 'Test Feed', diff --git a/apps/client/lib/index.js b/apps/client/lib/index.js index 18502d2..488b272 100644 --- a/apps/client/lib/index.js +++ b/apps/client/lib/index.js @@ -1,5 +1,12 @@ const { createRssCloudClient } = require('./client'); const { renderCloudFeed } = require('./feed'); const { buildNotifyResponse } = require('./notify'); +const { createWebSubClient, readVerification } = require('./websub'); -module.exports = { createRssCloudClient, renderCloudFeed, buildNotifyResponse }; +module.exports = { + createRssCloudClient, + renderCloudFeed, + buildNotifyResponse, + createWebSubClient, + readVerification +}; diff --git a/apps/client/lib/websub.js b/apps/client/lib/websub.js new file mode 100644 index 0000000..8b12d74 --- /dev/null +++ b/apps/client/lib/websub.js @@ -0,0 +1,80 @@ +const FORM_TYPE = 'application/x-www-form-urlencoded'; + +// Build a WebSub client bound to one hub. subscribe/unsubscribe/publish all POST +// an `hub.*` urlencoded form to the hub's single front door (default `/websub`) +// over an injectable fetch, and resolve to the hub's raw reply ({ status, body }) +// without throwing on a non-2xx — inspect `status` yourself. +function createWebSubClient(options) { + const doFetch = options.fetch ?? fetch; + const base = options.serverUrl.replace(/\/$/, ''); + const path = options.path ?? '/websub'; + + async function send(form) { + const res = await doFetch(`${base}${path}`, { + method: 'POST', + headers: { 'Content-Type': FORM_TYPE }, + body: form.toString() + }); + return { status: res.status, body: await res.text() }; + } + + // The callback+topic form both subscribe and unsubscribe open with. + function callbackForm(mode, opts) { + return new URLSearchParams({ + 'hub.mode': mode, + 'hub.callback': opts.callbackUrl, + 'hub.topic': opts.topicUrl + }); + } + + async function subscribe(opts) { + const form = callbackForm('subscribe', opts); + if (opts.leaseSeconds !== undefined) { + form.set('hub.lease_seconds', String(opts.leaseSeconds)); + } + if (opts.secret !== undefined) { + form.set('hub.secret', opts.secret); + } + return send(form); + } + + async function unsubscribe(opts) { + return send(callbackForm('unsubscribe', opts)); + } + + async function publish(opts) { + return send( + new URLSearchParams({ + 'hub.mode': 'publish', + 'hub.url': opts.topicUrl + }) + ); + } + + return { subscribe, unsubscribe, publish }; +} + +// Read a hub's intent-verification GET query (Express `req.query`). A WebSub +// verification always carries `hub.mode` and a `hub.challenge` the subscriber +// must echo verbatim; returns the parsed fields, or `null` when the query isn't +// a verification (so the callback can fall through). `hub.lease_seconds` rides +// along on subscribe verifications only. +function readVerification(query) { + const mode = query['hub.mode']; + const challenge = query['hub.challenge']; + if (typeof mode !== 'string' || typeof challenge !== 'string') { + return null; + } + const parsed = { + mode, + topic: query['hub.topic'], + challenge + }; + const lease = query['hub.lease_seconds']; + if (lease !== undefined) { + parsed.leaseSeconds = Number(lease); + } + return parsed; +} + +module.exports = { createWebSubClient, readVerification }; diff --git a/apps/client/lib/websub.test.js b/apps/client/lib/websub.test.js new file mode 100644 index 0000000..5c59295 --- /dev/null +++ b/apps/client/lib/websub.test.js @@ -0,0 +1,188 @@ +const test = require('node:test'); +const assert = require('node:assert/strict'); +const { createWebSubClient, readVerification } = require('./websub'); + +function fakeFetch(status = 202, responseBody = '') { + const calls = []; + const fn = async(url, init) => { + calls.push({ url: String(url), init: init ?? {} }); + return { status, text: async() => responseBody }; + }; + return { fn, calls }; +} + +function form(init) { + return new URLSearchParams(init.body); +} + +test('subscribe posts the hub.* subscribe form to /websub', async() => { + const { fn, calls } = fakeFetch(); + const client = createWebSubClient({ + serverUrl: 'http://hub.example:5337', + fetch: fn + }); + + const res = await client.subscribe({ + callbackUrl: 'http://sub.example:9000/websub-callback', + topicUrl: 'https://feed.example/rss' + }); + + assert.equal(calls[0].url, 'http://hub.example:5337/websub'); + assert.equal(calls[0].init.method, 'POST'); + assert.equal( + calls[0].init.headers['Content-Type'], + 'application/x-www-form-urlencoded' + ); + const body = form(calls[0].init); + assert.equal(body.get('hub.mode'), 'subscribe'); + assert.equal( + body.get('hub.callback'), + 'http://sub.example:9000/websub-callback' + ); + assert.equal(body.get('hub.topic'), 'https://feed.example/rss'); + assert.deepEqual(res, { status: 202, body: '' }); +}); + +test('subscribe carries hub.lease_seconds and hub.secret when supplied', async() => { + const { fn, calls } = fakeFetch(); + const client = createWebSubClient({ + serverUrl: 'http://hub.example:5337', + fetch: fn + }); + + await client.subscribe({ + callbackUrl: 'http://sub.example:9000/websub-callback', + topicUrl: 'https://feed.example/rss', + leaseSeconds: 3600, + secret: 's3cr3t' + }); + + const body = form(calls[0].init); + assert.equal(body.get('hub.lease_seconds'), '3600'); + assert.equal(body.get('hub.secret'), 's3cr3t'); +}); + +test('subscribe omits hub.lease_seconds and hub.secret when not supplied', async() => { + const { fn, calls } = fakeFetch(); + const client = createWebSubClient({ + serverUrl: 'http://hub.example:5337', + fetch: fn + }); + + await client.subscribe({ + callbackUrl: 'http://sub.example:9000/websub-callback', + topicUrl: 'https://feed.example/rss' + }); + + const body = form(calls[0].init); + assert.equal(body.has('hub.lease_seconds'), false); + assert.equal(body.has('hub.secret'), false); +}); + +test('publish posts hub.mode=publish with the topic as hub.url', async() => { + const { fn, calls } = fakeFetch(); + const client = createWebSubClient({ + serverUrl: 'http://hub.example:5337', + fetch: fn + }); + + const res = await client.publish({ topicUrl: 'https://feed.example/rss' }); + + assert.equal(calls[0].url, 'http://hub.example:5337/websub'); + assert.equal(calls[0].init.method, 'POST'); + const body = form(calls[0].init); + assert.equal(body.get('hub.mode'), 'publish'); + assert.equal(body.get('hub.url'), 'https://feed.example/rss'); + assert.deepEqual(res, { status: 202, body: '' }); +}); + +test('unsubscribe posts hub.mode=unsubscribe with callback and topic', async() => { + const { fn, calls } = fakeFetch(); + const client = createWebSubClient({ + serverUrl: 'http://hub.example:5337', + fetch: fn + }); + + await client.unsubscribe({ + callbackUrl: 'http://sub.example:9000/websub-callback', + topicUrl: 'https://feed.example/rss' + }); + + const body = form(calls[0].init); + assert.equal(body.get('hub.mode'), 'unsubscribe'); + assert.equal( + body.get('hub.callback'), + 'http://sub.example:9000/websub-callback' + ); + assert.equal(body.get('hub.topic'), 'https://feed.example/rss'); +}); + +test('targets a configurable hub path', async() => { + const { fn, calls } = fakeFetch(); + const client = createWebSubClient({ + serverUrl: 'http://hub.example:5337', + path: '/hub', + fetch: fn + }); + + await client.publish({ topicUrl: 'https://feed.example/rss' }); + + assert.equal(calls[0].url, 'http://hub.example:5337/hub'); +}); + +test('strips a trailing slash from the server URL', async() => { + const { fn, calls } = fakeFetch(); + const client = createWebSubClient({ + serverUrl: 'http://hub.example:5337/', + fetch: fn + }); + + await client.publish({ topicUrl: 'https://feed.example/rss' }); + + assert.equal(calls[0].url, 'http://hub.example:5337/websub'); +}); + +test('defaults to the global fetch when none is injected', () => { + const client = createWebSubClient({ + serverUrl: 'http://hub.example:5337' + }); + + assert.equal(typeof client.subscribe, 'function'); + assert.equal(typeof client.unsubscribe, 'function'); + assert.equal(typeof client.publish, 'function'); +}); + +test('readVerification parses a subscribe verification GET', () => { + const parsed = readVerification({ + 'hub.mode': 'subscribe', + 'hub.topic': 'https://feed.example/rss', + 'hub.challenge': 'abc123', + 'hub.lease_seconds': '3600' + }); + + assert.deepEqual(parsed, { + mode: 'subscribe', + topic: 'https://feed.example/rss', + challenge: 'abc123', + leaseSeconds: 3600 + }); +}); + +test('readVerification omits leaseSeconds for an unsubscribe verification', () => { + const parsed = readVerification({ + 'hub.mode': 'unsubscribe', + 'hub.topic': 'https://feed.example/rss', + 'hub.challenge': 'xyz789' + }); + + assert.deepEqual(parsed, { + mode: 'unsubscribe', + topic: 'https://feed.example/rss', + challenge: 'xyz789' + }); +}); + +test('readVerification returns null when the query carries no challenge', () => { + assert.equal(readVerification({ 'hub.mode': 'subscribe' }), null); + assert.equal(readVerification({}), null); +}); From b4c8df4b3f4944db4dd68497e54f4404723e8cb6 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Mon, 15 Jun 2026 15:39:57 -0500 Subject: [PATCH 21/35] docs(server): split protocol docs into per-dialect pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README was the only doc page and didn't cover XML-RPC. Break the protocol surface into dedicated pages under apps/server/docs/ and turn the README into an index, so the docs read well both on GitHub and in the app. - New pages: rssCloud over REST, rssCloud over XML-RPC, WebSub (incl. how to advertise a hub from a feed — Link header primary, atom:link backup), and a cross-protocol page on the unified fan-out. - markdown-doc: rewrite relative .md links to in-app routes (preserving #anchors) and add GitHub-style heading ids, so the same source links work as files on GitHub and as routes in the rendered docs. - controllers: DOC_PAGES table; /docs index plus /docs/:page (404 unknown). - home nav and e2e static coverage updated for the new pages. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/e2e/test/static.js | 19 ++++ apps/server/README.md | 91 +++------------ apps/server/controllers/index.js | 42 ++++++- apps/server/docs/cross-protocol.md | 58 ++++++++++ apps/server/docs/rsscloud-rest.md | 87 ++++++++++++++ apps/server/docs/rsscloud-xml-rpc.md | 66 +++++++++++ apps/server/docs/websub.md | 131 ++++++++++++++++++++++ apps/server/services/markdown-doc.js | 78 ++++++++++++- apps/server/services/markdown-doc.test.js | 52 +++++++++ apps/server/views/home.handlebars | 12 +- 10 files changed, 555 insertions(+), 81 deletions(-) create mode 100644 apps/server/docs/cross-protocol.md create mode 100644 apps/server/docs/rsscloud-rest.md create mode 100644 apps/server/docs/rsscloud-xml-rpc.md create mode 100644 apps/server/docs/websub.md diff --git a/apps/e2e/test/static.js b/apps/e2e/test/static.js index a2f796c..57d6234 100644 --- a/apps/e2e/test/static.js +++ b/apps/e2e/test/static.js @@ -12,6 +12,25 @@ describe('Static Pages', function() { expect(res).status(200); }); + for (const slug of [ + 'rsscloud-rest', + 'rsscloud-xml-rpc', + 'websub', + 'cross-protocol' + ]) { + it(`docs/${slug} should return 200`, async function() { + let res = await chai.request(SERVER_URL).get(`/docs/${slug}`); + + expect(res).status(200); + }); + } + + it('an unknown docs page should return 404', async function() { + let res = await chai.request(SERVER_URL).get('/docs/nonexistent'); + + expect(res).status(404); + }); + it('home should return 200', async function() { let res = await chai.request(SERVER_URL).get('/'); diff --git a/apps/server/README.md b/apps/server/README.md index 6ea2a63..a5cb7b3 100644 --- a/apps/server/README.md +++ b/apps/server/README.md @@ -4,7 +4,24 @@ [![CI](https://github.com/rsscloud/rsscloud-server/actions/workflows/ci.yml/badge.svg)](https://github.com/rsscloud/rsscloud-server/actions/workflows/ci.yml) [![Andrew Shell's Weblog](https://img.shields.io/badge/weblog-rssCloud-brightgreen)](https://andrewshell.org/search/?keywords=rsscloud) -rssCloud Server implementation in Node.js +rssCloud Server implementation in Node.js. + +Subscribers register a callback to be told when a feed updates; publishers announce a +change; the server fans the notification out to every subscriber. It speaks the +[rssCloud](http://rsscloud.org/) protocol (over REST and XML-RPC) **and** acts as a +[WebSub](https://www.w3.org/TR/websub/) hub — and a single publish reaches all of them +at once. + +## Documentation + +- **[rssCloud over REST](docs/rsscloud-rest.md)** — `POST /pleaseNotify` and + `POST /ping` as form posts. +- **[rssCloud over XML-RPC](docs/rsscloud-xml-rpc.md)** — `rssCloud.hello`, + `rssCloud.pleaseNotify`, and `rssCloud.ping` at `POST /RPC2`. +- **[WebSub](docs/websub.md)** — the hub endpoint, intent verification, leases, signed + delivery, and how to advertise your hub from a feed. +- **[How it fits together](docs/cross-protocol.md)** — why one ping notifies every + subscriber regardless of the protocol they used. ## How to install @@ -59,75 +76,3 @@ pnpm test This should build the appropriate containers and show the test output. Our tests create mock API endpoints so we can verify rssCloud server works correctly when reading resources and notifying subscribers. - -## How to use - -### POST /pleaseNotify - -Posting to /pleaseNotify is your way of alerting the server that you want to receive notifications when one or more resources are updated. - -The POST parameters are: - -1. domain -- optional, if omitted the requesting IP address is used -2. port -3. path -4. registerProcedure -- required, but isn't used in this server as it only applies to xml-rpc or soap. -5. protocol -- the spec allows for http-post, xml-rpc or soap but this server only supports http-post and xml-rpc. This server also supports https-post which is identical to http-post except it notifies using https as the scheme instead of http. _Note: if you specify http-post with port 443, the server will automatically use the https scheme for notifications._ For other ports that expect https, use https-post as the protocol. -6. url1, url2, ..., urlN this is the resource you're requesting to be notified about. In the case of an RSS feed you would specify the URL of the RSS feed. - -When you POST the server first checks if the urls you specifed are returning an [HTTP 2xx status code](http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2) then it attempts to notify the subscriber of an update to make sure it works. This is done in one of two ways. - -1. If you did not specify a domain parameter and we're using the requesting IP address we perform a POST request to the URL represented by `http://:` with a single parameter `url`. To accept the subscription that resource just needs to return an HTTP 2xx status code. -2. If you did specify a domain parameter then we perform a GET request to the URL represented by `http://:` with two query string parameters, url and challenge. To accept the subscription that resource needs to return an HTTP 2xx status code and have the challenge value as the response body. - -You will receive a response with two values: - -1. success -- true or false depending on whether or not the subscription suceeded -2. msg -- a string that explains either that you succeed or why it failed - -The default response type is text/xml but if you POST with an [accept header](http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1) specifying `application/json` we will return a JSON formatted response. - -Examples: - -```xml - - -``` - -```json -{ - "success": false, - "msg": "The subscription was cancelled because the call failed when we tested the handler." -} -``` - -### POST /ping - -Posting to /ping is your way of alerting the server that a resource has been updated. - -The POST parameters are: - -1. url - -When you POST the server first checks if the url has actually changed since the last time it checked. If it has, it will go through it's list of subscribers and POST to the subscriber with the parameter `url`. - -The default response type is text/xml but if you POST with an [accept header](http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1) specifying `application/json` we will return a JSON formatted response. - -Examples: - -```xml - - -``` - -```json -{ "success": true, "msg": "Thanks for the ping." } -``` - -### GET /pingForm - -The path /pingForm is an HTML form intented to allow you to ping via a web browser. - -### GET /viewLog - -The path /viewLog is a log of recent events that have occured on the server. It's very useful if you're trying to debug your tools. diff --git a/apps/server/controllers/index.js b/apps/server/controllers/index.js index 34f9e16..9da9eed 100644 --- a/apps/server/controllers/index.js +++ b/apps/server/controllers/index.js @@ -15,15 +15,32 @@ const NEGOTIATED_VIEWS = [ { path: '/pleaseNotifyForm', view: 'please-notify-form' } ]; +// Per-protocol documentation pages, rendered from docs/.md into the shared +// `docs` view at /docs/. README.md is the index at /docs. +const DOC_PAGES = [ + { slug: 'rsscloud-rest', label: 'rssCloud over REST' }, + { slug: 'rsscloud-xml-rpc', label: 'rssCloud over XML-RPC' }, + { slug: 'websub', label: 'WebSub' }, + { slug: 'cross-protocol', label: 'How It Fits Together' } +]; + +// Maps the relative `.md` basenames the Markdown links to (GitHub-relative) onto +// their in-app routes, so renderMarkdownDoc can rewrite them: README → /docs, +// each page → /docs/. Unmapped targets (LICENSE.md) are left untouched. +const DOC_LINKS = DOC_PAGES.reduce( + (links, { slug }) => ({ ...links, [slug]: `/docs/${slug}` }), + { README: '/docs' } +); + // Render a Markdown file into the shared `docs` view, mapping a read failure to -// a 500. The README/LICENSE routes differ only in source file, heading, and -// whether the redundant leading H1 is dropped. +// a 500. Routes differ only in source file, heading, and whether the redundant +// leading H1 is dropped; all share the DOC_LINKS rewrite map. function sendMarkdownDoc(res, { file, label, stripH1 }) { try { res.render('docs', { title: `rssCloud Server: ${label}`, heading: `rssCloud Server: ${label}`, - htmltext: renderMarkdownDoc(file, { stripH1 }) + htmltext: renderMarkdownDoc(file, { stripH1, docLinks: DOC_LINKS }) }); } catch (err) { console.error(`Error reading ${file}:`, err.message); @@ -73,6 +90,25 @@ function createControllers({ core }) { }); }); + router.get('/docs/:page', (req, res) => { + if (req.accepts('html') !== 'html') { + res.status(406).send('Not Acceptable'); + return; + } + // Only known slugs map to a file, so an unknown (or traversal) page is a + // plain 404 rather than a file read off disk. + const page = DOC_PAGES.find(p => p.slug === req.params.page); + if (page === undefined) { + res.status(404).send('Not Found'); + return; + } + sendMarkdownDoc(res, { + file: `docs/${page.slug}.md`, + label: page.label, + stripH1: true + }); + }); + router.get('/LICENSE.md', (req, res) => { sendMarkdownDoc(res, { file: 'LICENSE.md', diff --git a/apps/server/docs/cross-protocol.md b/apps/server/docs/cross-protocol.md new file mode 100644 index 0000000..a685337 --- /dev/null +++ b/apps/server/docs/cross-protocol.md @@ -0,0 +1,58 @@ +# How it fits together + +The server speaks three notification dialects — [rssCloud over REST](rsscloud-rest.md), +[rssCloud over XML-RPC](rsscloud-xml-rpc.md), and [WebSub](websub.md) — but underneath +they are **one hub over one subscriber list**. The headline consequence: + +> A publisher sends **one** signal, and **every** subscriber to that resource is +> notified — no matter which protocol each of them subscribed with. + +## One signal, three doors + +A change can be announced three ways, and all of them converge on the same internal +routine: + +| Publisher sends | Front door | +| ----------------------------------- | -------------- | +| `POST /ping` (form `url`) | rssCloud REST | +| `rssCloud.ping` (`resourceUrl`) | rssCloud XML-RPC | +| `POST /websub` with `hub.mode=publish` | WebSub | + +Each one hands the hub a resource URL and triggers the same sequence. + +## What the hub does + +1. **Re-fetch** the resource once. +2. **Detect change** — the hub hashes the body (and tracks its size). If neither the + hash nor the size differs from the last time it looked, nothing is sent. (A + resource the hub has never seen before counts as changed, so its first ping fans + out.) +3. **Fan out** — the hub loads *all* active subscriptions for that resource and + delivers to each one using the delivery method that subscription registered with. + +That third step is the whole point: the subscriber list is keyed by resource, not by +protocol. Each subscription carries a `protocol`, and the hub picks the matching +delivery for it: + +| Subscription `protocol` | How that subscriber is notified | +| ----------------------- | ------------------------------------------------------------ | +| `http-post` | HTTP `POST` to the callback with a `url` parameter | +| `https-post` | the same over HTTPS | +| `xml-rpc` | an XML-RPC `notify` call to the callback | +| `websub` | HTTP `POST` of the full feed body, with `Link` and optional `X-Hub-Signature` | + +## Why this matters + +You don't have to care which protocol your subscribers chose, and they don't have to +care which one your publisher speaks: + +- A publisher that speaks **rssCloud** (a `/ping`) still notifies **WebSub** + subscribers. +- A publisher that speaks **WebSub** (`hub.mode=publish`) still notifies **rssCloud** + subscribers. + +The protocol is a per-subscriber delivery detail. The change signal is shared. + +--- + +← [Back to the documentation index](../README.md) diff --git a/apps/server/docs/rsscloud-rest.md b/apps/server/docs/rsscloud-rest.md new file mode 100644 index 0000000..e5a1805 --- /dev/null +++ b/apps/server/docs/rsscloud-rest.md @@ -0,0 +1,87 @@ +# rssCloud over REST + +The rssCloud protocol spoken as plain `application/x-www-form-urlencoded` +HTTP POSTs. Subscribers register a callback with [`POST /pleaseNotify`](#post-pleasenotify); +publishers announce a change with [`POST /ping`](#post-ping). The same notifications +can also be driven [over XML-RPC](rsscloud-xml-rpc.md), and a single ping reaches +subscribers of **every** protocol — see [How it fits together](cross-protocol.md). + +## POST /pleaseNotify + +Tell the server you want to be notified when one or more resources (feeds) change. + +The form parameters are: + +| Parameter | Required | Meaning | +| -------------------- | -------- | ----------------------------------------------------------------------- | +| `protocol` | yes | How the server should notify you: `http-post`, `https-post`, or `xml-rpc`. | +| `port` | yes | Port of your callback. | +| `path` | yes | Path of your callback. | +| `url1`, `url2`, … `urlN` | yes | The resource URL(s) you want to watch. For a feed, its URL. | +| `domain` | no | Your callback host. If omitted, the server uses the requesting IP address. | +| `registerProcedure` | no | Accepted for spec compatibility but ignored (only meaningful for XML-RPC/SOAP). | + +`https-post` is identical to `http-post` except notifications are sent over HTTPS. +As a convenience, `http-post` with `port` 443 is also notified over HTTPS; for any +other HTTPS port, use `https-post`. + +### How the subscription is verified + +Before recording the subscription, the server confirms your resource URLs return an +HTTP 2xx, then proves your callback works — in one of two ways: + +1. **No `domain` given (IP-based).** The server `POST`s to `http://:` + with a single `url` parameter. Reply with any 2xx to accept. +2. **`domain` given (challenge).** The server `GET`s + `http://:?url=&challenge=`. Reply 2xx **and** + echo the `challenge` value verbatim as the body to accept. + +### Response + +Two values are returned: `success` (`true`/`false`) and `msg` (a human-readable +explanation). The default content type is `text/xml`; send `Accept: application/json` +for JSON. + +```xml + + +``` + +```json +{ + "success": false, + "msg": "The subscription was cancelled because the call failed when we tested the handler." +} +``` + +## POST /ping + +Tell the server a resource has been updated. + +| Parameter | Required | Meaning | +| --------- | -------- | ---------------------------- | +| `url` | yes | The resource URL that changed. | + +The server re-fetches the URL and, **only if the content actually changed**, fans the +notification out to every subscriber of that resource (see +[How it fits together](cross-protocol.md)). The default content type is `text/xml`; +send `Accept: application/json` for JSON. + +```xml + + +``` + +```json +{ "success": true, "msg": "Thanks for the ping." } +``` + +## Browser helpers + +- **`GET /pleaseNotifyForm`** — an HTML form for subscribing from a browser. +- **`GET /pingForm`** — an HTML form for pinging from a browser. +- **`GET /viewLog`** — a live log of recent server events, handy when debugging your tools. + +--- + +← [Back to the documentation index](../README.md) diff --git a/apps/server/docs/rsscloud-xml-rpc.md b/apps/server/docs/rsscloud-xml-rpc.md new file mode 100644 index 0000000..9e2f067 --- /dev/null +++ b/apps/server/docs/rsscloud-xml-rpc.md @@ -0,0 +1,66 @@ +# rssCloud over XML-RPC + +The same rssCloud operations as the [REST front door](rsscloud-rest.md), spoken as +XML-RPC method calls. Everything is handled at a single endpoint: + +**`POST /RPC2`** — send an XML-RPC `methodCall` with `Content-Type: text/xml`. The +response is an XML-RPC `methodResponse`, also `text/xml`. + +Three methods are recognised. + +## rssCloud.hello + +A connectivity check. No parameters; always returns boolean `true`. + +```xml +1 +``` + +## rssCloud.pleaseNotify + +Register a callback for one or more resources. Six positional parameters, in order: + +| # | Parameter | Type | Meaning | +| - | ----------------- | --------------- | -------------------------------------------------------------------- | +| 1 | `notifyProcedure` | string | The XML-RPC method the server calls on your callback (e.g. `rssCloud.notify`). Used for the `xml-rpc` protocol. | +| 2 | `port` | int (`i4`) | Port of your callback. | +| 3 | `path` | string | Path of your callback. | +| 4 | `protocol` | string | `http-post`, `https-post`, or `xml-rpc`. | +| 5 | `urlList` | array of string | The resource URL(s) to watch. | +| 6 | `domain` | string | Optional callback host; omit (or pass empty) to use the caller's address. | + +Parameters 1–5 are required (5 or 6 params total). On a successful, verified +subscription the response is boolean `true`. A subscription failure or a malformed +call returns a **fault** (see [Faults](#faults)). + +## rssCloud.ping + +Announce that a resource changed. One positional parameter: + +| # | Parameter | Type | Meaning | +| - | ------------- | ------ | ------------------------------ | +| 1 | `resourceUrl` | string | The resource URL that changed. | + +As in the rssCloud reference implementation, `rssCloud.ping` returns boolean `true` +whenever the call is **well-formed** — even if the re-fetch or fan-out later fails. Only a +malformed call (wrong number of parameters) returns a fault. As with REST, a +well-formed ping triggers a re-fetch and, on a real change, a fan-out to every +subscriber regardless of protocol (see [How it fits together](cross-protocol.md)). + +## Faults + +Errors are returned as a standard XML-RPC `fault`. rssCloud faults always use +`faultCode` `4`; the `faultString` carries a human-readable explanation. + +```xml + + + faultCode4 + faultStringCan't make the call because "rssCloud.frobnicate" is not defined. + + +``` + +--- + +← [Back to the documentation index](../README.md) diff --git a/apps/server/docs/websub.md b/apps/server/docs/websub.md new file mode 100644 index 0000000..48fcf58 --- /dev/null +++ b/apps/server/docs/websub.md @@ -0,0 +1,131 @@ +# WebSub + +This server is also a [WebSub](https://www.w3.org/TR/websub/) hub. WebSub (formerly +PubSubHubbub) is a W3C-standard publish/subscribe protocol for web content: +subscribers register a callback, the hub verifies their intent, and on each update the +hub delivers the **full feed body** to the callback — optionally signed. + +A single publish reaches WebSub subscribers **and** rssCloud subscribers alike; see +[How it fits together](cross-protocol.md). + +All hub operations share one endpoint: + +**`POST /websub`** — `application/x-www-form-urlencoded`, with `hub.*` fields. (The +path is configurable via `WEBSUB_PATH`; `/websub` is the default.) + +## Subscribe and unsubscribe + +| Field | Required | Meaning | +| ------------------ | -------- | ----------------------------------------------------------------------- | +| `hub.mode` | yes | `subscribe` or `unsubscribe`. | +| `hub.callback` | yes | Absolute URL the hub delivers to (and verifies against). | +| `hub.topic` | yes | Absolute URL of the feed you want. | +| `hub.lease_seconds`| no | Requested subscription lifetime; the hub clamps it (see [Leases](#leases)). | +| `hub.secret` | no | Shared secret that opts you into [signed delivery](#authenticated-delivery). | + +A well-formed request is acknowledged immediately with **`202 Accepted`**; a malformed +one (missing/relative `hub.callback`, empty `hub.topic`, unknown `hub.mode`) returns +**`400`**. The `202` only means the request was accepted — the subscription is not +active until intent verification succeeds. + +## Intent verification + +After a `202`, the hub confirms the request out of band by sending a `GET` to your +`hub.callback` with these query parameters: + +| Parameter | Meaning | +| ------------------- | -------------------------------------------------------------- | +| `hub.mode` | `subscribe` or `unsubscribe` (echoes the request). | +| `hub.topic` | The topic URL. | +| `hub.challenge` | A random token. | +| `hub.lease_seconds` | The lease the hub actually granted (subscribe only; see below).| + +To confirm, respond **`2xx`** with a body that is **exactly** the `hub.challenge` +value. Any other status, or a body that doesn't match, and the hub discards the +subscription. + +## Leases + +`hub.lease_seconds` is a request, not a guarantee. The hub clamps it to its configured +bounds and tells you the granted value in the verification GET's `hub.lease_seconds`: + +| Bound | Config key | Default | +| ------- | --------------------------- | ------------------ | +| default | `WEBSUB_LEASE_DEFAULT_SECS` | `86400` (1 day) | +| minimum | `WEBSUB_LEASE_MIN_SECS` | `300` (5 minutes) | +| maximum | `WEBSUB_LEASE_MAX_SECS` | `864000` (10 days) | + +If you omit `hub.lease_seconds` you get the default. Re-subscribe before the lease +expires to renew it. + +## Content distribution + +When the topic changes, the hub `POST`s to your `hub.callback`: + +- **Body** — the full fetched feed content. +- **`Content-Type`** — relayed from the origin feed (or `application/octet-stream` if + the origin sent none). +- **`Link`** — `; rel="hub", ; rel="self"`, advertising the hub and + the canonical topic. +- **`X-Hub-Signature`** — only when you subscribed with a `hub.secret` (see below). + +Respond with any `2xx` to acknowledge. Redirects are followed. + +### Authenticated delivery + +If you supplied `hub.secret`, the hub signs each delivery with an +`X-Hub-Signature: =` header, where `` is the HMAC of the request body +keyed by your secret. The algorithm is `sha256` by default (configurable via +`WEBSUB_SIGNATURE_ALGO`). Recompute the HMAC over the received body and compare to +reject spoofed deliveries. + +## Publishing + +A publisher can notify the hub natively over WebSub instead of an rssCloud ping: + +| Field | Required | Meaning | +| ---------- | -------- | --------------------------------------------------- | +| `hub.mode` | yes | `publish`. | +| `hub.url` | yes\* | The topic URL that changed. (`hub.topic` is accepted as a fallback.) | + +The hub answers `202`, then re-fetches the topic and fans the change out. This is +exactly the path an rssCloud `/ping` takes, so a WebSub publish also reaches rssCloud +subscribers — see [How it fits together](cross-protocol.md). + +## Using WebSub with your feed + +So that subscribers can **discover** this hub, advertise it from the resource you want +watched. Per the WebSub spec, advertise it two ways, in priority order: + +1. **HTTP `Link` header (primary).** When your server returns the feed, include: + + ```http + Link: ; rel="hub" + Link: ; rel="self" + ``` + + The header is authoritative and works for any content type, so it's the preferred + mechanism. + +2. **`` in the feed (backup).** Inside the feed document, declare the Atom + namespace and add the hub and self links — useful for consumers that only read the + body: + + ```xml + + + + + + + + ``` + +A subscriber reads the `rel="hub"` link to find this endpoint and the `rel="self"` +link to learn the canonical topic URL, then subscribes as above. (Note: discovery is a +subscriber-side concern — this hub accepts explicit `hub.topic` subscriptions +regardless of how the feed advertises itself.) + +--- + +← [Back to the documentation index](../README.md) diff --git a/apps/server/services/markdown-doc.js b/apps/server/services/markdown-doc.js index ba799c2..8ee1902 100644 --- a/apps/server/services/markdown-doc.js +++ b/apps/server/services/markdown-doc.js @@ -1,12 +1,84 @@ const fs = require('fs'); const md = require('markdown-it')(); +// Rewrite one relative `.md` link to its in-app route using the docLinks map +// (keyed by the file's basename without extension), preserving any `#anchor`. +// External links, non-`.md` links, and basenames absent from the map are +// returned unchanged — so the same source links resolve to files on GitHub and +// to routes in the rendered docs, and unmapped targets (e.g. LICENSE.md, which +// has its own route) are left alone. +function rewriteDocLink(href, docLinks) { + if (/^[a-z][a-z0-9+.-]*:|^\/\//i.test(href)) { + return href; + } + const match = /^[^?#]*?([^/?#]+)\.md(#.*)?$/.exec(href); + if (match === null) { + return href; + } + const target = docLinks[match[1]]; + return target === undefined ? href : `${target}${match[2] ?? ''}`; +} + +// Slugify heading text the way GitHub does, so the in-app anchors match the +// `#fragment` links the Markdown uses (which GitHub also honours): lowercase, +// drop punctuation, spaces to hyphens. +function slugify(text) { + return text + .trim() + .toLowerCase() + .replace(/[^\w\- ]+/g, '') + .replace(/\s+/g, '-'); +} + +// A heading slug made unique within one document, mirroring GitHub's `-1`/`-2` +// suffixing for repeated headings. +function uniqueSlug(text, used) { + const base = slugify(text); + const seen = used.get(base) ?? 0; + used.set(base, seen + 1); + return seen === 0 ? base : `${base}-${seen}`; +} + +const renderToken = (tokens, idx, options, env, self) => + self.renderToken(tokens, idx, options); +const baseLinkOpen = md.renderer.rules.link_open ?? renderToken; + +// Rewrite GitHub-relative `.md` links to in-app routes (when a docLinks map is +// supplied via env). +md.renderer.rules.link_open = function(tokens, idx, options, env, self) { + const docLinks = env && env.docLinks; + if (docLinks) { + const hrefIndex = tokens[idx].attrIndex('href'); + if (hrefIndex >= 0) { + const attr = tokens[idx].attrs[hrefIndex]; + attr[1] = rewriteDocLink(attr[1], docLinks); + } + } + return baseLinkOpen(tokens, idx, options, env, self); +}; + +// Give every heading a stable id so `#fragment` links resolve in the rendered +// docs as well as on GitHub. +md.renderer.rules.heading_open = function(tokens, idx, options, env, self) { + if (!env.usedSlugs) { + env.usedSlugs = new Map(); + } + const inline = tokens[idx + 1]; + if (inline && inline.content) { + tokens[idx].attrSet('id', uniqueSlug(inline.content, env.usedSlugs)); + } + return self.renderToken(tokens, idx, options); +}; + // Render a Markdown file to HTML for the shared `docs` view. `stripH1` drops a // leading

— the README keeps its own "# rssCloud Server" title for GitHub, // but the docs page supplies its own heading, so the rendered H1 is redundant. -// Throws if the file can't be read; the caller maps that to a 500. -function renderMarkdownDoc(filePath, { stripH1 = false } = {}) { - const html = md.render(fs.readFileSync(filePath, { encoding: 'utf8' })); +// `docLinks` rewrites relative `.md` links to in-app routes (see +// {@link rewriteDocLink}). Throws if the file can't be read; the caller maps +// that to a 500. +function renderMarkdownDoc(filePath, { stripH1 = false, docLinks = null } = {}) { + const content = fs.readFileSync(filePath, { encoding: 'utf8' }); + const html = md.render(content, { docLinks }); return stripH1 ? html.replace(/]*>[\s\S]*?<\/h1>\s*/i, '') : html; diff --git a/apps/server/services/markdown-doc.test.js b/apps/server/services/markdown-doc.test.js index 265d20c..acbdcd6 100644 --- a/apps/server/services/markdown-doc.test.js +++ b/apps/server/services/markdown-doc.test.js @@ -34,3 +34,55 @@ test('retains the H1 by default', () => { test('throws when the file cannot be read', () => { assert.throws(() => renderMarkdownDoc('/no/such/file.md')); }); + +test('rewrites relative .md links to in-app routes via docLinks', () => { + const file = writeTemp('See [WebSub](docs/websub.md) for details.'); + const html = renderMarkdownDoc(file, { + docLinks: { websub: '/docs/websub' } + }); + assert.match(html, /href="\/docs\/websub"/); +}); + +test('preserves a #anchor when rewriting a .md link', () => { + const file = writeTemp('[fan-out](cross-protocol.md#fan-out)'); + const html = renderMarkdownDoc(file, { + docLinks: { 'cross-protocol': '/docs/cross-protocol' } + }); + assert.match(html, /href="\/docs\/cross-protocol#fan-out"/); +}); + +test('maps a parent-relative README link to /docs', () => { + const file = writeTemp('[Home](../README.md)'); + const html = renderMarkdownDoc(file, { docLinks: { README: '/docs' } }); + assert.match(html, /href="\/docs"/); +}); + +test('leaves unmapped and external links unchanged', () => { + const file = writeTemp( + '[license](LICENSE.md) [spec](https://www.w3.org/TR/websub/)' + ); + const html = renderMarkdownDoc(file, { + docLinks: { websub: '/docs/websub' } + }); + assert.match(html, /href="LICENSE.md"/); + assert.match(html, /href="https:\/\/www.w3.org\/TR\/websub\/"/); +}); + +test('leaves links untouched when no docLinks map is given', () => { + const file = writeTemp('[WebSub](docs/websub.md)'); + const html = renderMarkdownDoc(file); + assert.match(html, /href="docs\/websub.md"/); +}); + +test('gives headings GitHub-style ids so #fragment links resolve', () => { + const file = writeTemp('## POST /pleaseNotify\n\nbody'); + const html = renderMarkdownDoc(file); + assert.match(html, /

POST \/pleaseNotify<\/h2>/); +}); + +test('disambiguates repeated heading ids like GitHub', () => { + const file = writeTemp('## Response\n\na\n\n## Response\n\nb'); + const html = renderMarkdownDoc(file); + assert.match(html, /

/); + assert.match(html, /

/); +}); diff --git a/apps/server/views/home.handlebars b/apps/server/views/home.handlebars index 76d74d6..b5a3f0a 100644 --- a/apps/server/views/home.handlebars +++ b/apps/server/views/home.handlebars @@ -10,9 +10,17 @@

A notification protocol server that allows RSS feeds to notify subscribers when they are updated.

-

Available Pages

+

Documentation

+ + +

Tools

    -
  • Documentation
  • Please Notify Form
  • Ping Form
  • View Log
  • From b7bdf2fde87affbef337141481ad191f53989572 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 10:27:01 -0500 Subject: [PATCH 22/35] build(deps): bump vite override to >=8.0.16 vite 8.0.14 (a dev/test-only transitive dep via vitest) is in range for two advisories, both Windows-only dev-server issues: - GHSA-fx2h-pf6j-xcff (high): server.fs.deny bypass on Windows alt paths - GHSA-v6wh-96g9-6wx3 (medium): launch-editor NTLMv2 hash disclosure Raise the pnpm override floor from >=6.4.2 to >=8.0.16; resolves to 8.1.1. Co-Authored-By: Claude Opus 4.8 (1M context) --- package.json | 2 +- pnpm-lock.yaml | 237 +++++++++++++++++++++++++------------------------ 2 files changed, 124 insertions(+), 115 deletions(-) diff --git a/package.json b/package.json index 6283f3b..84eebcb 100644 --- a/package.json +++ b/package.json @@ -21,7 +21,7 @@ "overrides": { "serialize-javascript": ">=7.0.5", "qs": ">=6.15.2", - "vite": ">=6.4.2", + "vite": ">=8.0.16", "esbuild": ">=0.28.1", "diff": ">=8.0.3" }, diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index e15de54..7c31230 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -7,7 +7,7 @@ settings: overrides: serialize-javascript: '>=7.0.5' qs: '>=6.15.2' - vite: '>=6.4.2' + vite: '>=8.0.16' esbuild: '>=0.28.1' diff: '>=8.0.3' @@ -176,7 +176,7 @@ importers: version: 9.39.4(jiti@2.6.1) tsup: specifier: ^8.3.5 - version: 8.5.1(jiti@2.6.1)(postcss@8.5.15)(typescript@5.9.3) + version: 8.5.1(jiti@2.6.1)(postcss@8.5.16)(typescript@5.9.3) typescript: specifier: ^5.7.2 version: 5.9.3 @@ -219,7 +219,7 @@ importers: version: 7.2.2 tsup: specifier: ^8.3.5 - version: 8.5.1(jiti@2.6.1)(postcss@8.5.15)(typescript@5.9.3) + version: 8.5.1(jiti@2.6.1)(postcss@8.5.16)(typescript@5.9.3) typescript: specifier: ^5.7.2 version: 5.9.3 @@ -253,7 +253,7 @@ importers: version: 9.39.4(jiti@2.6.1) tsup: specifier: ^8.3.5 - version: 8.5.1(jiti@2.6.1)(postcss@8.5.15)(typescript@5.9.3) + version: 8.5.1(jiti@2.6.1)(postcss@8.5.16)(typescript@5.9.3) typescript: specifier: ^5.7.2 version: 5.9.3 @@ -376,14 +376,14 @@ packages: conventional-commits-parser: optional: true - '@emnapi/core@1.10.0': - resolution: {integrity: sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==} + '@emnapi/core@1.11.1': + resolution: {integrity: sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==} - '@emnapi/runtime@1.10.0': - resolution: {integrity: sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==} + '@emnapi/runtime@1.11.1': + resolution: {integrity: sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==} - '@emnapi/wasi-threads@1.2.1': - resolution: {integrity: sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==} + '@emnapi/wasi-threads@1.2.2': + resolution: {integrity: sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==} '@esbuild/aix-ppc64@0.28.1': resolution: {integrity: sha512-Svl7tq8k/08+p6CXPpRjQ1fKX+1odH/BQbb48fV6fj3CWHhsoIOoY87w1oHXm0qEpkIK3ZfVgp0hed3XBXzXMQ==} @@ -649,8 +649,8 @@ packages: '@jridgewell/trace-mapping@0.3.31': resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==} - '@napi-rs/wasm-runtime@1.1.4': - resolution: {integrity: sha512-3NQNNgA1YSlJb/kMH1ildASP9HW7/7kYnRI2szWJaofaS1hWmbGI4H+d3+22aGzXXN9IJ+n+GiFVcGipJP18ow==} + '@napi-rs/wasm-runtime@1.1.6': + resolution: {integrity: sha512-ZLv/JdUfkvOy9eCnnBaGfiO+XimbjebAeO+MRQqD/B+FR1tnRN0tpKSJHRbE8sFfS6aqsXZ67TQjfwfsxULVbg==} peerDependencies: '@emnapi/core': ^1.7.1 '@emnapi/runtime': ^1.7.1 @@ -659,8 +659,8 @@ packages: resolution: {integrity: sha512-jCs9ldd7NwzpgXDIf6P3+NrHh9/sD6CQdxHyjQI+h/6rDNo88ypBxxz45UDuZHz9r3tNz7N/VInSVoVdtXEI4A==} engines: {node: ^14.21.3 || >=16} - '@oxc-project/types@0.132.0': - resolution: {integrity: sha512-FESMOxil5Se014ui/Eq8fT5uHJo6nIRwH0PfJrZJXs6Gek3ZVFOrpUv3YIZT20m+extU98Hg1Ym72U58rlsxUQ==} + '@oxc-project/types@0.137.0': + resolution: {integrity: sha512-WT+Gb24i8hmvo85AIv2oEYouEXkRlKAlT9WaCa3TfLgNCN+GhrJOGZuIlMouAh38Qe4QOx26eUOVsq70qXrywA==} '@paralleldrive/cuid2@2.3.1': resolution: {integrity: sha512-XO7cAxhnTZl0Yggq6jOgjiOHhbgcO4NqFqwSmQpjK3b6TEE6Uj/jfSk6wzYyemh3+I0sHirKSetjQwn5cZktFw==} @@ -669,91 +669,91 @@ packages: resolution: {integrity: sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==} engines: {node: '>=14'} - '@rolldown/binding-android-arm64@1.0.2': - resolution: {integrity: sha512-ZS4D1JPGn/MYQN/SYDWftIE/nVsM8j/AFOYEzAoOE2O3NktQOZru+/vYXGbR/qtdLdIfGCP0lcoJiYVzsEz+iQ==} + '@rolldown/binding-android-arm64@1.1.3': + resolution: {integrity: sha512-DT6Z3PhvioeHMvxo+xHc3KtqggrI7CCTXCmC2h/5zUlp5jVitv7XEy+9q5/7v8IolhlioawpMo8Kg0EEBy7J0g==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@rolldown/binding-darwin-arm64@1.0.2': - resolution: {integrity: sha512-vdFA9+C/rekyGce7WqHs/xoT0ioZEWaOFyZLIV1mEeNFaFDUQrPIo8Vs2GvJ6eetb3rzDUtUBgzto3ExpXJB3w==} + '@rolldown/binding-darwin-arm64@1.1.3': + resolution: {integrity: sha512-0NwgwsjM7LrsuVnXMK3koTpagBNOhloc/BNjKqZjv4V5zI5r13qx69uVhRx+o5Z0yy4Hzq+lpy7TAgUG/ocvrw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@rolldown/binding-darwin-x64@1.0.2': - resolution: {integrity: sha512-BewSOwTHazv77DTYiAZXSqqKZ4KP/KonFisDMVU7PImxoWfB2aepnPhd2E4SWz3zDzYgDNbs6jBmTdgNnF02GA==} + '@rolldown/binding-darwin-x64@1.1.3': + resolution: {integrity: sha512-YtiBp4disu6V560loT6PjMdiRaWmVvDNrUunAalbiFx2ggeJwxdAsgZMcoGP17uyAsTwAj5V1niksxlHnVQ1Sw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@rolldown/binding-freebsd-x64@1.0.2': - resolution: {integrity: sha512-m41o7M0YWtUdqk61Tb+jnKb2rN++iRdIASlExkUoKfIAH30DOHCB8fVLzSUpbWHHU8esmEioY62PxzexE8MBuA==} + '@rolldown/binding-freebsd-x64@1.1.3': + resolution: {integrity: sha512-yD3EkEdXk2LypPxnf/kSZHirarsI8gcPzc62SukhR9VJTyvV+F9Q/GxWNuCojc7sXyuVC4DxRGhdDK4X8VSsbw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@rolldown/binding-linux-arm-gnueabihf@1.0.2': - resolution: {integrity: sha512-jcojB9H7W/jS29pMKWAK1N+fU99vXodHDTatS3b3y/XSOCiHo0kkA74pL3jJmkoQtYpOCxDvaKs1fo2Ij/1X5w==} + '@rolldown/binding-linux-arm-gnueabihf@1.1.3': + resolution: {integrity: sha512-c+8vieQbsD7HNAHKIA34w0GJ9FedFFuJGD+7E6vz7Q3uqAIugL5p45fhlsj4UaAsHpcmlqugBWMhA0/j7o0sIg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@rolldown/binding-linux-arm64-gnu@1.0.2': - resolution: {integrity: sha512-1jn6qDU5iiOgFgygDzKUuKP0maTi0/f1+sBLgvij/76C77Nm3ts6ufz9Bjg5q5dduxiUIxtq86JIoBvo1xQ4Ig==} + '@rolldown/binding-linux-arm64-gnu@1.1.3': + resolution: {integrity: sha512-50jD0uUwLvur7Zz9LHz17kaAdTPjn5wN93hEgjvmYFRZwiR7ZJYovTd5ipyWJDAnXKvZ+wgc+/Ika6dwSF5OcA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] - '@rolldown/binding-linux-arm64-musl@1.0.2': - resolution: {integrity: sha512-QVLO/czFMdoMFSqlX3bcswcJNm/23r+qoa/jgtmFc/qEp6/jXmIkDjF/XIo8dPfGaiwy1xfQn8o77L79GeXFgw==} + '@rolldown/binding-linux-arm64-musl@1.1.3': + resolution: {integrity: sha512-BO9+oPL8K9poZJBfYPsXNtYjPE5uM3qeehT3aFcW4LITOl+iSqhp0abzjR2nWBUNjIZeKXjAEWBZ64WjNoHd6w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] - '@rolldown/binding-linux-ppc64-gnu@1.0.2': - resolution: {integrity: sha512-hgO5Abm0w5UL6FEa2iFnZqo2KlK7TQ5QhV5x09hujBf7t5KzHQ1VmfPuTpqRy/rNlSxua3eWH374xxiVrP+lcA==} + '@rolldown/binding-linux-ppc64-gnu@1.1.3': + resolution: {integrity: sha512-f3VpLB1vQ0Eo6ecr/6cekLnvYMFF4YBFoVGkfkvPLq1bAkbAwHYQPZKoAmG6OJyTcxxoC+AvezGx/S1obNC0Mw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] - '@rolldown/binding-linux-s390x-gnu@1.0.2': - resolution: {integrity: sha512-fy8rXxuYEu602abC8MUNaPjYLIFzReOaEIEMKMUa0rFEUxNpVXhs15KSSQ4qlqSaM7B6rcj9rDZgADh/IGDzLQ==} + '@rolldown/binding-linux-s390x-gnu@1.1.3': + resolution: {integrity: sha512-AmurZ26Pqx/RI9N1gzEOCklkKXl927yjfXWUUS0O7Puh8ARM/Ob8qfrD3qnWksScdw6cSrW5PSHE9DyLu7+PtA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] - '@rolldown/binding-linux-x64-gnu@1.0.2': - resolution: {integrity: sha512-0+bOkiQ779+r1WpoHOWHqncvyySci0vKph+myNDYb+im6meJAzHQXay6oEgnkHuUGouM1LKTZwqKpBow6Kj7CQ==} + '@rolldown/binding-linux-x64-gnu@1.1.3': + resolution: {integrity: sha512-JJpqs8bRGITDOdbkNKnlojzBabbOHrqjSvDr0IVsZObE1lBcPjxItUEY9eWIDbxaJ3cGrXPWGfGkIxFijg/URg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] - '@rolldown/binding-linux-x64-musl@1.0.2': - resolution: {integrity: sha512-mjSkrzZK5Qsl0a9d1JgILOiuZOSDTVdKENcSXBoqbzSrspLR/4/IRVDo5wd2GgZjNss/viBFJdeq+j7qH2nypw==} + '@rolldown/binding-linux-x64-musl@1.1.3': + resolution: {integrity: sha512-rSJcdjPxzA/by/6/rYs+v+bXU7UjvnbUWz8MJb6kh6+knqB1dCrtHg0uu7C/4haqJvqdkYHQ5IGn+tCH9GLW/g==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] - '@rolldown/binding-openharmony-arm64@1.0.2': - resolution: {integrity: sha512-1v5vHasdfQAZoEHakBV72LIFAC9JjnymsiKxp+GEr/ma3+NJCPSaYK+qavInOovJkgwFrs7GccX2d6IgDA3Z5w==} + '@rolldown/binding-openharmony-arm64@1.1.3': + resolution: {integrity: sha512-hQ3/PYkDJICgevvyNcVrihVeqq7k1Pp3VZ9lY+dauAYUJKO+auqApvANhvR1An9BhmqYKvW2Mu1F9u4DXSMLxQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@rolldown/binding-wasm32-wasi@1.0.2': - resolution: {integrity: sha512-mb1VobWn6NheziTk5/WEaR6AKVbrwT5sOi6C7zk3gy/pD1qtJfU1j4PgTo2NJnOtbL9Dl3Aeei8w9jJ7qC2jZQ==} + '@rolldown/binding-wasm32-wasi@1.1.3': + resolution: {integrity: sha512-Elcv/BtML9lXrV6JuKITc/grN2kYV9gjsQpW8Jfw4ioK0TOkjBjye0nnyqQNy9STNaI20lXNaQBRrD5gSgR0Yg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [wasm32] - '@rolldown/binding-win32-arm64-msvc@1.0.2': - resolution: {integrity: sha512-SqKonF56vA/L2yHwHYcEp2P34URpOZ7d1fS635cTkpDnUtEGdUbhI6NzsPdqeSWvAAeGDrxjWjNmibDIdFf9/A==} + '@rolldown/binding-win32-arm64-msvc@1.1.3': + resolution: {integrity: sha512-2DrEfhluH9yhiaFApmsjsjwrSYbNcY1oFTzYSP1a535jDbV98zCFanA/96TBUd0iDFcxGmw9QRExwGCXz3U+/g==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@rolldown/binding-win32-x64-msvc@1.0.2': - resolution: {integrity: sha512-v7qRI7gXLRINcOGXt+7YmAZ6iFuyZVMIoXAxhd8oP+DR9dLfL9GfNIx7PLMxmhZdvq8waUJBQiWN9EKNy+TRBQ==} + '@rolldown/binding-win32-x64-msvc@1.1.3': + resolution: {integrity: sha512-OL4OMk7UPXOeVGGd3qo5zJyPIljf4AFgk5QAkPPS+OoLuOOozhuaQGC18MxVTnw/06q93gShAJzlwnSCY9YtqA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] @@ -924,8 +924,8 @@ packages: cpu: [arm64] os: [win32] - '@tybys/wasm-util@0.10.2': - resolution: {integrity: sha512-RoBvJ2X0wuKlWFIjrwffGw1IqZHKQqzIchKaadZZfnNpsAYp2mM0h36JtPCjNDAHGgYez/15uMBpfGwchhiMgg==} + '@tybys/wasm-util@0.10.3': + resolution: {integrity: sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg==} '@types/body-parser@1.19.6': resolution: {integrity: sha512-HLFeCYgz89uk22N5Qg3dvGvsv46B8GLvKKo1zKG4NybA8U2DiEO3w9lqGg29t/tfLRJpJ6iQxnVw4OnB7MoM9g==} @@ -1080,7 +1080,7 @@ packages: resolution: {integrity: sha512-EZOrpDbkKotFAP7wPAQV1UIyoGOk4oX7ynWhBhLB7v+meMHbQhU16oPpIYGTTe4oFlhpryGpgpcZP/sin3hYuw==} peerDependencies: msw: ^2.4.9 - vite: '>=6.4.2' + vite: '>=8.0.16' peerDependenciesMeta: msw: optional: true @@ -2306,8 +2306,8 @@ packages: yaml: optional: true - postcss@8.5.15: - resolution: {integrity: sha512-FfR8sjd4em2T6fb3I2MwAJU7HWVMr9zba+enmQeeWFfCbm+UOC/0X4DS8XtpUTMwWMGbjKYP7xjfNekzyGmB3A==} + postcss@8.5.16: + resolution: {integrity: sha512-vuwillviilfKZsg0VGj5R/YwwcHx4SLsIOI/7K6mQkWx+l5cUHTjj5g0AasTBcyXsbfTgrwsUNmVUb5xVwyPwg==} engines: {node: ^10 || ^12 || >=14} prelude-ls@1.2.1: @@ -2374,8 +2374,8 @@ packages: resolution: {integrity: sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw==} engines: {node: '>=8'} - rolldown@1.0.2: - resolution: {integrity: sha512-oZx5zVDtVB44AW3eaifgDml1gWRDZGvjcfdxonE4swNPG98PrrXjaO/KrnUjzlMnztCCRVlUueA1kCXhARGk6g==} + rolldown@1.1.3: + resolution: {integrity: sha512-1F1eEtUBtFvcGm1HQ9TiUIUHPQG7mSAODrhIzjxoUEFuo8OcbrGLiVLkevNgj84TE4lnHvnumwFjhJO5Eu135g==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true @@ -2552,6 +2552,10 @@ packages: resolution: {integrity: sha512-pn99VhoACYR8nFHhxqix+uvsbXineAasWm5ojXoN8xEwK5Kd3/TrhNn1wByuD52UxWRLy8pu+kRMniEi6Eq9Zg==} engines: {node: '>=12.0.0'} + tinyglobby@0.2.17: + resolution: {integrity: sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==} + engines: {node: '>=12.0.0'} + tinypool@1.1.1: resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==} engines: {node: ^18.0.0 || >=20.0.0} @@ -2683,13 +2687,13 @@ packages: engines: {node: ^18.0.0 || ^20.0.0 || >=22.0.0} hasBin: true - vite@8.0.14: - resolution: {integrity: sha512-s4BJJ+5y1pYL6Otw51FHhVJQhPnuRinKig64g/1+EUNaJsd3gCKdD31IPFvswUgW9/60QT9oFHbZHbQK5imcxw==} + vite@8.1.1: + resolution: {integrity: sha512-X/05/cT+VITy2AeDc1der6smvGWWREtL4hPbPTaVbjSBuuWkmNOjR6HP3NzqcQA2nF6VHGUPaFRJyft/2AE9Kg==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true peerDependencies: '@types/node': ^20.19.0 || >=22.12.0 - '@vitejs/devtools': ^0.1.18 + '@vitejs/devtools': ^0.3.0 esbuild: '>=0.28.1' jiti: '>=1.21.0' less: ^4.0.0 @@ -2979,18 +2983,18 @@ snapshots: optionalDependencies: conventional-commits-parser: 6.4.0 - '@emnapi/core@1.10.0': + '@emnapi/core@1.11.1': dependencies: - '@emnapi/wasi-threads': 1.2.1 + '@emnapi/wasi-threads': 1.2.2 tslib: 2.8.1 optional: true - '@emnapi/runtime@1.10.0': + '@emnapi/runtime@1.11.1': dependencies: tslib: 2.8.1 optional: true - '@emnapi/wasi-threads@1.2.1': + '@emnapi/wasi-threads@1.2.2': dependencies: tslib: 2.8.1 optional: true @@ -3192,16 +3196,16 @@ snapshots: '@jridgewell/resolve-uri': 3.1.2 '@jridgewell/sourcemap-codec': 1.5.5 - '@napi-rs/wasm-runtime@1.1.4(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0)': + '@napi-rs/wasm-runtime@1.1.6(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.1)': dependencies: - '@emnapi/core': 1.10.0 - '@emnapi/runtime': 1.10.0 - '@tybys/wasm-util': 0.10.2 + '@emnapi/core': 1.11.1 + '@emnapi/runtime': 1.11.1 + '@tybys/wasm-util': 0.10.3 optional: true '@noble/hashes@1.8.0': {} - '@oxc-project/types@0.132.0': {} + '@oxc-project/types@0.137.0': {} '@paralleldrive/cuid2@2.3.1': dependencies: @@ -3210,53 +3214,53 @@ snapshots: '@pkgjs/parseargs@0.11.0': optional: true - '@rolldown/binding-android-arm64@1.0.2': + '@rolldown/binding-android-arm64@1.1.3': optional: true - '@rolldown/binding-darwin-arm64@1.0.2': + '@rolldown/binding-darwin-arm64@1.1.3': optional: true - '@rolldown/binding-darwin-x64@1.0.2': + '@rolldown/binding-darwin-x64@1.1.3': optional: true - '@rolldown/binding-freebsd-x64@1.0.2': + '@rolldown/binding-freebsd-x64@1.1.3': optional: true - '@rolldown/binding-linux-arm-gnueabihf@1.0.2': + '@rolldown/binding-linux-arm-gnueabihf@1.1.3': optional: true - '@rolldown/binding-linux-arm64-gnu@1.0.2': + '@rolldown/binding-linux-arm64-gnu@1.1.3': optional: true - '@rolldown/binding-linux-arm64-musl@1.0.2': + '@rolldown/binding-linux-arm64-musl@1.1.3': optional: true - '@rolldown/binding-linux-ppc64-gnu@1.0.2': + '@rolldown/binding-linux-ppc64-gnu@1.1.3': optional: true - '@rolldown/binding-linux-s390x-gnu@1.0.2': + '@rolldown/binding-linux-s390x-gnu@1.1.3': optional: true - '@rolldown/binding-linux-x64-gnu@1.0.2': + '@rolldown/binding-linux-x64-gnu@1.1.3': optional: true - '@rolldown/binding-linux-x64-musl@1.0.2': + '@rolldown/binding-linux-x64-musl@1.1.3': optional: true - '@rolldown/binding-openharmony-arm64@1.0.2': + '@rolldown/binding-openharmony-arm64@1.1.3': optional: true - '@rolldown/binding-wasm32-wasi@1.0.2': + '@rolldown/binding-wasm32-wasi@1.1.3': dependencies: - '@emnapi/core': 1.10.0 - '@emnapi/runtime': 1.10.0 - '@napi-rs/wasm-runtime': 1.1.4(@emnapi/core@1.10.0)(@emnapi/runtime@1.10.0) + '@emnapi/core': 1.11.1 + '@emnapi/runtime': 1.11.1 + '@napi-rs/wasm-runtime': 1.1.6(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.1) optional: true - '@rolldown/binding-win32-arm64-msvc@1.0.2': + '@rolldown/binding-win32-arm64-msvc@1.1.3': optional: true - '@rolldown/binding-win32-x64-msvc@1.0.2': + '@rolldown/binding-win32-x64-msvc@1.1.3': optional: true '@rolldown/pluginutils@1.0.1': {} @@ -3360,7 +3364,7 @@ snapshots: '@turbo/windows-arm64@2.9.14': optional: true - '@tybys/wasm-util@0.10.2': + '@tybys/wasm-util@0.10.3': dependencies: tslib: 2.8.1 optional: true @@ -3579,13 +3583,13 @@ snapshots: chai: 5.3.3 tinyrainbow: 2.0.0 - '@vitest/mocker@3.2.6(vite@8.0.14(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1))': + '@vitest/mocker@3.2.6(vite@8.1.1(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1))': dependencies: '@vitest/spy': 3.2.6 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: - vite: 8.0.14(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1) + vite: 8.1.1(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1) '@vitest/pretty-format@3.2.6': dependencies: @@ -4857,14 +4861,14 @@ snapshots: mlly: 1.8.2 pathe: 2.0.3 - postcss-load-config@6.0.1(jiti@2.6.1)(postcss@8.5.15): + postcss-load-config@6.0.1(jiti@2.6.1)(postcss@8.5.16): dependencies: lilconfig: 3.1.3 optionalDependencies: jiti: 2.6.1 - postcss: 8.5.15 + postcss: 8.5.16 - postcss@8.5.15: + postcss@8.5.16: dependencies: nanoid: 3.3.12 picocolors: 1.1.1 @@ -4919,26 +4923,26 @@ snapshots: resolve-from@5.0.0: {} - rolldown@1.0.2: + rolldown@1.1.3: dependencies: - '@oxc-project/types': 0.132.0 + '@oxc-project/types': 0.137.0 '@rolldown/pluginutils': 1.0.1 optionalDependencies: - '@rolldown/binding-android-arm64': 1.0.2 - '@rolldown/binding-darwin-arm64': 1.0.2 - '@rolldown/binding-darwin-x64': 1.0.2 - '@rolldown/binding-freebsd-x64': 1.0.2 - '@rolldown/binding-linux-arm-gnueabihf': 1.0.2 - '@rolldown/binding-linux-arm64-gnu': 1.0.2 - '@rolldown/binding-linux-arm64-musl': 1.0.2 - '@rolldown/binding-linux-ppc64-gnu': 1.0.2 - '@rolldown/binding-linux-s390x-gnu': 1.0.2 - '@rolldown/binding-linux-x64-gnu': 1.0.2 - '@rolldown/binding-linux-x64-musl': 1.0.2 - '@rolldown/binding-openharmony-arm64': 1.0.2 - '@rolldown/binding-wasm32-wasi': 1.0.2 - '@rolldown/binding-win32-arm64-msvc': 1.0.2 - '@rolldown/binding-win32-x64-msvc': 1.0.2 + '@rolldown/binding-android-arm64': 1.1.3 + '@rolldown/binding-darwin-arm64': 1.1.3 + '@rolldown/binding-darwin-x64': 1.1.3 + '@rolldown/binding-freebsd-x64': 1.1.3 + '@rolldown/binding-linux-arm-gnueabihf': 1.1.3 + '@rolldown/binding-linux-arm64-gnu': 1.1.3 + '@rolldown/binding-linux-arm64-musl': 1.1.3 + '@rolldown/binding-linux-ppc64-gnu': 1.1.3 + '@rolldown/binding-linux-s390x-gnu': 1.1.3 + '@rolldown/binding-linux-x64-gnu': 1.1.3 + '@rolldown/binding-linux-x64-musl': 1.1.3 + '@rolldown/binding-openharmony-arm64': 1.1.3 + '@rolldown/binding-wasm32-wasi': 1.1.3 + '@rolldown/binding-win32-arm64-msvc': 1.1.3 + '@rolldown/binding-win32-x64-msvc': 1.1.3 rollup@4.60.4: dependencies: @@ -5176,6 +5180,11 @@ snapshots: fdir: 6.5.0(picomatch@4.0.4) picomatch: 4.0.4 + tinyglobby@0.2.17: + dependencies: + fdir: 6.5.0(picomatch@4.0.4) + picomatch: 4.0.4 + tinypool@1.1.1: {} tinyrainbow@2.0.0: {} @@ -5201,7 +5210,7 @@ snapshots: tslib@2.8.1: optional: true - tsup@8.5.1(jiti@2.6.1)(postcss@8.5.15)(typescript@5.9.3): + tsup@8.5.1(jiti@2.6.1)(postcss@8.5.16)(typescript@5.9.3): dependencies: bundle-require: 5.1.0(esbuild@0.28.1) cac: 6.7.14 @@ -5212,7 +5221,7 @@ snapshots: fix-dts-default-cjs-exports: 1.0.1 joycon: 3.1.1 picocolors: 1.1.1 - postcss-load-config: 6.0.1(jiti@2.6.1)(postcss@8.5.15) + postcss-load-config: 6.0.1(jiti@2.6.1)(postcss@8.5.16) resolve-from: 5.0.0 rollup: 4.60.4 source-map: 0.7.6 @@ -5221,7 +5230,7 @@ snapshots: tinyglobby: 0.2.16 tree-kill: 1.2.2 optionalDependencies: - postcss: 8.5.15 + postcss: 8.5.16 typescript: 5.9.3 transitivePeerDependencies: - jiti @@ -5297,7 +5306,7 @@ snapshots: debug: 4.4.3 es-module-lexer: 1.7.0 pathe: 2.0.3 - vite: 8.0.14(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1) + vite: 8.1.1(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1) transitivePeerDependencies: - '@types/node' - '@vitejs/devtools' @@ -5313,13 +5322,13 @@ snapshots: - tsx - yaml - vite@8.0.14(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1): + vite@8.1.1(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1): dependencies: lightningcss: 1.32.0 picomatch: 4.0.4 - postcss: 8.5.15 - rolldown: 1.0.2 - tinyglobby: 0.2.16 + postcss: 8.5.16 + rolldown: 1.1.3 + tinyglobby: 0.2.17 optionalDependencies: '@types/node': 22.19.19 esbuild: 0.28.1 @@ -5330,7 +5339,7 @@ snapshots: dependencies: '@types/chai': 5.2.3 '@vitest/expect': 3.2.6 - '@vitest/mocker': 3.2.6(vite@8.0.14(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1)) + '@vitest/mocker': 3.2.6(vite@8.1.1(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1)) '@vitest/pretty-format': 3.2.6 '@vitest/runner': 3.2.6 '@vitest/snapshot': 3.2.6 @@ -5348,7 +5357,7 @@ snapshots: tinyglobby: 0.2.16 tinypool: 1.1.1 tinyrainbow: 2.0.0 - vite: 8.0.14(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1) + vite: 8.1.1(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1) vite-node: 3.2.4(@types/node@22.19.19)(esbuild@0.28.1)(jiti@2.6.1) why-is-node-running: 2.3.0 optionalDependencies: From 34a2d126b16f96f415e914aaa8b8ea6d690c2362 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 11:51:40 -0500 Subject: [PATCH 23/35] feat(core): add SSRF egress guard for outbound fetches Subscriber- and publisher-supplied URLs (hub.callback, hub.topic/url) are fetched and delivered to with no host check. Because WebSub content distribution relays a fetched body to the callback, that is a full-read SSRF: name an internal hub.topic, receive its response at a callback you control. Add createSafeFetch: a fetch that refuses non-http(s) schemes and screens the destination against classifyBlockedAddress (loopback, private, link-local/metadata, unique-local, CGNAT; IPv4-mapped decoded). It is rebinding-safe -- a custom DNS lookup pins the connection to the validated address, and a custom connector screens IP-literal hosts (which skip DNS entirely), both re-firing on every redirect hop. createCidrAllowList exempts operator-listed ranges. Adds undici (Agent + connector, to pin the socket) and ipaddr.js (range classification) as core deps. 100% coverage maintained. See ADR-0003. Co-Authored-By: Claude Opus 4.8 (1M context) --- CONTEXT.md | 13 + ...003-ssrf-egress-guard-on-outbound-fetch.md | 71 +++++ packages/core/package.json | 2 + packages/core/src/index.ts | 8 + packages/core/src/safe-fetch.test.ts | 296 ++++++++++++++++++ packages/core/src/safe-fetch.ts | 225 +++++++++++++ pnpm-lock.yaml | 18 ++ 7 files changed, 633 insertions(+) create mode 100644 docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md create mode 100644 packages/core/src/safe-fetch.test.ts create mode 100644 packages/core/src/safe-fetch.ts diff --git a/CONTEXT.md b/CONTEXT.md index 749e2f4..d1bab70 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -196,6 +196,19 @@ delivery. The algorithm is a config knob (default `sha256`); no `hub.secret` → _Avoid_: HMAC (that's the algorithm; the header is the wire artifact), auth token, signature (ambiguous — name the header). +**Egress guard** (`createSafeFetch`): +The screen on every outbound fetch — **Topic** re-fetch, **Intent verification** GET, and +**Content distribution** — that refuses a destination resolving to a non-public address +(loopback, private, link-local incl. cloud-metadata, ULA, CGNAT) or a non-`http(s)` scheme. +It exists because **Callback** and **Topic** are attacker-supplied, and Content distribution +relays a fetched body to the Callback (an SSRF exfiltration path absent from rssCloud's +URL-only **Notification**). Screens the *resolved IP* and pins the connection to it, so a +rebinding name or redirect that points inward is refused on every hop (see +[ADR-0003](docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md)). On by default; LAN feeds +opt in via `WEBSUB_FETCH_ALLOW_CIDRS`. +_Avoid_: firewall (that's the network layer), allowlist (the guard is deny-by-default; the +CIDR list is only the exemption), sanitizer. + ## Example dialogue > **Dev:** When a `pleaseNotify` comes in over XML-RPC, who decides the callback is `diffDomain`? diff --git a/docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md b/docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md new file mode 100644 index 0000000..8b84c0e --- /dev/null +++ b/docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md @@ -0,0 +1,71 @@ +# Outbound fetches are screened by an SSRF egress guard pinned to the resolved IP + +The hub fetches and delivers to URLs supplied by untrusted clients. A subscriber's +`hub.callback` (where WebSub content distribution **sends the fetched feed body**) and a +publisher's `hub.topic` / `hub.url` (which the hub **fetches**) are both attacker-controlled +and were validated only for shape (absolute URL / non-empty string). With WebSub content +distribution, that turned a pre-existing *blind* fetch of an arbitrary URL into a *full-read* +SSRF: an attacker subscribes a callback they control, names an internal `hub.topic` (e.g. +`http://169.254.169.254/…`), and the hub relays the internal response body to the callback. +This ADR records the egress guard added to close it. + +## Status + +accepted + +## Decision + +1. **Deny by resolved IP, not by URL text.** A public hub must accept arbitrary public feed + URLs, so an allowlist of topics is unworkable. Instead we enumerate the destinations the + hub must never reach and refuse them: an outbound request is blocked when its host + resolves to a non-public range (loopback, private, link-local incl. cloud-metadata + `169.254.169.254`, IPv6 unique-local, CGNAT, unspecified) or its scheme is not + `http`/`https`. Classification is on the resolved IP (via `ipaddr.js`), decoding + IPv4-mapped IPv6 first, so a hostname that points inward — or an IP-literal in disguise — + is caught. + +2. **Rebinding-safe: pin the connection to the validated address.** The guard is a + `createSafeFetch` wrapper over undici. A custom DNS lookup screens every resolved address + and the connection is pinned to that address (undici does not re-resolve), and a custom + connector screens IP-literal hosts (which skip DNS entirely). Both fire on every + connection the dispatcher opens, so each **redirect hop** is re-screened and there is no + resolve-then-connect TOCTOU window. + +3. **One guard, injected everywhere outbound.** `createSafeFetch` lives in `@rsscloud/core` + and is injected as the `fetch` for the engine's topic re-fetch **and** every protocol + plugin's deliveries / verification GETs. So topic fetch, the WebSub challenge GET, WebSub + content delivery, and the rssCloud REST/XML-RPC notifies are all covered uniformly — the + pre-existing rssCloud blind fetch is hardened for free. + +4. **Secure by default, with an operator escape hatch.** Protection is **on** by default + (`WEBSUB_SSRF_PROTECTION`); a hub that legitimately serves feeds on a private LAN exempts + specific ranges via `WEBSUB_FETCH_ALLOW_CIDRS` rather than disabling protection wholesale. + Local dev and the e2e suite (whose targets are loopback / private Docker IPs) keep working + by allowlisting those ranges — the e2e suite runs with protection on so the guarded fetch + is exercised end-to-end. + +## Why connector-level, not just a custom lookup + +undici's `connect.lookup` hook is the obvious place to screen DNS, but it is **skipped for +IP-literal hosts** — undici connects straight to a literal address without resolving — so a +lookup-only guard lets `http://169.254.169.254/` (the headline payload) straight through. +Screening must therefore also happen at the connector, which runs for every connection +regardless of how the host was specified. The lookup handles hostnames (and pins their +resolved IP); the connector handles literals; together they cover the initial request and +every redirect. + +## Consequences + +- `@rsscloud/core` gains two runtime dependencies — `undici` (the Agent + connector needed to + pin the socket; the same implementation as the platform's global `fetch`) and `ipaddr.js` + (range classification). The guard is injected, so a consumer that never builds it pays only + the install. +- A blocked request surfaces through existing error handling: the topic re-fetch reports a + read failure (`RESOURCE_READ_FAILED`), and a blocked callback counts as a failed delivery + (`notifyFailed`). Nothing new is thrown to the front door. +- A hub deployed on a network whose feeds live on private addresses must opt those ranges in + via `WEBSUB_FETCH_ALLOW_CIDRS`, or fetches to them will be refused. This is the intended + secure-by-default trade-off. +- The highest-value target is cloud instance metadata; operators should still defend it at + the infrastructure layer too (e.g. IMDSv2 with a hop limit), so the guard is defence in + depth rather than the only control. diff --git a/packages/core/package.json b/packages/core/package.json index 8c5b231..0dc198c 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -45,6 +45,8 @@ "sideEffects": false, "dependencies": { "@rsscloud/xml-rpc": "workspace:*", + "ipaddr.js": "^2.4.0", + "undici": "^7.28.0", "xml2js": "^0.6.2" }, "scripts": { diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index ff21eb3..2d5c17a 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -42,6 +42,14 @@ export { createWebSubProtocolPlugin, type WebSubProtocolPluginOptions } from './protocols/websub-plugin.js'; +export { + createSafeFetch, + createCidrAllowList, + classifyBlockedAddress, + SsrfBlockedError, + type SafeFetchOptions, + type GuardedLookupFn +} from './safe-fetch.js'; export { createDefaultFeedParser, type DefaultFeedParserOptions diff --git a/packages/core/src/safe-fetch.test.ts b/packages/core/src/safe-fetch.test.ts new file mode 100644 index 0000000..05214f5 --- /dev/null +++ b/packages/core/src/safe-fetch.test.ts @@ -0,0 +1,296 @@ +import { describe, expect, it, vi } from 'vitest'; +import { + classifyBlockedAddress, + createCidrAllowList, + createSafeFetch, + SsrfBlockedError, + type GuardedLookupFn, + type SafeFetchOptions +} from './safe-fetch.js'; + +type LooseConnector = ( + options: { hostname: string }, + callback: (err: Error | null, socket: unknown) => void +) => void; + +/** + * Build a `createSafeFetch` instance with the undici layers stubbed so the guard + * can be exercised without a socket: `buildConnector` captures the validating DNS + * lookup (the hostname path) and returns a recording base connector, while + * `agentFactory` captures the guarded connector (the IP-literal path). No + * `baseFetch` is supplied, so the undici-fetch default is selected (never called). + */ +function buildSafe(opts: Pick): { + base: ReturnType; + guardedLookup: GuardedLookupFn; + guardedConnector: LooseConnector; +} { + const base = vi.fn( + (_options: unknown, callback: (e: Error | null, s: unknown) => void) => + callback(null, {}) + ); + let guardedLookup: GuardedLookupFn | undefined; + let guardedConnector: LooseConnector | undefined; + createSafeFetch({ + ...opts, + buildConnector: ((buildOptions?: { lookup?: unknown }) => { + guardedLookup = buildOptions?.lookup as GuardedLookupFn; + return base as never; + }) as unknown as NonNullable, + agentFactory: connector => { + guardedConnector = connector as unknown as LooseConnector; + return {} as never; + } + }); + if (guardedLookup === undefined || guardedConnector === undefined) { + throw new Error('createSafeFetch did not build the guard'); + } + return { base, guardedLookup, guardedConnector }; +} + +describe('classifyBlockedAddress', () => { + it('flags an IPv4 loopback address with its range name', () => { + expect(classifyBlockedAddress('127.0.0.1')).toBe('loopback'); + }); + + it.each([ + ['127.0.0.1', 'loopback'], + ['10.0.0.1', 'private'], + ['172.16.0.1', 'private'], + ['192.168.1.1', 'private'], + ['169.254.169.254', 'linkLocal'], // cloud metadata endpoint + ['100.64.0.1', 'carrierGradeNat'], + ['0.0.0.0', 'unspecified'], + ['::1', 'loopback'], + ['fe80::1', 'linkLocal'], + ['fc00::1', 'uniqueLocal'], + ['::ffff:10.0.0.1', 'private'] // IPv4-mapped private, decoded + ])('blocks the non-public address %s as %s', (ip, reason) => { + expect(classifyBlockedAddress(ip)).toBe(reason); + }); + + it.each([ + ['8.8.8.8'], + ['1.1.1.1'], + ['2606:4700:4700::1111'], + ['::ffff:8.8.8.8'] // IPv4-mapped public, decoded + ])('allows the public unicast address %s', ip => { + expect(classifyBlockedAddress(ip)).toBeNull(); + }); +}); + +describe('createCidrAllowList', () => { + it('permits an address inside a configured CIDR', () => { + const allow = createCidrAllowList(['10.0.0.0/8']); + expect(allow('10.1.2.3')).toBe(true); + }); + + it('rejects an address outside every configured CIDR', () => { + const allow = createCidrAllowList(['10.0.0.0/8']); + expect(allow('192.168.1.1')).toBe(false); + }); + + it('matches against any of several configured CIDRs', () => { + const allow = createCidrAllowList(['10.0.0.0/8', '192.168.0.0/16']); + expect(allow('192.168.5.5')).toBe(true); + }); + + it('supports IPv6 CIDRs', () => { + const allow = createCidrAllowList(['fc00::/7']); + expect(allow('fc00::1234')).toBe(true); + }); + + it('does not match an address of a different family than the CIDR', () => { + const allow = createCidrAllowList(['fc00::/7']); + expect(allow('10.1.2.3')).toBe(false); + }); + + it('matches an IPv4-mapped address against an IPv4 CIDR', () => { + const allow = createCidrAllowList(['10.0.0.0/8']); + expect(allow('::ffff:10.1.2.3')).toBe(true); + }); + + it('permits nothing when the list is empty', () => { + const allow = createCidrAllowList([]); + expect(allow('10.1.2.3')).toBe(false); + }); +}); + +describe('createSafeFetch', () => { + it('rejects a non-http(s) scheme without calling the base fetch', async () => { + const baseFetch = vi.fn(); + const safeFetch = createSafeFetch({ + baseFetch: baseFetch as unknown as typeof fetch + }); + + await expect(safeFetch('file:///etc/passwd')).rejects.toThrow( + /http/i + ); + expect(baseFetch).not.toHaveBeenCalled(); + }); + + it('delegates an http(s) request to the base fetch with the pinning dispatcher', async () => { + const sentinel = {} as never; + const baseFetch = vi.fn(async () => new Response('ok')); + const safeFetch = createSafeFetch({ + baseFetch: baseFetch as unknown as typeof fetch, + agentFactory: () => sentinel, + lookup: () => {} + }); + + await safeFetch('https://feed.example/rss', { method: 'GET' }); + + expect(baseFetch).toHaveBeenCalledWith( + 'https://feed.example/rss', + expect.objectContaining({ method: 'GET', dispatcher: sentinel }) + ); + }); + + it('parses the URL from a Request object input', async () => { + const baseFetch = vi.fn(async () => new Response('ok')); + const safeFetch = createSafeFetch({ + baseFetch: baseFetch as unknown as typeof fetch, + agentFactory: () => ({}) as never, + lookup: () => {} + }); + + await safeFetch(new Request('https://feed.example/rss')); + + expect(baseFetch).toHaveBeenCalledTimes(1); + }); + + // --- hostname targets: screened during DNS resolution by the guarded lookup --- + + it('blocks a hostname that resolves to an internal address', () => { + const { guardedLookup } = buildSafe({ + lookup: (_hostname, _options, cb) => + cb(null, [{ address: '169.254.169.254', family: 4 }]) + }); + const cb = vi.fn(); + + guardedLookup('metadata.attacker.test', {}, cb); + + expect(cb).toHaveBeenCalledTimes(1); + expect(cb.mock.calls[0]?.[0]).toBeInstanceOf(SsrfBlockedError); + }); + + it('blocks when any address in a multi-record resolution is internal', () => { + const { guardedLookup } = buildSafe({ + lookup: (_hostname, _options, cb) => + cb(null, [ + { address: '8.8.8.8', family: 4 }, + { address: '10.0.0.7', family: 4 } + ]) + }); + const cb = vi.fn(); + + guardedLookup('mixed.attacker.test', {}, cb); + + expect(cb.mock.calls[0]?.[0]).toBeInstanceOf(SsrfBlockedError); + }); + + it('passes a public single-address resolution through unchanged', () => { + const { guardedLookup } = buildSafe({ + lookup: (_hostname, _options, cb) => cb(null, '1.1.1.1', 4) + }); + const cb = vi.fn(); + + guardedLookup('feed.example', {}, cb); + + expect(cb).toHaveBeenCalledWith(null, '1.1.1.1', 4); + }); + + it('passes a public multi-address resolution through unchanged', () => { + const addresses = [{ address: '1.1.1.1', family: 4 }]; + const { guardedLookup } = buildSafe({ + lookup: (_hostname, _options, cb) => cb(null, addresses) + }); + const cb = vi.fn(); + + guardedLookup('feed.example', {}, cb); + + expect(cb).toHaveBeenCalledWith(null, addresses, undefined); + }); + + it('propagates an underlying DNS resolution error', () => { + const dnsError = Object.assign(new Error('not found'), { + code: 'ENOTFOUND' + }); + const { guardedLookup } = buildSafe({ + lookup: (_hostname, _options, cb) => cb(dnsError) + }); + const cb = vi.fn(); + + guardedLookup('feed.example', {}, cb); + + expect(cb).toHaveBeenCalledWith(dnsError); + }); + + it('exempts an internal address permitted by the allow predicate', () => { + const addresses = [{ address: '10.0.0.7', family: 4 }]; + const { guardedLookup } = buildSafe({ + lookup: (_hostname, _options, cb) => cb(null, addresses), + allow: () => true + }); + const cb = vi.fn(); + + guardedLookup('lan.feed', {}, cb); + + expect(cb).toHaveBeenCalledWith(null, addresses, undefined); + }); + + it('still blocks an internal address the allow predicate rejects', () => { + const { guardedLookup } = buildSafe({ + lookup: (_hostname, _options, cb) => + cb(null, [{ address: '10.0.0.7', family: 4 }]), + allow: () => false + }); + const cb = vi.fn(); + + guardedLookup('lan.feed', {}, cb); + + expect(cb.mock.calls[0]?.[0]).toBeInstanceOf(SsrfBlockedError); + }); + + // --- IP-literal targets: screened at the connector (DNS is skipped) --- + + it('blocks a request to an internal IP-literal host', () => { + const { guardedConnector, base } = buildSafe({ lookup: () => {} }); + const cb = vi.fn(); + + guardedConnector({ hostname: '169.254.169.254' }, cb); + + expect(cb.mock.calls[0]?.[0]).toBeInstanceOf(SsrfBlockedError); + expect(base).not.toHaveBeenCalled(); + }); + + it('allows a request to a public IP-literal host', () => { + const { guardedConnector, base } = buildSafe({ lookup: () => {} }); + const cb = vi.fn(); + + guardedConnector({ hostname: '1.1.1.1' }, cb); + + expect(base).toHaveBeenCalledTimes(1); + }); + + it('exempts an internal IP-literal host on the allow list', () => { + const { guardedConnector, base } = buildSafe({ + lookup: () => {}, + allow: () => true + }); + const cb = vi.fn(); + + guardedConnector({ hostname: '10.0.0.7' }, cb); + + expect(base).toHaveBeenCalledTimes(1); + }); + + it('delegates a hostname target to the base connector', () => { + const { guardedConnector, base } = buildSafe({ lookup: () => {} }); + const cb = vi.fn(); + + guardedConnector({ hostname: 'feed.example' }, cb); + + expect(base).toHaveBeenCalledTimes(1); + }); +}); diff --git a/packages/core/src/safe-fetch.ts b/packages/core/src/safe-fetch.ts new file mode 100644 index 0000000..703f0ff --- /dev/null +++ b/packages/core/src/safe-fetch.ts @@ -0,0 +1,225 @@ +import { lookup as dnsLookup } from 'node:dns'; +import type { LookupAddress, LookupOptions } from 'node:dns'; +import { isIP, type LookupFunction } from 'node:net'; +import { + Agent, + buildConnector, + fetch as undiciFetch, + type Dispatcher +} from 'undici'; +import ipaddr from 'ipaddr.js'; + +/** + * Classify an IP literal for SSRF egress safety. Returns the non-public range + * name (e.g. `loopback`, `private`, `linkLocal`) when the address is anything + * other than a public unicast address, or `null` when it is safe to reach. + * + * IPv4-mapped IPv6 forms (`::ffff:10.0.0.1`) are decoded to their IPv4 address + * first via `ipaddr.process`, so they cannot smuggle an internal target past + * the check. + */ +export function classifyBlockedAddress(ip: string): string | null { + const range = ipaddr.process(ip).range(); + return range === 'unicast' ? null : range; +} + +/** + * Build an allow predicate from operator-configured CIDRs. An address that + * falls inside any listed range is exempted from {@link classifyBlockedAddress} + * — the escape hatch for a hub that legitimately serves feeds on a private LAN. + * An empty list permits nothing (the predicate always returns `false`). + */ +export function createCidrAllowList(cidrs: string[]): (ip: string) => boolean { + const ranges = cidrs.map(cidr => ipaddr.parseCIDR(cidr)); + return (ip: string): boolean => { + const addr = ipaddr.process(ip); + return ranges.some(([net, prefix]) => { + if (addr.kind() !== net.kind()) { + return false; + } + // kind() matched above; match's per-kind overload needs the cast. + return (addr as ipaddr.IPv4).match([net as ipaddr.IPv4, prefix]); + }); + }; +} + +/** Raised when an outbound request is refused on SSRF-egress grounds. */ +export class SsrfBlockedError extends Error { + constructor(message: string) { + super(message); + this.name = 'SsrfBlockedError'; + } +} + +/** The reason an address is refused, or `null` if it is allowed to be reached. */ +function blockedReason( + ip: string, + allow: ((ip: string) => boolean) | undefined +): string | null { + if (allow?.(ip)) { + return null; + } + return classifyBlockedAddress(ip); +} + +/** A `dns.lookup`-shaped callback (the single- and all-address forms). */ +type LookupResultCallback = ( + err: NodeJS.ErrnoException | null, + address?: string | LookupAddress[], + family?: number +) => void; + +/** A `dns.lookup`-shaped resolver. */ +export type GuardedLookupFn = ( + hostname: string, + options: LookupOptions, + callback: LookupResultCallback +) => void; + +type Connector = ReturnType; + +/** Construction-time dependencies for {@link createSafeFetch}. */ +export interface SafeFetchOptions { + /** Underlying fetch; defaults to undici's fetch (matches the injected agent). */ + baseFetch?: typeof fetch; + /** Builds the pinning dispatcher from the guarded connector (injectable for tests). */ + agentFactory?: (connector: ReturnType) => Dispatcher; + /** Builds the base socket connector (injectable for tests). */ + buildConnector?: typeof buildConnector; + /** Underlying DNS resolver; defaults to `node:dns` lookup (injectable for tests). */ + lookup?: GuardedLookupFn; + /** Optional allow predicate exempting specific addresses (e.g. a LAN range). */ + allow?: (ip: string) => boolean; +} + +const SUPPORTED_PROTOCOLS = new Set(['http:', 'https:']); + +/** + * Wrap a DNS resolver so every resolved address is screened before the socket + * connects. Because the connection is pinned to the addresses validated here + * (the dispatcher does not re-resolve), this closes DNS-rebinding: a name that + * resolves to an internal address is rejected at connect time, on the initial + * request and on every redirect hop alike. + */ +function createGuardedLookup(deps: { + lookup: GuardedLookupFn; + allow: ((ip: string) => boolean) | undefined; +}): GuardedLookupFn { + const { lookup, allow } = deps; + return (hostname, options, callback) => { + lookup(hostname, options, (err, address, family) => { + if (err) { + callback(err); + return; + } + // dns.lookup always yields an address on success (string, or a + // LookupAddress[] when called with `all`); the cast drops the + // error-path `undefined` the callback type carries. + const resolved = address as string | LookupAddress[]; + const addresses = Array.isArray(resolved) + ? resolved.map(entry => entry.address) + : [resolved]; + for (const ip of addresses) { + const reason = blockedReason(ip, allow); + if (reason !== null) { + callback( + new SsrfBlockedError( + `Refusing to connect to ${hostname} (${ip}): ${reason} address` + ) + ); + return; + } + } + callback(null, address, family); + }); + }; +} + +/** + * Wrap a socket connector so the destination address is screened. A hostname + * target is screened during DNS resolution by the guarded lookup; an IP-literal + * target skips DNS entirely (undici connects straight to it), so it is screened + * here — which also covers an auto-followed redirect that lands on an internal + * literal, since the connector runs for every connection the dispatcher opens. + */ +function createGuardedConnector(deps: { + lookup: GuardedLookupFn; + allow: ((ip: string) => boolean) | undefined; + build: typeof buildConnector; +}): Connector { + const base = deps.build({ + lookup: createGuardedLookup({ + lookup: deps.lookup, + allow: deps.allow + }) as unknown as LookupFunction + }); + return (options, callback) => { + const host = options.hostname; + if (isIP(host) !== 0) { + const reason = blockedReason(host, deps.allow); + if (reason !== null) { + callback( + new SsrfBlockedError( + `Refusing to connect to ${host}: ${reason} address` + ), + null + ); + return; + } + } + base(options, callback); + }; +} + +function defaultAgentFactory(connector: Connector): Dispatcher { + return new Agent({ connect: connector }); +} + +type FetchInput = Parameters[0]; +type FetchInit = Parameters[1]; + +function urlOf(input: FetchInput): URL { + if (typeof input === 'string' || input instanceof URL) { + return new URL(input); + } + return new URL(input.url); +} + +/** + * A `fetch` that is safe against SSRF egress. It refuses non-http(s) schemes and + * routes every request through an undici dispatcher whose connector validates the + * destination against {@link classifyBlockedAddress} (minus any {@link + * SafeFetchOptions.allow} exemptions), pinning the connection to the address it + * checked. Inject this as the `fetch` for the engine and every protocol plugin so + * topic re-fetch, the WebSub verification GET, and content delivery are all guarded. + */ +export function createSafeFetch(options: SafeFetchOptions = {}): typeof fetch { + const baseFetch = + options.baseFetch ?? (undiciFetch as unknown as typeof fetch); + const baseLookup = + options.lookup ?? (dnsLookup as unknown as GuardedLookupFn); + const build = options.buildConnector ?? buildConnector; + const agentFactory = options.agentFactory ?? defaultAgentFactory; + const dispatcher = agentFactory( + createGuardedConnector({ + lookup: baseLookup, + allow: options.allow, + build + }) + ); + + return (input: FetchInput, init?: FetchInit): Promise => { + const url = urlOf(input); + if (!SUPPORTED_PROTOCOLS.has(url.protocol)) { + return Promise.reject( + new SsrfBlockedError( + `Refusing to fetch ${url.protocol}// URL: only http and https are allowed` + ) + ); + } + // `dispatcher` is undici's per-request agent hook, absent from the DOM + // RequestInit the global fetch type advertises; the base fetch is undici's. + const guardedInit = { ...init, dispatcher } as unknown as FetchInit; + return baseFetch(input, guardedInit); + }; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 7c31230..5da0968 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -155,6 +155,12 @@ importers: '@rsscloud/xml-rpc': specifier: workspace:* version: link:../xml-rpc + ipaddr.js: + specifier: ^2.4.0 + version: 2.4.0 + undici: + specifier: ^7.28.0 + version: 7.28.0 xml2js: specifier: ^0.6.2 version: 0.6.2 @@ -1828,6 +1834,10 @@ packages: resolution: {integrity: sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==} engines: {node: '>= 0.10'} + ipaddr.js@2.4.0: + resolution: {integrity: sha512-9VGk3HGanVE6JoZXHiCpnGy5X0jYDnN4EA4lntFPj+1vIWlFhIylq2CrrCOJH9EAhc5CYhq18F2Av2tgoAPsYQ==} + engines: {node: '>= 10'} + is-arrayish@0.2.1: resolution: {integrity: sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==} @@ -2667,6 +2677,10 @@ packages: undici-types@7.24.6: resolution: {integrity: sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==} + undici@7.28.0: + resolution: {integrity: sha512-cRZYrTDwWznlnRiPjggAGxZXanty6M8RV1ff8Wm4LWXBp7/IG8v5DnOm74DtUBp9OONpK75YlPnIjQqX0dBDtA==} + engines: {node: '>=20.18.1'} + unpipe@1.0.0: resolution: {integrity: sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==} engines: {node: '>= 0.8'} @@ -4445,6 +4459,8 @@ snapshots: ipaddr.js@1.9.1: {} + ipaddr.js@2.4.0: {} + is-arrayish@0.2.1: {} is-binary-path@2.1.0: @@ -5290,6 +5306,8 @@ snapshots: undici-types@7.24.6: {} + undici@7.28.0: {} + unpipe@1.0.0: {} uri-js@4.4.1: From 10be43c1f9c61d15016909ce10a4669fb87f0687 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 11:51:55 -0500 Subject: [PATCH 24/35] feat(server): screen outbound fetches with the SSRF egress guard Build one createSafeFetch and inject it as the fetch for the engine's topic re-fetch and every protocol plugin's deliveries and verification GETs, so the REST/XML-RPC notify, the WebSub challenge GET, and WebSub content distribution are all guarded -- and the pre-existing rssCloud blind fetch is hardened along the way. On by default via WEBSUB_SSRF_PROTECTION; WEBSUB_FETCH_ALLOW_CIDRS exempts a private LAN that legitimately hosts feeds. The e2e suite runs with protection on and the Docker private ranges allowlisted, so the guarded path is exercised end to end. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/e2e/docker-compose.yml | 4 ++++ apps/server/config.js | 17 ++++++++++++++++- apps/server/core.js | 34 ++++++++++++++++++++++++++++++---- apps/server/docs/websub.md | 19 +++++++++++++++++++ 4 files changed, 69 insertions(+), 5 deletions(-) diff --git a/apps/e2e/docker-compose.yml b/apps/e2e/docker-compose.yml index 7f715ba..8530e6f 100644 --- a/apps/e2e/docker-compose.yml +++ b/apps/e2e/docker-compose.yml @@ -9,6 +9,10 @@ services: PORT: 5337 NODE_TLS_REJECT_UNAUTHORIZED: 0 ENABLE_TEST_API: "true" + # Keep SSRF egress protection ON so the suite exercises the guarded fetch, + # but exempt the Docker-network private ranges the mock servers live on so + # legitimate topic/callback traffic to rsscloud-tests still flows. + WEBSUB_FETCH_ALLOW_CIDRS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/8" expose: - 5337 diff --git a/apps/server/config.js b/apps/server/config.js index 9ec9d5b..9be9fad 100644 --- a/apps/server/config.js +++ b/apps/server/config.js @@ -41,5 +41,20 @@ module.exports = { // omitted, and the [min, max] a requested lease is clamped to. webSubLeaseDefaultSecs: getNumericConfig('WEBSUB_LEASE_DEFAULT_SECS', 86400), webSubLeaseMinSecs: getNumericConfig('WEBSUB_LEASE_MIN_SECS', 300), - webSubLeaseMaxSecs: getNumericConfig('WEBSUB_LEASE_MAX_SECS', 864000) + webSubLeaseMaxSecs: getNumericConfig('WEBSUB_LEASE_MAX_SECS', 864000), + // SSRF egress protection for outbound fetches (topic re-fetch, the WebSub + // verification GET, and content delivery). On by default; an outbound + // request whose host resolves to a non-public address (loopback, private, + // link-local / cloud-metadata, etc.) is refused. Set + // WEBSUB_SSRF_PROTECTION=off for local or containerised testing where + // targets are loopback/private. + webSubSsrfProtection: !['off', 'false', '0', 'no'].includes( + String(getConfig('WEBSUB_SSRF_PROTECTION', 'on')).toLowerCase() + ), + // CIDRs exempted from SSRF protection — for a hub that legitimately serves + // feeds on a private LAN. Comma-separated, e.g. "10.0.0.0/8,192.168.0.0/16". + webSubFetchAllowCidrs: String(getConfig('WEBSUB_FETCH_ALLOW_CIDRS', '')) + .split(',') + .map(value => value.trim()) + .filter(Boolean) }; diff --git a/apps/server/core.js b/apps/server/core.js index 38489fb..c765654 100644 --- a/apps/server/core.js +++ b/apps/server/core.js @@ -8,6 +8,8 @@ const { createXmlRpcProtocolPlugin, createWebSubProtocolPlugin, createFileStore, + createSafeFetch, + createCidrAllowList, resolveConfig } = require('@rsscloud/core'); const config = require('./config'); @@ -24,17 +26,40 @@ const coreConfig = resolveConfig({ webSubLeaseMaxSecs: config.webSubLeaseMaxSecs }); +// SSRF egress guard for every outbound call. Built once and injected into the +// engine's topic re-fetch and each plugin's deliveries/verification GETs, so a +// subscriber- or publisher-supplied URL that resolves to an internal address +// (loopback, private, link-local / cloud-metadata) is refused at connect time. +// When protection is off (dev/CI against loopback or private hosts), `fetch` is +// left unset so callers fall back to the platform's global fetch. +const fetchOption = config.webSubSsrfProtection + ? { + fetch: createSafeFetch( + config.webSubFetchAllowCidrs.length > 0 + ? { allow: createCidrAllowList(config.webSubFetchAllowCidrs) } + : {} + ) + } + : {}; + // Registers the 'websub' protocol so core.subscribe accepts WebSub subscriptions // (without it, core.subscribe → UNSUPPORTED_PROTOCOL). The plugin verifies // subscriber intent and, on fan-out, distributes the feed body to WebSub // callbacks — advertising this hub's public URL in the Link rel="hub" header. const plugins = [ - createRestProtocolPlugin({ requestTimeoutMs: config.requestTimeout }), - createXmlRpcProtocolPlugin({ requestTimeoutMs: config.requestTimeout }), + createRestProtocolPlugin({ + requestTimeoutMs: config.requestTimeout, + ...fetchOption + }), + createXmlRpcProtocolPlugin({ + requestTimeoutMs: config.requestTimeout, + ...fetchOption + }), createWebSubProtocolPlugin({ requestTimeoutMs: config.requestTimeout, hubUrl: config.hubUrl, - signatureAlgo: config.webSubSignatureAlgo + signatureAlgo: config.webSubSignatureAlgo, + ...fetchOption }) ]; @@ -53,7 +78,8 @@ const core = createRssCloudCore({ ) }), plugins, - config: coreConfig + config: coreConfig, + ...fetchOption }); module.exports = { core, events: core.events }; diff --git a/apps/server/docs/websub.md b/apps/server/docs/websub.md index 48fcf58..88e869d 100644 --- a/apps/server/docs/websub.md +++ b/apps/server/docs/websub.md @@ -92,6 +92,25 @@ The hub answers `202`, then re-fetches the topic and fans the change out. This i exactly the path an rssCloud `/ping` takes, so a WebSub publish also reaches rssCloud subscribers — see [How it fits together](cross-protocol.md). +## SSRF egress protection + +Both `hub.topic`/`hub.url` (which the hub **fetches**) and `hub.callback` (which the hub +**delivers** the fetched body to) are supplied by untrusted clients. To stop them being +pointed at the hub's own network, every outbound request — topic re-fetch, the intent +verification GET, and content delivery — is screened: the destination is rejected at +connect time if its host resolves to a non-public address (loopback, private, link-local +incl. cloud-metadata `169.254.169.254`, unique-local, CGNAT). Screening is done on the +resolved IP and re-applied on every redirect hop, so a hostname or redirect that points +inward is refused, not followed. + +| Config key | Default | Meaning | +| --------------------------- | ------- | ----------------------------------------------------------------------------------------- | +| `WEBSUB_SSRF_PROTECTION` | `on` | Set to `off` (or `false`/`0`/`no`) to disable screening — only for trusted/loopback test setups. | +| `WEBSUB_FETCH_ALLOW_CIDRS` | _(none)_| Comma-separated CIDRs exempted from screening, for a hub that legitimately serves feeds on a private LAN (e.g. `10.0.0.0/8,192.168.0.0/16`). | + +A blocked request surfaces as a failed fetch: the topic re-fetch reports a read failure +and a blocked callback counts as a failed delivery. + ## Using WebSub with your feed So that subscribers can **discover** this hub, advertise it from the resource you want From 4287b1101521c73dfbda9ab519e724fef97d0207 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 13:52:06 -0500 Subject: [PATCH 25/35] build(deps): bump form-data, ws, and markdown-it past advisories Clears the remaining Dependabot alerts: - ws ^8.20.1 -> ^8.21.0 (high; GHSA-96hv-2xvq-fx4p, memory-exhaustion DoS) - markdown-it ^14.1.1 -> ^14.2.0 (moderate; GHSA-6v5v-wf23-fmfq, quadratic smartquotes DoS) -- both direct apps/server deps - form-data >=4.0.6 via pnpm override (high; GHSA-hmw2-7cc7-3qxx, CRLF injection) -- transitive through superagent/chai-http Lockfile resolves to ws 8.21.0, markdown-it 14.2.0, form-data 4.0.6. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/server/package.json | 4 +-- package.json | 3 ++- pnpm-lock.yaml | 53 +++++++++++++++++++++++----------------- 3 files changed, 35 insertions(+), 25 deletions(-) diff --git a/apps/server/package.json b/apps/server/package.json index 37a7b59..7b176bc 100644 --- a/apps/server/package.json +++ b/apps/server/package.json @@ -22,9 +22,9 @@ "dotenv": "^17.4.2", "express": "^4.22.2", "express-handlebars": "^5.3.5", - "markdown-it": "^14.1.1", + "markdown-it": "^14.2.0", "morgan": "^1.10.1", - "ws": "^8.20.1", + "ws": "^8.21.0", "xmlbuilder": "^15.1.1" }, "devDependencies": { diff --git a/package.json b/package.json index 84eebcb..aae7925 100644 --- a/package.json +++ b/package.json @@ -23,7 +23,8 @@ "qs": ">=6.15.2", "vite": ">=8.0.16", "esbuild": ">=0.28.1", - "diff": ">=8.0.3" + "diff": ">=8.0.3", + "form-data": ">=4.0.6" }, "onlyBuiltDependencies": [ "esbuild" diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 5da0968..891fb1c 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -10,6 +10,7 @@ overrides: vite: '>=8.0.16' esbuild: '>=0.28.1' diff: '>=8.0.3' + form-data: '>=4.0.6' importers: @@ -125,14 +126,14 @@ importers: specifier: ^5.3.5 version: 5.3.5 markdown-it: - specifier: ^14.1.1 - version: 14.1.1 + specifier: ^14.2.0 + version: 14.2.0 morgan: specifier: ^1.10.1 version: 1.10.1 ws: - specifier: ^8.20.1 - version: 8.20.1 + specifier: ^8.21.0 + version: 8.21.0 xmlbuilder: specifier: ^15.1.1 version: 15.1.1 @@ -1662,8 +1663,8 @@ packages: resolution: {integrity: sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw==} engines: {node: '>=14'} - form-data@4.0.5: - resolution: {integrity: sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==} + form-data@4.0.6: + resolution: {integrity: sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==} engines: {node: '>= 6'} formidable@2.1.5: @@ -1769,6 +1770,10 @@ packages: resolution: {integrity: sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg==} engines: {node: '>= 0.4'} + hasown@2.0.4: + resolution: {integrity: sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==} + engines: {node: '>= 0.4'} + he@1.2.0: resolution: {integrity: sha512-F/1DnUGPopORZi0ni+CvrCgHQ5FyEAHRLSApuYWMmrbSwoN2Mn/7k+Gl38gJnR7yyDZk6WLXwiGod1JOWNDKGw==} hasBin: true @@ -2031,8 +2036,8 @@ packages: lines-and-columns@1.2.4: resolution: {integrity: sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==} - linkify-it@5.0.0: - resolution: {integrity: sha512-5aHCbzQRADcdP+ATqnDuhhJ/MRIqDkZX5pyjFHRRysS8vZ5AbqGEoFIb6pYHPZ+L/OC2Lc+xT8uHVVR5CAK/wQ==} + linkify-it@5.0.1: + resolution: {integrity: sha512-wVoTjP4Q6R0NW5hiZkVJaFZPWgtXfoGF+6LucL3/FtiNjmcHhYjEr5f1Kqjirc1nBW07J/ZuRFumqr2oqccEWg==} load-tsconfig@0.2.5: resolution: {integrity: sha512-IXO6OCs9yg8tMKzfPZ1YmheJbZCiEsnBdcB03l0OcfK9prKnJb96siuHCr5Fl37/yo9DnKU+TLpxzTUspw9shg==} @@ -2071,8 +2076,8 @@ packages: resolution: {integrity: sha512-hXdUTZYIVOt1Ex//jAQi+wTZZpUpwBj/0QsOzqegb3rGMMeJiSEu5xLHnYfBrRV4RH2+OCSOO95Is/7x1WJ4bw==} engines: {node: '>=10'} - markdown-it@14.1.1: - resolution: {integrity: sha512-BuU2qnTti9YKgK5N+IeMubp14ZUKUUw7yeJbkjtosvHiP0AZ5c8IAgEMk79D0eC8F23r4Ac/q8cAIFdm2FtyoA==} + markdown-it@14.2.0: + resolution: {integrity: sha512-1TGiQiJVRQ3NPmZH6sx5Cfnmg6GQm9jvC1ch4TK511NjSJvjzKLzn5pPfZRNZkRPZP0HqCioSndqH8v2nRaWVQ==} hasBin: true math-intrinsics@1.1.0: @@ -2803,8 +2808,8 @@ packages: wrappy@1.0.2: resolution: {integrity: sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==} - ws@8.20.1: - resolution: {integrity: sha512-It4dO0K5v//JtTXuPkfEOaI3uUN87iYPnqo/ZzqCoG3g8uhA66QUMs/SrM0YK7/NAu+r4LMh/9dq2A7k+rHs+w==} + ws@8.21.0: + resolution: {integrity: sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==} engines: {node: '>=10.0.0'} peerDependencies: bufferutil: ^4.0.1 @@ -3468,7 +3473,7 @@ snapshots: '@types/cookiejar': 2.1.5 '@types/methods': 1.1.4 '@types/node': 22.19.19 - form-data: 4.0.5 + form-data: 4.0.6 '@types/supertest@6.0.3': dependencies: @@ -4020,7 +4025,7 @@ snapshots: es-errors: 1.3.0 get-intrinsic: 1.3.0 has-tostringtag: 1.0.2 - hasown: 2.0.3 + hasown: 2.0.4 es-toolkit@1.46.1: {} @@ -4288,12 +4293,12 @@ snapshots: cross-spawn: 7.0.6 signal-exit: 4.1.0 - form-data@4.0.5: + form-data@4.0.6: dependencies: asynckit: 0.4.0 combined-stream: 1.0.8 es-set-tostringtag: 2.1.0 - hasown: 2.0.3 + hasown: 2.0.4 mime-types: 2.1.35 formidable@2.1.5: @@ -4409,6 +4414,10 @@ snapshots: dependencies: function-bind: 1.1.2 + hasown@2.0.4: + dependencies: + function-bind: 1.1.2 + he@1.2.0: {} html-escaper@2.0.2: {} @@ -4611,7 +4620,7 @@ snapshots: lines-and-columns@1.2.4: {} - linkify-it@5.0.0: + linkify-it@5.0.1: dependencies: uc.micro: 2.1.0 @@ -4652,11 +4661,11 @@ snapshots: dependencies: semver: 7.8.0 - markdown-it@14.1.1: + markdown-it@14.2.0: dependencies: argparse: 2.0.1 entities: 4.5.0 - linkify-it: 5.0.0 + linkify-it: 5.0.1 mdurl: 2.0.0 punycode.js: 2.3.1 uc.micro: 2.1.0 @@ -5128,7 +5137,7 @@ snapshots: cookiejar: 2.1.4 debug: 4.4.3 fast-safe-stringify: 2.1.1 - form-data: 4.0.5 + form-data: 4.0.6 formidable: 3.5.4 methods: 1.1.2 mime: 2.6.0 @@ -5142,7 +5151,7 @@ snapshots: cookiejar: 2.1.4 debug: 4.4.3 fast-safe-stringify: 2.1.1 - form-data: 4.0.5 + form-data: 4.0.6 formidable: 2.1.5 methods: 1.1.2 mime: 2.6.0 @@ -5424,7 +5433,7 @@ snapshots: wrappy@1.0.2: {} - ws@8.20.1: {} + ws@8.21.0: {} xml2js@0.5.0: dependencies: From b03fd14b84b3cf1968524e4c473a99d37f77b27c Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 14:52:50 -0500 Subject: [PATCH 26/35] fix(server): normalize WEBSUB_PATH to a leading slash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A WEBSUB_PATH set without a leading slash (e.g. "websub") produced a malformed hubUrl (http://host:portwebsub) — the URL advertised to subscribers in the Link rel="hub" header — and an off-root route mount. Normalize it at assignment so the mount and hubUrl stay well-formed regardless of how it is configured; absolute paths are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/server/config.js | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/apps/server/config.js b/apps/server/config.js index 9be9fad..4876ae7 100644 --- a/apps/server/config.js +++ b/apps/server/config.js @@ -16,7 +16,12 @@ function getNumericConfig(key, defaultValue) { // (consumed when content distribution lands), defaulting to domain/port/path. const domain = getConfig('DOMAIN', 'localhost'); const port = getNumericConfig('PORT', 5337); -const webSubPath = getConfig('WEBSUB_PATH', '/websub'); +// Normalize to a leading slash so the route mount and the hubUrl composition +// stay well-formed even if WEBSUB_PATH is set without one (e.g. "websub"). +const rawWebSubPath = getConfig('WEBSUB_PATH', '/websub'); +const webSubPath = rawWebSubPath.startsWith('/') + ? rawWebSubPath + : `/${rawWebSubPath}`; module.exports = { appName: 'rssCloudServer', From efe49c8ef71214a80dfc907d73efce82df50a0bd Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 14:52:50 -0500 Subject: [PATCH 27/35] test(e2e): drop loopback from the SSRF allowlist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Docker e2e reaches the mock servers and the app over Docker service names, which resolve to private (RFC1918) addresses — loopback is never a legitimate egress target here. Keep only the Docker-private ranges so the suite would catch an accidental loopback-egress regression. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/e2e/docker-compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/e2e/docker-compose.yml b/apps/e2e/docker-compose.yml index 8530e6f..e33adb7 100644 --- a/apps/e2e/docker-compose.yml +++ b/apps/e2e/docker-compose.yml @@ -12,7 +12,7 @@ services: # Keep SSRF egress protection ON so the suite exercises the guarded fetch, # but exempt the Docker-network private ranges the mock servers live on so # legitimate topic/callback traffic to rsscloud-tests still flows. - WEBSUB_FETCH_ALLOW_CIDRS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.0/8" + WEBSUB_FETCH_ALLOW_CIDRS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" expose: - 5337 From f1ce0356e16524a5c5d41848529bf4eb1f94bf61 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 15:08:09 -0500 Subject: [PATCH 28/35] test(core): assert WebSub lease bounds stay ordered Add an invariant check that webSubLeaseMinSecs <= webSubLeaseDefaultSecs <= webSubLeaseMaxSecs, so a future change to the lease defaults that breaks the ordering the clamp relies on is caught even if the literal-value assertions are updated to match. Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/core/src/config.test.ts | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/packages/core/src/config.test.ts b/packages/core/src/config.test.ts index be20316..2c0de33 100644 --- a/packages/core/src/config.test.ts +++ b/packages/core/src/config.test.ts @@ -16,6 +16,13 @@ describe('resolveConfig', () => { }); }); + it('keeps the lease bounds ordered (min <= default <= max)', () => { + const { webSubLeaseMinSecs, webSubLeaseDefaultSecs, webSubLeaseMaxSecs } = + resolveConfig(); + expect(webSubLeaseMinSecs).toBeLessThanOrEqual(webSubLeaseDefaultSecs); + expect(webSubLeaseDefaultSecs).toBeLessThanOrEqual(webSubLeaseMaxSecs); + }); + it('overrides only the provided keys', () => { const resolved = resolveConfig({ maxConsecutiveErrors: 5 }); expect(resolved.maxConsecutiveErrors).toBe(5); From a2d648889fae65e84cfdf3dd044b8dafd0eca4ec Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 15:08:09 -0500 Subject: [PATCH 29/35] fix(core): reject non-URL hub.topic/hub.url with a synchronous 400 hub.topic (subscribe/unsubscribe) and hub.url/hub.topic (publish) were only checked for a non-empty string, so a malformed value passed parsing, got a 202, and failed later in the async fetch. Validate them as absolute URLs with the existing isAbsoluteUrl helper (matching hub.callback) so a bad URL is a synchronous 400 at parse time. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../src/protocols/websub-dispatcher.test.ts | 33 +++++++++++++++++++ .../core/src/protocols/websub-dispatcher.ts | 14 ++++---- 2 files changed, 40 insertions(+), 7 deletions(-) diff --git a/packages/core/src/protocols/websub-dispatcher.test.ts b/packages/core/src/protocols/websub-dispatcher.test.ts index e803297..7d058ef 100644 --- a/packages/core/src/protocols/websub-dispatcher.test.ts +++ b/packages/core/src/protocols/websub-dispatcher.test.ts @@ -76,6 +76,16 @@ describe('parseSubscribe', () => { expect(result).toEqual({ ok: false, status: 400 }); }); + it('rejects a non-URL hub.topic as a 400', () => { + const result = parseSubscribe({ + 'hub.mode': 'subscribe', + 'hub.callback': 'https://sub.example.com/listener', + 'hub.topic': 'not-a-url' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); + it('carries a supplied hub.secret through as details.secret', () => { const result = parseSubscribe({ 'hub.mode': 'subscribe', @@ -230,6 +240,29 @@ describe('parsePublish', () => { expect(result).toEqual({ ok: false, status: 400 }); }); + + it('falls back to hub.topic when hub.url is not a valid URL', () => { + const result = parsePublish({ + 'hub.mode': 'publish', + 'hub.url': 'not-a-url', + 'hub.topic': 'http://feed.example/rss' + }); + + expect(result).toEqual({ + ok: true, + request: { resourceUrl: 'http://feed.example/rss' } + }); + }); + + it('rejects a publish whose hub.url and hub.topic are both invalid as a 400', () => { + const result = parsePublish({ + 'hub.mode': 'publish', + 'hub.url': 'not-a-url', + 'hub.topic': 'also-bad' + }); + + expect(result).toEqual({ ok: false, status: 400 }); + }); }); describe('createWebSubDispatcher', () => { diff --git a/packages/core/src/protocols/websub-dispatcher.ts b/packages/core/src/protocols/websub-dispatcher.ts index b09c613..00417d2 100644 --- a/packages/core/src/protocols/websub-dispatcher.ts +++ b/packages/core/src/protocols/websub-dispatcher.ts @@ -38,8 +38,8 @@ function isAbsoluteUrl(value: string): boolean { /** * The two fields every actionable `hub.*` request shares: a valid absolute - * `hub.callback` and a non-empty `hub.topic`. Returns `null` when either is - * malformed. + * `hub.callback` and a valid absolute `hub.topic`. Returns `null` when either is + * malformed, so a bad URL is a synchronous 400 rather than a later async failure. */ function parseHubCallbackTopic( body: Record @@ -49,7 +49,7 @@ function parseHubCallbackTopic( return null; } const topic = body['hub.topic']; - if (typeof topic !== 'string' || topic === '') { + if (typeof topic !== 'string' || !isAbsoluteUrl(topic)) { return null; } return { callback, topic }; @@ -57,16 +57,16 @@ function parseHubCallbackTopic( /** * The updated topic a publish names: `hub.url` preferred, falling back to - * `hub.topic` for compatibility. Returns `null` when neither is a non-empty - * string. + * `hub.topic` for compatibility. Returns `null` when neither is a valid absolute + * URL, so a bad URL is a synchronous 400 rather than a later async failure. */ function publishTopic(body: Record): string | null { const url = body['hub.url']; - if (typeof url === 'string' && url !== '') { + if (typeof url === 'string' && isAbsoluteUrl(url)) { return url; } const topic = body['hub.topic']; - if (typeof topic === 'string' && topic !== '') { + if (typeof topic === 'string' && isAbsoluteUrl(topic)) { return topic; } return null; From 99c2a0b8176f9f064fca6b75ae54c856b0be772e Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 15:08:09 -0500 Subject: [PATCH 30/35] fix(core): bound WebSub delivery redirects to prevent loops distribute() followed 3xx redirects recursively with no hop limit, so a self-redirecting callback looped the delivery task forever (one POST per hop), never settling that subscription's fan-out. Cap the chain at MAX_REDIRECTS (5); when exhausted, throw so deliver() reports a failed delivery rather than looping or silently succeeding. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../core/src/protocols/websub-plugin.test.ts | 28 +++++++++++++++++++ packages/core/src/protocols/websub-plugin.ts | 23 +++++++++++++-- 2 files changed, 48 insertions(+), 3 deletions(-) diff --git a/packages/core/src/protocols/websub-plugin.test.ts b/packages/core/src/protocols/websub-plugin.test.ts index f427fc4..de752f1 100644 --- a/packages/core/src/protocols/websub-plugin.test.ts +++ b/packages/core/src/protocols/websub-plugin.test.ts @@ -425,6 +425,34 @@ describe('createWebSubProtocolPlugin deliver', () => { expect(calls[1]?.init?.body).toBe('updated'); }); + it('reports failure when a callback redirects past the hop limit', async () => { + let calls = 0; + const fakeFetch = (async () => { + calls += 1; + return new Response(null, { + status: 302, + headers: { location: 'https://sub.example/loop' } + }); + }) as typeof fetch; + + const plugin = createWebSubProtocolPlugin({ + fetch: fakeFetch, + hubUrl: 'https://hub.example/websub' + }); + + const result = await plugin.deliver( + deliveryContext( + 'https://sub.example/loop', + 'http://feed.example/rss' + ) + ); + + expect(result.ok).toBe(false); + expect(result.error).toBeInstanceOf(Error); + // Bounded: the initial POST plus a fixed number of redirect hops, not ∞. + expect(calls).toBe(6); + }); + it('reports failure when the callback responds non-2xx', async () => { const fakeFetch = (async () => new Response('nope', { status: 404 })) as typeof fetch; diff --git a/packages/core/src/protocols/websub-plugin.ts b/packages/core/src/protocols/websub-plugin.ts index 92023de..4100537 100644 --- a/packages/core/src/protocols/websub-plugin.ts +++ b/packages/core/src/protocols/websub-plugin.ts @@ -35,6 +35,9 @@ const WEBSUB_PROTOCOLS: Protocol[] = ['websub']; /** Fallback request timeout when none is supplied (mirrors the server default). */ const DEFAULT_REQUEST_TIMEOUT_MS = 4000; +/** Max redirect hops a single content-distribution delivery will follow. */ +const MAX_REDIRECTS = 5; + /** Portable, hard-to-guess token for the intent-verification challenge. */ function defaultCreateChallenge(): string { const bytes = new Uint8Array(16); @@ -85,10 +88,15 @@ export function createWebSubProtocolPlugin( } } - /** POST the feed body to one callback, following redirects like rssCloud notify. */ + /** + * POST the feed body to one callback, following redirects like rssCloud + * notify, but bounded by `redirectsLeft` so a self-redirecting callback + * can't loop the delivery task forever. + */ async function distribute( targetUrl: string, - ctx: DeliveryContext + ctx: DeliveryContext, + redirectsLeft = MAX_REDIRECTS ): Promise { const headers: Record = { 'Content-Type': @@ -119,7 +127,16 @@ export function createWebSubProtocolPlugin( if (res.status >= 300 && res.status < 400) { const location = res.headers.get('location'); if (location) { - await distribute(new URL(location, targetUrl).toString(), ctx); + if (redirectsLeft <= 0) { + throw new Error( + 'WebSub content distribution exceeded the redirect limit' + ); + } + await distribute( + new URL(location, targetUrl).toString(), + ctx, + redirectsLeft - 1 + ); return; } } From 03fd7783e39e1055e83028648f00144665d750cb Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 15:08:10 -0500 Subject: [PATCH 31/35] fix(core): absorb synchronous throws in the verification scheduler The in-process scheduler routed async rejections to onError but let a synchronous throw from a non-async task escape as an uncaught microtask exception. Wrap the task() call in try/catch so both error kinds reach onError, honoring the scheduler's documented non-throwing/absorb contract. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../src/engine/verification-scheduler.test.ts | 16 ++++++++++++++++ .../core/src/engine/verification-scheduler.ts | 8 +++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/packages/core/src/engine/verification-scheduler.test.ts b/packages/core/src/engine/verification-scheduler.test.ts index 3dc631b..31f35d4 100644 --- a/packages/core/src/engine/verification-scheduler.test.ts +++ b/packages/core/src/engine/verification-scheduler.test.ts @@ -37,4 +37,20 @@ describe('createInProcessVerificationScheduler', () => { expect(seen).toEqual([boom]); }); + + it('routes a synchronous throw from a task to onError', async () => { + const seen: unknown[] = []; + const scheduler = createInProcessVerificationScheduler({ + onError: error => seen.push(error) + }); + + const boom = new Error('sync boom'); + scheduler.schedule(() => { + throw boom; + }); + + await new Promise(resolve => setTimeout(resolve, 0)); + + expect(seen).toEqual([boom]); + }); }); diff --git a/packages/core/src/engine/verification-scheduler.ts b/packages/core/src/engine/verification-scheduler.ts index fd467a6..52fbff0 100644 --- a/packages/core/src/engine/verification-scheduler.ts +++ b/packages/core/src/engine/verification-scheduler.ts @@ -31,7 +31,13 @@ export function createInProcessVerificationScheduler( return { schedule(task) { queueMicrotask(() => { - void task().catch(options.onError); + // Absorb both a synchronous throw (a non-async task can throw + // before returning its promise) and an async rejection. + try { + void task().catch(options.onError); + } catch (error) { + options.onError(error); + } }); } }; From 23491cf5d6bbea111c829d87859a7206f17ff1f1 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 15:12:57 -0500 Subject: [PATCH 32/35] fix(core): bound rssCloud REST notify redirects to prevent loops MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sendNotify() followed 3xx redirects recursively with no hop limit, so a self-redirecting subscriber callback looped forever (one POST per hop). Cap the chain at MAX_REDIRECTS (5) — matching the WebSub delivery fix — throwing when exhausted so it reports a failed notification rather than looping. This bounds both deliver() and the same-domain test-notify in verify(), which share sendNotify. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../core/src/protocols/rest-plugin.test.ts | 25 +++++++++++++++++++ packages/core/src/protocols/rest-plugin.ts | 17 ++++++++++--- 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/packages/core/src/protocols/rest-plugin.test.ts b/packages/core/src/protocols/rest-plugin.test.ts index 1b23e89..c98d981 100644 --- a/packages/core/src/protocols/rest-plugin.test.ts +++ b/packages/core/src/protocols/rest-plugin.test.ts @@ -126,6 +126,31 @@ describe('createRestProtocolPlugin deliver', () => { ]); }); + it('reports failure when a callback redirects past the hop limit', async () => { + let calls = 0; + const fakeFetch = (async () => { + calls += 1; + return new Response('', { + status: 302, + headers: { location: '/loop' } + }); + }) as typeof fetch; + + const plugin = createRestProtocolPlugin({ fetch: fakeFetch }); + + const result = await plugin.deliver( + deliveryContext( + 'https://subscriber.example/loop', + 'https://feed.example/rss' + ) + ); + + expect(result.ok).toBe(false); + expect(result.error).toBeInstanceOf(Error); + // Bounded: the initial POST plus a fixed number of redirect hops, not ∞. + expect(calls).toBe(6); + }); + it('treats a 3xx without a Location header as a failure', async () => { const fakeFetch = (async () => new Response('', { status: 302 })) as typeof fetch; diff --git a/packages/core/src/protocols/rest-plugin.ts b/packages/core/src/protocols/rest-plugin.ts index d825709..2f6dffe 100644 --- a/packages/core/src/protocols/rest-plugin.ts +++ b/packages/core/src/protocols/rest-plugin.ts @@ -22,6 +22,9 @@ const REST_PROTOCOLS: Protocol[] = ['http-post', 'https-post']; /** Fallback request timeout when none is supplied (mirrors the server default). */ const DEFAULT_REQUEST_TIMEOUT_MS = 4000; +/** Max redirect hops a single notification will follow. */ +const MAX_REDIRECTS = 5; + /** Portable, hard-to-guess token for the cross-domain challenge handshake. */ function defaultCreateChallenge(): string { const bytes = new Uint8Array(16); @@ -50,10 +53,14 @@ export function createRestProtocolPlugin( return body; } - /** POST the notification, following redirects; throws on timeout or non-2xx. */ + /** + * POST the notification, following redirects (bounded by `redirectsLeft` so a + * self-redirecting callback can't loop forever); throws on timeout or non-2xx. + */ async function sendNotify( targetUrl: string, - body: URLSearchParams + body: URLSearchParams, + redirectsLeft = MAX_REDIRECTS ): Promise { const res = await fetchWithTimeout(doFetch, requestTimeoutMs, targetUrl, { method: 'POST', @@ -65,9 +72,13 @@ export function createRestProtocolPlugin( if (res.status >= 300 && res.status < 400) { const location = res.headers.get('location'); if (location) { + if (redirectsLeft <= 0) { + throw new Error('Notification Failed: too many redirects'); + } await sendNotify( new URL(location, targetUrl).toString(), - body + body, + redirectsLeft - 1 ); return; } From 3de9aa2649454eca5c84a0a6a4fc2094223d0385 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 16:01:58 -0500 Subject: [PATCH 33/35] fix(server): scope the SSRF allowlist to topic fetch, keep callbacks strict A single guarded fetch (one allowlist) was injected into both the engine's topic re-fetch and every plugin's callback delivery/verification GET. So an operator exempting a private CIDR for their feeds also exempted attacker-chosen hub.callback targets in that range, reopening SSRF on the callback path. Split the guard by trust: - topic-fetch path (engine) honors WEBSUB_FETCH_ALLOW_CIDRS; - callback path (delivery + verification, both to attacker-supplied callbacks) is strict by default and honors only the new WEBSUB_CALLBACK_ALLOW_CIDRS. No @rsscloud/core change: createSafeFetch already takes a per-instance allow, so this is wiring plus a second config knob. Default config is unchanged (both empty = strict everywhere). The e2e sets both (mock topic+callback live on private Docker IPs); docs and ADR-0003 updated. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/e2e/docker-compose.yml | 5 ++- apps/server/config.js | 24 ++++++++--- apps/server/core.js | 43 ++++++++++++------- apps/server/docs/websub.md | 15 +++++-- ...003-ssrf-egress-guard-on-outbound-fetch.md | 29 +++++++------ 5 files changed, 77 insertions(+), 39 deletions(-) diff --git a/apps/e2e/docker-compose.yml b/apps/e2e/docker-compose.yml index e33adb7..7ffd08b 100644 --- a/apps/e2e/docker-compose.yml +++ b/apps/e2e/docker-compose.yml @@ -11,8 +11,11 @@ services: ENABLE_TEST_API: "true" # Keep SSRF egress protection ON so the suite exercises the guarded fetch, # but exempt the Docker-network private ranges the mock servers live on so - # legitimate topic/callback traffic to rsscloud-tests still flows. + # legitimate traffic to rsscloud-tests flows. Topic fetches and callback + # deliveries/verification GETs have separate allowlists; the mock serves + # both, so set both here. WEBSUB_FETCH_ALLOW_CIDRS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" + WEBSUB_CALLBACK_ALLOW_CIDRS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" expose: - 5337 diff --git a/apps/server/config.js b/apps/server/config.js index 4876ae7..9977814 100644 --- a/apps/server/config.js +++ b/apps/server/config.js @@ -11,6 +11,14 @@ function getNumericConfig(key, defaultValue) { return value ? parseInt(value, 10) : defaultValue; } +// Parse a comma-separated CIDR list, dropping blank entries. +function getCidrListConfig(key) { + return String(getConfig(key, '')) + .split(',') + .map(value => value.trim()) + .filter(Boolean); +} + // The hub's public base URL and mount path. The WebSub endpoint mounts at // webSubPath; hubUrl is the externally-reachable URL advertised to subscribers // (consumed when content distribution lands), defaulting to domain/port/path. @@ -56,10 +64,14 @@ module.exports = { webSubSsrfProtection: !['off', 'false', '0', 'no'].includes( String(getConfig('WEBSUB_SSRF_PROTECTION', 'on')).toLowerCase() ), - // CIDRs exempted from SSRF protection — for a hub that legitimately serves - // feeds on a private LAN. Comma-separated, e.g. "10.0.0.0/8,192.168.0.0/16". - webSubFetchAllowCidrs: String(getConfig('WEBSUB_FETCH_ALLOW_CIDRS', '')) - .split(',') - .map(value => value.trim()) - .filter(Boolean) + // CIDRs exempted from SSRF protection on the TOPIC-fetch path only — for a + // hub that legitimately fetches feeds on a private LAN. Comma-separated, + // e.g. "10.0.0.0/8,192.168.0.0/16". Deliberately does NOT apply to callback + // delivery/verification, so a trusted-feed exemption can't be abused to make + // the hub deliver to an attacker-chosen internal hub.callback. + webSubFetchAllowCidrs: getCidrListConfig('WEBSUB_FETCH_ALLOW_CIDRS'), + // CIDRs exempted on the CALLBACK path (delivery + verification GET) only — + // for a hub with genuine subscribers on a private LAN. Default empty (strict): + // attacker-supplied callbacks never inherit the topic-fetch allowlist above. + webSubCallbackAllowCidrs: getCidrListConfig('WEBSUB_CALLBACK_ALLOW_CIDRS') }; diff --git a/apps/server/core.js b/apps/server/core.js index c765654..ca68c4a 100644 --- a/apps/server/core.js +++ b/apps/server/core.js @@ -26,40 +26,50 @@ const coreConfig = resolveConfig({ webSubLeaseMaxSecs: config.webSubLeaseMaxSecs }); -// SSRF egress guard for every outbound call. Built once and injected into the -// engine's topic re-fetch and each plugin's deliveries/verification GETs, so a -// subscriber- or publisher-supplied URL that resolves to an internal address -// (loopback, private, link-local / cloud-metadata) is refused at connect time. -// When protection is off (dev/CI against loopback or private hosts), `fetch` is -// left unset so callers fall back to the platform's global fetch. -const fetchOption = config.webSubSsrfProtection - ? { +// SSRF egress guard for every outbound call: any URL whose host resolves to an +// internal address (loopback, private, link-local / cloud-metadata) is refused +// at connect time. The exemption allowlist is split by trust so a trusted-feed +// exemption can't be abused: the TOPIC path (the engine's feed re-fetch) honors +// WEBSUB_FETCH_ALLOW_CIDRS, while the CALLBACK path (each plugin's delivery and +// verification GET, both to attacker-supplied hub.callback URLs) is strict by +// default and only honors the separate WEBSUB_CALLBACK_ALLOW_CIDRS. When +// protection is off (dev/CI against loopback or private hosts), `fetch` is left +// unset so callers fall back to the platform's global fetch. +function guardedFetchOption(allowCidrs) { + if (!config.webSubSsrfProtection) { + return {}; + } + return { fetch: createSafeFetch( - config.webSubFetchAllowCidrs.length > 0 - ? { allow: createCidrAllowList(config.webSubFetchAllowCidrs) } + allowCidrs.length > 0 + ? { allow: createCidrAllowList(allowCidrs) } : {} ) - } - : {}; + }; +} + +const topicFetchOption = guardedFetchOption(config.webSubFetchAllowCidrs); +const callbackFetchOption = guardedFetchOption(config.webSubCallbackAllowCidrs); // Registers the 'websub' protocol so core.subscribe accepts WebSub subscriptions // (without it, core.subscribe → UNSUPPORTED_PROTOCOL). The plugin verifies // subscriber intent and, on fan-out, distributes the feed body to WebSub // callbacks — advertising this hub's public URL in the Link rel="hub" header. +// Plugins only ever reach subscriber callbacks, so they take the callback policy. const plugins = [ createRestProtocolPlugin({ requestTimeoutMs: config.requestTimeout, - ...fetchOption + ...callbackFetchOption }), createXmlRpcProtocolPlugin({ requestTimeoutMs: config.requestTimeout, - ...fetchOption + ...callbackFetchOption }), createWebSubProtocolPlugin({ requestTimeoutMs: config.requestTimeout, hubUrl: config.hubUrl, signatureAlgo: config.webSubSignatureAlgo, - ...fetchOption + ...callbackFetchOption }) ]; @@ -79,7 +89,8 @@ const core = createRssCloudCore({ }), plugins, config: coreConfig, - ...fetchOption + // The engine's only outbound call is the topic/feed re-fetch. + ...topicFetchOption }); module.exports = { core, events: core.events }; diff --git a/apps/server/docs/websub.md b/apps/server/docs/websub.md index 88e869d..bc36928 100644 --- a/apps/server/docs/websub.md +++ b/apps/server/docs/websub.md @@ -103,10 +103,17 @@ incl. cloud-metadata `169.254.169.254`, unique-local, CGNAT). Screening is done resolved IP and re-applied on every redirect hop, so a hostname or redirect that points inward is refused, not followed. -| Config key | Default | Meaning | -| --------------------------- | ------- | ----------------------------------------------------------------------------------------- | -| `WEBSUB_SSRF_PROTECTION` | `on` | Set to `off` (or `false`/`0`/`no`) to disable screening — only for trusted/loopback test setups. | -| `WEBSUB_FETCH_ALLOW_CIDRS` | _(none)_| Comma-separated CIDRs exempted from screening, for a hub that legitimately serves feeds on a private LAN (e.g. `10.0.0.0/8,192.168.0.0/16`). | +The exemption allowlist is **split by trust** so a trusted-feed exemption can't be turned +into a callback-SSRF: the topic-fetch path and the callback path (delivery + verification +GET, both to attacker-supplied `hub.callback` URLs) have separate allowlists. Exempting a +private range for your feeds does **not** let an attacker register a `hub.callback` in that +range. + +| Config key | Default | Meaning | +| ----------------------------- | ------- | ----------------------------------------------------------------------------------------- | +| `WEBSUB_SSRF_PROTECTION` | `on` | Set to `off` (or `false`/`0`/`no`) to disable screening — only for trusted/loopback test setups. | +| `WEBSUB_FETCH_ALLOW_CIDRS` | _(none)_| Comma-separated CIDRs exempted on the **topic-fetch** path only, for a hub that fetches feeds on a private LAN (e.g. `10.0.0.0/8,192.168.0.0/16`). | +| `WEBSUB_CALLBACK_ALLOW_CIDRS` | _(none)_| Comma-separated CIDRs exempted on the **callback** path (delivery + verification), for a hub with genuine subscribers on a private LAN. Strict by default. | A blocked request surfaces as a failed fetch: the topic re-fetch reports a read failure and a blocked callback counts as a failed delivery. diff --git a/docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md b/docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md index 8b84c0e..57eb372 100644 --- a/docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md +++ b/docs/adr/0003-ssrf-egress-guard-on-outbound-fetch.md @@ -31,18 +31,22 @@ accepted connection the dispatcher opens, so each **redirect hop** is re-screened and there is no resolve-then-connect TOCTOU window. -3. **One guard, injected everywhere outbound.** `createSafeFetch` lives in `@rsscloud/core` - and is injected as the `fetch` for the engine's topic re-fetch **and** every protocol - plugin's deliveries / verification GETs. So topic fetch, the WebSub challenge GET, WebSub - content delivery, and the rssCloud REST/XML-RPC notifies are all covered uniformly — the - pre-existing rssCloud blind fetch is hardened for free. +3. **One guard, injected on every outbound path.** `createSafeFetch` lives in + `@rsscloud/core` and is injected as the `fetch` for the engine's topic re-fetch **and** + every protocol plugin's deliveries / verification GETs. So topic fetch, the WebSub + challenge GET, WebSub content delivery, and the rssCloud REST/XML-RPC notifies are all + covered uniformly — the pre-existing rssCloud blind fetch is hardened for free. -4. **Secure by default, with an operator escape hatch.** Protection is **on** by default - (`WEBSUB_SSRF_PROTECTION`); a hub that legitimately serves feeds on a private LAN exempts - specific ranges via `WEBSUB_FETCH_ALLOW_CIDRS` rather than disabling protection wholesale. - Local dev and the e2e suite (whose targets are loopback / private Docker IPs) keep working - by allowlisting those ranges — the e2e suite runs with protection on so the guarded fetch - is exercised end-to-end. +4. **Secure by default, with trust-split operator escape hatches.** Protection is **on** by + default (`WEBSUB_SSRF_PROTECTION`). The exemption allowlist is **split by trust** so a + trusted-feed exemption can't reopen SSRF for callbacks: the topic-fetch path honors + `WEBSUB_FETCH_ALLOW_CIDRS`, while the callback path (delivery + verification GET, both to + attacker-supplied `hub.callback` URLs) is strict by default and honors only the separate + `WEBSUB_CALLBACK_ALLOW_CIDRS`. The engine takes the topic policy (its sole outbound call + is the feed re-fetch); the plugins take the callback policy (they only ever reach + subscriber callbacks). Local dev and the e2e suite (whose targets are loopback / private + Docker IPs) keep working by allowlisting those ranges on both paths — the e2e suite runs + with protection on so the guarded fetch is exercised end-to-end. ## Why connector-level, not just a custom lookup @@ -64,7 +68,8 @@ every redirect. read failure (`RESOURCE_READ_FAILED`), and a blocked callback counts as a failed delivery (`notifyFailed`). Nothing new is thrown to the front door. - A hub deployed on a network whose feeds live on private addresses must opt those ranges in - via `WEBSUB_FETCH_ALLOW_CIDRS`, or fetches to them will be refused. This is the intended + via `WEBSUB_FETCH_ALLOW_CIDRS` (and, for subscribers on private addresses, separately via + `WEBSUB_CALLBACK_ALLOW_CIDRS`), or those requests will be refused. This is the intended secure-by-default trade-off. - The highest-value target is cloud instance metadata; operators should still defend it at the infrastructure layer too (e.g. IMDSv2 with a hop limit), so the guard is defence in From 875f7b12c47243e99801f309a85470bc02f6fc0b Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 16:02:58 -0500 Subject: [PATCH 34/35] fix(core): fail WebSub delivery when the hub URL is unconfigured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit distribute() interpolated hubUrl into the Link rel="hub" header unconditionally, so a plugin built without hubUrl (the type allows it — hubUrl?: string) would POST a malformed `; rel="hub"` header to subscribers. Guard at the top of distribute and throw when hubUrl is missing so deliver() reports a failed delivery instead of emitting bad metadata; a host always injects hubUrl (see apps/server). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../core/src/protocols/websub-plugin.test.ts | 20 +++++++++++++++++++ packages/core/src/protocols/websub-plugin.ts | 6 ++++++ 2 files changed, 26 insertions(+) diff --git a/packages/core/src/protocols/websub-plugin.test.ts b/packages/core/src/protocols/websub-plugin.test.ts index de752f1..b86382d 100644 --- a/packages/core/src/protocols/websub-plugin.test.ts +++ b/packages/core/src/protocols/websub-plugin.test.ts @@ -425,6 +425,26 @@ describe('createWebSubProtocolPlugin deliver', () => { expect(calls[1]?.init?.body).toBe('updated'); }); + it('reports failure when no hub URL is configured', async () => { + let called = false; + const fakeFetch = (async () => { + called = true; + return new Response(null, { status: 204 }); + }) as typeof fetch; + const plugin = createWebSubProtocolPlugin({ fetch: fakeFetch }); + + const result = await plugin.deliver( + deliveryContext( + 'https://sub.example/listener', + 'http://feed.example/rss' + ) + ); + + expect(result.ok).toBe(false); + expect(result.error).toBeInstanceOf(Error); + expect(called).toBe(false); + }); + it('reports failure when a callback redirects past the hop limit', async () => { let calls = 0; const fakeFetch = (async () => { diff --git a/packages/core/src/protocols/websub-plugin.ts b/packages/core/src/protocols/websub-plugin.ts index 4100537..b5ae8be 100644 --- a/packages/core/src/protocols/websub-plugin.ts +++ b/packages/core/src/protocols/websub-plugin.ts @@ -98,6 +98,12 @@ export function createWebSubProtocolPlugin( ctx: DeliveryContext, redirectsLeft = MAX_REDIRECTS ): Promise { + if (hubUrl === undefined) { + // The Link rel="hub" advertisement needs the hub's own URL; without + // it we'd POST a malformed `` header. Fail the delivery + // instead (a host always injects hubUrl — see apps/server). + throw new Error('WebSub hub URL is not configured'); + } const headers: Record = { 'Content-Type': ctx.payload.contentType ?? 'application/octet-stream', From 89e6b03cac355165fe2a234b3a4b7fb9480c6842 Mon Sep 17 00:00:00 2001 From: Andrew Shell Date: Tue, 30 Jun 2026 16:20:37 -0500 Subject: [PATCH 35/35] build(server): add Docker Hub publish tooling and Dockge example Add scripts/docker-build-push.sh (multi-platform buildx build+push to andrewshell/rsscloud-server, version from apps/server/package.json) plus docker:build-push / docker:dry-run pnpm wrappers. Document the server port with EXPOSE 5337 and ship examples/dockge/compose.yaml for hosting, with persistence via a named volume mount rather than a Dockerfile VOLUME. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/server/Dockerfile | 6 + examples/dockge/compose.yaml | 50 +++++++ package.json | 3 + scripts/README.md | 63 +++++++++ scripts/docker-build-push.sh | 255 +++++++++++++++++++++++++++++++++++ 5 files changed, 377 insertions(+) create mode 100644 examples/dockge/compose.yaml create mode 100644 scripts/README.md create mode 100755 scripts/docker-build-push.sh diff --git a/apps/server/Dockerfile b/apps/server/Dockerfile index 92429b9..b0fab8f 100644 --- a/apps/server/Dockerfile +++ b/apps/server/Dockerfile @@ -33,4 +33,10 @@ COPY --from=build /app/packages/express/dist packages/express/dist WORKDIR /app/apps/server +# Default listen port (config.js PORT default). Documentation only — override +# PORT at runtime and publish the matching port if you change it. Persistence of +# ./data (subscriptions + stats) is handled by the runtime volume mount, e.g. +# the `volumes:` entry in examples/dockge/compose.yaml — not declared here. +EXPOSE 5337 + CMD ["node", "--use_strict", "app.js"] diff --git a/examples/dockge/compose.yaml b/examples/dockge/compose.yaml new file mode 100644 index 0000000..8a4a061 --- /dev/null +++ b/examples/dockge/compose.yaml @@ -0,0 +1,50 @@ +# rssCloud server — example stack for Dockge (https://github.com/louislam/dockge) +# +# Usage: +# 1. In Dockge, create a new stack (e.g. "rsscloud") and paste this file. +# 2. Edit DOMAIN and HUB_URL below to your public hostname. +# 3. Deploy. Subscriptions + stats persist in the named volume `rsscloud-data`. +# +# This pulls the published image rather than building, so it deploys without the +# source tree. To pin a release, replace `:latest` with a version tag +# (e.g. andrewshell/rsscloud-server:4.0.0). + +services: + rsscloud: + image: andrewshell/rsscloud-server:latest + container_name: rsscloud-server + restart: unless-stopped + ports: + # host:container — change the host side if 5337 is taken; usually you'd + # also front this with a reverse proxy terminating HTTPS. + - "5337:5337" + environment: + # Externally-reachable hostname for this hub (no scheme, no port). + DOMAIN: cloud.example.com + # Public WebSub hub URL advertised to subscribers — point it at your + # HTTPS endpoint. Must match how subscribers reach the /websub route. + HUB_URL: https://cloud.example.com/websub + # Optional: change the listen port (then update the ports mapping above). + # PORT: "5337" + # + # SSRF egress protection is ON by default (recommended) and refuses + # outbound requests to private/loopback addresses. Only relax it if your + # feeds or subscribers genuinely live on a private LAN: + # WEBSUB_FETCH_ALLOW_CIDRS: "10.0.0.0/8,192.168.0.0/16" # topic fetch + # WEBSUB_CALLBACK_ALLOW_CIDRS: "10.0.0.0/8,192.168.0.0/16" # delivery + volumes: + # Persist on-disk state across restarts/redeploys. + - rsscloud-data:/app/apps/server/data + healthcheck: + test: + - CMD + - node + - -e + - "require('http').get('http://localhost:5337/',r=>process.exit(r.statusCode<500?0:1)).on('error',()=>process.exit(1))" + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + +volumes: + rsscloud-data: diff --git a/package.json b/package.json index aae7925..e7abcdd 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,9 @@ "test:unit": "turbo run test", "test:core": "turbo run test --filter=@rsscloud/core", "clean": "turbo run clean", + "docker:build-push": "./scripts/docker-build-push.sh", + "docker:build-push-skip-quality": "./scripts/docker-build-push.sh --skip-quality", + "docker:dry-run": "./scripts/docker-build-push.sh --dry-run --skip-quality", "prepare": "husky" }, "packageManager": "pnpm@10.11.0", diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..f969841 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,63 @@ +# Scripts + +## docker-build-push.sh + +Builds the server Docker image (`apps/server/Dockerfile`) and pushes it to Docker +Hub as `andrewshell/rsscloud-server`. + +### Features + +- ✅ **Quality checks** — runs `typecheck`, `lint`, and unit tests before building +- 🐳 **Docker validation** — checks Docker is running and you're authenticated +- 🏷️ **Smart tagging** — tags with the version from `apps/server/package.json` + `latest` +- 🎯 **Custom tags** — pass an extra tag as a positional argument +- 🚀 **Multi-platform** — builds `linux/amd64` and `linux/arm64` via `docker buildx` +- 🔍 **Dry run** — preview the tags without building/pushing + +### Usage + +```bash +# Full build with quality checks +pnpm docker:build-push + +# Skip quality checks for quick iterations +pnpm docker:build-push-skip-quality + +# Dry run — show what would happen without building/pushing +pnpm docker:dry-run + +# Direct script usage (e.g. with a custom tag) +./scripts/docker-build-push.sh beta +./scripts/docker-build-push.sh --help +``` + +### Requirements + +- Docker installed and running, with `buildx` +- Docker Hub authentication (`docker login`) — the script prompts if needed +- Node.js + pnpm +- Run from the repository root + +### Tags pushed + +- `andrewshell/rsscloud-server:` (from `apps/server/package.json`) +- `andrewshell/rsscloud-server:latest` +- `andrewshell/rsscloud-server:` (if provided) + +### Running the published image + +The server keeps subscriptions/stats on disk under `/app/apps/server/data`, so +mount a volume there to persist state across restarts. Set `DOMAIN`/`HUB_URL` to +the externally-reachable host so the hub advertises the right callback URL. + +```bash +docker run -d -p 5337:5337 \ + -e DOMAIN=cloud.example.com \ + -e HUB_URL=https://cloud.example.com/websub \ + -v rsscloud-data:/app/apps/server/data \ + andrewshell/rsscloud-server:latest +``` + +> The `docker-compose.yml` under `apps/e2e/` relaxes the SSRF egress protection so +> the test mock servers are reachable. A real deployment should **not** copy those +> `WEBSUB_*_ALLOW_CIDRS` / SSRF env vars — keep the strict defaults. diff --git a/scripts/docker-build-push.sh b/scripts/docker-build-push.sh new file mode 100755 index 0000000..f925dec --- /dev/null +++ b/scripts/docker-build-push.sh @@ -0,0 +1,255 @@ +#!/bin/bash + +# Docker Build and Push Script for rsscloud-server +# Builds and pushes the Docker image to andrewshell/rsscloud-server on Docker Hub + +set -e # Exit on any error + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Configuration +DOCKER_REPO="andrewshell/rsscloud-server" +DOCKERFILE_PATH="apps/server/Dockerfile" +# The published version comes from the server package, not the private monorepo +# root (which stays at 0.0.0). +VERSION_PACKAGE_JSON="apps/server/package.json" + +# Helper functions +log_info() { + echo -e "${BLUE}ℹ️ $1${NC}" +} + +log_success() { + echo -e "${GREEN}✅ $1${NC}" +} + +log_warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +log_error() { + echo -e "${RED}❌ $1${NC}" +} + +# Parse command line arguments +SKIP_QUALITY=false +CUSTOM_TAG="" +DRY_RUN=false + +# Check for help first +if [[ "$1" == "-h" || "$1" == "--help" ]]; then + show_usage() { + echo "Usage: $0 [OPTIONS] [CUSTOM_TAG]" + echo "" + echo "Options:" + echo " --skip-quality Skip quality checks (typecheck, lint, unit tests)" + echo " --dry-run Show what would be done without executing" + echo " -h, --help Show this help message" + echo "" + echo "Examples:" + echo " $0 # Build and push with version + latest tags" + echo " $0 beta # Build and push with version + latest + beta tags" + echo " $0 --skip-quality # Build and push without running quality checks" + echo " $0 --skip-quality v1.2.3 # Build and push with custom tag, skip quality checks" + echo "" + } + show_usage + exit 0 +fi + +while [[ $# -gt 0 ]]; do + case $1 in + --skip-quality) + SKIP_QUALITY=true + shift + ;; + --dry-run) + DRY_RUN=true + shift + ;; + *) + CUSTOM_TAG="$1" + shift + ;; + esac +done + +# Function to check if Docker is running +check_docker() { + log_info "Checking if Docker is running..." + if ! docker info >/dev/null 2>&1; then + log_error "Docker is not running. Please start Docker and try again." + exit 1 + fi + log_success "Docker is running" +} + +# Function to check Docker Hub login +check_docker_login() { + log_info "Checking Docker Hub authentication..." + if ! docker info | grep -q "Username:"; then + log_warning "Not logged into Docker Hub. Attempting login..." + if ! docker login; then + log_error "Failed to login to Docker Hub. Please run 'docker login' manually." + exit 1 + fi + fi + log_success "Docker Hub authentication verified" +} + +# Function to run quality checks +run_quality_checks() { + if [ "$SKIP_QUALITY" = true ]; then + log_warning "Skipping quality checks as requested" + return 0 + fi + + log_info "Running quality checks..." + + log_info "Running TypeScript type checking..." + if ! pnpm typecheck; then + log_error "TypeScript type checking failed" + exit 1 + fi + + log_info "Running ESLint..." + if ! pnpm lint; then + log_error "ESLint checks failed" + exit 1 + fi + + log_info "Running unit tests..." + if ! pnpm test:unit; then + log_error "Unit tests failed" + exit 1 + fi + + log_success "All quality checks passed" +} + +# Function to get version from the server package.json +get_version() { + node -p "require('./${VERSION_PACKAGE_JSON}').version" +} + +# Function to build Docker image +build_image() { + local version=$(get_version) + + log_info "Building Docker image..." + log_info "Version: $version" + + if [ "$DRY_RUN" = true ]; then + log_info "[DRY RUN] Would build image with tag: ${DOCKER_REPO}:${version}" + log_info "[DRY RUN] Would tag as latest: ${DOCKER_REPO}:latest" + if [ -n "$CUSTOM_TAG" ]; then + log_info "[DRY RUN] Would tag with custom tag: ${DOCKER_REPO}:${CUSTOM_TAG}" + fi + log_success "[DRY RUN] Docker image build simulation completed" + return 0 + fi + + # Build multi-platform image with version tag + log_info "Building multi-platform image with tag: ${DOCKER_REPO}:${version}" + log_info "Building for platforms: linux/amd64,linux/arm64" + + # Create buildx builder if it doesn't exist + if ! docker buildx ls | grep -q multiplatform; then + log_info "Creating multiplatform buildx builder..." + docker buildx create --name multiplatform --use + else + docker buildx use multiplatform + fi + + # Build all tags at once + local tags="-t ${DOCKER_REPO}:${version} -t ${DOCKER_REPO}:latest" + if [ -n "$CUSTOM_TAG" ]; then + tags="$tags -t ${DOCKER_REPO}:${CUSTOM_TAG}" + fi + + # Build and push multi-platform image with all tags + # Using --no-cache to ensure fresh builds + if ! docker buildx build -f "$DOCKERFILE_PATH" \ + --platform linux/amd64,linux/arm64 \ + $tags \ + --no-cache \ + --push .; then + log_error "Docker multi-platform build failed" + exit 1 + fi + + log_success "Docker image built successfully" +} + +# Function to push Docker image (now handled in build step for multi-platform) +push_image() { + local version=$(get_version) + + if [ "$DRY_RUN" = true ]; then + log_info "[DRY RUN] Would push ${DOCKER_REPO}:${version}" + log_info "[DRY RUN] Would push ${DOCKER_REPO}:latest" + if [ -n "$CUSTOM_TAG" ]; then + log_info "[DRY RUN] Would push ${DOCKER_REPO}:${CUSTOM_TAG}" + fi + log_success "[DRY RUN] All tags push simulation completed" + return 0 + fi + + # Multi-platform images are pushed during build step + log_success "Multi-platform images already pushed to Docker Hub during build" +} + + +# Function to display image info +show_image_info() { + local version=$(get_version) + + echo "" + log_success "🐳 Docker image build and push completed!" + echo "" + echo "📦 Image Repository: ${DOCKER_REPO}" + echo "🏷️ Tags pushed:" + echo " • ${DOCKER_REPO}:${version}" + echo " • ${DOCKER_REPO}:latest" + if [ -n "$CUSTOM_TAG" ]; then + echo " • ${DOCKER_REPO}:${CUSTOM_TAG}" + fi + echo "" + echo "🚀 To run the image:" + echo " docker run -d -p 5337:5337 \\" + echo " -e DOMAIN=cloud.example.com \\" + echo " -e HUB_URL=https://cloud.example.com/websub \\" + echo " -v rsscloud-data:/app/apps/server/data \\" + echo " ${DOCKER_REPO}:latest" + echo "" +} + +# Main execution +main() { + echo "🐳 rsscloud-server Docker Build & Push Script" + echo "=============================================" + echo "" + + + # Verify we're in the right directory + if [ ! -f "package.json" ] || [ ! -f "$DOCKERFILE_PATH" ]; then + log_error "Please run this script from the project root directory" + exit 1 + fi + + # Run all checks and build steps + check_docker + check_docker_login + run_quality_checks + build_image + push_image + show_image_info +} + +# Run main function with all arguments +main "$@"