Skip to content

fix(planetscale): forward signed data-plane auth params to branch scrapes#212

Merged
Makisuo merged 1 commit into
mainfrom
fix/planetscale-metrics-signed-url
Jul 15, 2026
Merged

fix(planetscale): forward signed data-plane auth params to branch scrapes#212
Makisuo merged 1 commit into
mainfrom
fix/planetscale-metrics-signed-url

Conversation

@Makisuo

@Makisuo Makisuo commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

What & why

PlanetScale branch-metrics scraping was returning 403 invalid signature on every branch, silently flatlining ingest for 22h+ in production.

Root cause: the metrics data plane (metrics.psdb.cloud) does not authenticate with the service token or the OAuth bearer — it uses a signed, expiring URL. PlanetScale's http_sd discovery response mints per-branch __param_sig / __param_exp labels, and Prometheus convention promotes __param_<name>?<name>= query params on the scrape URL. The Authorization header (token/bearer) only authenticates the discovery listing on api.planetscale.com.

subTargetsFromGroup stripped every __-prefixed label except __scheme__/__metrics_path__, so it dropped sig/exp and built an unsigned URL → 403 on every scrape. Reconnecting and rotating service tokens never helped because the token was never the data-plane credential — this was a misdiagnosis baked into the integration's design.

The change

  • subTargetsFromGroup promotes __param_* labels onto a new PlanetScaleSubTarget.signedUrl (base url + ?sig=&exp=).
  • Base url and subTargetKey stay param-free on purpose: the signature rotates each 10-min discovery refresh, so keeping it out of the identity keeps the scraper's per-branch fiber (and instance label) stable — only the server-side fetch picks up the fresh signature.
  • ScrapeTargetsService.scrapeForCollector and PlanetScaleConnectionService.probeDataPlaneScrape now fetch signedUrl.
  • Corrected the now-wrong "data plane rejects OAuth bearers / only accepts service tokens" comments.

Verification

  • New discovery test asserts __param_*signedUrl, url/subTargetKey stay stable, and sig/exp never leak into metric labels.
  • 48 api tests pass, typecheck clean.
  • The URL built via URLSearchParams is byte-identical to one proven to return HTTP 200 with live branch metrics (base64 %3D%3D padding included).

Reviewer notes

  • Deploy scope: apps/api only. The scraper delegates the actual fetch to the API's prometheus-scrape proxy, and its target.url was already the base URL — no scraper redeploy.
  • Existing enabled targets recover on the next scrape after deploy — no reconnect, no service-token step.
  • Follow-up (not in this PR): this proves the hybrid "paste a service token" flow is now unnecessary — the OAuth bearer auths discovery and the data plane only needs the signed URL, so probeDataPlaneScrape will pass for bearer targets and pure-OAuth scraping works end-to-end. Ripping out the manual-token UI + setMetricsToken is a clean simplification.

🤖 Generated with Claude Code


Open in Devin Review

…apes

PlanetScale authenticates the metrics data plane (metrics.psdb.cloud) with a
signed, expiring URL — NOT the service token or OAuth bearer. Its Prometheus
http_sd response mints per-branch `__param_sig`/`__param_exp` labels, which
Prometheus convention promotes to `?sig=&exp=` query params on the scrape URL.
The Authorization header (token/bearer) only auths the discovery listing on
api.planetscale.com.

`subTargetsFromGroup` stripped every `__`-prefixed label except
`__scheme__`/`__metrics_path__`, dropping sig/exp and building an unsigned URL,
so every branch scrape returned `403 invalid signature` (a 22h+ metrics outage
in production). Reconnecting and rotating service tokens couldn't help because
the token was never the data-plane credential.

Fix: promote `__param_*` labels onto a new `PlanetScaleSubTarget.signedUrl`
(base `url` stays param-free so the scraper's per-branch fiber identity +
`instance` label don't churn as the signature rotates each 10-min refresh).
`ScrapeTargetsService.scrapeForCollector` and
`PlanetScaleConnectionService.probeDataPlaneScrape` now fetch `signedUrl`.

Verified: the URL built via URLSearchParams is byte-identical to one proven to
return HTTP 200 with live branch metrics. apps/api-only deploy; existing
targets recover on the next scrape with no reconnect.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pullfrog

pullfrog Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Your Pullfrog Router balance is exhausted.

You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

Pullfrog  | ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

🐛 1 issue in files not directly in the diff

🐛 External scraper receives unsigned URLs for PlanetScale targets, causing all scrapes to fail with 403 (apps/api/src/routes/scraper-internal.http.ts:45-51)

The target list served to the external scraper uses the unsigned base URL (subTarget?.url at scraper-internal.http.ts:79) instead of the signed URL that carries the required authentication query params, so every PlanetScale branch scrape by the standalone scraper returns a 403 signature error.

Impact: The standalone Prometheus scraper cannot collect any PlanetScale branch metrics — only the internal collector path works.

Incomplete transformation: signedUrl added to one scraping path but not the other

The PR adds signedUrl to PlanetScaleSubTarget and correctly updates the internal collector path in ScrapeTargetsService.scrapeForCollector (apps/api/src/services/ScrapeTargetsService.ts:863) to use match.signedUrl. However, the external scraper path through scraper-internal.http.ts was not updated:

  1. SubTargetOverride interface (scraper-internal.http.ts:45-51) only has url, subTargetKey, and labels — no signedUrl.
  2. toInternalScrapeTarget at scraper-internal.http.ts:79 sets url: subTarget?.url ?? row.url, which maps to PlanetScaleSubTarget.url (the unsigned base URL).
  3. InternalScrapeTarget schema (packages/domain/src/http/scraper-internal.ts:15) only has a url field.
  4. At scraper-internal.http.ts:170, the PlanetScaleSubTarget is passed as a SubTargetOverride, and TypeScript's structural typing silently drops the signedUrl field.

The external scraper receives InternalScrapeTarget objects whose url lacks the ?sig=…&exp=… params that PlanetScale's metrics data plane requires for authentication.

View 1 additional finding in Devin Review.

Open in Devin Review

@Makisuo Makisuo merged commit def2465 into main Jul 15, 2026
7 of 8 checks passed
@Makisuo Makisuo deleted the fix/planetscale-metrics-signed-url branch July 15, 2026 22:12
@Makisuo

Makisuo commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator Author

Re: Devin Review — "External scraper receives unsigned URLs … causing all scrapes to fail with 403"

This is a false positive — the standalone scraper never fetches target.url and never contacts the PlanetScale metrics data plane directly.

The scraper (apps/scraper) is a scheduler, not a fetcher. Its only outbound HTTP calls are to the API and the OTLP ingest (ApiClient.ts / OtlpIngest.ts). For each scrape it calls GET /api/internal/prometheus-scrape?targetId=…&sub=… (apps/scraper/src/ApiClient.ts:97-103), passing only the id + subTargetKey — not the URL. The API proxy then resolves match.signedUrl server-side in scrapeForCollector (apps/api/src/services/ScrapeTargetsService.ts:863, the exact line this PR changes) and performs the signed fetch there, behind SSRF validation + the discovery cache.

The unsigned url on InternalScrapeTarget is consumed only for the fiber fingerprint (ScrapeScheduler.ts:120) and the instance label host (ScrapeScheduler.ts:225, host only — query params are irrelevant to both). Keeping it param-free is deliberate: the sig/exp rotate every 10-minute discovery refresh, so folding them into the identity URL would restart every per-branch scrape fiber on each rotation. No change needed.

🤖 Addressed by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant