v2 public API: Stripe-style spec + foundation with api_keys pilot#211
Open
Makisuo wants to merge 3 commits into
Open
v2 public API: Stripe-style spec + foundation with api_keys pilot#211Makisuo wants to merge 3 commits into
Makisuo wants to merge 3 commits into
Conversation
Introduces the Maple v2 public API: docs/api-v2.md (conventions spec +
resource catalog + rollout phases) and the executable foundation:
- packages/domain/src/http/v2: MapleApiV2 HttpApi shell, prefixed public
object IDs (key_/dash_/... via a boundary base58 codec, no DB churn),
Stripe list envelope ({object:"list",data,has_more,next_cursor}) with
uniform limit/cursor pagination, {error:{type,code,message}} error
envelope with stable machine codes, snake_case wire fields, ISO-8601
timestamps, and the AuthorizationV2 + V2SchemaErrors middlewares
- Scoped API keys: api_keys.scopes jsonb column + migration, scope
grammar (<family>:read|write|*), mechanical enforcement derived from
method + first /v2 path segment (write implies read; session tokens
and legacy null-scope keys bypass)
- Pilot resource api_keys at /v2/api_keys (list/create/retrieve/roll/
revoke) as thin adapters over the existing ApiKeysService; mounted
alongside v1 with Scalar docs at /v2/docs
- Tests: public-ID round-trips, wire-shape/envelope encodes, scope
matrix, OpenAPI contract freeze, PGlite scope persistence, and
end-to-end HTTP tests over a real router (auth, 401/403/400/404
envelopes, pagination, CRUD)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTnfswT7zw6WhWgkS53WqM
Contributor
|
Your Pullfrog Router balance is exhausted. You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Ingest Rust Test + Benchmark ResultsCommit: Load Benchmark —
|
| Metric | main (median) | PR (median) | Delta |
|---|---|---|---|
| Requests/sec | 1934.68 | 2012.99 | +4.0% better |
| Rows/sec | 19346.76 | 20129.90 | +4.0% better |
| p50 latency | 32.00 ms | 30.88 ms | -3.5% better |
| p95 latency | 37.87 ms | 35.50 ms | -6.3% better |
| p99 latency | 39.47 ms | 38.73 ms | -1.9% better |
| Export catch-up | 0.027 s | 0.026 s | -2.1% better |
| Max RSS | 102.16 MiB | 100.47 MiB | -1.7% better |
| Failures | 0 | 0 | same |
Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.
PR load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 27,
"duration_seconds": 1.056480138,
"export_catchup_seconds": 0.025843914,
"request_rps": 1893.0786562501378,
"row_rps": 18930.786562501376,
"p50_ms": 32.853,
"p95_ms": 41.495,
"p99_ms": 47.205,
"max_rss_mb": 101.9375,
"max_cpu_percent": 70.1,
"avg_cpu_percent": 57.699999999999996
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 26,
"duration_seconds": 0.951806132,
"export_catchup_seconds": 0.026085175,
"request_rps": 2101.2682444033676,
"row_rps": 21012.68244403368,
"p50_ms": 29.925,
"p95_ms": 33.19,
"p99_ms": 34.591,
"max_rss_mb": 99.54296875,
"max_cpu_percent": 76.7,
"avg_cpu_percent": 46.650000000000006
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 24,
"duration_seconds": 0.993546881,
"export_catchup_seconds": 0.026329352,
"request_rps": 2012.9900644315949,
"row_rps": 20129.90064431595,
"p50_ms": 30.883,
"p95_ms": 35.498,
"p99_ms": 38.735,
"max_rss_mb": 100.47265625,
"max_cpu_percent": 73.2,
"avg_cpu_percent": 44.900000000000006
}
]main load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 25,
"duration_seconds": 1.065595782,
"export_catchup_seconds": 0.025949403,
"request_rps": 1876.884306210589,
"row_rps": 18768.84306210589,
"p50_ms": 33.406,
"p95_ms": 37.869,
"p99_ms": 39.467,
"max_rss_mb": 103.03125,
"max_cpu_percent": 69.7,
"avg_cpu_percent": 57.53333333333333
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 26,
"duration_seconds": 1.033764888,
"export_catchup_seconds": 0.026796597,
"request_rps": 1934.675885412738,
"row_rps": 19346.75885412738,
"p50_ms": 31.997,
"p95_ms": 39.182,
"p99_ms": 40.314,
"max_rss_mb": 102.16015625,
"max_cpu_percent": 71.2,
"avg_cpu_percent": 52.46666666666666
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 24,
"duration_seconds": 0.994095118,
"export_catchup_seconds": 0.026645646,
"request_rps": 2011.8799134873127,
"row_rps": 20118.799134873127,
"p50_ms": 31.117,
"p95_ms": 34.802,
"p99_ms": 37.219,
"max_rss_mb": 99.046875,
"max_cpu_percent": 75.0,
"avg_cpu_percent": 45.8
}
]WAL-acked microbench (cargo bench --bench ingest_bench)
Compiling serde v1.0.228
Compiling zerocopy v0.8.39
Compiling zerocopy-derive v0.8.39
Compiling num-traits v0.2.19
Compiling regex-automata v0.4.14
Compiling serde_urlencoded v0.7.1
Compiling axum v0.8.8
Compiling reqwest v0.12.28
Compiling ppv-lite86 v0.2.21
Compiling rand_chacha v0.9.0
Compiling rand v0.9.2
Compiling tonic v0.14.4
Compiling opentelemetry_sdk v0.31.0
Compiling tonic-prost v0.14.4
Compiling matchers v0.2.0
Compiling tracing-subscriber v0.3.22
Compiling opentelemetry-proto v0.31.0
Compiling opentelemetry-http v0.31.0
Compiling either v1.15.0
Compiling rayon-core v1.13.0
Compiling chrono v0.4.43
Compiling reqwest v0.13.2
Compiling opentelemetry-otlp v0.31.1
Compiling tracing-opentelemetry v0.32.1
Compiling half v2.7.1
Compiling crossbeam-deque v0.8.6
Compiling plotters-backend v0.3.7
Compiling ciborium-io v0.2.2
Compiling clap_lex v1.1.0
Compiling anstyle v1.0.14
Compiling clap_builder v4.5.60
Compiling ciborium-ll v0.2.2
Compiling plotters-svg v0.3.7
Compiling maple-ingest v0.1.0 (/home/runner/work/maple/maple/apps/ingest)
Compiling itertools v0.10.5
Compiling futures-io v0.3.32
Compiling same-file v1.0.6
Compiling cast v0.3.0
Compiling criterion-plot v0.5.0
Compiling walkdir v2.5.0
Compiling futures v0.3.31
Compiling clap v4.5.60
Compiling rayon v1.12.0
Compiling plotters v0.3.7
Compiling ciborium v0.2.2
Compiling regex v1.12.3
Compiling tinytemplate v1.2.1
Compiling is-terminal v0.4.17
Compiling anes v0.1.6
Compiling oorandom v11.1.5
Compiling criterion v0.5.1
Finished `bench` profile [optimized] target(s) in 1m 25s
Running benches/ingest_bench.rs (target/release/deps/ingest_bench-581d2100de893627)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench: 546128 ns/iter (+/- 97556)
test ingest_accept/traces_10_spans_wal_ack ... bench: 518434 ns/iter (+/- 31954)
cargo test
test telemetry::tests::metrics_summary_data_points_are_dropped ... ok
test telemetry::tests::metrics_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::migrate_legacy_shard_relocates_frames_into_lanes ... ok
test telemetry::tests::clickhouse_export_drops_passworded_non_https_endpoint_without_sending ... ok
test telemetry::tests::pipeline_can_start_for_clickhouse_only_without_tinybird_credentials ... ok
test telemetry::tests::pipeline_e2e_exports_gzip_ndjson_to_fake_tinybird ... ok
test telemetry::tests::pipeline_e2e_exports_traces_to_fake_tinybird ... ok
test telemetry::tests::sampling_keeps_errors_even_when_ratio_low ... ok
test telemetry::tests::scraper_contract::scraper_otlp_json_decodes_with_gateway_serde_and_encodes_to_rows ... ok
test telemetry::tests::signal_tag_round_trips_all_variants ... ok
test telemetry::tests::pipeline_e2e_exports_metrics_to_fake_tinybird ... ok
test telemetry::tests::telemetry_signal_as_str_is_canonical_lowercase ... ok
test telemetry::tests::timestamp_has_nano_precision ... ok
test telemetry::tests::timestamps_match_clickhouse_datetime64_nine_format ... ok
test telemetry::tests::trace_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::traces_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::wal_partial_drain_advances_cursor_without_truncating ... ok
test telemetry::tests::wal_round_trips_frame ... ok
test telemetry::tests::wal_truncates_after_full_drain_allowing_further_appends ... ok
test telemetry::tests::pipeline_exports_ready_org_to_clickhouse_without_tinybird_calls ... ok
test telemetry::tests::slow_clickhouse_lane_does_not_block_cosharded_tinybird_org ... ok
test telemetry::tests::clickhouse_breaker_sheds_after_threshold_failures ... ok
test result: ok. 36 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.78s
Running unittests src/bin/load_test.rs (target/debug/deps/load_test-661a0aa1eb3f6d6d)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/main.rs (target/debug/deps/maple_ingest-c33bf80c577edb95)
running 38 tests
test autumn::tests::allowed_only_no_balance_field ... ok
test autumn::tests::flat_hardcap_with_remaining_allows ... ok
test autumn::tests::flat_hardcap_depleted_blocks ... ok
test autumn::tests::flat_overage_allows ... ok
test autumn::tests::flat_unlimited_allows ... ok
test autumn::tests::flat_sub_one_gb_remaining_still_allows ... ok
test autumn::tests::nested_balance_object_depleted_blocks ... ok
test autumn::tests::nested_overage_allows ... ok
test autumn::tests::nested_balance_object_with_remaining_allows ... ok
test autumn::tests::null_balance_no_subscription_blocks ... ok
test autumn::tests::unrecognized_shape_returns_none ... ok
test tests::api_error_from_pipeline_maps_variants_to_status ... ok
test tests::api_error_kind_maps_status_to_stable_label ... ok
test tests::clickhouse_destination_uses_native_pipeline_even_in_forward_mode ... ok
test tests::clickhouse_destination_is_terminal_in_dual_mode ... ok
test tests::clickhouse_target_resolver_rejects_password_over_http ... ok
test tests::clickhouse_target_resolver_decrypts_current_schema_password ... ok
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::clickhouse_target_resolver_requires_current_schema ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::decrypt_aes256_gcm_matches_node_crypto_fixture ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::rejection_span_status_is_error_only_for_5xx ... ok
test tests::hash_is_deterministic ... ok
test tests::resolve_ingest_key_keeps_stale_schema_on_managed_native_path ... ok
test tests::resolve_connector_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_refreshes_routing_before_auth_cache_expires ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::tinybird_destination_keeps_forward_mode_on_forward_path ... ok
test autumn::tests::fails_open_on_transport_error ... ok
test tests::resolve_ingest_key_serves_last_known_routing_when_refresh_fails ... ok
test tests::forward_mode_switches_ready_org_to_clickhouse_without_forwarding_again ... ok
test result: ok. 38 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.23s
Doc-tests maple_ingest
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
JSON-decoded HTTP payload schemas use optionalKey per repo convention (review finding on #211). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTnfswT7zw6WhWgkS53WqM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces the Maple v2 public API — a Stripe-style, documented, stability-committed HTTP surface — as spec + foundation, with one pilot resource (
api_keys) shipped end-to-end. Full spec and rollout phases indocs/api-v2.md.The spec (
docs/api-v2.md)/v2/<resource>snake_case nouns, GET/POST/PATCH/DELETE (PATCH updates, not Stripe's legacy POST), prefixed public object IDs,{object:"list",data,has_more,next_cursor}list envelope,{error:{type,code,message,param?,doc_url?}}error envelope with a closed type set + stable codes, snake_case wire fields, ISO-8601 UTC timestamps,objectfield on every resource.txidconstraint, and phases 1–5 (core resources → telemetry reads → internal RPC + dashboard migration → idempotency/rate limits → events & webhooks).The foundation (
packages/domain/src/http/v2/)MapleApiV2HttpApi mounted alongside v1 at/v2, Scalar docs at/v2/docs, OpenAPI derived automatically.key_…,dash_…, registry inpublic-id.ts): reversible base58 boundary codec over internal UUIDs/strings — zero DB migration; Clerkorg_…/user_…pass through.limit1–100 default 20, opaque cursors)._tagon the wire) +V2SchemaErrorsmiddleware rewriting request-decode failures intoinvalid_request_errorviaHttpApiMiddleware.layerSchemaErrorTransform.api_keys.scopesjsonb + migration; grammar<family>:read|write|*;AuthorizationV2enforces mechanically from method + first/v2path segment (write implies read; session tokens and legacy null-scope keys bypass, matching Stripe's dashboard model).Pilot:
/v2/api_keyslist / create (with scopes) / retrieve / roll / revokeas thin adapters over the existingApiKeysService— proves every convention end-to-end. v1 (/api/...) is untouched apart from additive optionalscopesfields.Testing
packages/domain: 199 tests pass (public-ID round-trips incl. unicode/malformed, wire-shape + envelope encodes, scope grammar + enforcement matrix, pagination, OpenAPI contract-freeze test asserting the exact v2 surface).apps/api: 822 tests pass, including new PGlite scope-persistence tests and end-to-end HTTP tests over a real router (list envelope, 401/403/400/404 envelopes, scope enforcement incl. write-implies-read, full CRUD round-trip with public IDs, cursor pagination).packages/domain,packages/db,apps/api,apps/web.🤖 Generated with Claude Code
https://claude.ai/code/session_01TTnfswT7zw6WhWgkS53WqM
Generated by Claude Code