Skip to content

Real performance benchmark + fix dead Technical Methodology CTA#56

Merged
bedus-creation merged 6 commits into
mainfrom
task/perf-benchmark-methodology
Jul 8, 2026
Merged

Real performance benchmark + fix dead Technical Methodology CTA#56
bedus-creation merged 6 commits into
mainfrom
task/perf-benchmark-methodology

Conversation

@bedus-creation

Copy link
Copy Markdown
Contributor

Problem

The home-page performance-benchmark section had two honesty problems:

  1. The View Technical Methodology CTA was a dead <button> with no handler or href (PerformanceBenchmarks.vue), and no methodology page existed.
  2. The figures (94,203 req/s, plus Django/Express bars) were unsubstantiated and compared against unrelated stacks — apples-to-oranges.

Approach

Fixed it properly with an honest overhead-delta framing: the baseline is raw FastAPI only. Both apps serve byte-identical responses, so any difference is framework overhead — nothing else.

What's included

  • benchmarks/ harness — a reproducible uv project. apps/raw.py (plain FastAPI) and apps/startkit.py (same endpoints via the framework's Router + FastAPIProvider) serve TechEmpower-style /json and /plaintext. bench.py boots each under a single uvicorn worker, warms up, drives ab under keep-alive across N trials, and reports peak throughput (peak is the meaningful capacity number — background contention can only slow a run, never inflate it).

  • Real measured numbers — ran on Apple M3 Pro (single worker, peak of 8 trials):

    Endpoint Raw FastAPI FastAPI Startkit Difference
    JSON serialization 18,552 req/s 19,116 req/s −3.0%
    Plaintext 19,745 req/s 20,038 req/s −1.5%

    → within ±3% (both directions) = no measurable overhead over raw FastAPI.

  • docs/benchmarks.md — methodology page: test types, method, load-testing hygiene, known limitations, and reproduction steps. Added to the sidebar.

  • PerformanceBenchmarks.vue — rewritten to show the honest raw-vs-Startkit overhead delta with the measured numbers, and the CTA now links to /docs/benchmarks.

Verification

  • npm run docs:build passes.
  • Built home page contains href="/docs/benchmarks"; no 94,203/Django/Express strings remain.
  • Harness re-runnable via cd benchmarks && uv sync && uv run python bench.py.

Follow-up (out of scope)

The left-column feature bullets on this section still contain unrelated marketing claims (e.g. "Hardware-accelerated TLS termination"). Flagging for a separate copy pass.

…y page, fix dead CTA

The home-page performance section previously showed unsubstantiated figures
and compared against unrelated stacks, and its 'View Technical Methodology'
CTA was a dead <button> pointing nowhere.

- Add benchmarks/ harness: raw FastAPI baseline vs FastAPI Startkit serving
  identical TechEmpower-style endpoints, driven by ApacheBench under keep-alive,
  reporting peak throughput across trials.
- Run it and publish real measured numbers (Apple M3 Pro, single worker):
  Startkit is within +/-3% of raw FastAPI, i.e. no measurable overhead.
- Add docs/benchmarks.md methodology page documenting test types, method,
  load-testing hygiene, limitations, and reproduction steps.
- Rewrite PerformanceBenchmarks.vue to show the honest overhead delta and wire
  the CTA to /docs/benchmarks.
- Add the page to the sidebar nav.
Drop the unsubstantiated 99.9% Uptime SLA stat from the hero section and
the Observability Stack module from the core modules grid. Observability
is documented in the Logging docs, so it should not be advertised as a
standalone home-page claim. Adjust the surrounding grid layouts.
Keep the third core-module slot but repurpose it from the observability
stack (documented in the Logging docs) to an AI & Agents module,
describing LLM providers, tool orchestration, and streaming on the async
core via the container/provider pattern.
Drop the remaining <20ms P99 Latency stat; with no substantiated metrics
left, remove the hero stats block entirely.

@bedus-creation bedus-creation left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review verdict: APPROVE ✅ (posted as comment — GitHub blocks self-approval on own PR)

Reviewed full range main...HEAD at commit b36d608.

Verified

  • Numbers are real, not fabricated. Every published figure matches benchmarks/results/results.json exactly (JSON 18,552→19,116 req/s / −3.0%; Plaintext 19,745→20,038 / −1.5%; 60k req · c64 · peak of 8 trials · M3 Pro). Django/Express bars and the 99.9% uptime SLA stat are removed.
  • Dead CTA fixed. "View Technical Methodology" is now <a href="/docs/benchmarks"> (was a no-op <button>); nav link added in config.mts.
  • Site builds clean. vitepress build succeeds — no dead links, Bot icon resolves, docs/benchmarks.md renders.
  • Harness is valid & reproducible. apps/startkit.py uses real framework API (Application(providers=[...]), Router().get(), include_router, .fastapi all exist).

Non-blocking follow-ups (optional)

  1. pyproject.toml declares httpx>=0.27 but the harness never imports it (uses urllib + ab). Drop it.
  2. bench.py docstring says "takes the median of N trials" but reports the peak everywhere else. Reword.
  3. summary.md load_tool shows the raw ab -V line; docs page cleaned it to "ApacheBench 2.3" — regenerating would reintroduce the mismatch.
  4. HeroSection.vue still shows "<20ms P99 Latency", not backed by the published table (p99 measured but not surfaced).

None block merge.

Delete the local benchmark harness (benchmarks/) and rewire the docs to
the external github.com/fastapi-startkit/web-frameworks harness, which is
now the single source of truth for benchmarking.

- Replace the old local-harness figures (parity / no measurable overhead)
  with the real web-frameworks relative-delta results for the shared
  GET /, GET /user/{id}, and POST /user routes, attributed to the source.
- Update methodology to match how web-frameworks measures (Docker
  python:3.14-slim, uvicorn --workers=nproc, oha, keepalive, c64/c256) and
  document the FastAPI 0.139 include_router overhead behind POST /user.
- Point the Reproduce section at web-frameworks; remove the cd benchmarks
  / bench.py steps and all dangling harness references.
- Update PerformanceBenchmarks.vue so the home-page card matches the page
  (per-route deltas, honest framing, web-frameworks attribution).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant