Real performance benchmark + fix dead Technical Methodology CTA#56
Merged
Conversation
…y page, fix dead CTA The home-page performance section previously showed unsubstantiated figures and compared against unrelated stacks, and its 'View Technical Methodology' CTA was a dead <button> pointing nowhere. - Add benchmarks/ harness: raw FastAPI baseline vs FastAPI Startkit serving identical TechEmpower-style endpoints, driven by ApacheBench under keep-alive, reporting peak throughput across trials. - Run it and publish real measured numbers (Apple M3 Pro, single worker): Startkit is within +/-3% of raw FastAPI, i.e. no measurable overhead. - Add docs/benchmarks.md methodology page documenting test types, method, load-testing hygiene, limitations, and reproduction steps. - Rewrite PerformanceBenchmarks.vue to show the honest overhead delta and wire the CTA to /docs/benchmarks. - Add the page to the sidebar nav.
Drop the unsubstantiated 99.9% Uptime SLA stat from the hero section and the Observability Stack module from the core modules grid. Observability is documented in the Logging docs, so it should not be advertised as a standalone home-page claim. Adjust the surrounding grid layouts.
Keep the third core-module slot but repurpose it from the observability stack (documented in the Logging docs) to an AI & Agents module, describing LLM providers, tool orchestration, and streaming on the async core via the container/provider pattern.
Drop the remaining <20ms P99 Latency stat; with no substantiated metrics left, remove the hero stats block entirely.
bedus-creation
commented
Jul 8, 2026
bedus-creation
left a comment
Contributor
Author
There was a problem hiding this comment.
Code Review verdict: APPROVE ✅ (posted as comment — GitHub blocks self-approval on own PR)
Reviewed full range main...HEAD at commit b36d608.
Verified
- Numbers are real, not fabricated. Every published figure matches
benchmarks/results/results.jsonexactly (JSON 18,552→19,116 req/s / −3.0%; Plaintext 19,745→20,038 / −1.5%; 60k req · c64 · peak of 8 trials · M3 Pro). Django/Express bars and the 99.9% uptime SLA stat are removed. - Dead CTA fixed. "View Technical Methodology" is now
<a href="/docs/benchmarks">(was a no-op<button>); nav link added inconfig.mts. - Site builds clean.
vitepress buildsucceeds — no dead links,Boticon resolves,docs/benchmarks.mdrenders. - Harness is valid & reproducible.
apps/startkit.pyuses real framework API (Application(providers=[...]),Router().get(),include_router,.fastapiall exist).
Non-blocking follow-ups (optional)
pyproject.tomldeclareshttpx>=0.27but the harness never imports it (usesurllib+ab). Drop it.bench.pydocstring says "takes the median of N trials" but reports the peak everywhere else. Reword.summary.mdload_toolshows the rawab -Vline; docs page cleaned it to "ApacheBench 2.3" — regenerating would reintroduce the mismatch.HeroSection.vuestill shows "<20ms P99 Latency", not backed by the published table (p99 measured but not surfaced).
None block merge.
Delete the local benchmark harness (benchmarks/) and rewire the docs to
the external github.com/fastapi-startkit/web-frameworks harness, which is
now the single source of truth for benchmarking.
- Replace the old local-harness figures (parity / no measurable overhead)
with the real web-frameworks relative-delta results for the shared
GET /, GET /user/{id}, and POST /user routes, attributed to the source.
- Update methodology to match how web-frameworks measures (Docker
python:3.14-slim, uvicorn --workers=nproc, oha, keepalive, c64/c256) and
document the FastAPI 0.139 include_router overhead behind POST /user.
- Point the Reproduce section at web-frameworks; remove the cd benchmarks
/ bench.py steps and all dangling harness references.
- Update PerformanceBenchmarks.vue so the home-page card matches the page
(per-route deltas, honest framing, web-frameworks attribution).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The home-page performance-benchmark section had two honesty problems:
<button>with no handler or href (PerformanceBenchmarks.vue), and no methodology page existed.94,203 req/s, plus Django/Express bars) were unsubstantiated and compared against unrelated stacks — apples-to-oranges.Approach
Fixed it properly with an honest overhead-delta framing: the baseline is raw FastAPI only. Both apps serve byte-identical responses, so any difference is framework overhead — nothing else.
What's included
benchmarks/harness — a reproducibleuvproject.apps/raw.py(plain FastAPI) andapps/startkit.py(same endpoints via the framework'sRouter+FastAPIProvider) serve TechEmpower-style/jsonand/plaintext.bench.pyboots each under a single uvicorn worker, warms up, drivesabunder keep-alive across N trials, and reports peak throughput (peak is the meaningful capacity number — background contention can only slow a run, never inflate it).Real measured numbers — ran on Apple M3 Pro (single worker, peak of 8 trials):
→ within ±3% (both directions) = no measurable overhead over raw FastAPI.
docs/benchmarks.md— methodology page: test types, method, load-testing hygiene, known limitations, and reproduction steps. Added to the sidebar.PerformanceBenchmarks.vue— rewritten to show the honest raw-vs-Startkit overhead delta with the measured numbers, and the CTA now links to/docs/benchmarks.Verification
npm run docs:buildpasses.href="/docs/benchmarks"; no94,203/Django/Express strings remain.cd benchmarks && uv sync && uv run python bench.py.Follow-up (out of scope)
The left-column feature bullets on this section still contain unrelated marketing claims (e.g. "Hardware-accelerated TLS termination"). Flagging for a separate copy pass.