Skip to content

docs: add benchmark results matrix to README#769

Merged
waltsims merged 2 commits into
masterfrom
docs/benchmark-results-matrix
Jul 20, 2026
Merged

docs: add benchmark results matrix to README#769
waltsims merged 2 commits into
masterfrom
docs/benchmark-results-matrix

Conversation

@waltsims

@waltsims waltsims commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a ## Benchmarks section to the front-page README (docs/README.md) with a 10-row matrix covering (backend, OS, accelerator) combinations against three grid sizes: 64³, 128³, 256³.

All cells ship empty (em-dash) as a scaffold. The plan is for contributors to fill rows via follow-up commits on this branch:

  • Magda: macOS (Apple Silicon), python backend, CPU
  • Farid: any hardware you have available
  • Anyone else with a machine that fits a row

Grid-size columns are drawn from the isotropic sizes in the default x/y/z_scale_array in benchmarks/helpers.py (indices 0/3/6/9 = 32³/64³/128³/256³, dropped 32³ as too small to be interesting). 512³ was excluded because many workstations OOM.

Rows omitted intentionally:

  • python/cpp + macOS + GPU — no CUDA on macOS.
  • cpp + Intel Mac + CPU — rare; can be added when someone runs it.

Reproducer command lives in benchmarks/README.md; the section links to it.

Test plan

  • Section renders correctly on GitHub (table alignment, em-dash characters)
  • Section renders correctly on PyPI once the next release ships (docs/README.md is the readme = in pyproject.toml)
  • Contributors can fill a row without merge conflicts on other rows

🤖 Generated with Claude Code

Greptile Summary

This PR adds benchmark results to the public README. The main changes are:

  • A backend, operating system, accelerator, and hardware matrix.
  • Runtime columns for 64³, 128³, and 256³ grids.
  • An initial Apple M1 result and contribution instructions.

Confidence Score: 5/5

The documentation is mergeable after small fixes to the benchmark links and reproduction guidance.

  • The matrix dimensions and stated benchmark defaults match the implementation.
  • The script link opens a different file than its label names.
  • The default reproducer runs large cases that the matrix intentionally excludes.

docs/README.md

Important Files Changed

Filename Overview
docs/README.md Adds the benchmark matrix and contribution guidance; the benchmark defaults are accurate, but one link is mislabeled and the linked default reproducer runs cases outside the matrix.

Reviews (1): Last reviewed commit: "docs: fill python/macOS/CPU 64³ benchmar..." | Re-trigger Greptile

Greptile also left 2 inline comments on this PR.

Adds a Benchmarks section to the front-page README with a 10-row
matrix covering (backend, OS, accelerator) combinations against
three grid sizes (64³, 128³, 256³). All cells ship empty (em-dash)
as a scaffold — contributors fill rows via follow-up commits with
their hardware, k-wave-python version, and BINARY_VERSION.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@waltsims
waltsims requested a review from faridyagubbayli July 16, 2026 15:02
@waltsims

Copy link
Copy Markdown
Owner Author

@faridyagubbayli @djps

it would be great if you two could add entries here!

@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.83%. Comparing base (dc21546) to head (9451b8a).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #769   +/-   ##
=======================================
  Coverage   75.83%   75.83%           
=======================================
  Files          58       58           
  Lines        8274     8274           
  Branches     1614     1614           
=======================================
  Hits         6275     6275           
  Misses       1373     1373           
  Partials      626      626           
Flag Coverage Δ
3.10 75.80% <ø> (ø)
3.11 75.80% <ø> (ø)
3.12 75.80% <ø> (ø)
3.13 75.80% <ø> (ø)
macos-latest 75.71% <ø> (ø)
ubuntu-latest 75.71% <ø> (ø)
windows-latest 75.64% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Measured on Apple M1 MacBook Air (8 GB) against k-wave-python 0.6.3rc1:
64³ = 81 s (python backend, CPU, 1000 timesteps, 3 averages).
128³/256³ left empty — 256³ exceeds 8 GB RAM on this machine.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@waltsims
waltsims marked this pull request as ready for review July 20, 2026 22:08
@waltsims
waltsims merged commit 011c8ed into master Jul 20, 2026
34 checks passed
@waltsims
waltsims deleted the docs/benchmark-results-matrix branch July 20, 2026 22:09
Comment thread docs/README.md

## Benchmarks

Reference runtimes for the 3D scaling benchmark in [`benchmarks/benchmark.py`](../benchmarks/README.md). Values are total elapsed seconds for a single default run (3D initial-value problem, heterogeneous absorbing medium, 1000 timesteps, averaged over 3 repeats).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Benchmark Link Opens Wrong File

The link is labeled benchmarks/benchmark.py, but it opens benchmarks/README.md. Readers clicking it to inspect the benchmark implementation are sent to the instructions instead; the instructions already have a separate link below.

Suggested change
Reference runtimes for the 3D scaling benchmark in [`benchmarks/benchmark.py`](../benchmarks/README.md). Values are total elapsed seconds for a single default run (3D initial-value problem, heterogeneous absorbing medium, 1000 timesteps, averaged over 3 repeats).
Reference runtimes for the 3D scaling benchmark in [`benchmarks/benchmark.py`](../benchmarks/benchmark.py). Values are total elapsed seconds for a single default run (3D initial-value problem, heterogeneous absorbing medium, 1000 timesteps, averaged over 3 repeats).

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread docs/README.md
| cpp | Windows | CPU (OMP) | — | — | — | — |
| cpp | Windows | NVIDIA GPU (CUDA) | — | — | — | — |

Contributions welcome — open a PR filling a row with your k-wave-python version, and (for `cpp` backend) `BINARY_VERSION`. See [`benchmarks/README.md`](../benchmarks/README.md) for the reproducer command.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Reproducer Runs Excluded Grid Sizes

The linked default command runs all 12 benchmark cases, including the large cases excluded from this matrix because workstations may run out of memory. A contributor following this instruction to fill a 64³, 128³, or 256³ cell can therefore spend time on unrelated cases or encounter an out-of-memory failure before the command completes; provide a bounded command that reproduces the listed sizes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant