Skip to content

chore: DO NOT MERGE - benchmark tsam develop-v4 vs v3.4.x#110

Draft
FBumann wants to merge 2 commits into
mainfrom
chore/tsam-v4-bench
Draft

chore: DO NOT MERGE - benchmark tsam develop-v4 vs v3.4.x#110
FBumann wants to merge 2 commits into
mainfrom
chore/tsam-v4-bench

Conversation

@FBumann

@FBumann FBumann commented Jul 23, 2026

Copy link
Copy Markdown
Owner

DO NOT MERGE — measurement experiment, not a dependency change proposal.

Pins tsam to the develop-v4 git branch via [tool.uv.sources] so the Benchmarks workflow's head-vs-base comparison measures tsam v4 against released v3.4.x on the identical suite (no lockfile is committed, so base resolves the v3 release line while head resolves develop-v4). Includes the #109 benchmark suite commits since the workflow and cases must exist on the head.

Early local signal (90-day config cases, min of 5):

  • dist_global: ~87 ms → ~53 ms (~1.6× faster — consistent with the v4 rework around durationRepresentation)
  • medoid / mean / dist_cluster: ~35 ms → ~19-21 ms (~1.7× faster)
  • v4's result object now holds lazy _accuracy_metrics / _reconstructed_data internals, so the eager accuracy cost (62% of a v3 run) may have moved or vanished — the CI table will show where.

All 542 wrapper tests pass against develop-v4 locally (v4-only features like preserve_n_clusters from FZJ-IEK3-VSA/tsam#410 are not yet on that branch — the wrapper's feature detection correctly reports them absent).

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8e92b137-9d9c-490c-a653-97f40c5777c9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch chore/tsam-v4-bench

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown

Benchmarks

Δ% vs base, one table sorted by biggest change first
(min walltime and memray peak). Peak memory is
deterministic — any delta there is real; walltime on GitHub
runners is noisy, so only large time deltas are meaningful.
The interactive plot is attached as the benchmark-plot artifact.

Full table

benchmarks/test_bench_aggregate.py

name metric base head
test_wrapper_to_dataframe time 0.0003684s +17.2%
peak 8.55 MiB 0.0%
test_user_disaggregate time 0.01331s +3.4%
peak 26.7 MiB 0.0%
test_wrapper_concat_results time 0.0223s -0.3%
peak 8.72 MiB 0.0%
test_e2e_wide[dist_global] time 2.034s -1.7%
peak 104 MiB -0.9%
test_e2e_full time 0.1646s -3.4%
peak 6.6 MiB -2.1%
test_e2e_wide[medoid] time 0.557s -3.8%
peak 113 MiB -8.2%
test_user_apply time 0.4492s -4.0%
peak 104 MiB -0.9%
test_config_representation[dist_global] time 0.1399s -5.6%
peak 1.78 MiB -4.1%
test_config_representation[dist_global_minmax] time 0.1427s -6.1%
peak 1.78 MiB -4.1%
test_e2e_slices time 1.327s -6.1%
peak 14.3 MiB 0.0%
test_wrapper_result_conversion time 0.04064s -10.1%
peak 60.3 MiB 0.0%
test_e2e_default time 0.08795s -13.1%
peak 7.11 MiB -9.1%
test_config_representation[medoid] time 0.05438s -13.4%
peak 1.9 MiB -9.9%
test_config_extremes[replace] time 0.05605s -14.1%
peak 1.9 MiB -10.1%
test_config_representation[mean] time 0.05148s -14.3%
peak 1.78 MiB -4.1%
test_config_representation[dist_cluster] time 0.05351s -14.9%
peak 1.78 MiB -4.1%
test_config_extremes[append] time 0.05628s -14.9%
peak 1.9 MiB -10.2%
test_e2e_multidim time 0.2545s -20.7%
peak 26.7 MiB -8.9%

FBumann and others added 2 commits July 23, 2026 22:31
The head run leaves an untracked uv.lock behind, and uv keeps a
satisfying lockfile as-is — so a head that changes dependencies leaked
its resolution into the base run (both sides benchmarked the head's
deps). Remove it before the base run and log each side's resolved tsam
version for transparency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DO NOT MERGE — experiment to measure tsam v4 against v3.4.x with the
head-vs-base benchmark workflow. Head resolves tsam from the develop-v4
git branch; base (main) resolves the released v3 line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant