perf: opt-in blob sizes, single-parse load, parallel stats/report

README.md

@@ -6,17 +6,17 @@
 
 See [`docs/PERF.md`](docs/PERF.md) for extended benchmarks.
 
-Benchmarked on open-source repositories. `extract` reads bare clones; `stats` and `report` read the resulting JSONL. Measurements taken with a pre-built v2.11.0 binary on a single NVMe-SSD machine (not a controlled lab benchmark; directional, not absolute). `stats`/`report` now include the test-to-source ratio section, so they do slightly more work than pre-v2.11 figures.
+Benchmarked on open-source repositories. `extract` reads bare clones; `stats` and `report` read the resulting JSONL. Measurements taken on a single NVMe-SSD machine with a development build after v2.11.0 — blob-size resolution is now opt-in (`--blob-sizes`), and `stats`/`report` fan their independent metric passes across CPU cores (12 here). Not a controlled lab benchmark — directional, not absolute; `extract` is I/O-bound and carries ~10% run-to-run variance.
 
 | Repository | Commits | Devs | Extract | Stats (JSON) | Report (HTML) | JSONL size |
 |------------|---------|------|---------|-------------|--------------|------------|
-| [Pi-hole](https://github.com/pi-hole/pi-hole) | 7,077 | 281 | 0.9s | 0.21s | 0.23s | 23K lines / 6.4 MB |
-| [Praat](https://github.com/praat/praat) | 10,221 | 19 | 24s | 1.1s | 1.2s | 95K lines / 29 MB |
-| [WordPress](https://github.com/WordPress/WordPress) | 52,466 | 131 | 46s | 3.0s | 3.2s | 298K lines / 96 MB |
-| [Kubernetes](https://github.com/kubernetes/kubernetes) | 137,016 | 5,295 | 1m 58s | 11.1s | 12.0s | 943K lines / 313 MB |
-| [Linux kernel](https://github.com/torvalds/linux) | 1,438,634 | 38,832 | 11m 34s | 1m 24s | 1m 29s | 6.1M lines / 1.9 GB |
+| [Pi-hole](https://github.com/pi-hole/pi-hole) | 7,077 | 281 | 0.9s | 0.11s | 0.14s | 23K lines / 6.2 MB |
+| [Praat](https://github.com/praat/praat) | 10,221 | 19 | 20s | 0.6s | 0.6s | 95K lines / 27 MB |
+| [WordPress](https://github.com/WordPress/WordPress) | 52,466 | 131 | 40s | 1.6s | 1.8s | 298K lines / 90 MB |
+| [Kubernetes](https://github.com/kubernetes/kubernetes) | 137,016 | 5,295 | 1m 51s | 8.1s | 8.5s | 943K lines / 295 MB |
+| [Linux kernel](https://github.com/torvalds/linux) | 1,438,634 | 38,832 | 12m 46s | 47.6s | 48.8s | 6.1M lines / 1.74 GB |
 
-`extract`, `stats`, and `report` scale roughly linearly with dataset size. The per-dev collaborator map in `report` is pre-computed in a single pass over files (O(F × D_per_file²)); on the kubernetes snapshot that adds ~2 seconds over `stats`, on linux ~40 seconds. A previous implementation computed this nested inside the per-dev loop (O(D × F × D_per_file)) and was 6× slower on kubernetes and 11× slower on linux. If you only need the aggregate data, `stats --format json` is always the fastest path; reach for `report` when you actually want the HTML dashboard.
+`extract`, `stats`, and `report` scale roughly linearly with dataset size. Since the post-v2.11.0 work, `stats` and `report` run their independent metric passes concurrently across cores — on this 12-core machine that cut their wall time ~30–49% versus v2.11.0 (e.g. WordPress report 3.2s → 1.8s, Linux 1m29s → 49s) and brings the two within a few percent of each other (`report` does a little more — repo tree, dev network — but it overlaps the other passes). `extract` is unchanged within run-to-run variance: it's bound by the `git log` stream, not the now-optional blob-size lookup. `stats --format json` is the leanest path when you only need aggregate data; reach for `report` when you want the HTML dashboard.
 
 ## Privacy and reliability
 
@@ -107,11 +107,13 @@ The `--mailmap` flag uses git's built-in `.mailmap` support to unify developer i
 
 ### What gitcortex collects from git
 
-Extraction runs two git commands against the local repository and streams their output. No source-code bytes are read.
+Extraction runs one git command against the local repository by default (a
+second only when `--blob-sizes` is set) and streams the output. No source-code
+bytes are read.
 
 ```
 git log -M --raw --numstat --format=<metadata> <branch>    → commits, parents, per-file diffs (counts only)
-git cat-file --batch-check                                 → blob sizes (old/new) for each file change
+git cat-file --batch-check   (only with --blob-sizes)      → blob sizes (old/new) for each file change
 ```
 
 Per-commit metadata (populates the `commit` record):
@@ -130,7 +132,8 @@ Per-file-change metadata (populates the `commit_file` record):
 |---|---|---|
 | `path_current`, `path_previous`, `status` | `git log --raw` | hotspots, directories, extensions, rename tracking (`R100` / `C075` trigger merges) |
 | `additions`, `deletions` | `git log --numstat` | per-file churn, recent churn, coupling |
-| `old_hash`, `new_hash`, `old_size`, `new_size` | `git cat-file --batch-check` | retained but not currently used in stats |
+| `old_hash`, `new_hash` | `git log --raw` | emitted but not consumed by any stat |
+| `old_size`, `new_size` | `git cat-file --batch-check` (opt-in: `--blob-sizes`) | blob byte sizes; **off by default** (the lookup dominated extract time and no stat reads them), so absent from the JSONL unless `--blob-sizes` is passed |
 
 **Not collected:**
 - File contents / diff hunks — only line counts from `--numstat`.
@@ -143,6 +146,7 @@ Per-file-change metadata (populates the `commit_file` record):
 - `--mailmap` — normalizes author/committer names+emails via git's `.mailmap` before recording (off by default; warned when a `.mailmap` exists but the flag is omitted).
 - `--ignore <glob>` — drops matching `commit_file` records entirely at extract time (counts in the `commit` record are recomputed so totals remain consistent).
 - `--first-parent` — traverses only the first-parent chain, skipping merged branch history.
+- `--blob-sizes` — resolves per-blob byte sizes via `git cat-file --batch-check` and emits `old_size`/`new_size` (off by default). The lookup is the bulk of any cat-file cost and no gitcortex stat consumes the sizes, so enable this only when an external consumer of the JSONL needs them.
 
 Full per-record schema (every field, types, enums): see [`docs/RUNBOOK.md`](docs/RUNBOOK.md#jsonl-format).
 
@@ -151,7 +155,7 @@ Output is a JSONL file with one record per line. Four record types:
 ```jsonl
 {"type":"commit","sha":"abc...","tree":"def...","parents":["ghi..."],"author_name":"Alice","author_email":"alice@example.com","author_date":"2024-01-15T10:30:00Z","committer_name":"Alice","committer_email":"alice@example.com","committer_date":"2024-01-15T10:30:00Z","message":"","additions":42,"deletions":7,"files_changed":3}
 {"type":"commit_parent","sha":"abc...","parent_sha":"ghi..."}
-{"type":"commit_file","commit":"abc...","path_current":"src/main.go","path_previous":"src/main.go","status":"M","old_hash":"111...","new_hash":"222...","old_size":1024,"new_size":1087,"additions":10,"deletions":3}
+{"type":"commit_file","commit":"abc...","path_current":"src/main.go","path_previous":"src/main.go","status":"M","old_hash":"111...","new_hash":"222...","additions":10,"deletions":3}
 {"type":"dev","dev_id":"sha256hash...","name":"Alice","email":"alice@example.com"}
 ```
 
@@ -534,12 +538,13 @@ internal/
 
 ### Extraction pipeline
 
-Two long-running git processes for the entire extraction, regardless of repository size:
+One long-running git process for the entire extraction (a second only with
+`--blob-sizes`), regardless of repository size:
 
 ```
 git log --raw --numstat -M --- single stream ---- parse ---- emit JSONL
                                                     |
-git cat-file --batch-check -- long-running ---- resolve blob sizes
+git cat-file --batch-check -- long-running ---- resolve blob sizes  (only with --blob-sizes)
 ```
 
 ### Stats pipeline

cmd/gitcortex/main.go

@@ -12,6 +12,7 @@ import (
 	"runtime"
 	"sort"
 	"strings"
+	"sync"
 	"syscall"
 	"time"
 
@@ -84,6 +85,7 @@ func extractCmd() *cobra.Command {
 	cmd.Flags().DurationVar(&cfg.CommandTimeout, "command-timeout", extract.DefaultCommandTimeout, "Maximum duration for git commands")
 	cmd.Flags().BoolVar(&cfg.FirstParent, "first-parent", false, "Restrict to first-parent chain")
 	cmd.Flags().BoolVar(&cfg.Mailmap, "mailmap", false, "Use .mailmap to normalize author/committer identities")
+	cmd.Flags().BoolVar(&cfg.BlobSizes, "blob-sizes", false, "Resolve per-blob byte sizes via cat-file (off by default; dominates extract time and is unused by stats)")
 	cmd.Flags().StringSliceVar(&cfg.IgnorePatterns, "ignore", nil, "Glob patterns to exclude files (e.g. package-lock.json, *.min.js)")
 
 	return cmd
@@ -423,59 +425,83 @@ func renderStatsJSON(f *stats.Formatter, ds *stats.Dataset, sf *statsFlags) erro
 	showAll := sf.stat == ""
 	report := make(map[string]interface{})
 
+	// Each selected stat is an independent pure read over the immutable
+	// Dataset, so compute them concurrently and guard only the brief map
+	// write. In the common showAll path this fans ~13 passes across cores
+	// instead of running serially. JSON output is unaffected: encoding/json
+	// sorts map keys, so completion order doesn't change the bytes.
+	var (
+		mu sync.Mutex
+		wg sync.WaitGroup
+	)
+	put := func(key string, compute func() interface{}) {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			v := compute()
+			mu.Lock()
+			report[key] = v
+			mu.Unlock()
+		}()
+	}
+
 	if showAll || sf.stat == "summary" {
-		report["summary"] = stats.ComputeSummary(ds)
+		put("summary", func() interface{} { return stats.ComputeSummary(ds) })
 	}
 	if showAll || sf.stat == "contributors" {
-		report["contributors"] = stats.TopContributors(ds, sf.topN)
+		put("contributors", func() interface{} { return stats.TopContributors(ds, sf.topN) })
 	}
 	if showAll || sf.stat == "hotspots" {
-		report["hotspots"] = stats.FileHotspots(ds, sf.topN)
+		put("hotspots", func() interface{} { return stats.FileHotspots(ds, sf.topN) })
 	}
 	if showAll || sf.stat == "directories" {
-		report["directories"] = stats.DirectoryStats(ds, sf.topN)
+		put("directories", func() interface{} { return stats.DirectoryStats(ds, sf.topN) })
 	}
 	if showAll || sf.stat == "extensions" {
-		report["extensions"] = stats.ExtensionStats(ds, sf.topN)
+		put("extensions", func() interface{} { return stats.ExtensionStats(ds, sf.topN) })
 	}
 	if showAll || sf.stat == "tests" {
-		cfg := stats.NewTestConfig(sf.testGlobs)
-		report["tests"] = map[string]interface{}{
-			"summary": stats.ComputeTestSummary(ds, cfg, sf.topN),
-			"trend":   stats.TestRatioOverTime(ds, cfg, sf.granularity),
-		}
+		put("tests", func() interface{} {
+			cfg := stats.NewTestConfig(sf.testGlobs)
+			return map[string]interface{}{
+				"summary": stats.ComputeTestSummary(ds, cfg, sf.topN),
+				"trend":   stats.TestRatioOverTime(ds, cfg, sf.granularity),
+			}
+		})
 	}
 	if showAll || sf.stat == "activity" {
-		report["activity"] = stats.ActivityOverTime(ds, sf.granularity)
+		put("activity", func() interface{} { return stats.ActivityOverTime(ds, sf.granularity) })
 	}
 	if showAll || sf.stat == "busfactor" {
-		report["busfactor"] = stats.BusFactor(ds, sf.topN)
+		put("busfactor", func() interface{} { return stats.BusFactor(ds, sf.topN) })
 	}
 	if showAll || sf.stat == "coupling" {
-		report["coupling"] = stats.FileCoupling(ds, sf.topN, sf.couplingMinChanges)
+		put("coupling", func() interface{} { return stats.FileCoupling(ds, sf.topN, sf.couplingMinChanges) })
 	}
 	if showAll || sf.stat == "churn-risk" {
-		report["churn_risk"] = stats.ChurnRisk(ds, sf.topN)
+		put("churn_risk", func() interface{} { return stats.ChurnRisk(ds, sf.topN) })
 	}
 	if showAll || sf.stat == "working-patterns" {
-		report["working_patterns"] = stats.WorkingPatterns(ds)
+		put("working_patterns", func() interface{} { return stats.WorkingPatterns(ds) })
 	}
 	if showAll || sf.stat == "dev-network" {
-		report["dev_network"] = stats.DeveloperNetwork(ds, sf.topN, sf.networkMinFiles)
+		put("dev_network", func() interface{} { return stats.DeveloperNetwork(ds, sf.topN, sf.networkMinFiles) })
 	}
 	if sf.stat == "profile" {
-		report["profiles"] = stats.DevProfiles(ds, sf.email, 0, stats.NewTestConfig(sf.testGlobs))
+		put("profiles", func() interface{} { return stats.DevProfiles(ds, sf.email, 0, stats.NewTestConfig(sf.testGlobs)) })
 	}
 	if showAll || sf.stat == "pareto" {
-		report["pareto"] = reportpkg.ComputePareto(ds)
+		put("pareto", func() interface{} { return reportpkg.ComputePareto(ds) })
 	}
 	if showAll || sf.stat == "top-commits" {
-		report["top_commits"] = stats.TopCommits(ds, sf.topN)
+		put("top_commits", func() interface{} { return stats.TopCommits(ds, sf.topN) })
 	}
 	if sf.stat == "structure" {
-		report["structure"] = reportpkg.BuildRepoTree(stats.FileHotspots(ds, 0), sf.treeDepth)
+		put("structure", func() interface{} { return reportpkg.BuildRepoTree(stats.FileHotspots(ds, 0), sf.treeDepth) })
 	}
 
+	wg.Wait()
+
 	return f.PrintReport(report)
 }
 
@@ -1190,6 +1216,7 @@ func scanCmd() *cobra.Command {
 		mailmap            bool
 		firstParent        bool
 		includeMessages    bool
+		blobSizes          bool
 		couplingMaxFiles   int
 		couplingMinChanges int
 		churnHalfLife      int
@@ -1272,6 +1299,7 @@ breakdown — handy for showing aggregated work across many repos.`,
 					CommandTimeout:  extract.DefaultCommandTimeout,
 					FirstParent:     firstParent,
 					Mailmap:         mailmap,
+					BlobSizes:       blobSizes,
 					IgnorePatterns:  extractIgnore,
 					StartOffset:     -1,
 				},
@@ -1369,6 +1397,7 @@ breakdown — handy for showing aggregated work across many repos.`,
 	cmd.Flags().IntVar(&batchSize, "batch-size", 1000, "Per-repo extract checkpoint interval")
 	cmd.Flags().BoolVar(&mailmap, "mailmap", false, "Use .mailmap (per repo) to normalize identities")
 	cmd.Flags().BoolVar(&firstParent, "first-parent", false, "Restrict extracts to the first-parent chain")
+	cmd.Flags().BoolVar(&blobSizes, "blob-sizes", false, "Resolve per-blob byte sizes via cat-file (off by default; dominates extract time and is unused by stats)")
 	cmd.Flags().BoolVar(&includeMessages, "include-commit-messages", false, "Include commit messages in JSONL (needed for Top Commits in the consolidated report)")
 	cmd.Flags().IntVar(&couplingMaxFiles, "coupling-max-files", 50, "Max files per commit for coupling analysis (consolidated report)")
 	cmd.Flags().IntVar(&couplingMinChanges, "coupling-min-changes", 5, "Min co-changes for coupling results (consolidated report)")

cmd/gitcortex/main_test.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"bytes"
+	"encoding/json"
 	"fmt"
 	"os"
 	"os/exec"
@@ -13,6 +14,86 @@ import (
 	"github.com/lex0c/gitcortex/internal/stats"
 )
 
+// renderStatsJSON fans the (showAll) stats across goroutines and writes them
+// into a shared map under a mutex. This guards that path two ways that the
+// rest of the suite did not cover: (1) run under `go test -race` it trips the
+// detector on any unsynchronized access; (2) running it repeatedly and
+// asserting byte-identical output catches a dropped/racy map write that would
+// silently omit a stat or reorder bytes. encoding/json sorts map keys, so a
+// correct parallel run is deterministic — any diff here is a real defect.
+func TestRenderStatsJSON_ParallelDeterministicAndComplete(t *testing.T) {
+	// Non-trivial fixture: 4 commits, 2 devs, 3 files, with a.go/b.go
+	// co-changing twice (coupling edge) and alice/bob sharing files
+	// (dev-network edge) so every showAll stat produces real output.
+	fixture := strings.Join([]string{
+		`{"type":"dev","dev_id":"d1","name":"Alice","email":"alice@x.com"}`,
+		`{"type":"dev","dev_id":"d2","name":"Bob","email":"bob@x.com"}`,
+		`{"type":"commit","sha":"c1","author_name":"Alice","author_email":"alice@x.com","author_date":"2024-01-01T10:00:00Z","additions":15,"deletions":0,"files_changed":2}`,
+		`{"type":"commit_file","commit":"c1","path_current":"a.go","path_previous":"a.go","status":"M","additions":10,"deletions":0}`,
+		`{"type":"commit_file","commit":"c1","path_current":"b.go","path_previous":"b.go","status":"M","additions":5,"deletions":0}`,
+		`{"type":"commit","sha":"c2","author_name":"Bob","author_email":"bob@x.com","author_date":"2024-01-02T10:00:00Z","additions":11,"deletions":1,"files_changed":2}`,
+		`{"type":"commit_file","commit":"c2","path_current":"a.go","path_previous":"a.go","status":"M","additions":3,"deletions":1}`,
+		`{"type":"commit_file","commit":"c2","path_current":"c.go","path_previous":"c.go","status":"M","additions":8,"deletions":0}`,
+		`{"type":"commit","sha":"c3","author_name":"Alice","author_email":"alice@x.com","author_date":"2024-01-03T10:00:00Z","additions":6,"deletions":3,"files_changed":2}`,
+		`{"type":"commit_file","commit":"c3","path_current":"a.go","path_previous":"a.go","status":"M","additions":2,"deletions":2}`,
+		`{"type":"commit_file","commit":"c3","path_current":"b.go","path_previous":"b.go","status":"M","additions":4,"deletions":1}`,
+		`{"type":"commit","sha":"c4","author_name":"Bob","author_email":"bob@x.com","author_date":"2024-02-01T10:00:00Z","additions":1,"deletions":1,"files_changed":1}`,
+		`{"type":"commit_file","commit":"c4","path_current":"c.go","path_previous":"c.go","status":"M","additions":1,"deletions":1}`,
+	}, "\n") + "\n"
+
+	dir := t.TempDir()
+	path := filepath.Join(dir, "fixture.jsonl")
+	if err := os.WriteFile(path, []byte(fixture), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	ds, err := stats.LoadJSONL(path)
+	if err != nil {
+		t.Fatalf("LoadJSONL: %v", err)
+	}
+
+	sf := &statsFlags{
+		format: "json", topN: 10, granularity: "month", stat: "",
+		couplingMaxFiles: 50, couplingMinChanges: 1, churnHalfLife: 90,
+		networkMinFiles: 1, treeDepth: 3,
+	}
+
+	render := func() string {
+		var buf bytes.Buffer
+		f := stats.NewFormatter(&buf, "json")
+		if err := renderStatsJSON(f, ds, sf); err != nil {
+			t.Fatalf("renderStatsJSON: %v", err)
+		}
+		return buf.String()
+	}
+
+	// Determinism: many runs over the same Dataset must be byte-identical.
+	// A racy map write or lost goroutine result would diverge here, and
+	// -race would flag the access.
+	first := render()
+	for i := 0; i < 12; i++ {
+		if got := render(); got != first {
+			t.Fatalf("run %d diverged from first run — parallel assembly is not deterministic", i)
+		}
+	}
+
+	// Completeness: every showAll stat must be present. A dropped parallel
+	// write would silently omit one — this asserts the full inventory landed.
+	var out map[string]json.RawMessage
+	if err := json.Unmarshal([]byte(first), &out); err != nil {
+		t.Fatalf("output is not valid JSON: %v", err)
+	}
+	want := []string{
+		"summary", "contributors", "hotspots", "directories", "extensions",
+		"tests", "activity", "busfactor", "coupling", "churn_risk",
+		"working_patterns", "dev_network", "pareto", "top_commits",
+	}
+	for _, k := range want {
+		if _, ok := out[k]; !ok {
+			t.Errorf("showAll output missing key %q (parallel write dropped?)", k)
+		}
+	}
+}
+
 // scanCmd must validate --since BEFORE running the discovery walk
 // and extract pool. Without the early check, an obvious typo like
 // `--since 1yy` only surfaces after scan.Run has already walked