diff --git a/evaluations/OVERALL_BENCHMARK_EVALUATION.md b/evaluations/OVERALL_BENCHMARK_EVALUATION.md new file mode 100644 index 0000000..a94c9f2 --- /dev/null +++ b/evaluations/OVERALL_BENCHMARK_EVALUATION.md @@ -0,0 +1,302 @@ +# Overall Codex Benchmark Evaluation + +## Scope + +This report summarizes the 13 leakage-controlled Codex benchmark runs created +for the grouped leave-one-out folds. Each fold has its own draft PR containing +the sub-agent trace, generated artifacts, and fold-level evaluation report. + +The report focuses on the question: with access to the gold code and mapping, +is it clear what went wrong, and are the failures systematic? + +## PR Index + +| Fold | Held-out index/indices | PR | Key precision | Gold recall | +| --- | --- | --- | ---: | ---: | +| 01 | 1, 2, 3, 6, 16, 27 | https://github.com/bioepic-data/data-harmonization-eval/pull/37 | 0.364 | 0.621 | +| 02 | 15, 26 | https://github.com/bioepic-data/data-harmonization-eval/pull/27 | 0.000 | 0.000 | +| 03 | 4 | https://github.com/bioepic-data/data-harmonization-eval/pull/15 | 0.551 | 1.000 | +| 04 | 5 | https://github.com/bioepic-data/data-harmonization-eval/pull/24 | 1.000 | 0.528 | +| 05 | 7 | https://github.com/bioepic-data/data-harmonization-eval/pull/20 | 1.000 | 1.000 | +| 06 | 8 | https://github.com/bioepic-data/data-harmonization-eval/pull/26 | 0.994 | 0.994 | +| 07 | 9 | https://github.com/bioepic-data/data-harmonization-eval/pull/28 | 0.000 | 0.000 | +| 08 | 10 | https://github.com/bioepic-data/data-harmonization-eval/pull/29 | 0.000 | 0.000 | +| 09 | 17 | https://github.com/bioepic-data/data-harmonization-eval/pull/30 | 1.000 | 0.998 | +| 10 | 18 | https://github.com/bioepic-data/data-harmonization-eval/pull/32 | 0.000 | 0.000 | +| 11 | 23 | https://github.com/bioepic-data/data-harmonization-eval/pull/34 | 0.000 | 0.000 | +| 12 | 24 | https://github.com/bioepic-data/data-harmonization-eval/pull/35 | 0.559 | 1.000 | +| 13 | 25 | https://github.com/bioepic-data/data-harmonization-eval/pull/36 | 0.000 | 0.000 | + +All sub-agent traces passed the targeted local no-gold path scan. No trace showed +root `data/gold`, root `data/processed`, other fold sandboxes, or held-out +`dataset_NN.py` reads under the scanned patterns. + +## Big Picture + +The benchmark is not measuring only whether an agent can map columns. In many +folds, the hard part is reproducing undocumented expert curation choices: +which rows, sites, treatments, files, and derived identifiers the expert decided +belong in the target. When those choices were simple and visible in the raw +files, the agents often got close. When the expert used hidden row filters, +external/implicit site policies, manuscript-derived constants, or exact +site-id canonicalization, the agents frequently produced plausible but +non-gold-equivalent outputs. + +The most important result is that failures are not random. They cluster into a +small number of repeated failure modes: + +1. Hidden row/site inclusion policy. +2. Site identifier canonicalization. +3. Wide-to-long parsing and replicate semantics. +4. Interval calculation policy. +5. Sentinel/QC handling and gold-data defects. +6. Literal JSON mapping mismatch despite semantically useful mappings. + +## Common Failure Modes + +### 1. Hidden Row Or Site Inclusion Policy + +This was the most common cause of major precision/recall loss. + +- Fold 03 / index 4 recovered all gold rows and all shared values, but retained + 449 extra `tb` rows. The gold code simply drops `site_id == "tb"` and hardcodes + locations for `ph1`, `ph2`, and `sg5`. That exclusion is clear in gold code, + but it was not discoverable from the fold-local skill instructions. +- Fold 04 / index 5 produced a high-precision subset but missed almost half the + gold key set. The agent followed the general treatment/control guidance and + kept control rows only; the gold for this dataset includes a broader row set. +- Fold 11 / index 23 went the other direction and over-included heavily. The + gold filters to `Sensor.Type == "SWC"` and `Treatment == "control"`, merges + sensor metadata, constructs composite site IDs, and derives replicates from + sensor/location/depth. The agent output was much larger than gold. +- Fold 12 / index 24 recovered all gold rows exactly but included 2,532 extra + keys. Gold selects only matric-potential columns matching a narrow pattern + (`P1/P2` at 15/30/60 cm). + +These are expert-label decisions, not merely transformation mechanics. The +skills contain general guidance about treatment/control and usable soil-moisture +signals, but they do not enumerate the per-dataset inclusion choices that the +gold code applies. + +### 2. Site Identifier Canonicalization + +Several near-row-count matches scored as zero keyed overlap because site IDs did +not match gold exactly. + +- Fold 08 / index 10 had the same row count as gold, but zero keyed overlap. The + gold parses site IDs as `PLM1`, `PLM2`, `PLM3` from data column names and only + uses `ER-*` identifiers in the location lookup. An agent that emits `ER-PLM1` + style site IDs is scientifically understandable but fails literal key + matching. +- Fold 10 / index 18 also had a similar row count but zero keyed overlap. Gold + constructs `site_id` from `Field_Site`, `Plot`, and `Topographic_Position` + with exact underscore/string behavior. +- Fold 13 / index 25 had a close row count but zero keyed overlap, consistent + with a site/depth/replicate canonicalization mismatch rather than a simple + missing-file problem. + +The current benchmark key is strict: `datetime_UTC`, `site_id`, `depth_m`, and +`replicate` must match. That is appropriate for target equivalence, but it means +minor-looking naming differences dominate the score. + +### 3. Wide-To-Long And Replicate Semantics + +Many datasets encode site, depth, variable type, and replicate in column names. +The agents often found the right payload files but missed the exact expert +interpretation of those encodings. + +Examples: + +- Fold 01 combines several wide micromet datasets. It achieved only 36% key + precision and 62% gold recall across the cluster. The gold code contains + dataset-specific parsing rules for file-derived site IDs, depth suffixes, + replicate counters, sentinel cleanup, and pivoting VWC/SWP into a single row. +- Fold 06 / index 8 did well overall, with roughly 99.4% precision and recall, + but remaining mismatches were concentrated in values and a small number of + keys, consistent with subtle parsing/QC differences. +- Fold 09 / index 17 was very strong: 100% key precision and 99.8% recall. + This suggests the agent can reproduce wide-to-long transforms when the naming + pattern and inclusion rule are learnable from exemplars and raw headers. + +### 4. Interval Policy + +The skill instructions recommend sorting and computing intervals within a +site/depth stream. That is usually scientifically cleaner. Gold does not always +do that. + +The clearest example is fold 05 / index 7. The agent matched all rows and all +non-interval values, but differed in six `interval_min` cells. Gold computes a +raw sequential timestamp difference across the input file. At depth transitions +that creates large negative intervals. The agent grouped by depth, so it avoided +those negative wraparound values. This is a gold-equivalence failure but arguably +not a scientific failure. + +This pattern matters because interval calculation is simultaneously: + +- an executable-output metric, +- a semantic curation choice, +- and in some gold modules, a place where questionable values are preserved. + +### 5. Sentinel Values And Gold-Data Defects + +Gold sentinel handling is inconsistent across datasets. Some modules explicitly +clean sentinel values: + +- index 1 cleans `-9999`; +- index 8 cleans `-9999`; +- index 9 cleans `-9999`; +- index 16 cleans `9999` and `-9999`. + +But the benchmark also includes known gold-data issues, including sentinel +values incorrectly retained in some gold outputs. In those cases an agent that +does the scientifically right thing and drops sentinel values can be penalized, +while an agent that imitates gold literally would preserve bad data. + +This should be split out in future scoring: + +- **gold-equivalence score**: did the agent reproduce the current expert output? +- **data-quality score**: did the agent avoid known bad values, impossible + intervals, and invalid observations? + +At present, the benchmark mostly reports gold equivalence, so it cannot +distinguish "agent error" from "agent fixed a gold bug" without manual review. + +### 6. Literal JSON Mapping Mismatch + +JSON mapping scores were generally worse than executable behavior. Common +patterns: + +- Agents often added `gravimetric_water_content` mappings because the target + schema has `gravimetric_water_content_gH2O_gs`; several gold mapping entries + omit this category even when the code fills the output column with nulls. +- Gold mapping `source_pattern` strings sometimes use normalized names + (`Depth..cm.`, `Volumetric.Water.Content`) while raw files and code use names + like `Depth (cm)` or `Volumetric Water Content`. +- Transformation prose rarely matched literally, even when the executable + transform was correct. +- Some gold mapping entries disagree with the gold code. For example, fold 03 / + index 4 has a datetime source-file mismatch between JSON and actual code/data. + +This means literal JSON comparison should be treated as a weak proxy for +semantic mapping quality. A structured semantic scorer would be more useful +than text equality. + +## Given Gold Access, Is It Clear What Went Wrong? + +Mostly yes, but not always for fair reasons. + +With the gold code open, most failures are explainable: + +- Extra or missing rows usually trace to a visible filter or missing filter in + `dataset_NN.py`. +- Zero key overlap usually traces to site ID, depth, replicate, or timestamp + canonicalization. +- Small value mismatches usually trace to sentinel handling, interval policy, or + unit conversion. +- JSON mismatches usually trace to literal wording and category coverage rather + than a totally wrong transform. + +However, many gold decisions were not recoverable from the materials available +to the sub-agents: + +- dropping particular sites such as `tb`; +- including only some sites from a package, for example Colorado-relevant + locations, when that policy is not documented in the skill; +- selecting control rows in some datasets but not applying the same heuristic in + others; +- using manuscript-derived constants such as sampling depth or date when they + are not explicit in staged raw files; +- using an exact site-id spelling that differs from a reference location ID; +- preserving questionable gold interval/sentinel behavior. + +That means some benchmark failures reflect missing task specification rather +than model inability. + +## Fold-Level Interpretation + +### Strong Or Nearly Strong Runs + +- **Fold 05 / index 7**: Essentially correct except for six interval cells where + the agent used grouped interval logic instead of gold's raw-order diff. +- **Fold 06 / index 8**: High precision and recall, with remaining subtle key + and value differences. +- **Fold 09 / index 17**: High precision and recall; most transformation logic + matched. +- **Fold 12 / index 24**: Perfect recall and exact shared-row values, but many + extra rows because the agent included more source columns/observations than + gold. +- **Fold 03 / index 4**: Perfect recall and exact shared values, but retained + the undocumented `tb` site that gold drops. + +### Weak Runs With Diagnostic Value + +- **Fold 02 / indices 15, 26**: Cluster handling failed. Dataset 15 is especially + brittle because gold relies on metadata/default constants and produces an + unexpected row profile relative to agent output. +- **Fold 07 / index 9**: Row count close but zero key overlap, likely + timestamp/site/depth/replicate canonicalization. +- **Fold 08 / index 10**: Row count exact but zero key overlap, likely site-id + canonicalization (`PLM*` vs `ER-PLM*`) and manuscript-derived depth labels. +- **Fold 10 / index 18**: Row count close but zero key overlap, likely composite + site-id/depth parsing. +- **Fold 11 / index 23**: Major over-inclusion; gold's exact SWC/control/sensor + metadata logic was not reproduced. +- **Fold 13 / index 25**: Row count close but zero key overlap, likely + canonicalization of keys. +- **Fold 01 / indices 1, 2, 3, 6, 16, 27**: Cluster-level run had moderate + recall but low precision. The cluster combines multiple difficult wide-format + parsers and sentinel/QC policies. + +## Recommendations + +1. **Separate hidden curation labels from harmonization skill.** + Add a benchmark-visible curation contract for each holdout: include/exclude + sites, treatments, files, sensors, and known bad rows. Without this, agents + are guessing hidden expert choices. + +2. **Score row inclusion and value transformation separately.** + Fold 03 and fold 12 show perfect shared-row values but poor precision due to + extra rows. This is different from wrong transformations. + +3. **Add a site-id canonicalization metric.** + Several zero-key folds likely contain useful transformed values under + non-gold site IDs. A diagnostic scorer should align candidate site IDs or + report canonicalization-only failures. + +4. **Add a "gold defect" annotation layer.** + Known sentinel-value and interval defects should be marked so the evaluator + can distinguish reproducing gold from producing scientifically cleaner data. + +5. **Make interval policy explicit.** + Decide whether intervals should be raw-row diffs or grouped by + site/depth/replicate. The current gold uses both patterns. + +6. **Use semantic JSON scoring.** + Literal comparison of prose and raw/normalized source names is too brittle. + Score destination variable, source variable identity, unit conversion, null + fill, and filtering semantics separately. + +7. **Document external metadata permissions.** + Some agents used or wanted package metadata for DOI/date/depth/location + information. The benchmark should state whether ESS-DIVE API lookups are + allowed or whether all metadata must be staged locally. + +## Bottom Line + +The agents often understood the basic soil-moisture harmonization task. The +largest failures were not generic inability to write pandas code; they were +failures to infer hidden gold curation policy, exact site-id canonicalization, +and idiosyncratic gold behavior. With gold access, most failures are clear. +Without gold access, several were not fairly recoverable from the fold-local +skills and raw files alone. + +The benchmark is therefore useful, but it should be interpreted as a combined +test of: + +- curation-policy inference, +- exact gold-code imitation, +- tabular transformation, +- and robustness to imperfect gold labels. + +Those should be disentangled before using the aggregate score as a model-quality +measure.