[codex] add fold04 agent evaluation bundle

evaluations/fold-04-holdout-5/AGENT_ACTION_LOG.md

@@ -0,0 +1,24 @@
+- 2026-06-30T15:54:33-07:00 | action=command | subject=`mkdir -p /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/agent_outputs` | reason=initialize benchmark output directory and action log
+- 2026-06-30T15:54:42-07:00 | action=read | subject=`/scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/{AGENT_INSTRUCTIONS.md,MANIFEST.json,skills/essdive_sm_curator/SKILL.md,skills/essdive_sm_harmonizer/SKILL.md}` | reason=load fold-local task instructions and allowed skills
+- 2026-06-30T15:54:51-07:00 | action=read | subject=`/scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/skills/essdive_sm_curator/SKILL.md lines 261-520 and skills/essdive_sm_harmonizer/SKILL.md lines 261-620` | reason=complete fold-local skill instructions before acting
+- 2026-06-30T15:54:58-07:00 | action=command | subject=`wc -l /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/skills/essdive_sm_curator/SKILL.md /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/skills/essdive_sm_harmonizer/SKILL.md` | reason=verify whether skill reads reached end of files
+- 2026-06-30T15:55:06-07:00 | action=read | subject=`/scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/skills/essdive_sm_curator/SKILL.md lines 521-556` | reason=finish reading fold-local curator skill
+- 2026-06-30T15:55:13-07:00 | action=directory_listing | subject=`find /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5 -maxdepth 5 -type f` | reason=identify fold-local allowed exemplars and metadata files
+- 2026-06-30T15:55:23-07:00 | action=read | subject=`/scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/data/processed/ess-dive_wfsfa_soil_datasets/sm_data_harmonization_mapping.json and selected fold-local expert code` | reason=inspect allowed schema examples and code patterns for harmonizer output
+- 2026-06-30T15:55:35-07:00 | action=directory_listing | subject=`find /h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-8ac2940c708a515-20230504T210140482233 -maxdepth 3 -type f` | reason=inspect allowed held-out raw package files
+- 2026-06-30T15:55:42-07:00 | action=directory_listing | subject=`find /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/data -maxdepth 6 -type f` | reason=locate fold-local cached metadata and allowed exemplar files
+- 2026-06-30T15:56:34-07:00 | action=read | subject=`python inspect CSV headers and previews in /h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-8ac2940c708a515-20230504T210140482233` | reason=classify raw held-out payload, sensor metadata, and units
+- 2026-06-30T15:56:55-07:00 | action=read | subject=`python summarize /h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-8ac2940c708a515-20230504T210140482233 CSVs` | reason=determine held-out payload row counts, unique sites, sensors, depths, and interval
+- 2026-06-30T15:57:10-07:00 | action=search | subject=`rg -n "Microclimate|SM_SWC|SFA_Location|Sensor.SN|Treatment|Subplot|SWC" /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/data` | reason=find allowed exemplar patterns for this raw file structure
+- 2026-06-30T15:57:19-07:00 | action=read | subject=`/scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/data/gold/expert_code/harmonize_sm/dataset_23.py and mapping lines 1500-1635` | reason=inspect allowed analogous treatment-filtered SWC harmonization pattern
+- 2026-06-30T15:57:32-07:00 | action=directory_listing | subject=`find /h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive_b924ba040ecefee_20250707T154402200 -maxdepth 4 -type f` | reason=locate allowed raw reference location registry cited by fold-local mapping index 0
+- 2026-06-30T15:57:46-07:00 | action=read | subject=`python inspect /h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive_b924ba040ecefee_20250707T154402200/data/East_Taylor_Watershed_Community_Observatory_Sites___Point_Locations__Surface_v3_2_20250327.csv and match held-out SFA_Location values` | reason=resolve held-out site coordinates using allowed raw location registry
+- 2026-06-30T15:58:02-07:00 | action=read | subject=`python search matched prefixes in /h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive_b924ba040ecefee_20250707T154402200/data/East_Taylor_Watershed_Community_Observatory_Sites___Point_Locations__Surface_v3_2_20250327.csv for missing held-out locations` | reason=resolve or document unmatched location IDs
+- 2026-06-30T15:58:27-07:00 | action=read | subject=`python dry-run held-out control filter and coordinate coverage for /h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-8ac2940c708a515-20230504T210140482233` | reason=validate harmonization decisions before writing outputs
+- 2026-06-30T15:58:45-07:00 | action=command | subject=`curl -L https://api.ess-dive.lbl.gov/packages/ess-dive-8ac2940c708a515-20230504T210140482233` | reason=retrieve public ESS-DIVE metadata DOI because no fold-local cache is present
+- 2026-06-30T15:58:54-07:00 | action=command | subject=`curl -L https://api.ess-dive.lbl.gov/packages/ess-dive-8ac2940c708a515-20230504T210140482233` | reason=retry public ESS-DIVE metadata DOI lookup after sandbox DNS failure
+- 2026-06-30T16:01:05-07:00 | action=write | subject=`python generate curator_bundle.json mapping.json mapping_notes.md harmonize_heldout.py in /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/agent_outputs` | reason=create required benchmark output artifacts from allowed raw files and fold-local exemplars
+- 2026-06-30T16:01:26-07:00 | action=command | subject=`python /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/agent_outputs/harmonize_heldout.py` | reason=run generated harmonizer to create heldout_harmonized.csv
+- 2026-06-30T16:01:48-07:00 | action=read | subject=`python validate files in /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/agent_outputs` | reason=verify required output files, schema, row count, and basic harmonized values
+- 2026-06-30T16:02:05-07:00 | action=command | subject=`python -m py_compile /scratch/jmc/data-harmonization-eval/.runs/fold-04-holdout-5/agent_outputs/harmonize_heldout.py` | reason=syntax-check generated harmonization script
+- 2026-06-30T16:02:54-07:00 | action=command | subject=`python action-log forbidden-path scan` | reason=verify trace contains no forbidden outside-fold access patterns

evaluations/fold-04-holdout-5/AGENT_INSTRUCTIONS.md

@@ -0,0 +1,10 @@
+# Run environment — leave-one-cluster-out
+
+Harmonize the held-out dataset(s) below using ONLY:
+- the skills in `skills/`,
+- the exemplars in `data/processed/.../sm_data_harmonization_mapping.json` and the code patterns in `data/gold/expert_code/harmonize_sm/` (both have the held-out cluster removed),
+- the shared raw inputs under `~/ess-dive_wfsfa_soil_datasets/` and the cached metadata under `data/external/ess-dive_meta/`.
+
+Do NOT look up the held-out dataset's existing harmonized output, expert code, or mapping entry from any other location. The held-out datasets are:
+
+- `ess-dive-8ac2940c708a515-20230504T210140482233`

evaluations/fold-04-holdout-5/EVALUATION.md

@@ -0,0 +1,65 @@
+# Fold 04 Holdout 5 Agent Evaluation
+
+## Scope
+
+This directory records one sandboxed sub-agent attempt on fold 04, where dataset index 5 was held out.
+
+- Dataset index: `5`
+- Sandbox: `.runs/fold-04-holdout-5`
+- Tracked artifact copy: `evaluations/fold-04-holdout-5`
+- Successful agent trace: `AGENT_ACTION_LOG.md`
+
+## Anti-Leakage Audit
+
+The successful trace was scanned for root gold data, root processed data, other fold sandboxes, `dataset_05.py`, and explicit `violation` markers. Result: `0` matches.
+
+## Output Equivalence Against Gold
+
+| Metric | Agent | Gold |
+| --- | ---: | ---: |
+| Rows | 515,825 | 1,089,621 |
+| Columns | 9 | 9 |
+| Schema exact match | true | n/a |
+| Unique keyed rows recovered | 515,795 | 976,805 |
+| Missing gold keyed rows | 461,010 | n/a |
+| Extra agent keyed rows | 0 | n/a |
+| Row precision | 1.000 | n/a |
+| Gold recall | 0.528 | n/a |
+
+Shared-row value matches after key merge:
+
+| Column | Matched rows |
+| --- | ---: |
+| `interval_min` | 59007/516005 |
+| `volumetric_water_content_m3_m3` | 515933/516005 |
+| `gravimetric_water_content_gH2O_gs` | 516005/516005 |
+| `water_potential_kPa` | 516005/516005 |
+| `is_timeseries` | 516005/516005 |
+
+Interpretation: this run did not match the gold row set. The agent retained control-treatment SWC rows only, producing 515,825 rows versus 1,089,621 gold rows. Because `depth_m` is missing and timestamps/sites repeat, the key merge has duplicate expansions; row-set metrics above use NaN-safe unique keys.
+
+## JSON Mapping Match
+
+- Top-level exact matches: `5/7`
+- Gold mapping categories present: `8/8`
+- Extra agent categories: `gravimetric_water_content`
+- Missing gold categories: `none`
+- Field-level exact matches: `24/40`
+
+Field exact matches by name:
+
+| Field | Exact matches |
+| --- | ---: |
+| `destination_variable` | 8/8 |
+| `source_pattern` | 4/8 |
+| `source_files` | 4/8 |
+| `transformation` | 0/8 |
+| `unit_conversion` | 8/8 |
+
+## Python Match
+
+The generated script produced the required target schema and parsed successfully, but its curation choice diverged from gold by filtering to `Treatment == "ctl"` rows only. Gold includes a much larger row set for this dataset.
+
+## Overall Assessment
+
+The fold 04 run passed the local no-gold trace audit and generated executable artifacts, but output equivalence was poor because the row-inclusion decision diverged from gold.

evaluations/fold-04-holdout-5/MANIFEST.json

@@ -0,0 +1,35 @@
+{
+  "name": "fold-04-holdout-5",
+  "holdout_indices": [
+    5
+  ],
+  "holdout_identifiers": [
+    "ess-dive-8ac2940c708a515-20230504T210140482233"
+  ],
+  "exemplar_indices": [
+    1,
+    2,
+    3,
+    4,
+    6,
+    7,
+    8,
+    9,
+    10,
+    15,
+    16,
+    17,
+    18,
+    23,
+    24,
+    25,
+    26,
+    27
+  ],
+  "n_exemplars": 27,
+  "sources": {
+    "package": "data/gold/expert_code/harmonize_sm",
+    "mapping": "data/gold/sm_data_harmonization_mapping.json",
+    "skills": "skills"
+  }
+}