[codex] add fold02 agent evaluation bundle

evaluations/fold-02-holdout-15-26/AGENT_ACTION_LOG.md

@@ -0,0 +1,63 @@
+- 2026-06-30T15:54:23-07:00 | action=command | subject=mkdir -p .runs/fold-02-holdout-15-26/agent_outputs | reason=initialize action log and output directory
+- 2026-06-30T15:54:39-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/AGENT_INSTRUCTIONS.md | reason=read mandatory fold-local benchmark instructions
+- 2026-06-30T15:54:39-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/MANIFEST.json | reason=read fold manifest for held-out datasets and allowed exemplars
+- 2026-06-30T15:54:39-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/skills/essdive_sm_curator/SKILL.md | reason=read fold-local curator skill
+- 2026-06-30T15:54:39-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/skills/essdive_sm_harmonizer/SKILL.md | reason=read fold-local harmonizer skill
+- 2026-06-30T15:54:50-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/skills/essdive_sm_curator/SKILL.md lines 261-520 | reason=complete fold-local curator skill read
+- 2026-06-30T15:54:50-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/skills/essdive_sm_harmonizer/SKILL.md lines 261-620 | reason=complete fold-local harmonizer skill read
+- 2026-06-30T15:55:00-07:00 | action=command | subject=wc -l fold-local instruction and skill files | reason=verify complete local skill file lengths before proceeding
+- 2026-06-30T15:55:09-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/skills/essdive_sm_curator/SKILL.md lines 521-556 | reason=finish complete fold-local curator skill read
+- 2026-06-30T15:55:16-07:00 | action=directory_listing | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26 | reason=inventory fold-local sandbox files and allowed exemplars
+- 2026-06-30T15:55:23-07:00 | action=directory_listing | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/data | reason=locate fold-local exemplar mapping, code, and metadata only
+- 2026-06-30T15:55:40-07:00 | action=command | subject=python summarize fold-local sm_data_harmonization_mapping.json | reason=inspect allowed exemplar mapping schema and dataset patterns
+- 2026-06-30T15:55:40-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/data/processed/ess-dive_wfsfa_soil_datasets/sm_data_harmonization_mapping.json | reason=read fold-local exemplar mapping only
+- 2026-06-30T15:56:34-07:00 | action=directory_listing | subject=/h/jmc/ess-dive_wfsfa_soil_datasets held-out package directories | reason=locate allowed raw held-out files for package classification
+- 2026-06-30T15:56:54-07:00 | action=command | subject=python inspect two held-out raw CSV files | reason=summarize columns, row counts, and sample values for harmonization
+- 2026-06-30T15:56:54-07:00 | action=read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-987726ef1235abc-20230504T210342929747/TDR_survey_ER_20190701.csv | reason=inspect held-out dataset 15 raw payload
+- 2026-06-30T15:56:54-07:00 | action=read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-f782da867133296-20230504T211008637996/ER18_soil_physical.csv | reason=inspect held-out dataset 26 raw payload
+- 2026-06-30T15:57:13-07:00 | action=command | subject=sed read fold-local common.py and exemplar dataset scripts | reason=learn allowed code patterns and schema helper behavior
+- 2026-06-30T15:57:13-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/data/gold/expert_code/harmonize_sm/common.py | reason=inspect fold-local shared helper functions
+- 2026-06-30T15:57:13-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/data/gold/expert_code/harmonize_sm/dataset_09.py | reason=inspect similar discrete TDR/VWC exemplar
+- 2026-06-30T15:57:13-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/data/gold/expert_code/harmonize_sm/dataset_18.py | reason=inspect similar discrete lab water-content exemplar
+- 2026-06-30T15:57:31-07:00 | action=command | subject=curl ESS-DIVE API metadata for held-out package IDs | reason=retrieve DOI/title metadata for mapping entries without reading forbidden local gold outputs
+- 2026-06-30T15:57:42-07:00 | action=command | subject=curl ESS-DIVE API metadata for held-out package IDs with network access | reason=sandbox DNS blocked public metadata lookup needed for DOI/title fields
+- 2026-06-30T15:58:04-07:00 | action=command | subject=python parse downloaded ESS-DIVE metadata JSON from /tmp | reason=extract DOI, descriptions, distributions, spatial coverage for held-out curation
+- 2026-06-30T15:58:04-07:00 | action=read | subject=/tmp/ess_meta_15.json | reason=read public ESS-DIVE metadata response for held-out dataset 15
+- 2026-06-30T15:58:04-07:00 | action=read | subject=/tmp/ess_meta_26.json | reason=read public ESS-DIVE metadata response for held-out dataset 26
+- 2026-06-30T16:00:46-07:00 | action=command | subject=python generate harmonize_heldout.py | reason=create benchmark harmonization script without apply_patch
+- 2026-06-30T16:00:46-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=write final harmonization script under agent_outputs
+- 2026-06-30T16:02:54-07:00 | action=command | subject=python regenerate harmonize_heldout.py | reason=fix script generation quoting after failed attempt
+- 2026-06-30T16:02:54-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=write final harmonization script under agent_outputs
+- 2026-06-30T16:03:02-07:00 | action=command | subject=python agent_outputs/harmonize_heldout.py | reason=run held-out harmonization script to generate final outputs
+- 2026-06-30T16:03:02-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=execute generated harmonization script
+- 2026-06-30T16:03:34-07:00 | action=command | subject=python edit harmonize_heldout.py remove pyproj dependency | reason=replace unavailable pyproj dependency with local UTM conversion
+- 2026-06-30T16:03:34-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=read generated script for dependency fix
+- 2026-06-30T16:03:34-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=write generated script with local UTM conversion
+- 2026-06-30T16:03:55-07:00 | action=command | subject=python agent_outputs/harmonize_heldout.py | reason=rerun held-out harmonization after dependency fix
+- 2026-06-30T16:03:55-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=execute generated harmonization script after dependency fix
+- 2026-06-30T16:03:55.620789-07:00 | action=command | subject=harmonize_heldout.py main | reason=generate all held-out benchmark outputs
+- 2026-06-30T16:03:55.621017-07:00 | action=read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-987726ef1235abc-20230504T210342929747/TDR_survey_ER_20190701.csv | reason=read held-out dataset 15 raw CSV for harmonization
+- 2026-06-30T16:04:08-07:00 | action=command | subject=python edit harmonize_heldout.py qc_flag dtype | reason=fix NumPy string/NaN dtype promotion issue
+- 2026-06-30T16:04:08-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=read generated script for qc_flag fix
+- 2026-06-30T16:04:08-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=write generated script with qc_flag dtype fix
+- 2026-06-30T16:04:17-07:00 | action=command | subject=python agent_outputs/harmonize_heldout.py | reason=rerun held-out harmonization after qc_flag fix
+- 2026-06-30T16:04:17-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/harmonize_heldout.py | reason=execute generated harmonization script after qc_flag fix
+- 2026-06-30T16:04:17.543179-07:00 | action=command | subject=harmonize_heldout.py main | reason=generate all held-out benchmark outputs
+- 2026-06-30T16:04:17.543412-07:00 | action=read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-987726ef1235abc-20230504T210342929747/TDR_survey_ER_20190701.csv | reason=read held-out dataset 15 raw CSV for harmonization
+- 2026-06-30T16:04:17.573359-07:00 | action=read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-f782da867133296-20230504T211008637996/ER18_soil_physical.csv | reason=read held-out dataset 26 raw CSV for harmonization
+- 2026-06-30T16:04:17.592195-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/heldout_harmonized_dataset_15.csv | reason=write optional per-dataset harmonized rows for held-out index 15
+- 2026-06-30T16:04:17.594437-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/heldout_harmonized_dataset_26.csv | reason=write optional per-dataset harmonized rows for held-out index 26
+- 2026-06-30T16:04:17.597450-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/heldout_harmonized.csv | reason=write concatenated held-out target-schema harmonized rows
+- 2026-06-30T16:04:17.601624-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/mapping.json | reason=write held-out mapping entries
+- 2026-06-30T16:04:17.603208-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/curator_bundle.json | reason=write curator decisions and structured inputs
+- 2026-06-30T16:04:17.603503-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/mapping_notes.md | reason=write harmonization decisions and unresolved assumptions
+- 2026-06-30T16:04:40-07:00 | action=directory_listing | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs | reason=verify final output files are present under agent_outputs
+- 2026-06-30T16:04:40-07:00 | action=command | subject=python validate held-out output files | reason=check parseability, row counts, and target schema
+- 2026-06-30T16:04:40-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/heldout_harmonized.csv | reason=validate combined held-out harmonized CSV
+- 2026-06-30T16:04:40-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/heldout_harmonized_dataset_15.csv | reason=validate optional dataset 15 CSV
+- 2026-06-30T16:04:40-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/heldout_harmonized_dataset_26.csv | reason=validate optional dataset 26 CSV
+- 2026-06-30T16:04:40-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/mapping.json | reason=validate mapping JSON
+- 2026-06-30T16:04:40-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/curator_bundle.json | reason=validate curator bundle JSON
+- 2026-06-30T16:04:40-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/agent_outputs/mapping_notes.md | reason=validate mapping notes presence
+- 2026-06-30T16:04:56-07:00 | action=command | subject=python scan AGENT_ACTION_LOG.md for forbidden path patterns | reason=perform mandatory anti-leakage log scan
+- 2026-06-30T16:04:56-07:00 | action=read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-02-holdout-15-26/AGENT_ACTION_LOG.md | reason=scan own action log for forbidden path patterns

evaluations/fold-02-holdout-15-26/AGENT_INSTRUCTIONS.md

@@ -0,0 +1,11 @@
+# Run environment — leave-one-cluster-out
+
+Harmonize the held-out dataset(s) below using ONLY:
+- the skills in `skills/`,
+- the exemplars in `data/processed/.../sm_data_harmonization_mapping.json` and the code patterns in `data/gold/expert_code/harmonize_sm/` (both have the held-out cluster removed),
+- the shared raw inputs under `~/ess-dive_wfsfa_soil_datasets/` and the cached metadata under `data/external/ess-dive_meta/`.
+
+Do NOT look up the held-out dataset's existing harmonized output, expert code, or mapping entry from any other location. The held-out datasets are:
+
+- `ess-dive-987726ef1235abc-20230504T210342929747`
+- `ess-dive-f782da867133296-20230504T211008637996`

evaluations/fold-02-holdout-15-26/EVALUATION.md

@@ -0,0 +1,45 @@
+# fold-02-holdout-15-26 Agent Evaluation
+
+## Scope
+
+This directory records one sandboxed sub-agent attempt for held-out dataset indices `[15, 26]`.
+
+- Sandbox: `.runs/fold-02-holdout-15-26`
+- Tracked artifact copy: `evaluations/fold-02-holdout-15-26`
+- Successful agent trace: `AGENT_ACTION_LOG.md`
+
+## Anti-Leakage Audit
+
+The successful trace was scanned for root gold data, root processed data, other fold sandboxes, held-out expert module names, and explicit `violation` markers. Result: `0` matches.
+
+## Output Equivalence Against Gold
+
+| Metric | Agent | Gold |
+| --- | ---: | ---: |
+| Rows | 711 | 586 |
+| Target columns present | 9 / 9 | 9 / 9 |
+| Schema exact match | true | n/a |
+| Unique keyed rows recovered | 0 | 584 |
+| Missing gold keyed rows | 584 | n/a |
+| Extra agent keyed rows | 711 | n/a |
+| Row precision | 0.000 | n/a |
+| Gold recall | 0.000 | n/a |
+
+Shared-row value matches after key merge:
+
+| Column | Matched rows |
+| --- | ---: |
+| `interval_min` | 0/0 |
+| `volumetric_water_content_m3_m3` | 0/0 |
+| `gravimetric_water_content_gH2O_gs` | 0/0 |
+| `water_potential_kPa` | 0/0 |
+| `is_timeseries` | 0/0 |
+
+## JSON Mapping Match
+
+- Summary: `{"heldout_indices": [15, 26], "note": "cluster mapping literal comparison not computed"}`
+
+
+## Overall Assessment
+
+See `evaluation_metrics.json` for machine-readable details. The run passed the targeted local no-gold path audit. Output equivalence should be interpreted with the key policy recorded in the metrics (`key_includes_dataset_index=True`).

evaluations/fold-02-holdout-15-26/MANIFEST.json

@@ -0,0 +1,36 @@
+{
+  "name": "fold-02-holdout-15-26",
+  "holdout_indices": [
+    15,
+    26
+  ],
+  "holdout_identifiers": [
+    "ess-dive-987726ef1235abc-20230504T210342929747",
+    "ess-dive-f782da867133296-20230504T211008637996"
+  ],
+  "exemplar_indices": [
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    16,
+    17,
+    18,
+    23,
+    24,
+    25,
+    27
+  ],
+  "n_exemplars": 26,
+  "sources": {
+    "package": "data/gold/expert_code/harmonize_sm",
+    "mapping": "data/gold/sm_data_harmonization_mapping.json",
+    "skills": "skills"
+  }
+}