[codex] add fold08 agent evaluation bundle

evaluations/fold-08-holdout-10/AGENT_ACTION_LOG.md

@@ -0,0 +1,24 @@
+2026-06-30T16:01:26-07:00 | action=command+write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-08-holdout-10/AGENT_ACTION_LOG.md and agent_outputs | reason=initialize mandatory trace log and output directory before any further work
+2026-06-30T16:01:41-07:00 | action=command+file_read | subject=.runs/fold-08-holdout-10/AGENT_INSTRUCTIONS.md | reason=read fold-local benchmark instructions
+2026-06-30T16:01:41-07:00 | action=file_read | subject=.runs/fold-08-holdout-10/MANIFEST.json | reason=read fold-local manifest for held-out dataset and allowed files
+2026-06-30T16:01:41-07:00 | action=file_read | subject=.runs/fold-08-holdout-10/skills/essdive_sm_curator/SKILL.md | reason=read required fold-local curator skill
+2026-06-30T16:01:41-07:00 | action=file_read | subject=.runs/fold-08-holdout-10/skills/essdive_sm_harmonizer/SKILL.md | reason=read required fold-local harmonizer skill
+2026-06-30T16:01:52-07:00 | action=command+directory_listing | subject=.runs/fold-08-holdout-10 sandbox top-level files | reason=identify fold-local allowed exemplars and metadata paths
+2026-06-30T16:01:52-07:00 | action=directory_listing | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-01092fc392bc46d-20240819T143818677 | reason=inspect raw held-out package files allowed by benchmark
+2026-06-30T16:02:02-07:00 | action=command+directory_listing | subject=.runs/fold-08-holdout-10/data | reason=locate fold-local exemplar mapping and expert code directories
+2026-06-30T16:02:54-07:00 | action=command+file_read | subject=fold-local mapping JSON and held-out Soil_water_content_Fig4e.csv | reason=inspect exemplar mapping summaries and raw held-out columns/sample rows
+2026-06-30T16:03:14-07:00 | action=command+external_metadata_lookup | subject=ESS-DIVE API package metadata for ess-dive-01092fc392bc46d-20240819T143818677 | reason=retrieve DOI and package-level metadata per fold-local curator skill
+2026-06-30T16:03:31-07:00 | action=command+external_metadata_lookup | subject=ESS-DIVE API package metadata for ess-dive-01092fc392bc46d-20240819T143818677 | reason=retry metadata lookup after sandbox DNS failure to retrieve DOI
+2026-06-30T16:03:55-07:00 | action=command+file_read | subject=fold-local mapping entries 5, 15, 23, 25, 26 and exemplar code dataset_05/15/23/25/26.py | reason=inspect structurally related VWC examples for mapping and code conventions
+2026-06-30T16:04:11-07:00 | action=command+file_read | subject=fold-local expert_code/harmonize_sm/common.py | reason=inspect schema helper behavior and column order convention
+2026-06-30T16:04:25-07:00 | action=command+directory_listing | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive_b924ba040ecefee_20250707T154402200 | reason=locate raw reference location metadata named in fold-local exemplar mapping index 0
+2026-06-30T16:04:38-07:00 | action=command+file_read+search | subject=raw reference location registry and held-out Soil_water_content_Fig4e.csv | reason=find PLM site coordinates and quantify held-out sampling interval
+2026-06-30T16:06:17-07:00 | action=command+write | subject=.runs/fold-08-holdout-10/agent_outputs/{curator_bundle.json,mapping.json,mapping_notes.md,harmonize_heldout.py} | reason=write benchmark held-out curation, mapping, notes, and harmonization script
+2026-06-30T16:06:17.958597-07:00 | action=file_write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-08-holdout-10/agent_outputs/curator_bundle.json | reason=write curator_bundle.json benchmark artifact
+2026-06-30T16:06:17.958904-07:00 | action=file_write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-08-holdout-10/agent_outputs/mapping.json | reason=write mapping.json benchmark artifact
+2026-06-30T16:06:17.959075-07:00 | action=file_write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-08-holdout-10/agent_outputs/mapping_notes.md | reason=write mapping_notes.md benchmark artifact
+2026-06-30T16:06:17.959234-07:00 | action=file_write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-08-holdout-10/agent_outputs/harmonize_heldout.py | reason=write harmonize_heldout.py benchmark artifact
+2026-06-30T16:06:27-07:00 | action=command | subject=.runs/fold-08-holdout-10/agent_outputs/harmonize_heldout.py | reason=run held-out harmonization script to generate heldout_harmonized.csv
+2026-06-30T16:06:27.949299-07:00 | action=file_read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-01092fc392bc46d-20240819T143818677/Soil_water_content_Fig4e.csv | reason=read held-out raw soil water content payload
+2026-06-30T16:06:27.967549-07:00 | action=file_write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-08-holdout-10/agent_outputs/heldout_harmonized.csv | reason=write held-out harmonized CSV
+2026-06-30T16:06:49-07:00 | action=command+file_read+directory_listing+search | subject=agent_outputs, heldout_harmonized.csv, and AGENT_ACTION_LOG.md | reason=validate artifacts, row count, schema, and scan trace for forbidden path usage

evaluations/fold-08-holdout-10/AGENT_INSTRUCTIONS.md

@@ -0,0 +1,10 @@
+# Run environment — leave-one-cluster-out
+
+Harmonize the held-out dataset(s) below using ONLY:
+- the skills in `skills/`,
+- the exemplars in `data/processed/.../sm_data_harmonization_mapping.json` and the code patterns in `data/gold/expert_code/harmonize_sm/` (both have the held-out cluster removed),
+- the shared raw inputs under `~/ess-dive_wfsfa_soil_datasets/` and the cached metadata under `data/external/ess-dive_meta/`.
+
+Do NOT look up the held-out dataset's existing harmonized output, expert code, or mapping entry from any other location. The held-out datasets are:
+
+- `ess-dive-01092fc392bc46d-20240819T143818677`

evaluations/fold-08-holdout-10/EVALUATION.md

@@ -0,0 +1,52 @@
+# fold-08-holdout-10 Agent Evaluation
+
+## Scope
+
+This directory records one sandboxed sub-agent attempt for held-out dataset indices `[10]`.
+
+- Sandbox: `.runs/fold-08-holdout-10`
+- Tracked artifact copy: `evaluations/fold-08-holdout-10`
+- Successful agent trace: `AGENT_ACTION_LOG.md`
+
+## Anti-Leakage Audit
+
+The successful trace was scanned for root gold data, root processed data, other fold sandboxes, held-out expert module names, and explicit `violation` markers. Result: `0` matches.
+
+## Output Equivalence Against Gold
+
+| Metric | Agent | Gold |
+| --- | ---: | ---: |
+| Rows | 309 | 309 |
+| Target columns present | 9 / 9 | 9 / 9 |
+| Schema exact match | true | n/a |
+| Unique keyed rows recovered | 0 | 103 |
+| Missing gold keyed rows | 103 | n/a |
+| Extra agent keyed rows | 309 | n/a |
+| Row precision | 0.000 | n/a |
+| Gold recall | 0.000 | n/a |
+
+Shared-row value matches after key merge:
+
+| Column | Matched rows |
+| --- | ---: |
+| `interval_min` | 0/0 |
+| `volumetric_water_content_m3_m3` | 0/0 |
+| `gravimetric_water_content_gH2O_gs` | 0/0 |
+| `water_potential_kPa` | 0/0 |
+| `is_timeseries` | 0/0 |
+
+## JSON Mapping Match
+
+- Summary: `{"extra_agent_categories": ["gravimetric_water_content"], "field_exact_matches_by_name": {"destination_variable": "8/8", "source_files": "6/8", "source_pattern": "4/8", "transformation": "0/8", "unit_conversion": "6/8"}, "field_level_exact_matches": "24/40", "gold_mapping_categories_present": "8/8", "missing_gold_categories": [], "top_level_exact_matches": "7/7", "top_level_matches": {"archive_repository": true, "data_payload_files": true, "dataset_identifier": true, "doi": true, "index": true, "location_metadata_files": true, "sensor_metadata_files": true}}`
+
+| Field | Exact matches |
+| --- | ---: |
+| `destination_variable` | 8/8 |
+| `source_pattern` | 4/8 |
+| `source_files` | 6/8 |
+| `transformation` | 0/8 |
+| `unit_conversion` | 6/8 |
+
+## Overall Assessment
+
+See `evaluation_metrics.json` for machine-readable details. The run passed the targeted local no-gold path audit. Output equivalence should be interpreted with the key policy recorded in the metrics (`key_includes_dataset_index=False`).

evaluations/fold-08-holdout-10/MANIFEST.json

@@ -0,0 +1,35 @@
+{
+  "name": "fold-08-holdout-10",
+  "holdout_indices": [
+    10
+  ],
+  "holdout_identifiers": [
+    "ess-dive-01092fc392bc46d-20240819T143818677"
+  ],
+  "exemplar_indices": [
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    15,
+    16,
+    17,
+    18,
+    23,
+    24,
+    25,
+    26,
+    27
+  ],
+  "n_exemplars": 27,
+  "sources": {
+    "package": "data/gold/expert_code/harmonize_sm",
+    "mapping": "data/gold/sm_data_harmonization_mapping.json",
+    "skills": "skills"
+  }
+}

evaluations/fold-08-holdout-10/agent_outputs/curator_bundle.json

@@ -0,0 +1,67 @@
+{
+  "package_id": "ess-dive-01092fc392bc46d-20240819T143818677",
+  "doi": "doi:10.15485/2322567",
+  "curator_decision": "INCLUDE",
+  "exclusion_reason": null,
+  "package_title": "Hydrological control of chemical weathering and rock-carbon fluxes: East River, Colorado",
+  "data_payload_files": [
+    {
+      "filename": "Soil_water_content_Fig4e.csv",
+      "columns": [
+        "Date",
+        "PLM1 _vol_water_content",
+        "PLM2 _vol_water_content",
+        "PLM3 _vol_water_content"
+      ],
+      "row_count_estimate": 104,
+      "data_row_count_after_units_row": 103,
+      "file_size_mb": 0.0031,
+      "column_preview": "Date,PLM1 _vol_water_content,PLM2 _vol_water_content,PLM3 _vol_water_content\\nYYYY-MM-DD,m3/m3,m3/m3,m3/m3\\n2017-05-06,0.387,0.39,0.39",
+      "classification_reason": "CSV contains date-stamped direct volumetric water content observations in m3/m3 for PLM1, PLM2, and PLM3."
+    }
+  ],
+  "location_metadata_files": [],
+  "sensor_metadata_files": [],
+  "readme_content": null,
+  "location_resolution": {
+    "source": "varadharajan_lookup",
+    "qc_flag_recommendation": "g1",
+    "site_coordinates": [
+      {
+        "site_id": "ER-PLM1",
+        "latitude": 38.919771,
+        "longitude": -106.949273
+      },
+      {
+        "site_id": "ER-PLM2",
+        "latitude": 38.920157,
+        "longitude": -106.948714
+      },
+      {
+        "site_id": "ER-PLM3",
+        "latitude": 38.920784,
+        "longitude": -106.948367
+      }
+    ],
+    "note": "Coordinates are not present in the held-out payload; PLM IDs were resolved against the fold-local reference location dataset cited by mapping index 0."
+  },
+  "time_series_inference": {
+    "is_timeseries": true,
+    "interval_min": 20160.0,
+    "reasoning": "Each PLM site has repeated observations from 2017-05-06 to 2021-05-02; sampling is irregular but the median interval is 14 days."
+  },
+  "experimental_context": {
+    "manipulation_detected": false,
+    "manipulation_type": null,
+    "has_control_data": null,
+    "recommendation": "include_all"
+  },
+  "similar_dataset_reference": {
+    "index": 25,
+    "reason": "Similar code pattern: wide-format volumetric water content time series reshaped to long format."
+  },
+  "open_questions": [
+    "The CSV does not encode measurement depth; depth_m is populated as missing.",
+    "The package-level metadata gives watershed bounds, while site-specific PLM coordinates require the Varadharajan location registry lookup and should carry g1 location provenance if location output is generated."
+  ]
+}

evaluations/fold-08-holdout-10/agent_outputs/harmonize_heldout.py

@@ -0,0 +1,81 @@
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from pathlib import Path
+import re
+import numpy as np
+import pandas as pd
+
+FOLD_DIR = Path('/scratch/jmc/data-harmonization-eval/.runs/fold-08-holdout-10')
+LOG_PATH = FOLD_DIR / 'AGENT_ACTION_LOG.md'
+RAW_PATH = Path('/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-01092fc392bc46d-20240819T143818677/Soil_water_content_Fig4e.csv')
+OUT_PATH = FOLD_DIR / 'agent_outputs' / 'heldout_harmonized.csv'
+
+SCHEMA = [
+    'datetime_UTC',
+    'site_id',
+    'depth_m',
+    'replicate',
+    'is_timeseries',
+    'interval_min',
+    'volumetric_water_content_m3_m3',
+    'gravimetric_water_content_gH2O_gs',
+    'water_potential_kPa',
+]
+
+
+def trace(action: str, subject: str, reason: str) -> None:
+    ts = datetime.now(timezone.utc).astimezone().isoformat()
+    with LOG_PATH.open('a', encoding='utf-8') as f:
+        f.write(f'{ts} | action={action} | subject={subject} | reason={reason}\n')
+
+
+def parse_local_to_utc(series: pd.Series, fmt: str, tz: str) -> pd.Series:
+    dt = pd.to_datetime(series, format=fmt, errors='coerce')
+    dt = dt.dt.tz_localize(tz, ambiguous='NaT', nonexistent='shift_forward')
+    return dt.dt.tz_convert('UTC')
+
+
+def ensure_harmonized_cols(df: pd.DataFrame) -> pd.DataFrame:
+    for col in SCHEMA:
+        if col not in df.columns:
+            df[col] = np.nan
+    return df[SCHEMA]
+
+
+def harmonize() -> pd.DataFrame:
+    trace('file_read', str(RAW_PATH), 'read held-out raw soil water content payload')
+    raw = pd.read_csv(RAW_PATH)
+    data = raw.iloc[1:].copy()
+
+    value_cols = [c for c in data.columns if re.search(r'vol_water_content', c, flags=re.IGNORECASE)]
+    long = data.melt(
+        id_vars=['Date'],
+        value_vars=value_cols,
+        var_name='source_variable',
+        value_name='volumetric_water_content_m3_m3',
+    )
+
+    long['datetime_UTC'] = parse_local_to_utc(long['Date'], '%Y-%m-%d', 'America/Denver')
+    long['site_id'] = 'ER-' + long['source_variable'].str.extract(r'(PLM\d+)')[0]
+    long['volumetric_water_content_m3_m3'] = pd.to_numeric(long['volumetric_water_content_m3_m3'], errors='coerce')
+    long['depth_m'] = np.nan
+    long['replicate'] = 1
+    long['is_timeseries'] = True
+    long['gravimetric_water_content_gH2O_gs'] = np.nan
+    long['water_potential_kPa'] = np.nan
+
+    long = long.sort_values(['site_id', 'datetime_UTC']).reset_index(drop=True)
+    long['interval_min'] = long.groupby('site_id')['datetime_UTC'].diff().dt.total_seconds() / 60.0
+
+    out = ensure_harmonized_cols(long)
+    out = out[out['datetime_UTC'].notna() & out['site_id'].notna() & out['volumetric_water_content_m3_m3'].notna()].copy()
+    out['datetime_UTC'] = out['datetime_UTC'].dt.strftime('%Y-%m-%dT%H:%M:%SZ')
+    return out
+
+
+if __name__ == '__main__':
+    df = harmonize()
+    trace('file_write', str(OUT_PATH), 'write held-out harmonized CSV')
+    df.to_csv(OUT_PATH, index=False)
+    print(f'wrote {OUT_PATH} rows={len(df)} columns={list(df.columns)}')