[codex] add fold07 agent evaluation bundle

[jmchandonia]

Summary

• add tracked fold-07 holdout-9 sub-agent artifacts, including action trace and generated outputs
• add an evaluation report and machine-readable metrics against held-out gold

Evaluation Highlights

• targeted forbidden-path scan found zero matches
• agent produced 664 rows vs 672 gold rows
• no keyed rows matched gold under the evaluator key policy
• generated JSON matched all 7/7 top-level fields and covered all 8/8 gold mapping categories

Validation

• python -m py_compile evaluations/fold-07-holdout-9/agent_outputs/harmonize_heldout.py
• targeted rg forbidden-path scan over evaluations/fold-07-holdout-9/AGENT_ACTION_LOG.md

[claude]

Claude finished @jmchandonia's task —— View job

---

PR Review: fold07 agent evaluation bundle

• Gather context
• Read changed files
• Review code and artifacts
• Post review feedback

---

Summary

This PR adds a complete agent evaluation bundle for fold-07 holdout-9. The artifacts are well-structured and the action log demonstrates proper sandboxing discipline. Below are detailed observations.

---

harmonize_heldout.py — Code Quality

Hardcoded absolute path (significant issue)

RAW_BASE = Path("/h/jmc/ess-dive_wfsfa_soil_datasets")  # line 8

The raw data path is hardcoded to a specific user's home directory. This makes the script non-reproducible on any other machine. Consider accepting the path via an environment variable or CLI argument (e.g., argparse), with the current value as a documented default.

Floating-point precision noise in output

0.055999999999999994  # heldout_harmonized.csv line 2

VWC values like 5.6 / 100 produce float representation artifacts. The gold dataset likely stores clean rounded values. This is likely a key contributor to the 0 key intersection (though the key policy issue below is the primary cause). Rounding to a sensible precision (e.g., round(x, 6)) before writing would improve output cleanliness.

Silent NaN pass-through for VWC values

At line 52–55, after filtering -9999 sentinels, only notna() NaN rows are dropped. However, the errors="coerce" in pd.to_numeric (line 49–51) silently converts any other non-numeric string to NaN before the drop. This is acceptable behavior, but a warning/log for coerced-to-NaN values would improve debuggability.

No __all__ or module-level docstring

Minor: the script has no top-level docstring explaining its purpose or expected inputs/outputs, which would help future evaluators understand the artifact.

---

Key-Match Failure — Root Cause Analysis

The evaluation reports key_intersection = 0 (0 out of 672 gold rows matched). The metrics record key_includes_dataset_index = false. The most likely causes:

1. Timezone rounding: Collection Date is a date-only field parsed as local midnight America/Denver → UTC, producing T06:00:00Z (MDT offset). If the gold uses a different UTC offset (e.g., MST = T07:00:00Z) or stores date-only values, all 664 rows would fail to key-match on datetime_UTC.

2. site_id format mismatch: The agent uses SampleSiteCode as-is (e.g., "001-ER18"). If the gold normalizes these differently (e.g., strips leading zeros, appends the dataset identifier, or uses a different column), every key would miss.

These would be worth investigating — the row count mismatch (664 vs. 672) suggests 8 rows were additionally filtered differently by the gold harmonizer (possibly different sentinel handling or date parsing).

---

mapping.json — Correctness

transformation field: 0/8 exact matches

The evaluation notes "transformation": "0/8" exact matches. The agent's transformation descriptions are clear and correct in substance, but differ in phrasing from the gold. This is expected for agent-generated text, but worth noting as a systematic gap if transformation strings are compared literally.

gravimetric_water_content as extra category

The agent included a mapping entry for gravimetric_water_content (marked as null/missing), which the gold apparently omits. The agent correctly identifies no GWC is present in the source — the gold may simply not include mapping entries for absent variables. This is a minor schema policy difference.

latitude/longitude mappings are correct but absent from output CSV

The mapping.json documents coordinate reprojection (UTM EPSG:32613 → WGS84), but harmonize_heldout.py does not emit latitude or longitude columns. This is consistent with HARMONIZED_COLUMNS (line 12–22) not including them, which is correct if the target schema omits coordinate columns. The mapping entry is informational only.

---

AGENT_ACTION_LOG.md — Process Integrity

The action log is thorough and shows the agent correctly:

• read only fold-local and allowed raw paths
• avoided the held-out gold path
• ran a self-audit forbidden-path scan at the end (line 38)

One minor note: the log shows a /tmp cleanup action (line 36–37) and a self-correction log entry (line 37–38). This transparency is good practice.

---

EVALUATION.md / evaluation_metrics.json — Completeness

The evaluation report is well-formatted and machine-readable metrics are consistent with the markdown table. The key_includes_dataset_index = false policy caveat is correctly documented. No issues found.

---

Security / Path Concerns

No forbidden-path leakage was detected. The hardcoded /h/jmc/ path is a reproducibility concern but not a security concern for this evaluation context.

---

Recommendations (Priority Order)

1. Investigate the key-match failure — determine if the UTC offset or site_id format differ between agent and gold, and document findings in mapping_notes.md or EVALUATION.md.
2. Parameterize RAW_BASE — accept via env var or CLI arg to make the script portable.
3. Round VWC output values — e.g., out["volumetric_water_content_m3_m3"] = out["..."].round(8) before writing CSV.
4. Add a script docstring — briefly document expected environment and outputs.

The bundle is otherwise complete and well-structured. The 0 key-match result is notable and warrants follow-up, but does not block merging if the evaluation framework treats this fold as a captured baseline.

---