[codex] add fold13 agent evaluation bundle

[jmchandonia]

Summary

• add tracked fold-13 holdout-25 sub-agent artifacts, including action trace and generated outputs
• add an evaluation report and machine-readable metrics against held-out gold

Evaluation Highlights

• targeted forbidden-path scan found zero matches
• agent produced 471,671 rows vs 471,893 gold rows
• no keyed rows matched gold under evaluator key policy
• generated JSON matched all 7/7 top-level fields and covered all 8/8 gold mapping categories

Validation

• python -m py_compile evaluations/fold-13-holdout-25/agent_outputs/harmonize_heldout.py
• targeted rg forbidden-path scan over evaluations/fold-13-holdout-25/AGENT_ACTION_LOG.md

[claude]

Claude finished @jmchandonia's task —— View job

---

PR Review: fold13 Agent Evaluation Bundle

Tasks

• Initialize review
• Read repository guidelines (CLAUDE.md — not present)
• Review Python script (harmonize_heldout.py)
• Review JSON files (mapping.json, curator_bundle.json, evaluation_metrics.json)
• Review evaluation documentation (EVALUATION.md, AGENT_ACTION_LOG.md)
• Provide comprehensive feedback

---

Code Quality — harmonize_heldout.py

Strengths:

• Clean functional structure; read_source, local_timestamp_to_utc, harmonize_one, main are well-separated.
• Correct use of pd.to_numeric(errors='coerce') and pd.to_datetime(errors='coerce') for robust parsing.
• VWC.between(0, 1, inclusive="both") is the right physical bound check.
• tz_localize with ambiguous="NaT", nonexistent="NaT" properly handles DST edge cases in a fixed-offset timezone (though Etc/GMT+7 is already DST-free).

Issues and suggestions:

1. Hard-coded absolute paths (RAW_DIR, OUT_PATH at lines 8–9) make the script non-portable and will fail on any machine other than the author's. Consider accepting paths via argparse or environment variables, or at minimum documenting the expected layout in a comment.

2. interval_min key mismatch likely explains 0 row recall. The script computes interval_min per (site_id, depth_m) group (line 80) but sorts the entire frame before grouping (line 78). Because the frame is sorted by ["site_id", "depth_m", "datetime_UTC"], the .diff() will produce NaN at the first row of each new group — which is correct — but the final output column order is determined after the sort, so the key used for joining with gold must include (datetime_UTC, site_id, depth_m, replicate). If the gold uses dataset_index as part of the composite key (key_includes_dataset_index=False noted in metrics), the 0-intersection is expected and explained. The evaluation already surfaces this; no bug per se, but worth noting in documentation.

3. replicate is always 1 (integer, line 73) — if gold stores it as a string "1", the join will fail silently. The evaluation's key policy should clarify the expected type.

4. from __future__ import annotations is unused (line 1) — the script uses no PEP 563 annotations.

5. No error handling for missing files. If RAW_DIR / filename does not exist, read_csv raises an unhelpful FileNotFoundError. A short existence check or a try/except with a descriptive message would improve debuggability.

6. COLUMNS list (line 11–18) duplicates knowledge from VWC_COLUMNS (line 30) and DEPTH_M (line 31). A single source of truth (e.g., building DEPTH_M from COLUMNS) would reduce drift risk if column names change.

7. out.to_csv(OUT_PATH, index=False) (line 93) writes NaN as empty strings by default in pandas, which is fine for this use case but should be stated explicitly (na_rep='') if downstream consumers depend on it.

---

Mapping JSON — mapping.json

• Extra gravimetric_water_content category is included despite it not being in gold. The evaluation flags this as extra_agent_categories: ["gravimetric_water_content"]. Since gravimetric_water_content_gH2O_gs is a target column (set to NaN), including a mapping entry for it is defensible, but reviewers should confirm whether gold intentionally omits no-data mapping entries.
• latitude and longitude categories are mapped but those columns are absent from TARGET_COLUMNS in the script. This is an inconsistency: the mapping document records spatial metadata that the harmonized CSV does not emit. Either the script should add those columns to output, or the mapping should note them as curator-only metadata.
• transformation: "0/8" exact matches suggests the free-text transformation descriptions diverge significantly from gold. This is expected for a generative agent but worth noting as a qualitative weakness.

---

Evaluation Metrics — evaluation_metrics.json

• row_precision: 0.0 and gold_recall: 0.0 with key_intersection: 0 despite 471k agent rows vs 471k gold rows is the most notable finding. The PR description correctly attributes this to key_includes_dataset_index=False. However, the metrics file does not record what keys were used for joining, making it hard to audit independently. Recommend adding a key_columns field to evaluation_metrics.json.
• unique_agent_keys: 467624 vs unique_gold_keys: 467529 — the agent produces more unique keys, implying the agent's timestamp/depth combinations are distinct from gold's. This likely stems from a timezone offset difference (agent uses Etc/GMT+7; gold may use a different offset or daylight saving treatment).

---

Documentation

• AGENT_ACTION_LOG.md is thorough and well-structured. The action log correctly records API lookup attempts (including DNS failure) and validates outputs before finishing.
• EVALUATION.md clearly presents the key divergence between rows and zeroed key metrics. The note pointing readers to evaluation_metrics.json for the key policy is good, but the EVALUATION.md itself should state what the key columns are (currently only implied).
• mapping_notes.md is concise and captures important assumptions (headerless CSV, invalid Feb 29 rows, timezone assumption, coordinates).

---

Security

No security concerns. The script reads local CSV files and writes a local CSV; no network calls, no shell injection, no secret exposure.

---

Summary

The bundle is well-organized and the Python script is readable and functionally correct for the intended task. The primary concern is the 0% row recall against gold, which is likely a timezone offset mismatch causing every (datetime_UTC, site_id, depth_m, replicate) key to diverge. This should be investigated before concluding the agent's approach is incorrect — the evaluation metrics should record the join key columns to make this auditable. The hard-coded absolute paths in the script are a minor but real portability issue.