[codex] add fold05 agent evaluation bundle

[jmchandonia]

Summary

• add tracked fold-05 holdout-7 sub-agent artifacts, including action trace and generated outputs
• add an evaluation report comparing the agent output, JSON mapping, and Python logic against the held-out gold standard
• add machine-readable evaluation metrics for the run

Evaluation Highlights

• targeted forbidden-path scan found zero matches
• agent recovered all 56,861 gold keyed rows with no extras or omissions
• all non-interval target values matched gold exactly
• interval_min matched 56,855/56,861 rows; the six mismatches come from the agent computing intervals within depth streams while gold computes raw sequential diffs across depth transitions
• generated JSON matched all 7/7 top-level gold fields and covered all 8/8 gold mapping categories, with one extra gravimetric_water_content category

Validation

• python -m py_compile evaluations/fold-05-holdout-7/agent_outputs/harmonize_heldout.py
• targeted rg forbidden-path scan over evaluations/fold-05-holdout-7/AGENT_ACTION_LOG.md
• UV_CACHE_DIR=/scratch/jmc/data-harmonization-eval/.uv-cache uv run pytest

[claude]

Claude finished @jmchandonia's task —— View job

---

PR Review: fold-05-holdout-7 Agent Evaluation Bundle

Tasks

• Gather context and understand the request
• Read CLAUDE.md for repo-specific guidelines (no CLAUDE.md present)
• Review changed files
• Post comprehensive review feedback

---

Summary

This PR adds the evaluation artifacts for a sub-agent run on fold-05, holdout dataset 7. The overall result is strong — perfect row/column recovery, exact match on all non-interval columns, and a clean anti-leakage trace. Below is a detailed review of each artifact type.

---

harmonize_heldout.py — Code Quality

Positive observations:

• Clean, readable structure with proper type annotations and from __future__ import annotations
• Good use of pd.to_numeric(..., errors='coerce') to safely handle malformed values
• Appropriate NaN filtering before groupby interval computation
• ensure_harmonized_cols provides a safe contract for output schema

Issues and suggestions:

1. Hardcoded absolute paths (harmonize_heldout.py:9-14): RUN_DIR, RAW_DIR, and OUT_DIR are all hardcoded to /scratch/jmc/... and /h/jmc/.... This means the script cannot be run portably by another evaluator on a different machine. If this script is intended to be a reproducible artifact, consider parameterizing paths via argparse or environment variables. This is the most significant practical issue in the file.

2. log() side-effects on import (harmonize_heldout.py:30-33): The log() function writes to LOG_PATH inside the sandbox RUN_DIR. If someone tries to run this on another machine, it will fail with a FileNotFoundError since /scratch/jmc/.../AGENT_ACTION_LOG.md doesn't exist. The script would benefit from making logging optional or gracefully skipping if the log path doesn't exist.

3. interval_min grouping vs. raw order (harmonize_heldout.py:67-72): The evaluation correctly identifies this as a scientifically defensible choice, but the 6-row mismatch is a known behavioral difference. The inline comment in the script could document why per-group groupby was chosen over raw sequential diff, to make the trade-off explicit.

4. Negative interval guard (harmonize_heldout.py:73): x.loc[x['interval_min'] < 0, 'interval_min'] = np.nan — within a per-group .diff(), negative intervals should not occur unless timestamps are non-monotonic within a group. Consider adding an assertion or warning here to catch potential input data quality issues rather than silently setting to NaN.

5. No error handling around file read (harmonize_heldout.py:51): pd.read_csv(PAYLOAD) will raise a raw FileNotFoundError if the path doesn't exist. A clear user-facing error message would improve debuggability.

---

mapping.json — JSON Mapping Quality

Positive observations:

• All 7/7 top-level fields match gold exactly
• 8/8 gold mapping categories are present
• destination_variable perfectly matched across all 8 fields
• Coordinate metadata correctly inferred from BM_EGM_Well_CO2.csv

Issues and observations:

1. Extra gravimetric_water_content category (mapping.json:68-76): The agent added this mapping (correctly noting it should be null/NA), but it's absent from the gold mapping JSON. This is arguably better behavior (more complete documentation), but the evaluator should decide whether this is counted as a "false positive" category in the benchmark metric.

2. source_pattern field uses raw column names (mapping.json:39, 59): The agent used "Depth (cm)" and "Volumetric Water Content", while gold uses normalized strings "Depth..cm." and "Volumetric.Water.Content". This is a systematic difference in normalization convention that reduces the source_pattern match rate to 3/8. This difference should be documented in the benchmark's known-variance notes, since both representations are factually correct.

3. transformation field: 0/8 literal matches: The field-level exact match rate of 0/8 for transformation text is the largest gap in the mapping. This is an expected outcome given free-form prose, but it would be useful to know whether a semantic similarity score was computed, since 0/8 on literal match understates the quality of the transformation descriptions.

4. site_id source attribution (mapping.json:27-35): The agent cites both the BM file prefix and BM_EGM_Well_CO2.csv, while gold cites only the location file. The agent's more complete attribution is arguably more accurate.

---

curator_bundle.json — Curation Artifact

Positive observations:

• curator_decision: "INCLUDE" with null exclusion reason is correct
• time_series_inference correctly identifies 60-minute intervals with proper reasoning
• similar_dataset_reference pointing to index 23 (a fold-local exemplar) is a sound and transparent reasoning choice
• open_questions documents the timezone inference limitation clearly — good practice

Minor observations:

1. interval_min: 60.0 in time_series_inference (curator_bundle.json:69): The curator bundle states the median interval as 60 min, while the Python script computes per-group diffs that produce 6 non-60-minute values at depth transitions. These are consistent with each other, but the bundle's claim is a median/typical value, not a guaranteed exact value — which is appropriate.

2. BM_EGM_Surface_Flux.csv correctly excluded (curator_bundle.json:26-31): The reasoning that 36 non-null Soil.Moisture values with no depth metadata should not be harmonized is correct and well-documented.

---

EVALUATION.md — Evaluation Report

Positive observations:

• Thorough, well-structured evaluation across three dimensions (output, JSON, Python)
• The interval_min discrepancy is correctly identified and contextualized
• The NumPy 2 compatibility shim for add_loc_qc is documented — important for reproducibility

Potential issues:

1. NumPy 2 compatibility shim (EVALUATION.md:45-47): The shim affects the add_loc_qc side output. The report states it "did not alter the soil-moisture transformation being compared," but this claim isn't backed by a test. Consider adding a note about how this was verified (e.g., column-by-column comparison before and after shim application).

2. Transformation match metric (EVALUATION.md:119): transformation: 0/8 is reported without any qualifier. If the goal is to evaluate semantic correctness (not literal text equality), this metric as presented may be misleading. A note that "0/8 literal matches does not imply 0/8 semantic matches" would be valuable.

---

evaluation_metrics.json — Machine-Readable Metrics

Positive observations:

• external_metadata_access is correctly captured with classified_as_gold_leakage: false
• python_equivalence flags are specific and actionable
• exact_equal_sorted_except_interval_min: true is a clean and honest summary

Suggestions:

1. missed_gold_interval_order: true (evaluation_metrics.json:78): The field name is slightly ambiguous — "order" suggests row ordering, but the actual issue is grouping strategy for interval computation. A more precise key like missed_gold_interval_grouping or a description field would improve clarity.

2. Missing semantic similarity scores: The json_mapping_equivalence section captures exact match counts but no semantic or fuzzy match scores. Given that transformation: 0/8 on literal match, a BLEU/BERTScore or even manual quality rating per field would be more informative for benchmark calibration.

---

AGENT_ACTION_LOG.md — Action Trace

Positive observations:

• Clean chronological trace with well-formed log entries
• The agent correctly removed the temporary package_metadata_probe.json artifact before finalizing
• Self-scan for forbidden paths is documented (line 33)
• ESS-DIVE API access is explicitly noted and justified (lines 19-21)

One observation:

1. ESS-DIVE API access (AGENT_ACTION_LOG.md:19-21): The evaluation correctly flags this in EVALUATION.md as an external metadata access that is not gold leakage but may violate strict benchmark protocol. The PR summary mentions this should be "accounted for separately." It would be worth clarifying in the evaluation or experiment config whether external metadata access is permitted or should be penalized in future runs.

---

Security & Safety

No security concerns. The code does not:

• Execute arbitrary shell commands
• Expose secrets or credentials
• Read outside the defined sandbox paths (per the anti-leakage audit)

The hardcoded paths in harmonize_heldout.py raise portability concerns but not security concerns.

---

Test Coverage

The PR references uv run pytest in the validation section. The existing test files (test_build_env.py, test_expert_harmonizer.py, test_stage_raw_data.py) do not appear to directly test the agent output equivalence logic. A test that mechanically reproduces the output comparison performed in EVALUATION.md would strengthen the evaluation pipeline's reproducibility.

---

Overall Assessment

This is a high-quality evaluation bundle. The agent output is nearly gold-equivalent, the evaluation report is thorough and honest about discrepancies, and the anti-leakage audit is well-documented. The main actionable items are:

• Blocker-level: Hardcoded absolute paths in harmonize_heldout.py prevent portability
• Moderate: ESS-DIVE API access policy should be clarified in the benchmark protocol
• Minor: interval_min grouping choice and transformation literal match metric deserve inline documentation