[codex] add fold03 agent evaluation bundle

[jmchandonia]

Summary

• add tracked fold-03 holdout-4 sub-agent artifacts, including action traces and generated outputs
• add an evaluation report comparing the agent output, JSON mapping, and Python logic against the held-out gold standard
• add machine-readable evaluation metrics for the run

Validation

• python -m py_compile evaluations/fold-03-holdout-4/agent_outputs/harmonize_heldout.py
• UV_CACHE_DIR=/scratch/jmc/data-harmonization-eval/.uv-cache uv run pytest

Notes

• The successful trace passed the targeted forbidden-path scan with zero matches.
• The agent recovered all 552 gold rows but retained 449 extra tb rows, giving perfect recall and 55.1% row precision.

[claude]

Claude finished @jmchandonia's task —— View job

---

PR Review

Tasks

• Gather context and understand the PR
• Read and analyze changed files
• Provide comprehensive review feedback

---

Summary

This PR adds a well-structured evaluation bundle for fold-03-holdout-4, documenting a sub-agent's attempt at harmonizing the ess-dive-6c7085e9c544cc6-20250424T164534831 dataset. The artifacts are clean and the evaluation methodology is thorough. Below are observations by category.

---

Code Quality & Best Practices

harmonize_heldout.py

• Hardcoded absolute paths (/scratch/jmc/..., /h/jmc/...) at lines 9–17 make the script non-portable. If run on a different machine or by a different user, it will fail immediately. Consider parameterizing these paths via environment variables or CLI arguments for reproducibility.
• ensure_harmonized_cols is a no-op here: since out is constructed column-by-column from TARGET_COLS in harmonize(), the function at line 38–42 will never add a missing column — it effectively just reorders. This is fine for correctness, but the function's intent (guarding against missing columns) is already satisfied before it's called.
• log_line appends to a hardcoded log path (LOG_PATH at line 11), which is the fold sandbox path. If this script is re-run outside the sandbox, it will create or corrupt the wrong log file. The logging side-effect is tightly coupled to the absolute path.
• The from __future__ import annotations import (line 2) is unused given the type annotations in this file. Harmless, but unnecessary.
• Missing tb site filter: as documented in EVALUATION.md, the script does not drop site_id == "tb" rows, resulting in 449 extra rows in the output (55.1% row precision). This is the primary correctness gap vs. the gold standard.
• No sort: gold sorts by datetime_UTC, site_id; the agent omits this. Downstream consumers depending on ordering would get inconsistent results.

mapping.json

• "doi": null — the DOI (doi:10.15485/2561511) was known to the gold standard but not found by the agent. The mapping_notes.md correctly documents this as an open issue.
• "location_metadata_files" includes an extra East Taylor reference file not in the gold. This is documented in EVALUATION.md and is a defensible inclusion given the agent's location-resolution workflow, but it is a literal divergence from gold.
• "transformation" field matches 0/8 entries literally against gold — this is a known limitation of free-text prose fields. The semantic content appears correct even if the phrasing differs.

---

Potential Bugs / Issues

1. tb rows not filtered (harmonize_heldout.py:85): out = out[out['datetime_UTC'].notna()].copy() filters only on missing timestamps. The gold expert module drops the tb site entirely. This is the root cause of 449 extra rows and ~45% precision loss. A one-line fix: out = out[out['site_id'] != 'tb'] before the notna filter.

2. Timestamp format divergence: The agent writes ISO strings ending in Z (e.g., "2022-07-22T06:00:00Z") while the gold stores timezone-aware UTC Timestamp objects in the DataFrame. When written to CSV these would serialize differently. The EVALUATION.md notes they parse equivalently, but any string comparison (rather than datetime comparison) in downstream evaluation would fail.

3. parse_local_date_to_utc_iso uses format='%Y-%m-%d' (line 46): This is correct for the source data but will silently coerce (to NaT) any row with a malformed date due to errors='coerce'. There's no warning emitted for coerced dates, which could silently drop rows. The final notna() filter at line 85 catches this but without logging the count dropped.

4. Row count validation (line 64): The check len(ddf) != len(mdf) raises an error only if total row counts differ. It does not validate that the join is row-aligned beyond count equality — if the files were sorted differently, the concatenation would silently misalign site/date metadata with measurement values.

---

Performance Considerations

• The script loads the entire df_data.csv (1001 rows, ~0.1 MB) and df_meta.csv into memory, which is trivially small. No performance concerns.
• The log_line function opens and closes the log file on every call. For small scripts this is fine, but it's worth noting for larger-scale evaluations.

---

Security Concerns

• The hardcoded paths (/h/jmc/..., /scratch/jmc/...) reference user-specific locations. If this script were run in a CI/CD environment, these paths would not exist and would fail fast — not a security risk, but a portability one.
• The AGENT_ACTION_LOG.failed-write-location.md at line 50 records a violation marker: the first agent attempt wrote artifacts outside the fold sandbox. This is correctly preserved for the audit trail. The anti-leakage scan on the successful trace correctly found 0 forbidden path matches.
• No secrets, credentials, or sensitive data appear to be hardcoded.

---

Evaluation Methodology Observations

• Anti-leakage audit is commendable and well-documented. The rg scan command is included in EVALUATION.md making it reproducible.
• The failed trace (AGENT_ACTION_LOG.failed-write-location.md) is preserved for audit purposes — good practice.
• evaluation_metrics.json is well-structured and machine-readable. The python_equivalence boolean flags (missed_drop_tb, missed_sort, etc.) clearly identify the gaps.
• The "transformation": "0/8" JSON metric correctly reflects that free-text fields are essentially uncomputable for exact string matching; it may be worth adding a semantic/fuzzy match score in future iterations.

---

Suggested Improvements for Future Runs

1. Drop tb rows: out = out[out['site_id'] != 'tb']
2. Add sort: out = out.sort_values(['datetime_UTC', 'site_id'])
3. Parameterize paths to improve portability
4. Log count of coerced/dropped rows from parse_local_date_to_utc_iso
5. Consider a (site, datetime) key-level alignment check in addition to total row count

---

Overall Assessment

The bundle is well-organized and the evaluation is honest about the agent's gaps. The core transformation (timezone conversion, column mapping, units) is correct. The primary miss is the tb site filter — a small but impactful omission that explains the ~45% precision penalty. The JSON mapping is a moderate match to gold (25/40 field-level exact matches), with the zero-match on transformation being an expected limitation of prose-text comparison. The audit trail is thorough and the failed trace is appropriately preserved.