Skip to content

[codex] add fold11 agent evaluation bundle #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
jmchandonia wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[codex] add fold11 agent evaluation bundle #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions evaluations/fold-11-holdout-23/AGENT_ACTION_LOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
2026-06-30T16:04:39-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/AGENT_ACTION_LOG.md | reason=initialize mandatory trace log
2026-06-30T16:04:39-07:00 | action=directory create | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs | reason=initialize output directory
2026-06-30T16:04:39-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/AGENT_INSTRUCTIONS.md | reason=read benchmark instructions
2026-06-30T16:04:39-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/MANIFEST.json | reason=read sandbox manifest
2026-06-30T16:04:39-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/skills/essdive_sm_curator/SKILL.md | reason=read fold-local curator skill
2026-06-30T16:04:39-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/skills/essdive_sm_harmonizer/SKILL.md | reason=read fold-local harmonizer skill
2026-06-30T16:04:47-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/skills/essdive_sm_harmonizer/SKILL.md lines 261-520 | reason=complete fold-local harmonizer skill read
2026-06-30T16:04:56-07:00 | action=directory listing | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23 | reason=inspect sandbox top-level files
2026-06-30T16:04:56-07:00 | action=directory listing | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data | reason=inspect fold-local data files only
2026-06-30T16:04:56-07:00 | action=directory listing | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379 | reason=inspect allowed held-out raw package files
2026-06-30T16:05:20-07:00 | action=command | subject=python inspect fold-local exemplars and held-out raw files | reason=summarize allowed mapping/code/raw inputs
2026-06-30T16:05:21.293887-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/processed/ess-dive_wfsfa_soil_datasets/sm_data_harmonization_mapping.json | reason=inspect fold-local mapping schema and exemplar entries
2026-06-30T16:05:21.294834-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/gold/expert_code/harmonize_sm/common.py | reason=inspect allowed common harmonizer helpers
2026-06-30T16:05:21.294976-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/gold/expert_code/harmonize_sm/dataset_24.py | reason=inspect allowed exemplar code dataset_24
2026-06-30T16:05:21.295094-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/gold/expert_code/harmonize_sm/dataset_25.py | reason=inspect allowed exemplar code dataset_25
2026-06-30T16:05:21.295189-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/gold/expert_code/harmonize_sm/dataset_26.py | reason=inspect allowed exemplar code dataset_26
2026-06-30T16:05:21.295276-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/gold/expert_code/harmonize_sm/dataset_27.py | reason=inspect allowed exemplar code dataset_27
2026-06-30T16:05:21.295389-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/gold/expert_code/harmonize_sm/dataset_18.py | reason=inspect allowed exemplar code dataset_18
2026-06-30T16:05:21.295489-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/gold/expert_code/harmonize_sm/dataset_16.py | reason=inspect allowed exemplar code dataset_16
2026-06-30T16:05:21.295576-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/data/gold/expert_code/harmonize_sm/dataset_15.py | reason=inspect allowed exemplar code dataset_15
2026-06-30T16:05:21.295668-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/WM_SWC.csv | reason=inspect held-out raw CSV header and sample rows
2026-06-30T16:05:21.472686-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/WM_SWC.csv | reason=inspect held-out raw CSV shape and dtypes
2026-06-30T16:05:23.952391-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/plot_metadata.csv | reason=inspect held-out raw CSV header and sample rows
2026-06-30T16:05:23.957491-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/plot_metadata.csv | reason=inspect held-out raw CSV shape and dtypes
2026-06-30T16:05:23.986915-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/sensor_metadata.csv | reason=inspect held-out raw CSV header and sample rows
2026-06-30T16:05:23.993878-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/sensor_metadata.csv | reason=inspect held-out raw CSV shape and dtypes
2026-06-30T16:05:52-07:00 | action=command | subject=curl ESS-DIVE package metadata for ess-dive-a99be52b7a6114c-20230504T210134503379 | reason=retrieve DOI and package-level metadata when fold cache is absent
2026-06-30T16:05:52-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/_essdive_meta_probe.json | reason=temporary metadata probe inside agent_outputs
2026-06-30T16:07:59-07:00 | action=command | subject=curl ESS-DIVE package metadata for ess-dive-a99be52b7a6114c-20230504T210134503379 with network | reason=retrieve DOI and package-level metadata when fold cache is absent
2026-06-30T16:07:59-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/_essdive_meta_probe.json | reason=temporary metadata probe inside agent_outputs
2026-06-30T16:08:01-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/_essdive_meta_probe.json | reason=inspect retrieved ESS-DIVE metadata probe
2026-06-30T16:08:15-07:00 | action=command | subject=python parse ESS-DIVE metadata probe | reason=extract DOI, spatial coverage, temporal coverage, distribution filenames
2026-06-30T16:08:15.212319-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/_essdive_meta_probe.json | reason=parse retrieved metadata fields
2026-06-30T16:08:40-07:00 | action=command | subject=python profile held-out sensor joins and cadence | reason=verify merge keys, depth mapping, replicate strategy, interval
2026-06-30T16:08:41.315174-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/sensor_metadata.csv | reason=profile sensor metadata keys
2026-06-30T16:08:41.319556-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/WM_SWC.csv | reason=profile payload keys and timestamp cadence
2026-06-30T16:09:38-07:00 | action=command | subject=python inspect post-merge output candidates | reason=verify missing metadata, exact duplicates, replicate grouping
2026-06-30T16:09:38.502999-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/WM_SWC.csv | reason=inspect payload for harmonization candidate rows
2026-06-30T16:09:39.852104-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/sensor_metadata.csv | reason=inspect sensor metadata for harmonization candidate rows
2026-06-30T16:10:01-07:00 | action=command | subject=python inspect missing-depth payload rows | reason=document handling for rows without point/depth metadata
2026-06-30T16:10:02.383473-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/WM_SWC.csv | reason=read missing-depth payload rows
2026-06-30T16:10:03.729952-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/sensor_metadata.csv | reason=read missing-depth sensor metadata rows
2026-06-30T16:11:18-07:00 | action=command | subject=write harmonize_heldout.py | reason=create held-out harmonization script
2026-06-30T16:11:18-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/harmonize_heldout.py | reason=create standalone harmonizer output
2026-06-30T16:12:21-07:00 | action=command | subject=python write curator_bundle mapping and notes | reason=create required JSON and markdown outputs
2026-06-30T16:12:22.118709-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/_essdive_meta_probe.json | reason=read public metadata probe for DOI/title and package context
2026-06-30T16:12:22.118940-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/WM_SWC.csv | reason=compute payload summary for curator bundle
2026-06-30T16:12:23.780294-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/plot_metadata.csv | reason=compute plot metadata summary for curator bundle
2026-06-30T16:12:23.783355-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/sensor_metadata.csv | reason=compute sensor metadata summary for curator bundle
2026-06-30T16:12:24.813846-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/curator_bundle.json | reason=write curator bundle JSON
2026-06-30T16:12:24.814199-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/mapping.json | reason=write mapping JSON entry
2026-06-30T16:12:24.814444-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/mapping_notes.md | reason=write mapping notes markdown
2026-06-30T16:12:32-07:00 | action=command | subject=python /scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/harmonize_heldout.py | reason=run held-out harmonization script
2026-06-30T16:12:32.615715-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/WM_SWC.csv | reason=read held-out SWC payload for harmonization
2026-06-30T16:12:33.945648-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/sensor_metadata.csv | reason=read held-out sensor metadata for depth/site joins
2026-06-30T16:12:53-07:00 | action=command | subject=perl edit harmonize_heldout.py sensor serial casting | reason=fix SWC-only Sensor.SN normalization
2026-06-30T16:12:53-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/harmonize_heldout.py | reason=update harmonizer to cast Sensor.SN after SWC filtering
2026-06-30T16:12:53-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/harmonize_heldout.py | reason=verify modified harmonizer section
2026-06-30T16:13:01-07:00 | action=command | subject=python /scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/harmonize_heldout.py rerun | reason=generate held-out harmonized CSV after cast fix
2026-06-30T16:13:01.671650-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/WM_SWC.csv | reason=read held-out SWC payload for harmonization
2026-06-30T16:13:03.334949-07:00 | action=file read | subject=/h/jmc/ess-dive_wfsfa_soil_datasets/ess-dive-a99be52b7a6114c-20230504T210134503379/sensor_metadata.csv | reason=read held-out sensor metadata for depth/site joins
2026-06-30T16:13:21.848342-07:00 | action=write | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/heldout_harmonized.csv | reason=write held-out harmonized CSV
2026-06-30T16:13:45-07:00 | action=command | subject=rm -f temporary metadata probe | reason=remove non-deliverable temporary file from agent_outputs
2026-06-30T16:13:45-07:00 | action=delete | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/_essdive_meta_probe.json | reason=remove non-deliverable temporary file
2026-06-30T16:14:09-07:00 | action=command | subject=python validate outputs and scan action log | reason=final schema/count validation and anti-leakage scan
2026-06-30T16:14:09.672234-07:00 | action=directory listing | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs | reason=verify only required deliverables are present
2026-06-30T16:14:09.672441-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/curator_bundle.json | reason=validate curator_bundle.json parses as JSON
2026-06-30T16:14:09.672699-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/mapping.json | reason=validate mapping.json parses as JSON
2026-06-30T16:14:09.672901-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/mapping_notes.md | reason=validate mapping notes exists and preview
2026-06-30T16:14:09.673045-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/harmonize_heldout.py | reason=validate harmonizer script exists and contains schema
2026-06-30T16:14:09.673229-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/agent_outputs/heldout_harmonized.csv | reason=validate harmonized CSV schema and row count
2026-06-30T16:14:11.809979-07:00 | action=file read | subject=/scratch/jmc/data-harmonization-eval/.runs/fold-11-holdout-23/AGENT_ACTION_LOG.md | reason=scan mandatory action log for forbidden path patterns
10 changes: 10 additions & 0 deletions evaluations/fold-11-holdout-23/AGENT_INSTRUCTIONS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Run environment — leave-one-cluster-out

Harmonize the held-out dataset(s) below using ONLY:
- the skills in `skills/`,
- the exemplars in `data/processed/.../sm_data_harmonization_mapping.json` and the code patterns in `data/gold/expert_code/harmonize_sm/` (both have the held-out cluster removed),
- the shared raw inputs under `~/ess-dive_wfsfa_soil_datasets/` and the cached metadata under `data/external/ess-dive_meta/`.

Do NOT look up the held-out dataset's existing harmonized output, expert code, or mapping entry from any other location. The held-out datasets are:

- `ess-dive-a99be52b7a6114c-20230504T210134503379`
58 changes: 58 additions & 0 deletions evaluations/fold-11-holdout-23/EVALUATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# fold-11-holdout-23 Agent Evaluation

## Scope

This directory records one sandboxed sub-agent attempt for held-out dataset indices `[23]`.

- Sandbox: `.runs/fold-11-holdout-23`
- Tracked artifact copy: `evaluations/fold-11-holdout-23`
- Successful agent trace: `AGENT_ACTION_LOG.md`

Note: the raw `heldout_harmonized.csv` is 116,186,889 bytes, which exceeds
GitHub's 100 MB per-file limit. It is tracked as
`agent_outputs/heldout_harmonized.csv.gz`; see
`agent_outputs/heldout_harmonized_csv_manifest.json` for raw and compressed
artifact hashes.

## Anti-Leakage Audit

The successful trace was scanned for root gold data, root processed data, other fold sandboxes, held-out expert module names, and explicit `violation` markers. Result: `0` matches.

## Output Equivalence Against Gold

| Metric | Agent | Gold |
| --- | ---: | ---: |
| Rows | 1,722,716 | 912,417 |
| Target columns present | 9 / 9 | 9 / 9 |
| Schema exact match | true | n/a |
| Unique keyed rows recovered | 0 | 912,226 |
| Missing gold keyed rows | 912,226 | n/a |
| Extra agent keyed rows | 1,722,706 | n/a |
| Row precision | 0.000 | n/a |
| Gold recall | 0.000 | n/a |

Shared-row value matches after key merge:

| Column | Matched rows |
| --- | ---: |
| `interval_min` | 0/0 |
| `volumetric_water_content_m3_m3` | 0/0 |
| `gravimetric_water_content_gH2O_gs` | 0/0 |
| `water_potential_kPa` | 0/0 |
| `is_timeseries` | 0/0 |

## JSON Mapping Match

- Summary: `{"extra_agent_categories": ["gravimetric_water_content"], "field_exact_matches_by_name": {"destination_variable": "8/8", "source_files": "6/8", "source_pattern": "5/8", "transformation": "0/8", "unit_conversion": "6/8"}, "field_level_exact_matches": "25/40", "gold_mapping_categories_present": "8/8", "missing_gold_categories": [], "top_level_exact_matches": "5/7", "top_level_matches": {"archive_repository": true, "data_payload_files": true, "dataset_identifier": true, "doi": true, "index": true, "location_metadata_files": false, "sensor_metadata_files": false}}`

| Field | Exact matches |
| --- | ---: |
| `destination_variable` | 8/8 |
| `source_pattern` | 5/8 |
| `source_files` | 6/8 |
| `transformation` | 0/8 |
| `unit_conversion` | 6/8 |

## Overall Assessment

See `evaluation_metrics.json` for machine-readable details. The run passed the targeted local no-gold path audit. Output equivalence should be interpreted with the key policy recorded in the metrics (`key_includes_dataset_index=False`).
35 changes: 35 additions & 0 deletions evaluations/fold-11-holdout-23/MANIFEST.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"name": "fold-11-holdout-23",
"holdout_indices": [
23
],
"holdout_identifiers": [
"ess-dive-a99be52b7a6114c-20230504T210134503379"
],
"exemplar_indices": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
15,
16,
17,
18,
24,
25,
26,
27
],
"n_exemplars": 27,
"sources": {
"package": "data/gold/expert_code/harmonize_sm",
"mapping": "data/gold/sm_data_harmonization_mapping.json",
"skills": "skills"
}
}
Loading
Loading