Verify DeepTempo Direct-Edit + GPT-5-Codex Location Oracle result

[samarmstrong]

Please verify the DeepTempo Direct-Edit + GPT-5-Codex Location Oracle submission in PR #2.

Submitted result:

• Strict PoC+Unit: 77/230 = 33.48%
• PoC-only: 82/230 = 35.65%
• Setting: PatchEval Location Oracle
• Model: gpt-5-codex via OpenRouter
• Entry type: LLM / non-agent direct edit harness
• Returned API cost: approximately $0.181/CVE

Generation method:

• Direct search/replace edits -> git diff
• Official PatchEval problem statement
• Oracle vulnerable file content
• One compiler-feedback repair round for compiler/build failures
• No PoC or unit-test feedback during generation

Step-by-step reproduction (the generation harness is public at https://github.com/samarmstrong/patcheval-direct-edit; the submitted patch and usage artifacts are committed under its outputs/ for comparison):

1. Clone the public reproduction harness:

   git clone https://github.com/samarmstrong/patcheval-direct-edit
   cd patcheval-direct-edit

2. Clone and set up PatchEval at the path expected by the harness:

   git clone https://github.com/bytedance/PatchEval patcheval-upstream
   python3 -m venv .venv
   .venv/bin/python -m pip install -r patcheval-upstream/requirements.txt

3. Download the 230 PatchEval Docker images:

   scripts/prefetch_full230_images.sh

4. Set the OpenRouter API key without printing it:

   export OPENROUTER_API_KEY=...

5. Use the pinned model id:

   openai/gpt-5-codex
   

6. Generate the initial 230 patches (fresh output names so the submitted artifacts in outputs/ remain available for comparison):

   PYTHONUNBUFFERED=1 .venv/bin/python \
     scripts/generate_edit_patches_v2.py \
     --upstream patcheval-upstream \
     --model openai/gpt-5-codex \
     --cve-file full230_cves.txt \
     --output outputs/reproduction_patches.json \
     --usage-output outputs/reproduction_usage.json \
     --raw-output outputs/reproduction_raw.json \
     --request-timeout-seconds 1200 \
     --max-generation-retries 2

7. Run initial PatchEval validation:

   PYTHONUNBUFFERED=1 .venv/bin/python \
     scripts/run_disk_safe_validation.py \
     --upstream patcheval-upstream \
     --patch-file outputs/reproduction_patches.json \
     --output reproduction_initial \
     --cve-file full230_cves.txt

8. Generate one compiler-feedback repair round for compiler/build failures only:

   PYTHONUNBUFFERED=1 .venv/bin/python \
     scripts/generate_compile_feedback_repairs.py \
     --upstream patcheval-upstream \
     --model openai/gpt-5-codex \
     --validation-output patcheval-upstream/patcheval/evaluation/evaluation_output/reproduction_initial \
     --base-patch-file outputs/reproduction_patches.json \
     --output outputs/reproduction_compile_repairs.json \
     --usage-output outputs/reproduction_compile_repairs_usage.json \
     --raw-output outputs/reproduction_compile_repairs_raw.json \
     --request-timeout-seconds 1200

9. Merge repaired patches into outputs/reproduction_final_patches.json (same logic as scripts/supervise_full230_v2_pipeline.sh), validate the final patch set, and compare against the submitted summary.json:

   PYTHONUNBUFFERED=1 .venv/bin/python \
     scripts/run_disk_safe_validation.py \
     --upstream patcheval-upstream \
     --patch-file outputs/reproduction_final_patches.json \
     --output reproduction_final \
     --cve-file full230_cves.txt

   The canonical orchestration script for the full run is scripts/supervise_full230_v2_pipeline.sh.

Reproduction notes:

• The submitted patches are plain diffs.
• Local macOS/Colima validation required Docker/bind-mount fixes so /workspace/fix.patch remained available after container startup; Linux hosts matching the PatchEval README may not need those local harness fixes.
• Positive controls using embedded reference patches passed 3/3 strict locally.
• LLM generation is not guaranteed bit-for-bit deterministic; please expect possible variance of roughly 1-2 CVEs on a fresh generation run.

[samarmstrong]

Updated the reproduction steps above: the generation harness is now public at https://github.com/samarmstrong/patcheval-direct-edit (the original steps referenced a private repository). The submitted patch and usage artifacts are committed under its outputs/ directory for direct comparison, and PR #2's REPRODUCTION.md has been updated to match.