Vally harness and CI workflow updates by LarryOsterman · Pull Request #380 · microsoft/skills

LarryOsterman · 2026-07-15T17:23:14Z

Split from #375 (2 of 5).

Updates the Vally test harness code and CI workflows:

tests/harness/ runner/output-capture changes, new copilot-client.test.ts, ralph-loop and types updates
.github/workflows/ — renamed vally-evaluation.yml -> run-vally-evaluations.yml, test-harness.yml update
.github/aw/actions-lock.json

Split from #375: harness output capture and runner updates, new copilot-client tests, and CI workflow updates (rename vally-evaluation to run-vally-evaluations, test-harness tweaks, actions-lock). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9429ad33-6f31-4ec5-ba00-0f11b5afb82e

Copilot

Pull request overview

This PR updates the Vally test harness to capture richer generation output (including raw model responses) and to better handle Copilot generation failures, while also adjusting CI workflows to run Vally evaluations under updated tooling/permissions.

Changes:

Extend harness result types and reporting to include rawResponse from generation.
Add Copilot generation retry/error-classification logic and improve runner error output behavior (especially for JSON output).
Update GitHub Actions workflows for harness link-sync checks and Vally evaluation execution (renamed workflow, updated Node/Vally versions, permissions, and PR scoping).

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/harness/types.ts	Adds `rawResponse` to evaluation results and updates defaults/signatures.
tests/harness/runner.ts	Captures `rawResponse`, adds generation failure findings/suggestions, and adds JSON error output helpers.
tests/harness/ralph-loop.test.ts	Updates mocks/tests to satisfy the new `rawResponse` field and formatting changes.
tests/harness/copilot-client.ts	Adds typed Copilot generation errors, retry/backoff, configurable timeout/retries, and improved code extraction heuristics.
tests/harness/copilot-client.test.ts	Adds unit tests for `extractCode` and `classifyCopilotError`.
.github/workflows/test-harness.yml	Expands skill link sync verification to include Python in addition to Rust.
.github/workflows/run-vally-evaluations.yml	Renames/updates the Vally workflow, updates tool versions, permissions, and PR scoping behavior.
.github/aw/actions-lock.json	Adds a locked entry for `github/gh-aw-actions/setup@v0.81.6`.

+        if (s && typeof s.disconnect === "function") {
          await s.disconnect();
-        } else if (typeof s.destroy === "function") {
+        } else if (s && typeof s.destroy === "function") {
          await s.destroy();
        }


+function classifyCliError(error: unknown): CliErrorKind {
+  if (error instanceof CopilotGenerationError) {
+    return error.kind;
+  }
+
+  const message = error instanceof Error ? error.message.toLowerCase() : "";
+  if (
+    message.includes("authentication failed") ||
+    message.includes("bad credentials") ||
+    message.includes("401")
+  ) {
+    return "auth";
+  }
+  if (message.includes("timeout") || message.includes("session.idle")) {
+    return "timeout";
+  }
+  if (message.includes("rate limit") || message.includes("429")) {
+    return "transient";
+  }
+  return "fatal";
+}


LarryOsterman · 2026-07-15T17:35:18Z

Split from #375 (see #375 for the full split plan). Do not merge #375 until this and the sibling split PRs (#379, #380, #381, #382, #383) are merged.

richardpark-msft

Bringing over some comments to this PR.

The file was removed on this split branch under the assumption it belonged only to PR #380, but #380 already merged it into main unchanged. Since the merge-base copy matched main, merging this branch would have deleted the file from main. Restoring it to main's version so this branch carries it through untouched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…383) * Add Vally skill-effectiveness experiments and Python test scenarios Split from #375: adds the Vally skill-effectiveness experiment framework (run-skill-experiments.ps1, run-all-evals.ps1, generate-skill-experiments.ps1, compare-experiment.mjs), standardized vally eval/experiment yaml and scenarios.yaml for ~40 Python skills, shared Python grader tooling, and harness runner script/dependency updates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9429ad33-6f31-4ec5-ba00-0f11b5afb82e * Update vally version * Update package info * Fix review comments: delete artifact, fix exception handling, add exclude-pattern support * Fix review 4708288806 comments: exceptions, duplicate files, tab check, PS1 fixes * Removed experiment specific evaluations; cleaned up existing eval.yaml files * Removed bogus files * test: cover python static graders * fix: address all 10 comments from review 4739490318 * fix: rename grader_model to judge_model in all Python eval specs (Vally 0.10) * Improved reliability of prompts * Fixed test harness for azure-appconfiguration-py skill * Fix --junit output path and update Node.js version prerequisite * Fix fastapi-router-py template.py to use {{}} placeholder syntax * fix: update Node.js version to 22 in test-harness workflow * fix: skip template Python files containing {{...}} placeholders in syntax check * fix: remove invalid 'execution' and 'portability' fields from GraderMetadata * fix: allow esbuild builds in tests workspace * fix: restore required vally grader metadata * fix: remove obsolete execution and portability fields from rust-cargo-build-failure grader * Fix remaining review thread feedback * Fixes from rust branch * Apply remaining changes * Fix grader metadata typing * Remove test-harness.yml workflow (belongs to PR #380) * Fix remaining review thread feedback * Restore test-harness.yml deleted in error The file was removed on this split branch under the assumption it belonged only to PR #380, but #380 already merged it into main unchanged. Since the merge-base copy matched main, merging this branch would have deleted the file from main. Restoring it to main's version so this branch carries it through untouched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add @microsoft/vally-cli to test devDependencies and update lockfile * Fix Windows-only backslashes and README pnpm --dir path - tests/run-all-evals.ps1: Replace backslashes with forward slashes in customGraderPluginDir, customGraderDistEntry, and vallyCmd paths so that Join-Path resolves them correctly on Linux/macOS. - tests/scenarios/README.md: Fix 'pnpm --dir ../../../..' (repository root) to 'pnpm --dir ../../..' (tests/) and update the experiment file and output-dir paths to be relative to tests/. * Move Vally expect_skills constraints into each stimulus * Fix azure-eventhub-rust review feedback * Fix remaining fastapi and eval review issues * Updated timeouts on skills * Fix review 4766529446: experiment evals, vally probe, timestamps, dead params, version gate * Removed skill_effectiveness_eval.yaml files for python * Repoint python skill experiments to eval.yaml and remove generator script Completes 27ae1ab: the eval.yaml deletions were committed but the experiment repoints (skill_effectiveness_eval.yaml -> eval.yaml) and the generate-skill-experiments.ps1 deletion were left unstaged, so git status kept showing them. Commit them so the python experiments reference the single general-purpose eval.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 82323067-0922-476e-8f08-4659317988ab * Regenerate rust skills and add guidance on structures. * Fix 5 issues from review 4768872057: pnpm-workspace bool, cargo clippy -D warnings, compare-experiment score check, run-skill-experiments local vally, README pylint install step * Cleaned up eval job; simplified storage skills somewhat * Fix azure-mgmt-apimanagement-py: remove invalid require_envvar from DefaultAzureCredential * Fix cargo clippy --D warnings in Rust eval files and fix -rs typo in run-skill-experiments.ps1 * Fix pydantic-models-py: simplify template to prevent file truncation in evals The base_create_response_indb stimulus was failing Python syntax and idiomatic checks because the LLM generated oversized Python files (with helpers and section comments) that got truncated during writing. Changes: - Reduce template.py from 124 to 48 lines using concise ConfigDict pattern, inline Field calls, and brief docstrings - Update SKILL.md examples to use ConfigDict (modern Pydantic v2) - Remove verbose section separators and boilerplate comments * Fix pydantic vally camel_case_aliases workspace output path * Fix vally CI: add file save instructions and improve error handling examples - pydantic-models-py eval: add file save instruction to base_create_response_indb prompt so agents write a .py file, fixing "No Python files found" grader failure - azure-ai-translation-document-py eval: add file save instructions to both stimuli for deterministic grader behavior - azure-ai-translation-document-py skill: rewrite Basic Document Translation section with complete self-contained example including try/except HttpResponseError; remove invalid require_envvar=True parameter from DefaultAzureCredential() * Fix review 4790830267: add file save instructions and fix eval output key collision - Thread 97: Add 'Save the code to <file>.py in the current workspace.' to all eval stimuli that use Python graders but lacked explicit file output paths, preventing deterministic grader failures when models respond without saving files. Affected: agent-framework-azure-ai-py, azure-ai-contentsafety-py, azure-ai-contentunderstanding-py, azure-ai-ml-py, azure-ai-projects-py, azure-ai-textanalytics-py, azure-ai-transcription-py, azure-ai-translation-text-py, azure-ai-vision-imageanalysis-py, azure-ai-voicelive-py, azure-containerregistry-py, azure-cosmos-db-py, fastapi-router-py, m365-agents-py, azure-storage-queue-py - Thread 98: Include eval filename in output key in run-all-evals.ps1 so that eval.yaml and skill_effectiveness_eval.yaml in the same vally/ folder write to separate output directories, preventing JUnit file and run-dir lookup collisions * Removed experimental files that are no longer needed * fix: add #Requires -Version 7.0 to run-all-evals.ps1 (review 4739841645) * Fix 6 issues from review 4791599641: workspace file requirements and producer open() &str fix * Fix all review #4791806947 comments: syntax checks, compare-experiment, package/workflow versions, dead params, exit code tracking * Fix azure-ai-projects-py: wrong import paths and invalid credential parameter * Fix review 4791806947: add save instructions to eval.yaml stimuli, fix blob-rust ignore_for_fail, fix skill-creator Rust guidance - Add 'Save the code to file.py' instructions to all eval.yaml stimuli missing them (31 stimuli across 18 Python scenario files) so the workspace-only syntax/pylint graders can find the .py files - Remove error_exceptions from Post-prompt graders in azure-storage-blob-rust/vally/eval.yaml (keep only on trajectory analysis), preventing genuine E0277 compilation failures from being masked as successful builds - Fix #[non_exhaustive] guidance in skill-creator/SKILL.md: distinguish between externally constructible structs (use ..Default::default()) and truly non_exhaustive structs (use constructor/builder or default-then-mutate) to avoid E0639 compile errors * Fix azure-monitor-opentelemetry-py: add explicit connection_string example, fix invalid credential param - Replace "Explicit Configuration" section with "Explicit Connection String" section showing connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"] with try/except - Remove invalid DefaultAzureCredential(require_envvar=True) parameter (not a valid API) - Update capabilities.md to reflect renamed section This fixes the azure-monitor-opentelemetry-py-eval explicit_connection_string scenario which was failing Python syntax and idiomatic checks because the skill lacked a canonical example for the connection_string parameter pattern. * Fix --junit flag, add datetime import, add rust grader plugin support in experiments - run-all-evals.ps1: Remove filename arg from --junit flag; Vally 0.10 treats it as boolean - pydantic-models-py/SKILL.md: Add missing 'from datetime import datetime' import in camelCase Aliases example - run-skill-experiments.ps1: Mirror rust-cargo-build-failure grader plugin detection/build/pass from run-all-evals.ps1 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Copilot-Session: 9429ad33-6f31-4ec5-ba00-0f11b5afb82e Copilot-Session: 82323067-0922-476e-8f08-4659317988ab

Copilot AI review requested due to automatic review settings July 15, 2026 17:23

LarryOsterman requested a review from thegovind as a code owner July 15, 2026 17:23

Copilot started reviewing on behalf of LarryOsterman July 15, 2026 17:24 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

LarryOsterman requested a review from a team July 15, 2026 17:27

This was referenced Jul 15, 2026

Add Vally skill-effectiveness experiments and Python test scenarios #383

Merged

Add python skills and test. #375

Closed

skill-creator: lifecycle guidance and Azure SDK patterns reference #379

Merged

This was referenced Jul 15, 2026

Standardize Rust Vally scenarios and remove azure-servicebus-rust #381

Merged

Add capability and non-hero references for Python skills #382

Merged

richardpark-msft reviewed Jul 16, 2026

View reviewed changes

Comment thread .github/workflows/run-vally-evaluations.yml Outdated

Comment thread .github/workflows/run-vally-evaluations.yml Outdated

Copilot started work on behalf of LarryOsterman July 16, 2026 20:28 View session

Copilot started work on behalf of LarryOsterman July 16, 2026 20:30 View session

Add missing .github/skills symlinks for azure-sdk-python plugin

8dd8f99

Copilot AI review requested due to automatic review settings July 16, 2026 20:32

LarryOsterman removed the request for review from Copilot July 16, 2026 20:32

fix: address workflow review suggestions

bdd55c1

Copilot AI review requested due to automatic review settings July 16, 2026 20:32

LarryOsterman removed the request for review from Copilot July 16, 2026 20:32

Copilot finished work on behalf of LarryOsterman July 16, 2026 20:32

Copilot started work on behalf of LarryOsterman July 16, 2026 20:33 View session

Copilot finished work on behalf of LarryOsterman July 16, 2026 20:34

LarryOsterman merged commit 27fbd3b into main Jul 16, 2026
5 checks passed

Copilot AI added a commit that referenced this pull request Jul 22, 2026

Remove test-harness.yml workflow (belongs to PR #380)

14e6067

LarryOsterman deleted the larryo/split/harness-and-ci branch July 28, 2026 23:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Vally harness and CI workflow updates - #380

Vally harness and CI workflow updates#380
LarryOsterman merged 3 commits into
mainfrom
larryo/split/harness-and-ci

LarryOsterman commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

LarryOsterman commented Jul 15, 2026

Uh oh!

richardpark-msft left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

LarryOsterman commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

LarryOsterman commented Jul 15, 2026

Uh oh!

richardpark-msft left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants