Add Vally skill-effectiveness experiments and Python test scenarios - #383
Merged
Conversation
Loading
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split from #375 (5 of 5).
Adds the Vally skill-effectiveness experiment framework and Python test scenarios:
run-skill-experiments.ps1,run-all-evals.ps1,compare-experiment.mjsvally/eval + experiment yaml andscenarios.yamlfor ~40 Python skillstests/scenarios/_shared/run-harness-by-language.ps1and test dependency manifests