diff --git a/.github/workflows/benchmark-latency.yml b/.github/workflows/benchmark-latency.yml new file mode 100644 index 00000000..04eee48e --- /dev/null +++ b/.github/workflows/benchmark-latency.yml @@ -0,0 +1,125 @@ +# Latency Budget Enforcement CI +# +# Runs the CLI cold-start and command-latency Criterion benchmarks on every +# push / pull request and checks the results against the project's latency +# budgets. Fails the workflow when a statistically meaningful regression is +# detected, helping to keep the CLI snappy for all users. +# +# See CLI_LATENCY_BUDGETS.md for budget definitions and environment overrides. + +name: Latency Budget + +on: + push: + branches: [master, main] + paths: + - '**.rs' + - 'Cargo.toml' + - 'Cargo.lock' + - 'benches/**' + - '.github/workflows/benchmark-latency.yml' + pull_request: + paths: + - '**.rs' + - 'Cargo.toml' + - 'Cargo.lock' + - 'benches/**' + - '.github/workflows/benchmark-latency.yml' + + # Allow manual trigger for ad-hoc budget checks. + workflow_dispatch: + +jobs: + latency-budget: + name: Latency Budget Check + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + + - name: Install Rust toolchain + uses: dtolnay/rust-toolchain@stable + + - name: Install system dependencies + run: sudo apt-get update && sudo apt-get install -y libudev-dev + + # Build the project first so benchmark compilation is faster. + - name: Build project + run: cargo build --locked + + # Run only the latency-relevant benchmark groups. This keeps CI fast + # while still exercising cold-start and command-dispatch paths. + - name: Run latency benchmarks + run: | + cargo bench --locked -- \ + cli_cold_start \ + cli_command_latency \ + latency_budget \ + 2>&1 | tee target/criterion/latency-bench-output.txt + + # Parse Criterion output and check against latency budgets. + # The check-latency-budgets.sh script extracts median values from the + # default Criterion stdout format and compares them against the budgets + # defined in the bash script (which mirror src/utils/latency_budget.rs). + - name: Parse and check latency budget report + id: budget-check + run: | + chmod +x scripts/check-latency-budgets.sh + REPORT=$(bash scripts/check-latency-budgets.sh \ + --input target/criterion/latency-bench-output.txt \ + --report-path target/criterion/latency-budget-report.json \ + 2>&1) || EXIT_CODE=$? + echo "::group::Latency Budget Report" + echo "$REPORT" + echo "::endgroup::" + ALL_PASS=$(echo "$REPORT" | jq -r '.all_pass') + echo "all_pass=$ALL_PASS" >> "$GITHUB_OUTPUT" + if [ "$ALL_PASS" = "false" ]; then + echo "❌ Latency budget violations detected!" + echo "failures=true" >> "$GITHUB_OUTPUT" + exit 1 + fi + echo "✅ All latency budgets met." + + - name: Upload Criterion report (artefact) + if: always() + uses: actions/upload-artifact@v4 + with: + name: criterion-latency-report + path: | + target/criterion/cli_cold_start/ + target/criterion/cli_command_latency/ + target/criterion/latency_budget/ + target/criterion/latency-bench-output.txt + target/criterion/latency-budget-report.json + + # Post a PR comment with the budget check summary when run on a PR. + - name: Comment PR with budget summary + if: github.event_name == 'pull_request' && always() + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const reportPath = 'target/criterion/latency-budget-report.json'; + let summary = '## ⏱ CLI Latency Budget Check\n\n'; + if (fs.existsSync(reportPath)) { + const report = JSON.parse(fs.readFileSync(reportPath, 'utf8')); + let details = ''; + for (const check of report.checks) { + const icon = check.status === 'PASS' ? '✅' : + check.status === 'FAIL' ? '❌' : + check.status === 'NOISY' ? '⚠️' : + check.status === 'SKIPPED' ? '⏭️' : '❗'; + summary += `| ${icon} ${check.budget} | ${check.budget_max_ms} ms | ${check.actual_median_ms} ms | ${check.status} |\n`; + } + if (report.any_fail) summary += '\n❌ **Some budgets were violated.**'; + else summary += '\n✅ **All budgets met.**'; + } else { + summary += '_No latency budget report was generated._'; + } + github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: summary + }); diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 394c856c..55707d4a 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -70,6 +70,9 @@ the next baseline so pull requests can compare performance consistently. |---|---| | `simulate_ops_10k` | Baseline wrapping-add loop — regression guard | | `cli_arg_parsing` | Argument tokenisation cost for common subcommands | +| `cli_cold_start` | Cold-start latency: `info`, `--help`, `--version` | +| `cli_command_latency` | Command dispatch latency for wallet, network, config, template, deploy | +| `latency_budget` | Latency budget check engine overhead | | `template_registry_deserialise` | JSON deserialisation at 10 / 50 / 200 / 1000 entries | | `template_registry_search` | Linear search over registry at 10 / 100 / 500 entries | | `wallet_entry_format` | KV formatting and JSON serialisation for wallet lists | @@ -125,3 +128,17 @@ To catch performance regressions in pull requests, compare the Criterion Criterion's `--output-format bencher` emits a machine-readable format compatible with [github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark). + +### Latency budget enforcement + +A dedicated [latency budget](CLI_LATENCY_BUDGETS.md) CI workflow +(`benchmark-latency.yml`) runs cold-start and command-latency benchmarks on +every push / PR and **fails the pipeline** if any budget is exceeded: + +```yaml +- name: Run latency benchmarks + run: cargo bench --locked -- cli_cold_start cli_command_latency latency_budget +``` + +See [`CLI_LATENCY_BUDGETS.md`](./CLI_LATENCY_BUDGETS.md) for the full list of +budgets, environment variable overrides, and how to add new budgets. diff --git a/CLI_LATENCY_BUDGETS.md b/CLI_LATENCY_BUDGETS.md new file mode 100644 index 00000000..ab15f442 --- /dev/null +++ b/CLI_LATENCY_BUDGETS.md @@ -0,0 +1,160 @@ +# CLI Latency Budgets + +StarForge enforces **latency budgets** for its CLI's cold-start and command +dispatch paths. This ensures that every `starforge` invocation feels snappy +and that performance regressions are caught in CI before they reach users. + +--- + +## What is a latency budget? + +A **latency budget** is a per-command upper bound on wall-clock execution time +(e.g. `cli_info` must complete within **300 ms** median). Budgets are defined +in [`src/utils/latency_budget.rs`](./src/utils/latency_budget.rs) and checked +against Criterion benchmark results. + +| Budget label | Default max (ms) | CV threshold | Description | +|-----------------------------|------------------|--------------|--------------------------------------------| +| `cli_cold_start_info` | 500 | 0.15 | Cold start: `starforge -q info` | +| `cli_cold_start_help` | 350 | 0.15 | Cold start: `starforge --help` | +| `cli_cold_start_version` | 300 | 0.15 | Cold start: `starforge --version` | +| `cli_wallet_list` | 300 | 0.20 | `starforge wallet list` | +| `cli_wallet_show` | 350 | 0.20 | `starforge wallet show ` | +| `cli_network_show` | 250 | 0.20 | `starforge network show` | +| `cli_network_switch` | 300 | 0.20 | `starforge network switch ` | +| `cli_config_show` | 300 | 0.20 | `starforge config show` | +| `cli_info` | 300 | 0.20 | `starforge info` | +| `cli_template_list` | 350 | 0.20 | `starforge template list` | +| `cli_template_search` | 400 | 0.20 | `starforge template search ` | +| `cli_deploy_help` | 350 | 0.20 | `starforge deploy --help` | +| `cli_benchmark_wasm` | 500 | 0.20 | `starforge benchmark wasm` | + +> **CV threshold**: the maximum **coefficient of variation** (std_dev / mean) +> before a measurement is flagged as _noisy_ (yellow / warning). High noise +> means results are unreliable — the run should be retried with more samples. + +--- + +## How budgets are checked + +1. **Criterion benchmarks** measure median latency for each path + ([`benches/benchmarks.rs`](./benches/benchmarks.rs), groups + `cli_cold_start`, `cli_command_latency`, `latency_budget`). + +2. The **latency budget engine** ([`src/utils/latency_budget.rs`](./src/utils/latency_budget.rs)) + compares the measured median against the budget: + + - **PASS** — median ≤ budget, CV ≤ threshold + - **FAIL** — median > budget + - **NOISY** — median OK, but CV > threshold (retry recommended) + - **SKIPPED** — budget inactive for this environment + - **ERROR** — missing measurement or invalid input + +3. A **JSON report** is written to `target/criterion/latency-budget-report.json` + and consumed by CI to gate the pipeline. + +--- + +## Running budget checks locally + +```bash +# Run only the latency benchmarks +cargo bench -- locked -- cli_cold_start cli_command_latency latency_budget + +# Run the full suite (includes gas analyzer benchmarks) +cargo bench + +# Override a budget via environment variable (no code change needed) +STARFORGE_LATENCY_BUDGET_CLI_INFO=800 cargo bench -- cli_cold_start +``` + +--- + +## Environment variable overrides + +Each budget can be overridden at runtime with the environment variable +`STARFORGE_LATENCY_BUDGET_`. + +| Value | Effect | +|------------------------|-----------------------------------------| +| A positive integer (ms)| Sets the budget's `max_median` | +| `off` or `0` | Deactivates the budget (always skipped) | +| Any other string | Silently ignored (fallback to default) | + +Examples: + +```bash +# Double the info budget to 600 ms (useful on slower CI runners) +STARFORGE_LATENCY_BUDGET_CLI_INFO=600 cargo bench -- cli_cold_start + +# Deactivate the wallet list budget entirely +STARFORGE_LATENCY_BUDGET_CLI_WALLET_LIST=off cargo bench -- cli_cold_start +``` + +--- + +## CI integration + +The [`benchmark-latency.yml`](.github/workflows/benchmark-latency.yml) workflow +runs on every push / PR that touches `.rs` files, `Cargo.toml`, or the benches +directory. It: + +1. Builds the project +2. Runs the latency benchmark groups +3. Generates and inspects the JSON budget report +4. **Fails the workflow** if any budget is violated +5. Uploads the Criterion report as a build artefact +6. Comments on the PR with a budget summary table + +--- + +## Adding a new budget + +1. Add a new entry to the `default()` impl of `LatencyBudgets` in + [`src/utils/latency_budget.rs`](./src/utils/latency_budget.rs): + + ```rust + LatencyBudget::new_static("cli_my_new_command", 400, Some(0.20), true), + ``` + +2. Add a corresponding Criterion benchmark function in + [`benches/benchmarks.rs`](./benches/benchmarks.rs) with a matching label. + +3. Register the function in the `criterion_group!` macro at the bottom of + `benches/benchmarks.rs`. + +4. Run `cargo test --test latency_budget_test` to verify the new budget is + picked up by the tests. + +5. Run `cargo bench -- cli_cold_start cli_command_latency` to establish a + baseline for the new path. + +--- + +## Security notes + +- Latency budgets are **purely local measurements**. No measurement data is + transmitted over the network. +- Environment variable overrides are scoped to the current process; they are + never persisted or saved to config files. +- Budget definitions are compiled into the binary. There is no external + configuration file that could be tampered with to alter performance guarantees. + +--- + +## Migration notes + +If you are upgrading from a version without latency budgets (before this +feature was introduced), the new CI workflow will start checking budgets +automatically. To avoid breakage on a slow CI runner: + +1. Set the env var overrides listed in the workflow file to generous values. +2. Run one CI cycle, inspect the generated report, then tighten budgets. + +--- + +## Compatibility + +- **Platforms**: Linux (CI), macOS, and Windows are all supported. +- **Rust version**: Rust 1.70+ (stable). +- **No external services** required. Budget checks work fully offline. diff --git a/PR_DESCRIPTION.md b/PR_DESCRIPTION.md index b9405dd5..4a079e13 100644 --- a/PR_DESCRIPTION.md +++ b/PR_DESCRIPTION.md @@ -1,77 +1,54 @@ -## Summary - -This PR implements four AI-driven features for StarForge to enhance security, deployment optimization, multi-network support, and automation capabilities. - -### Implemented Features - -**#533 AI Template Security Scanning** -- Added comprehensive security scanning for Soroban contract templates -- Detects vulnerabilities: reentrancy, missing authorization, integer overflow, code injection, access control, cryptographic issues, data leakage -- Malicious code detection with pattern matching -- Security anti-pattern detection (unwrap usage, expect usage) -- Security scoring (0-100) and risk level assessment -- Fix suggestions with code examples and implementation effort estimates -- Continuous monitoring configuration -- CLI command: `starforge template-security scan ` - -**#543 AI Deployment Optimization** -- AI-driven optimization for deployment processes -- Cost optimization: gas cost estimation and savings analysis -- Speed optimization: deployment time estimation and improvement -- Reliability optimization: retry logic and pre-deployment validation suggestions -- Resource utilization analysis (CPU, memory, network, storage) -- Network selection recommendations with cost/speed/reliability scores -- Batch optimization for multiple deployments -- Scheduling optimization with optimal deployment times -- CLI command: `starforge deployment-optimize analyze --wasm ` - -**#550 AI Deployment Multi-Network Support** -- AI-driven multi-network deployment support (testnet, mainnet, custom networks) -- Network-specific configuration management -- Cross-network deployment with parallel/sequential/testnet-first strategies -- Network comparison with cost, speed, and reliability metrics -- Cost optimization across networks -- Risk assessment before deployment -- Synchronization status tracking across networks -- CLI commands: `starforge multi-network deploy`, `compare`, `add-network`, `list-networks`, `switch` - -**#540 AI Deployment Automation** -- AI-driven automation for deployment processes -- Pre-deployment validation: WASM file checks, size validation, network connectivity, wallet balance -- Automated testing with coverage reporting -- Deployment execution with gas estimation -- Post-deployment verification: contract inspection, storage verification -- Automated rollback on failure -- Monitoring setup with event monitoring and alert thresholds -- Complete automation pipeline with configurable levels (basic, standard, full) -- CLI command: `starforge deployment-automate run --wasm ` - -### Code Changes - -- Created `src/utils/template_security_scanner.rs` - Security scanning engine -- Created `src/commands/template_security.rs` - CLI commands for security scanning -- Created `src/utils/deployment_optimizer.rs` - Deployment optimization engine -- Created `src/commands/deployment_optimize.rs` - CLI commands for optimization -- Created `src/utils/multi_network_deploy.rs` - Multi-network deployment engine -- Created `src/commands/multi_network.rs` - CLI commands for multi-network -- Created `src/utils/deployment_automation.rs` - Deployment automation engine -- Created `src/commands/deployment_automate.rs` - CLI commands for automation -- Updated `src/commands/mod.rs` - Added new command modules -- Updated `src/utils/mod.rs` - Added new utility modules -- Updated `src/main.rs` - Integrated new CLI commands - -## Test plan - -- [ ] `cargo test` - Run all unit tests -- [ ] `starforge template-security scan ` -- [ ] `starforge deployment-optimize analyze --wasm ` -- [ ] `starforge multi-network deploy --wasm --networks testnet,mainnet` -- [ ] `starforge multi-network compare` -- [ ] `starforge deployment-automate run --wasm --network testnet` - -## Related Issues - -close #533 AI Template Security Scanning -close #543 AI Deployment Optimization -close #550 AI Deployment Multi-Network Support -close #540 AI Deployment Automation +## Enforce CLI Startup and Command Latency Budgets (Issue #698) + +### Summary + +Adds a comprehensive latency budget enforcement system for the StarForge CLI. Tracks representative cold-start and command paths via Criterion benchmarks and flags statistically meaningful regressions in CI. + +### Changes + +**Core Engine** — `src/utils/latency_budget.rs` (new) +- `LatencyBudget` / `LatencyBudgets` — per-command threshold definitions with defaults for 13 CLI paths +- `check_measurement()` — checks a single benchmark measurement against its budget (Pass/Fail/Noisy/Skipped/Error) +- `check_latency_budget()` — runs a full budget check across all active budgets +- `is_significant_regression()` — CV-based z-score statistical regression detection +- `budget_report_to_json()` / `print_budget_summary()` — JSON and human-readable output +- Environment variable overrides via `STARFORGE_LATENCY_BUDGET_