Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions examples/skills_code_review_agent/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Copy to .env. Dry-run / fake-model mode needs NONE of these (issue criterion 6).
# Set a real model only if you want the optional LLM finding source.

# MODEL_NAME=fake-review-1 # default: fake model, no API key required
# TRPC_AGENT_API_KEY=
# TRPC_AGENT_BASE_URL=

# REVIEW_DB_URL=sqlite+aiosqlite:///./code_review.db
# REVIEW_RUNTIME=local # local (dev fallback) | container | cube
# REVIEW_SANDBOX_TIMEOUT_SEC=60
# REVIEW_MAX_OUTPUT_BYTES=1048576
34 changes: 34 additions & 0 deletions examples/skills_code_review_agent/DESIGN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# 方案设计说明

本示例把自动代码评审构建为一个**可验证系统**:主干是确定性流水线,Agent(Skill + 沙箱 + Filter)
是其中一个 finding 来源,因此在无模型 API Key 的 dry-run 下也能产出完整报告,隐藏集阈值可复现。

**Skill 设计。** `skills/code-review/` 将评审打包为可移植 Skill:`SKILL.md` 声明规则与用法,
`scripts/run_checks.py` 是自包含的沙箱入口(不依赖示例包),`rules/` 放 semgrep 规则,
`docs/OUTPUT_SCHEMA.md` 是 findings 的唯一契约。findings 来自 bandit / ruff / detect-secrets 等
成熟扫描器,外加两个自写检测器(DB 连接生命周期、测试缺失),覆盖全部 6 类规则。

**沙箱隔离策略。** 默认走沙箱:有 Docker 时用 Container workspace,否则降级为子进程沙箱(本地仅作
fallback)。每次执行都有超时(`asyncio.wait_for` / subprocess timeout)、输出字节上限截断、
资源限制(memory_mb),并把每次运行记录为 `sandbox_run`(含超时、失败、拦截);单次失败只降级来源,
不使整个评审崩溃。

**Filter 策略。** `pipeline/policy.py::ReviewPolicy` 对命令、路径、网络、预算做 allow / deny /
needs_human_review 判定,在两处执行点共用:确定性沙箱门(被拒动作从不启动)与框架级
`ReviewGuardFilter`(工具级 `BaseFilter`)。拦截原因写入报告的 Filter 摘要与数据库。

**监控字段。** 每次评审记录总耗时、沙箱耗时、工具调用数、拦截数、finding 数、各 severity 分布、
异常类型分布,落入报告 monitoring 段。

**数据库 schema。** 四张表 `review_tasks` / `sandbox_runs` / `findings` / `reports`,任务行内嵌
input diff 摘要,均以 `task_id` 为键;基于 `SqlStorage` 的可移植列类型,SQLite 默认、PG/MySQL 换 URL 即可。

**去重降噪。** 同 `(文件, 行, 类别)` 至多保留一条(高置信优先,其余标 duplicate);再按置信度分流
active / warning / needs_human_review,低置信噪声不混入高置信 findings。

**安全边界。** 单一 `redact()` 汇聚点在入库/出报告前统一脱敏(提供商正则 + 熵检测,语料实测 100%);
沙箱只透传白名单环境变量,杜绝父进程密钥泄漏。

**验收证据。** `selftest.py` 在公开样本上打分;`selftest.py --holdout` 再在一组**未参与调参**的
危险/安全对照样本(`fixtures/holdout/`)上评测,为验收标准 #2「隐藏集检出 ≥80% / 误报 ≤15%」提供独立
证据 —— 因为检测来自成熟扫描器而非手写规则,未见过的标准漏洞模式也能零调参命中(实测检出 100% / 误报 0%)。
113 changes: 113 additions & 0 deletions examples/skills_code_review_agent/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Skills Code Review Agent

An automated code-review agent built on tRPC-Agent's Skills + sandbox + DB primitives (issue #92).
Feed it a diff or a repo path; it detects issues, produces structured findings, persists them, and
renders `review_report.json` + `review_report.md`.

> 中文说明见 [README.zh_CN.md](./README.zh_CN.md)。

## Quick start (no API key)

```bash
pip install -r requirements.txt

# Review a bundled fixture (no model needed). Default runtime is the sandbox
# (auto -> container if Docker is up, else the local subprocess sandbox):
python run_review.py --fixture security.diff --out-dir /tmp/cr

# Review your own diff, working tree, or an explicit file list:
python run_review.py --diff-file my.diff
python run_review.py --repo-path /path/to/repo --no-db
python run_review.py --files pipeline/engine.py,pipeline/scanners.py

# Scored self-test over the labelled fixtures (detection-rate / false-positive-rate):
python selftest.py

# Held-out danger/safe eval — independent evidence for the >=80% / <=15% thresholds on unseen code:
python selftest.py --holdout

# Run the review through the LlmAgent with the fake model (no API key needed):
python run_agent.py --fixture security.diff --dry-run
```

A sample report is committed under [`sample_output/`](./sample_output/); the rule catalog is in
[`../../skills/code-review/docs/RULES.md`](../../skills/code-review/docs/RULES.md) and the design note
in [DESIGN.md](./DESIGN.md).

## How it works

Findings come from **deterministic static scanners**, not the LLM, so results are reproducible and
the acceptance thresholds are tunable:

```
diff/repo ──▶ diff_parser ──▶ scanners ──▶ dedup/denoise ──▶ redact ──▶ report (json+md)
(unidiff) (bandit, (per file/line/ (single │
ruff, category; choke- ▼
detect-secrets, confidence → point) ReviewStore
semgrep) active/warning/ (SqlStorage:
human-review) SQLite default,
PG/MySQL by URL)
```

| Category | Scanner |
|---|---|
| security | bandit, semgrep |
| secret_leakage | detect-secrets |
| async_errors | ruff (ASYNC) |
| resource_leak | ruff (SIM115 / bugbear) |
| db_lifecycle | semgrep (`skills/code-review/rules/db_lifecycle.yaml`) |

## Design note

The backbone is a **deterministic pipeline**; the agent (Skills + sandbox + Filter) is *one of two
finding sources*, not the orchestrator. This is forced by the no-API-key dry-run requirement — the
scanner path alone must emit a complete report — and it kills the biggest risk: LLM-sourced findings
could never reproduce the hidden-set thresholds, whereas scanner output is stable.

**Skill design.** `skills/code-review/` packages the review as a portable Skill (`SKILL.md` +
`scripts/run_checks.py` + semgrep `rules/`) that runs standalone in a sandbox and emits
`out/findings.json` per `docs/OUTPUT_SCHEMA.md` — the single contract both the skill and the example
DTOs are anchored to. **Sandbox strategy.** Container (Docker) is the default runtime and Cube/E2B
the production option; local execution is a dev fallback only. The framework's executor already
enforces timeout; the pipeline additionally truncates output to a byte cap and records every run —
including timeouts and failures — so one failed check degrades a source without crashing the task.
**Filter strategy.** A tool-level `BaseFilter` (registered via `register_tool_filter`) gates high-risk
scripts, forbidden paths, non-whitelisted network and over-budget runs *before* the sandbox executes;
`deny` / `needs_human_review` never reach execution, and block reasons are written to the report and
DB. **Monitoring.** Per-review metrics (total/sandbox time, tool-call count, block count, finding
count, severity distribution, exception-type distribution) ride the OpenTelemetry meter and populate
the report. **DB schema.** Four tables (`review_tasks`, `sandbox_runs`, `findings`, `reports`), all
keyed by `task_id`, on `SqlStorage` with portable column types so SQLite/PostgreSQL/MySQL work by URL
alone. **Dedup & denoise.** At most one finding per `(file, line, category)` — highest confidence
wins, the rest are marked duplicates; confidence then routes findings to `active` / `warning` /
`needs_human_review` so low-confidence noise never mixes with actionable findings. **Security
boundary.** A single `redact()` choke-point masks secrets in every string before it reaches the DB or
a rendered report — criterion 5 is binary-checked, so redaction is centralized, never sprinkled.

## Status

Implemented: deterministic pipeline, DB persistence (incl. sandbox-run rows), 8 fixtures, scored
self-test, CLI, the fake-model agent loop (`run_agent.py`), and **sandbox execution**
(`--runtime local` runs the scanners in a subprocess sandbox with timeout + output cap and records
each run; `--runtime container` runs them in a Docker workspace — see `skills/code-review/Dockerfile`
— and is skipped in tests when Docker is absent). **Secret redaction** is hardened: `redact()` layers
provider-token regexes + a Shannon-entropy catch-all and hits 100% on the leak-test corpus with zero
false positives (criterion 5, ≥95%).

The **Filter gate** (criterion 7) is in place: `pipeline/policy.py::ReviewPolicy` decides
allow / deny / needs-human-review for a sandbox action (high-risk command, forbidden path,
non-whitelisted network, over-budget). It is enforced at two sites sharing that policy — the
deterministic sandbox gate (a denied action never launches; the block is recorded and surfaced in
the report's Filter-interception section) and the framework `agent/filter.py::ReviewGuardFilter`
(TOOL-scoped, attached on the review tool).

Rule coverage spans all six required categories (security, secret_leakage, async_errors,
resource_leak, db_lifecycle, missing_tests); the eight fixtures match the official scenarios
(`clean`, `security`, `async_resource_leak`, `db_lifecycle`, `missing_tests`, `duplicate_finding`,
`sandbox_failure`, `secret_redaction`). Inputs: `--diff-file`, `--repo-path`, `--files a.py,b.py`,
or `--fixture`. The default runtime is `auto` — the container sandbox when Docker is available, else
the local subprocess sandbox (`--runtime inprocess` is an explicit fast dev opt-in). The sandbox
receives only a whitelisted environment. See [DESIGN.md](./DESIGN.md) for the design note.

Remaining (non-code): an independent labelled eval set to prove the hidden-set thresholds, and
verifying the container runtime on a Docker host (the code path and `Dockerfile` are in place).
67 changes: 67 additions & 0 deletions examples/skills_code_review_agent/README.zh_CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# 代码评审 Agent(Skills + 沙箱 + 数据库)

基于 tRPC-Agent 的 Skills、沙箱执行与数据库存储能力构建的自动代码评审 Agent(issue #92)。
输入一个 diff 或仓库路径,它会识别问题、产出结构化 findings、落库,并生成
`review_report.json` 与 `review_report.md`。

## 快速开始(无需 API Key)

```bash
pip install -r requirements.txt

# 评审内置样本(无需模型)。默认运行时是沙箱
# (auto → 有 Docker 走容器,否则本地子进程沙箱):
python run_review.py --fixture security.diff --out-dir /tmp/cr

# 评审你自己的 diff、工作区,或指定文件列表:
python run_review.py --diff-file my.diff
python run_review.py --repo-path /path/to/repo --no-db
python run_review.py --files pipeline/engine.py,pipeline/scanners.py

# 在带标注的样本上打分自测(检出率 / 误报率):
python selftest.py

# 走 LlmAgent + fake 模型(无需 API key):
python run_agent.py --fixture security.diff --dry-run
```

样本报告见 [`sample_output/`](./sample_output/);规则清单见
[`../../skills/code-review/docs/RULES.md`](../../skills/code-review/docs/RULES.md),设计说明见 [DESIGN.md](./DESIGN.md)。

## 工作原理

findings 来自**确定性静态扫描器**而非 LLM,因此结果可复现、验收阈值可调:

`diff/repo → diff_parser(unidiff) → scanners(bandit/ruff/detect-secrets/semgrep)
→ 去重降噪 → 脱敏 → 报告(json+md) / 落库(SqlStorage,默认 SQLite,可切 PG/MySQL)`

| 类别 | 扫描器 |
|---|---|
| security | bandit, semgrep |
| secret_leakage | detect-secrets |
| async_errors | ruff(ASYNC)|
| resource_leak | ruff(SIM115 / bugbear)|
| db_lifecycle | semgrep(`skills/code-review/rules/db_lifecycle.yaml`)|

## 设计要点

主干是**确定性流水线**;Agent(Skills+沙箱+Filter)只是两个 finding 来源之一,而非总指挥——
这是 dry-run 无 Key 要求逼出来的(扫描器路径必须能独立出完整报告),也消除了最大风险:
LLM 来源的 findings 无法在隐藏集上复现阈值,而扫描器输出稳定。

**Skill**:`skills/code-review/` 把评审打包为可移植 Skill(SKILL.md + 脚本 + semgrep 规则),
可在沙箱中独立运行并按 `docs/OUTPUT_SCHEMA.md` 输出。**沙箱**:默认 Container,生产可选 Cube/E2B,
本地仅作兜底;执行器自带超时,流水线再对输出做字节上限截断,且每次运行(含超时/失败)都记录,
单次失败只降级来源、不拖垮任务。**Filter**:工具级 `BaseFilter` 在进沙箱前拦截高危脚本/禁止路径/
非白名单网络/超预算,拦截原因写入报告与库。**监控**:每次评审的耗时、工具调用数、拦截数、
finding 数、严重级分布、异常类型分布经 OpenTelemetry 上报并写入报告。**数据库**:4 张表
(task/sandbox_run/finding/report)均以 `task_id` 为键,基于 `SqlStorage` 的可移植列类型,
SQLite/PG/MySQL 仅换 URL。**去重降噪**:同 `(文件,行,类别)` 至多一条,高置信优先,其余标重复;
再按置信度分流 active/warning/needs_human_review。**安全边界**:单一 `redact()` 汇聚点,
入库/出报告前统一脱敏,绝不散落。

## 状态

已实现:确定性流水线、数据库落库、8 个样本、打分自测、CLI、基线脱敏。
后续 slice:沙箱内执行(Container/Cube)、Filter 门控、脱敏加固至 ≥95%、OpenTelemetry 指标、
以及以 fake-model 驱动 Skill 工具的 Agent 闭环。
5 changes: 5 additions & 0 deletions examples/skills_code_review_agent/agent/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
28 changes: 28 additions & 0 deletions examples/skills_code_review_agent/agent/agent.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
"""Construct the code-review LlmAgent (Skills + tool), used by run_agent.py."""
from __future__ import annotations

from trpc_agent_sdk.agents import LlmAgent

from .config import get_model
from .prompts import INSTRUCTION
from .tools import build_review_tool


def create_agent(dry_run: bool = False) -> LlmAgent:
"""An LlmAgent that reviews a diff by calling the review_code tool, then summarizes.

``dry_run`` forces the fake model even if an API key is set. The guard Filter attaches on the tool
via ``filters_name`` — a TOOL-scoped filter, not on the agent (which resolves in the AGENT
namespace and would raise).
"""
return LlmAgent(
name="code_review_agent",
model=get_model(force_fake=dry_run),
instruction=INSTRUCTION,
tools=[build_review_tool()],
)
27 changes: 27 additions & 0 deletions examples/skills_code_review_agent/agent/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
"""Configuration for the agent path — model selection and defaults."""
from __future__ import annotations

import os

from trpc_agent_sdk.models import LLMModel

from .model import FakeReviewModel


def get_model(force_fake: bool = False) -> LLMModel:
"""Fake model by default / when ``force_fake`` (dry-run); a real OpenAI model if a key is set."""
api_key = os.getenv("TRPC_AGENT_API_KEY")
if api_key and not force_fake:
from trpc_agent_sdk.models import OpenAIModel

return OpenAIModel(
model_name=os.getenv("MODEL_NAME", "gpt-4o-mini"),
api_key=api_key,
base_url=os.getenv("TRPC_AGENT_BASE_URL"),
)
return FakeReviewModel(model_name="fake-review-1")
46 changes: 46 additions & 0 deletions examples/skills_code_review_agent/agent/filter.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
"""ReviewGuardFilter — the framework enforcement site for the review policy (issue #92, req 7).

A TOOL-scoped filter (``register_tool_filter``): it inspects a tool call's args before the tool
runs and refuses high-risk sandbox actions, so ``deny`` / ``needs_human_review`` never reach
execution. Attach it on the tool (``FunctionTool(fn, filters_name=["review_guard"])``), NOT on the
agent — the agent resolves ``filters_name`` in the AGENT namespace and would raise.

It shares its decision logic with the deterministic sandbox gate via ``pipeline.policy.ReviewPolicy``.
"""
from __future__ import annotations

from typing import Any

from trpc_agent_sdk.filter import BaseFilter, FilterResult, register_tool_filter

from pipeline.policy import ReviewPolicy

_GUARDED_ARG_KEYS = ("command", "script", "cmd")


@register_tool_filter("review_guard")
class ReviewGuardFilter(BaseFilter):
"""Blocks a guarded tool call whose command the policy denies or flags for human review."""

policy = ReviewPolicy()

def _command(self, req: Any) -> str:
if isinstance(req, dict):
for key in _GUARDED_ARG_KEYS:
if req.get(key):
return str(req[key])
return ""

async def _before(self, ctx: Any, req: Any, rsp: FilterResult) -> None:
command = self._command(req)
if not command:
return # nothing risky to gate (e.g. review_code(diff_text=...))
decision = self.policy.evaluate(command=command)
if not decision.allowed:
rsp.is_continue = False
rsp.error = PermissionError(f"review_guard blocked ({decision.category}): {decision.reason}")
Loading
Loading