Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,13 @@
## Agent workflow

- Follow `skills/xprobe-measure-latency/SKILL.md` for measurement tasks.
- For an unknown workload, progress from readiness and baseline through a broad,
representative bounded inventory, evidence-based selector narrowing,
validation, and one detailed bounded measurement per stated hypothesis. Scope
breadth and capture duration are independent. Do not guess selectors.
- Classify a workload as CPU-only or GPU/mixed before choosing collectors.
`discover` is only for CUDA context holders; CPU-only work proceeds with its
selected PID and host selectors. For GPU/mixed work, progress from readiness
and baseline through a broad, representative bounded inventory,
evidence-based selector narrowing, validation, and one detailed bounded
measurement per stated hypothesis. Scope breadth and capture duration are
independent. Do not guess selectors.
- Read evidence pairs, artifact analysis, stream identity, collection quality,
and profiler overhead before interpreting latency. Summed concurrent GPU
duration is not wall time.
Expand Down
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
cmake_minimum_required(VERSION 3.20)

project(xprobe VERSION 0.3.2 LANGUAGES C)
project(xprobe VERSION 0.3.3 LANGUAGES C)

include(CTest)

Expand Down
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ members = [
resolver = "2"

[workspace.package]
version = "0.3.2"
version = "0.3.3"
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/itdevwu/xprobe"
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ only installation action required from the user:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.3/skills/xprobe-measure-latency \
--global
```

Expand Down Expand Up @@ -83,7 +83,7 @@ for safe reactivation.

## Support

| Surface | 0.3.2 support |
| Surface | 0.3.3 support |
| --- | --- |
| OS/architecture | Linux x86_64, glibc 2.34 or newer |
| Host events | ELF function entry/return through PID-scoped uprobes |
Expand Down
2 changes: 1 addition & 1 deletion docs/agent-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ repairs the matching xprobe CLI itself:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.3/skills/xprobe-measure-latency \
--global
```

Expand Down
2 changes: 1 addition & 1 deletion docs/cli-contract.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# CLI contract

xprobe 0.3.2 exposes exactly four public commands: `doctor`, `discover`,
xprobe 0.3.3 exposes exactly four public commands: `doctor`, `discover`,
`validate`, and `measure`.

## Common behavior
Expand Down
12 changes: 6 additions & 6 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ user needs to run:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.3/skills/xprobe-measure-latency \
--global
```

Expand All @@ -28,7 +28,7 @@ The versioned bootstrap installs to `~/.local` without root access:

```bash
curl --proto '=https' --tlsv1.2 -fsSL \
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.2/install.sh | sh
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.3/install.sh | sh
```

The bootstrap downloads the release archive and its SHA256 file, verifies the
Expand All @@ -47,7 +47,7 @@ prefix, download the script and pass `--prefix`:

```bash
curl --proto '=https' --tlsv1.2 -fsSLO \
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.2/install.sh
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.3/install.sh
sh install.sh --prefix /opt/xprobe
```

Expand All @@ -59,7 +59,7 @@ script with `sudo`. The installer never elevates privileges itself.
For a fully explicit archive workflow:

```bash
version=0.3.2
version=0.3.3
base=https://github.com/itdevwu/xprobe/releases/download/v$version
archive=xprobe-$version-linux-x86_64.tar.gz

Expand Down Expand Up @@ -92,7 +92,7 @@ missing or damaged, manually install the complete version-matched directory:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.3/skills/xprobe-measure-latency \
--global
```

Expand All @@ -101,7 +101,7 @@ installation names the target explicitly:

```bash
npx --yes skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.3/skills/xprobe-measure-latency \
--agent codex --global --copy --yes
```

Expand Down
4 changes: 2 additions & 2 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -eu

repository=${XPROBE_REPOSITORY:-itdevwu/xprobe}
version=${XPROBE_VERSION:-0.3.2}
version=${XPROBE_VERSION:-0.3.3}
if [ -n "${XPROBE_PREFIX:-}" ]; then
prefix=$XPROBE_PREFIX
elif [ -n "${HOME:-}" ]; then
Expand All @@ -19,7 +19,7 @@ Install a released xprobe binary and its CUDA Agents.
Usage: install.sh [--version VERSION] [--prefix DIR] [--uninstall]

Options:
--version VERSION Release to install (default: 0.3.2)
--version VERSION Release to install (default: 0.3.3)
--prefix DIR Installation prefix (default: $HOME/.local)
--uninstall Remove xprobe from the selected prefix
-h, --help Show this help
Expand Down
39 changes: 23 additions & 16 deletions skills/xprobe-measure-latency/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,30 +15,37 @@ CLI and selector syntax is in [references/cli-contract.md](references/cli-contra

## Workflow

1. Run `xprobe --version`. When the command is absent or not 0.3.2, read
1. Run `xprobe --version`. When the command is absent or not 0.3.3, read
[references/setup.md](references/setup.md) and install or repair the CLI
yourself; do not ask the user to perform a separate CLI installation.
Confirm every JSON response has `schema_version: "2.0"`; do not assume a
pre-0.3 or future protocol is compatible with this Skill.
2. Establish an application-level latency baseline. Wait for process readiness,
CUDA context creation, JIT compilation, and warmup before discovery. Keep a
repeatable request or batch trigger ready for the measurement window.
2. Establish an application-level latency baseline, process readiness, and
warmup. Classify the workload before selecting collectors: for CPU-only work,
choose the owning PID and skip CUDA discovery; for GPU or mixed work, wait
for CUDA context creation and JIT warmup before discovery. Keep a repeatable
request or batch trigger ready for the measurement window.
3. Run `xprobe doctor --json --non-interactive --no-color`. Check individual
capabilities; `ok: true` only means diagnosis completed.
4. Run `xprobe discover --pid ROOT_PID --limit 200 --json --non-interactive
--no-color`. It returns NVML-confirmed CUDA context holders under that process
tree. Choose a worker from workload, PID/start-time, command line, and GPU
UUID evidence. Measure workers separately when several ranks are relevant.
5. Map the workload before choosing a name. Validate broad kernel, memcpy, or
memset activity endpoints, then collect one bounded, representative coarse
inventory per event family with `--events-out`. Scope breadth and collection
duration are independent: keep the selector broad, but choose a duration that
4. For GPU or mixed work, run `xprobe discover --pid ROOT_PID --limit 200 --json
--non-interactive --no-color`. It returns NVML-confirmed CUDA context holders
under that process tree. Choose a worker from workload, PID/start-time,
command line, and GPU UUID evidence. Measure workers separately when several
ranks are relevant. For CPU-only work, do not run `discover`; continue with
the selected process PID.
5. Map GPU or mixed work before choosing a name. Validate broad kernel, memcpy,
or memset activity endpoints, then collect one bounded, representative coarse
inventory per event family with `--events-out`. For CPU-only work, resolve and
validate the intended host-function boundary directly; do not require CUDA or
CUPTI. Scope breadth and collection duration are independent: keep the
selector broad where an activity inventory exists, but choose a duration that
covers the workload cycle being diagnosed. Give `--max-events` headroom.
6. Run `scripts/analyze_trace.py` on each coarse artifact. Use kernel names,
selector hints, duration aggregates, launch variants, stream distribution,
busy union, overlap factor, and adjacent gaps to form one narrow hypothesis. Read
6. For GPU artifacts, run `scripts/analyze_trace.py`. Use kernel names, selector
hints, duration aggregates, launch variants, stream distribution, busy union,
overlap factor, and adjacent gaps to form one narrow hypothesis. Read
[references/trace-analysis.md](references/trace-analysis.md) when interpreting
the report.
the report. For CPU-only work, use resolved host selectors and result evidence
to form the hypothesis instead.
7. Run `xprobe validate --pid WORKER_PID --from SELECTOR --to SELECTOR --match
POLICY --json --non-interactive --no-color`. Stop when `valid` is false. If
`agent_activation` is `injection_required`, disclose that `measure` will
Expand Down
4 changes: 2 additions & 2 deletions skills/xprobe-measure-latency/agents/openai.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
interface:
display_name: "Xprobe Latency Measurement"
short_description: "Map broad CPU and GPU latency evidence"
default_prompt: "Use $xprobe-measure-latency to map this workload broadly, then measure the suspected latency boundary precisely."
short_description: "Route CPU and GPU latency investigations"
default_prompt: "Use $xprobe-measure-latency to classify this workload as CPU-only or GPU/mixed, then measure the suspected latency boundary."
45 changes: 40 additions & 5 deletions skills/xprobe-measure-latency/references/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ command unless their environment prevents the agent from writing a usable prefix

## Check and bootstrap xprobe

Check the executable first. Continue only when it reports 0.3.2; otherwise run
Check the executable first. Continue only when it reports 0.3.3; otherwise run
the bootstrap below:

```bash
Expand All @@ -21,7 +21,7 @@ installing under `~/.local`:

```bash
curl --proto '=https' --tlsv1.2 -fsSL \
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.2/install.sh \
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.3/install.sh \
-o /tmp/xprobe-install.sh
sh /tmp/xprobe-install.sh
export PATH="$HOME/.local/bin:$PATH"
Expand All @@ -36,23 +36,58 @@ PATH, permission, driver, CUDA, or CUPTI failures explicitly and adjust from the
reported detail; do not continue to measurement on an unverified installation.
The CLI needs no Node.js. CUDA is optional until a GPU selector is measured.

## Build locally when the release is unsuitable

The `glibc 2.34` requirement applies to the precompiled release archive. When
that archive cannot run on the host, build the matching source locally with the
host glibc instead. This is a local-use fallback, not permission to weaken the
release package's `GLIBC_2.34` ceiling.

```bash
git clone --depth 1 --branch v0.3.3 https://github.com/itdevwu/xprobe.git
cd xprobe
mamba env create --file environment.yml
mamba run -n xprobe-dev just build
export PATH="$PWD/target/debug:$PATH"
xprobe --version
xprobe doctor --json --non-interactive --no-color
```

For CPU-only work, this build is sufficient; no CUDA toolkit or CUPTI Agent is
required. Do not use `scripts/package-release.sh` for this path because it
enforces the distributable archive compatibility policy.

For GPU or mixed work, build one local Agent against the target's CUDA 12 or
CUDA 13 toolkit, then point the CLI at it:

```bash
mamba run -n xprobe-dev env CUDA_PATH=/opt/cuda-12 cmake -S . -B build/cuda12 -G Ninja \
-DXPROBE_BUILD_BPF=ON -DXPROBE_REQUIRE_CUPTI=ON -DXPROBE_CUDA_MAJOR=12
mamba run -n xprobe-dev cmake --build build/cuda12 --target xprobe-bpf xprobe-cupti
export XPROBE_CUPTI_AGENT_PATH="$PWD/build/cuda12/cupti/libxprobe-cupti.so"
```

Replace `12` and the toolkit path with `13` for CUDA 13. CUDA/CUPTI majors other
than 12 or 13 are not supported: do not force an Agent build or bypass the
version check.

## Repair the Skill only when needed

The user normally installed this Skill before invoking the agent. When its files
are missing or the version is not 0.3.2, install the complete version-matched
are missing or the version is not 0.3.3, install the complete version-matched
directory through the Agent Skills CLI:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.3/skills/xprobe-measure-latency \
--global
```

For non-interactive automation, select the host explicitly:

```bash
npx --yes skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.3/skills/xprobe-measure-latency \
--agent codex --global --copy --yes
```

Expand Down
11 changes: 9 additions & 2 deletions tests/agent-contract/test_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ def check_skill(workspace: pathlib.Path) -> None:
assert quality_field in normalized_skill
for investigation_step in (
"application-level latency baseline",
"CPU-only",
"GPU or mixed",
"do not run `discover`",
"representative coarse inventory",
"Scope breadth and collection duration are independent",
"scripts/analyze_trace.py",
Expand Down Expand Up @@ -112,6 +115,7 @@ def check_skill(workspace: pathlib.Path) -> None:
normalized_investigation = re.sub(r"\s+", " ", investigation)
normalized_quality = re.sub(r"\s+", " ", quality)
normalized_trace_analysis = re.sub(r"\s+", " ", trace_analysis)
normalized_setup = re.sub(r"\s+", " ", setup)
for required in (
"Triton",
"procfs start time",
Expand Down Expand Up @@ -139,12 +143,15 @@ def check_skill(workspace: pathlib.Path) -> None:
):
assert required in normalized_trace_analysis
for required in (
"v0.3.2/install.sh",
"v0.3.3/install.sh",
"npx skills@1 add",
"xprobe --version",
"xprobe doctor",
"host glibc",
"CUDA/CUPTI majors other than 12 or 13 are not supported",
"scripts/package-release.sh",
):
assert required in setup
assert required in normalized_setup

engineering_rules = re.sub(
r"\s+", " ", (workspace / "AGENTS.md").read_text()
Expand Down
Loading