Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,10 @@
## Agent workflow

- Follow `skills/xprobe-measure-latency/SKILL.md` for measurement tasks.
- For an unknown workload, progress from readiness and baseline through a short
survey artifact, evidence-based selector narrowing, validation, and one
bounded measurement per stated hypothesis. Do not guess selectors.
- For an unknown workload, progress from readiness and baseline through a broad,
representative bounded inventory, evidence-based selector narrowing,
validation, and one detailed bounded measurement per stated hypothesis. Scope
breadth and capture duration are independent. Do not guess selectors.
- Read evidence pairs, artifact analysis, stream identity, collection quality,
and profiler overhead before interpreting latency. Summed concurrent GPU
duration is not wall time.
Expand Down
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
cmake_minimum_required(VERSION 3.20)

project(xprobe VERSION 0.3.1 LANGUAGES C)
project(xprobe VERSION 0.3.2 LANGUAGES C)

include(CTest)

Expand Down
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ members = [
resolver = "2"

[workspace.package]
version = "0.3.1"
version = "0.3.2"
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/itdevwu/xprobe"
Expand Down
30 changes: 10 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,33 +14,23 @@ in a process, on the CPU, NVIDIA GPU, or across both. Its bounded native profile
combines eBPF uprobes and NVIDIA CUPTI with an agent-friendly CLI, strict JSON
contracts, explicit correlation quality, and no daemon or server lifecycle.

## Install xprobe
## Install

```bash
curl --proto '=https' --tlsv1.2 -fsSL \
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.1/install.sh | sh
```

This installs the released CLI and CUDA 12/13 Agents under `~/.local`. xprobe
supports Linux x86_64 with glibc 2.34 or newer. NVIDIA CUDA is optional unless
GPU events are selected. See [Installation](docs/installation.md) for checksum
verification, custom prefixes, upgrades, and removal.

## Install the Agent Skill

Install the version-matched Skill with the open Agent Skills CLI:
Install the version-matched Skill with the open Agent Skills CLI. This is the
only installation action required from the user:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.1/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
--global
```

The installer detects Codex, Claude Code, Cursor, and other compatible agents.
Node.js is only needed for this Skill installation, not for xprobe itself. Set
`DISABLE_TELEMETRY=1` when anonymous `skills` CLI telemetry is not wanted.
The Skill guides a short survey, derives selectors from trace evidence, and
includes a deterministic multi-stream JSONL analyzer.
When invoked, the Skill checks for the matching `xprobe` CLI and installs or
repairs it under a writable prefix before profiling. It can then diagnose and
adjust path, permission, NVIDIA, CUDA, or CUPTI problems from live evidence.
Node.js is only needed for Skill installation, not for xprobe itself. See
[Installation](docs/installation.md) for direct CLI use and archive verification.

## Measure

Expand Down Expand Up @@ -93,7 +83,7 @@ for safe reactivation.

## Support

| Surface | 0.3.1 support |
| Surface | 0.3.2 support |
| --- | --- |
| OS/architecture | Linux x86_64, glibc 2.34 or newer |
| Host events | ELF function entry/return through PID-scoped uprobes |
Expand Down
17 changes: 9 additions & 8 deletions docs/agent-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ sequences or correlation rules.
## Install the Skill

The canonical directory follows the open Agent Skills format and is shared by
Codex, Claude Code, Cursor, and other compatible clients. Install the released
version through the `skills` CLI so client-specific discovery paths remain the
installer's responsibility:
Codex, Claude Code, Cursor, and other compatible clients. The user installs the
released Skill through the `skills` CLI; the activated Skill then bootstraps or
repairs the matching xprobe CLI itself:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.1/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
--global
```

Expand All @@ -26,8 +26,9 @@ For automation, add `--agent codex|claude-code|cursor --copy --yes`. Omit
install its whole directory so its references, examples, and analysis script
remain available.

The Skill moves from a short survey artifact to evidence-based selector
narrowing and includes `scripts/analyze_trace.py` for deterministic kernel,
The Skill checks, installs, and verifies the CLI before it first runs `doctor`.
It then maps a representative workload broadly, uses artifact evidence to narrow
selectors, and includes `scripts/analyze_trace.py` for deterministic kernel,
copy, overlap, stream, and gap summaries. The xprobe repository tests
installation with `skills` CLI 1.5.20 in isolated
home directories. This pinned test protects released behavior while the
Expand All @@ -45,8 +46,8 @@ The test requires the visible command set to be exactly `doctor`, `discover`,
checks injection requirements, verifies schemas, exercises the bundled trace
analyzer, and checks that the Skill uses only the four-command bounded workflow
and inspects result quality/evidence.
The installation test uses the real third-party CLI with telemetry disabled and
verifies byte-for-byte copies for Codex, Claude Code, and Cursor.
The installation test uses the real third-party CLI in isolated home directories
and verifies byte-for-byte copies for Codex, Claude Code, and Cursor.

This is interface conformance, not model evaluation. External harnesses may
evaluate task success, command count, cleanup, mutation disclosure, and result
Expand Down
2 changes: 1 addition & 1 deletion docs/cli-contract.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# CLI contract

xprobe 0.3.1 exposes exactly four public commands: `doctor`, `discover`,
xprobe 0.3.2 exposes exactly four public commands: `doctor`, `discover`,
`validate`, and `measure`.

## Common behavior
Expand Down
38 changes: 27 additions & 11 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,30 @@ contains the CLI and separate CUDA 12 and CUDA 13 CUPTI Agents. A CUDA toolkit
is not required to install xprobe; NVIDIA driver and CUPTI availability matter
only when measuring GPU events.

## User installation
## Agent-managed installation

For agent use, install the version-matched Skill and let the agent bootstrap or
repair the CLI under a writable prefix. This is the only installation command a
user needs to run:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
--global
```

The Skill verifies `xprobe --version`, installs the matching release when needed,
then runs `doctor` before it profiles. It has the live context needed to adjust
PATH, prefix, permission, NVIDIA, CUDA, and CUPTI issues. Node.js is required
only for Skill installation.

## Direct CLI installation

The versioned bootstrap installs to `~/.local` without root access:

```bash
curl --proto '=https' --tlsv1.2 -fsSL \
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.1/install.sh | sh
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.2/install.sh | sh
```

The bootstrap downloads the release archive and its SHA256 file, verifies the
Expand All @@ -30,7 +47,7 @@ prefix, download the script and pass `--prefix`:

```bash
curl --proto '=https' --tlsv1.2 -fsSLO \
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.1/install.sh
https://raw.githubusercontent.com/itdevwu/xprobe/v0.3.2/install.sh
sh install.sh --prefix /opt/xprobe
```

Expand All @@ -42,7 +59,7 @@ script with `sudo`. The installer never elevates privileges itself.
For a fully explicit archive workflow:

```bash
version=0.3.1
version=0.3.2
base=https://github.com/itdevwu/xprobe/releases/download/v$version
archive=xprobe-$version-linux-x86_64.tar.gz

Expand All @@ -68,24 +85,23 @@ sh install.sh --prefix "$HOME/.local" --uninstall
The uninstall operation removes only xprobe-owned files below `bin/xprobe`,
`lib/xprobe`, `include/xprobe`, and `share/xprobe`.

## Agent Skill
## Manual Skill repair

The xprobe CLI works independently of an AI agent. To let an Agent discover,
validate, and run bounded measurements using the released contract, install the
version-matched Skill:
The xprobe CLI works independently of an AI agent. When a Skill installation is
missing or damaged, manually install the complete version-matched directory:

```bash
npx skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.1/skills/xprobe-measure-latency \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
--global
```

The command interactively selects among detected agents. A non-interactive
installation names the target explicitly:

```bash
DISABLE_TELEMETRY=1 npx --yes skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.1/skills/xprobe-measure-latency \
npx --yes skills@1 add \
https://github.com/itdevwu/xprobe/tree/v0.3.2/skills/xprobe-measure-latency \
--agent codex --global --copy --yes
```

Expand Down
4 changes: 2 additions & 2 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -eu

repository=${XPROBE_REPOSITORY:-itdevwu/xprobe}
version=${XPROBE_VERSION:-0.3.1}
version=${XPROBE_VERSION:-0.3.2}
if [ -n "${XPROBE_PREFIX:-}" ]; then
prefix=$XPROBE_PREFIX
elif [ -n "${HOME:-}" ]; then
Expand All @@ -19,7 +19,7 @@ Install a released xprobe binary and its CUDA Agents.
Usage: install.sh [--version VERSION] [--prefix DIR] [--uninstall]

Options:
--version VERSION Release to install (default: 0.3.1)
--version VERSION Release to install (default: 0.3.2)
--prefix DIR Installation prefix (default: $HOME/.local)
--uninstall Remove xprobe from the selected prefix
-h, --help Show this help
Expand Down
39 changes: 24 additions & 15 deletions skills/xprobe-measure-latency/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,21 @@ description: Investigate unknown Linux CPU/CUDA latency with bounded xprobe capt

# Investigate latency with xprobe

Use JSON mode and move from a short survey to a narrow measurement. Read
Use JSON mode to make a wide, coarse workload inventory before measuring one
boundary narrowly and finely. Read [references/setup.md](references/setup.md)
to install or repair the CLI and this Skill. Read
[references/investigation.md](references/investigation.md) before profiling an
unknown workload. Read [references/result-quality.md](references/result-quality.md)
before interpreting correlation, clocks, concurrency, or overhead. The exact
CLI and selector syntax is in [references/cli-contract.md](references/cli-contract.md).

## Workflow

1. Run `xprobe --version`. Require a version-matched xprobe 0.3.x CLI at 0.3.1
or later and confirm every JSON response has `schema_version: "2.0"`; do not
assume a pre-0.3 or future protocol is compatible with this Skill.
1. Run `xprobe --version`. When the command is absent or not 0.3.2, read
[references/setup.md](references/setup.md) and install or repair the CLI
yourself; do not ask the user to perform a separate CLI installation.
Confirm every JSON response has `schema_version: "2.0"`; do not assume a
pre-0.3 or future protocol is compatible with this Skill.
2. Establish an application-level latency baseline. Wait for process readiness,
CUDA context creation, JIT compilation, and warmup before discovery. Keep a
repeatable request or batch trigger ready for the measurement window.
Expand All @@ -25,13 +29,14 @@ CLI and selector syntax is in [references/cli-contract.md](references/cli-contra
--no-color`. It returns NVML-confirmed CUDA context holders under that process
tree. Choose a worker from workload, PID/start-time, command line, and GPU
UUID evidence. Measure workers separately when several ranks are relevant.
5. If selectors are unknown, validate broad activity endpoints, then make one
short duration-bounded survey with `--events-out survey.jsonl`. For a kernel
inventory, use `cuda:kernel_start` to `cuda:kernel_end` with `exact`; keep the
survey short and give `--max-events` headroom.
6. Run `scripts/analyze_trace.py survey.jsonl`. Use its kernel names, selector
hints, duration aggregates, launch variants, stream distribution, busy union,
overlap factor, and adjacent gaps to form one narrow hypothesis. Read
5. Map the workload before choosing a name. Validate broad kernel, memcpy, or
memset activity endpoints, then collect one bounded, representative coarse
inventory per event family with `--events-out`. Scope breadth and collection
duration are independent: keep the selector broad, but choose a duration that
covers the workload cycle being diagnosed. Give `--max-events` headroom.
6. Run `scripts/analyze_trace.py` on each coarse artifact. Use kernel names,
selector hints, duration aggregates, launch variants, stream distribution,
busy union, overlap factor, and adjacent gaps to form one narrow hypothesis. Read
[references/trace-analysis.md](references/trace-analysis.md) when interpreting
the report.
7. Run `xprobe validate --pid WORKER_PID --from SELECTOR --to SELECTOR --match
Expand All @@ -45,12 +50,15 @@ CLI and selector syntax is in [references/cli-contract.md](references/cli-contra
9. Check `status`, matched/unmatched/ambiguous/dropped counts, collection
completeness, buffer utilization, clock alignment, estimated error,
correlation method/confidence/score, warnings, and every evidence pair.
10. Repeat only with a stated reason: narrow the selector, select another worker
or stream, change an explicitly compatible policy, or test the next boundary.
10. Repeat only with a stated reason: select another event family, narrow the
selector, select another worker or stream, change an explicitly compatible
policy, or test the next boundary.
Recheck application latency after profiling and report observed overhead.

For completed captures, replace `--pid` with one or more `--input` arguments.
Use the [kernel duration](examples/kernel-duration.json),
Begin with the [coarse kernel inventory](examples/coarse-kernel-inventory.json)
or [coarse memcpy inventory](examples/coarse-memcpy-inventory.json), then use the
[kernel duration](examples/kernel-duration.json),
[same-stream gap](examples/same-stream-kernel-gap.json),
[host span](examples/host-function-span.json), and
[memcpy duration](examples/memcpy-duration.json) specs, plus the
Expand All @@ -69,4 +77,5 @@ hypothesis; orchestration remains the agent framework's responsibility.
duration cannot explain warp stalls, cache misses, occupancy, instruction mix,
or Tensor Core utilization; hand that question to NCU or PC sampling.
- Avoid continuous or repeated exploratory capture in one production process.
Use short surveys, narrow formal measurements, and a post-profile baseline.
Use representative bounded inventories, narrow formal measurements, and a
post-profile baseline.
4 changes: 2 additions & 2 deletions skills/xprobe-measure-latency/agents/openai.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
interface:
display_name: "Xprobe Latency Measurement"
short_description: "Investigate bounded CPU and GPU latency"
default_prompt: "Use $xprobe-measure-latency to survey this workload, derive evidence-based selectors, and measure the suspected latency boundary."
short_description: "Map broad CPU and GPU latency evidence"
default_prompt: "Use $xprobe-measure-latency to map this workload broadly, then measure the suspected latency boundary precisely."
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"schema_version": "2.0",
"name": "coarse_kernel_inventory",
"target": {
"pid": 1234,
"process_start_time": 987654
},
"start_selector": "cuda:kernel_start",
"end_selector": "cuda:kernel_end",
"match_policy": "exact",
"samples": null,
"duration_ms": 1000,
"timeout_ms": 30000,
"max_events": 200000
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"schema_version": "2.0",
"name": "coarse_memcpy_inventory",
"target": {
"pid": 1234,
"process_start_time": 987654
},
"start_selector": "cuda:memcpy_start",
"end_selector": "cuda:memcpy_end",
"match_policy": "exact",
"samples": null,
"duration_ms": 1000,
"timeout_ms": 30000,
"max_events": 200000
}
Loading
Loading