feat(events)!: bound LLM history with context baselines#398
Conversation
Bound raw and annotated LLM event histories after guardrails and codecs run while preserving provider and caller-visible payloads. Update ATOF guidance and observability skills to describe the bounded canonical event contract. BREAKING CHANGE: LLM start and end event contents no longer expose complete conversation histories. Explicit histories now retain system instructions and only the latest user turn, or the final item when no user message exists. Signed-off-by: Will Killian <wkillian@nvidia.com>
WalkthroughThis PR truncates emitted LLM histories to the current user turn, preserves full sanitized baselines at agent and compaction boundaries, adds provider-history codec support and ordered emission reservations, updates tests and documentation, and normalizes PostCompact hooks as compaction events. ChangesLLM history truncation
PostCompact normalization
Estimated code review effort: 5 (Critical) | ~100 minutes Sequence Diagram(s)sequenceDiagram
participant LlmHandle
participant ScopeStack
participant ProviderSurfaceDescriptor
participant AgentEmissionQueue
participant EventSubscribers
LlmHandle->>ScopeStack: reserve baseline or delta start
LlmHandle->>ProviderSurfaceDescriptor: resolve and project request history
LlmHandle->>AgentEmissionQueue: reserve ordered emission
AgentEmissionQueue->>EventSubscribers: deliver start or mark event
LlmHandle->>ScopeStack: rearm baseline on compaction
LlmHandle->>EventSubscribers: emit end event with current user turn
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
github-actions
Bot
added
size:M
willkill07
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/core/src/api/llm.rs`:
- Around line 282-290: The history filter in
limit_json_history_to_current_user_turn is dropping OpenAI Responses instruction
items with role developer by only retaining system markers. Update the retention
predicate passed to retain_current_user_turn so it preserves both system and
developer roles alongside the current user turn, using the existing Json role
checks in this function.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 0025c458-0ef6-4506-8991-15d40437f679
📒 Files selected for processing (9)
crates/core/src/api/llm.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/observability/atof_tests.rsdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdxdocs/instrument-applications/instrument-llm-call.mdxskills/nemo-relay-export-atif-trajectories/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.md
📜 Review details
🧰 Additional context used
📓 Path-based instructions (28)
**/*.{md,rst,html,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
**/*.{md,rst,html,txt}: Always spellNVIDIAin all caps. Do not useNvidia,nvidia,nVidia,nVIDIA, orNV.
Usean NVIDIAbefore a noun because the name starts with an 'en' sound.
Do not add a registered trademark symbol afterNVIDIAwhen referring to the company.
Use trademark symbols with product names only when the document type or legal guidance requires them.
Verify official capitalization, spacing, and hyphenation for product names.
Precede NVIDIA product names withNVIDIAon first mention when it is natural and accurate.
Do not rewrite product names for grammar or title-case rules.
Preserve third-party product names according to the owner's spelling.
Include the company name and full model qualifier on first use when it helps identify the model.
Preserve the official capitalization and punctuation of model names.
Use shorter family names only after the full name is established.
Spell out a term on first use and put the acronym in parentheses unless the acronym is widely understood by the intended audience.
Use the acronym on later mentions after it has been defined.
For long documents, reintroduce the full term if readers might lose context.
Form plurals of acronyms withs, not an apostrophe, such asGPUs.
In headings, common acronyms can remain abbreviated. Spell out the term in the first or second sentence of the body.
Common terms such asCPU,GPU,PC,API, andUIusually do not need to be spelled out for developer audiences.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,rst,html}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
Link the first mention of a product name when the destination helps the reader.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,rst,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
Spell
NVIDIAin all caps. Do not useNvidia,nvidia, orNV.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
**/*.{md,rst}: Format commands, code elements, expressions, package names, file names, and paths as inline code.
Use descriptive link text. Avoid raw URLs and weak anchors such as "here" or "read more."
Use title case consistently for technical documentation headings.
Introduce code blocks, lists, tables, and images with complete sentences.
Write procedures as imperative steps. Keep steps parallel and split long procedures into smaller tasks.
Prefer active voice, present tense, short sentences, contractions, and plain English.
Usecanfor possibility and reservemayfor permission.
Useafterfor temporal relationships instead ofonce.
Preferrefer tooverseewhen the wording points readers to another resource.
Avoid culture-specific idioms, unnecessary Latinisms, jokes, and marketing exaggeration in technical docs.
Spell out months in body text, avoid ordinal dates, and use clear time zones.
Spell out whole numbers from zero through nine unless they are technical values, parameters, versions, or UI values.
Use numerals for 10 or greater and include commas in thousands.
Do not add trademark symbols to learning-oriented docs unless the source, platform, or legal guidance explicitly requires them.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.md
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-technical-docs.md)
**/*.md: Use title case consistently in technical documentation headings
Avoid quotation marks, ampersands, and exclamation marks in headings
Keep product, event, research, and whitepaper names in their official title case
Use title case for table headers
Do not force social-media sentence case into technical docs
Format code elements, commands, parameters, package names, and expressions in monospace
Format directories, file names, and paths in monospace using backticks
Use angle brackets inside monospace for variables inside paths, such as/home/<username>/.login
Format error messages and strings in quotation marks, keeping literal code strings in code formatting when clearer
Format UI buttons, menus, fields, and labels in bold
Use angle brackets between UI labels for menu paths, such as File > Save As
Use italics for new terms on first use, sparingly and only when introducing the term
Use italics for publication titles
Format keyboard shortcuts in plain text, such as Press Ctrl+Alt+Delete
Use owner/repo link text for GitHub repositories, preferring[NVIDIA/NeMo](link)over prose references like 'the GitHub repo'
Introduce every code block with a complete sentence
Do not make a code block complete the grammar of the previous sentence
Do not continue a sentence after a code block
Use syntax highlighting when the format supports it for code blocks
Avoid the word 'snippet' unless the surrounding docs already use it as a term of art
Keep inline method, function, and class references consistent with nearby docs, omitting empty parentheses for prose readability when no call is shown
Use descriptive anchor text that matches the destination title when possible for links
Avoid raw URLs in running text
Avoid generic anchor text such as 'here,' 'this page,' and 'read more'
Include acronyms in link text when a linked term includes an acronym
Do not link long sentences or multiple sentences
Avoid links that pull readers away from a procedure unless the link is a p...
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/SKILL.md
📄 CodeRabbit inference engine (AGENTS.md)
SKILL.mdfiles are skill entrypoints and must start with YAML frontmatter containing at leastnameanddescription.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
⚙️ CodeRabbit configuration file
**/SKILL.md: Do not flag SKILL.md files for missing SPDX headers. Skill entrypoints intentionally start with YAML frontmatter instead.
Verify that every SKILL.md keeps valid YAML frontmatter with at least name and description fields before the Markdown body.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Update
README.md,fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.
**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such asRELEASING.md, not as user-facing docs pages orCHANGELOG.md
Keep stable user-facing wrappers atscripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pagesIf links in documentation change, run
just docs-linkcheck.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mddocs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdxskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,markdown,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mddocs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdxskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mddocs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdxskills/nemo-relay-export-atif-trajectories/SKILL.mdcrates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings,
documentation, integrations, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile skills/ # Published Codex/agent skills for NeMo Relay usage patternsPrerequisites
Insta...
Files:
skills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.mddocs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdxskills/nemo-relay-export-atif-trajectories/SKILL.mdcrates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
In MDX files, top-of-file comments must use JSX comment delimiters (
{/*to open and*/}to close); do not use HTML comments for MDX SPDX headers
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdx
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rs
crates/core/src/api/{tool,llm}.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Wire the new middleware chain into the execute path in
crates/core/src/api/tool.rsorcrates/core/src/api/llm.rsat the appropriate pipeline stage
Files:
crates/core/src/api/llm.rs
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/api/llm.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/core/src/api/llm.rs
🔇 Additional comments (9)
crates/core/tests/unit/llm_api_tests.rs (1)
14-25: LGTM!Also applies to: 48-208, 210-257
crates/core/tests/unit/observability/atof_tests.rs (1)
11-11: LGTM!Also applies to: 1257-1326
docs/about-nemo-relay/concepts/events.mdx (1)
83-91: LGTM!Also applies to: 106-107
docs/configure-plugins/observability/atof.mdx (1)
104-105: LGTM!docs/instrument-applications/instrument-llm-call.mdx (1)
241-245: LGTM!skills/nemo-relay-export-atif-trajectories/SKILL.md (1)
26-28: LGTM!skills/nemo-relay-export-openinference/SKILL.md (1)
32-33: LGTM!skills/nemo-relay-setup-observability/SKILL.md (1)
41-43: LGTM!crates/core/src/api/llm.rs (1)
293-345: 📐 Maintainability & Code QualityVerify the required Rust and core-runtime validation pass.
This touches
crates/core, so please confirmcargo fmt --all,cargo clippy --workspace --all-targets -- -D warnings,just test-rust, and the full binding validation matrix required for core changes were run. As per coding guidelines, “Any Rust change must runjust test-rust,” “Any Rust change must runcargo fmt --all,” and “Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings.” As per path instructions, “Changes tocrates/coreorcrates/adaptivemust run the full language matrix.”Also applies to: 388-400, 473-475, 527-529, 574-603
Sources: Coding guidelines, Path instructions
Address CodeRabbit feedback by retaining developer-role instruction items alongside system instructions before the current user turn. Signed-off-by: Will Killian <wkillian@nvidia.com>
willkill07
changed the title
Preserve complete sanitized request histories for the first LLM start in each agent scope and after canonical compaction marks. Keep ordinary starts and all end-event histories bounded to the current user turn. Refs: RELAY-445 Signed-off-by: Will Killian <wkillian@nvidia.com>
willkill07
force-pushed
the
feat/relay-445-limit-llm-event-context
branch
from
815bc5e to
a36fd97
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/core/tests/unit/llm_api_tests.rs (1)
131-170: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winSnapshot the codec annotations too.
The current assertions only prove the raw request and response JSON stay untouched. They do not catch an in-place mutation of
annotated_requestorannotated_response, which is the boundary this test is meant to protect.🧪 Suggested check
+ let original_annotated_request = annotated_request.as_ref().clone();Repeat the same snapshot for the response annotation before wrapping it in
Arc.Also applies to: 284-285
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/core/tests/unit/llm_api_tests.rs` around lines 131 - 170, The test only snapshots the raw request/response and still misses in-place mutation of the codec annotations. In llm_api_tests around the llm_call / llm_call_end flow, add a snapshot for the response annotation before wrapping it in Arc, similar to original_request and original_response. Use the existing annotated_request and annotated_response symbols to verify both annotations remain unchanged across the call boundary.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@crates/core/tests/unit/llm_api_tests.rs`:
- Around line 131-170: The test only snapshots the raw request/response and
still misses in-place mutation of the codec annotations. In llm_api_tests around
the llm_call / llm_call_end flow, add a snapshot for the response annotation
before wrapping it in Arc, similar to original_request and original_response.
Use the existing annotated_request and annotated_response symbols to verify both
annotations remain unchanged across the call boundary.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 410d12ed-2321-47b5-854a-952f26aed438
📒 Files selected for processing (17)
crates/cli/src/adapters/mod.rscrates/cli/tests/coverage/adapters_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/context_tests.rscrates/core/tests/unit/llm_api_tests.rscrates/core/tests/unit/observability/atof_tests.rsdocs/about-nemo-relay/concepts/events.mdxdocs/configure-plugins/observability/atof.mdxdocs/instrument-applications/instrument-llm-call.mdxdocs/nemo-relay-cli/basic-usage.mdxdocs/nemo-relay-cli/claude-code.mdxdocs/nemo-relay-cli/codex.mdxskills/nemo-relay-export-atif-trajectories/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-setup-observability/SKILL.md
📜 Review details
⏰ Context from checks skipped due to timeout. (20)
- GitHub Check: Node.js / Test (windows-amd64)
- GitHub Check: Node.js / Test (macos-arm64)
- GitHub Check: Node.js / Test (windows-arm64)
- GitHub Check: Python / Test (linux-arm64)
- GitHub Check: Node.js / Test (linux-arm64)
- GitHub Check: Go / Test (macos-arm64)
- GitHub Check: Python / Test (macos-arm64)
- GitHub Check: Node.js / Test (linux-amd64)
- GitHub Check: Python / Test (windows-arm64)
- GitHub Check: Go / Test (windows-arm64)
- GitHub Check: Python / Test (linux-amd64)
- GitHub Check: Go / Test (windows-amd64)
- GitHub Check: Rust / Test (linux-amd64)
- GitHub Check: Python / Test (windows-amd64)
- GitHub Check: Rust / Test (macos-arm64)
- GitHub Check: Rust / Test (linux-arm64)
- GitHub Check: Rust / Test (windows-arm64)
- GitHub Check: Go / Test (linux-amd64)
- GitHub Check: Go / Test (linux-arm64)
- GitHub Check: Rust / Test (windows-amd64)
🧰 Additional context used
📓 Path-based instructions (29)
**/*.{md,rst,html,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
**/*.{md,rst,html,txt}: Always spellNVIDIAin all caps. Do not useNvidia,nvidia,nVidia,nVIDIA, orNV.
Usean NVIDIAbefore a noun because the name starts with an 'en' sound.
Do not add a registered trademark symbol afterNVIDIAwhen referring to the company.
Use trademark symbols with product names only when the document type or legal guidance requires them.
Verify official capitalization, spacing, and hyphenation for product names.
Precede NVIDIA product names withNVIDIAon first mention when it is natural and accurate.
Do not rewrite product names for grammar or title-case rules.
Preserve third-party product names according to the owner's spelling.
Include the company name and full model qualifier on first use when it helps identify the model.
Preserve the official capitalization and punctuation of model names.
Use shorter family names only after the full name is established.
Spell out a term on first use and put the acronym in parentheses unless the acronym is widely understood by the intended audience.
Use the acronym on later mentions after it has been defined.
For long documents, reintroduce the full term if readers might lose context.
Form plurals of acronyms withs, not an apostrophe, such asGPUs.
In headings, common acronyms can remain abbreviated. Spell out the term in the first or second sentence of the body.
Common terms such asCPU,GPU,PC,API, andUIusually do not need to be spelled out for developer audiences.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,rst,html}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)
Link the first mention of a product name when the destination helps the reader.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,rst,txt}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
Spell
NVIDIAin all caps. Do not useNvidia,nvidia, orNV.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,rst}
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)
**/*.{md,rst}: Format commands, code elements, expressions, package names, file names, and paths as inline code.
Use descriptive link text. Avoid raw URLs and weak anchors such as "here" or "read more."
Use title case consistently for technical documentation headings.
Introduce code blocks, lists, tables, and images with complete sentences.
Write procedures as imperative steps. Keep steps parallel and split long procedures into smaller tasks.
Prefer active voice, present tense, short sentences, contractions, and plain English.
Usecanfor possibility and reservemayfor permission.
Useafterfor temporal relationships instead ofonce.
Preferrefer tooverseewhen the wording points readers to another resource.
Avoid culture-specific idioms, unnecessary Latinisms, jokes, and marketing exaggeration in technical docs.
Spell out months in body text, avoid ordinal dates, and use clear time zones.
Spell out whole numbers from zero through nine unless they are technical values, parameters, versions, or UI values.
Use numerals for 10 or greater and include commas in thousands.
Do not add trademark symbols to learning-oriented docs unless the source, platform, or legal guidance explicitly requires them.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.md
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-technical-docs.md)
**/*.md: Use title case consistently in technical documentation headings
Avoid quotation marks, ampersands, and exclamation marks in headings
Keep product, event, research, and whitepaper names in their official title case
Use title case for table headers
Do not force social-media sentence case into technical docs
Format code elements, commands, parameters, package names, and expressions in monospace
Format directories, file names, and paths in monospace using backticks
Use angle brackets inside monospace for variables inside paths, such as/home/<username>/.login
Format error messages and strings in quotation marks, keeping literal code strings in code formatting when clearer
Format UI buttons, menus, fields, and labels in bold
Use angle brackets between UI labels for menu paths, such as File > Save As
Use italics for new terms on first use, sparingly and only when introducing the term
Use italics for publication titles
Format keyboard shortcuts in plain text, such as Press Ctrl+Alt+Delete
Use owner/repo link text for GitHub repositories, preferring[NVIDIA/NeMo](link)over prose references like 'the GitHub repo'
Introduce every code block with a complete sentence
Do not make a code block complete the grammar of the previous sentence
Do not continue a sentence after a code block
Use syntax highlighting when the format supports it for code blocks
Avoid the word 'snippet' unless the surrounding docs already use it as a term of art
Keep inline method, function, and class references consistent with nearby docs, omitting empty parentheses for prose readability when no call is shown
Use descriptive anchor text that matches the destination title when possible for links
Avoid raw URLs in running text
Avoid generic anchor text such as 'here,' 'this page,' and 'read more'
Include acronyms in link text when a linked term includes an acronym
Do not link long sentences or multiple sentences
Avoid links that pull readers away from a procedure unless the link is a p...
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/SKILL.md
📄 CodeRabbit inference engine (AGENTS.md)
SKILL.mdfiles are skill entrypoints and must start with YAML frontmatter containing at leastnameanddescription.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
⚙️ CodeRabbit configuration file
**/SKILL.md: Do not flag SKILL.md files for missing SPDX headers. Skill entrypoints intentionally start with YAML frontmatter instead.
Verify that every SKILL.md keeps valid YAML frontmatter with at least name and description fields before the Markdown body.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdskills/nemo-relay-export-atif-trajectories/SKILL.md
**/*.{md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Update
README.md,fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.
**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such asRELEASING.md, not as user-facing docs pages orCHANGELOG.md
Keep stable user-facing wrappers atscripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pagesIf links in documentation change, run
just docs-linkcheck.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mddocs/nemo-relay-cli/codex.mdxdocs/instrument-applications/instrument-llm-call.mdxdocs/nemo-relay-cli/basic-usage.mdxskills/nemo-relay-export-atif-trajectories/SKILL.mddocs/configure-plugins/observability/atof.mdxdocs/nemo-relay-cli/claude-code.mdxdocs/about-nemo-relay/concepts/events.mdx
**/*.{md,markdown,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mddocs/nemo-relay-cli/codex.mdxdocs/instrument-applications/instrument-llm-call.mdxdocs/nemo-relay-cli/basic-usage.mdxskills/nemo-relay-export-atif-trajectories/SKILL.mddocs/configure-plugins/observability/atof.mdxdocs/nemo-relay-cli/claude-code.mdxdocs/about-nemo-relay/concepts/events.mdx
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdcrates/core/tests/unit/context_tests.rsdocs/nemo-relay-cli/codex.mdxdocs/instrument-applications/instrument-llm-call.mdxcrates/cli/tests/coverage/adapters_tests.rsdocs/nemo-relay-cli/basic-usage.mdxskills/nemo-relay-export-atif-trajectories/SKILL.mddocs/configure-plugins/observability/atof.mdxcrates/core/src/api/runtime/state.rsdocs/nemo-relay-cli/claude-code.mdxcrates/cli/src/adapters/mod.rsdocs/about-nemo-relay/concepts/events.mdxcrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings,
documentation, integrations, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile skills/ # Published Codex/agent skills for NeMo Relay usage patternsPrerequisites
Insta...
Files:
skills/nemo-relay-setup-observability/SKILL.mdskills/nemo-relay-export-openinference/SKILL.mdcrates/core/tests/unit/context_tests.rsdocs/nemo-relay-cli/codex.mdxdocs/instrument-applications/instrument-llm-call.mdxcrates/cli/tests/coverage/adapters_tests.rsdocs/nemo-relay-cli/basic-usage.mdxskills/nemo-relay-export-atif-trajectories/SKILL.mddocs/configure-plugins/observability/atof.mdxcrates/core/src/api/runtime/state.rsdocs/nemo-relay-cli/claude-code.mdxcrates/cli/src/adapters/mod.rsdocs/about-nemo-relay/concepts/events.mdxcrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/core/tests/unit/context_tests.rscrates/cli/tests/coverage/adapters_tests.rscrates/core/src/api/runtime/state.rscrates/cli/src/adapters/mod.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/tests/unit/context_tests.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/tests/unit/context_tests.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/core/tests/unit/context_tests.rscrates/cli/tests/coverage/adapters_tests.rscrates/core/src/api/runtime/state.rscrates/cli/src/adapters/mod.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/core/tests/unit/context_tests.rscrates/cli/tests/coverage/adapters_tests.rscrates/core/src/api/runtime/state.rscrates/cli/src/adapters/mod.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/core/tests/unit/context_tests.rscrates/cli/tests/coverage/adapters_tests.rscrates/core/src/api/runtime/state.rscrates/cli/src/adapters/mod.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/core/tests/unit/context_tests.rscrates/cli/tests/coverage/adapters_tests.rscrates/core/src/api/runtime/state.rscrates/cli/src/adapters/mod.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/tests/unit/context_tests.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/core/tests/unit/context_tests.rscrates/cli/tests/coverage/adapters_tests.rscrates/core/src/api/runtime/state.rscrates/cli/src/adapters/mod.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/tests/unit/context_tests.rscrates/core/src/api/runtime/state.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/core/tests/unit/context_tests.rscrates/cli/tests/coverage/adapters_tests.rscrates/core/tests/unit/observability/atof_tests.rscrates/core/tests/unit/llm_api_tests.rs
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
In MDX files, top-of-file comments must use JSX comment delimiters (
{/*to open and*/}to close); do not use HTML comments for MDX SPDX headers
Files:
docs/nemo-relay-cli/codex.mdxdocs/instrument-applications/instrument-llm-call.mdxdocs/nemo-relay-cli/basic-usage.mdxdocs/configure-plugins/observability/atof.mdxdocs/nemo-relay-cli/claude-code.mdxdocs/about-nemo-relay/concepts/events.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/nemo-relay-cli/codex.mdxdocs/instrument-applications/instrument-llm-call.mdxdocs/nemo-relay-cli/basic-usage.mdxdocs/configure-plugins/observability/atof.mdxdocs/nemo-relay-cli/claude-code.mdxdocs/about-nemo-relay/concepts/events.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/nemo-relay-cli/codex.mdxdocs/instrument-applications/instrument-llm-call.mdxdocs/nemo-relay-cli/basic-usage.mdxdocs/configure-plugins/observability/atof.mdxdocs/nemo-relay-cli/claude-code.mdxdocs/about-nemo-relay/concepts/events.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/nemo-relay-cli/codex.mdxdocs/instrument-applications/instrument-llm-call.mdxdocs/nemo-relay-cli/basic-usage.mdxdocs/configure-plugins/observability/atof.mdxdocs/nemo-relay-cli/claude-code.mdxdocs/about-nemo-relay/concepts/events.mdx
crates/core/src/api/runtime/state.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
crates/core/src/api/runtime/state.rs: Add registry fields asSortedRegistry<GuardrailEntry<T>>orSortedRegistry<Intercept<T>>toNemoRelayContextStateincrates/core/src/api/runtime/state.rs
Add chain execution helpers toNemoRelayContextStatefollowing the pattern of existing methods liketool_sanitize_request_chainortool_request_intercepts_chain
Files:
crates/core/src/api/runtime/state.rs
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/api/runtime/state.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/core/src/api/runtime/state.rscrates/cli/src/adapters/mod.rscrates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rs
crates/core/src/api/{tool,llm}.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Wire the new middleware chain into the execute path in
crates/core/src/api/tool.rsorcrates/core/src/api/llm.rsat the appropriate pipeline stage
Files:
crates/core/src/api/llm.rs
🔇 Additional comments (23)
crates/cli/src/adapters/mod.rs (1)
898-898: LGTM!crates/cli/tests/coverage/adapters_tests.rs (1)
933-933: LGTM!docs/nemo-relay-cli/basic-usage.mdx (1)
309-310: LGTM!docs/nemo-relay-cli/claude-code.mdx (1)
125-129: LGTM!docs/nemo-relay-cli/codex.mdx (1)
184-185: LGTM!crates/core/tests/unit/llm_api_tests.rs (3)
14-42: LGTM!
58-89: LGTM!
288-397: LGTM!Also applies to: 399-448, 451-611
crates/core/tests/unit/observability/atof_tests.rs (2)
12-45: LGTM!
1260-1347: LGTM!docs/about-nemo-relay/concepts/events.mdx (1)
83-96: LGTM!Also applies to: 111-113
docs/configure-plugins/observability/atof.mdx (1)
104-107: LGTM!docs/instrument-applications/instrument-llm-call.mdx (1)
241-246: LGTM!skills/nemo-relay-export-atif-trajectories/SKILL.md (1)
26-28: LGTM!skills/nemo-relay-export-openinference/SKILL.md (1)
32-34: LGTM!skills/nemo-relay-setup-observability/SKILL.md (1)
41-43: LGTM!crates/core/src/api/runtime/scope_stack.rs (2)
12-12: LGTM!Also applies to: 30-31, 45-49, 58-60, 144-144, 255-258
160-193: 🩺 Stability & AvailabilityCheck scope-stack affinity for
owning_agent_uuid().take_full_llm_context()/rearm_full_llm_context()assumeparent_uuidis resolved in the sameScopeStackinstance that owns the agent scope. Confirm that compactionMarkemission andemit_llm_start_with_subscribers()never cross a task/thread boundary; otherwise the fallback can consume or re-arm the wrong baseline.crates/core/src/api/runtime/state.rs (2)
29-29: LGTM!
194-199: 🗄️ Data Integrity & IntegrationCentralize the
compactionmark name
Thecompactionmark name should come from a shared constant used by both the runtime and the CLI adapter so the re-arm path can’t drift silently.crates/core/tests/unit/context_tests.rs (1)
98-136: LGTM!crates/core/src/api/llm.rs (2)
373-425: LGTM!Also applies to: 488-501, 541-552
265-351: 🎯 Functional CorrectnessInspect the current-turn truncation helpers directly. The summary points to
retain_current_user_turnand thelimit_*_to_current_user_turnpaths, but the implementation isn’t in the visible snippet. That logic decides whether prior-turn history is retained or dropped, so it needs a direct pass before relying on the change summary.
…t-llm-event-context Signed-off-by: Will Killian <wkillian@nvidia.com> # Conflicts: # crates/core/src/api/llm.rs # crates/core/src/api/runtime/state.rs # crates/core/tests/unit/llm_api_tests.rs
mnajafian-nv
left a comment
mnajafian-nv
left a comment
There was a problem hiding this comment.
I left four focused questions on contract consistency, concurrency, and compatibility. The post-sanitization projection direction looks sound; these comments are intended to make the breaking event contract consistent across every path before approval.
Project request histories exclusively through available codecs, apply end-event bounding at the shared event builder, and preserve baseline-before-delta ordering for concurrent starts.\n\nRefs: RELAY-445 Signed-off-by: Will Killian <wkillian@nvidia.com>
github-actions
Bot
added
size:XL
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/core/src/api/llm.rs`:
- Around line 307-320: retain_current_user_turn currently drops instruction
messages in the no-user branch by draining everything before the last item,
which can collapse a system/developer/assistant history down to only the
assistant turn. Update the no-user path in retain_current_user_turn so
instruction messages are preserved consistently with the user-found branch, and
make sure the helper still only trims non-instruction history when appropriate.
Add a regression test covering a user-less history containing system and
developer messages to verify those instructions remain after calling
retain_current_user_turn.
In `@crates/core/tests/unit/llm_api_tests.rs`:
- Around line 537-600: Add a baseline Start-event assertion in
custom_request_codecs_project_convertible_histories so the codec-driven
managed-execution path verifies both requests. Keep the existing delta check,
but also locate the custom-codec-baseline event in the collected events and
assert that its projected conversation preserves the full 4-message history and
the annotated request reflects all messages, matching the behavior already
covered in the other history tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 4edb5638-eebb-4a3b-84bc-ef2b5f596304
📒 Files selected for processing (10)
crates/core/src/api/llm.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/tests/unit/llm_api_tests.rsdocs/about-nemo-relay/concepts/events.mdxdocs/instrument-applications/instrument-llm-call.mdx
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (23)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/api/llm.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/api/llm.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/core/src/codec/openai_chat.rsdocs/instrument-applications/instrument-llm-call.mdxcrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rsdocs/about-nemo-relay/concepts/events.mdxcrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings,
documentation, integrations, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile skills/ # Published Codex/agent skills for NeMo Relay usage patternsPrerequisites
Insta...
Files:
crates/core/src/codec/openai_chat.rsdocs/instrument-applications/instrument-llm-call.mdxcrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rsdocs/about-nemo-relay/concepts/events.mdxcrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/scope_stack.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
In MDX files, top-of-file comments must use JSX comment delimiters (
{/*to open and*/}to close); do not use HTML comments for MDX SPDX headers
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Update
README.md,fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.
**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such asRELEASING.md, not as user-facing docs pages orCHANGELOG.md
Keep stable user-facing wrappers atscripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pagesIf links in documentation change, run
just docs-linkcheck.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
**/*.{md,markdown,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
crates/core/src/api/runtime/state.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
crates/core/src/api/runtime/state.rs: Add registry fields asSortedRegistry<GuardrailEntry<T>>orSortedRegistry<Intercept<T>>toNemoRelayContextStateincrates/core/src/api/runtime/state.rs
Add chain execution helpers toNemoRelayContextStatefollowing the pattern of existing methods liketool_sanitize_request_chainortool_request_intercepts_chain
Files:
crates/core/src/api/runtime/state.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/core/tests/unit/llm_api_tests.rs
crates/core/src/api/{tool,llm}.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Wire the new middleware chain into the execute path in
crates/core/src/api/tool.rsorcrates/core/src/api/llm.rsat the appropriate pipeline stage
Files:
crates/core/src/api/llm.rs
🔇 Additional comments (18)
docs/about-nemo-relay/concepts/events.mdx (1)
94-97: LGTM!Also applies to: 115-117
docs/instrument-applications/instrument-llm-call.mdx (1)
247-250: LGTM!crates/core/src/api/runtime/scope_stack.rs (1)
46-51: LGTM!Also applies to: 60-64, 148-149, 165-209, 271-278
crates/core/src/api/runtime/state.rs (2)
548-576: LGTM!
196-201: 🩺 Stability & AvailabilityConfirm compaction marks rearm the originating agent
Compaction marks should only callrearm_full_llm_context(event.parent_uuid())when the activecurrent_scope_stack()is guaranteed to contain that parent agent; otherwise the fallback can arm the wrong baseline for the next LLM call.crates/core/src/api/llm.rs (1)
486-506: LGTM!crates/core/src/codec/resolve.rs (1)
50-50: LGTM!Also applies to: 137-146
crates/core/src/codec/anthropic.rs (1)
58-58: LGTM!crates/core/src/codec/openai_chat.rs (1)
35-35: LGTM!crates/core/src/codec/openai_responses.rs (1)
50-50: LGTM!crates/core/tests/unit/llm_api_tests.rs (8)
8-38: LGTM!Also applies to: 50-61
78-154: LGTM!
ConversationCodecandHistoryResponseCodeccorrectly round-trip throughOpenAIChatCodecand populate theextracatch-all, matching the annotated request/response contract used elsewhere in the suite.
157-350: LGTM!Solid coverage: baseline-then-delta projection,
Arccopy-on-write isolation between the first (full-baseline) and second (reduced) Start events, and preservation ofmetadata/tools/content/output/usagefields in bounded events.
502-535: LGTM!Good coverage of no-op edge cases: empty history, opaque/non-matching message shapes, and scalar
input— all correctly assert no mutation.
615-666: LGTM!Manual, managed, and streaming end-path projections are exercised consistently, including annotated-response
extra.messagesreduction for the codec-backed cases.
712-807: LGTM!The emission-gate/mutex-holding design is correctly exploited here: the second thread can't reach its own sanitize-start guardrail until the first thread's entire
emit_llm_start_with_subscriberscall (including subscriber push) completes, which is exactly what's needed to prove reserved-baseline-before-delta ordering under concurrency.
490-500: 🎯 Functional CorrectnessConfirm the no-user fallback for assistant/tool turns. The current expectation drops the assistant message and keeps only the trailing tool result. If a no-user-boundary response is still meant to preserve tool context, this fallback should retain the assistant half of the pair or be documented explicitly.
667-710: 🗄️ Data Integrity & IntegrationCheck the failure-path end output projection. The
failed-history-endcase relies on.data(multi_turn_response()), butLlmCallExecuteParams::datais documented as handle-only state. Confirmemit_llm_end_without_outputis meant to project that payload intoend.output()on failure, and that this assertion cannot beNone.
Preserve provider-native request and response fields while bounding any codec-resolved conversation history. Order compaction marks and LLM starts without holding locks across sanitizer callbacks. Refs: RELAY-445 Signed-off-by: Will Killian <wkillian@nvidia.com>
Queue LLM-generated marks behind their captured agent start reservation, acknowledge buffered optimization delivery without duplication, and retain raw end-history projection when annotations contain no history. Refs: RELAY-445 Signed-off-by: Will Killian <wkillian@nvidia.com>
| }; | ||
| let Some(start) = items.iter().rposition(is_user) else { | ||
| let changed = last_index > 0; | ||
| items.drain(..last_index); |
There was a problem hiding this comment.
Could we split the no-user fallback by history kind here? For request projection, a system/developer-only conversation is still the complete current context, but this branch trims it to the final instruction by draining everything before last_index. That seems to contradict the new start-event contract to preserve system/developer instructions. A regression with messages: [system, developer] and a Responses input that contains only instruction items would make the intended behavior explicit.
Keep complete system and developer instruction histories on request events when no user message is present, while retaining the end-event final-item fallback.\n\nRefs: RELAY-445 Signed-off-by: Will Killian <wkillian@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
docs/instrument-applications/instrument-llm-call.mdx (1)
241-243: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winCorrect the subsequent-start history description.
“Other starts contain the current user turn” is incomplete: they also retain system/developer instructions and subsequent assistant/tool messages, as documented in
docs/about-nemo-relay/concepts/events.mdxlines 85-87. Update this bullet so validation expectations match the emitted event contract.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/instrument-applications/instrument-llm-call.mdx` around lines 241 - 243, The subsequent-start description incorrectly says these events contain only the current user turn; update the relevant bullet in the instrument-llm-call documentation to state that they retain system/developer instructions, the current user turn, and subsequent assistant/tool messages, matching the event contract in events.mdx.Source: Path instructions
crates/core/tests/unit/llm_api_tests.rs (1)
500-572: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winMissing regression coverage for
retain_current_user_turn's no-user + instruction case.
current_user_turn_projection_handles_responses_input_and_edge_cases'sno_usercase only usesassistant/toolentries (nosystem/developer), so it doesn't exercise the no-user-with-instructions bug inretain_current_user_turn(seellm.rscomment). Add a case with e.g.[{"role": "system", ...}, {"role": "assistant", ...}]and assert the system message survives, matching the pattern used for the request-side instruction-preservation tests elsewhere in this file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/core/tests/unit/llm_api_tests.rs` around lines 500 - 572, Add regression coverage in current_user_turn_projection_handles_responses_input_and_edge_cases for a no-user response containing an instruction: include a system or developer message followed by an assistant message, call project_llm_response_to_current_user_turn, and assert the instruction plus the expected retained entries remain. This should exercise retain_current_user_turn’s no-user instruction-preservation behavior.
♻️ Duplicate comments (1)
crates/core/src/api/llm.rs (1)
359-377: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
retain_current_user_turn's no-user branch still drops leading instructions — same bug flagged previously, now scoped to the response/end-event path.The no-user branch (
items.drain(..last_index)) unconditionally removes everything except the last item, regardless of whether earlier items aresystem/developerinstructions. This is the exact issue raised in a prior review ("Preserve instructions when no user turn exists") that has no "Addressed" acknowledgment here. The sibling functionretain_current_request_turn(lines 379-402), added in this same PR for request-side truncation, correctly fixes this by retaining instruction items plus the last item — butretain_current_user_turn(used only for the response/end-event path vialimit_role_history_to_current_user_turn) was left with the old behavior. A response history with nouserturn but a leadingsystem/developermessage would have that instruction silently dropped from the emitted end event.🐛 Suggested fix — mirror `retain_current_request_turn`'s no-user handling
let Some(start) = items.iter().rposition(is_user) else { - let changed = last_index > 0; - items.drain(..last_index); - return changed; + let original_len = items.len(); + let mut index = 0; + items.retain(|item| { + let retain = index == last_index || is_instruction(item); + index += 1; + retain + }); + return items.len() != original_len; };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/core/src/api/llm.rs` around lines 359 - 377, Update retain_current_user_turn’s no-user branch to preserve all instruction items while retaining the final item, matching retain_current_request_turn’s handling. Replace the unconditional drain behavior with filtering or equivalent logic that keeps items satisfying is_instruction plus the last item, and ensure changed accurately reflects whether non-instruction items were removed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/core/src/api/llm.rs`:
- Around line 523-541: Replace the blind deep-equality traversal in
replace_first_matching_history with targeted replacement at the exact source
location used to extract the history, preferably by tracking and passing its
JSON pointer/path or restricting traversal to known history keys such as
"messages" and "conversation". Ensure the function reports failure when that
expected path is missing or no longer matches, so truncation cannot silently
patch an unrelated duplicate field.
In `@crates/core/src/codec/anthropic.rs`:
- Around line 443-451: Update the Anthropic re-encoding logic that handles
UNPARSED_MESSAGES_KEY so normalized message edits are preserved instead of
writing the cached raw messages verbatim. When rebuilding the payload, merge or
apply mutations from the current normalized messages to the cached unparsed
content, while retaining unsupported provider-native blocks; reference the
parsing logic producing unparsed_messages and the corresponding encode path
around UNPARSED_MESSAGES_KEY.
In `@crates/core/src/codec/openai_chat.rs`:
- Around line 248-256: Update OpenAI chat codec encode handling so
`extra[CHAT_UNPARSED_MESSAGES_KEY]` does not cause `annotated.messages` edits to
be discarded. In `encode()`, merge or reconstruct the raw unparsed entries with
the current typed messages while preserving unsupported provider message
content, and ensure the same behavior covers the related decode/projection paths
around `messages` and `unparsed_messages`. Add or update coverage for
developer-role messages and verify intercept-driven rewrites persist after
re-encoding.
---
Outside diff comments:
In `@crates/core/tests/unit/llm_api_tests.rs`:
- Around line 500-572: Add regression coverage in
current_user_turn_projection_handles_responses_input_and_edge_cases for a
no-user response containing an instruction: include a system or developer
message followed by an assistant message, call
project_llm_response_to_current_user_turn, and assert the instruction plus the
expected retained entries remain. This should exercise
retain_current_user_turn’s no-user instruction-preservation behavior.
In `@docs/instrument-applications/instrument-llm-call.mdx`:
- Around line 241-243: The subsequent-start description incorrectly says these
events contain only the current user turn; update the relevant bullet in the
instrument-llm-call documentation to state that they retain system/developer
instructions, the current user turn, and subsequent assistant/tool messages,
matching the event contract in events.mdx.
---
Duplicate comments:
In `@crates/core/src/api/llm.rs`:
- Around line 359-377: Update retain_current_user_turn’s no-user branch to
preserve all instruction items while retaining the final item, matching
retain_current_request_turn’s handling. Replace the unconditional drain behavior
with filtering or equivalent logic that keeps items satisfying is_instruction
plus the last item, and ensure changed accurately reflects whether
non-instruction items were removed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: fabeff89-094f-4cad-8855-fb3df7892cb7
📒 Files selected for processing (13)
crates/core/src/api/llm.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/api/runtime/state.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/api/scope.rscrates/core/src/codec/anthropic.rscrates/core/src/codec/openai_chat.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/tests/unit/llm_api_tests.rsdocs/about-nemo-relay/concepts/events.mdxdocs/instrument-applications/instrument-llm-call.mdx
📜 Review details
⏰ Context from checks skipped due to timeout. (11)
- GitHub Check: Rust / Package (windows-amd64)
- GitHub Check: Rust / Package (windows-arm64)
- GitHub Check: Rust / Package (linux-arm64)
- GitHub Check: Rust / Package (macos-arm64)
- GitHub Check: Rust / Package (linux-amd64)
- GitHub Check: CodeRabbit / Review
- GitHub Check: Node.js / Package (windows-amd64)
- GitHub Check: Node.js / Package (windows-arm64)
- GitHub Check: Node.js / Package (macos-arm64)
- GitHub Check: Node.js / Package (linux-amd64)
- GitHub Check: Python / Test (windows-arm64)
🧰 Additional context used
📓 Path-based instructions (23)
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
In MDX files, top-of-file comments must use JSX comment delimiters (
{/*to open and*/}to close); do not use HTML comments for MDX SPDX headers
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Update
README.md,fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.
**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such asRELEASING.md, not as user-facing docs pages orCHANGELOG.md
Keep stable user-facing wrappers atscripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pagesIf links in documentation change, run
just docs-linkcheck.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
**/*.{md,markdown,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxcrates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings,
documentation, integrations, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile skills/ # Published Codex/agent skills for NeMo Relay usage patternsPrerequisites
Insta...
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdxcrates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/instrument-applications/instrument-llm-call.mdxdocs/about-nemo-relay/concepts/events.mdx
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/core/src/{api/**/*.rs,api/runtime/**/*.rs,codec/**/*.rs,json.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Implement the new or changed public runtime behavior first in the Rust core, especially under
crates/core/src/api/and related core modules such ascrates/core/src/api/runtime/,crates/core/src/codec/, andcrates/core/src/json.rs.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/src/api/llm.rs
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/src/api/scope.rscrates/core/src/api/runtime/subscriber_dispatcher.rscrates/core/src/codec/openai_responses.rscrates/core/src/codec/resolve.rscrates/core/src/api/runtime/state.rscrates/core/src/codec/anthropic.rscrates/core/src/api/optimization.rscrates/core/src/api/runtime/scope_stack.rscrates/core/src/codec/openai_chat.rscrates/core/tests/unit/llm_api_tests.rscrates/core/src/api/llm.rs
crates/core/src/api/runtime/state.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
crates/core/src/api/runtime/state.rs: Add registry fields asSortedRegistry<GuardrailEntry<T>>orSortedRegistry<Intercept<T>>toNemoRelayContextStateincrates/core/src/api/runtime/state.rs
Add chain execution helpers toNemoRelayContextStatefollowing the pattern of existing methods liketool_sanitize_request_chainortool_request_intercepts_chain
Files:
crates/core/src/api/runtime/state.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/core/tests/unit/llm_api_tests.rs
crates/core/src/api/{tool,llm}.rs
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Wire the new middleware chain into the execute path in
crates/core/src/api/tool.rsorcrates/core/src/api/llm.rsat the appropriate pipeline stage
Files:
crates/core/src/api/llm.rs
🔇 Additional comments (14)
docs/about-nemo-relay/concepts/events.mdx (1)
83-99: LGTM!Also applies to: 115-117
docs/instrument-applications/instrument-llm-call.mdx (1)
244-250: LGTM!crates/core/src/api/runtime/scope_stack.rs (3)
37-160: LGTM!
292-354: LGTM!
178-190: LGTM!Also applies to: 275-276, 417-424
crates/core/src/api/scope.rs (1)
331-362: LGTM!crates/core/src/codec/openai_responses.rs (1)
28-30: LGTM!Also applies to: 46-49
crates/core/src/api/optimization.rs (1)
38-38: LGTM!Also applies to: 194-254
crates/core/src/api/runtime/subscriber_dispatcher.rs (1)
42-72: LGTM!Also applies to: 171-178
crates/core/src/codec/resolve.rs (1)
27-42: LGTM!Also applies to: 63-63, 153-162
crates/core/src/api/llm.rs (2)
20-95: LGTM!Reservation-based ordered emission wiring, provider-agnostic truncation dispatch (
limit_json_history_to_current_user_turn), and the request/annotation projection helper otherwise look correct and are well covered by tests (concurrency, reentrancy, compaction re-arming).Also applies to: 139-152, 342-358, 404-521, 552-587, 588-628, 630-707, 892-905, 943-946
719-744: 🩺 Stability & AvailabilityNo issue here: uncommitted
AgentEmissionReservations are canceled on drop, so sanitization returningNonedoes not stall the emission queue.> Likely an incorrect or invalid review comment.crates/core/src/api/runtime/state.rs (1)
20-22: LGTM!Also applies to: 541-569
crates/core/tests/unit/llm_api_tests.rs (1)
8-190: LGTM!Coverage is thorough for the projection/ordering contract otherwise (baseline vs. delta, compaction re-arming, concurrency, reentrancy, and end-path parity across manual/managed/streaming/failure).
Also applies to: 193-499, 574-1517
| fn replace_first_matching_history( | ||
| value: &mut Json, | ||
| original_history: &Json, | ||
| projected_history: &Json, | ||
| ) -> bool { | ||
| if value == original_history { | ||
| *value = projected_history.clone(); | ||
| return true; | ||
| } | ||
| match value { | ||
| Json::Array(items) => items | ||
| .iter_mut() | ||
| .any(|item| replace_first_matching_history(item, original_history, projected_history)), | ||
| Json::Object(fields) => fields.values_mut().any(|field| { | ||
| replace_first_matching_history(field, original_history, projected_history) | ||
| }), | ||
| _ => false, | ||
| } | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Deep-equality history replacement can patch the wrong field or silently leave the real history untruncated.
replace_first_matching_history walks data and replaces the first value that is structurally equal to original_history. If the raw response payload contains another field whose value happens to equal the extracted history (duplicated/mirrored content, or a custom response codec that stores the same array under two keys), the DFS traversal order determines which field gets patched — potentially the wrong one, leaving the real "messages" field exposed in full while a look-alike field is truncated instead. Since this function underwrites the PR's core guarantee ("end events include only the current user turn"), a mismatch here silently defeats that guarantee rather than failing loudly.
Consider matching on the known source key(s) (e.g. only replace under a literal "messages"/"conversation" key path) rather than a blind whole-tree equality search, or track the JSON pointer path where original_history was extracted from and replace at that exact path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/core/src/api/llm.rs` around lines 523 - 541, Replace the blind
deep-equality traversal in replace_first_matching_history with targeted
replacement at the exact source location used to extract the history, preferably
by tracking and passing its JSON pointer/path or restricting traversal to known
history keys such as "messages" and "conversation". Ensure the function reports
failure when that expected path is missing or no longer matches, so truncation
cannot silently patch an unrelated duplicate field.
There was a problem hiding this comment.
Agree this looks worth tightening. Since we extract history from a specific field, patching by deep equality could truncate the wrong copy if the same array appears elsewhere in the payload. Could we replace at the exact extracted path, or constrain this to known history keys?
There was a problem hiding this comment.
@mnajafian-nv Agreed — constraining by known key names is the more robust fix here, since exact-path tracking would need plumbing through the codec/annotation extraction, and the annotation-sourced history (from annotated_response.extra["messages"]) doesn't always have a matching literal path in data for provider-specific shapes anyway.
Suggested approach: change replace_first_matching_history to only descend into object fields whose key matches a known history key ("messages", and the provider "unparsed" keys already used elsewhere in this file: CHAT_UNPARSED_MESSAGES_KEY, ANTHROPIC_UNPARSED_MESSAGES_KEY, "_openai_responses_unparsed_input_items"), and only fall back to equality check within that constrained field rather than searching every field in the tree:
fn replace_first_matching_history(
value: &mut Json,
original_history: &Json,
projected_history: &Json,
) -> bool {
match value {
Json::Object(fields) => {
for key in [
"messages",
CHAT_UNPARSED_MESSAGES_KEY,
ANTHROPIC_UNPARSED_MESSAGES_KEY,
"_openai_responses_unparsed_input_items",
] {
if let Some(field) = fields.get_mut(key)
&& field == original_history
{
*field = projected_history.clone();
return true;
}
}
fields.values_mut().any(|field| {
replace_first_matching_history(field, original_history, projected_history)
})
}
Json::Array(items) => items.iter_mut().any(|item| {
replace_first_matching_history(item, original_history, projected_history)
}),
_ => false,
}
}This keeps the recursive traversal (needed since provider responses nest the history at varying depths) but restricts matching to fields keyed by a recognized history field, so a look-alike array under an unrelated key won't get silently patched instead of the real one.
Want me to draft this as a concrete diff via the planning flow, or would you rather land it yourselves in a follow-up commit?
There was a problem hiding this comment.
@mnajafian-nv Sounds good — I'll wait for @willkill07's input on the proposed approach (constraining replace_first_matching_history to known history keys rather than blind deep-equality search) before this is finalized. Happy to draft the concrete diff via `` once there's agreement on the approach.
Failed to handle agent chat message. Please try again.
| // Preserve provider-native tool and future content blocks that the | ||
| // normalized message schema does not yet model. | ||
| let (mut messages, unparsed_messages) = match obj.get("messages") { | ||
| Some(raw) => match serde_json::from_value(raw.clone()) { | ||
| Ok(messages) => (messages, None), | ||
| Err(_) => (Vec::new(), Some(raw.clone())), | ||
| }, | ||
| None => (Vec::new(), None), | ||
| }; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n 'fn run_request_intercepts_with_codec_and_recorder' -A 60 crates/core/src/api/llm.rs
rg -n '\.encode\(' crates/core/src/api/llm.rsRepository: NVIDIA/NeMo-Relay
Length of output: 224
🏁 Script executed:
sed -n '420,590p' crates/core/src/codec/anthropic.rs
printf '\n--- llm.rs ---\n'
sed -n '470,560p' crates/core/src/api/llm.rsRepository: NVIDIA/NeMo-Relay
Length of output: 9740
🏁 Script executed:
rg -n 'encode\(&annotated|codec\.encode|project_llm_request_to_current_user_turn|request_history_with_hint' crates/core/srcRepository: NVIDIA/NeMo-Relay
Length of output: 961
🏁 Script executed:
sed -n '220,300p' crates/core/src/api/shared.rs
printf '\n--- python plugin ---\n'
sed -n '160,230p' crates/core/src/plugins/nemo_guardrails/python.rsRepository: NVIDIA/NeMo-Relay
Length of output: 4748
🏁 Script executed:
sed -n '180,280p' crates/core/src/api/shared.rs
sed -n '180,220p' crates/core/src/plugins/nemo_guardrails/python.rsRepository: NVIDIA/NeMo-Relay
Length of output: 4959
Preserve annotated.messages edits when raw messages are cached. In crates/core/src/codec/anthropic.rs:562-568, UNPARSED_MESSAGES_KEY is written back verbatim, so any rewrite to the normalized message list is dropped on re-encode. This path is used when rebuilding outbound requests and in guardrail rewrites, so conversations with unmodeled content blocks can silently bypass message mutations.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/core/src/codec/anthropic.rs` around lines 443 - 451, Update the
Anthropic re-encoding logic that handles UNPARSED_MESSAGES_KEY so normalized
message edits are preserved instead of writing the cached raw messages verbatim.
When rebuilding the payload, merge or apply mutations from the current
normalized messages to the cached unparsed content, while retaining unsupported
provider-native blocks; reference the parsing logic producing unparsed_messages
and the corresponding encode path around UNPARSED_MESSAGES_KEY.
| // Preserve valid provider messages that the normalized schema does not | ||
| // yet model (for example, developer-role or future content parts). | ||
| let (messages, unparsed_messages) = match obj.get("messages") { | ||
| Some(raw) => match serde_json::from_value(raw.clone()) { | ||
| Ok(messages) => (messages, None), | ||
| Err(_) => (Vec::new(), Some(raw.clone())), | ||
| }, | ||
| None => (Vec::new(), None), | ||
| }; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Same discard-on-unparsed encode() pattern as anthropic.rs.
Identical concern: when extra[CHAT_UNPARSED_MESSAGES_KEY] is present, encode() (lines 342-346) writes the raw messages verbatim, ignoring any edits made to annotated.messages. Given OpenAI's "developer" role is a documented, real message role and this codec's typed Message decode appears to fail wholesale on any array containing it (per request_projection_preserves_instruction_only_context in the test file), conversations using developer-role messages would always bypass intercept-driven message rewrites on re-encode.
Also applies to: 301-308, 342-346, 389-392
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/core/src/codec/openai_chat.rs` around lines 248 - 256, Update OpenAI
chat codec encode handling so `extra[CHAT_UNPARSED_MESSAGES_KEY]` does not cause
`annotated.messages` edits to be discarded. In `encode()`, merge or reconstruct
the raw unparsed entries with the current typed messages while preserving
unsupported provider message content, and ensure the same behavior covers the
related decode/projection paths around `messages` and `unparsed_messages`. Add
or update coverage for developer-role messages and verify intercept-driven
rewrites persist after re-encoding.
mnajafian-nv
left a comment
mnajafian-nv
left a comment
There was a problem hiding this comment.
Great work! Conditional approval pending addressing the last comment!
2 participants
Overview
Warning
BREAKING CHANGE (Behavioral): LLM event contents intentionally use a snapshot-and-delta history contract.
The first LLM start in each agent scope and the first start after compaction contain the complete sanitized request history. Other starts and every end-event history contain only the current user turn. Consumers must not assume that every LLM event contains the complete conversation.
Implement RELAY-445 by bounding ordinary emitted LLM histories while retaining full request baselines at agent and compaction boundaries. Provider execution, middleware inputs, caller-visible values, and public APIs remain unchanged.
Details
compactionmark and normalize bothPreCompactandPostCompactcoding-agent hooks to that mark.PreCompactto baseline a compaction model request when it passes through Relay, then usePostCompactto re-arm the first normal request containing the compacted history. If no LLM call occurs between the hooks, the repeated re-arm remains idempotent and only the next request receives the full baseline.Validation:
cargo fmt --alljust test-rustcargo clippy --workspace --all-targets -- -D warningsjust test-python— 496 passedjust test-gojust test-node— 246 passedjust docs— passed with zero errors; remote redirect comparison was skipped after FDR returned HTTP 403uv run pre-commit run --all-filesWhere should the reviewer start?
Start with
crates/core/src/api/runtime/scope_stack.rsfor per-agent baseline state,crates/core/src/api/runtime/state.rsfor compaction re-arming at central event emission, andcrates/core/src/api/llm.rsfor the start-event snapshot-or-delta decision.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
messages/inputwhen decoding fails and round-trip them safely.