Skip to content

Trace-level LLM (GenAI) enrichment for OpenRouter OTel spans#202

Open
Makisuo wants to merge 1 commit into
mainfrom
claude/busy-planck-l24zkk
Open

Trace-level LLM (GenAI) enrichment for OpenRouter OTel spans#202
Makisuo wants to merge 1 commit into
mainfrom
claude/busy-planck-l24zkk

Conversation

@Makisuo

@Makisuo Makisuo commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Context

OpenRouter's Broadcast feature already ingests into Maple: it POSTs OTLP/HTTP-JSON traces to /v1/traces (documented in docs/openrouter-tracing.md) using the OTel gen_ai.* semantic conventions (gen_ai.operation.name, gen_ai.provider.name/legacy gen_ai.system, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens/output_tokens, gen_ai.usage.cost, gen_ai.response.finish_reasons). But those attributes landed in the generic SpanAttributes map with no LLM-aware rendering — this is roadmap item #13.

This PR closes that gap for trace-level LLM enrichment: open any trace with OpenRouter/LLM spans and immediately see which spans are LLM calls, on what model/provider, how many tokens, and at what cost — with prompts/completions inspectable inline and graceful handling of OpenRouter Privacy Mode. It is pure query-engine projection + frontend work — no ingest or storage changes.

What changed

  • Projection — added the gen_ai.* keys to TREE_SPAN_ATTR_KEYS (packages/query-engine/src/ch/queries/errors.ts) so the waterfall/timeline/flow views can badge LLM spans without the per-span lazy detail fetch.
  • Detector — new getGenAiInfo (packages/ui/src/lib/gen-ai.ts) normalizes gen_ai.* into one shape, coalescing gen_ai.systemgen_ai.provider.name and gen_ai.response.modelgen_ai.request.model, and tolerating Privacy Mode (absent prompt/completion/finish reasons). Detection keys on a real non-empty value, never key-presence (the trimmed-projection footgun).
  • Cross-view badge + detail summary — a new genAiAdapter in the cloud-platform registry (packages/ui/src/lib/cloud-platforms/gen-ai.ts) lights up the sparkle/provider badge across every trace view and the span-detail summary block (provider, operation, model, tokens, cost, params, finish reason) with no component edits, since both are driven by getCloudPlatform.
  • Filter aliasgen_ai.systemgen_ai.provider.name now coalesce in attribute filters (packages/query-engine/src/traces-shared.ts), so a provider filter matches either spelling.
  • Value formattinggen_ai.request.model/response.model render as model chips, token counts are grouped, and gen_ai.usage.cost renders as currency in the attribute table (apps/web/src/components/attributes/commit-sha-attribute.tsx).
  • Added SparklesIcon; updated docs/openrouter-tracing.md and roadmap Alerts #13.

Testing

  • packages/ui — 14 new unit tests for the detector, adapter, and formatters (full OpenRouter span, provider coalescing, empty-value rejection, Privacy Mode, finish-reason parsing, bad-outcome flagging, no shadowing of DB/non-LLM spans). All 103 ui-lib tests pass.
  • packages/query-engine — extended the span-hierarchy projection test to assert the gen_ai.* keys, and added a tracesTimeseriesQuery test asserting the gen_ai.provider.name/gen_ai.system coalescing compiles. All 105 ch tests pass.
  • turbo typecheck clean across @maple/ui, @maple/query-engine, @maple/web; oxfmt + oxlint clean.
  • Not run: full browser E2E (seed a gen_ai span → view enrichment); the pure logic is unit-covered and the React integration is typecheck-covered.

Follow-ups (deferred)

Aggregate "AI-first ops" surfaces — a dedicated AI page, token/cost dashboards + numeric-attribute aggregation, a gen_ai_usage_hourly rollup, MCP tools, and cost/budget alerts — are scoped in the plan for a phased follow-up (query-time first, typed rollup MV as a fast-follow).

🤖 Generated with Claude Code

https://claude.ai/code/session_01QdSppGz2eTovugvSVRQeBq


Generated by Claude Code


Open in Devin Review

OpenRouter Broadcast already ingests into Maple via /v1/traces using the OTel
gen_ai.* semantic conventions, but those attributes landed in the generic
SpanAttributes map with no LLM-aware rendering (roadmap #13). This makes every
LLM span read as a first-class LLM call across the trace views and span detail.

- Project the gen_ai.* keys into TREE_SPAN_ATTR_KEYS so the waterfall/timeline/
  flow views can badge LLM spans without the per-span lazy detail fetch.
- Add getGenAiInfo (packages/ui/src/lib/gen-ai.ts) normalizing gen_ai.* into a
  single shape, coalescing gen_ai.system -> gen_ai.provider.name and
  gen_ai.response.model -> gen_ai.request.model, and tolerating OpenRouter
  Privacy Mode (absent prompt/completion/finish reasons).
- Add a genAiAdapter to the cloud-platform registry so the sparkle/provider
  badge lights up across every trace view and the span-detail summary block
  (provider, operation, model, tokens, cost, params, finish reason) with no
  component edits.
- Coalesce gen_ai.system <-> gen_ai.provider.name in attribute filters
  (traces-shared.ts) so a provider filter matches either spelling.
- Format gen_ai model/token/cost values in the attribute table.
- Add SparklesIcon; unit tests for the detector, adapter, formatters, the
  provider-alias SQL, and the tree projection; update the OpenRouter tracing doc
  and roadmap #13.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdSppGz2eTovugvSVRQeBq
@pullfrog

pullfrog Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Your Pullfrog Router balance is exhausted.

You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

Pullfrog  | ⚠️ this action is pinned to a commit SHA, which freezes the cleanup step — switch to @v0 or keep the SHA fresh with Dependabot | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants