Skip to content

feat: Add LiteLLM provider instrumentation for cross-provider LLM tracking#1410

Closed
icohangar-ops wants to merge 1 commit into
AgentOps-AI:mainfrom
icohangar-ops:fix/litellm-provider-tracking
Closed

feat: Add LiteLLM provider instrumentation for cross-provider LLM tracking#1410
icohangar-ops wants to merge 1 commit into
AgentOps-AI:mainfrom
icohangar-ops:fix/litellm-provider-tracking

Conversation

@icohangar-ops

Copy link
Copy Markdown

Description

This PR adds a new litellm provider instrumentation module to AgentOps that wraps litellm.completion() and litellm.acompletion() directly at the LiteLLM entry point. This ensures LLM calls are tracked regardless of which underlying provider LiteLLM routes to.

Problem

When using LiteLLM with litellm.completion(model="anthropic/claude-3-5-sonnet-..."), LiteLLM uses its own internal Anthropic handler that makes direct HTTP calls. It does NOT call the official anthropic Python SDK, so AgentOps' Anthropic instrumentor cannot capture those calls. The same issue affects other non-OpenAI providers like Bedrock, Gemini, Cohere, etc.

For OpenAI models, this works because LiteLLM internally uses the openai Python SDK, which AgentOps' OpenAI instrumentor wraps successfully.

Solution

Add LiteLLM as its own provider in AgentOps' instrumentation system. Instead of wrapping individual provider SDKs, the LiteLLM instrumentor wraps the LiteLLM entry points (completion() and acompletion()) directly, using attribute extraction that handles:

  • Provider detection: Extracts provider from model strings (e.g. "anthropic/..." → provider="anthropic")
  • Standard ModelResponse format: Normalized by LiteLLM for most providers
  • Responses API format: Handles the OpenAI /v1/responses response structure
  • Streaming: Both sync and async streaming with chunk aggregation
  • Token usage: Handles both OAI-style (prompt_tokens) and Anthropic-style (input_tokens) usage formats

Files Changed

  • agentops/instrumentation/providers/litellm/__init__.py - Module init
  • agentops/instrumentation/providers/litellm/attributes.py - Attribute extraction for LiteLLM requests/responses
  • agentops/instrumentation/providers/litellm/instrumentor.py - Instrumentor class with sync/async wrappers
  • agentops/instrumentation/__init__.py - Register litellm in PROVIDERS dict

Related Issues

Fixes #1079

@icohangar-ops icohangar-ops closed this by deleting the head repository Jun 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LiteLLM + AgentOps integration: Provider-specific tracking issue with Anthropic models

2 participants