fix(agent): report per-call usage metrics on kickoff results

[joaomdmoura]

Summary

Fixes EPD-177: Agent.kickoff() populated result.usage_metrics from llm.get_token_usage_summary(), which returns the LLM instance's lifetime accumulator. Counts therefore grew across calls and pooled across agents sharing one LLM object — with a shared module-level LLM, a second agent's first turn appeared to "cost" the whole preceding session. Broken cost attribution and dashboards, hit by an enterprise customer in production migration.

Fix

• New UsageMetrics.delta_since(baseline): field-wise difference between two snapshots of the same accumulator, clamped at zero.
• Agent (sync + async kickoff): snapshot the accumulator right after kickoff preparation and thread the baseline through output building and the guardrail-retry path, so the result reports usage for that call only — including all guardrail retry attempts.
• LiteAgent (deprecated, same bug): snapshot at kickoff start (_kickoff_usage_baseline) and report deltas at both output-building sites.
• Cumulative counters are untouched: get_token_usage_summary() still returns instance-lifetime totals (crew-level aggregation in crew.py and the event-based flow aggregation depend on the existing semantics). Its docstring now states the instance-lifetime scope explicitly, and LiteAgentOutput.usage_metrics's field description documents the per-call semantics.

Known limitation (inherent to counter diffing): two kickoffs running concurrently against the same LLM instance can still attribute each other's overlapping usage; sequential calls — the reported scenario — are now exact.

Testing

• The customer's clean-room repro from EPD-177 now prints identical per-call usage (110 tokens) for shared-LLM and own-LLM agents, with the shared instance's lifetime counters still cumulative.
• New regression tests (TestKickoffUsageMetricsArePerCall, TestUsageMetricsDeltaSince): shared-LLM agents, repeated kickoffs on one agent, async kickoff, LiteAgent, plus delta_since field-wise math and negative-clamp.
• Green: tests/agents/test_lite_agent.py (35), tests/agents/test_agent.py (97), usage-related tests in test_crew.py / test_llm.py, test_flow_usage_metrics.py + events/test_llm_usage_event.py (44), ruff, and mypy.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes how kickoff results report token usage (observable API for dashboards/billing); cumulative LLM counters and crew aggregation paths are intentionally untouched but consumers that assumed cumulative kickoff metrics will see different numbers.

Overview
Agent.kickoff() / kickoff_async() no longer copy the LLM instance’s lifetime token counters into LiteAgentOutput.usage_metrics. They snapshot usage at kickoff start and set metrics to the delta for that call only (including all guardrail re-runs), via new UsageMetrics.delta_since().

get_token_usage_summary() behavior is unchanged (still cumulative); its docstring and LiteAgentOutput.usage_metrics now document per-call vs lifetime semantics.

Regression tests cover shared LLM, repeated kickoffs, async kickoff, guardrail retries, and delta_since math.

^{Reviewed by Cursor Bugbot for commit 4f6ec67. Bugbot is set up for automated code reviews on this repo. Configure here.}

[linear]

EPD-177

[coderabbitai]

📝 Walkthrough

Walkthrough

Agent kickoff usage is now measured relative to a kickoff-start snapshot. Synchronous, asynchronous, and guardrail-retry paths propagate this baseline, while UsageMetrics provides clamped deltas and tests cover per-call reporting.

Changes

Per-kickoff usage tracking

Layer / File(s)	Summary
Usage delta contract lib/crewai/src/crewai/types/usage_metrics.py, lib/crewai/src/crewai/llms/base_llm.py, lib/crewai/tests/agents/test_lite_agent.py	UsageMetrics.delta_since computes non-negative field-wise usage differences, with tests covering optional fields and counter resets. LLM documentation defines cumulative counters and per-call deltas.
Agent kickoff baseline propagation lib/crewai/src/crewai/agent/core.py, lib/crewai/src/crewai/lite_agent_output.py	Sync and async kickoffs snapshot usage, propagate the baseline through execution and guardrail retries, and report kickoff-relative metrics in output metadata.
Per-kickoff usage regression coverage lib/crewai/tests/agents/test_lite_agent.py	An offline fixed-usage LLM verifies per-invocation metrics across shared agents, repeated kickoffs, asynchronous kickoffs, and guardrail retries.

Sequence Diagram(s)

sequenceDiagram
  participant Agent
  participant LLM
  participant Guardrail
  participant UsageMetrics
  participant KickoffOutput
  Agent->>LLM: snapshot cumulative usage
  Agent->>LLM: execute kickoff
  LLM->>Guardrail: return result
  Guardrail->>LLM: retry when validation fails
  Agent->>LLM: snapshot final cumulative usage
  LLM->>UsageMetrics: calculate delta_since(baseline)
  UsageMetrics->>KickoffOutput: attach kickoff usage metrics

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 59.38% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: kickoff results now report per-call usage metrics.
Description check	✅ Passed	The description accurately describes the usage-metrics fix, the new delta logic, and the related tests.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch joao/epd-177-agentkickoff-usage-counters-are-cumulative-and-scoped-to-the

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (3)

lib/crewai/src/crewai/lite_agent.py (1)

293-293: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Baseline stored on self instead of threaded as a parameter.

Unlike Agent.kickoff() (core.py), which keeps usage_baseline as a local variable passed through the call chain, LiteAgent stores it on self._kickoff_usage_baseline. Since kickoff() already mutates other shared instance state without synchronization, this isn't a regression in isolation, but it's inconsistent with the safer pattern used elsewhere in this PR and would corrupt usage attribution if the same LiteAgent instance's kickoff()/kickoff_async() is invoked concurrently.

Consider threading the baseline through _execute_core(..., usage_baseline=...) as a parameter instead of an instance attribute, matching Agent's approach.

Also applies to: 521-524

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/crewai/src/crewai/lite_agent.py` at line 293, Replace the instance-level
_kickoff_usage_baseline state with a local baseline in LiteAgent.kickoff() and
kickoff_async(), and thread it explicitly through _execute_core(...,
usage_baseline=...) and any intermediate call sites, matching Agent’s
implementation. Remove the PrivateAttr declaration and update usage attribution
to read only from the passed parameter.

lib/crewai/src/crewai/agent/core.py (2)

1686-1696: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate logic with LiteAgent._current_usage_summary.

This method is byte-for-byte identical (code and docstring) to LiteAgent._current_usage_summary in lib/crewai/src/crewai/lite_agent.py (lines 551-561). Since Agent and LiteAgent don't share a common base for this concern, consider extracting a small shared helper (e.g. a module-level function taking llm and token_process) to avoid maintaining two copies.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/crewai/src/crewai/agent/core.py` around lines 1686 - 1696, Extract the
duplicated usage-summary logic from Agent._current_usage_summary and
LiteAgent._current_usage_summary into a shared module-level helper accepting the
LLM and token process, then update both methods to delegate to it while
preserving their UsageMetrics behavior and documentation as appropriate.

---

1611-1634: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a guardrail-retry usage_metrics regression test.

The retry path correctly threads usage_baseline through _process_kickoff_guardrail so retries report whole-call usage, but the new tests only cover sync/async/shared-LLM/LiteAgent flows — not a guardrail-retry scenario, which is the most usage-sensitive path here (each retry adds more LLM usage that must roll up into one delta).

🧪 Suggested test outline

def test_guardrail_retry_usage_includes_all_attempts(self):
    # LLM that fails guardrail once, succeeds second time
    ...
    result = agent.kickoff("question")
    # usage_metrics should reflect 2 LLM calls' worth of tokens, not just the last
    assert result.usage_metrics["successful_requests"] == 2

As per path instructions, **/tests/**/*.py should "Write unit tests for new functionality that focus on behavior rather than implementation details."

Also applies to: 1812-1888

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/crewai/src/crewai/agent/core.py` around lines 1611 - 1634, The
usage_metrics coverage is missing the guardrail-retry path. Add a
behavior-focused regression test in the relevant tests module that uses an LLM
returning an output rejected by the guardrail once and accepted on the second
attempt, then calls the agent kickoff flow and asserts result.usage_metrics
reflects both LLM calls, including successful_requests == 2; ensure the test
exercises the public behavior rather than internal implementation details.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@lib/crewai/src/crewai/agent/core.py`:
- Around line 1686-1696: Extract the duplicated usage-summary logic from
Agent._current_usage_summary and LiteAgent._current_usage_summary into a shared
module-level helper accepting the LLM and token process, then update both
methods to delegate to it while preserving their UsageMetrics behavior and
documentation as appropriate.
- Around line 1611-1634: The usage_metrics coverage is missing the
guardrail-retry path. Add a behavior-focused regression test in the relevant
tests module that uses an LLM returning an output rejected by the guardrail once
and accepted on the second attempt, then calls the agent kickoff flow and
asserts result.usage_metrics reflects both LLM calls, including
successful_requests == 2; ensure the test exercises the public behavior rather
than internal implementation details.

In `@lib/crewai/src/crewai/lite_agent.py`:
- Line 293: Replace the instance-level _kickoff_usage_baseline state with a
local baseline in LiteAgent.kickoff() and kickoff_async(), and thread it
explicitly through _execute_core(..., usage_baseline=...) and any intermediate
call sites, matching Agent’s implementation. Remove the PrivateAttr declaration
and update usage attribution to read only from the passed parameter.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 66707a74-f8ed-47f6-8bc0-c456d530540e

📥 Commits

Reviewing files that changed from the base of the PR and between 7baf8f9 and 439c32b.

📒 Files selected for processing (6)

• lib/crewai/src/crewai/agent/core.py
• lib/crewai/src/crewai/lite_agent.py
• lib/crewai/src/crewai/lite_agent_output.py
• lib/crewai/src/crewai/llms/base_llm.py
• lib/crewai/src/crewai/types/usage_metrics.py
• lib/crewai/tests/agents/test_lite_agent.py

[joaomdmoura]

Addressed the review:

• Dropped the lite_agent.py diff entirely per @lorenzejay — the per-call usage snapshot now lives only in agent/core.py. That also resolves both CodeRabbit nitpicks on that file (instance-attr baseline, duplicated _current_usage_summary helper).
• Added the guardrail-retry regression test CodeRabbit suggested: a guardrail rejecting the first attempt and accepting the second yields usage_metrics equal to exactly 2× a single-attempt kickoff (requests and tokens), confirming retries roll up into one per-call delta.

Re-verified the EPD-177 clean-room repro against the branch (Agent = core path): still fixed — shared-LLM and own-LLM agents report identical per-call usage while the shared instance keeps cumulative lifetime counters.