feat: add streaming support to OCI Generative AI provider#4961
feat: add streaming support to OCI Generative AI provider#4961fede-kamel wants to merge 7 commits into
Conversation
a2fdc11 to
1562c65
Compare
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add multimodal content handling for generic model families: - vision.py: model lists, data URI helpers, image encoding utilities - _build_generic_content handles image_url, document_url, video_url, audio_url content types mapped to OCI SDK content objects - _message_has_multimodal_content detects non-text payloads - Cohere models reject multimodal with clear error message - supports_multimodal() returns True Depends on: crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add multimodal content handling for generic model families: - vision.py: model lists, data URI helpers, image encoding utilities - _build_generic_content handles image_url, document_url, video_url, audio_url content types mapped to OCI SDK content objects - _message_has_multimodal_content detects non-text payloads - Cohere models reject multimodal with clear error message - supports_multimodal() returns True Depends on: crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Add OCI embedding support integrated with CrewAI's RAG pipeline: - OCIEmbeddingFunction: ChromaDB-compatible embedding callable with batching, config serialization, image embedding support - OCIProvider: Pydantic-based provider with alias validation for env vars and config keys - Factory registration in embeddings/factory.py + types.py - Supports text and image embeddings, output dimensions, custom endpoints, all 4 OCI auth modes Tested live against cohere.embed-english-v3.0 with API_KEY auth. Depends on: crewAIInc#4964, crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add multimodal content handling for generic model families: - vision.py: model lists, data URI helpers, image encoding utilities - _build_generic_content handles image_url, document_url, video_url, audio_url content types mapped to OCI SDK content objects - _message_has_multimodal_content detects non-text payloads - Cohere models reject multimodal with clear error message - supports_multimodal() returns True Depends on: crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Add OCI embedding support integrated with CrewAI's RAG pipeline: - OCIEmbeddingFunction: ChromaDB-compatible embedding callable with batching, config serialization, image embedding support - OCIProvider: Pydantic-based provider with alias validation for env vars and config keys - Factory registration in embeddings/factory.py + types.py - Supports text and image embeddings, output dimensions, custom endpoints, all 4 OCI auth modes Tested live against cohere.embed-english-v3.0 with API_KEY auth. Depends on: crewAIInc#4964, crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Replace asyncio.to_thread wrappers with true async I/O using aiohttp for acall() and astream(). The OCI SDK is sync-only, so we bypass it for HTTP and use its signer for request authentication directly. - oci_async.py: OCIAsyncClient with aiohttp, OCI request signing, native SSE parsing, connection pooling - acall(): true async chat completion (no thread pool) - astream(): true async SSE streaming (no thread+queue bridge) - Graceful fallback to asyncio.to_thread when aiohttp unavailable or client is mocked (unit tests) - aiohttp + certifi added to crewai[oci] optional deps Temporary measure until OCI SDK ships native async support. Tested live: acall, astream, and concurrent acall against meta.llama-3.3-70b-instruct with API_KEY auth. Depends on: crewAIInc#4966, crewAIInc#4964, crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
|
OCI GenAI PR series (tracking issue: #4944)
Depends on #4959. |
25c696e to
fa07670
Compare
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add multimodal content handling for generic model families: - vision.py: model lists, data URI helpers, image encoding utilities - _build_generic_content handles image_url, document_url, video_url, audio_url content types mapped to OCI SDK content objects - _message_has_multimodal_content detects non-text payloads - Cohere models reject multimodal with clear error message - supports_multimodal() returns True Depends on: crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add multimodal content handling for generic model families: - vision.py: model lists, data URI helpers, image encoding utilities - _build_generic_content handles image_url, document_url, video_url, audio_url content types mapped to OCI SDK content objects - _message_has_multimodal_content detects non-text payloads - Cohere models reject multimodal with clear error message - supports_multimodal() returns True Depends on: crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Add OCI embedding support integrated with CrewAI's RAG pipeline: - OCIEmbeddingFunction: ChromaDB-compatible embedding callable with batching, config serialization, image embedding support - OCIProvider: Pydantic-based provider with alias validation for env vars and config keys - Factory registration in embeddings/factory.py + types.py - Supports text and image embeddings, output dimensions, custom endpoints, all 4 OCI auth modes Tested live against cohere.embed-english-v3.0 with API_KEY auth. Depends on: crewAIInc#4964, crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Replace asyncio.to_thread wrappers with true async I/O using aiohttp for acall() and astream(). The OCI SDK is sync-only, so we bypass it for HTTP and use its signer for request authentication directly. - oci_async.py: OCIAsyncClient with aiohttp, OCI request signing, native SSE parsing, connection pooling - acall(): true async chat completion (no thread pool) - astream(): true async SSE streaming (no thread+queue bridge) - Graceful fallback to asyncio.to_thread when aiohttp unavailable or client is mocked (unit tests) - aiohttp + certifi added to crewai[oci] optional deps Temporary measure until OCI SDK ships native async support. Tested live: acall, astream, and concurrent acall against meta.llama-3.3-70b-instruct with API_KEY auth. Depends on: crewAIInc#4966, crewAIInc#4964, crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
- OCICompletion(BaseLLM): sync call() and async acall() for generic (Meta, Google, OpenAI, xAI) and Cohere model families - Shared OCI auth utilities (utilities/oci.py): API key, security token, instance principal, and resource principal auth - Provider routing in llm.py: oci/ prefix and OCI model-id patterns - oci registered as optional dependency (crewai[oci]) - Configurable timeout via DEFAULT_OCI_TIMEOUT constant - Cohere and generic request/response paths fully separated Tracking issue: crewAIInc#4944 Part 1 of series: crewAIInc#4959 → crewAIInc#4961 → crewAIInc#4962 → crewAIInc#4963 → crewAIInc#4964 → crewAIInc#4966 → crewAIInc#4982
a6a950d to
db77428
Compare
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add response_model (Pydantic) support for structured output: - _build_response_format converts Pydantic schema to OCI JsonSchemaResponseFormat (generic) or CohereResponseJsonFormat - _parse_structured_response validates and returns typed models - response_model threaded through call, _call_impl, _stream_call_impl, and _handle_tool_calls for full coverage - Handles JSON in markdown fences via base class _validate_structured_output Tested live against meta.llama-3.3-70b-instruct and google.gemini-2.5-flash. Depends on: crewAIInc#4962 (tool calling), crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
Add multimodal content handling for generic model families: - vision.py: model lists, data URI helpers, image encoding utilities - _build_generic_content handles image_url, document_url, video_url, audio_url content types mapped to OCI SDK content objects - _message_has_multimodal_content detects non-text payloads - Cohere models reject multimodal with clear error message - supports_multimodal() returns True Depends on: crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Add OCI embedding support integrated with CrewAI's RAG pipeline: - OCIEmbeddingFunction: ChromaDB-compatible embedding callable with batching, config serialization, image embedding support - OCIProvider: Pydantic-based provider with alias validation for env vars and config keys - Factory registration in embeddings/factory.py + types.py - Supports text and image embeddings, output dimensions, custom endpoints, all 4 OCI auth modes Tested live against cohere.embed-english-v3.0 with API_KEY auth. Depends on: crewAIInc#4964, crewAIInc#4963, crewAIInc#4962, crewAIInc#4961, crewAIInc#4959 Tracking issue: crewAIInc#4944
Add native function calling for generic and Cohere model families: - _format_tools converts CrewAI tool specs to OCI SDK format - _extract_tool_calls normalizes responses back to CrewAI shape - _handle_tool_calls executes tools and recurses until model finishes - Cohere tool message handling with trailing tool results - Tool choice control (auto/none/required/function) - Passthrough parameter filtering via SDK introspection - Streaming tool call accumulation from SSE fragments - supports_function_calling() returns True Tested live against meta.llama-3.3-70b-instruct with raw tool call return and recursive tool execution. Depends on: crewAIInc#4961 (streaming), crewAIInc#4959 (basic text) Tracking issue: crewAIInc#4944
📝 WalkthroughWalkthroughAdds OCI as a native CrewAI LLM provider with optional SDK installation, configurable authentication, generic and Cohere request handling, synchronous/asynchronous completion, streaming, metadata tracking, and unit/live integration tests. ChangesOCI provider integration
Sequence Diagram(s)sequenceDiagram
participant CrewAILLM
participant OCICompletion
participant OCIClient
participant AsyncQueue
CrewAILLM->>OCICompletion: call or astream with messages
OCICompletion->>OCICompletion: normalize messages and build request
OCICompletion->>OCIClient: execute chat or streaming request
OCIClient-->>OCICompletion: response or streaming events
OCICompletion->>AsyncQueue: forward asynchronous chunks
OCICompletion-->>CrewAILLM: completion text, chunks, and metadata
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This PR is stale because it has been open for 45 days with no activity. |
- OCICompletion(BaseLLM): sync call() and async acall() for generic (Meta, Google, OpenAI, xAI) and Cohere model families - Shared OCI auth utilities (utilities/oci.py): API key, security token, instance principal, and resource principal auth - Provider routing in llm.py: oci/ prefix and OCI model-id patterns - oci registered as optional dependency (crewai[oci]) - Configurable timeout via DEFAULT_OCI_TIMEOUT constant - Cohere and generic request/response paths fully separated Tracking issue: crewAIInc#4944 Part 1 of series: crewAIInc#4959 → crewAIInc#4961 → crewAIInc#4962 → crewAIInc#4963 → crewAIInc#4964 → crewAIInc#4966 → crewAIInc#4982
- 15 unit tests (mocked OCI SDK, no credentials needed) - 2 parametrized integration tests across 4 model families: meta.llama-3.3-70b-instruct, cohere.command-r-plus-08-2024, google.gemini-2.5-flash, openai.gpt-5.2-chat-latest - conftest.py with shared fixtures for both test suites
Previously: any `openai.*` model on the GENERIC api_format route was sent `max_completion_tokens` instead of `max_tokens`. That works for the GPT-5 family (which OCI requires it on with HTTP 400 if you send `max_tokens`), but is inconsistent with `_is_openai_gpt5_family()` already used elsewhere in the same method for temperature and stop. Switch the param-name routing to the same predicate so GPT-4o / GPT-4.1 keep the standard `max_tokens` field, while only `openai.gpt-5*` gets `max_completion_tokens`. Also reword the test pair to reflect the two distinct behaviours. Verified live against OCI us-chicago-1 — all 7 model families pass: openai.gpt-5.5 ok (max_completion_tokens) openai.gpt-5 ok (max_completion_tokens) openai.gpt-4o Ok. (max_tokens) openai.gpt-4.1 ok (max_tokens) meta.llama-3.3-70b-instruct ok meta.llama-4-scout-17b-16e-instruct ok cohere.command-latest ok (CohereChatRequest)
Add reasoning_effort: str | None = None as a constructor kwarg. When set, it's passed as `reasoning_effort` on the OCI GenericChatRequest — honoured by GPT-5 family, Gemini 2.5, Grok reasoning variants, and Cohere Command-A-Reasoning; ignored by non-reasoning models. Single biggest cost knob for reasoning-capable models on OCI: setting "LOW" typically cuts reasoning-token spend 5-10×. Other GenericChatRequest fields (verbosity, parallel_tool_calls, logit_bias, n, metadata, etc.) aren't exposed — they either duplicate existing controls or aren't missing primitives.
Coderabbit's pre-merge check flagged 45.10% docstring coverage on this PR; required threshold is 80%. Add concise docstrings for the remaining 14 methods on OCICompletion (init, provider helpers, message helpers, response extractors, call/acall, _chat). Coverage now 100% per ast walk; no behavioural change.
Drop unreachable os.getenv defaults in the live-config fixture and fix the pytest path in the run instructions to include the lib/crewai/ prefix.
Add streaming text completion via OCI SSE events: - stream=True in call() routes to _stream_call_impl with chunk events - iter_stream() yields raw text chunks (sync generator) - astream() wraps iter_stream via thread+queue for async callers - _stream_chat_events holds client lock for full stream duration - SSE event parsing handles both string and mapping payloads Tested live against meta.llama-3.3-70b-instruct, cohere.command-r-plus-08-2024, google.gemini-2.5-flash, and openai.gpt-5.2-chat-latest. Depends on: crewAIInc#4959 Tracking issue: crewAIInc#4944
|
Rebased onto current main and un-drafted — mergeable again. Streaming verified live against OCI on-demand (us-chicago-1), including SSE accumulation and usage metadata. Unit suite: 20 passed. |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
lib/crewai/src/crewai/llms/providers/oci/completion.py (3)
209-231: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winUnsupported message roles are silently dropped.
Messages whose role isn't
user/system/assistantare skipped in both_build_generic_messagesand_build_cohere_chat_history, logged only atdebuglevel (invisible under default logging config). If upstream agent/task flows ever appendtool-role entries to conversation history before native tool-calling support lands for OCI, those messages vanish without any visible signal, silently losing context sent to the model.Also applies to: 233-260
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/src/crewai/llms/providers/oci/completion.py` around lines 209 - 231, Unsupported roles in _build_generic_messages and _build_cohere_chat_history are dropped with only debug logging, potentially losing conversation context. Update both methods to emit a visible warning when skipping unsupported roles, including the role and enough message context to diagnose the issue; preserve the existing supported-role mapping and skip behavior until OCI tool-role support is implemented.
636-712: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
response_modelis accepted but silently ignored.
call()/acall()acceptresponse_model: type[BaseModel] | Nonebut never reference it — passing it is a silent no-op that always returns plain text instead of a parsed model or a clear error. Given structured output is explicitly deferred to a later PR in this series (#4963), consider failing fast now rather than silently ignoring the argument, to avoid confusing callers who expect enforcement.♻️ Suggested guard
) -> str | BaseModel | list[dict[str, Any]]: + if response_model is not None: + raise NotImplementedError( + "response_model is not yet supported by the OCI native provider." + ) with llm_call_context():🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/src/crewai/llms/providers/oci/completion.py` around lines 636 - 712, Fail fast when a non-None response_model is supplied to call(), rather than silently ignoring it; add the same validation to acall() or ensure it is enforced before delegation, using a clear NotImplementedError or ValueError stating structured outputs are unsupported. Keep the existing behavior unchanged when response_model is None and reference the call() and acall() methods.
593-635: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy lift
astreambackground thread has no cancellation path on early abandonment.If the caller stops consuming the async generator early (e.g.
break, timeout, cancellation),_producer's background thread keeps callingiter_streamto completion — continuing to hit the OCI API and (once the locking gap above is fixed) holdingself._client_lockfor the remaining stream duration, blocking any othercall()/astream()on the same instance until the orphaned stream finishes server-side.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/src/crewai/llms/providers/oci/completion.py` around lines 593 - 635, Update astream and its _producer in the OCI provider to support cancellation when the async generator is closed, cancelled, or abandoned: create a stop event, have _producer and iter_stream consumption check it and stop promptly, and ensure generator cleanup signals the event, closes/cancels the underlying stream if supported, and joins or otherwise releases the worker without blocking the event loop. Preserve normal completion and exception propagation while preventing orphaned OCI requests and locks.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai/src/crewai/llms/providers/oci/completion.py`:
- Around line 347-357: Update _extract_usage and
_extract_usage_from_stream_event to read OCI reasoning-token usage from
reasoningTokens or completionTokensDetails.reasoningTokens, and include it as
reasoning_tokens when present. Preserve existing prompt, completion, and total
token extraction and ensure the resulting usage data reaches
_track_token_usage_internal for both non-streaming and streaming responses.
- Around line 555-591: Update iter_stream to hold self._client_lock throughout
response event consumption by reusing _stream_chat_events(chat_details), rather
than calling _chat and iterating response.data.events() outside the lock.
Preserve the existing event parsing, usage tracking, metadata updates, and final
response metadata assignment while consuming events from the locked generator;
apply the same locking correction to the related streaming paths identified by
_stream_chat_events, iter_stream, and the referenced call sites.
In `@lib/crewai/src/crewai/utilities/oci.py`:
- Around line 41-54: The SECURITY_TOKEN branch currently discards configured
passphrases when loading the private key. Update the private-key loading call in
the authentication logic to pass config.get("pass_phrase") instead of None,
preserving support for encrypted OCI session keys.
---
Nitpick comments:
In `@lib/crewai/src/crewai/llms/providers/oci/completion.py`:
- Around line 209-231: Unsupported roles in _build_generic_messages and
_build_cohere_chat_history are dropped with only debug logging, potentially
losing conversation context. Update both methods to emit a visible warning when
skipping unsupported roles, including the role and enough message context to
diagnose the issue; preserve the existing supported-role mapping and skip
behavior until OCI tool-role support is implemented.
- Around line 636-712: Fail fast when a non-None response_model is supplied to
call(), rather than silently ignoring it; add the same validation to acall() or
ensure it is enforced before delegation, using a clear NotImplementedError or
ValueError stating structured outputs are unsupported. Keep the existing
behavior unchanged when response_model is None and reference the call() and
acall() methods.
- Around line 593-635: Update astream and its _producer in the OCI provider to
support cancellation when the async generator is closed, cancelled, or
abandoned: create a stop event, have _producer and iter_stream consumption check
it and stop promptly, and ensure generator cleanup signals the event,
closes/cancels the underlying stream if supported, and joins or otherwise
releases the worker without blocking the event loop. Preserve normal completion
and exception propagation while preventing orphaned OCI requests and locks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 396494e4-4357-474f-87ae-5a65ea12defc
📒 Files selected for processing (11)
lib/crewai/pyproject.tomllib/crewai/src/crewai/llm.pylib/crewai/src/crewai/llms/providers/oci/__init__.pylib/crewai/src/crewai/llms/providers/oci/completion.pylib/crewai/src/crewai/utilities/oci.pylib/crewai/tests/llms/oci/__init__.pylib/crewai/tests/llms/oci/conftest.pylib/crewai/tests/llms/oci/test_oci.pylib/crewai/tests/llms/oci/test_oci_integration_basic.pylib/crewai/tests/llms/oci/test_oci_integration_streaming.pylib/crewai/tests/llms/oci/test_oci_streaming.py
| def _extract_usage(self, response: Any) -> dict[str, int]: | ||
| """Return ``{prompt_tokens, completion_tokens, total_tokens}`` from the OCI response, or an empty dict if unavailable.""" | ||
| chat_response = response.data.chat_response | ||
| usage = getattr(chat_response, "usage", None) | ||
| if usage is None: | ||
| return {} | ||
| return { | ||
| "prompt_tokens": getattr(usage, "prompt_tokens", 0), | ||
| "completion_tokens": getattr(usage, "completion_tokens", 0), | ||
| "total_tokens": getattr(usage, "total_tokens", 0), | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Reasoning-token usage is never extracted, despite being a stated PR feature.
Neither _extract_usage nor _extract_usage_from_stream_event reads a reasoningTokens/completionTokensDetails.reasoningTokens field from OCI responses — both only surface prompt_tokens/completion_tokens/total_tokens. This contradicts the PR objectives ("Accumulation of reasoning-token usage details") and the author's own comment that streaming "accumulates reasoningTokens for reasoning-enabled models." Since reasoning_effort is sent on the request (line 303-304) for reasoning-capable models (GPT-5, Gemini 2.5, Cohere Command-A-Reasoning per the inline comment), but the corresponding usage accounting is dropped, callers get no visibility into reasoning-token spend for these models, and _track_token_usage_internal never sees a reasoning_tokens key to propagate downstream.
Also applies to: 428-436
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai/src/crewai/llms/providers/oci/completion.py` around lines 347 -
357, Update _extract_usage and _extract_usage_from_stream_event to read OCI
reasoning-token usage from reasoningTokens or
completionTokensDetails.reasoningTokens, and include it as reasoning_tokens when
present. Preserve existing prompt, completion, and total token extraction and
ensure the resulting usage data reaches _track_token_usage_internal for both
non-streaming and streaming responses.
| def iter_stream( | ||
| self, | ||
| messages: str | list[LLMMessage], | ||
| tools: list[dict[str, BaseTool]] | None = None, | ||
| callbacks: list[Any] | None = None, | ||
| available_functions: dict[str, Any] | None = None, | ||
| from_task: Task | None = None, | ||
| from_agent: Agent | None = None, | ||
| ) -> Any: | ||
| """Yield raw text chunks from OCI without triggering tool recursion.""" | ||
| normalized_messages = self._normalize_messages(messages) | ||
| chat_request = self._build_chat_request(normalized_messages, is_stream=True) | ||
| chat_details = self._oci.generative_ai_inference.models.ChatDetails( | ||
| compartment_id=self.compartment_id, | ||
| serving_mode=self._build_serving_mode(), | ||
| chat_request=chat_request, | ||
| ) | ||
| response = self._chat(chat_details) | ||
| usage_data: dict[str, int] = {} | ||
| response_metadata: dict[str, Any] = {} | ||
|
|
||
| for event in response.data.events(): | ||
| event_data = self._parse_stream_event(event) | ||
| if not event_data: | ||
| continue | ||
| text_chunk = self._extract_text_from_stream_event(event_data) | ||
| if text_chunk: | ||
| yield text_chunk | ||
| usage_chunk = self._extract_usage_from_stream_event(event_data) | ||
| if usage_chunk: | ||
| usage_data = usage_chunk | ||
| response_metadata.update(self._extract_metadata_from_stream_event(event_data)) | ||
|
|
||
| if usage_data: | ||
| self._track_token_usage_internal(usage_data) | ||
| response_metadata["usage"] = usage_data | ||
| self.last_response_metadata = response_metadata or None |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
iter_stream doesn't hold the client lock for the full streaming duration.
_stream_chat_events (used by _stream_call_impl) correctly holds self._client_lock for the entire yield from response.data.events():
def _stream_chat_events(self, chat_details: Any) -> Any:
with self._client_lock:
response = self.client.chat(chat_details)
yield from response.data.events()But iter_stream (which astream() is built on) instead calls self._chat(chat_details), which releases the lock immediately after client.chat() returns, and then iterates response.data.events() completely unlocked:
response = self._chat(chat_details) # lock released right after this returns
...
for event in response.data.events(): # unlocked iterationSince _chat's own docstring states the OCI client "is not re-entrant," this means a concurrent call()/another astream()/iter_stream() on the same OCICompletion instance can invoke self.client while a previous iter_stream/astream call is still consuming its event stream — exactly the race condition the lock was introduced to prevent. This directly contradicts the PR's stated "ordered client locking for the full streaming duration" objective.
🔒 Proposed fix — reuse `_stream_chat_events` for lock consistency
def iter_stream(
self,
messages: str | list[LLMMessage],
tools: list[dict[str, BaseTool]] | None = None,
callbacks: list[Any] | None = None,
available_functions: dict[str, Any] | None = None,
from_task: Task | None = None,
from_agent: Agent | None = None,
) -> Any:
"""Yield raw text chunks from OCI without triggering tool recursion."""
normalized_messages = self._normalize_messages(messages)
chat_request = self._build_chat_request(normalized_messages, is_stream=True)
chat_details = self._oci.generative_ai_inference.models.ChatDetails(
compartment_id=self.compartment_id,
serving_mode=self._build_serving_mode(),
chat_request=chat_request,
)
- response = self._chat(chat_details)
usage_data: dict[str, int] = {}
response_metadata: dict[str, Any] = {}
- for event in response.data.events():
+ for event in self._stream_chat_events(chat_details):
event_data = self._parse_stream_event(event)Also applies to: 723-727, 718-721
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai/src/crewai/llms/providers/oci/completion.py` around lines 555 -
591, Update iter_stream to hold self._client_lock throughout response event
consumption by reusing _stream_chat_events(chat_details), rather than calling
_chat and iterating response.data.events() outside the lock. Preserve the
existing event parsing, usage tracking, metadata updates, and final response
metadata assignment while consuming events from the locked generator; apply the
same locking correction to the related streaming paths identified by
_stream_chat_events, iter_stream, and the referenced call sites.
| elif auth_type_upper == "SECURITY_TOKEN": | ||
| config = oci.config.from_file( | ||
| file_location=auth_file_location, | ||
| profile_name=auth_profile, | ||
| ) | ||
| key_file = config["key_file"] | ||
| security_token_file = config["security_token_file"] | ||
| private_key = oci.signer.load_private_key_from_file(key_file, None) | ||
| with open(security_token_file, encoding="utf-8") as file: | ||
| security_token = file.read() | ||
| client_kwargs["config"] = config | ||
| client_kwargs["signer"] = oci.auth.signers.SecurityTokenSigner( | ||
| security_token, private_key | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file and nearby context
file="lib/crewai/src/crewai/utilities/oci.py"
wc -l "$file"
sed -n '1,220p' "$file"
# Search for OCI config/pass_phrase usage in the repo
rg -n "pass_phrase|load_private_key_from_file|SECURITY_TOKEN" lib/crewai/src -S
# If the OCI SDK is installed in the sandbox, inspect the signature/docs directly
python3 - <<'PY'
import inspect
try:
import oci
print("oci module:", oci.__file__)
print("load_private_key_from_file signature:", inspect.signature(oci.signer.load_private_key_from_file))
print("from_file signature:", inspect.signature(oci.config.from_file))
except Exception as e:
print("OCI SDK inspection unavailable:", repr(e))
PYRepository: crewAIInc/crewAI
Length of output: 3029
🌐 Web query:
OCI Python SDK load_private_key_from_file pass_phrase config key SecurityTokenSigner documentation
💡 Result:
In the OCI Python SDK, handling a private key with a passphrase is supported through the pass_phrase parameter, which is available in various signing and configuration methods [1][2][3]. load_private_key_from_file The oci.signer.load_private_key_from_file(filename, pass_phrase=None) function is used to load a private key from a file on disk [2][3]. If your private key is encrypted, you must provide the passphrase via the pass_phrase argument to successfully load and decrypt the key [2][3]. Configuration File When using the OCI configuration file (typically ~/.oci/config), you can specify the passphrase by adding a pass_phrase key to your profile [4][5]. The SDK automatically reads this value if present. This is particularly useful when using oci.config.from_file() to load your configuration, as the SDK handles the passphrase integration when creating signers [1][6]. Signer and SecurityTokenSigner - Signer: The standard oci.signer.Signer class accepts a pass_phrase argument in its constructor [2][6]. If you are creating a signer manually, you can pass the passphrase directly there [1][6]. - SecurityTokenSigner: The oci.auth.signers.SecurityTokenSigner is designed for scenarios where the API key is a token (such as for instance principals or service-to-service authentication) [7][8]. Unlike the standard Signer used for user-based authentication, SecurityTokenSigner expects the caller to provide the token and the pre-loaded private key directly [7]. If you are using this class, you would typically load the private key yourself using load_private_key_from_file (with the pass_phrase provided) before passing the key object to the SecurityTokenSigner constructor [2][3][7].
Citations:
- 1: https://oracle-cloud-infrastructure-python-sdk.readthedocs.io/en/latest/raw-requests.html
- 2: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html
- 3: https://oracle-cloud-infrastructure-python-sdk.readthedocs.io/en/latest/api/signing.html
- 4: https://docs.oracle.com/en-us/iaas/tools/python/latest/configuration.html
- 5: https://github.com/oracle/oci-python-sdk/blob/master/src/oci/config.py
- 6: https://github.com/oracle/oci-python-sdk/blob/master/src/oci/signer.py
- 7: https://github.com/oracle/oci-python-sdk/blob/9b6357cb/src/oci/auth/signers/security_token_signer.py
- 8: https://github.com/oracle/oci-python-sdk/blob/master/docs/api/signing.rst
Pass pass_phrase through for SECURITY_TOKEN private keys. load_private_key_from_file(key_file, None) drops support for encrypted OCI keys; use config.get("pass_phrase") so passphrase-protected session keys can authenticate.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 48-48: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(security_token_file, encoding="utf-8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai/src/crewai/utilities/oci.py` around lines 41 - 54, The
SECURITY_TOKEN branch currently discards configured passphrases when loading the
private key. Update the private-key loading call in the authentication logic to
pass config.get("pass_phrase") instead of None, preserving support for encrypted
OCI session keys.
There was a problem hiding this comment.
🧹 Nitpick comments (3)
lib/crewai/tests/llms/oci/test_oci_streaming.py (3)
20-20: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMove
import jsonto the top of the module.The
import jsonstatement inside_make_fake_stream_eventis non-idiomatic. Python best practices recommend top-of-file imports.♻️ Suggested refactor
from __future__ import annotations +import json from unittest.mock import MagicMock @@ - import json event = MagicMock()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/tests/llms/oci/test_oci_streaming.py` at line 20, Move the json import from inside _make_fake_stream_event to the module-level import section at the top of the file, and remove the nested import from the function.Source: Coding guidelines
51-53: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the
StreamOptionsmock into a shared fixture.The
StreamOptionsmock setup is duplicated across three tests (lines 51-53, 84-86, 114-116). Extracting it into a fixture (or extending the existingpatch_oci_modulefixture) would reduce duplication and ensure consistency.♻️ Suggested refactor
+# In conftest.py or a local fixture: +@pytest.fixture(autouse=True) +def mock_stream_options(patch_oci_module): + patch_oci_module.generative_ai_inference.models.StreamOptions = MagicMock( + side_effect=lambda **kw: MagicMock(**kw) + ) + # Then remove these blocks from each test: - patch_oci_module.generative_ai_inference.models.StreamOptions = MagicMock( - side_effect=lambda **kw: MagicMock(**kw) - )Also applies to: 84-86, 114-116
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/tests/llms/oci/test_oci_streaming.py` around lines 51 - 53, Extract the duplicated StreamOptions MagicMock setup from the tests into a shared fixture or extend the existing patch_oci_module fixture. Configure StreamOptions once with the current side_effect behavior, then have all three streaming tests reuse that fixture and remove their local setup.Source: Coding guidelines
129-158: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueLock test accesses internal implementation details.
test_oci_stream_chat_events_holds_client_lockdirectly tests_client_lockand_stream_chat_eventsrather than observable behavior. Per coding guidelines, unit tests should focus on behavior rather than implementation details. That said, lock-holding semantics are inherently internal, so this is a reasonable trade-off — just flagging for awareness.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/tests/llms/oci/test_oci_streaming.py` around lines 129 - 158, The lock test intentionally verifies internal lock-holding semantics, so retain it but clearly document that this is an accepted exception to behavior-focused testing guidelines. In test_oci_stream_chat_events_holds_client_lock, keep the assertions around _client_lock and _stream_chat_events, and add a concise comment or docstring explaining why these implementation details must be tested.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@lib/crewai/tests/llms/oci/test_oci_streaming.py`:
- Line 20: Move the json import from inside _make_fake_stream_event to the
module-level import section at the top of the file, and remove the nested import
from the function.
- Around line 51-53: Extract the duplicated StreamOptions MagicMock setup from
the tests into a shared fixture or extend the existing patch_oci_module fixture.
Configure StreamOptions once with the current side_effect behavior, then have
all three streaming tests reuse that fixture and remove their local setup.
- Around line 129-158: The lock test intentionally verifies internal
lock-holding semantics, so retain it but clearly document that this is an
accepted exception to behavior-focused testing guidelines. In
test_oci_stream_chat_events_holds_client_lock, keep the assertions around
_client_lock and _stream_chat_events, and add a concise comment or docstring
explaining why these implementation details must be tested.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 78010feb-b285-4624-a509-d5dd3cc86d39
📒 Files selected for processing (5)
lib/crewai/src/crewai/llms/providers/oci/completion.pylib/crewai/tests/llms/oci/conftest.pylib/crewai/tests/llms/oci/test_oci_integration_basic.pylib/crewai/tests/llms/oci/test_oci_integration_streaming.pylib/crewai/tests/llms/oci/test_oci_streaming.py
🚧 Files skipped from review as they are similar to previous changes (4)
- lib/crewai/tests/llms/oci/test_oci_integration_streaming.py
- lib/crewai/tests/llms/oci/conftest.py
- lib/crewai/tests/llms/oci/test_oci_integration_basic.py
- lib/crewai/src/crewai/llms/providers/oci/completion.py
Summary
stream=True)iter_stream()— sync generator yielding raw text chunksastream()— async generator wrapping sync stream via thread + queue_stream_chat_eventsholds ordered client lock for full stream durationDepends on #4959 (basic text completion provider). Draft until #4959 merges.
Tracking issue: #4944
Diff breakdown
completion.py(streaming methods)test_oci_streaming.py(4 unit tests)test_oci_integration_streaming.py(2 parametrized integration tests)Test plan
meta.llama-3.3-70b-instructcohere.command-r-plus-08-2024google.gemini-2.5-flashopenai.gpt-5.2-chat-latest