Skip to content

[V1.4.1] changing model (not provider) during a task wipes session caches and history #380

Description

@ahmad-ajmal

What happened:
Changing the LLM model in Settings while a task is running breaks caching for that task. Even when the provider stays the same and only the model name changes, all accumulated per-task cache state is discarded — cache hit rates drop to zero and the cache has to re-warm from scratch mid-task. On providers with stateless chat-completions APIs, the wiped session history is also the task's conversational memory, so the impact is worse than cost: the in-flight task can lose earlier context.

What I expected:
A model-only change within the same provider should not destroy per-task session state. The session message histories are provider-agnostic message lists that the new model can consume as-is — the prefix cache would take one miss and re-warm, but accumulated history (and correctness for stateless providers) should be preserved. A full reset is only justified when the provider actually changes.

Steps to reproduce:

  1. Start a long-running task so the session cache warms up (watch for [CACHE METRICS] ... HIT lines in the logs).
  2. While the task is running, open Settings → Models and change only the model (keep the same provider). Save.
  3. Observe the logs: [LLM] Reinitializing with provider: ... fires, and subsequent calls for the running task show cache misses / rebuilt session history. On BytePlus, chat_with_session can raise No session cache found for <task_id>:<call_type>.

Environment: Browser UI, macOS, commit 07ddba44, v0.1.0

Logs / screenshots:
[LLM] Reinitializing with provider: <same provider>, model: <new model> followed by [CACHE METRICS] ... MISS for a task that previously had hits.

Root cause (from reading the code):

  1. The frontend always includes llmProvider in the model_settings_update payload, even when only the model changed (src/pages/Settings/ModelSettings.tsx:254).
  2. The handler calls agent.reinitialize_llm(new_provider) whenever llmProvider is present — it does not distinguish "provider changed" from "model changed within the same provider" (app/ui_layer/adapters/browser_adapter.py:5831).
  3. LLMInterface.reinitialize() unconditionally resets _session_system_prompts, all per-provider session message histories (_anthropic_session_messages, _openai_compat_session_messages, etc.), and replaces the BytePlus/Gemini cache managers — dropping the BytePlus _session_cache_registry keyed by task_id:call_type (agent_core/core/impl/llm/interface.py:332-346).

So any Settings save — including a no-op save — nukes all mid-task cache state.

Suggested fix direction:
In reinitialize(), compare the old (provider, model) against the new one:

  • Provider changed → full reset (current behavior is correct).
  • Model-only change → keep _session_system_prompts and the session message histories (they are plain message lists usable by the new model); recreate the BytePlus/Gemini cache managers with the new model since their server-side caches are model-bound.
  • Nothing changed → skip the reset entirely, so an unrelated Settings save doesn't disturb running tasks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions