WikiGenerator.ExecuteAgentWithRetryAsync has no max-tool-call guard — a stuck-loop model burns MaxRetryAttempts × (TimeoutSeconds / per-call) paid LLM calls per generation

[Fr3ya]

WikiGenerator.ExecuteAgentWithRetryAsync at src/OpenDeepWiki/Services/Wiki/WikiGenerator.cs#L1127-L1265 wraps chatClient.RunStreamingAsync(...) in an outer retry loop (while (retryCount < _options.MaxRetryAttempts)). The streaming loop increments toolCallCount on every observed tool-call update but never checks it against a cap:

while (retryCount < _options.MaxRetryAttempts)        // outer retry
{
    ...
    var toolCallCount = 0;
    await foreach (var update in chatClient.RunStreamingAsync(messages, thread, ...))
    {
        ...
        var functionCallContents = update.Contents.OfType<FunctionCallContent>().ToList();
        if (functionCallContents.Count > 0)
        {
            foreach (var functionCall in functionCallContents)
            {
                toolCallCount++;   // ← counted via decoded view
            }
        }
        ...
        if (update.RawRepresentation is StreamingChatCompletionUpdate chatCompletionUpdate &&
            chatCompletionUpdate.ToolCallUpdates.Count > 0)
        {
            foreach (var tool in chatCompletionUpdate.ToolCallUpdates)
            {
                if (!string.IsNullOrEmpty(tool.FunctionName))
                {
                    toolCallCount++;   // ← counted AGAIN via raw representation
                }
            }
        }
    }
    ...
    return;
}

toolCallCount is used only in two _logger.LogInformation calls (lines 1242 and 1248). There is no if (toolCallCount > maxCalls) break anywhere in the streaming loop, and the underlying MAF RunStreamingAsync does not bound by tool-call count either.

The only thing keeping a stuck-loop model from running forever is the CancellationTokenSource.CreateLinkedTokenSource(...).CancelAfter(TimeSpan.FromSeconds(_options.TimeoutSeconds)) wired in at the caller (see e.g. AgentExecutor.cs#L103-L104 for the chat path). When TimeoutSeconds fires, the OperationCanceledException falls through, IsTransientException(ex) treats it as transient, and the outer retry loop re-runs the same stuck-loop model up to MaxRetryAttempts times, with exponential backoff in between (lines 1280-1289).

Net: a single wiki-document generation against a misbehaving / adversarial model can burn MaxRetryAttempts * (TimeoutSeconds / per-LLM-call-latency) paid LLM calls + tool dispatches before the operation fails.

There is also a secondary bug: lines 1198-1227 double-count the same tool call (once via update.Contents.OfType<FunctionCallContent>(), again via update.RawRepresentation.ToolCallUpdates). The logged ToolCalls number is therefore ~2× reality, which affects monitoring and any per-document cost-budget alerts a deployment puts on top.

To Reproduce

The PoC mirrors ExecuteAgentWithRetryAsync's loop structure verbatim, with a StuckLoopChatClient standing in for a model that keeps emitting tool calls until the cancellation token fires.

import asyncio

_PAID = {"llm_calls": 0, "tool_dispatches": 0, "tokens": 0}


class FakeUpdate:
    def __init__(self, name):
        self.contents = [{"type": "FunctionCallContent", "name": name}]
        self.raw_tool_calls = [{"function_name": name}]


class CancelledByTimeout(Exception):
    pass


class StuckLoopChatClient:
    async def run_streaming(self, cancellation):
        i = 0
        while True:
            if cancellation.is_set():
                raise CancelledByTimeout("timeout")
            i += 1
            _PAID["llm_calls"] += 1
            _PAID["tool_dispatches"] += 1
            _PAID["tokens"] += 100
            yield FakeUpdate(f"search_call_{i}")
            await asyncio.sleep(0.02)


async def execute_agent_with_retry(max_retries=3, timeout_s=2):
    """Faithful port of WikiGenerator.ExecuteAgentWithRetryAsync:1127-1265."""
    retry, logged = 0, 0
    while retry < max_retries:
        cancellation = asyncio.Event()
        loop = asyncio.get_event_loop()
        handle = loop.call_later(timeout_s, cancellation.set)
        try:
            async for update in StuckLoopChatClient().run_streaming(cancellation):
                # Mirror WikiGenerator.cs:1198-1208 (decoded view count)
                for _ in [c for c in update.contents if c.get("type") == "FunctionCallContent"]:
                    logged += 1
                # Mirror WikiGenerator.cs:1212-1225 (raw representation count, DOUBLE-COUNTS)
                for tool in update.raw_tool_calls:
                    if tool.get("function_name"):
                        logged += 1
                # NOTICE: no `if logged > some_max: break` anywhere
            return logged, "voluntary-stop"
        except CancelledByTimeout:
            retry += 1
            if retry < max_retries:
                await asyncio.sleep(0.1 * (2 ** (retry - 1)))
        finally:
            handle.cancel()
    return logged, f"all-{max_retries}-retries-exhausted"


async def main():
    logged, reason = await execute_agent_with_retry()
    print(f"Exit reason:        {reason}")
    print(f"Paid LLM calls:     {_PAID['llm_calls']}")
    print(f"Tool dispatches:    {_PAID['tool_dispatches']}")
    print(f"Output tokens:      {_PAID['tokens']}")
    print(f"Logged tool-calls:  {logged}  (≈ 2x actual due to double-counting)")


asyncio.run(main())

Save as repro.py, run python repro.py. Observed output:

Exit reason:        all-3-retries-exhausted
Paid LLM calls:     297
Tool dispatches:    297
Output tokens:      29700
Logged tool-calls:  198  (≈ 2x actual due to double-counting)

Expected behavior

1. The streaming loop should bound the number of tool calls per RunStreamingAsync invocation — e.g. if (toolCallCount > _options.MaxToolCallsPerAttempt) break;. Once it breaks, the call should be treated as a non-transient failure (the model isn't converging — retrying with the same prompt is unlikely to help and just multiplies cost).
2. The logged toolCallCount should reflect a single observation per tool call. Use either update.Contents or update.RawRepresentation.ToolCallUpdates, not both. The two are redundant views of the same stream event.

---

Actual behavior

1. The loop runs until the caller-supplied cancellation token fires (i.e. TimeoutSeconds is exhausted), then the outer retry loop classifies that as transient and re-runs the whole generation up to MaxRetryAttempts times.
2. Each tool call in the stream is counted twice — once via update.Contents.OfType<FunctionCallContent>(), once via the raw ToolCallUpdates representation. Production telemetry dashboards that key on this number show 2× actual tool-call rate.