Extend the Message format to support tool calling, extended thinking, and multimodality

[nico-martin]

This issue aims to bring together several related discussions (including #138, #149, and #50) into one cohesive proposal around restructuring the Message type to better reflect the realities of modern on-device LLMs.

Motivation

Even sub-10B models increasingly support multimodal inputs, tool calling, and extended thinking. The current Message shape (a role and a string content) doesn't have room for any of these, which makes it hard to build real agent-style applications on top of the Prompt API. Rather than solving each capability in isolation, it seems worth discussing a more unified message structure that could accommodate all of them cleanly.

Proposed message structure

The key idea is to make content a union of typed content parts (text, image, audio), and to introduce distinct message interfaces per role, since the properties that make sense differ significantly between user, assistant, system, and tool messages:

export type MessageContent = string | MessageContentPart[];

export type MessageContentPart = TextContentPart | ImageContentPart | AudioContentPart;

export interface TextContentPart {
    type: 'text';
    text: string;
}

export interface ImageContentPart {
    type: 'image';
    data: string | Blob | ArrayBuffer | Uint8Array;
    mimeType?: 'image/png' | 'image/jpeg' | 'image/webp' | string;
}

export interface AudioContentPart {
    type: 'audio';
    data: string | Blob | ArrayBuffer | Uint8Array;
    mimeType?: 'audio/wav' | 'audio/mpeg' | 'audio/ogg' | string;
}

export interface SystemMessage {
    role: 'system';
    content: MessageContent;
}

export interface UserMessage {
    role: 'user';
    content: MessageContent;
}

export interface AssistantMessage {
    role: 'assistant';
    content?: MessageContent;
    thinking?: string;
    toolCalls?: Array<{
        id: string;
        type?: 'function';
        function: {
            name: string;
            arguments: Record<string, unknown>;
        };
    }>;
}

export interface ToolMessage {
    role: 'tool';
    toolCallId: string;
    name?: string;
    content: MessageContent;
}

export type Message = SystemMessage | UserMessage | AssistantMessage | ToolMessage;

Note that MessageContent is shared across user/system messages and tool call responses keeping the content shape consistent regardless of role.

Because messages are fully serializable, they can also be stored and reused across sessions via initialPrompts, which would address #50.

Proposed response structure

On the output side, a flat text response also doesn't map well to agentic interactions. A single user prompt can trigger multiple model turns: one or more to emit tool calls, others to synthesize the result. A more flexible ModelResponse could model this as a list of RunResults:

export interface Usage {
    promptTokens: number;
    completionTokens: number;
    totalTokens: number;
}

export interface RunResult {
    thinkingText: string;
    text: string;
    tools: ToolCallResult[];
    usage: Usage;
}

export interface RequestResult {
    done: boolean;
    runs: RunResult[];
    usage: Usage;
}

export type StreamChunk = RequestResult;

A response to "What's the weather in London?" with tool calling and thinking enabled might look like:

{
  done: true,
  runs: [
    {
      thinkingText:
        '1.  **Analyze the user\'s request:** The user is asking ...,
      text: "",
      tools: [
        {
          id: "toolcall_3",
          name: "get_weather",
          args: {
            location: "London",
          },
          output: {
            content: [
              {
                type: "text",
                text: "The weather in London in Sunny, 20 degrees celsius.",
              },
            ],
          },
          durationMs: 0.13499999791383743,
        },
      ],
      usage: {
        promptTokens: 123,
        completionTokens: 156,
        totalTokens: 279,
      },
    },
    {
      thinkingText:
        'The user is asking for the current weather in ...',
      text: "The current weather in London is sunny with a temperature of 20 degrees.",
      tools: [],
      usage: {
        promptTokens: 167,
        completionTokens: 152,
        totalTokens: 319,
      },
    },
  ],
  usage: {
    promptTokens: 290,
    completionTokens: 308,
    totalTokens: 598,
  },
};

Two runs: the first produces the tool call, the second synthesizes the final answer.

Reference implementation

I've been exploring a very similar shape while building an AgentSDK for Transformers.js, which tries to align closely with the Prompt API. A minimal working example with Gemma 4 running in-browser via WebGPU looks like this:

const model = await Model.load(
  { modelId: 'onnx-community/gemma-4-E2B-it-ONNX', device: 'webgpu', dtype: 'q4f16' },
  (info) => (info.status === 'progress_total') && console.log(info),
);

const agent = new Agent({
  model,
  initialPrompts: [{ role: 'system', content: 'You are a helpful research assistant.' }],
});

const result = await agent.run('Who are you?');

This isn't meant as a concrete implementation proposal, more as a working reference point that the two APIs are converging on similar problems, and that aligning on message shape early would benefit both.

---

Looking forward to hearing thoughts, especially on whether the per-role interface split feels right, and whether the multi-run response model is a direction worth pursuing.

[nico-martin]

I wrote my interfaces in TypeScript because thats how they are defined in my Transformers.js implementation. For the specs, here are the WebIDL definitions

Messages

typedef (
  ImageBitmapSource
  or AudioBuffer
  or BufferSource
  or DOMString
) MessageValue;

typedef (
  MessageValue
  or sequence<MessageContentPart>
) MessageContent;

dictionary MessageContentPart {
  required DOMString type;          // "text" | "image" | "audio"
  required MessageValue value;
};

dictionary ToolCallFunction {
  required DOMString name;
  required object arguments;
};

dictionary ToolCall {
  required DOMString id;
  DOMString type;                   // "function"
  required ToolCallFunction function;
};

dictionary SystemMessage {
  required DOMString role;          // "system"
  required MessageContent content;
};

dictionary UserMessage {
  required DOMString role;          // "user"
  required MessageContent content;
};

dictionary AssistantMessage {
  required DOMString role;          // "assistant"
  MessageContent content;
  DOMString thinking;
  sequence<ToolCall> toolCalls;
};

dictionary ToolMessage {
  required DOMString role;          // "tool"
  required DOMString toolCallId;
  DOMString name;
  required MessageContent content;
};

typedef (
  SystemMessage
  or UserMessage
  or AssistantMessage
  or ToolMessage
) Message;

Results

dictionary ToolCallResult {
  // extend as needed
};

dictionary Usage {
  required long long promptTokens;
  required long long completionTokens;
  required long long totalTokens;
};

dictionary RunResult {
  required DOMString thinkingText;
  required DOMString text;
  required sequence<ToolCallResult> tools;
  required Usage usage;
};

dictionary RequestResult {
  required boolean done;
  required sequence<RunResult> runs;
  required Usage usage;
};

typedef RequestResult StreamChunk;

[Mrww305]

[tomayac]

@Mrww305 Please don't post AI-generated infographics. It's totally fine if they help you personally understand the proposal, but please refrain from posting them here.

[Mrww305]

This is a very great idea to further increase synthesis and decrease drift. Especially with long context AND MULTIMODEL

[Mrww305]

This proposal is exactly what the industry needs, and it addresses the fundamental tension in on-device AI right now: web APIs were built for chat, but the world is moving to agents.

Here’s my honest breakdown of why this architecture is smart, where the friction points will be, and what's missing.

The Smart Moves (What Works)

1. The "Multi-Run Response Model" is the killer feature.
   Current APIs (like the Web LLM API) assume a single synchronous request/response. By explicitly returning a list of RunResults, you’re acknowledging that agentic loops are first-class citizens. This allows the browser's task scheduler to pause, run a tool (like navigator.clipboard.read()), and resume the LLM call without losing the context stack.
2. Unified Content Union over "Patches".
   Right now, browsers are bolting on image inputs via experimental flags, and tools via separate JSON schemas. Your architecture correctly recognizes that modality and tool-calling are not separate features—they are the same thing. An image is just a vision tool; a tool call is just a text prompt with structured output. Unifying them under a single content union is the only way to prevent API bloat.
3. Role-Specific Interfaces (Tool/Assistant/User).
   This is crucial for on-device memory management. By separating the schema, the browser engine can optimize memory allocation—e.g., keeping System messages in high-priority cache, while aggressively garbage-collecting Tool outputs.

The Friction Points (The Hard Parts)

1. The "Bloat" Problem.
   If content is a union of text | image | audio, you will need to stream these parts. Are you proposing WebCodecs integration for audio/video chunks, or base64 blobs? If you use base64, the memory footprint of a 10-second audio clip will crash a mobile browser tab. Suggestion: The union should accept Blob or ReadableStream references, not raw data.
2. State Management Hell.
   The RunResults list implies a graph of dependencies (Tool A's output feeds Tool B's input). The current diagram doesn't show a run_id or parent_run_id. Without this, the browser can't prune the execution tree when a user clicks "Stop generating," leading to zombie event listeners.
3. Privacy at the Tool Layer.
   Granting an LLM access to navigator APIs (camera, location, clipboard) via this unified structure requires a new permission model. The current Permissions API isn't granular enough for "Tool A can read contacts, but Tool B cannot." You'll need a Tool Manifest that ties the tool call to a specific permission policy.

What's Missing (The 2026 Reality)

· Token Budget Headers: On-device models (Gemini Nano, Phi) have tiny context windows. This architecture needs a truncation_strategy field in the System message to tell the browser how to compress older RunResults (e.g., "summarize after 5 turns") before the model hits OOM.
· Embedding Support: You mention "multimodal," but you omitted vector embeddings. A true agent needs RAG. The content union should include an embedding part so the browser can do semantic search locally without round-tripping the text back to the model.

Final Verdict

This is a correct and necessary evolution. It moves the browser from being a text renderer to being an orchestrator for local AI.

The only way this fails is if the Chrome/Safari teams implement the Tool role as a simple JSON string instead of a fully typed interface. If they get the types right (strongly typed function signatures with Zod/TypeScript-like validation built into the browser), this will be the foundation for the next generation of web apps.

Score: 9/10 for vision. The missing 1 point is for the lack of a streaming partial-tool-call protocol (since agents often need to show "thinking" in real-time before the final RunResults list is compiled).