Skip to content

fix(tools): surface non-UTF-8 files as ToolError in read/edit#1745

Open
sean-kim05 wants to merge 1 commit into
anthropics:mainfrom
sean-kim05:fix/agent-tools-non-utf8
Open

fix(tools): surface non-UTF-8 files as ToolError in read/edit#1745
sean-kim05 wants to merge 1 commit into
anthropics:mainfrom
sean-kim05:fix/agent-tools-non-utf8

Conversation

@sean-kim05

Copy link
Copy Markdown

Summary

The agent read and edit tools (src/anthropic/lib/tools/agent_toolset.py) decode files with Path.read_text() (UTF-8) inside a try that only catches ToolError and OSError:

text = target.read_text()
except ToolError:
    raise
except OSError as e:
    raise _fs_error("read", file_path, e) from e

UnicodeDecodeError is a subclass of ValueError, not OSError, so it escapes uncaught. The moment the model calls read/edit on a file that isn't valid UTF-8 — a binary artifact, or an ordinary latin-1 / UTF-16 text file — the exception bypasses the tools' error-normalization: instead of the intended clean "read: <path>: ..." ToolError, the messages runner logs a full traceback (log.exception) and hands the model a raw UnicodeDecodeError repr.

The sibling grep and bash tools in the same module already decode with errors="replace", so non-UTF-8 files are a known, expected input here — read/edit just don't handle them.

Reproduction

env = AgentToolContext(workdir=d)
(Path(d) / "notes.txt").write_bytes(b"caf\xe9 menu\n")   # latin-1 text
await beta_read_tool(env).call({"file_path": "notes.txt"})
# -> UnicodeDecodeError leaks out instead of a ToolError

Fix

Catch UnicodeDecodeError in both tools and raise a clean ToolError. The edit path deliberately does not fall back to a lossy errors="replace" decode, since the replacement round-trip would corrupt the file on write-back.

Testing

Added test_read_rejects_non_utf8_file and test_edit_rejects_non_utf8_file in tests/lib/tools/test_agent_toolset.py (a latin-1 file must raise ToolError matching "not valid UTF-8"). Verified RED (raw UnicodeDecodeError) → GREEN. Full test_agent_toolset.py passes (42 tests); ./scripts/lint clean (ruff + pyright + mypy).

The agent `read` and `edit` tools decode files with `Path.read_text()`
(UTF-8) inside a `try` that only catches `ToolError` and `OSError`.
`UnicodeDecodeError` is a subclass of `ValueError`, not `OSError`, so a
non-UTF-8 file (a binary artifact, or an ordinary latin-1 / UTF-16 text file)
escapes the tools' error contract: instead of the intended clean
`"<op>: <path>: ..."` `ToolError`, the runner logs a full traceback and hands
the model a raw `UnicodeDecodeError` repr. The sibling `grep`/`bash` tools in
the same module already decode with `errors="replace"`, so non-UTF-8 files are
a known, expected input here.

Catch `UnicodeDecodeError` in both tools and raise a clean `ToolError`. Editing
deliberately does not fall back to a lossy `errors="replace"` decode, since the
replacement round-trip would corrupt the file on write-back.
@sean-kim05 sean-kim05 requested a review from a team as a code owner July 7, 2026 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant