Fuzz the syscall emulation layer; fix 10 guest-reachable bugs

[perbu]

Fuzzes the host-side Linux syscall emulation

fuzz/syscall_fuzz.cpp drives Machine::system_call() directly on a CoW fork: no VM entry, ~8k exec/s under ASan+UBSan. Inputs are record streams — POKE writes fuzzer-chosen bytes into guest memory, CALL sets the six argument registers and dispatches. Both matter, since most handlers read structs out of the guest. Pointer args come from a table of page boundaries, end-of-mapping addresses, unmapped holes and non-canonical values; uniformly random pointers just bounce off copy_from_guest().

MachineException escaping a handler is treated as correct. Sanitizer reports are findings. Other std::exception types are logged once per unique type+message.

~125M executions on the default table, ~47M with TINYKVM_FUZZ_UNSAFE=1, 21k edges. Ten bugs, one commit each, all reachable from an unprivileged guest with legal syscalls. Regression tests in tests/unit/syscalls.cpp, each verified to fail before its fix.

Memory safety

• sendmsg — msg_namelen unchecked into a 128-byte stack sockaddr_storage. Guest-controlled length and bytes; segfaults a plain release build.
• sendmmsg — vlen only checked > 0 before use as a copy length into a 64KB stack array. Same, segfaults release.
• getdents64 — host -1 stored in __u64 sysret() then tested > 0, so it became a ~2^64 copy_to_guest length. Host stack streamed into guest memory, then a wild copy. getdents64 on any regular fd (ENOTDIR) triggers it.
• zero-length I/O — buffers[0] on an empty gather list. write(fd, p, 0), a legal no-op, null-dereferences and kills the VMM. UB in nine other handlers.
• prctl(PR_GET_NAME) — up to 16 bytes read out of the 8-byte literal "tinykvm"; adjacent .rodata to the guest.

Availability

• madvise(MADV_DONTNEED) → memzero() — guest length walked at one page-table lookup per 4K, missing pages skipped, unbounded. ~9M pages/s, so 2^64 pins the host thread for years. Inside the syscall handler, so the execution timeout never fires.

Correctness / UB

• ppoll — tv_sec * 1000 on a raw guest value: signed-overflow UB, then truncation to an arbitrary timeout.
• recvmsg/recvfrom — socklen_t& bound to an arbitrarily-aligned guest pointer.
• timerfd_create/eventfd2/inotify_init1 — failed host fd passed to manage(), which throws on a negative fd, so the if (vfd < 0) below each was dead code and a std::runtime_error escaped. clockid and flags are guest-controlled. epoll_create1 already had the right shape.
• openat — non-zero open_how.mode without O_CREAT; openat2 requires 0, so guests could never open an existing file for writing. Found writing a test, not by the fuzzer.

Questions for @fwsGonzo

Five things I deliberately did not touch, each changes behaviour outside the library's own scope. Details and repros in docs/syscall-fuzzing.md.

1. SIGPIPE kills the VMM. pipe2(fds); close(fds[0]); write(fds[1], buf, 16) — three legal syscalls, process dies. MSG_NOSIGNAL is passed on the sendmsg/sendto paths, so the hazard is known, but plain write/writev/pwritev64 have no equivalent and pipes cannot be protected that way. Options: signal(SIGPIPE, SIG_IGN) in Machine::init() (what most servers want, but imposes a side effect on the host — the src/ CLIs would stop dying on EPIPE into head); send(..., MSG_NOSIGNAL) for sockets (needs FileDescriptors to keep the is_socket flag it currently discards, and still leaves pipes); or block SIGPIPE per write (a syscall pair per I/O). Which?

2. Guest-triggerable std::runtime_error. The contract is that guest misbehaviour arrives as MachineException, but futex() with an unimplemented op (FUTEX_REQUEUE suffices), manage() at max_files, and translate_writable_vfd() on a read-only fd all throw bare std::runtime_error. Retyping to MachineException matches the rest of the layer — is anything relying on the current types?

3. clock_nanosleep. Passes the guest timespec straight through, so a guest can pin a host thread indefinitely; same blind spot as the madvise hang. Unlike that one, bounding it changes guest-visible semantics. Cap it, make it a policy hook, or leave it to embedders? (The fuzzer value-clamps it so the marshalling still gets covered.)

4. Remote-VM madvise. The memzero clamp widens its bound to include a connected remote's range, preserving existing behaviour. But should a guest be able to MADV_DONTNEED a storage VM's pages at all? I'd expect not, and narrowing it would be the isolation-correct change.

5. Callbacks returning without sysret(). poll, ppoll, epoll_wait and accept4 all return early when their callback declines, leaving the return register untouched — at syscall entry that is the syscall number, so a declined accept4 looks like it returned fd 288. Fine if the embedder also pauses the guest, but it is an undocumented part of the callback contract.

Notes

• readv (19) has no handler at all and returns -ENOSYS, while writev is implemented. glibc uses readv in some configurations.
• poll/ppoll still bind writable_memarray<pollfd> to an arbitrarily-aligned guest pointer (same UB as the recvmsg fix). Fixing it means copying in/out or rejecting misaligned arrays with -EFAULT; the latter changes guest-visible semantics.
• Harness only ever uses a plain fork. Remote-connected VMs are untouched — the obvious next extension.
• ARM64 untested; the harness is arch-neutral but has only run on AMD64.

test_reset and test_tegridy fail on master too, unrelated to this branch.

🤖 Generated with Claude Code

[fwsGonzo]

Thanks! Merged manually.