vcpu: lazy per-fork kvm_run mapping + kvm_vcpu_id/guest_cpu_index split - #93
Open
perbu wants to merge 3 commits into
Open
vcpu: lazy per-fork kvm_run mapping + kvm_vcpu_id/guest_cpu_index split#93perbu wants to merge 3 commits into
perbu wants to merge 3 commits into
Conversation
Each vCPU's kvm_run mapping is one non-coalescible VMA (a distinct anon_inode per vcpu fd, so vma_merge can never merge them), and every subsequent KVM_CREATE_VM registers an MMU notifier through mm_take_all_locks, which walks and locks every VMA in the address space. With N forks of one master sharing one address space that makes building the pool O(N^2): measured ~0.105 us per resident VMA per KVM_CREATE_VM here, i.e. ~8-10 ms/fork at 100k parked forks. A parked fork does not need the mapping: only two of the pieces of vCPU state a fork copies from its master go through it (GPRs and sregs), and KVM has no ordering requirement between mmap(vcpu_fd) and the vCPU ioctls. With MachineOptions::lazy_vcpu_mmap, a fork's vCPU therefore skips the mmap at construction and keeps GPRs and sregs in a userspace shadow instead. The four register accessors are re-pointed at that shadow, so it is canonical storage -- necessary because the accessors hand out mutable references that callers read-modify-write, and the fork path itself reads freshly staged state back (set_tls_base, and reset_to's enter_usermode reading sregs.cs.dpl). Pure ioctl staging would read zeros back. On the first run the mapping is created, the shadow is blitted into kvm_run->s.regs with both KVM_SYNC_X86_REGS and KVM_SYNC_X86_SREGS marked dirty (without which the guest would enter with CR3=0 and fail entry), and the accessor pointers flip permanently. The flip happens in run_once() only, so no caller can hold a register reference across it. Cold pool builds then keep one VMA per fork that has *ever run* rather than one per fork that exists. The mapping is never released early: munmap fires an mmu-notifier invalidation to every VM registered in the address space, which is the same O(N) fan-out in the other direction. Notes: - Masters always map eagerly. Fork construction reads the master's registers through the mapping precisely because a process that inherited the master over fork() may not ioctl its vCPU fd (-EIO), see the #error at the top of vcpu.cpp. - init() gained an explicit m_initialized flag: the old "if (kvm_run == nullptr)" guard doubled as "not initialized yet" and wraps KVM_SET_CPUID2, which the master reset path relies on being issued once. - SMP vCPUs keep mapping eagerly; they are only ever created to run. - vCPU gained a destructor that releases the shadow: deinit() runs from ~Machine only, which a constructor that throws after vCPU init (a fork running out of working memory in setup_cow_mode) never reaches. - AMD64 only. The arm64 run loop writes kvm_run->immediate_exit, and already has its own ioctl-plus-cache register model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Covers a lazily mapped fork that is constructed and destroyed without
ever running, one that is reset_to() repeatedly while parked (which
enters usermode, reading the staged sregs back), and one that runs after
staging: the shadowed CR3 has to survive the transition, or the guest
fails entry.
The load-bearing check is the VMA accounting: the number of
anon_inode:kvm-vcpu lines in /proc/self/maps must equal the number of
vCPUs that have ever run, not the number that exist. A control case
with the option off asserts the opposite (one mapping per fork at
construction), so the counter cannot pass vacuously.
TINYKVM_LAZY_VCPU_MMAP_DEFAULT flips the MachineOptions default, so the
whole suite can be built and run against the lazy path:
cmake ../unit -DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_FLAGS_DEBUG="-g -DTINYKVM_LAZY_VCPU_MMAP_DEFAULT=true"
Both ways round the suite gives identical results.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The two roles of cpu_id are unrelated: the KVM_CREATE_VCPU argument (unique per struct kvm) and the guest-visible CPU index (PerVCPUTable slot, SMP TSS/IST math). Today they are always equal; under VM pooling (VmGroup) many forks share one struct kvm, so the KVM id becomes group-unique while every fork's main vCPU stays guest CPU 0. No behavior change: all call sites pass (id, id) or (0, 0). The only live guest-visible consumer is vcpu_table_addr(); the two dead-code sites (vcpu_run.cpp integrity check, remote.cpp thread selection) are converted to guest_cpu_index since both express the guest-visible role. cpu_id is removed, not aliased, so stale uses fail to compile. Unit suite: failure set identical to baseline (fork.cpp:411/reset/ tegridy, the known environmental failures on this host). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 2 of the ladder in #91. Applies to master independently of the VmGroup core.
MachineOptions::lazy_vcpu_mmap): each vCPU's kvm_run page is a non-coalescible VMA, and every subsequent KVM_CREATE_VM walks all VMAs under mm_take_all_locks — building a pool of N forks in one address space is O(N²), ~8–10 ms/fork at 100k parked forks. A parked fork doesn't need the mapping (only GPRs/sregs go through it, kept in a userspace shadow that the register accessors re-point at); it is materialized on first run. A fork that never runs costs zero host VMAs.kvm_vcpu_id(KVM_CREATE_VCPU argument, unique per struct kvm) andguest_cpu_index(PerVCPUTable slot, SMP TSS/IST math). Always equal today; under VmGroup the KVM id becomes group-unique while every fork's main vCPU stays guest CPU 0. No behavior change;cpu_idis removed, not aliased, so stale uses fail to compile.Unit suite matches the master baseline on x86-64.
🤖 Generated with Claude Code