| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
KVM fixes for 7.3-rcN
- Fix a brown paper bag bug where KVM would incorrectly treat Intel PMU MSRs
as valid on AMD.
- Fix a regression in the hardware disable selftest where it checked the wrong
macro when detecting glibc support (breaks at least musl).
- Never clear KVM_REQ_VM_DEAD so that dead VMs stay dead, which is especially
important for KVM_BUG_ON() flows, which often guard more dangerous bugs.
- Re-pend GET_NESTED_STATE_PAGES if getting the pages fails, to fix a bug
where KVM would let userspace run a broken setup with stale vmcs12 pages.
- Fix a class of bugs where KVM would fail to fill kvm_run exit fields if
getting nested pages failed.
- Treat reserved entries in the memory attributes xarray as "no attributes",
to fix false positives when checking for mixed attributes.
- Fix memcg accounting for the memory attributes xarray (the xarray library
subtly requires the xarray to be configured for accounting upfront; the gfp
flags taken at runtime are used only rarely).
- Don't pre-reserve xarray entries when storing empty attributes, as storing
NULL must not require memory allocation (KVM and other subsystems heavily
rely on this behavior).
|
|
Manually perform cache maintenance on the source VM during intra-host
migration to ensure no stale data is left in CPU caches after the VM is
destroyed. Because the source VM is "converted" to a non-SEV VM, KVM's
memory reclaim flows won't trigger cache maintenance, e.g. when all guest
memory is reclaimed in response to detaching from the mmu_notifier.
Note, relying on the destination VM to do cache maintenance isn't an option
as KVM doesn't require identical guest memory configurations, i.e. the
source VM may have access to memory that the destination VM does not.
Enforcing equivalent memory configurations is infeasible, as it would
require a *deep* comparison of memslots, e.g. to verify that not only are
the memslot identical, but what the memslots point at is also identical.
Fixes: b56639318bb2 ("KVM: SEV: Add support for SEV intra host migration")
Cc: stable@vger.kernel.org
Reported-by: Stefan Teodorescu <fane@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260923163721.1584779-3-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Unconditionally free SEV's "have run CPUs" cpumask in the VM destroy path,
i.e. even for what appear to be non-SEV VMs, as an SEV VM becomes a non-SEV
VM if its state is intra-host migrated. Alternatively, the mask could be
freed in sev_migrate_from() when "converting" the source VM, but that gets
annoying because ideally KVM would nullify the mask to guard against UAF,
and nullifying the mask would need be conditioned on CPUMASK_OFFSTACK=y.
Freeing the mask during sev_migrate_from() is also not robust against other
KVM bugs, though that's kind of a moot point since any such bugs would show
up even if sev->active is never set. I.e. KVM must get that side of things
correct. But, that's not a great reason to add more code just to make
things marginally less robust.
Fixes: 6f38f8c57464 ("KVM: SVM: Flush cache only on CPUs running SEV guest")
Cc: stable@vger.kernel.org
Reported-by: Stefan Teodorescu <fane@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260923163721.1584779-2-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Fill kvm_run with "internal error, emulation" in the common error handling
paths for getting nested state pages, as requiring each check to manually
fill kvm_run is error prone and requires a non-trivial amount of copy+paste.
Specifically, both SVM and VMX fail to fill kvm_run if load_pdptrs() fails,
and SVM fails to fill kvm_run if kvm_hv_verify_vp_assist() fails. If those
flows fail, the *best* case scenario is that KVM will exit to userspace
with KVM_EXIT_UNKNOWN. The worst case scenario is that KVM exits with a
stale exit_reason and confuses userspace.
Note, SVM never exits to userspace if something goes sideways when dealing
with vmcb12 assets while emulating VMRUN, i.e. lack of SVM-specific code is
not a bug.
Fixes: 0f85722341b0 ("KVM: nVMX: delay loading of PDPTRs to KVM_REQ_GET_NESTED_STATE_PAGES")
Fixes: 232f75d3b4b5 ("KVM: nSVM: call nested_svm_load_cr3 on nested state load")
Fixes: 3f4a812edf5c ("KVM: nSVM: hyper-v: Enable L2 TLB flush")
Cc: stable@vger.kernel.org
Reported-by: Jinwoo Lee <rkskek9254@gmail.com>
Closes: https://lore.kernel.org/all/20260813043932.3214460-1-rkskek9254@gmail.com
Reported-by: Stefan Teodorescu <fane@google.com>
Link: https://patch.msgid.link/20260921211608.1030158-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Re-pend GET_NESTED_STATE_PAGES before exiting to userspace if getting the
nested pages fails in the KVM_RUN path. If userspace re-runs the vCPU, and
vmcs02 holds valid PFNs from the *previous* run of L2, then KVM could
re-enter L2 with stale, unpinned PFNs mapped into e.g. the vAPIC page.
Note, both SVM and VMX (as of commit 11722439fb20 ("KVM: nVMX: Ensure
KVM_REQ_GET_NESTED_STATE_PAGES is cleared on VM-Exit") ensure the request
is cleared on VM-Exit (including the "forced" case), i.e. there is no risk
of double-mapping due to emulated VMLAUNCH/VMRESUME/VMRUN *and* the request
trying to map the nested pages.
Fixes: 671ddc700fd0 ("KVM: nVMX: Don't leak L1 MMIO regions to L2")
Cc: stable@vger.kernel.org
Reported-by: Jinwoo Lee <rkskek9254@gmail.com>
Closes: https://lore.kernel.org/all/20260813043932.3214460-1-rkskek9254@gmail.com
Reported-by: Stefan Teodorescu <fane@google.com>
Link: https://patch.msgid.link/20260921211608.1030158-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Use kvm_test_request() instead of kvm_check_request() when querying
KVM_REQ_VM_DEAD, i.e. don't clear KVM_REQ_VM_DEAD, as the entire purpose
of KVM_REQ_VM_DEAD is to prevent the vCPU from enterring the guest ever
again, even if userspace insists on redoing KVM_RUN.
Ensuring KVM_REQ_VM_DEAD is never cleared will allow relaxing KVM's rule
that ioctls can't be invoked on dead VMs, to only disallow ioctls if the
VM is bugged, i.e. if KVM hit a KVM_BUG_ON().
Opportunistically add compile-time assertions to guard against clearing
KVM_REQ_VM_DEAD through the standard APIs.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Acked-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260806214618.82180-1-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Commit c85cdc1cc1ea ("KVM: x86/pmu: Move handling PERF_GLOBAL_CTRL and
friends to common x86") moved the existence check for the following Intel
PMU MSRs to kvm_pmu_is_valid_msr():
- MSR_CORE_PERF_GLOBAL_STATUS
- MSR_CORE_PERF_GLOBAL_CTRL
- MSR_CORE_PERF_GLOBAL_OVF_CTRL
That commit deemed these MSRs valid whenever pmu->version > 1. It intended
to share the check with AMD PerfMonV2 because both vendor implementations
require version 2 or greater for global PMU controls. However, as noted in
the commit message, AMD uses different MSR indices for its global PMU
registers.
Commit 4a2771895ca6 ("KVM: x86/svm/pmu: Add AMD PerfMonV2 support")
subsequently added AMD PerfMonV2 support and set pmu->version = 2. Because
kvm_pmu_is_valid_msr() validated the Intel MSRs whenever pmu->version > 1,
KVM incorrectly permitted AMD guests with PerfMonV2 to access these Intel
MSRs without a #GP.
Move the validation of these Intel MSRs to intel_is_valid_msr() and remove
the common switch statement from kvm_pmu_is_valid_msr(). AMD already
validates its own global PMU MSRs in amd_is_valid_msr().
Fixes: 4a2771895ca6 ("KVM: x86/svm/pmu: Add AMD PerfMonV2 support")
Signed-off-by: Jim Mattson <jmattson@google.com>
Reviewed-by: Like Xu <likexu@tencent.com>
Reviewed-by: Sandipan Das <sandipan.das@amd.com>
Link: https://patch.msgid.link/20260902184711.138538-1-jmattson@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Pull kvm updates from Paolo Bonzini:
"ARM64:
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when
FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits
are stateful in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the
scope of those instructions remains 'in host' (i.e. L1
kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly
in the VGIC, our in-kernel generator of bugs (and sometimes
interrupts)
LoongArch:
- Advertise already-supported capabilities
- Some bug fixes about timer and MMIO
- Some hardening about interrupt injection
- Replace kvm_err() with kvm_pr_unimpl()
- Add FPU/LSX/LASX test cases for selftests
RISC-V:
- Svadu/Zicfiss/Zicfilp FWFT support for Guest
- Use try_cmpxchg for IMSIC MRIF RMW
- More arch-specific tracepoints in KVM RISC-V
- Eager page splitting when enabling dirty logging
- Optimize hfence request handling for SMP Guests
- Improve dirty log clearing by skipping zero bits in mask
- Guard HFENCE range loops against overflow
- CPU PM notifiers in KVM RISC-V for non-retentive idle states
- Fix kernel-mode vector context save/restore for Guest
s390:
- Fixes for vfio-ap
- Fixes for the gmap rework
- Fixes for vsie
- AI triggered fixes all over
- diag9c tracing
- code move preparation for the additional arm64 support
- enable CONTEXT_ANALYSIS
x86:
- Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding
regs.c (the kvm_cache_regs.h => regs.h is already applied) and
msrs.{c,h}, and moving relevant code out of x86.c
- Split kvm_mmu in three parts, respectively to describe the format
of page tables, walking the guest page tables and building the page
tables. Always use the same page table walker kvm->arch.gva_walk as
the entry point to convert a guest's virtual address, where the
previous code used two different kvm_mmu structs depending on
whether the walk included nested EPT/NPT or not. Make page fault
vmexits reuse the permission checking machinery that is used for
guest page faults. This is both a cleanup and a baby step towards
supporting XS/XU memory permissions
- Document some of the "fun" gotchas with the APIC base when creating
IRQCHIPs on x86
- Remove a defunct masterclock update from kvm_xen_shared_info_init().
It could result in incorrect kvmclock due to triggering an
unnecessary switch to/from masterclock mode
- Skip Xen runstate time updates if time has effectively gone
backwards, so that the guest doesn't report 100% steal time for
a very, very long time
- Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM
was updating the wrong sub-leaf, and upstream KVM will soon provide
all the information needed by userspace to populate the CPUID field
itself
- Fix a bug where KVM would walk a newly created rmap without holding
the rmap lock (or mmu_lock) during aging
- Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs
- Fix a variety of #DB priority bugs
- Fix a class of races related to enabling Hyper-V emulation on a
vCPU after the vCPU is visible to the rest of KVM
- Use static calls for nested virtualization ops
- Move more KVM-internal code out of x86's kvm_host.h
- Enumerate support for a variety of Zhaoxin instructions that don't
require explicit virtualization
- Fix missing EFER validation bugs, including in the KVM_SET_SREGS*
path
- Harden kvm_vcpu_map() against double-mapping and thus leaking
references
- Misc fixes and cleanups, e.g. for largely benign syzkaller splats
x86 (Intel):
- Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used
for IPI virtualization when the vCPU is freed, to fix a
use-after-free where hardware will write to a freed vCPU's PID
- Service local TLB flushes on a failed nested VM-Enter to fix a bug
where KVM could miss a TLB on a future, successful VM-Enter with
the same L2 VPID
- Cap the maximum value shoved into the VMX Preemption Timer to
workaround an erratum that affects all existing Intel CPUs that
support CPUID 0x15
- Fix VPID virtualization bugs where KVM would fail to flush hardware
TLBs
- Harden the TDX "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
x86 (AMD):
- Forcefully invalidate SNP VMSA pages if their backing guest_memfd
page is zapped/invalidated, e.g. due to a PUNCH_HOLE in response to
a Page-State Change request
- Remove a dying VM from the GA Log notifier list before the VM is
actually destroyed, to fix a potential use-after-free
- While FOLL_WRITE was needed in the past to trigger CoW unsharing,
nowadays FOLL_LONGTERM does that already even without FOLL_WRITE,
and in fact, get_user_pages() actually disallows FOLL_WRITE
together with FOLL_LONGTERM. So don't pass FOLL_WRITE when
registering encrypted memory regions, i.e. when pinning SEV/SEV-ES
guest memory, to fix a regression with file-backed memory
introduced by KVM's (correct) usage of long-term pins
(This was reviewed by mm maintainers; for more information, see
commit ee1a586dd1fa "KVM: SEV: Drop FOLL_WRITE for encrypted region
registration")
- Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled
hosts to fix a data corruption issue due to the PSP driver
assigning to-be-written pages to firmware (as required by the SNP
specs)
- Unconditionally intercept ICBEP so that KVM generates the correct
guest RIP when handling an ICEBP-induced TASK_SWITCH #VMEXIT
- Harden the SNP "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
Generic:
- Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's
debugfs
- Add a per-VM bitmap to track which vCPU IDs have been "claimed" but
for which the vCPU isn't yet online, and use the bitmap to reject
duplicate IDs before calling into arch code. This allows arch code
to consume vcpu_id without having to worry about cross-vCPU
clobbering (at least s390 and x86 have had related bugs)
- Rework the so called "prepare" and "invalidate" guest_memfd hooks
to prepare for in-place private<=>shared conversion, and clean up a
few warts along the way
Selftests:
- Automatically allocate a full page for L2 guest stacks on x86
instead of requiring test-specific L1 guest code to carve out a
portion of the L1 stack for L2 usage, and to ensure the L2 stack
also adheres to the x86-64 calling convention ABI
- Add a selftest to verify {Guest,Host}-Only behavior in x86's
mediated PMU
- Clean up nested SVM's handling of GPRs on L2<=>L1 transitions,
reuse the functionality for nested VMX, and drop the ucall hack
that was fudging around the lack of GPR switching on nVMX
- Add a stress test to verify KVM doesn't clobber/drop #PF state,
e.g. CR2, across save/restore, including when L2 is active
- Add a test to verify KVM_CREATE_VM accepts exactly what is reported
by KVM_CAP_VM_TYPES
- Misc selftests fixes and cleanups
- Fix several issues with seeding the pRNG, and rework the pRNG APIs
to that the pRNG can be sanely used in host code, not just guest
code
- Add an IRQ test to validate virtual IRQ deliverty for IRQs wired up
via KVM_IRQFD + KVM_SET_GSI_ROUTING, with optional support for
triggering IRQs via writes to an assigned VFIO device
- Add syscall wrappers to assert success on a variety of pthreads and
CPU affinity APIs
- Set vCPU pthread affinity as early as possible to reduce contention
issues that were surfaced by PREEMPT_LAZY, which result in runtimes
of over a minute on large hosts, versus the expected ~5 seconds
- Rework the PMU counters test to run each testcase using a single VM
with many vCPUs for each sub-testcase, instead of using a unique VM
for each sub-testcase. This cuts the runtime by ~20x
Miscellaneous:
- MAINTAINERS updates for vfio-ap, guest_memfd, kvm-x86. Mostly
representing the status quo more accurately, but also... welcome
David Hildenbrand as guest_memfd reviewer!"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (413 commits)
KVM: arm64: Validate GICv5 timer PPIs before claiming ownership
KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs
KVM: arm64: vgic: Prevent speculative SPI array underflow
KVM: arm64: vgic: Free gic_kvm_info on initialization failure
KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'
s390/vfio-ap: Fix NULL deref in status_show() during queue probe
s390/vfio-ap: Fix hot-unplug skipped when last AP adapter or domain removed
s390/vfio-ap: fix potential use of uninitialized apm_filtered bitmap
s390/vfio-ap: Fix control domain removal in vfio_ap_mdev_cfg_remove
s390/vfio-ap: Fix required lock not held during update of ap_matrix_mdev object
s390/vfio-ap: Fix missing lock required to access list of ap_matrix_mdev objects
s390/vfio-ap: Fix dereference matrix_mdev->kvm without checking for NULL
s390/vfio-ap: Fix stale do_remove flag across iterations in vfio_ap_mdev_cfg_remove
RISC-V: KVM: fix vcpu vector context handling for kernel-mode vector
riscv: vector: allow non-preemptible kernel-mode vector with IRQs off
riscv: vector: refactor riscv_v_start_kernel_context
KVM: s390: gmap: Make prefix handling optional
KVM: s390: gmap: Make CMMA optional
KVM: s390: gmap: Make storage keys optional
KVM: s390: Prepare gmap for a second KVM implementation
...
|
|
Pull MM updates from Andrew Morton:
- "mm: drop "sub" prefix from various places" (Dev Jain)
page->folio conversion and a naming cleanup
- "mm/kasan: remove redundant initialization for kasan_flag_write_only"
(Igor Putko)
KASAN cleanup work
- "mm/filemap: reduce unnecessary xarray lookups" (Chi Zhiling)
Small speedup in the pagecaache read code
- "mm/percpu: Fix possible NOFS/NOIO reclaim recursion" (Kaitao Cheng)
Improve the vmalloc code - mainly the avoidance of GFP_KERNEL
allocations when the caller asked for GFP_NOFS or GFP_NOIO
- "mm/kmemleak: avoid soft lockup when scanning task stacks" (Breno
Leitao)
Avoid a soft lockup watchdog trigger from the kmemleak scanning code
in extreme situations
- "mm/page_owner: misc cleanups" (Ye Liu)
Cleanups to the page_owner code. For some reason lots of people have
been working on the page_owner code this cycle.
- "mm: convert to walk_page_range_vma() to eliminate find_vma()"
(Kefeng Wang)
Simplify and accelerate the page walking library function
- "mm/migrate: preparatory cleanups for batch copy and offload"
(Shivank Garg)
Cleanups in the migration code
- "mm/page_owner: add per-fd filter infrastructure for print_mode and
NUMA filtering" (Zhen Ni)
Per-fd filtering to page_owner in order to reduce the sometimes vast
amount of output it can produce
- "mm: Refactor bootmem gigantic hugepage allocation" (Muchun Song)
Fixes and preparatory cleanups around bootmem HugeTLB handling,
sparse initialization ordering, and related vmemmap setup
- "mm/zsmalloc: reduce lock contention in zs_free()" (Wenchao Hao)
Reduce lock contention in zs_free(), which dominates the unmap path
under memory pressure on Android (LMK kills) and on x86 servers
running zswap-heavy workloads.
Up to 1.83x improvement in microbenchmarking.
- "move alloc_tag.c file under mm/" (Suren Baghdasaryan)
- "samples/damon: handle damon_{start,stop}() failures" (SJ Park)
Fix improper handling of damon_start(), damon_stop(), and
damon_call() failures across DAMON sample modules to prevent
potential memory leaks, operation disruptions and use-after-free
bugs
- "mm/damon/sysfs: kobject_del() directories that users can
create/remove" (SJ Park)
Fix delayed sysfs directory removal under DEBUG_KOBJECT_RELEASE
causeing creation failures due to duplicate directory names by adding
missing kobject_del() calls before creating new directories
- "mm: cleanup clear_not_present_full_ptes()" (David Hildenbrand)
Clean up the core pte handling code
- "selftests/damon: misc fixes for test bugs" (Kunwu Chan)
Fix several bugs in the DAMON selftests
- "selftests/damon: fix memcg_path staging handling" (Cheng Nie)
Fix a bug in _damon_sysfs.py for damos_filter memcg_path setup, and
add a test case for it in sysfs.py.
- "selftests/damon: test kdamond refresh_ms" (Ruslan Valiyev)
Selftest coverage for DAMON's refresh_ms sysfs feature by updating
the test control module and verifying that scheme stats update
automatically without manual intervention
- "mm/damon: five misc fixups" (Akinobu Mita)
Miscellaneous DAMON fixups.
- "mm/damon/core: detect internal variation above max_nr_regions/2"
(Jiayuan Chen)
Fix DAMON's region splitting behavior when region counts exceed half
the maximum budget by dynamically scaling down the split fraction as
the limit approaches, preventing large regions from staying un-split,
and add corresponding KUnit test coverage
- "mm: preparatory patches for PMD level swap entries" (Usama Arif)
Refactor and clean up PMD softleaf helpers, call sites, and
architecture flags to lay the groundwork for a follow-up series that
introduces PMD page table swap entries
- "mm/damon: update, optimize, and clean up doc, tests, and code" (SJ
Park)
Update DAMON design and ABI documentation, expands unit and selftest
coverage, optimize damon_commit_target_regions(), and clean up
recently added sysfs interface code for better readability
- "mm/vmpressure: reduce CPU, memory and code overhead on cgroup v2"
(Usama Arif)
Optimize vmpressure() by skipping unnecessary work on cgroup v2 for
userspace event notifications and refactor v1-only eventfd handling
into mm/memcontrol-v1.c to reduce memory overhead and code complexity
- "selftests/mm: refactor pkey helpers and fix mmap error handling"
(Hongfu Li)
Refactor pkeys shared tracing and assertion helpers into a common
file, unify protection key selftests to use consistent diagnostic
logging and assertions, and enforce standardized MAP_FAILED return
checks for mmap() calls across the tests
- "mm/damon: optimize out nr_accesses_bp" (SJ Park)
Replace the error-prone, continuously updated nr_accesses_bp field in
damon_region with an on-demand moving sum function, reducing
structure memory overhead and avoiding state corruption bugs
- "Open HugeTLB allocation routine for more generic use" (Ackerley Tng)
Decouple HugeTLB folio allocation from VMA dependencies by
introducing hugetlb_alloc_folio(), enabling subsystems like
guest_memfd to allocate HugeTLB folios without standard VMA
reservations or pseudo-VMAs
- "mm/damon: provide pseudo moving sum probe_hits" (SJ Park)
Integrate DAMON's probe_hits attribute counter into the pseudo moving
sum infrastructure, enabling real-time, online monitoring without
waiting for full aggregation intervals
- "mm: Some cleanups for page allocator APIs" (Brendan Jackman)
Simplify and refactor the page allocator entry points and flags by
unifying allocation paths, adding internal alloc_flags arguments, and
eliminating redundant __ prefixed alloc_pages variants.
- "Fix incorrect access of hugetlb pte entries" (Dev Jain)
Enforce the consistent use of huge_ptep_get() instead of ptep_get()
for HugeTLB entries and fixes an unaligned address issue in arm64's
huge_ptep_get() implementation
- "mm/damon: validate all parameters in the core" (SJ Park)
Consolidate parameter validation into the DAMON core specifically
within damon_start() and damon_commit_ctx() to centralize error
checking, eliminate caller-side redundant checks and to improve
maintenance efficiency
- "tools/mm/page_owner_sort: fix filtering and cleanup issues" (Yichong
Chen)
Rename is_need() to filter_record() for clearer return semantics, fix
per-record allocation memory leaks and bound output copies in
search_pattern() to address an existing buffer issue
- "memcg: bail out reclaim when memcg is dying" (Jiayuan Chen)
Mitigate a system-wide stall which occurs when a cgroup is removed
while one of its memory control files is doing synchronous reclaim
- "mm/memory-failure: add panic option for unrecoverable pages" (Breno
Leitao)
Introduce an opt-in vm.panic_on_unrecoverable_memory_failure sysctl
that immediately panics the kernel on unrecoverable memory errors in
kernel-owned pages to preserve error context and prevent delayed,
silent data corruption
- "mm/damon: refactor damon_{start,stop,commit}() for simple error
handling" (SJ Park)
Refactor the DAMON core API functions to guarantee that all contexts
are fully stopped when damon_start(), damon_stop(), or damon_commit()
fail, eliminating the need for complex and error-prone caller-side
cleanup code
- "Keep tail page private zero at free and folio split" (Zi Yan)
Add checks to ensure tail_page->private is zero when freeing compound
or high-order pages and when promoting tail pages during large folio
splits. By validating these fields at free and split time, it allows
the removal of redundant private field clearing inside
prep_compound_tail()
- "mm: drop redundant lru_add_drain in anon folio reuse paths" (Barry
Song)
Eliminate redundant lru_add_drain() calls in
wp_can_reuse_anon_folio() and do_swap_page() to reduce LRU lock
contention and system overhead
By validating folio refcounts against the LRU cache before draining
and removing unnecessary drains in the swap path, it achieves up to a
30.5% reduction in drain calls during heavy swap workloads
- "mm: clean up folio LRU and swap declarations" (Jianyue Wu)
Reorganize folio LRU and swap code by relocating page-cluster state
to mm/swap_state.c, renaming mm/swap.c to mm/folio.c, and moving
MM-internal reclaim declarations into mm/internal.h.
- "userfaultfd: working set tracking for VM guest memory" (Kiryl
Shutsemau)
Add userfaultfd support for tracking the working set of VM guest
memory, so a VMM can identify hot pages and reclaim cold ones to
tiered or remote storage
- "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 2)" (David
Hildenbrand)
Remove the remaining pieces of CONFIG_HAVE_BOOTMEM_INFO_NODE,
performing some smaller cleanups around freeing of reserved vmemmap
pages on the way.
- "mm/damon: update probe hits for runtime parameter commits" (SJ Park)
Ensure that DAMON's probe_hits attribute counter is properly updated
when monitoring intervals are changed at runtime, matching the
behavior of nr_accesses. To achieve this, it refactors and renames
existing helper functions for shared use, applies the updates to
probe_hits, and handles edge cases in damon_probe_hits_mvsum() to
maintain measurement accuracy.
- "KSM: performance optimizations for rmap_walk_ksm" (xu xin)
Resolve a severe KSM reverse-mapping performance bottleneck where
thousands of split VMAs sharing a single anon_vma cause extended lock
contention.
By adding an interval-filtering check during the rmap walk, it
reduces worst-case anon_vma lock hold times from over 500ms down to
under 2ms, preventing application freezes and latency spikes under
memory pressure.
- "mm: split a couple of headers from internal.h" (Mike Rapoport)
Split declarations related to mm_init, memblock, vmalloc and sparse
into new headers
- "KSM: use linear_page_index in collect_procs_ksm()" (xu xin)
Apply the interval tree optimization from rmap_walk_ksm() to
collect_procs_ksm() to avoid iterating over non-matching VMAs during
KSM memory error handling.
It hoists loop-invariant address initialization and restricts the
anon_vma_interval_tree_foreach walk to a targeted page offset range,
reducing redundant checks and improving lookup efficiency.
- "selftests/mm: avoid false failures in hugetlb and KSM tests" (Sayali
Patil)
Fix issues in the hugetlb and KSM MM selftest categories that can
report failures when the prerequisites for the tests are not
satisfied
- "mm/damon: introduce data attributes only monitoring" (SJ Park)
Introduce attribute-weighted region management in DAMON, allowing
users to prioritize specific data attributes (such as page sizes or
cgroups) over or instead of access monitoring.
By assigning weights to attribute probes, DAMON can completely
disable access tracking and adjust monitoring regions based on
weighted probe-hit counters to optimize monitoring quality for
attribute-focused workloads.
- "mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings"
(Stanislav Kinsburskii)
Extend hmm_range_fault() to support userfaultfd-backed regions by
allowing the mmap lock to be dropped during fault handling via a new
hmm_range_fault_locked() helper.
By accepting a locked pointer and signaling retry status when lock
release occurs, it enables page fault resolution in userfaultfd
regions while preserving backward compatibility for existing callers.
- "mm: make VMA page offset handling more consistent" (Lorenzo Stoakes)
Clean up and standardize how vma->vm_pgoff is accessed and
manipulated across file-backed and anonymous mappings in the kernel
It introduces dedicated helper functions such as vma_start_pgoff(),
vma_end_pgoff(), vma_set_pgoff() and linear_page_delta() while
renaming rmap interval tree helpers to better reflect their
functionality.
These changes establish a cleaner foundation for future work that
will unify virtual page offset indexing for all anonymous and CoW'd
folios.
- "mm: handle device-private PMDs in walk callbacks" (Usama Arif)
Address kernel panics and state corruption caused by MM walk
callbacks reaching non-present device-private PMD swap entries
created during HMM migrations
It ensures that functions which acquire pmd_trans_huge_lock()
properly recognize device-private PMDs instead of assuming a present
THP or a standard migration entry.
- "mm/rmap: Refactor try_to_unmap_one" (Dev Jain)
Refactor try_to_unmap_one by modularizing Hugetlb,
anonymous-lazyfree, and anonymous-swapbacked logic into dedicated
functions, laying the structural groundwork for batched anonymous
large folio unmapping.
- "Docs/ABI/damon: sysfs ABI document fixes and additions" (Song Hu)
Fix typos and fills in missing entries in the DAMON sysfs ABI
document
- "dax/kmem: atomic whole-device hotplug via sysfs" (Gregory Price)
Introduce an atomic sysfs state attribute and supporting DAX/MM
infrastructure to prevent userland races when offlining and removing
entire memory regions
By adding an unplugged state alongside standard online modes, it
enables whole-device atomic hotplug control while preserving backward
compatibility.
- "mm: convert more vm_flags_t users to vma_flags_t" (Lorenzo Stoakes)
Continue transitioning the kernel from the deprecated vm_flags_t type
to vma_flags_t across core memory management infrastructure.
It replaces legacy type usage in core functions such as do_mmap(),
unmapped area allocation, mm->def_vma_flags, and VMA operations like
mlock, mprotect, and mremap.
- "Two small patches to clean up mm/mm_slot.h" (xu xin)
Refactor mm_slot.h by introducing mm_slot_remove() to unify duplicate
slot deletion sequences in khugepaged and KSM. It also adds code
documentation explaining why mm_slot_lookup and mm_slot_insert must
remain as preprocessor macros rather than static inline functions.
- "mm/damon/core: hide core-private struct fields" (SJ Park)
Clean up DAMON core structures by consistently marking internal-only
fields with private: comment tags to prevent improper direct access
from outer layers.
It enforces encapsulation across core structures including
damon_region, damon_target, and damon_ctx and updates DAMON_SYSFS to
interact through approved access APIs instead of exposing raw struct
members.
- "mm/damon: unurgent fixes for infinite loop, NULL de-ref and races"
(SJ Park)
Address potential infinite loops, NULL dereferences, and race
conditions identified in DAMON
It fixes an infinite loop triggered by extreme user configurations, a
NULL pointer dereference within unit tests and minor monitoring
accuracy degradation caused by subtle runtime races.
- "mm/page_alloc: fixes for free_pages_nolock() on RT/UP" (Brendan
Jackman)
Fix an NMI safety flaw in __free_frozen_pages() where freeing pages
on non-SMP or PREEMPT_RT kernels can bypass can_spin_trylock() checks
via non-PCP or isolated migration paths.
It also resolves potential kernel crashes and privilege escalation
risks triggered when BPF tracing runs in NMI context alongside memory
hotplug or large allocation frees.
- "mm/page_alloc: couple of followups for recent cleanups" (Brendan
Jackman)
Clean up and update page allocator nomenclature, documentation, and
debug assertions.
It aligns internal FPI_ flags with the public "nolock" naming
convention, removes outdated internal implementation details from
high-level page allocator comments, and eliminates obsolete
VM_BUG_ON() assertions in allocation paths.
- "mm/mseal: further cleanups" (Lorenzo Stoakes)
Refactor and simplify the mseal implementation by clarifying API
boundaries and removing unnecessary code complexity.
It replaces generic do_mseal() usage outside the syscall with a
dedicated mseal_mmap_page_zero() helper for MMAP_PAGE_ZERO,
eliminates mm_struct parameters to enforce that sealing applies only
to current->mm, and streamlines overall logic and comments with no
functional changes intended.
- "mm/vmscan: fix swappiness=max and clean up per-node proactive
reclaim" (Ridong Chen)
Resolve reclaim behavior bugs and clean up function parameters across
memory reclaim paths
It fixes swappiness=max in both standard reclaim and MGLRU so
unswappable anonymous memory no longer falls back to evicting page
cache, ensures reclaim_store() returns accurate error codes instead
of collapsing all failures into -EAGAIN, and removes the obsolete
gfp_mask parameter from __node_reclaim().
- "mm: mincore: misc cleanups" (Kefeng Wang)
Clean up and simplifies the mincore code. Most importantly, it
removes the historical special behavior that always reports VM_PFNMAP
pages as non-resident.
- "mm/huge_memory: drop dead split helper variants" (Kiryl Shutsemau)
Two trivial cleanups in the folio split API
- "mm/damon: fix uninitialized DAMOS field and kunit exec expectation
bugs" (SJ Park)
Resolve minor operational and testing bugs in DAMON identified by
Sashiko. It initializes the damos->last_applied field to prevent
occasional efficiency degradation and fixes invalid memory accesses
in DAMON KUnit tests during test failure handling.
- "cleanup for stable_page_flags()" (Jinjiang Tu)
Clean up and refactor stable_page_flags() used by /proc/kpageflags
without altering functionality.
It uses BIT_ULL() to prevent shift-overflow warnings on 64-bit flag
bits, converts folio-specific flag checks to standard folio_test_*()
helpers, and removes redundant CONFIG_PAGE_IDLE_FLAG handling.
- "Batch unmap of uffd-wp file folios" (Dev Jain)
Extend batched folio unmapping support to file folios within
userfaultfd write-protect (uffd-wp) VMAs by adding batching
capabilities to pte_install_uffd_wp_if_needed().
This removes special-case restrictions on uffd-wp VMAs in
try_to_unmap_one(), significantly simplifying the function's control
flow and complexity.
- "mm/early_ioremap: clarify and clean up early_ioremap_reset()"
(Sang-Heon Jeon)
Clarify and clean up the architecture-specific usage of
__late_set_fixmap() and __late_clear_fixmap() after
early_ioremap_reset()
It adds explicit documentation regarding when early_ioremap_reset()
must be called and removes redundant macro definitions and reset
calls in the RISC-V and ARM64 architectures.
- "mm: fix reclaim storms in defrag_mode" (Johannes Weiner)
Address severe performance regressions, swap storms, and spurious
OOMs caused by vm.defrag_mode=1 under high memory pressure in Meta
production
It updates the page allocator slowpath so non-movable allocation
requests actively trigger direct reclaim and direct compaction at
pageblock_order scale, allowing them to claim whole pageblocks rather
than spinning unproductively.
- "zram: lockmap tweaks" (Sebastian Siewior)
Optimize and fix lockdep tracking for zram devices by consolidating
per-entry lockmaps and isolate lock classes across multiple instances
This reduces memory overhead by replacing per-entry lockdep_map
instances with a single map per struct zram, and assigns a dynamic
lock_class_key to each instance to prevent false deadlock reports
when different zram devices are backed by distinct filesystems.
* tag 'mm-stable-2026-08-18-18-39' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (501 commits)
selftests/mm: thuge-gen: fix test_shmget() for PAGE_SIZE check
selftests/mm: unpoison pages in memory-failure teardown
mm/shmem: downgrade final i_blocks check in shmem_evict_inode() to pr_warn()
mm/khugepaged: replace mutex_lock/mutex_unlock usage with guard macro
mm/zsmalloc: fix release order of locks in zs_page_migrate()
Documentation: zram: remove sections numbering
ksm: stop iterating VMAs when ksm_test_exit returns true
mm: fold userfaultfd_rwp() to false without CONFIG_ARCH_HAS_PTE_PROTNONE
mm/migrate: report RCU-tasks quiescent states in migrate_pages_batch()
zram: use a custom key for each zram object
zram: move lockmap to be per-zram instead per table
selftests/mm: fix gup_longterm EINVAL error message
mm: page_alloc: fix non-movable reclaim storm in defrag_mode
mm: page_alloc: move capture_control to the page allocator
mm: compaction: support non-movable compaction for pageblock requests
mm: page_alloc: __GFP_FS lockdep annotation for direct compaction
hugetlb: evaluate subpool free state while locked
mm/damon: remove trailing semicolons after function definitions
mm/damon/ops-common: prevent migration fallback to non-target nodes
mm/damon: update outdated comment about DAMOS filter handling
...
|
|
Pull x86 cpuid updates from Borislav Petkov:
- Get rid of static_cpu_has() - one less API to care about testing CPU
features
- Unify the handling of CPU core types (performance, efficient, etc) by
mapping the vendor-specific types to Linux ones
- Continuation of the work of Ahmed Darwish to centralize CPUID leaf
representation
* tag 'x86_cpu_for_v7.3_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/CPU: Rename struct cpuid_read_output to struct cpuid_output
x86/cpu/scattered: Sort it properly
x86/cpu: Use parsed CPUID(0x1)
x86/lib: Add CPUID(0x1) family and model calculation
x86/cpu: Use parsed CPUID(0x0)
x86/cpu/transmeta: Rescan CPUID(0x1) after modifying capabilities
x86/topology: Add TOPO_CPU_TYPE_LOW_POWER
x86/topology: Name the AMD core-type values
x86/topo: Map vendor CPU types to generic Linux such types
x86/bugs: Don't use cpu-type matching in cpu_vuln_blacklist
x86/cpu: Hide and rename static_cpu_has()
|
|
Pull x86 TDX updates from Dave Hansen:
"TDX guests cause #VE exceptions whenever they do port I/O. The guest
then does some instruction decoding and makes a call up to the host
for help. That decoding had a couple of bugs.
Fix those bugs. Use an existing and now shared KVM helper for one of
them"
* tag 'x86_tdx_for_7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/tdx: Fix zero-extension for 32-bit port I/O
x86/insn-eval: Move assign_register() out of KVM as insn_assign_reg()
x86/tdx: Fix off-by-one in port I/O handling
|
|
Pull x86 MSR updates from Ingo Molnar:
- Streamline the x86 MSR handling APIs along the 64-bit variants,
simplifying the interfaces.
Removal of the old APIs is planned for the next cycle, to reduce
churn & integration pain (Juergen Gross)
* tag 'x86-msr-2026-08-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (21 commits)
x86/mce: Work around build warning after MSR-interface switch
cpufreq: Stop using 32-bit MSR interfaces
x86/featctl: Stop using 32-bit MSR interfaces
KVM/x86: Stop using 32-bit MSR interfaces
x86/mtrr: Stop using 32-bit MSR interfaces
acpi: Stop using 32-bit MSR interfaces
powercap: Stop using 32-bit MSR interfaces
thermal/intel: Stop using 32-bit MSR interfaces
x86/olpc: Stop using 32-bit MSR interfaces
x86/hyperv: Stop using 32-bit MSR interfaces
hwmon: Stop using 32-bit MSR interfaces
EDAC: Stop using 32-bit MSR interfaces
x86/cpu: Stop using 32-bit MSR interfaces
x86/apic: Stop using 32-bit MSR interfaces
x86/resctrl: Stop using 32-bit MSR interfaces
x86/tsc: Stop using 32-bit MSR interfaces
x86/amd: Stop using 32-bit MSR interfaces
x86/pci: Stop using 32-bit MSR interfaces
x86/hygon: Stop using 32-bit MSR interfaces
x86/mce: Stop using 32-bit MSR interfaces
...
|
|
KVM x86 misc changes for 7.3
- Fix VPID virtualization bugs where KVM would fail to flush hardware TLBs.
- Harden the SNP and TDX "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion.
- Fix a variety of #DB priority bugs.
- Fix a class of races related to enabling Hyper-V emulation on a vCPU after
the vCPU is visible to the rest of KVM.
- Use static calls for nested virtualization ops.
- Move more KVM-internal code out of x86's kvm_host.h.
- Enumerate support for a variety of Zhaoxin instructions that don't require
explicit virtualization.
- Fix missing EFER validation bugs, including in the KVM_SET_SREGS* path.
- Harden kvm_vcpu_map() against double-mapping and thus leaking references.
- Misc fixes and cleanups, e.g. for largely benign syzkaller splats.
|
|
KVM SVM changes for 7.3
- Remove a dying VM from the GA Log notifier list before the VM is actually
destroyed, to fix a potential use-after-free.
- Don't pass FOLL_WRITE when registering encrypted memory regions, i.e. when
pinning SEV/SEV-ES guest memory, to fix a regression with file-backed memory
introduced by KVM's (correct) usage of long-term pins.
[This is correct because, while FOLL_WRITE was needed in the past to trigger
CoW unsharing, nowadays FOLL_LONGTERM does that already even without
FOLL_WRITE. And in fact, get_user_pages() actually disallows FOLL_WRITE
together with FOLL_LONGTERM. This change was acked by the MM maintainers.
For more inforamtion see commit ee1a586dd1fa2f245b3b753a3e44d9263a49240b.
- Paolo]
- Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled hosts to
fix a data corruption issue due to the PSP driver assigning to-be-written
pages to firmware (as required by the SNP specs).
- Unconditionally intercept ICBEP so that KVM generates the correct guest RIP
when handling an ICEBP-induced TASK_SWITCH #VMEXIT.
|
|
KVM VMX changes for 7.3
- Service local TLB flushes on a failed nested VM-Enter to fix a bug where KVM
could miss a TLB on a future, successful VM-Enter with the same L2 VPID.
- Cap the maximum value shoved into the VMX Preemption Timer to workaround an
erratum that affects all existing Intel CPUs that support CPUID 0x15.
|
|
KVM x86 MMU changes for 7.3
- Fix a bug where KVM would walk a newly created rmap without holding the rmap
lock (or mmu_lock) during aging.
- Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs.
|
|
KVM x86 PV clocks and timekeeping related changes for 7.3
- Remove a defunct masterclock update from kvm_xen_shared_info_init() that
could result in corrupting kvmclock, for a lose definition or "corrupting",
due to triggering an unnecessary switch to/from masterclock mode.
- Skip Xen runstate time updates if time has effectively gone backwards, so
that the guest doesn't report 100% steal time for a very, very long time.
- Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM was
updating the wrong sub-leaf, and upstream KVM will soon provide all the
information needed by userspace to populate the CPUID field itself.
|
|
KVM guest_memfd and x86 CoCo changes for 7.3
- Forcefully invalidate SNP VMSA pages if their backing guest_memfd page is
zapped/invalidated, e.g. due to a PUNCH_HOLE in response to a Page-State
Change request.
- Rework the so called "prepare" and "invalidate" guest_memfd hooks to prepare
for in-place private<=>shared conversion, and clean up a few warts along the
way.
|
|
KVM arch-neutral and documentation changes for 7.3
- Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's debugfs.
- Document some of the "fun" gotchas with the APIC base when creating IRQCHIPs
on x86.
- Add a per-VM bitmap to track which vCPU IDs have been "claimed" but for
which the vCPU isn't yet online, and use the bitmap to reject duplicate IDs
before calling into arch code. This allows arch code to consume vcpu_id
without having to worry about cross-vCPU clobbering (at least s390 and x86
have had related bugs).
- Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used for IPI
virtualization when the vCPU is freed to fix a use-after-free where hardware
will write to a freed vCPU's PID.
|
|
LoongArch KVM changes for v7.3
1. Advertise already-supported capabilities.
2. Some bug fixes about timer and MMIO.
3. Some hardening about interrupt injection.
4. Replace kvm_err() with kvm_pr_unimpl().
5. Add FPU/LSX/LASX test cases for selftests.
|
|
KVM/riscv changes for 7.3
- Svadu/Zicfiss/Zicfilp FWFT support for Guest
- Use try_cmpxchg for IMSIC MRIF RMW
- More arch-specific tracepoints in KVM RISC-V
- Eager Page Splitting for KVM RISC-V
- Optimize hfence request handling for SMP Guests
- Improve dirty log clearing by skipping zero bits in mask
- Guard HFENCE range loops against overflow
- CPU PM notifiers in KVM RISC-V for non-retentive idle states
- Fix kernel-mode vector context save/restore for Guest
|
|
Remove the code in kvm_cpuid() that dynamically updates the Xen TSC
info CPUID leaf at runtime, as KVM is updating the wrong sub-leaf anyway
(0x40000x03/2 EAX is the *host* TSC frequency per the Xen ABI, not the
guest frequency which belongs in 0x40000x03/0 ECX).
Simply drop the code instead of fixing it to fill the correct sub-leaf, as
modifying guest CPUID entries/output at runtime is generally undesirable,
and providing userspace the necessary data to fill the sub-leaf itself is
useful for other reasons, e.g. to fill the generic 0x40000010 timing leaf
and to provide exact scaling information to aid save/restore.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Link: https://patch.msgid.link/20260728144954.355376-23-dwmw2@infradead.org
[sean: tweak changelog to take this patch before the new uAPI]
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
When kvm_xen_update_runstate() is invoked to set a vCPU's runstate, the
time spent in the previous runstate is accounted. This is based on the
delta between the current KVM clock time, and the previous value stored
in vcpu->arch.xen.runstate_entry_time.
If the KVM clock goes backwards, that delta will be negative. Or, since
it's an unsigned 64-bit integer, very *large*. Linux guests deal with
that particularly badly, reporting 100% steal time for ever more (well,
for *centuries* at least, until the delta has been consumed).
So when a negative delta is detected, just refrain from updating the
runstate times until the KVM clock catches up with runstate_entry_time
again.
Also clamp steal_ns to delta_ns to prevent steal time from exceeding
the total elapsed time, and handle negative steal_ns (which can happen
if run_delay goes backwards across a scheduler update).
The userspace APIs for setting the runstate times do not allow them to
be set past the current KVM clock, but userspace can still adjust the
KVM clock *after* setting the runstate times, which would cause this
situation to occur.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Paul Durrant <paul@xen.org>
Link: https://patch.msgid.link/20260728144954.355376-21-dwmw2@infradead.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Explicitly clear role.invalid when deriving a child shadow page's role from
its parent to harden against bugs elsewhere in KVM, as violating KVM's
invariant that invalid pages are NOT on the list of active MMU pages leads
to use-after-free due to __kvm_mmu_prepare_zap_page() using list_add()
instead of list_move() when processing an invalid shadow page, i.e. makes a
bad situation far worse.
Yell loudly if the parent is invalid, as it means KVM has missed a validity
check, i.e. KVM is attempting to map memory using an invalid/obsolete root,
but continue on as the child is otherwise still a valid shadow page.
==================================================================
BUG: KASAN: slab-use-after-free in __kvm_mmu_get_shadow_page+0x1817/0x1860 [kvm]
Write of size 8 at addr ff11000153dd1368 by task repro/853
CPU: 1 UID: 1000 PID: 853 Comm: repro Not tainted 7.2.0-rc2-3aec122bdcaf-next-vm #5 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Call Trace:
<TASK>
dump_stack_lvl+0x4b/0x70
print_report+0x153/0x49c
kasan_report+0xbc/0xf0
__kvm_mmu_get_shadow_page+0x1817/0x1860 [kvm]
mmu_alloc_root+0x141/0x320 [kvm]
kvm_mmu_load+0x612/0x20f0 [kvm]
kvm_arch_vcpu_ioctl_run+0x3dd5/0x6150 [kvm]
kvm_vcpu_ioctl+0x5e4/0x10d0 [kvm]
__x64_sys_ioctl+0x131/0x1b0
do_syscall_64+0x67/0x5f0
entry_SYSCALL_64_after_hwframe+0x4b/0x53
</TASK>
Allocated by task 853:
kasan_save_stack+0x20/0x40
kasan_save_track+0x14/0x30
__kasan_slab_alloc+0x5f/0x70
kmem_cache_alloc_noprof+0xfe/0x2e0
__kvm_mmu_topup_memory_cache+0x135/0x530 [kvm]
paging64_page_fault+0x318/0x1e30 [kvm]
kvm_mmu_do_page_fault+0x21d/0x630 [kvm]
kvm_mmu_page_fault+0x18c/0x17b0 [kvm]
kvm_arch_vcpu_ioctl_run+0x1f35/0x6150 [kvm]
kvm_vcpu_ioctl+0x5e4/0x10d0 [kvm]
__x64_sys_ioctl+0x131/0x1b0
do_syscall_64+0x67/0x5f0
entry_SYSCALL_64_after_hwframe+0x4b/0x53
Freed by task 853:
kasan_save_stack+0x20/0x40
kasan_save_track+0x14/0x30
kasan_save_free_info+0x3b/0x60
__kasan_slab_free+0x43/0x70
kmem_cache_free+0xe2/0x400
kvm_mmu_commit_zap_page.part.0+0x1e2/0x310 [kvm]
kvm_mmu_free_roots+0x283/0x560 [kvm]
kvm_arch_vcpu_ioctl_run+0x33c8/0x6150 [kvm]
kvm_vcpu_ioctl+0x5e4/0x10d0 [kvm]
__x64_sys_ioctl+0x131/0x1b0
do_syscall_64+0x67/0x5f0
entry_SYSCALL_64_after_hwframe+0x4b/0x53
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Fixes: a770f6f28b1a ("KVM: MMU: Inherit a shadow page's guest level count from vcpu setup")
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Interaction between KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM and
KVM_CAP_VM_COPY_ENC_CONTEXT_FROM can cause two separate issues:
- in sev_migrate_from(), when the destination KVM is a mirror, the mirror
entry is moved from the source's list to the owner's mirror_vms list,
without holding the owner's lock unlike other writers of the owner's
mirror list (sev_vm_copy_enc_context_from(), sev_vm_destroy()).
A concurrent COPY or destroy can race with sev_migrate_from() and
corrupt the list.
- In sev_vm_destroy(), the *owner* is still active and could receive
concurrently a KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM that causes
sev->enc_context_owner to change. In this case the incorrect VM
receives kvm_put_kvm().
The second issue needs particular care because the owner could disappear
altogether (even though the race window is impossibly small) between
reading it and locking it. There is thus no way to perform the checks
under the owner lock without putting struct kvm under SLAB_TYPESAFE_BY_RCU
(which would allow kvm_get_kvm_safe() under RCU critical section).
It is much simpler to just use a global lock, since the critical
sections are so small and the new lock is always a leaf lock.
Fixes: b2125513dfc0 ("KVM: SEV: Allow SEV intra-host migration of VM with mirrors")
Cc: stable@vger.kernel.org
Reported-by: Shen Yongchao <grayhat@foxmail.com>
Link: https://lore.kernel.org/kvm/tencent_625C0F42824E542C72B34733392AF2C49709@qq.com/
Link: https://lore.kernel.org/kvm/tencent_DDC4E4352EC91CAC05A9A8F4E55E8C96730A@qq.com/
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Red Hat is seeing multiple reports of Windows memory corruptions
(and consequent BSODs) with hv-tlbflush=on, on AMD processors only.
The crashes, while extremely rare, happen even with a stock configuration,
but with Driver Verifier enabled they can be detected after approximately
200 VM hours. In particular, Alexander Lougovski measured the following:
- on AMD Turin, 15 crashes in 3300 VM hours
- on AMD Milan, 2 crashes in 500 VM hours (there are fewer hours
here due to the host being smaller)
- on Intel Sapphire Rapids, 0 crashes in 8000 VM hours
- on AMD Turin with full TLB flush (not exactly this patch but
similar), no crashes in ~2 weeks of run time which should also
be ~7000 VM hours
For Turin, the microcode version was 0x0b002162, which (assuming
this is the same issue) should not be affected by the problem listed in
https://knowledge.broadcom.com/external/article/419026/bsod-on-virtual-machines-running-on-amd.html;
on the other hand that problem should not apply to earlier processors.
AMD has not provided any information or analysis yet, and when we asked
we didn't know yet that it reproduced on Milan as well.
As to the workload, Alexander threw more or less everything at the same
time at the VM:
- a full Windows Defender scan every 30 minutes
- a disk I/O job
- a loop doing repeated mmap of system files (mostly to hope that
it triggers some consistency check in the Windows memory manager)
- SQL Express 2022 + StressDB (1.6M rows), with the host doing queries
(75% write/25% read) via sqlcmd
Driver Verifier is able to detect BSODs more or less at the same time as
the pages are freed. They mostly happen in the Windows Defender filter
driver, but occasionally also in the networking stack (e.g., afd.sys)
or elsewhere in the filesystem stack (e.g., fltmgr.sys).
The flush is issued from kvm_hv_vcpu_flush_tlb(), which receives the
cross-CPU requests from the Hyper-V TLB flush hypercalls via a kfifo
and is invoked by the KVM_REQ_HV_TLB_FLUSH request. The mechanism is
the same for both Intel and AMD, and the handler for both vendors is
a simple INVVPID(ADDR)/INVLPGA instruction.
Because the request is handled on the destination CPU, there is a question
of what happens if the VM is migrated across physical CPUs. In that case,
the INVLPGA instruction would use a stale svm->vmcb->control.asid; but
if anything that might do an *unnecessary* flush (on an asid that's being
used for another VM) and then pre_svm_run() would force a full TLB rebuild.
So, for lack of better ideas, this patch forces a full ASID bump in
svm_flush_tlb_gva(). To avoid paying the price on Intel and also to
avoid unnecessary loops on AMD, the flush_tlb_gva op now returns whether
it did a full flush or not; kvm_hv_vcpu_flush_tlb() takes note and exits
its loops immediately. While there is an obvious performance impact,
about half of the benefit from Hyper-V tlbflush is preserved (10% vs. 20%
on the SQL Server workload).
kvm_mmu_invalidate_addr() is the only other caller of the flush_tlb_gva op.
The change would have a performance impact on every intercepted INVLPG and,
for nested SVM, on every L1 INVLPGA. For INVLPGA specifically, this covers
the same suspected issue but for nested hypervisors, so it is correct to
apply the workaround; for INVLPG on shadow paging, instead, the impact
would be stronger and, due to lack of data, for now the use of INVLPGA is
left in place in svm_flush_tlb_gva().
Analyzed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Analyzed-by: Alexander Lougovski <alougovs@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Tidy up the x86 vPMU emulation code and its selftests by fixing comments
that no longer reflect the code along with assorted spelling and grammar
mistakes, and by dropping redundant blank lines reported by checkpatch.
No functional change intended.
Signed-off-by: Like Xu <likexu@tencent.com>
Link: https://patch.msgid.link/20260625090155.6326-1-likexu@tencent.com
[sean: keep quotes around "hit", shove sequences fix into separate patch]
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
The KVM clock is an interesting thing. It is defined as "nanoseconds
since the guest was created", but in practice it runs at two *different*
rates — or three different rates, if you count implementation bugs.
Definition A is that it runs synchronously with the CLOCK_MONOTONIC_RAW
of the host, with a delta of kvm->arch.kvmclock_offset.
But that version doesn't actually get used in the common case, where the
host has a reliable TSC and the guest TSCs are all running at the same
rate and in sync with each other, and kvm->arch.use_master_clock is set.
In that common case, definition B is used: There is a reference point in
time at kvm->arch.master_kernel_ns (again a CLOCK_MONOTONIC_RAW time),
and a corresponding host TSC value kvm->arch.master_cycle_now. This
fixed point in time is converted to guest units (the time offset by
kvmclock_offset and the TSC Value scaled and offset to be a guest TSC
value) and advertised to the guest in the pvclock structure. While in
this 'use_master_clock' mode, the fixed point in time never needs to be
changed, and the clock runs precisely in time with the guest TSC, at the
rate advertised in the pvclock structure.
The third definition C is implemented in kvm_get_wall_clock_epoch() and
__get_kvmclock(), using the master_cycle_now and master_kernel_ns fields
but converting the *host* TSC cycles directly to a value in nanoseconds
instead of scaling via the guest TSC.
One might naïvely think that all three definitions are identical, since
CLOCK_MONOTONIC_RAW is not skewed by NTP frequency corrections; all
three are just the result of counting the host TSC at a known frequency,
or the scaled guest TSC at a known precise fraction of the host's
frequency. The problem is with arithmetic precision, and the way that
frequency scaling is done in a division-free way by multiplying by a
scale factor, then shifting right. In practice, all three ways of
calculating the KVM clock will suffer a systemic drift from each other.
Eventually, definition C should just be eliminated. Commit 451a707813ae
("KVM: x86/xen: improve accuracy of Xen timers") worked around it for
the specific case of Xen timers, which are defined in terms of the KVM
clock and suffered from a continually increasing error in timer expiry
times. That commit notes that get_kvmclock_ns() is non-trivial to fix
and says "I'll come back to that", which remains true.
Definitions A and B do need to coexist, the former to handle the case
where the host or guest TSC is suboptimally configured. But KVM should
be more careful about switching between them, and the discontinuity in
guest time which could result.
In particular, KVM_REQ_MASTERCLOCK_UPDATE will take a new snapshot of
time as the reference in master_kernel_ns and master_cycle_now, yanking
the guest's clock back to match definition A at that moment.
When invoked from in 'use_master_clock' mode, kvm_update_masterclock()
should probably *adjust* kvm->arch.kvmclock_offset to account for the
drift, instead of yanking the clock back to definition A. But in the
meantime there are a bunch of places where it just doesn't need to be
invoked at all.
To start with: there is no need to do such an update when a Xen guest
populates the shared_info page. This seems to have been a hangover from
the very first implementation of shared_info which automatically
populated the vcpu_info structures at their default locations, but even
then it should just have raised KVM_REQ_CLOCK_UPDATE on each vCPU
instead of using KVM_REQ_MASTERCLOCK_UPDATE. And now that userspace is
expected to explicitly set the vcpu_info even in its default locations,
there's not even any need for that either.
Fixes: 629b5348841a ("KVM: x86/xen: update wallclock region")
Reviewed-by: Paul Durrant <paul@xen.org>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
vCPU creation in kvm_vm_ioctl_create_vcpu() may fail after
kvm_arch_vcpu_create() -> vmx_vcpu_create() already succeeded. In such
case kvm_vm_ioctl_create_vcpu() destroys the newly created vCPU in the
failure path. However, that leaves a side effect: the IPIv pid_table
entry remains configured with this vCPU's pi_desc address. As a result,
when another vCPU sends an IPI to the APIC ID of this failed-to-create
vCPU, it will cause HW to write to this (freed!) pi_desc memory. [*]
Fix this by clearing the pid_table entry when destroying the vCPU.
Note that the same issue exists for SVM AVIC as well [1], to be fixed.
[*] Although, since this memory is freed into the kvm_vcpu_cache kmem
cache which is only used for allocating kvm_vcpus, _maybe_ this
memory will only be reused for pi_desc of another vCPU, not for
anything else. So _maybe_ this will only result in delivering the
IPI to a wrong vCPU (possibly of another VM) in the worst case, not
in a random corruption of kernel memory.
Link: https://lore.kernel.org/kvm/al4rNqpBYy8FGKPw@blrnaveerao1 [1]
Signed-off-by: Dmytro Maluka <dmaluka@chromium.org>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Link: https://patch.msgid.link/20260729170621.308809-3-dmaluka@chromium.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Rework .vcpu_pre_run() into a more specific .vcpu_needs_initialization() to
consolidate the SNP and TDX control flows, and to eliminate the potentially
confusing almost-collision between svm_vcpu_pre_run() and pre_sev_run(): the
former is SEV specific, but is pre-KVM_RUN, whereas the latter is pre-VMRUN.
Link: https://patch.msgid.link/20260731173340.2644656-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Synthesize SHUTDOWN (for L1) if L2 requires unhandleable emulation after
loading guest state from SMRAM during RSM to prevent a misbehaving L1 (or
userspace via L1) from tripping the sanity check that KVM doesn't try to
cancel a pending nested VM-Enter. If SMRAM is modified such that RSM will
load what should be impossible state for L2, then KVM will detect that it
needs to emulate the current code stream and will abort VM-Entry to L2.
And because KVM (rightly) expects such a scenario to be impossible, KVM
WARNs and bugs the VM.
__ret && !(vcpu->kvm)->vm_bugged
WARNING: arch/x86/kvm/vmx/vmx.c:6741 at vmx_handle_exit+0x65/0x790 [kvm_intel], CPU#13: vmx_invalid_nes/2902
Modules linked in: kvm_intel kvm irqbypass [last unloaded: kvm]
CPU: 13 UID: 1000 PID: 2902 Comm: vmx_invalid_nes Tainted: G W 7.2.0-rc2 #124 PREEMPT
Tainted: [W]=WARN
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:vmx_handle_exit+0x65/0x790 [kvm_intel]
Call Trace:
<TASK>
kvm_arch_vcpu_ioctl_run+0xdf8/0x1d00 [kvm]
kvm_vcpu_ioctl+0x2d5/0x960 [kvm]
__x64_sys_ioctl+0x8a/0xd0
do_syscall_64+0xb7/0x570
entry_SYSCALL_64_after_hwframe+0x4b/0x53
</TASK>
Alternatively, KVM could suppress the WARN for the RSM case, but that would
still leave the vCPU in a "bad" state that KVM doesn't know how to handle
(which is also why KVM rejects attempts to do KVM_RUN when the vCPU is
loaded with invalid state). And architecturally, the Intel SDM explicitly
states that RSM leads to shutdown if the CPU detects invalid state.
Fixes: 2bb8cafea80b ("KVM: vVMX: signal failure for nested VMEntry if emulation_required")
Reported-by: Hao Zhang <zhanghao1@kylinos.cn>
Link: https://patch.msgid.link/20260731173340.2644656-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Expose VMX's check for unhandleable emulation as its own kvm_x86_ops hook,
and move the actual pre-KVM_RUN check into common x86. This will allow
sharing the core logic with KVM's RSM emulation without needed to add a
post-RSM hook, and is a step towards removing the .vcpu_pre_run() hook
entirely.
Alternatively, KVM could provide a post-RSM hook as mentioned, but pre/post
hooks tend to be unwieldy as the exact "timing" of the call often matters
greatly. E.g. in this case, the call must slot in exactly between loading
guest state from SMRAM and the hack to force the vCPU out of L2 on SHUTDOWN.
Link: https://patch.msgid.link/20260731173340.2644656-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Due to a widespread Intel erratum (e.g. EMR158), programming the
VMX-preemption timer with certain large values may cause the timer to
expire earlier than expected. The recommended workaround is to cap the
VMX-preemption timer value to strictly less than:
2^25 * CPUID.15H:EBX[31:0] / CPUID.15H:EAX[31:0].
Calculate the maximum "safe" preemption timer value during hardware setup
based on CPUID 15H when available, and use the adjusted max value in all
locations where KVM currently hardcodes the max architectural value,
including in the subtle case where KVM soft-disables the timer.
Don't apply the workaround when running as a VM, because absent explicit
enumeration to state the bug is present (or not), it's L0's responsibility
to faithfully emulate/virtualize the VMX preemption timer.
WARN if the above logic would result in a max value of zero and fall back
to the maximum architectural value, as the expectation is that real
hardware will never provide problematic EAX/EBX values (which is another
reason to ignore the erratum when running as a VM; there's less chance of
a false positive on the WARN due to L0 providing an unanticipated ratio).
Reported-by: Sean Christopherson <seanjc@google.com>
Closes: https://lore.kernel.org/all/Zn9X0yFxZi_Mrlnt@google.com/
Suggested-by: Chao Gao <chao.gao@intel.com>
Assisted-by: Gemini:Gemini-Next
Reviewed-by: Chao Gao <chao.gao@intel.com>
Signed-off-by: Jim Mattson <jmattson@google.com>
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
[sean: track inclusive max instead of exclusive limit, massage changelog]
Link: https://patch.msgid.link/20260731171926.2629627-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Double down on using the VMX preemption timer only for 64-bit kernels, and
bury the setup and runtime adjustment code, and all global variables, under
CONFIG_X86_64=y. This will allow addressing a widespread Intel erratum
without running afoul of unused-but-set-variable and __udivdi3() errors on
32-bit kernels.
No functional change intended.
Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
Link: https://patch.msgid.link/20260731171926.2629627-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
The difference between __alloc_pages_node() and alloc_pages_node() is
that the latter allows you to pass NUMA_NO_NODE.
The former is going away and the latter works fine here so switch over.
No functional change intended.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-9-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Assisted-by: Gemini:unknown-version
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Always update x2APIC MSR intercepts for L1 when AVIC is deactivated, even
if L2 is active and KVM is using a separate MSR bitmap to run L2. If AVIC
is fully enabled prior to running L2, and is then inhibited while L2 is
active (for a VM-scoped inhibit), then KVM will run L1 with AVIC disabled,
but with x2APIC MSR intercepts disabled, i.e. will allow L1 to read most of
the host's APIC state, send arbitrary interrupts, change task priority, and
ultimately trivially DoS the host.
E.g. sending a self-IPI in L1 on HYPERV_REENLIGHTENMENT_VECTOR, 0xee, with
CONFIG_HYPERV=n in the host kernel as a "safe" PoC, yields:
Spurious interrupt (vector 0xee) on CPU#425. Acked
And hacking KVM to abuse kvm_set_posted_intr_wakeup_handler() to register a
handler and WARN on POSTED_INTR_WAKEUP_VECTOR yields:
------------[ cut here ]------------
WARNING: arch/x86/kvm/svm/svm.c:5594 at pi_wakeup_handler+0x9/0x10 [kvm_amd], CPU#156: nested_x2apic_t/316940
CPU: 156 UID: 0 PID: 316940 Comm: nested_x2apic_t Tainted: G S U
Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER
Hardware name: Google Astoria-Turin/astoria, BIOS 0.20260209.0-0 02/09/2026
RIP: 0010:pi_wakeup_handler+0x9/0x10 [kvm_amd]
Call Trace:
<IRQ>
sysvec_kvm_posted_intr_wakeup_ipi+0x64/0x80
</IRQ>
<TASK>
asm_sysvec_kvm_posted_intr_wakeup_ipi+0x1a/0x20
RIP: 0010:vcpu_run+0x1430/0x1e40 [kvm]
kvm_arch_vcpu_ioctl_run+0x2c1/0x600 [kvm]
kvm_vcpu_ioctl+0x580/0x6b0 [kvm]
__se_sys_ioctl+0x6d/0xb0
do_syscall_64+0x10a/0x480
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x46ff4b
</TASK>
---[ end trace 0000000000000000 ]---
Fixes: 091abbf578f9 ("KVM: x86: nSVM: optimize svm_set_x2apic_msr_interception")
Cc: stable@vger.kernel.org
Cc: Yosry Ahmed <yosry@kernel.org>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Link: https://patch.msgid.link/20260729213558.639074-1-pbonzini@redhat.com/
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Use CMPXCHG instead of clear_bit(), which currently emits a LOCK BTR since
the to-be-cleared bit isn't a compile-time constant, when aging SPTEs in
the shadow MMU to align with the approach taken by the TDP MMU, and because
using CMPXCHG is far more robust against bugs in KVM. E.g. if the SPTE is
somehow no longer an SPTE due to a KVM bug, CMPXCHG will fail gracefully,
whereas clear_bit() would potentially corrupt/clobber memory.
Clearing the Accessed bit without atomically ensuring the SPTE is still the
old SPTE is "fine", as holding the rmap's lock ensures zapping the old SPTE
can't fully complete, which in turn ensures a new, different SPTE can't be
installed. But that chain of logic isn't exactly obvious, and there's zero
reason to avoid CMPXCHG as its cost on modern hardware is within ~1-2 uops
of LOCK BTR (and may even be cheaper on some microarchitectures). Doing a
64-bit CMPXCHG on 32-bit kernels does require a more expensive CMPXCHG8B,
but 32-bit KVM is all but dead at this point.
Cc: James Houghton <jthoughton@google.com>
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: James Houghton <jthoughton@google.com>
Link: https://patch.msgid.link/20260728002236.869865-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Use LOCK CMPXCHG instead of LOCK AND to clear the Accessed bit when aging
SPTEs in the TDP MMU, as doing a LOCK AND can corrupt a FROZEN SPTE and
allow a third CPU to effectively overwrite the FROZEN SPTE. As pointed
out by AI of some kind, because the magic FROZEN_SPTE value is a "full"
SPTE, not a single bit, and includes the Accessed bit, clearing the
Accessed bit in a FROZEN SPTE will result in is_frozen_spte() getting a
false negative.
E.g. if CPU0 freezes an SPTE, and CPU1 clears the Accessed bit in the
frozen SPTE, then CPU2 could come along and overwrite the frozen SPTE with
a shadow-present SPTE.
Thankfully, the false negative is largely benign, because outside of TDX,
which doesn't support aging, KVM only freezes leaf SPTEs when removing an
upper level shadow page. So while KVM could clobber a frozen SPTE back to
a shadow-present SPTE, and could even use the new SPTE, the subsequent TLB
flush will make the orphaned, shadow-present SPTE unreachable. Failure to
ever zap the orphaned leaf SPTE would show up in KVM's stats, but otherwise
is benign (because KVM no longer keeps an elevated refcount for leaf SPTEs).
Opportunistically add a comment to warn future developers away from using
kvm_tdp_mmu_write_spte_atomic() and tdp_mmu_clear_spte_bits_atomic(), as
they are generally unsafe. Keep the helpers, e.g. instead of open-coding
the atomic64_fetch_and() in tdp_mmu_clear_spte_bits(), as scary warnings
usually are more effective deterrent against recidivism than removal of the
dangerous code.
Alternatively, KVM could use different bits for the magic FROZEN_SPTE value,
e.g. setting the Dirty bits (with effective IPAT and Global aliases) would
likely be "ok", as IPAT/Global are extremely unlikely to be cleared without
doing a full SPTE write, and KVM's clearing of Dirty bits shares logic with
Write-Protection, which must do a full SPTE write (via cmpxchg64() in the
TDP MMU) to ensure KVM isn't clobbering state. But there is zero reason to
carry that risk (beyond stubbornness in wanting to preserve a "cute" idea),
as the cost of LOCK CMPXCHG and LOCK AND are within 1-2 uops of each other
on modern hardware.
Fixes: b146a9b34aed ("KVM: x86/mmu: Age TDP MMU SPTEs without holding mmu_lock")
Cc: stable@vger.kernel.org
Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: James Houghton <jthoughton@google.com>
Link: https://patch.msgid.link/20260728002236.869865-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Convert the kvm_vcpu_map_readonly() usage in nVMX's MSR bitmap merging to
the new CLASS(kvm_vcpu_map_local_readonly) implementation, to eliminate the
last of the open-coded on-stack "struct kvm_host_map" declarations (in x86,
PPC still has one more to convert).
No functional change intended.
Link: https://patch.msgid.link/20260724004757.131420-5-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Convert the kvm_vcpu_map() usage in the enter/leave SMM flows to the new
CLASS(kvm_vcpu_map_local) implementations, to eliminate the need to
manually do unmap() in error paths, and more importantly to eliminate more
of the open-coded on-stack "struct kvm_host_map" declarations.
No functional change intended.
Link: https://patch.msgid.link/20260724004757.131420-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Add CLASS() definitions for locally mapping a PFN using kvm_vcpu_map()
given a vCPU+gfn pair. In addition to eliminating the need to manually
do unmap(), e.g. in error paths, this will allow hardening KVM against
double-mapping without having to manually ensure every on-stack declaration
is zero-initialized.
Use "map local" as the primary terminology as the basic concept is more or
less the same as kmap_local(): ensure the current context has a kernel
mapping to the underlying memory.
Immediately convert the relatively straightforward nested SVM flows, and
defer converting the more involved SMM flows to a separate change.
No functional change intended.
Cc: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260724004757.131420-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Always check and clear KVM_REQ_GET_NESTED_STATE_PAGES when emulating a
nested VM-Exit to ensure the request is cleared, even when KVM was built
with CONFIG_KVM_HYPERV=n, as KVM subtly relies on the "check" to clear
the flag and thus avoid double-mapping the vmcs12 pages, e.g. if KVM
manages to bail from VM-Enter without processing the request, and then
emulates VMLAUNCH or VMRESUME.
Fixes: b4f69df0f65e ("KVM: x86: Make Hyper-V emulation optional")
Cc: stable@vger.kernel.org
Reported-by: Yosry Ahmed <yosry@kernel.org>
Reviewed-by: Yosry Ahmed <yosry@kernel.org>
Link: https://patch.msgid.link/20260724004757.131420-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
mmu_try_to_unsync_pages() skips the shadow-page lookup when the
supplied memslot allows a hugepage, because a shadow page would disallow
hugepages. But hugepage metadata is per-address-space while shadow pages
are shared across all address spaces. With SMM, the other address space
can therefore have a shadow page even when the supplied memslot allows a
hugepage.
Check the corresponding memslot in the other address space before
taking the fast path. Skip the shadow-page lookup only when all address
spaces allow a hugepage.
Fixes: b3ae3ceb5569 ("KVM: x86/mmu: KVM: x86/mmu: Skip unsync when large pages are allowed")
Assisted-by: Codex:GPT-5
Signed-off-by: Jinu Kim <kimjw04271234@gmail.com>
[invert direction of the conditional. - Paolo]
Message-ID: <20260721103512.2136240-3-kimjw04271234@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
kvm_gfn_is_write_tracked() checks only the supplied memslot, but page
tracking is per-address-space and shadow pages are shared across all
address spaces. With SMM, a GFN can therefore be write-tracked in one
address space and appear untracked through the other.
Check the supplied slot first, then the slot for the other address space.
This ensures all callers honor write tracking regardless of the active
address space. In particular, it prevents mmu_try_to_unsync_pages() from
marking an upper-level shadow page unsync and eventually triggering the
BUG in pte_list_remove().
Fixes: 699023e23965 ("KVM: x86: add SMM to the MMU role, support SMRAM address space")
Assisted-by: Codex:GPT-5
Signed-off-by: Jinu Kim <kimjw04271234@gmail.com>
Message-ID: <20260721103512.2136240-2-kimjw04271234@gmail.com>
[invert direction of the conditional. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Cancel (and flush) the I/O APIC's delayed EOI handling work during the
"pre VM destroy" phase, before vCPUs are destroyed, as processing the EOI
broadcast will inject another IRQ if the line is asserted, i.e. will try
to deliver an IRQ to the target vCPU(s). Canceling the work after vCPUs
are destroyed leads to UAF if the delayed work is processed after vCPUs are
destroyed.
BUG: KASAN: slab-use-after-free in __kvm_irq_delivery_to_apic_fast+0x9bf/0xa20 arch/x86/kvm/lapic.c:1250
Read of size 8 at addr ffff8880499abea0 by task kworker/1:2/1218
CPU: 1 UID: 0 PID: 1218 Comm: kworker/1:2 Not tainted 7.1.0-rc7 #5 PREEMPT(lazy)
Hardware name: QEMU Ubuntu 25.10 PC v2 (i440FX + PIIX, + 10.1 machine, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events kvm_ioapic_eoi_inject_work
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:94
dump_stack_lvl+0x100/0x190 lib/dump_stack.c:120
print_address_description mm/kasan/report.c:378
print_report+0x139/0x4ad mm/kasan/report.c:482
kasan_report+0xe4/0x1d0 mm/kasan/report.c:595
__kvm_irq_delivery_to_apic_fast+0x9bf/0xa20 arch/x86/kvm/lapic.c:1250
__kvm_irq_delivery_to_apic+0xd8/0xbf0 arch/x86/kvm/lapic.c:1345
kvm_irq_delivery_to_apic arch/x86/kvm/lapic.h:129
ioapic_service+0x308/0x590 arch/x86/kvm/ioapic.c:492
kvm_ioapic_eoi_inject_work+0x13c/0x190 arch/x86/kvm/ioapic.c:532
process_one_work+0xa59/0x19a0 kernel/workqueue.c:3314
process_scheduled_works kernel/workqueue.c:3397
worker_thread+0x5eb/0xe50 kernel/workqueue.c:3478
kthread+0x370/0x450 kernel/kthread.c:436
ret_from_fork+0x72b/0xd30 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Note, the VM is unreachable once kvm_destroy_vm() starts, and scheduling
new work via kvm_ioapic_send_eoi() can only be done via KVM_RUN, i.e.
requires a live vCPU.
Alternatively, KVM could simply destroy the I/O APIC during the "pre" phase
of VM destruction, but that gets more than a bit sketchy as KVM expects the
I/O APIC to exist if ioapic_in_kernel() is true, and nested virtualization
in particular has a bad habit of touching VM-scope state during vCPU
destruction. E.g. attempting to free the PIC during the pre phase would
lead to a NULL pointer dereference in kvm_cpu_has_extint(), and it's not
hard to imagine the I/O APIC having a similar flaw.
Fixes: 17bcd7144263 ("KVM: x86: Free vCPUs before freeing VM state")
Reported-by: <zdi-disclosures@trendmicro.com>
Reported-by: Zhong Wang <wangzhong.c0ss4ck@bytedance.com>
Reported-by: Xuanqing Shi <shixuanqing.11@bytedance.com>
Cc: stable@vger.kernel.org
Signed-off-by: Weiming Shi <bestswngs@gmail.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Message-ID: <20260727171718.543491-1-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
VMCLEAR/VMREAD/VMWRITE/VMPTRLD access the internal VMCS cache, which
is not visible to the compiler; without a memory clobber, the compiler
can reorder them in troublesome ways because "asm volatile" and "asm goto"
only protect against removal of the asm. For example, placing a VMWRITE
before the corresponding VMCS pointer is loaded can lead to corruption.
While none of this has been observed, it is better to prevent than cure.
Likewise, INVEPT and INVVPID access the TLB and, even though in their
case the effect is only visible to the next VMLAUNCH/VMRESUME, it is
technically correct to add the clobber there too. So avoid any urge to
special case them, and simply hardcode "memory" into the clobber list
of vmx_asm1() and vmx_asm2(). __vmcs_readl() open-codes its own asm,
so add the clobber there as well.
Link: https://lore.kernel.org/kvm/CABgObfbL3t21yVeSwiLSjjOUER+rTYDPHYAH9YU4TWGRjx6XHg@mail.gmail.com/
Cc: Sean Christopherson <seanjc@google.com>
Cc: stable@vger.kernel.org
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
When getting a to-be-injected IRQ, don't WARN if the IRQ disappeared and
Xen emulation is supported, as a guest could concurrently toggle its
evtchn_upcall_pending flag in shared memory and deassert the IRQ. Even
more annoyingly, userspace could disable Xen emulation for the entire VM
KVM_XEN_HVM_CONFIG. So, suppress WARNs on lost IRQs if Xen emulation is
supported to prevent false positives.
Alternatively, KVM could track if the VM has ever used Xen emulation, but
the added complexity isn't worth carrying given that the vast majority of
deployments can and should disable Xen emulation.
Fixes: bf672720e83c ("KVM: x86: check the kvm_cpu_get_interrupt result before using it")
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260625212001.3B6561F000E9@smtp.kernel.org
Link: https://patch.msgid.link/20260724173425.278753-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
When getting a to-be-injected IRQ, don't WARN if the IRQ disappeared and
the VM has an in-kernel PIC, as the ExtINT handling that's routed through
KVM's virtual PIC is tracked per-VM, not per-vCPU. If another vCPU grabs
the IRQ, or deasserts the interrupt (which is level-triggered), then it's
both expected and "fine" for a
Keep the assert for split IRQCHIP VMs to help detect KVM bugs, as userspace
is responsible for routing ExtINT to the intended vCPU, i.e. once an ExtINT
is pending, it can't be cleared without holding the vCPU's mutex, and thus
false positives are impossible.
Fixes: bf672720e83c ("KVM: x86: check the kvm_cpu_get_interrupt result before using it")
Debugged-by: Alexander Potapenko <glider@google.com>
Reported-by: syzbot+dd769db18693736eee89@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=dd769db18693736eee89
Closes: https://lore.kernel.org/all/6a360fdf.871e809a.2d6dda.0000.GAE@google.com
Link: https://patch.msgid.link/20260724173425.278753-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
The default case in __apic_accept_irq() has carried a printk("TODO:
unsupported delivery mode") since the original LAPIC emulation was
introduced in commit 97222cc83163 ("KVM: Emulate local APIC in
kernel").
The switch now handles all eight delivery modes defined by the x86
architecture and is always constrained to the three-bit field
defined by the architecture: Either by masking with APIC_MODE_MASK,
by three-bit bitfield widths in the IOAPIC and MSI structs, or by
using APIC_DM_* constants directly.
Replace the unreachable printk with WARN_ON_ONCE(1) to match the
existing pattern for impossible defaults elsewhere in the same file.
Signed-off-by: Tim Wiederhake <twiederh@redhat.com>
Link: https://patch.msgid.link/20260715120341.2661873-1-twiederh@redhat.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Tweak the guest_memfd make_private() hook to explicitly pass the number of
pages to align with the signature of the make_shared() hook, and because
the existing code is outright broken if a guest_memfd folio is comprised of
more than one page (which can't happen, yet). The SNP code *tries* to
create a corresponding huge entry, but if the RMP must use 4KiB entries for
whatever reason, KVM will only convert the first pfn, and not the entire
range of pfns that will be mapped into the guest.
Alternatively, @max_order could simply be repurposed as _the_ @order, but
that will fall apart when in-place conversion comes along, at which point
KVM will need to deal with conversions that aren't bound 1:1 to a folio.
I.e. the number of pages to convert may not be exactly a power-of-2 (and
folios don't strictly guarantee power-of-2 pages anyways).
WARN in the SNP code if the number of pages to prepare is anything other
than '1', i.e. if guest_memfd is trying to prepare/convert more than a
single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion
greater than order-0 folios.
Opportunistically swap the ordering of @pfn and @gfn params for
kvm_x86_ops.gmem_make_private() to match kvm_arch_gmem_make_private().
Fixes: b85524314a3d ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Link: https://patch.msgid.link/20260723210811.72720-9-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
Paolo Bonzini
Sean Christopherson
Jim Mattson
Linus Torvalds
David Woodhouse
Like Xu
Dmytro Maluka
Brendan Jackman
Jinu Kim
Weiming Shi
Tim Wiederhake