| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
KVM fixes for 7.3-rcN
- Fix a brown paper bag bug where KVM would incorrectly treat Intel PMU MSRs
as valid on AMD.
- Fix a regression in the hardware disable selftest where it checked the wrong
macro when detecting glibc support (breaks at least musl).
- Never clear KVM_REQ_VM_DEAD so that dead VMs stay dead, which is especially
important for KVM_BUG_ON() flows, which often guard more dangerous bugs.
- Re-pend GET_NESTED_STATE_PAGES if getting the pages fails, to fix a bug
where KVM would let userspace run a broken setup with stale vmcs12 pages.
- Fix a class of bugs where KVM would fail to fill kvm_run exit fields if
getting nested pages failed.
- Treat reserved entries in the memory attributes xarray as "no attributes",
to fix false positives when checking for mixed attributes.
- Fix memcg accounting for the memory attributes xarray (the xarray library
subtly requires the xarray to be configured for accounting upfront; the gfp
flags taken at runtime are used only rarely).
- Don't pre-reserve xarray entries when storing empty attributes, as storing
NULL must not require memory allocation (KVM and other subsystems heavily
rely on this behavior).
|
|
The consolidation of the breakpoint and watchpoint register accessors
switched DBGBXVR<n> from trap_bvr() to trap_dbg_wb_reg(). The latter
selects backing storage with demux_wb_reg(), which only handles Op2 values
4 through 7. Since DBGBXVR<n> uses Op2 1, an AArch32 guest access hits
KVM_BUG_ON() and marks the VM dead.
DBGBXVR<n> aliases DBGBVR<n>_EL1[63:32], and its AA32(HI) descriptor
already selects the upper half. Map Op2 1 to dbg_bvr[] alongside Op2 4,
restoring the pre-regression behavior.
Fixes: 3ce9f3357e9e ("KVM: arm64: Fold DBGxVR/DBGxCR accessors into common set")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-5
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260810005616.13227-1-kmehltretter@gmail.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
The encoding to trap mapping currently maps a FGT on OP_GCSPOPX to
HFGITR_EL2.nGCSEPP but as per DDI0601 2026-06 this FGT controls trapping
of GCSPUSHX and GCSPOPCX, and not the separate GCSPOPX instruction.
Update the mapping to reflect the architecture.
Fixes: 863ac38984a82 ("KVM: arm64: Add missing HFGITR_EL2 FGT entries to nested virt")
Reviewed-by: Leonardo Bras <leo.bras@arm.com>
Signed-off-by: Mark Brown <broonie@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://patch.msgid.link/20260901-arm64-gcs-v20-2-f31750bdfadb@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
We free the shadow S2 structures from kvm_arch_flush_shadow_all(), which
is a Bad Idea(tm). Freeing the page tables is fair game (this is what
this callback is for), but freeing the container that could still be
referenced by another part of the system is not great.
Instead, grow separate destructors that gets called when we tear the VM
down for good. From there, we can nuke both the individual MMUs as well
as the global array that points to them, safe in the knowledge that the
vcpus themselves have been destroyed already.
Fixes: 4f128f8e1aaac ("KVM: arm64: nv: Support multiple nested Stage-2 mmu structures")
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260911162203.1919330-3-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
The nested_mmus array holds the shadow page tables that are used when
a guest is running a nested context. These structures are allocated on
VCPU_INIT for whole guest, which implies that they may have to be
relocated as the array grows.
Should a VCPU_INIT occur whilst a vcpu is actively running an L2 and
that the allocation requires relocation, that vcpu will still be
running with a pointer to the previous structure, which will have been
freed.
Fix this by turning the array of structures to an array of pointers,
which is now allocated at VM creation, sized to the absolute maximum
that KVM can handle.
In turn, each VCPU_INIT contributes S2_MMU_PER_VCPU to the pool. No
reallocation is ever performed, and the life cycle of each object is
much clearer:
- the nested_mmus array is allocated in kvm_init_nested(), and freed
in kvm_arch_destroy_vm()
- s2_mmu structures are allocated in kvm_vcpu_init_nested(), and freed
on kvm_arch_flush_shadow_all()
Finally, the freeing of vcpu->arch.vncr_array is made consistent
rather than being done on some failure paths, but not others.
Fixes: 4f128f8e1aaa ("KVM: arm64: nv: Support multiple nested Stage-2 mmu structures")
Reported-by: Shen Yongchao <grayhat@foxmail.com>
Reported-by: Karl Mehltretter <kmehltretter@gmail.com>
Suggested-by: Karl Mehltretter <kmehltretter@gmail.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://lore.kernel.org/r/20260803224405.41468-1-kmehltretter@gmail.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260911162203.1919330-2-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
fix_host_ownership() transfers only what it walks, so a hyp mapping
outside the linear map is not manipulated by the walk.
Walk the quarter of the VA space holding the private range and the
vmemmap once the transfer is done, and fail init unless every valid
leaf is hyp-owned: in the vmemmap when the page is memory, and in the
host stage-2, where hyp text may instead be mapped without write
access. A leaf that is not memory has no vmemmap entry and is checked
against the host stage-2 alone. Hyp text is matched by physical
address, since the only executable mapping in the range is the
Spectre-v3a vectors, whose VA is a private allocation, and an
executable mapping of anything else must not be host-readable. The
vmemmap can be block-mapped, so the walker checks each page of a leaf.
Suggested-by: Will Deacon <will@kernel.org>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-5-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
__io_map_base is the start of the private VA range only until the first
allocation from it, after which it is the allocation cursor. Keep it as
the start and move the cursor to __io_map_next, for the walk of the
range the next patch adds.
No functional change intended.
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-4-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
On a non-hVHE host, fix_host_ownership_walker()'s test for PAGE_HYP_EXEC
never matches: KVM_PGTABLE_PROT_UX is cleared at map time and only PX is
reported on read-back. Hyp text is therefore donated rather than left
read-only in the host stage-2, and the instruction dump in
nvhe_hyp_panic_handler() reads a page the host has no access to.
Match the text by physical address instead, in a helper a later patch
reuses. A test on the permissions would leave any other executable
mapping host-readable too.
Fixes: 80cbfd7174f31 ("KVM: arm64: Honor UX/PX attributes for EL2 S1 mappings")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
fix_host_ownership() walks only the linear-map alias of each memblock
region, and the per-CPU hyp stack, mapped in the private VA range for
its guard page, has none.
Walk each stack's VA range with the same walker.
Fixes: 1a919b17ef012 ("KVM: arm64: Add guard pages for pKVM (protected nVHE) hypervisor stack")
Reported-by: Hiroyuki Katsura <hk590@cam.ac.uk>
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Tested-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260908110713.1540304-2-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
kvm_smccc_set_filter() only rejects a range if its inclusive end,
base + nr_functions - 1, is below base. That catches an empty range
(nr_functions == 0) at every nonzero base, but at base 0 the end wraps
to U32_MAX and KVM tries to insert [0, U32_MAX], which overlaps the
reserved Arm Architecture Calls ranges. KVM_ARM_VM_SMCCC_FILTER then
returns -EEXIST instead of the -EINVAL that the smccc_filter selftest
expects for an empty range.
Reject a zero function count explicitly.
Tested with a userspace reproducer on an arm64 VHE host under QEMU TCG:
EEXIST before, EINVAL after.
Fixes: 821d935c87bc ("KVM: arm64: Introduce support for userspace SMCCC filtering")
Assisted-by: LLM
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Reviewed-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260829054856.70549-2-kmehltretter@gmail.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
pkvm_init_features_from_host() takes KVM_ARCH_FLAG_GUEST_HAS_SVE and
KVM_ARM_VCPU_SVE from the host separately, but pkvm_vcpu_init_sve()
tests the bit while vcpu_has_sve() reads the flag. A host that sets the
flag without the bit gets a vCPU with a NULL sve_state that the world
switch loads the guest's SVE state from.
Derive the flag from the bit, and drop the protected path's copy of the
host's flag, which is dead code since protected VMs are not allowed SVE.
Fixes: 41d6028e28bd ("KVM: arm64: Convert the SVE guest vcpu flag to a vm flag")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260825085948.1674721-5-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
unpin_host_sve_state() gates on the VM's SVE feature bit, but what it
unpins is the state pkvm_vcpu_init_sve() pinned. A vCPU that completed
init has sve_state set exactly when that bit is set, so the two agree.
Gate on sve_state, which is what is being unpinned.
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260825085948.1674721-4-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
pkvm_vcpu_init_sve() clears KVM_ARM_VCPU_SVE in kvm->arch.vcpu_features
when it fails, but vcpu_has_sve() tests KVM_ARCH_FLAG_GUEST_HAS_SVE,
which is left set. Later vCPUs on that VM then skip the SVE setup and
register with a NULL sve_state, which the guest's first FP access hands
to sve_load_state().
Return the error without touching vcpu_features.
Fixes: 5db1bef93342 ("KVM: arm64: Track SVE state in the hypervisor vcpu structure")
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260825085948.1674721-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
pkvm_vcpu_init_sve() clamps only the upper bound of the host-provided
sve_max_vl, so an invalid vector length reaches sve_state_size_from_vl()
and the WARN_ON() there, which is fatal at EL2. The existing
!sve_state_size test rejects such a length, but only after the macro has
run.
Check sve_vl_valid() before deriving the state size. A valid length
cannot yield a zero size, so the !sve_state_size test goes with it.
Fixes: 5db1bef93342 ("KVM: arm64: Track SVE state in the hypervisor vcpu structure")
Reported-by: Stefan Teodorescu <fane@google.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260825085948.1674721-2-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
Commit 7270cc9157f4 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU
notifiers") introduced VNCR_EL2 invalidation in both kvm_nested_s2_unmap()
and kvm_nested_s2_wp().
However at the point of this being performed concurrent stage 2 teardown of
a nested guest can cause kvm->arch.mmu.pgt to be set to NULL.
This happens in kvm_flush_shadow_all() -> kvm_arch_flush_shadow_all() ->
kvm_free_stage2_pgd() and is performed under the kvm->mmu_lock.
Commit ec14c272408a ("KVM: arm64: nv: Unmap/flush shadow stage 2 page
tables") introduced the teardown of the entire nested MMU range, which then
invokes stage2_apply_range() with resched=true:
mmu_notifier_invalidate_range_start()
-> ... -> kvm_mmu_notifier_invalidate_range_start()
-> kvm_mmu_unmap_gfn_range()
-> kvm_unmap_gfn_range()
-> kvm_nested_s2_unmap()
-> kvm_stage2_unmap_range()
-> __unmap_stage2_range()
-> stage2_apply_range()
This means that stage2_apply_range() can drop the kvm->mmu_lock and thus
concurrent progress can be made in lockstep with
kvm_arch_flush_shadow_all().
If kvm_arch_flush_shadow_all() advances ahead of stage2_apply_range() and
completes its operation it guarantees a NULL pointer deref.
Since kvm_free_stage2_pgd() is performed under the kvm->mmu_lock this will
either be observed NULL or not and serialised against
kvm_free_stage2_pgd().
Resolve the issue by abstracting the invalidation to a new function,
kvm_invalidate_vncr_ipa_all(), and check that the pgt is non-NULL before
dereferencing it.
Fixes: 7270cc9157f4 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers")
Cc: stable@vger.kernel.org
Reviewed-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Tested-by: Jonathan Davies <jonathan.davies@nutanix.com>
Link: https://patch.msgid.link/20260901-kvm-arm-nested-virt-fix-v3-2-b154676f7e4c@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
kvmtool was used to establish an L1 guest with 8 CPUs and 8 GiB of RAM, an
L2 guest with 4 CPUs and 4 GiB of RAM and an L3 guest with 2 CPUs and 2 GiB
of RAM, all of which was then exited.
Under memory pressure in the L0 host warnings were observed due to
migration triggered by compaction:
WARNING: arch/arm64/kvm/mmu.c:336 at __unmap_stage2_range+0x64/0x80,
CPU#5: kcompactd0/66
Which was, in turn, triggered by an MMU notifier for the host invalidation:
mmu_notifier_invalidate_range_start()
-> ... -> kvm_mmu_notifier_invalidate_range_start()
-> kvm_mmu_unmap_gfn_range()
-> kvm_unmap_gfn_range()
-> kvm_nested_s2_unmap()
-> kvm_stage2_unmap_range()
-> __unmap_stage2_range()
-> stage2_apply_range()
<- -EINVAL, triggering a WARN_ON()
Racing with L0's teardown of stage 2 page tables:
exit_mm()
-> mmput()
-> __mmput()
-> exit_mmap()
-> mmu_notifier_release()
-> ... -> kvm_mmu_notifier_release()
-> kvm_flush_shadow_all()
-> kvm_arch_flush_shadow_all()
-> kvm_free_stage2_pgd()
-> [ acquire kvm->mmu_lock for write ]
-> mmu->pgt = NULL [ among other tasks ]
-> [ release kvm->mmu_lock for write ]
It turns out there is a benign race resulting in a spurious warning:
Thread A - notify: migration | Thread B - notify: release
-------------------------------|---------------------------------
< kvm->mmu_lock held > |
stage2_apply_range() |
get mmu->pgt, check !NULL |
... | kvm_arch_flush_shadow_all()
cond_resched_rwlock_write(); | < contend, sleep kvm->mmu_lock >
< drop kvm->mmu_lock > | < acquire kvm->mmu_lock>
| ...
| kvm_free_stage2_pgd()
| mmu->pgt = NULL
| < invalidate MMU >
| ...
| < release kvm->mmu_lock >
[ scheduled ] |
stage2_apply_range() |
< loop to next > |
get, mmu->pgt, check !NULL |
is NULL, return -EINVAL |
__unmap_stage2_range() |
WARN_ON(-EINVAL) <--- entirely spurious - the race was handled
correctly.
Fix the spurious warning by updating stage2_apply_range() to no longer
treat concurrent PGT teardown on lock release as an error - whether the
walker is tearing down page tables or doing something else this is a
legitimate reason to abort the operation without error.
This keeps the warning in place for all other circumstances.
In practice only __unmap_stage2_range() actually does anything with the
error so this only impacts that.
Fixes: ec14c272408a ("KVM: arm64: nv: Unmap/flush shadow stage 2 page tables")
Cc: stable@vger.kernel.org
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Link: https://patch.msgid.link/20260901-kvm-arm-nested-virt-fix-v3-1-b154676f7e4c@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
vgic_its_save_device_tables() aborts with -EINVAL when a device's entry
falls outside the device table, which a guest can arrange on its own: an
indirect table lets it clear an L1 entry's valid bit without touching
GITS_BASER. That fails a save userspace should be able to issue
reliably.
Skip the device instead, and point the saved DTE chain past it, as
commit ad1e686e2378d ("KVM: arm64: vgic-its: Point saved ITEs at the
next valid entry") does for ITEs. compute_next_devid_offset() takes the
next device off the list whether or not it was saved, so the predecessor
would otherwise point at an entry the save never wrote. Restore follows
that offset while it stays inside the table being scanned: within an L2
block, or anywhere in a flat table. Both need userspace to remove a
memslot under the table, since dropping an L1 entry takes the whole
block with it and scan_its_table() stops at the block boundary.
Fixes: 57a9a117154c9 ("KVM: arm64: vgic-its: Device table save/restore")
Suggested-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/all/86bjaz5s6v.wl-maz@kernel.org/
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260821064445.615838-4-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
This reverts commit 9b10fb74e4b661543d188701bd4d024fc5c18f58.
Freeing the collections when GITS_BASER<coll> changes removes the state
this check rejected: vgic_its_cmd_handle_mapi(),
vgic_its_cmd_handle_mapc() and vgic_its_restore_cte() all validate the
ID against the current table before allocating, and the table can no
longer change under the list. What remains is a collection whose entry
is not backed by a memslot, which the write fails on anyway, so the
check costs a save userspace should be able to issue reliably and buys
nothing.
The reverted commit credited the check with bounding the walk as well.
It stays bounded without it: collection IDs are unique and each is below
the table's capacity, so the list cannot be longer than the table.
Suggested-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/all/87ecg9owwa.wl-maz@kernel.org/
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260821064445.615838-3-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
A guest that disables the ITS and re-points or shrinks GITS_BASER<n>
with VALID still set keeps the devices and collections it mapped
against the old table, as KVM frees them only when VALID is cleared.
The contents of the table are IMPLEMENTATION DEFINED, so a write that
gives GITS_BASER<n> a different address or size may lose whatever the
old value described. Free the list whenever the stored value changes,
and drop the translation cache with it.
The cache is not empty just because the ITS is disabled: its->enabled
is written under the cmd_lock, while vgic_its_resolve_lpi() tests it
under the its_lock, so an injection can still cache an entry after the
ITS was disabled. Hence the invalidation inside the its_lock section.
Test for a change rather than a write: its_restore_enable() rewrites
GITS_BASER<n> from its probe-time cache on resume, and KVM reports
GITS_TYPER.HCC as 0, so nothing re-maps the boot CPU's collection
afterwards.
Fixes: 36d6961c2b481 ("KVM: arm/arm64: vgic-its: Free caches when GITS_BASER Valid bit is cleared")
Suggested-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/all/87ecg9owwa.wl-maz@kernel.org/
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260821064445.615838-2-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
Use kvm_test_request() instead of kvm_check_request() when querying
KVM_REQ_VM_DEAD, i.e. don't clear KVM_REQ_VM_DEAD, as the entire purpose
of KVM_REQ_VM_DEAD is to prevent the vCPU from enterring the guest ever
again, even if userspace insists on redoing KVM_RUN.
Ensuring KVM_REQ_VM_DEAD is never cleared will allow relaxing KVM's rule
that ioctls can't be invoked on dead VMs, to only disallow ioctls if the
VM is bugged, i.e. if KVM hit a KVM_BUG_ON().
Opportunistically add compile-time assertions to guard against clearing
KVM_REQ_VM_DEAD through the standard APIs.
Reviewed-by: Kai Huang <kai.huang@intel.com>
Acked-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260806214618.82180-1-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
|
|
Pull arm64 fixes from Will Deacon:
"A mixture of ptdump, compat and MTE fixes that came in during the
merge window:
- Fix address handling of final memory region in ptdump
- Fix emulation of decrementing load/store multiple from 32-bit task
- Fix SCTLR context-switching for store-only MTE mode
- Fix numerous issues in MTE selftests"
* tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux:
selftests/arm64: Add MTE test config fragment
selftests/arm64: Fix MTE prctl TAP plan
selftests/arm64: Treat KSM merge_across_nodes as optional
selftests/arm64: Print missing MTE TAP headers
arm64: compat: Fix decrementing LDM/STM alignment emulation
arm64: process: Fix context switching MTE store-only tag check
KVM: arm64: ptdump: Flush the last region
arm64: ptdump: Make note_page_flush() range aware
|
|
KVM/arm64 changes for 7.3
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1
is present, guaranteeing that EL2-specific register bits are stateful
in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope
of those instructions remains 'in host' (i.e. L1 kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly in
the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
|
|
Currently the stage-2 ptdump calls note_page() at each leaf entry visit.
This simply misses the output of the last region, because note_page()
only dumps output when it detects a change in level/prot, or when the
walk enters a next marker section. The last region in the guest IPA
space with the same level/prot is not dumped since there is no change
after it.
Call note_page_flush() to dump the final region. note_page_flush()
uses ptdump_pg_state.end_address to call the final note_page(), so also
provide the end address.
Also change the second marker's start address to ULONG_MAX so we never
cross it. This avoids dumping redundant marker names (which are NULL),
and advancing beyond the end of the marker array.
Fixes: 7c4f73548ed1 ("KVM: arm64: Register ptdump with debugfs on guest creation")
Reported-by: Sashiko AI <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/kvmarm/20260630122758.891011F00A3A@smtp.kernel.org/
Reviewed-by: Dev Jain <dev.jain@arm.com>
Tested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
|
|
* kvm-arm64/misc-7.3:
: Miscellaneous fixes for KVM/arm64, 7.3
:
: - Fixes for saving invalid table entries as part of saving the ITS
: tables (Fuad Tabba)
:
: - Don't reallocate the SPI array for re-attempted vgic_init(), avoiding
: a memory leak (Fuad Tabba)
:
: - Hold a reference on an LPI when saving the pending state (Qihang)
:
: - Don't WARN for out-of-range, guest-supplied INTID (Karl)
:
: - Avoid corrupting GPRs for 32-bit CP64 reads (Karl)
:
: - Reset 'in kernel' VGIC state when private IRQ allocation fails (Fuad)
:
: - Avoid kallsyms lookup in nVHE panic unless the host stage-2 is also
: disabled (Vincent)
:
: - Disregard Pending+Active state when computing maintenance IRQ for
: ICH_MISR_EL2.NP (Kajetan)
:
: - Various Sashiko-identified issues dealing with GICv5 (Sascha)
:
: - Fix CPU onlining in pKVM due to mismatched accesses when the MMU is
: disabled (Will)
KVM: arm64: Validate GICv5 timer PPIs before claiming ownership
KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs
KVM: arm64: vgic: Prevent speculative SPI array underflow
KVM: arm64: vgic: Free gic_kvm_info on initialization failure
KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'
KVM: arm64: vgic: Fix detection of MI on no pending LR
KVM: arm64: Drop %pB on nVHE panic when stage-2 is active
KVM: arm64: vgic: Reset in_kernel on private IRQ allocation failure
KVM: arm64: GICv2: Don't WARN on out-of-range GICV_DIR INTID
KVM: arm64: Preserve GPRs for AArch32 CP64 reads generating an UNDEF
KVM: arm64: vgic-v3: take an LPI reference in vgic_v3_save_pending_tables
KVM: arm64: vgic-its: Point saved ITEs at the next valid entry
KVM: arm64: vgic-its: Don't save collections the table cannot hold
KVM: arm64: vgic: Don't leak the SPI array when init is retried
KVM: arm64: vgic-its: Don't dereference a NULL collection on ITT save
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
* kvm-arm64/vtr-patch:
: Inline patching of ICH_VTR_EL2 constant, courtesy of Marc Zyngier
:
: Unify readers of ICH_VTR_EL2 on an instruction-patched constant value,
: avoiding system register accesses known to trap under nested
: virtualization and sharing the implementation between pKVM and 'regular'
: KVM.
KVM: arm64: vgic-v3: Kill kvm_vgic_global_state.ich_vtr_el2
KVM: arm64: vgic-v3: Simplify initial GICv3 configuration sampling
KVM: arm64: Convert most ICH_VTR_EL2 accesses to inlined literal value
KVM: arm64: Add a helper providing an inlined literal value for ICH_VTR_EL2
KVM: arm64: Move GICv3 broken SEIS implementation detection to a CPU errrata
KVM: arm64: vgic-v3: Make vtr_to_* helpers use architectural field symbols
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
* kvm-arm64/vncr-fixes:
: Fixes for handling L1 VNCR mappings, courtesy of Marc Zyngier
:
: - Recompute the VNCR software TLB when the MMU is toggled at stage-1
:
: - Improve TLB invalidation intersection to handle TLBIs affecting the
: end of the VA space
:
: - Fix race to invalidate the VNCR fixmap between TLBI emulation and
: vcpu_put()
:
: - Add missing sign extension for computing TLBI ranges
:
: - Make VNCR invalidation participate in the MMU notifier seqcount,
: preventing a concurrent VNCR TLB fill from consuming a stale
: translation
KVM: arm64: Correctly cap TLBI Range to the architural limit
KVM: arm64: Add VNCR TLB tracking again
KVM: arm64: Make VNCR invalidation participate in MMU invalidation retry
KVM: arm64: Sign-extend VA for range-based TLBI invalidation
KVM: arm64: Handle VNCR TLB invalidation race with vcpu_put() VNCR unmapping
KVM: arm64: Correctly handle end of VA space TLBI invalidation
KVM: arm64: Consider SCTLR_EL2.M when mapping the L1 VNCR page
KVM: arm64: Handle negative S1 walk levels in VNCR TLB size evaluation
KVM: arm64: Remove VM-wide VNCR mapping counter
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
* kvm-arm64/feat-nv3: (29 commits)
: Support for FEAT_NV2p1 and FEAT_NV3, courtesy of Marc Zyngier
:
: Couple of accelerations for nested virtualization:
:
: - Elide traps for EL1 alias registers when hardware guarantees that
: EL2-only bits are stateful (NV2p1)
:
: - Use TGE-aware hardware to avoid unnecesary ERET/TLBI traps that
: remain 'in-context' (i.e. L1 kernel/userspace)
arm64: Add override for ID_AA64MMFR4_EL1.NV_frac
KVM: arm64: selftest: Add NVHCR_EL2 to get-reg-list
KVM: arm64: Expose FEAT_NV3 to guests
KVM: arm64: Add FEAT_NV3 detection
KVM: arm64: Engage NV3 TLBI trap elision
KVM: arm64: Engage NV3 ERET trap elision
KVM: arm64: Add NVHCR_EL2 context switching
KVM: arm64: Add routing for NVHCR_EL2 trap
KVM: arm64: Add NVHCR_EL2 handling to the sysreg array
KVM: arm64: Add sanitisation for NVHCR_EL2
KVM: arm64: Make HCR_EL2 a non-VNCR register
KVM: arm64: Add kvm_has_nv{2,3}() predicates
KVM: arm64: Add NV3 control bits to HCRX_EL2 sanitisation
KVM: arm64: Split NV-specific exit fixups from the non-NV handling
arm64: Add ARM64_HAS_NV3 capability
arm64: sysreg: Add HCRX_EL2 bits related to FEAT_NV3
arm64: sysreg: Add NVHCR_EL2 description as a mirror of HCR_EL2
arm64: Add FEAT_NV2p1 detection
KVM: arm64: Expose FEAT_NV2p1 to NV guests
KVM: arm64: Relax CNTHCTL_EL2 handling when FEAT_NV2p1 is present
...
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
* kvm-arm64/pkvm-7.3: (26 commits)
: pKVM updates for 7.3
:
: - Avoid name collision on trace_clock() when CONFIG_NVHE_EL2_TRACING is
: disabled (Mostafa Saleh)
:
: - Clean up state tracking for whether the EL2 shadow VM has been
: created (Fuad Tabba)
:
: - Synchronize SCTLR_EL1 when injecting an exception to use current
: PAN/SSBS state (Fuad Tabba)
:
: - Avoid unnecessary cache maintenance when I/D-cache are known to be
: coherent in pKVM (Mostafa Saleh)
:
: - Lazy vCPU context save/restore for pKVM (Fuad Tabba)
:
: - Various fixes to the stage-2 MMU for pKVM (Fuad Tabba)
:
: - Allow counter offsetting of non-protected guests in protected mode
: (Mostafa Saleh)
:
: - Condition the 'broken CNTVOFF' mitigation on a VM actually having a
: nonzero offset, fixing boot failures of pVMs on affected hardware
: (Mostafa Saleh)
KVM: arm64: Fix hvhe and broken CNTVOFF_EL2
KVM: arm64: Fix timer offsets for non-protected VMs
KVM: arm64: Make timer_get_offset() work in all contexts
KVM: arm64: selftests: Add stage-2 block transition test
KVM: arm64: Don't advertise eager page splitting under pKVM
KVM: arm64: Don't WARN on pKVM stage-2 map failures
KVM: arm64: Skip pKVM stage-2 flush when FWB is enabled
KVM: arm64: Top up stage-2 memcache for dirty logging faults
KVM: arm64: Top up the memcache for pKVM permission faults
KVM: arm64: Skip cache maintenance for non-cacheable pKVM mappings
KVM: arm64: Implement lazy vCPU state sync for non-protected guests
KVM: arm64: Add primitives to flush/sync the VGIC state at EL2
KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch
KVM: arm64: Add host and hypervisor vCPU lookup primitives
KVM: arm64: Move PSCI helper functions to a shared header
KVM: arm64: Factor out reusable vCPU reset helpers
KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code
KVM: arm64: Extract MPIDR computation into a shared header
KVM: arm64: selftests: Add a userspace watchpoint test
KVM: arm64: Flush external_mdscr_el1 to the pKVM hyp vCPU
...
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
* kvm-arm64/pmu-7.3:
: vPMU updates for 7.3
:
: - Support for slot-based PMU events, relying on new UAPI that makes
: selection of a vPMU implementation mandatory (Congkai Tan)
KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature
KVM: arm64: Ignore writes to PMCR_EL0.N when using strict UAPI
KVM: arm64: Advertise STALL_SLOT* in PMCEID1 under strict PMUv3 UAPI
KVM: arm64: Expose PMMIR_EL1.SLOTS under strict PMUv3 UAPI
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
With GICv5, timer PPIs use architecturally defined IDs, and therefore
we require userspace to use the architected IDs too. We perform this
validation in timer_irqs_are_valid(), which validates the configured
PPI after claiming it with kvm_vgic_set_owner(). This is obviously the
wrong order, and could both result in a NULL pointer dereference for
an ID >= 64 and in an incorrect ownership claim.
Switch the order to first validate the PPI before setting the
owner. This both prevents an invalid GICv5 timer PPI from reaching the
IRQ lookup, and avoids claiming an IRQ for a timer configuration that
will be rejected.
Fixes: 06c85b58e0b1 ("KVM: arm64: Move GICv5 timer PPI validation into timer_irqs_are_valid()")
Link: https://sashiko.dev/#/message/20260810111614.DDC731F000E9%40smtp.kernel.org
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260811150941.941295-5-sascha.bischoff@arm.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
GICv5 supports up to 128 PPIs, but KVM currently implements only the
first 64, which contain the architected PPIs it supports.
An encoded PPI with an ID outside that range passes irq_is_ppi(),
which only checks the encoded interrupt type. vgic_get_vcpu_irq()
therefore looks it up in private_irqs[], where array_index_nospec()
clamps the out-of-range index to zero and aliases PPI 0.
Include the supported PPI range in irq_is_ppi() so that KVM interfaces
reject unsupported PPIs. Also reject an out-of-range PPI in the lookup
as a safeguard against callers bypassing the predicate.
Fixes: 4d591252bacb ("KVM: arm64: gic-v5: Implement PPI interrupt injection")
Fixes: eb8bce08ecb1 ("KVM: arm64: gic: Introduce interrupt type helpers")
Link: https://sashiko.dev/#/patchset/20260724104819.1296803-1-sascha.bischoff@arm.com?part=27
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260811150941.941295-4-sascha.bischoff@arm.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
For a non-GICv5 VM, SPI interrupt IDs include the private-interrupt
offset, while KVM's SPI array is indexed from zero. The lookup applies
array_index_nospec() to the absolute interrupt ID and subtracts the
private-interrupt offset afterwards.
On a speculative bypass of the range check for an interrupt ID below
the private range, the clamp preserves the small absolute value and
the subtraction underflows to an out-of-bounds SPI array index.
Convert the interrupt ID to a zero-based index into the SPI array
before applying array_index_nospec(). This way, we ensure that we
clamp to a reachable SPI ID, rather than an out-of-range SPI index.
Fixes: 41b87599c743 ("KVM: arm/arm64: vgic: fix possible spectre-v1 in vgic_get_irq()")
Link: https://sashiko.dev/#/patchset/20260724104819.1296803-1-sascha.bischoff@arm.com?part=27
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260811150941.941295-3-sascha.bischoff@arm.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
vgic_set_kvm_info() allocates gic_kvm_info for use by
kvm_vgic_hyp_init(). When a maintenance interrupt is mandatory but not
provided, kvm_vgic_hyp_init() returns -ENXIO before reaching the
common cleanup path, leaking said allocation.
Route this error through that cleanup path so that gic_kvm_info is
freed and the global pointer is cleared.
Fixes: 0e5cb7770684 ("irqchip/gic: Split vGIC probing information from the GIC code")
Link: https://sashiko.dev/#/patchset/20260724104819.1296803-1-sascha.bischoff@arm.com?part=1
Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260811150941.941295-2-sascha.bischoff@arm.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
When running with hVHE enabled, ___kvm_hyp_init() calls
__kvm_init_el2_state() on the CPU initialisation path during onlining
and resume from suspend.
In order to avoid clobbering the link register across this call, it is
stashed away in the 'tmp' member of 'struct kvm_nvhe_init_params',
however this save/restore operation is performed with the stage-1 MMU
disabled at EL2 and therefore gives rise to coherency problems because
the field is not aligned or padded to the CWG. For example, a cacheable
write to a physically-adjacent structure sharing the same cacheline
could lead to an eviction and subsequent write-back, overwriting the
saved LR while the incoming CPU is executing __kvm_init_el2_state().
Save the lr in far_el2 and remove the 'tmp' member from
'struct kvm_nvhe_init_params' altogether.
Cc: Oliver Upton <oupton@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Fixes: afa9b48f327c ("KVM: arm64: Shave a few bytes from the EL2 idmap code")
Signed-off-by: Will Deacon <will@kernel.org>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260813131717.5885-1-will@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
As per the ARM GICv3 spec, the maintenance interrupt identified by
ICH_MISR_EL2.NP is asserted when it is enabled and no List register is
in pending state. This is further described in the document as "no List
registers with the State field set to 0b01 (pending)". By checking only
the pending bit of the LR (bit 62), KVM currently asserts the MI when
there are no LRs in "pending" or "pending and active" states.
Fix the detection logic to consider only the "pending" state.
Cc: stable@vger.kernel.org
Fixes: 96c2f03311de ("KVM: arm64: nv: Plumb handling of GICv3 EL2 accesses")
Signed-off-by: Kajetan Puchalski <kajetan.puchalski@arm.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260810102923.2426475-2-kajetan.puchalski@arm.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
With pKVM, the host stage-2 is lazily mapped. kallsyms, accessed via
the modifier %pB is therefore potentially unmapped. If the hyp panic
occurred while the host stage-2 lock was held, handling this fault
deadlocks.
Skip %pB formatting unless the host stage-2 is disabled.
Fixes: 6ccf9cb557bd ("KVM: arm64: Symbolize the nVHE HYP addresses")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260803093906.3531699-1-vdonnefort@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
kvm_vgic_create() sets vgic.in_kernel before allocating the per-vCPU
private IRQs, but the allocation-failure path resets only vgic_model and
leaves in_kernel set. As irqchip_in_kernel() is !!in_kernel, the VM is
left with an in-kernel irqchip but no model, and the -EEXIST guard at the
top of kvm_vgic_create() rejects every retry, so userspace cannot recover
from a transient -ENOMEM.
Reset in_kernel alongside vgic_model on the failure path.
Fixes: 9435c1e1431003 ("KVM: arm64: gic: Set vgic_model before initing private IRQs")
Cc: stable@vger.kernel.org
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Acked-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260802150845.3485757-1-fuad.tabba@linux.dev
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
vgic_v2_deactivate() passes the INTID a guest wrote to GICV_DIR straight
to vgic_get_vcpu_irq(), and treats a failed lookup as a "can't happen"
condition with WARN_ON_ONCE().
The guest can make it happen at will, though: for any INTID outside of
the implemented SGI, PPI and SPI ranges the lookup returns NULL, since
GICv2 has no LPIs. A guest running with EOImode==1 writing such an INTID
to GICV_DIR triggers the WARN, and panics hosts running with
panic_on_warn.
Drop the WARN and ignore failed lookups.
Fixes: 255de897e7fb ("KVM: arm64: GICv2: Handle deactivation via GICV_DIR traps")
Cc: stable@vger.kernel.org
Suggested-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Link: https://patch.msgid.link/20260726174803.5880-1-kmehltretter@gmail.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
kvm_handle_cp_64() only seeds params.regval for writes. If a CP64 read is
decoded but UNDEFs, emulate_cp() still returns handled and the caller
writes params.regval back to Rt/Rt2.
This can happen for PMU counter read accesses generating an UNDEF. KVM
injects the exception into the guest, so the MRRC GPRs must remain
unchanged.
Instead, the uninitialised regval is copied into the guest GPRs. With stack
auto-initialisation this is a deterministic zero or pattern value. With
CONFIG_INIT_STACK_NONE it may be stale host stack data.
Match kvm_handle_cp_32() and kvm_handle_sys_reg() by seeding regval from
the GPRs before emulation.
Fixes: 62a89c44954f0 ("arm64: KVM: 32bit handling of coprocessor traps")
Assisted-by: Claude:claude-opus-5
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260802182222.2239-1-kmehltretter@gmail.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
When running on a setup affected with broken CNTVOFF_EL2
(has_broken_cntvoff())
Booting with VHE or protected mode(nvhe) (id_aa64mmfr1.vh=0
and arm64_sw.hvhe=0) works fine.
However launching a protected VM with protected hvhe mode panics the
guest kernel:
[ 0.000000] Internal error: Oops - Undefined instruction: 0000000000000000 [#1] SMP
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.2.0-rc3-g05f75bd71e0e-dirty #29 PREEMPT
[ 0.000000] Hardware name: linux,dummy-virt (DT)
[ 0.000000] pstate: 000003c5 (nzcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 0.000000] pc : arch_timer_shutdown_virt+0x4/0x1c
[ 0.000000] lr : arch_timer_starting_cpu+0x1c4/0x2d4
[ 0.000000] sp : ffffa6bd9a193c00
[ 0.000000] x29: ffffa6bd9a193c20 x28: ffffa6bd9a1bcf88 x27: 0000000000000000
[ 0.000000] x26: ffff00001be70dd8 x25: ffffa6bd99d85000 x24: ffffa6bd99d85ee4
[ 0.000000] x23: ffffa6bd99d85000 x22: ffffa6bd9a1499c0 x21: ffffa6bd9a1ab900
[ 0.000000] x20: 00ffffffffffffff x19: ffff00001be8b600 x18: 000000000000028c
[ 0.000000] x17: 00000000510f0010 x16: 00000000510f0010 x15: 00000000500f0000
[ 0.000000] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000018
[ 0.000000] x11: ffffa6bd9a8ac000 x10: 0000000000f0000f x9 : ffffffffffffffff
[ 0.000000] x8 : ffffa6bd98822e18 x7 : 0070752d65746174 x6 : 00111ff76e007261
[ 0.000000] x5 : ffffa6bd9ad68078 x4 : 0000000000000000 x3 : ffffa6bd98822a0c
[ 0.000000] x2 : 0000000000000073 x1 : 0000000000000001 x0 : ffff00001be8b600
[ 0.000000] Call trace:
[ 0.000000] arch_timer_shutdown_virt+0x4/0x1c (P)
[ 0.000000] cpuhp_invoke_callback+0x11c/0x280
[ 0.000000] cpuhp_issue_call+0x1e8/0x224
[ 0.000000] __cpuhp_setup_state_cpuslocked+0x1d8/0x2b8
[ 0.000000] __cpuhp_setup_state+0x50/0x74
[ 0.000000] arch_timer_register+0xc0/0x148
[ 0.000000] arch_timer_of_init+0x148/0x170
[ 0.000000] timer_probe+0x74/0x124
[ 0.000000] time_init+0x18/0x58
[ 0.000000] start_kernel+0x1c0/0x3ac
[ 0.000000] __primary_switched+0x88/0x90
[ 0.000000] Code: c80b7d2a 35ffffab 17ffffeb d503245f (d53be328)
The workaround avoids setting non-zero CNTVOFF_EL2 and trapping the
virtual counter to emulate the offset.
In the VHE path (timer_set_traps()), traps are only enabled when the
guest actually has a non-zero virtual timer offset.
However, __timer_enable_traps() in hyp/nvhe/timer-sr.c unconditionally
set CNTHCTL_EL1TVT and CNTHCTL_EL1TVCT whenever has_broken_cntvoff()
was true.
Which causes 2 issues:
1) Protected VMs: kvm_handle_pvm_sysreg() does not find "cntv_ctl_el0"
in pvm_sys_reg_descs and injects undefined instruction exceptions.
2) non-protected guests are trapped all the time even with offset of
zero.
Fix this by adding a check in __timer_enable_traps() similar to the one in
timer_set_traps()
Fixes: 0bc9a9e85fcf ("KVM: arm64: Work around x1e's CNTVOFF_EL2 bogosity")
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Link: https://patch.msgid.link/20260808085824.732659-4-smostafa@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
With pKVM, protected VMs always have offset of zero.
However, timer offsets for non-protected guests fail to take effect
for two reasons:
1) In __timer_enable_traps(), enabling of traps check for
is_protected_kvm_enabled() rather than vcpu_is_protected(vcpu)
2) The vcpu timer offsets were never initialised and kept as NULL.
This is problematic for cases when the timer is trapped in the
hypervisor as the with the case of broken CNTVOFF_EL2, which leads
to the hypervisor and host using different offsets and causing VM
hangs.
This can be confirmed by running the arch_timer selftest which fails:
./arch_timer -o 100000000
Random seed: 0x6b8b4567
Guest assert failed, vcpu 0; stage; 3; iter: 0
==== Test Assertion Failure ====
arm64/arch_timer.c:137: config_iter + 1 == irq_iter
pid=310 tid=312 errno=4 - Interrupted system call
Guest assert failed, vcpu 3; stage; 3; iter: 0
Guest assert failed, vcpu 1; stage; 3; iter: 0
==== Test Assertion Failure ====
arm64/arch_timer.c:137: config_iter + 1 == irq_iter
pid=310 tid=313 errno=4 - Interrupted system call
Guest assert failed, vcpu 2; stage; 3; iter: 0
==== Test Assertion Failure ====
arm64/arch_timer.c:137: config_iter + 1 == irq_iter
pid=310 tid=314 errno=4 - Interrupted system call
[...]
After the fix:
./arch_timer -o 100000000
Random seed: 0x6b8b4567
PASS(vCPU-1).
PASS(vCPU-3).
PASS(vCPU-0).
PASS(vCPU-2)
Reported-by: Sashiko <sashiko-bot@kernel.org>
Fixes: cb0c272acebd ("KVM: arm64: Initialize the hypervisor's VM state at EL2")
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Link: https://patch.msgid.link/20260808085824.732659-3-smostafa@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
We currently have two implementations of get_timer offset(), one
in arm_arch_timer.h, and another one in switch.h.
These two only differ by a pair of kern_hyp_va(), which seems a
pretty weak reason to open-code it.
Turn this function into a macro to avoid the include dependency hell
on kern_hyp_va(), and make it work correctly in all contexts.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Link: https://patch.msgid.link/20260808085824.732659-2-smostafa@google.com
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
Having established that our VNCR TLB tracking was flawed and having
dropped it from KVM, it is time to replace it with something that
actually works in order to get some performance back, as the TLBI
overhead is otherwise pretty high.
The goal of such tracking is to hit the TLBI slow path if there are
any VNCR TLBs in the guest, irrespective of their mapping state.
For this purpose, we introduce an VM wide counter (vncr_tlb_count)
that tracks how many valid VNCR TLB are present. This means that
creating such TLB must increment the counter, and invalidation
decrement it, and both these operations must be done with the MMU lock
held for write.
On TLBI handling affecting EL2 S1, a non-zero counter forces the
handling to take the slow path to consider the VNCR TLBs. Note that
the bumping up is done "speculatively" in order to make sure that a
concurrent invalidation will be taken on the slow path.
Not exactly rocket science. Hopefully I got it right this time.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260806091026.620700-9-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
A VNCR TLB invalidation can occur on one vcpu while another vcpu is
faulting in this same page. Without correctly handling this, we can
end up with the following scenario:
- vcpu A walks the PTs to translate VNCR
- before vcpu A is able to grab the MMU lock to insert the TLB,
vcpu B updates the S1 PTs with an invalid entry, and issues
a TLBI S1E2 for this VA
- vcpu A inserts the TLB for something that is now invalid
This isn't a new problem, and we manage S2 by having the MMU notifier
to bump up mmu_invalidate_seq on invalidation so that the fault can be
replayed.
We can perform something similar here, and extend invalidate_vncr_va() to
update the same counter, clearly indicating that the context has
changed under our feet. This is safe as the invalidation always happen
while holding the MMU lock for write, and that we sample the sequence
number before walking S1.
Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2")
Reported-by: sashiko-bot@kernel.org
Link: https://lore.kernel.org/r/20260801130454.5D9F11F00AC4@smtp.kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260806091026.620700-8-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
When the decode_range_tlbi() helper was moved to be used for S1 TLBIs,
the required sign extension was omitted. Add it.
As a result, special care must be taken to not overflow PA bits when
this is used for S2 invalidation.
Fixes: 85bba00425ae0 ("KVM: arm64: nv: Move TLBI range decoding to a helper")
Reported-by: sashiko-bot@kernel.org
Link: https://lore.kernel.org/r/20260801130337.EB2BA1F00AC4@smtp.kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260806091026.620700-7-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
While VNCR TLB invalidation always occurs under the MMU lock,
vcpu_put() doesn't, while it unmaps the VNCR page.
The problem is that the invalidation evaluates vncr_tlb::cpu to
decide whether an unmapping needs to take place (cpu != -1) before
performing it. On the other hand, this_cpu_reset_vncr_fixmap()
unconditionally unmaps if L1_VNCR_MAPPED is set.
These two obviously can race, with a TOCTOU pattern on the TLBI
path, and a BUG_ON() on the vcpu_put() path. And the two can end-up
calling vncr_fixmap(-1), with extra lethal effects.
Move the reset of vncr_tlb::cpu to -1 to a common function, and make
this update atomic so that only a single thread can reset the field
and perform the corresponding unmap. The vcpu_put() still need to
unconditionally unmap the current VNCR to close another ugly race.
Finally, the assignment of vncr_tlb::cpu is moved to be kept in sync
with the actual mapping, similar to L1_VNCR_MAPPED being set.
Fixes: 7270cc9157f47 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers")
Reported-by: sashiko-bot@kernel.org
Link: https://lore.kernel.org/r/20260801130237.0FD8F1F00ACA@smtp.kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Link: https://patch.msgid.link/20260806091026.620700-6-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
Our TLB invalidation by VA code is based on comparing two ranges,
one defined by the TLB, and one defined by the TLBI instruction.
Each range is defined by a start and a size. However, the way the
comparison is done doesn't account for address rollover, as it
compares an address with (base + size). This works nicely until
this expression represent the last page/block in the TTBR1 VA space,
as the result is a big fat 0. And a failed TLB invalidation.
Rewrite the comparison in a way that is immune to the address
rollover (making the end address inclusive instead of exclusive),
and move this into a common helper that is used by both VA and IPA
invalidations, as suggested by Hyunwoo Kim (although the IPA version
didn't suffer from this particular problem, obviously).
Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2")
Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260806091026.620700-5-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
We record a VNCR TLB even when SCTLR_EL2.M is 0 in order to make
our life easier. But this is not something that the architecture
anticipate.
As a consequence, a hypervisor is free to set VNCR_EL2 to
some PA when SCTLR_EL2.M==0, use it to run a guest which indirectly
accesses the VNCR page, then eventually set SCTLR_EL2.M==1 with
the same VA. Yes, this is odd, but apparently legal.
A common trick in HW is to invalidate the TLBs on SCTLR_ELx.M being
flipped. But doing this is a not a good idea for us (we'd need to
trap SCTLR accesses), and wouldn't scale as we nest deeper.
Instead, use the fact that the S1 MMU being off at the point of
translation is cached in our TLB, and if it doesn't match the current
MMU state, leave the VNCR unmapped.
Fixes: 2a359e072596f ("KVM: arm64: nv: Handle mapping of VNCR_EL2 at EL2")
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260806091026.620700-4-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
|
Computing the effects of a TLB invalidation involves looking at
the size of the mapping cached by the TLB. For S1 mappings such as
VNCR, this is deducted from the combination of the base granule size
and the mapping level.
However, this implies that the S1 MMU is *on*. When the MMU is off,
we indicate this with the level being set to a "creative" value of
-127 (S1_MMU_DISABLED).
This ends-up being misinterpreted by pgshift_level_to_ttl() as it
doesn't handle negative levels at all (the level is immediately cast
to a u8 and only the bottom two bits considered), leading to an
invalidation size of 0. Not helpful.
Tidy-up pgshift_level_to_ttl() to handle these negative levels, and
ttl_to_size() to always return SZ_1G when no valid TTL is present.
This allows the removal of open-coded checks for similar situations.
Note that the check for a negative value not explicitely checking for
S1_MMU_DISABLED is deliberate, so that actual negative levels introduced
with LVA2 and D128 can take the same path if we ever support them.
Fixes: 7270cc9157f47 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers")
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Link: https://lore.kernel.org/r/ameGoxbn2wzBq2kL@v4bel
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260806091026.620700-3-maz@kernel.org
Signed-off-by: Oliver Upton <oupton@kernel.org>
|
Paolo Bonzini
Karl Mehltretter
Mark Brown
Marc Zyngier
Fuad Tabba
Lorenzo Stoakes (ARM)
Sean Christopherson
Kees Cook
Linus Torvalds
Wei-Lin Chang
Oliver Upton
Sascha Bischoff
Will Deacon
Kajetan Puchalski
Vincent Donnefort
Mostafa Saleh