ANDROID: KVM: arm64: Initialize pkvm_pgtable.mm_ops earlier
The `init` callback of an IOMMU driver is called just before
`finalize_host_mappings` so that EL2 mappings created by drivers are
subsequently unmapped from host stage-2. However, at this point hyp has
already switched to the buddy allocator, having reserved pages allocated
by the early allocator, but `pkvm_pgtable.mm_ops` have not been switched
to buddy allocator callbacks. As a result, pages allocated for EL2
mappings of the IOMMU driver are allocated by the obsoleted early
allocator and remain treated as free by the buddy allocator. This likely
leads to a corruption in the free page lists and a later hyp panic.
Move the initialization of `pkvm_pgtable.mm_ops` before
`finalize_host_mappings` and the call to IOMMU's `init`.
Test: run a VM
Test: adb shell cmd jobscheduler run -f android 5132250
Bug: 190463801
Bug: 209004831
Signed-off-by: David Brazdil <dbrazdil@google.com>
Change-Id: I42ad28e6b4cfb014d50dbbf722e4c02d5f684178
diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
index 5c33ffc..7ba941f 100644
--- a/arch/arm64/kvm/hyp/nvhe/setup.c
+++ b/arch/arm64/kvm/hyp/nvhe/setup.c
@@ -246,6 +246,15 @@
if (ret)
goto out;
+ pkvm_pgtable_mm_ops = (struct kvm_pgtable_mm_ops) {
+ .zalloc_page = hyp_zalloc_hyp_page,
+ .phys_to_virt = hyp_phys_to_virt,
+ .virt_to_phys = hyp_virt_to_phys,
+ .get_page = hpool_get_page,
+ .put_page = hpool_put_page,
+ };
+ pkvm_pgtable.mm_ops = &pkvm_pgtable_mm_ops;
+
if (kvm_iommu_ops.init) {
ret = kvm_iommu_ops.init();
if (ret)
@@ -256,15 +265,6 @@
if (ret)
goto out;
- pkvm_pgtable_mm_ops = (struct kvm_pgtable_mm_ops) {
- .zalloc_page = hyp_zalloc_hyp_page,
- .phys_to_virt = hyp_phys_to_virt,
- .virt_to_phys = hyp_virt_to_phys,
- .get_page = hpool_get_page,
- .put_page = hpool_put_page,
- };
- pkvm_pgtable.mm_ops = &pkvm_pgtable_mm_ops;
-
out:
/*
* We tail-called to here from handle___pkvm_init() and will not return,
