summaryrefslogtreecommitdiffstats
path: root/arch/x86/kvm/mmu.c
Commit message (Collapse)AuthorAgeFilesLines
* KVM: MMU: disable global page optimizationMarcelo Tosatti2009-04-221-1/+1
| | | | | | | | | Complexity to fix it not worthwhile the gains, as discussed in http://article.gmane.org/gmane.comp.emulators.kvm.devel/28649. Cc: stable@kernel.org Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: fix sparse warnings: Should it be static?Hannes Eder2009-03-241-5/+6
| | | | | | | | | | | | | | | Impact: Make symbols static. Fix this sparse warnings: arch/x86/kvm/mmu.c:992:5: warning: symbol 'mmu_pages_add' was not declared. Should it be static? arch/x86/kvm/mmu.c:1124:5: warning: symbol 'mmu_pages_next' was not declared. Should it be static? arch/x86/kvm/mmu.c:1144:6: warning: symbol 'mmu_pages_clear_parents' was not declared. Should it be static? arch/x86/kvm/x86.c:2037:5: warning: symbol 'kvm_read_guest_virt' was not declared. Should it be static? arch/x86/kvm/x86.c:2067:5: warning: symbol 'kvm_write_guest_virt' was not declared. Should it be static? virt/kvm/irq_comm.c:220:5: warning: symbol 'setup_routing_entry' was not declared. Should it be static? Signed-off-by: Hannes Eder <hannes@hanneseder.net> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: remove assertion in kvm_mmu_alloc_pageJoerg Roedel2009-03-241-1/+0
| | | | | | | | The assertion no longer makes sense since we don't clear page tables on allocation; instead we clear them during prefetch. Signed-off-by: Joerg Roedel <joerg.roedel@amd.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: remove redundant check in mmu_set_spteJoerg Roedel2009-03-241-6/+2
| | | | | | | | | | | | | | | | | | The following code flow is unnecessary: if (largepage) was_rmapped = is_large_pte(*shadow_pte); else was_rmapped = 1; The is_large_pte() function will always evaluate to one here because the (largepage && !is_large_pte) case is already handled in the first if-clause. So we can remove this check and set was_rmapped to one always here. Signed-off-by: Joerg Roedel <joerg.roedel@amd.com> Acked-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Rename "metaphysical" attribute to "direct"Avi Kivity2009-03-241-16/+16
| | | | | | | This actually describes what is going on, rather than alerting the reader that something strange is going on. Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: drop zeroing on mmu_memory_cache_allocMarcelo Tosatti2009-03-241-1/+0
| | | | | | | | | | | | | | Zeroing on mmu_memory_cache_alloc is unnecessary since: - Smaller areas are pre-allocated with kmem_cache_zalloc. - Page pointed by ->spt is overwritten with prefetch_page and entries in page pointed by ->gfns are initialized before reading. [avi: zeroing pages is unnecessary] Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Optimize page unshadowingAvi Kivity2009-03-241-3/+12
| | | | | | | Using kvm_mmu_lookup_page() will result in multiple scans of the hash chains; use hlist_for_each_entry_safe() to achieve a single scan instead. Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Drop walk_shadow()Avi Kivity2009-03-241-20/+0
| | | | | | No longer used. Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Use for_each_shadow_entry() in __direct_map()Avi Kivity2009-03-241-54/+29
| | | | | | Eliminating a callback and a useless structure. Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Add for_each_shadow_entry(), a simpler alternative to walk_shadow()Avi Kivity2009-03-241-20/+49
| | | | | | | | | Using a for_each loop style removes the need to write callback and nasty casts. Implement the walk_shadow() using the for_each_shadow_entry(). Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Initialize a shadow page's global attribute from cr4.pgeAvi Kivity2009-03-241-3/+1
| | | | | | | If cr4.pge is cleared, we ought to treat any ptes in the page as non-global. This allows us to remove the check from set_spte(). Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Inherit a shadow page's guest level count from vcpu setupAvi Kivity2009-03-241-6/+11
| | | | | | | | | | Instead of "calculating" it on every shadow page allocation, set it once when switching modes, and copy it when allocating pages. This doesn't buy us much, but sets up the stage for inheriting more information related to the mmu setup. Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Map device MMIO as UC in EPTSheng Yang2009-02-151-2/+7
| | | | | | | | | Software are not allow to access device MMIO using cacheable memory type, the patch limit MMIO region with UC and WC(guest can select WC using PAT and PCD/PWT). Signed-off-by: Sheng Yang <sheng@linux.intel.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: handle large host sptes on invlpg/resyncMarcelo Tosatti2008-12-311-1/+1
| | | | | | | | | | | | The invlpg and sync walkers lack knowledge of large host sptes, descending to non-existant pagetable level. Stop at directory level in such case. Fixes SMP Windows XP with hugepages. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Don't treat a global pte as such if cr4.pge is clearedAvi Kivity2008-12-311-0/+2
| | | | | | | | | | The pte.g bit is meaningless if global pages are disabled; deferring mmu page synchronization on these ptes will lead to the guest using stale shadow ptes. Fixes Vista x86 smp bootloader failure. Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: check for present pdptr shadow page in walk_shadowMarcelo Tosatti2008-12-311-0/+2
| | | | | | | | | | walk_shadow assumes the caller verified validity of the pdptr pointer in question, which is not the case for the invlpg handler. Fixes oops during Solaris 10 install. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: prepopulate the shadow on invlpgMarcelo Tosatti2008-12-311-12/+13
| | | | | | | | | | | | If the guest executes invlpg, peek into the pagetable and attempt to prepopulate the shadow entry. Also stop dirty fault updates from interfering with the fork detector. 2% improvement on RHEL3/AIM7. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: skip global pgtables on sync due to cr3 switchMarcelo Tosatti2008-12-311-6/+47
| | | | | | | | | Skip syncing global pages on cr3 switch (but not on cr4/cr0). This is important for Linux 32-bit guests with PAE, where the kmap page is marked as global. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: collapse remote TLB flushes on root syncMarcelo Tosatti2008-12-311-5/+14
| | | | | | | | | | Collapse remote TLB flushes on root sync. kernbench is 2.7% faster on 4-way guest. Improvements have been seen with other loads such as AIM7. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: use page array in unsync walkMarcelo Tosatti2008-12-311-55/+140
| | | | | | | | | | Instead of invoking the handler directly collect pages into an array so the caller can work with it. Simplifies TLB flush collapsing. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: optimize set_spte for page syncMarcelo Tosatti2008-12-311-0/+9
| | | | | | | | | | | | | | | | | | The write protect verification in set_spte is unnecessary for page sync. Its guaranteed that, if the unsync spte was writable, the target page does not have a write protected shadow (if it had, the spte would have been write protected under mmu_lock by rmap_write_protect before). Same reasoning applies to mark_page_dirty: the gfn has been marked as dirty via the pagefault path. The cost of hash table and memslot lookups are quite significant if the workload is pagetable write intensive resulting in increased mmu_lock contention. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: VMX: move vmx.h to include/asmEduardo Habkost2008-12-311-1/+1
| | | | | | | | vmx.h will be used by core code that is independent of KVM, so I am moving it outside the arch/x86/kvm directory. Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Fix aliased gfns treated as unaliasedIzik Eidus2008-12-311-4/+10
| | | | | | | | | | | Some areas of kvm x86 mmu are using gfn offset inside a slot without unaliasing the gfn first. This patch makes sure that the gfn will be unaliased and add gfn_to_memslot_unaliased() to save the calculating of the gfn unaliasing in case we have it unaliased already. Signed-off-by: Izik Eidus <ieidus@redhat.com> Acked-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Extend kvm_mmu_page->slot_bitmap sizeSheng Yang2008-12-311-3/+3
| | | | | | | | Otherwise set_bit() for private memory slot(above KVM_MEMORY_SLOTS) would corrupted memory in 32bit host. Signed-off-by: Sheng Yang <sheng@linux.intel.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: Enable MTRR for EPTSheng Yang2008-12-311-1/+10
| | | | | | | | The effective memory type of EPT is the mixture of MSR_IA32_CR_PAT and memory type field of EPT entry. Signed-off-by: Sheng Yang <sheng@linux.intel.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: Add local get_mtrr_type() to support MTRRSheng Yang2008-12-311-0/+104
| | | | | | | For EPT memory type support. Signed-off-by: Sheng Yang <sheng@linux.intel.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: fix sync of ptes addressed at owner pagetableMarcelo Tosatti2008-11-231-1/+1
| | | | | | | | | | | During page sync, if a pagetable contains a self referencing pte (that points to the pagetable), the corresponding spte may be marked as writable even though all mappings are supposed to be write protected. Fix by clearing page unsync before syncing individual sptes. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: increase per-vcpu rmap cache alloc sizeMarcelo Tosatti2008-11-111-1/+1
| | | | | | | | | | | | | | The page fault path can use two rmap_desc structures, if: - walk_addr's dirty pte update allocates one rmap_desc. - mmu_lock is dropped, sptes are zapped resulting in rmap_desc being freed. - fetch->mmu_set_spte allocates another rmap_desc. Increase to 4 for safety. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: sync root on paravirt TLB flushMarcelo Tosatti2008-10-281-0/+1
| | | | | | | | The pvmmu TLB flush handler should request a root sync, similarly to a native read-write CR3. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: add "oos_shadow" parameter to disable oosMarcelo Tosatti2008-10-151-1/+4
| | | | | | | Subject says it all. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: speed up mmu_unsync_walkMarcelo Tosatti2008-10-151-12/+60
| | | | | | | Cache the unsynced children information in a per-page bitmap. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: out of sync shadow coreMarcelo Tosatti2008-10-151-17/+193
| | | | | | | | | | | Allow guest pagetables to go out of sync. Instead of emulating write accesses to guest pagetables, or unshadowing them, we un-write-protect the page table and allow the guest to modify it at will. We rely on invlpg executions to synchronize individual ptes, and will synchronize the entire pagetable on tlb flushes. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: mmu_convert_notrap helperMarcelo Tosatti2008-10-151-0/+14
| | | | | | | | Need to convert shadow_notrap_nonpresent -> shadow_trap_nonpresent when unsyncing pages. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: awareness of new kvm_mmu_zap_page behaviourMarcelo Tosatti2008-10-151-4/+9
| | | | | | | | kvm_mmu_zap_page will soon zap the unsynced children of a page. Restart list walk in such case. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: mmu_parent_walkMarcelo Tosatti2008-10-151-0/+27
| | | | | | | Introduce a function to walk all parents of a given page, invoking a handler. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: x86: trap invlpgMarcelo Tosatti2008-10-151-0/+18
| | | | | | | | | | With pages out of sync invlpg needs to be trapped. For now simply nuke the entry. Untested on AMD. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: sync roots on mmu reloadMarcelo Tosatti2008-10-151-0/+36
| | | | | Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: mode specific sync_pageMarcelo Tosatti2008-10-151-0/+10
| | | | | | | | Examine guest pagetable and bring the shadow back in sync. Caller is responsible for local TLB flush before re-entering guest mode. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: do not write-protect large mappingsMarcelo Tosatti2008-10-151-2/+8
| | | | | | | | | | | | | There is not much point in write protecting large mappings. This can only happen when a page is shadowed during the window between is_largepage_backed and mmu_lock acquision. Zap the entry instead, so the next pagefault will find a shadowed page via is_largepage_backed and fallback to 4k translations. Simplifies out of sync shadow. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: move local TLB flush to mmu_set_spteMarcelo Tosatti2008-10-151-4/+4
| | | | | | | Since the sync page path can collapse flushes. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: split mmu_set_spteMarcelo Tosatti2008-10-151-44/+57
| | | | | | | Split the spte entry creation code into a new set_spte function. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: switch to get_user_pages_fastMarcelo Tosatti2008-10-151-14/+9
| | | | | | | | | Convert gfn_to_pfn to use get_user_pages_fast, which can do lockless pagetable lookups on x86. Kernel compilation on 4-way guest is 3.7% faster on VMX. Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com>
* KVM: MMU: Modify kvm_shadow_walk.entry to accept u64 addrSheng Yang2008-10-151-5/+5
| | | | | | | | | | | | | EPT is 4 level by default in 32pae(48 bits), but the addr parameter of kvm_shadow_walk->entry() only accept unsigned long as virtual address, which is 32bit in 32pae. This result in SHADOW_PT_INDEX() overflow when try to fetch level 4 index. Fix it by extend kvm_shadow_walk->entry() to accept 64bit addr in parameter. Signed-off-by: Sheng Yang <sheng.yang@intel.com> Signed-off-by: Avi Kivity <avi@qumranet.com>
* KVM: MMU: Fix setting the accessed bit on non-speculative sptesAvi Kivity2008-10-151-1/+1
| | | | | | | | | | | The accessed bit was accidentally turned on in a random flag word, rather than, the spte itself, which was lucky, since it used the non-EPT compatible PT_ACCESSED_MASK. Fix by turning the bit on in the spte and changing it to use the portable accessed mask. Signed-off-by: Avi Kivity <avi@qumranet.com>
* KVM: MMU: Flush tlbs after clearing write permission when accessing dirty logAvi Kivity2008-10-151-0/+1
| | | | | | | Otherwise, the cpu may allow writes to the tracked pages, and we lose some display bits or fail to migrate correctly. Signed-off-by: Avi Kivity <avi@qumranet.com>
* KVM: MMU: Add locking around kvm_mmu_slot_remove_write_access()Avi Kivity2008-10-151-0/+2
| | | | | | | It was generally safe due to slots_lock being held for write, but it wasn't very nice. Signed-off-by: Avi Kivity <avi@qumranet.com>
* KVM: MMU: Account for npt/ept/realmode page faultsAvi Kivity2008-10-151-0/+1
| | | | | | | Now that two-dimensional paging is becoming common, account for tdp page faults. Signed-off-by: Avi Kivity <avi@qumranet.com>
* KVM: MMU: Convert direct maps to use the generic shadow walkerAvi Kivity2008-10-151-38/+55
| | | | Signed-off-by: Avi Kivity <avi@qumranet.com>
* KVM: MMU: Add generic shadow walkerAvi Kivity2008-10-151-0/+34
| | | | | | | | | | | | | | We currently walk the shadow page tables in two places: direct map (for real mode and two dimensional paging) and paging mode shadow. Since we anticipate requiring a third walk (for invlpg), it makes sense to have a generic facility for shadow walk. This patch adds such a shadow walker, walks the page tables and calls a method for every spte encountered. The method can examine the spte, modify it, or even instantiate it. The walk can be aborted by returning nonzero from the method. Signed-off-by: Avi Kivity <avi@qumranet.com>
* KVM: MMU: Infer shadow root level in direct_map()Avi Kivity2008-10-151-5/+4
| | | | | | | In all cases the shadow root level is available in mmu.shadow_root_level, so there is no need to pass it as a parameter. Signed-off-by: Avi Kivity <avi@qumranet.com>
OpenPOWER on IntegriCloud