In the common case, pmap_enter_quick() completes without sleeping.

In such cases, the busying of the page and the unlocking of the
containing object by vm_map_pmap_enter() and vm_fault_prefault() is
unnecessary overhead.  To eliminate this overhead, this change
modifies pmap_enter_quick() so that it expects the object to be locked
on entry and it assumes the responsibility for busying the page and
unlocking the object if it must sleep.  Note: alpha, amd64, i386 and
ia64 are the only implementations optimized by this change; arm,
powerpc, and sparc64 still conservatively busy the page and unlock the
object within every pmap_enter_quick() call.

Additionally, this change is the first case where we synchronize
access to the page's PG_BUSY flag and busy field using the containing
object's lock rather than the global page queues lock.  (Modifications
to the page's PG_BUSY flag and busy field have asserted both locks for
several weeks, enabling an incremental transition.)

1 files changed, 4 insertions, 9 deletions

diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
index 3788a4f..46ed849 100644
--- a/sys/vm/vm_map.c
+++ b/sys/vm/vm_map.c
@@ -1417,22 +1417,17 @@ vm_map_pmap_enter(vm_map_t map, vm_offset_t addr, vm_prot_t prot,
 		    cnt.v_free_count < cnt.v_free_reserved) {
 			break;
 		}
-		vm_page_lock_queues();
 		if ((p->valid & VM_PAGE_BITS_ALL) == VM_PAGE_BITS_ALL &&
 		    (p->busy == 0) &&
 		    (p->flags & (PG_BUSY | PG_FICTITIOUS)) == 0) {
-			if ((p->queue - p->pc) == PQ_CACHE)
+			if ((p->queue - p->pc) == PQ_CACHE) {
+				vm_page_lock_queues();
 				vm_page_deactivate(p);
-			vm_page_busy(p);
-			vm_page_unlock_queues();
-			VM_OBJECT_UNLOCK(object);
+				vm_page_unlock_queues();
+			}
 			mpte = pmap_enter_quick(map->pmap,
 				addr + ptoa(tmpidx), p, mpte);
-			VM_OBJECT_LOCK(object);
-			vm_page_lock_queues();
-			vm_page_wakeup(p);
 		}
-		vm_page_unlock_queues();
 	}
 unlock_return:
 	VM_OBJECT_UNLOCK(object);