Consider three objects, O, BO, and BBO, where BO is O's backing object

and BBO is BO's backing object.  Now, suppose that O and BO are being
collapsed.  Furthermore, suppose that BO has been marked dead
(OBJ_DEAD) by vm_object_backing_scan() and that either
vm_object_backing_scan() has been forced to sleep due to encountering
a busy page or vm_object_collapse() has been forced to sleep due to
memory allocation in the swap pager.  If vm_object_deallocate() is
then called on BBO and BO is BBO's only shadow object,
vm_object_deallocate() will collapse BO and BBO.  In doing so, it adds
a necessary temporary reference to BO.  If this collapse also sleeps
and the prior collapse resumes first, the temporary reference will
cause vm_object_collapse to panic with the message "backing_object %p
was somehow re-referenced during collapse!"

Resolve this race by changing vm_object_deallocate() such that it
doesn't collapse BO and BBO if BO is marked dead.  Once O and BO are
collapsed, vm_object_collapse() will attempt to collapse O and BBO.
So, vm_object_deallocate() on BBO need do nothing.

Reported by: Peter Holm on 20050107
URL: http://www.holm.cc/stress/log/cons102.html

In collaboration with: tegge@
Candidate for RELENG_4 and RELENG_5
MFC after: 2 weeks

1 files changed, 8 insertions, 1 deletions

diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c
index 178cfce..887df6a 100644
--- a/sys/vm/vm_object.c
+++ b/sys/vm/vm_object.c
@@ -491,7 +491,14 @@ vm_object_deallocate(vm_object_t object)
 					tsleep(&proc0, PVM, "vmo_de", 1);
 					continue;
 				}
-				if ((robject->handle == NULL) &&
+				/*
+				 * Collapse object into its shadow unless its
+				 * shadow is dead.  In that case, object will
+				 * be deallocated by the thread that is
+				 * deallocating its shadow.
+				 */
+				if ((robject->flags & OBJ_DEAD) == 0 &&
+				    (robject->handle == NULL) &&
 				    (robject->type == OBJT_DEFAULT ||
 				     robject->type == OBJT_SWAP)) {