Correct two error cases in vm_map_unwire():

1. Contrary to the Single Unix Specification our implementation of
   munlock(2) when performed on an unwired virtual address range has
   returned an error.  Correct this.  Note, however, that the behavior
   of "system" unwiring is unchanged, only "user" unwiring is changed.
   If "system" unwiring is performed on an unwired virtual address
   range, an error is still returned.

2. Performing an errant "system" unwiring on a virtual address range
   that was "user" (i.e., mlock(2)) but not "system" wired would
   incorrectly undo the "user" wiring instead of returning an error.
   Correct this.

Discussed with:  green@
Reviewed by:     tegge@

1 files changed, 5 insertions, 4 deletions

diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
index 5324400..d27df6f 100644
--- a/sys/vm/vm_map.c
+++ b/sys/vm/vm_map.c
@@ -1696,10 +1696,10 @@ vm_map_unwire(vm_map_t map, vm_offset_t start, vm_offset_t end,
 			goto done;
 		}
 		/*
-		 * Require that the entry is wired.
+		 * If system unwiring, require that the entry is system wired.
 		 */
-		if (entry->wired_count == 0 || (user_unwire &&
-		    (entry->eflags & MAP_ENTRY_USER_WIRED) == 0)) {
+		if (!user_unwire && entry->wired_count < ((entry->eflags &
+		    MAP_ENTRY_USER_WIRED) ? 2 : 1)) {
 			end = entry->end;
 			rv = KERN_INVALID_ARGUMENT;
 			goto done;
@@ -1718,7 +1718,8 @@ done:
 	}
 	entry = first_entry;
 	while (entry != &map->header && entry->start < end) {
-		if (rv == KERN_SUCCESS) {
+		if (rv == KERN_SUCCESS && (!user_unwire ||
+		    (entry->eflags & MAP_ENTRY_USER_WIRED))) {
 			if (user_unwire)
 				entry->eflags &= ~MAP_ENTRY_USER_WIRED;
 			entry->wired_count--;