Implement preemption of kernel threads natively in the scheduler rather

than as one-off hacks in various other parts of the kernel:
- Add a function maybe_preempt() that is called from sched_add() to
  determine if a thread about to be added to a run queue should be
  preempted to directly.  If it is not safe to preempt or if the new
  thread does not have a high enough priority, then the function returns
  false and sched_add() adds the thread to the run queue.  If the thread
  should be preempted to but the current thread is in a nested critical
  section, then the flag TDF_OWEPREEMPT is set and the thread is added
  to the run queue.  Otherwise, mi_switch() is called immediately and the
  thread is never added to the run queue since it is switch to directly.
  When exiting an outermost critical section, if TDF_OWEPREEMPT is set,
  then clear it and call mi_switch() to perform the deferred preemption.
- Remove explicit preemption from ithread_schedule() as calling
  setrunqueue() now does all the correct work.  This also removes the
  do_switch argument from ithread_schedule().
- Do not use the manual preemption code in mtx_unlock if the architecture
  supports native preemption.
- Don't call mi_switch() in a loop during shutdown to give ithreads a
  chance to run if the architecture supports native preemption since
  the ithreads will just preempt DELAY().
- Don't call mi_switch() from the page zeroing idle thread for
  architectures that support native preemption as it is unnecessary.
- Native preemption is enabled on the same archs that supported ithread
  preemption, namely alpha, i386, and amd64.

This change should largely be a NOP for the default case as committed
except that we will do fewer context switches in a few cases and will
avoid the run queues completely when preempting.

Approved by:	scottl (with his re@ hat)

2 files changed, 7 insertions, 0 deletions

diff --git a/sys/conf/NOTES b/sys/conf/NOTES
index 2607142..f944152 100644
--- a/sys/conf/NOTES
+++ b/sys/conf/NOTES
@@ -186,6 +186,11 @@ options 	MUTEX_WAKE_ALL
 
 # SMP Debugging Options:
 #
+# FULL_PREEMPTION instructs the kernel to preempt non-realtime kernel
+#	  threads.  It sole use is to expose race conditions and other
+#	  bugs during development.  Enabling this option will reduce
+#	  performance and increase the frequency of kernel panics by
+#	  design.  If you aren't sure that you need it then you don't.
 # MUTEX_DEBUG enables various extra assertions in the mutex code.
 # SLEEPQUEUE_PROFILING enables rudimentary profiling of the hash table
 #	  used to hold active sleep queues.
@@ -197,6 +202,7 @@ options 	MUTEX_WAKE_ALL
 #	  a lock hierarchy violation occurs or if locks are held when going to
 #	  sleep.
 # WITNESS_SKIPSPIN disables the witness checks on spin mutexes.
+options 	FULL_PREEMPTION
 options 	MUTEX_DEBUG
 options 	WITNESS
 options 	WITNESS_DDB
diff --git a/sys/conf/options b/sys/conf/options
index d086703..2fb6582 100644
--- a/sys/conf/options
+++ b/sys/conf/options
@@ -61,6 +61,7 @@ DDB_NUMSYM	opt_ddb.h
 DDB_TRACE
 DDB_UNATTENDED
 DIRECTIO	opt_directio.h
+FULL_PREEMPTION
 GDB_REMOTE_CHAT	opt_ddb.h
 GDBSPEED	opt_ddb.h
 GEOM_AES	opt_geom.h