Extend taskqueue(9) to enable per-taskqueue callbacks.

The scope of these callbacks is primarily to support actions that affect the
taskqueue's thread environments.  They are entirely optional, and
consequently are introduced as a new API: taskqueue_set_callback().

This interface allows the caller to specify that a taskqueue requires a
callback and optional context pointer for a given callback type.

The callback types included in this commit can be used to register a
constructor and destructor for thread-local storage using osd(9).  This
allows a particular taskqueue to define that its threads require a specific
type of TLS, without the need for a specially-orchestrated task-based
mechanism for startup and shutdown in order to accomplish it.

Two callback types are supported at this point:

- TASKQUEUE_CALLBACK_TYPE_INIT, called by every thread when it starts, prior
  to processing any tasks.
- TASKQUEUE_CALLBACK_TYPE_SHUTDOWN, called by every thread when it exits,
  after it has processed its last task but before the taskqueue is
  reclaimed.

While I'm here:

- Add two new macros, TQ_ASSERT_LOCKED and TQ_ASSERT_UNLOCKED, and use them
  in appropriate locations.
- Fix taskqueue.9 to mention taskqueue_start_threads(), which is a required
  interface for all consumers of taskqueue(9).

Reviewed by:	kib (all), eadler (taskqueue.9), brd (taskqueue.9)
Approved by:	ken (mentor)
Sponsored by:	Spectra Logic
MFC after:	1 month

1 files changed, 28 insertions, 0 deletions

diff --git a/share/man/man9/taskqueue.9 b/share/man/man9/taskqueue.9
index c8e21d3..5f6131a 100644
--- a/share/man/man9/taskqueue.9
+++ b/share/man/man9/taskqueue.9
@@ -53,12 +53,23 @@ struct task {
 	void			*ta_context;	/* argument for handler */
 };
 
+enum taskqueue_callback_type {
+	TASKQUEUE_CALLBACK_TYPE_INIT,
+	TASKQUEUE_CALLBACK_TYPE_SHUTDOWN,
+};
+
+typedef void (*taskqueue_callback_fn)(void *context);
+
 struct timeout_task;
 .Ed
 .Ft struct taskqueue *
 .Fn taskqueue_create "const char *name" "int mflags" "taskqueue_enqueue_fn enqueue" "void *context"
 .Ft struct taskqueue *
 .Fn taskqueue_create_fast "const char *name" "int mflags" "taskqueue_enqueue_fn enqueue" "void *context"
+.Ft int
+.Fn taskqueue_start_threads "struct taskqueue **tqp" "int count" "int pri" "const char *name" "..."
+.Ft void
+.Fn taskqueue_set_callback "struct taskqueue *queue" "enum taskqueue_callback_type cb_type" "taskqueue_callback_fn callback" "void *context"
 .Ft void
 .Fn taskqueue_free "struct taskqueue *queue"
 .Ft int
@@ -127,6 +138,23 @@ should be used to free the memory used by the queue.
 Any tasks that are on the queue will be executed at this time after
 which the thread servicing the queue will be signaled that it should exit.
 .Pp
+Once a taskqueue has been created, its threads should be started using
+.Fn taskqueue_start_threads .
+Callbacks may optionally be registered using
+.Fn taskqueue_set_callback .
+Currently, callbacks may be registered for the following purposes:
+.Bl -tag -width TASKQUEUE_CALLBACK_TYPE_SHUTDOWN
+.It Dv TASKQUEUE_CALLBACK_TYPE_INIT
+This callback is called by every thread in the taskqueue, before it executes
+any tasks.
+This callback must be set before the taskqueue's threads are started.
+.It Dv TASKQUEUE_CALLBACK_TYPE_SHUTDOWN
+This callback is called by every thread in the taskqueue, after it executes
+its last task.
+This callback will always be called before the taskqueue structure is
+reclaimed.
+.El
+.Pp
 To add a task to the list of tasks queued on a taskqueue, call
 .Fn taskqueue_enqueue
 with pointers to the queue and task.