MFC r303562,303563,r303584,r303643,r303652,r303655,r303707:

    rwlock: s/READER/WRITER/ in wlock lockstat annotation

==

    sx: increment spin_cnt before cpu_spinwait in xlock

    The change is a no-op only done for consistency with the rest of the file.

==

    locks: change sleep_cnt and spin_cnt types to u_int

    Both variables are uint64_t, but they only count spins or sleeps.
    All reasonable values which we can get here comfortably hit in 32-bit range.

==

    Implement trivial backoff for locking primitives.

    All current spinning loops retry an atomic op the first chance they get,
    which leads to performance degradation under load.

    One classic solution to the problem consists of delaying the test to an
    extent. This implementation has a trivial linear increment and a random
    factor for each attempt.

    For simplicity, this first thouch implementation only modifies spinning
    loops where the lock owner is running. spin mutexes and thread lock were
    not modified.

    Current parameters are autotuned on boot based on mp_cpus.

    Autotune factors are very conservative and are subject to change later.

==

    locks: fix up ifdef guards introduced in r303643

    Both sx and rwlocks had copy-pasted ADAPTIVE_MUTEXES instead of the correct
    define.

==

    locks: fix compilation for KDTRACE_HOOKS && !ADAPTIVE_* case

==

    locks: fix sx compilation on mips after r303643

    The kernel.h header is required for the SYSINIT macro, which apparently
    was present on amd64 by accident.

Approved by:	re (gjb)

1 files changed, 24 insertions, 0 deletions

diff --git a/sys/sys/lock.h b/sys/sys/lock.h
index 8d7a068..dbe715a 100644
--- a/sys/sys/lock.h
+++ b/sys/sys/lock.h
@@ -201,9 +201,33 @@ extern struct lock_class lock_class_lockmgr;
 
 extern struct lock_class *lock_classes[];
 
+struct lock_delay_config {
+	u_int initial;
+	u_int step;
+	u_int min;
+	u_int max;
+};
+
+struct lock_delay_arg {
+	struct lock_delay_config *config;
+	u_int delay;
+	u_int spin_cnt;
+};
+
+static inline void
+lock_delay_arg_init(struct lock_delay_arg *la, struct lock_delay_config *lc) {
+	la->config = lc;
+	la->delay = 0;
+	la->spin_cnt = 0;
+}
+
+#define	LOCK_DELAY_SYSINIT(func) \
+	SYSINIT(func##_ld, SI_SUB_LOCK, SI_ORDER_ANY, func, NULL)
+
 void	lock_init(struct lock_object *, struct lock_class *,
 	    const char *, const char *, int);
 void	lock_destroy(struct lock_object *);
+void	lock_delay(struct lock_delay_arg *);
 void	spinlock_enter(void);
 void	spinlock_exit(void);
 void	witness_init(struct lock_object *, const char *);