Solve the dynamic linker's problems with multithreaded programs once

and for all (I hope).  Packages such as wine, JDK, and linuxthreads
should no longer have any problems with re-entering the dynamic
linker.

This commit replaces the locking used in the dynamic linker with a
new spinlock-based reader/writer lock implementation.  Brian
Fundakowski Feldman <green> argued for this from the very beginning,
but it took me a long time to come around to his point of view.
Spinlocks are the only kinds of locks that work with all thread
packages.  But on uniprocessor systems they can be inefficient,
because while a contender for the lock is spinning the holder of the
lock cannot make any progress toward releasing it.  To alleviate
this disadvantage I have borrowed a trick from Sleepycat's Berkeley
DB implementation.  When spinning for a lock, the requester does a
nanosleep() call for 1 usec. each time around the loop.  This will
generally yield the CPU to other threads, allowing the lock holder
to finish its business and release the lock.  I chose 1 usec. as the
minimum sleep which would with reasonable certainty not be rounded
down to 0.

The formerly machine-independent file "lockdflt.c" has been moved
into the architecture-specific subdirectories by repository copy.
It now contains the machine-dependent spinlocking code.  For the
spinlocks I used the very nifty "simple, non-scalable reader-preference
lock" which I found at

  <http://www.cs.rochester.edu/u/scott/synchronization/pseudocode/rw.html>

on all CPUs except the 80386 (the specific CPU model, not the
architecture).  The 80386 CPU doesn't support the necessary "cmpxchg"
instruction, so on that CPU a simple exclusive test-and-set lock
is used instead.  80386 CPUs are detected at initialization time by
trying to execute "cmpxchg" and catching the resulting SIGILL
signal.

To reduce contention for the locks, I have revamped a couple of
key data structures, permitting all common operations to be done
under non-exclusive (reader) locking.  The only operations that
require exclusive locking now are the rare intrusive operations
such as dlopen() and dlclose().

The dllockinit() interface is now deprecated.  It still exists,
but only as a do-nothing stub.  I plan to remove it as soon as is
reasonably possible.  (From the very beginning it was clearly
labeled as experimental and subject to change.)  As far as I know,
only the linuxthreads port uses dllockinit().  This interface turned
out to have several problems.  As one example, when the dynamic
linker called a client-supplied locking function, that function
sometimes needed lazy binding, causing re-entry into the dynamic
linker and a big looping mess.  And in any case, it turned out to be
too burdensome to require threads packages to register themselves
with the dynamic linker.

1 files changed, 18 insertions, 5 deletions

diff --git a/libexec/rtld-elf/rtld.h b/libexec/rtld-elf/rtld.h
index 6d1ebbf..ab00437 100644
--- a/libexec/rtld-elf/rtld.h
+++ b/libexec/rtld-elf/rtld.h
@@ -77,6 +77,23 @@ typedef struct Struct_Needed_Entry {
     unsigned long name;		/* Offset of name in string table */
 } Needed_Entry;
 
+/* Lock object */
+typedef struct Struct_LockInfo {
+    void *context;		/* Client context for creating locks */
+    void *thelock;		/* The one big lock */
+    /* Debugging aids. */
+    volatile int rcount;	/* Number of readers holding lock */
+    volatile int wcount;	/* Number of writers holding lock */
+    /* Methods */
+    void *(*lock_create)(void *context);
+    void (*rlock_acquire)(void *lock);
+    void (*wlock_acquire)(void *lock);
+    void (*rlock_release)(void *lock);
+    void (*wlock_release)(void *lock);
+    void (*lock_destroy)(void *lock);
+    void (*context_destroy)(void *context);
+} LockInfo;
+
 /*
  * Shared object descriptor.
  *
@@ -149,7 +166,6 @@ typedef struct Struct_Obj_Entry {
     Objlist dagmembers;		/* DAG has these members (%) */
     dev_t dev;			/* Object's filesystem's device */
     ino_t ino;			/* Object's inode number */
-    unsigned long mark;		/* Set to "curmark" to avoid repeat visits */
 } Obj_Entry;
 
 #define RTLD_MAGIC	0xd550b87a
@@ -170,10 +186,7 @@ unsigned long elf_hash(const char *);
 const Elf_Sym *find_symdef(unsigned long, Obj_Entry *, const Obj_Entry **,
   bool);
 void init_pltgot(Obj_Entry *);
-void lockdflt_acquire(void *);
-void *lockdflt_create(void *);
-void lockdflt_destroy(void *);
-void lockdflt_release(void *);
+void lockdflt_init(LockInfo *);
 void obj_free(Obj_Entry *);
 Obj_Entry *obj_new(void);
 int reloc_non_plt(Obj_Entry *, Obj_Entry *);