summaryrefslogtreecommitdiffstats
path: root/sys/kern/kern_rwlock.c
Commit message (Collapse)AuthorAgeFilesLines
* When releasing a read/shared lock we need to use a write memory barrierattilio2009-09-301-3/+4
| | | | | | | | | | in order to avoid, on architectures which doesn't have strong ordered writes, CPU instructions reordering. Diagnosed by: fabio Reviewed by: jhb Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com>
* * Change the scope of the ASSERT_ATOMIC_LOAD() from a generic check toattilio2009-08-171-2/+3
| | | | | | | | | | | | | | | a pointer-fetching specific operation check. Consequently, rename the operation ASSERT_ATOMIC_LOAD_PTR(). * Fix the implementation of ASSERT_ATOMIC_LOAD_PTR() by checking directly alignment on the word boundry, for all the given specific architectures. That's a bit too strict for some common case, but it assures safety. * Add a comment explaining the scope of the macro * Add a new stub in the lockmgr specific implementation Tested by: marcel (initial version), marius Reviewed by: rwatson, jhb (comment specific review) Approved by: re (kib)
* Add a new macro to test that a variable could be loaded atomically.bz2009-08-141-0/+2
| | | | | | | | | | | | | | | | | | | Check that the given variable is at most uintptr_t in size and that it is aligned. Note: ASSERT_ATOMIC_LOAD() uses ALIGN() to check for adequate alignment -- however, the function of ALIGN() is to guarantee alignment, and therefore may lead to stronger alignment enforcement than necessary for types that are smaller than sizeof(uintptr_t). Add checks to mtx, rw and sx locks init functions to detect possible breakage. This was used during debugging of the problem fixed with r196118 where a pointer was on an un-aligned address in the dpcpu area. In collaboration with: rwatson Reviewed by: rwatson Approved by: re (kib)
* Handle lock recursion differenty by always checking against LO_RECURSABLEattilio2009-06-021-6/+6
| | | | | | instead the lock own flag itself. Tested by: pho
* Remove extra cpu_spinwait() invocations. This should really only be usedjhb2009-05-291-8/+0
| | | | | | | in tight spin loops, not in these edge cases where we restart a much larger loop only a few times. Reviewed by: attilio
* Tweak a few comments on adaptive spinning.jhb2009-05-291-4/+10
|
* Add the OpenSolaris dtrace lockstat provider. The lockstat providersson2009-05-261-7/+90
| | | | | | | | | | adds probes for mutexes, reader/writer and shared/exclusive locks to gather contention statistics and other locking information for dtrace scripts, the lockstat(1M) command and other potential consumers. Reviewed by: attilio jhb jb Approved by: gnn (mentor)
* - Wrap lock profiling state variables in #ifdef LOCK_PROFILING blocks.jeff2009-03-151-1/+5
|
* Remove even more unneeded variable assignments.ed2009-02-261-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | kern_time.c: - Unused variable `p'. kern_thr.c: - Variable `error' is always caught immediately, so no reason to initialize it. There is no way that error != 0 at the end of create_thread(). kern_sig.c: - Unused variable `code'. kern_synch.c: - `rval' is always assigned in all different cases. kern_rwlock.c: - `v' is always overwritten with RW_UNLOCKED further on. kern_malloc.c: - `size' is always initialized with the proper value before being used. kern_exit.c: - `error' is always caught and returned immediately. abort2() never returns a non-zero value. kern_exec.c: - `len' is always assigned inside the if-statement right below it. tty_info.c: - `td' is always overwritten by FOREACH_THREAD_IN_PROC(). Found by: LLVM's scan-build
* add RW_SYSINIT_FLAGS macro and rw_sysinit_flags initialization functionkmacy2008-12-081-0/+8
|
* Teach WITNESS about the interlocks used with lockmgr. This removes a bunchjhb2008-09-101-2/+2
| | | | | | | | of spurious witness warnings since lockmgr grew witness support. Before this, every time you passed an interlock to a lockmgr lock WITNESS treated it as a LOR. Reviewed by: attilio
* Various whitespace fixes.jhb2008-09-101-1/+0
|
* Improve a comment which, in the actual CVS stock, doesn't completelyattilio2008-05-271-2/+5
| | | | explain the logic of the code chunk.
* - Add sysctls at debug.rwlock to control the behavior of the speculativejeff2008-04-041-3/+26
| | | | | | | | | | | | spinning when readers hold a lock. This spinning is speculative because, unlike the write case, we can not test whether the owners are running. - Add speculative read spinning for readers who are blocked by pending writers while a read lock is still held. This allows the thread to spin until the write lock succeeds after which it may spin until the writer has released the lock. This prevents excessive context switches when readers and writers both hold the lock for brief periods. Sponsored by: Nokia
* Add rw_try_rlock() and rw_try_wlock() to rwlocks.attilio2008-04-011-0/+49
| | | | | | | | | | These functions try the specified operation (rlocking and wlocking) and true is returned if the operation completes, false otherwise. The KPI is enriched by this commit, so __FreeBSD_version bumping and manpage updating will happen soon. Requested by: jeff, kris
* - In rw_wunlock_hard prefer to wakeup writers if there are both readersjeff2008-02-071-4/+4
| | | | | | | and writers available. Doing otherwise can cause deadlocks as no read locks can proceed while there are write waiters. Sponsored by: Nokia
* Adaptive spinning in write path with readers and writer starvation avoidance.jeff2008-02-061-154/+170
| | | | | | | | | | | | | | | | - Move recursion checking into rwlock inlines to free a bit for use with adaptive spinners. - Clear the RW_LOCK_WRITE_SPINNERS flag whenever the lock state changes causing write spinners to restart their loop. - Write spinners are limited by a count while readers hold the lock as there is no way to know for certain whether readers are running still. - In the read path block if there are write waiters or spinners to avoid starving writers. Use a new per-thread count, td_rw_rlocks, to skip starvation avoidance if it might cause a deadlock. - Remove or change invalid assertions in turnstiles. Reviewed by: attilio (developed parts of the patch as well) Sponsored by: Nokia
* Remove a conditional that is always true.jhb2008-01-171-1/+1
| | | | MFC after: 2 weeks
* - Re-implement lock profiling in such a way that it no longer breaksjeff2007-12-151-24/+7
| | | | | | | | | | | | | | | | | | | | | | the ABI when enabled. There is no longer an embedded lock_profile_object in each lock. Instead a list of lock_profile_objects is kept per-thread for each lock it may own. The cnt_hold statistic is now always 0 to facilitate this. - Support shared locking by tracking individual lock instances and statistics in the per-thread per-instance lock_profile_object. - Make the lock profiling hash table a per-cpu singly linked list with a per-cpu static lock_prof allocator. This removes the need for an array of spinlocks and reduces cache contention between cores. - Use a seperate hash for spinlocks and other locks so that only a critical_enter() is required and not a spinlock_enter() to modify the per-cpu tables. - Count time spent spinning in the lock statistics. - Remove the LOCK_PROFILE_SHARED option as it is always supported now. - Specifically drop and release the scheduler locks in both schedulers since we track owners now. In collaboration with: Kip Macy Sponsored by: Nokia
* Simplify the adaptive spinning algorithm in rwlock and mutex:attilio2007-11-261-112/+72
| | | | | | | | | | | | | | | | | | | | | currently, before to spin the turnstile spinlock is acquired and the waiters flag is set. This is not strictly necessary, so just spin before to acquire the spinlock and to set the flags. This will simplify a lot other functions too, as now we have the waiters flag set only if there are actually waiters. This should make wakeup/sleeping couplet faster under intensive mutex workload. This also fixes a bug in rw_try_upgrade() in the adaptive case, where turnstile_lookup() will recurse on the ts_lock lock that will never be really released [1]. [1] Reported by: jeff with Nokia help Tested by: pho, kris (earlier, bugged version of rwlock part) Discussed with: jhb [2], jeff MFC after: 1 week [2] John had a similar patch about 6.x and/or 7.x about mutexes probabilly
* Expand lock class with the "virtual" function lc_assert which will offerattilio2007-11-181-0/+9
| | | | | | | | | an unified way for all the lock primitives to express lock assertions. Currenty, lockmgrs and rmlocks don't have assertions, so just panic in that case. This will be a base for more callout improvements. Ok'ed by: jhb, jeff
* Remove a bogus KASSERT which will prevent rwlock to be acquiredattilio2007-11-141-3/+0
| | | | | | | recursively in exclusive mode with debugging kernels. Submitted by: kmacy Approved by: jeff
* generally we are interested in what thread did something asjulian2007-11-141-1/+1
| | | | | | opposed to what process. Since threads by default have teh name of the process unless over-written with more useful information, just print the thread name instead.
* Fix some problems with lock profiling in rw locks:attilio2007-07-201-8/+28
| | | | | | | | | | | | | | | | | - Adjust lock_profiling stubs semantic in the hard functions in order to be more accurate and trustable - As for sx locks, disable shared paths for lock_profiling. Actually, lock_profiling has a subtle race which makes results caming from shared paths not completely trustable. A macro stub (LOCK_PROFILING_SHARED) can be actually used for re-enabling this paths, but is currently intended for developing use only. - style(9) fixes Approved by: jeff, kmacy, jhb[1] Approved by: re [1] Had initial reservations not shared by others, conceded in the end.
* Introduce a new rwlocks initialization function: rw_init_flags.attilio2007-06-261-9/+85
| | | | | | | | | | | | | | | This is very similar to sx_init_flags: it initializes the rwlock using special flags passed as third argument (RW_DUPOK, RW_NOPROFILE, RW_NOWITNESS, RW_QUIET, RW_RECURSE). Among these, the most important new feature is probabilly that rwlocks can be acquired recursively now (for both shared and exclusive paths). Because of the recursion counter, the ABI is changed. Tested by: Timothy Redaelli <drizzt@gufi.org> Reviewed by: jhb Approved by: jeff (mentor) Approved by: re
* Commit 3/14 of sched_lock decomposition.jeff2007-06-041-26/+28
| | | | | | | | | | | | | | | | | - Add a per-turnstile spinlock to solve potential priority propagation deadlocks that are possible with thread_lock(). - The turnstile lock order is defined as the exact opposite of the lock order used with the sleep locks they represent. This allows us to walk in reverse order in priority_propagate and this is the only place we wish to multiply acquire turnstile locks. - Use the turnstile_chain lock to protect assigning mutexes to turnstiles. - Change the turnstile interface to pass back turnstile pointers to the consumers. This allows us to reduce some locking and makes it easier to cancel turnstile assignment while the turnstile chain lock is held. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)
* Move lock_profile_object_{init,destroy}() into lock_{init,destroy}().jhb2007-05-181-2/+0
|
* Add destroyed cookie values for sx locks and rwlocks as well as extrajhb2007-05-081-1/+17
| | | | | KASSERTs so that any lock operations on a destroyed lock will panic or hang.
* - Drop memory barriers in rw_try_upgrade(). We don't need an 'acq' memoryjhb2007-03-301-4/+4
| | | | | barrier here as the earlier rw_rlock() already contained one. - Comment fix.
* - Simplify the #ifdef's for adaptive mutexes and rwlocks by conditionallyjhb2007-03-221-23/+29
| | | | | defining a macro earlier in the file. - Add NO_ADAPTIVE_RWLOCKS option to disable adaptive spinning for rwlocks.
* Rename the 'mtx_object', 'rw_object', and 'sx_object' members of mutexes,jhb2007-03-211-72/+72
| | | | rwlocks, and sx locks to 'lock_object'.
* Print readers count as unsigned in ddb 'show lock'.jhb2007-03-131-2/+2
| | | | Submitted by: attilio
* Fix a typo.jhb2007-03-121-1/+1
|
* Add two new function pointers 'lc_lock' and 'lc_unlock' to lock classes.jhb2007-03-091-0/+33
| | | | | | | | | | | | | These functions are intended to be used to drop a lock and then reacquire it when doing an sleep such as msleep(9). Both functions accept a 'struct lock_object *' as their first parameter. The 'lc_unlock' function returns an integer that is then passed as the second paramter to the subsequent 'lc_lock' function. This can be used to communicate state. For example, sx locks and rwlocks use this to indicate if the lock was share/read locked vs exclusive/write locked. Currently, spin mutexes and lockmgr locks do not provide working lc_lock and lc_unlock functions.
* Use C99-style struct member initialization for lock classes.jhb2007-03-091-3/+3
|
* Fix some nits in lock profiling for rwlocks:jhb2007-03-071-6/+9
| | | | | | | | | - Properly note when a read lock is released. - Always note when we contest on a read lock. - Only note success of obtaining read locks for the first reader to match the behavior of sx(9). Reviewed by: kmacy
* Further improvements to LOCK_PROFILING:kmacy2007-02-271-2/+2
| | | | | | | | | | | | - Fix missing initialization in kern_rwlock.c causing bogus times to be collected - Move updates to the lock hash to after the lock is released for spin mutexes, sleep mutexes, and sx locks - Add new kernel build option LOCK_PROFILE_FAST - only update lock profiling statistics when an acquisition is contended. This reduces the overhead of LOCK_PROFILING to increasing system time by 20%-25% which on "make -j8 kernel-toolchain" on a dual woodcrest is unmeasurable in terms of wall-clock time. Contrast this to enabling lock profiling without LOCK_PROFILE_FAST and I see a 5x-6x slowdown in wall-clock time.
* Add rw_wowned() interface to rwlock(9), allowing a kernel thread torwatson2007-02-261-0/+7
| | | | | | | | determine if it holds an exclusive rwlock reference or not. This is non-ideal, but recursion scenarios in the network stack currently require it. Approved by: jhb
* general LOCK_PROFILING cleanupkmacy2007-02-261-11/+3
| | | | | | | | | | | | - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
* track lock class name in a way that doesn't break WITNESSkmacy2006-11-131-1/+1
|
* MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profilekmacy2006-11-111-2/+20
| | | | | | | | | | | wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
* Adjust td_locks for non-spin mutexes, rwlocks, and sx locks so that it isjhb2006-07-271-0/+4
| | | | | | | | a count of all non-spin locks, not just lockmgr locks. This can give us a much cheaper way to see if we have any locks held (such as when returning to userland via userret()) without requiring WITNESS. MFC after: 1 week
* Implement rw_try_upgrade() and rw_downgrade(). rw_try_upgrade() makes ajhb2006-04-191-2/+133
| | | | | single attempt at upgrading a read lock to a write lock, and rw_downgrade() converts curthread's write lock into a read lock.
* 'owner' is not used without SMP. Fix kernel build for such kernelwkoszek2006-04-181-0/+4
| | | | | | configurations. Approved by: jhb
* Adaptively spin before blocking on the turnstile if an rwlock is writejhb2006-04-181-7/+88
| | | | | | | | | | | | | | locked. In general the adaptive spinning is similar to the same code for mutexes with some extra trickiness in rw_wunlock_hard(). Specifically, even though both wait bits might be set and we might have a turnstile with at least one waiting thread, there might not be any threads blocked on the queue we are not waking up (they might all be spinning), and we should only preserve the waiting flag for the queue we aren't waking up if there are in fact threads blocked on that queue. Secondly, there might not be any threads blocked on the queue we have chosen to waken threads from (there might only be threads blocked on the other queue and the threads for this queue are all spinning) in which case we disown the turnstile instead of doing a braodcast and unpend.
* - Add a rw_wowner() macro that just returns the owner of a write lock andjhb2006-04-171-26/+38
| | | | | | | | use it in places that only care about the write owner instead of rw_owner() as a baby step towards limited read-lock owner. - Tidy the code that sets the WAITER flag bits to not duplicate a test around the atomic operation and the KTR trace in both of the lock functions.
* Fix another compile problem. If I find any more, this file is going in thescottl2006-02-011-1/+1
| | | | Attic until it is properly fixed.
* Regroup order of operations to better reflect what was probably intended.scottl2006-01-301-3/+3
| | | | Submitted by: Peter Jeremy
* Take a stab at making this compile when WITNESS is not defined. gcc can'tscottl2006-01-291-4/+4
| | | | | figure out the order of operations at line 519, and neither can I, but this is my best guess. Also correct a number of typos and syntax errors.
* Unbreak on archs where %d doesn't print uintptr_t arithmetic.mlaier2006-01-291-1/+2
|
OpenPOWER on IntegriCloud