summaryrefslogtreecommitdiffstats
path: root/sys/kern/kern_timeout.c
Commit message (Collapse)AuthorAgeFilesLines
* Fix a race which could result in some timeout buckets being skipped.jeff2008-07-191-6/+11
| | | | | | | | | | | | - When a tick occurs on a cpu, iterate from cs_softticks until ticks. The per-cpu tick processing happens asynchronously with the actual adjustment of the 'ticks' variable. Sometimes the results may be visible before the local call and sometimes after. Previously this could cause a one tick window where we didn't evaluate the bucket. - In softclock fetch curticks before incrementing cc_softticks so we don't skip insertions which were made for the current time. Sponsored by: Nokia
* - Correct a major error introduced in the per-cpu timeout commit. Sleepjeff2008-04-061-5/+5
| | | | | | | and wakeup require the same wait channel to function properly. Found by: kris Pointy hat: me
* Implement per-cpu callout threads, wheels, and locks.jeff2008-04-021-124/+246
| | | | | | | | | | | | | | | | | | | | | - Move callout thread creation from kern_intr.c to kern_timeout.c - Call callout_tick() on every processor via hardclock_cpu() rather than inspecting callout internal details in kern_clock.c. - Remove callout implementation details from callout.h - Package up all of the global variables into a per-cpu callout structure. - Start one thread per-cpu. Threads are not strictly bound. They prefer to execute on the native cpu but may migrate temporarily if interrupts are starving callout processing. - Run all callouts by default in the thread for cpu0 to maintain current ordering and concurrency guarantees. Many consumers may not properly handle concurrent execution. - The new callout_reset_on() api allows specifying a particular cpu to execute the callout on. This may migrate a callout to a new cpu. callout_reset() schedules on the last assigned cpu while callout_reset_curcpu() schedules on the current cpu. Reviewed by: phk Sponsored by: Nokia
* Fix a race where timeout/untimeout could cause crashes for Giant lockedalfred2008-03-221-4/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | code. The bug: There exists a race condition for timeout/untimeout(9) due to the way that the softclock thread dequeues timeouts. The softclock thread sets the c_func and c_arg of the callout to NULL while holding the callout lock but not Giant. It then drops the callout lock and acquires Giant. It is at this point where untimeout(9) on another cpu/thread could be called. Since c_arg and c_func are cleared, untimeout(9) does not touch the callout and returns as if the callout is canceled. The softclock then tries to acquire Giant and likely blocks due to the other cpu/thread holding it. The other cpu/thread then likely deallocates the backing store that c_arg points to and finishes working and hence drops Giant. Softclock resumes and acquires giant and calls the function with the now free'd c_arg and we have corruption/crash. The fix: We need to track curr_callout even for timeout(9) (LOCAL_ALLOC) callouts. We need to free the callout after the softclock processes it to deal with the race here. Obtained from: Juniper Networks, iedowse Reviewed by: jhb, iedowse MFC After: 2 weeks.
* - Pass the priority argument from *sleep() into sleepq and down intojeff2008-03-121-1/+1
| | | | | | | | | | | | | | | | | sched_sleep(). This removes extra thread_lock() acquisition and allows the scheduler to decide what to do with the static boost. - Change the priority arguments to cv_* to match sleepq/msleep/etc. where 0 means no priority change. Catch -1 in cv_broadcastpri() and convert it to 0 for now. - Set a flag when sleeping in a way that is compatible with swapping since direct priority comparisons are meaningless now. - Add a sysctl to ule, kern.sched.static_boost, that defaults to on which controls the boost behavior. Turning it off gives better performance in some workloads but needs more investigation. - While we're modifying sleepq, change signal and broadcast to both return with the lock held as the lock was held on enter. Reviewed by: jhb, peter
* Really, no explicit checks against against lock_class_* object should beattilio2008-02-061-2/+2
| | | | | | | | | | | | | done in consumers code: using locks properties is much more appropriate. Fix current code doing these bogus checks. Note: Really, callout are not usable by all !(LC_SPINLOCK | LC_SLEEPABLE) primitives like rmlocks doesn't implement the generic lock layer functions, but they can be equipped for this, so the check is still valid. Tested by: matteo, kris (earlier version) Reviewed by: jhb
* Cache the value of c_lock as it can change, in the struct,attilio2007-11-221-5/+7
| | | | | | | | while the global callout spinlock is not held, and can lead to PF#. Reported by: dougb, Mark Atkinson <atkin901 at yahoo dot com> Tested by: dougb Diagnosed by: jhb
* Add the function callout_init_rw() to callout facility in order to useattilio2007-11-201-53/+58
| | | | | | | | | | | | | | | | | | | | | rwlocks in conjuction with callouts. The function does basically what callout_init_mtx() alredy does with the difference of using a rwlock as extra argument. CALLOUT_SHAREDLOCK flag can be used, now, in order to acquire the lock only in read mode when running the callout handler. It has no effects when used in conjuction with mtx. In order to implement this, underlying callout functions have been made completely lock type-unaware, so accordingly with this, sysctl debug.to_avg_mtxcalls is now changed in the generic debug.to_avg_lockcalls. Note: currently the allowed lock classes are mutexes and rwlocks because callout handlers run in softclock swi, so they cannot sleep and they cannot acquire sleepable locks like sx or lockmgr. Requested by: kmacy, pjd, rwatson Reviewed by: jhb
* Remove the definition and implementation of 'CALLOUT_NETGIANT', a now- (andrwatson2007-09-151-11/+2
| | | | | | | possibly always-) unused define. Reported by: kmacy Approved by: re (kensmith)
* Close a race that snuck in with the recent changes to fix a LOR betweenjhb2007-08-311-13/+27
| | | | | | | | | | | | | | | | the callout_lock spin lock and the sleepqueue spin locks. In the fix, callout_drain() has to drop the callout_lock so it can acquire the sleepqueue lock. The state of the callout can change while the callout_lock is held however (for example, it can be rescheduled via callout_reset()). The previous code assumed that the only state change that could happen is that the callout could finish executing. This change alters callout_drain() to effectively restart and recheck everything after it acquires the sleepqueue lock thus handling all the possible states that the callout could be in after any changes while callout_lock was dropped. Approved by: re (kensmith) Tested by: kris
* Fix an old standing LOR between callout_lock and sleepqueues chain (whichattilio2007-06-261-3/+35
| | | | | | | | | | | | | | | | | | could lead to a deadlock). - sleepq_set_timeout acquires callout_lock (via callout_reset()) only with sleepq chain lock held - msleep_spin in _callout_stop_safe lock the sleepqueue chain with callout_lock held In order to solve this don't use msleep_spin in _callout_stop_safe() but use directly sleepqueues as inline msleep_spin code. Rearrange the wakeup path in order to have it consistent too. Reported by: kris (via stress2 test suite) Tested by: Timothy Redaelli <drizzt@gufi.org> Reviewed by: jhb Approved by: jeff (mentor) Approved by: re
* Make the TCP timer callout obtain Giant if the network stack is markedandre2007-05-111-2/+11
| | | | | | as non-mpsafe. This change is to be removed when all protocols are mp-safe.
* Improve ktr(4) logging for callout(9) subsystem. Log all inserts andglebius2006-10-111-7/+23
| | | | | | | | | removals, including failures, into the callwheel. XXX: Most of the CTR() macros are called with callout_lock spin mutex held, thus won't be logged into file, if KTR_ALQ is used. Moving the CTR() macros out from the spinlocked code would require copying of all arguments. I'm too lazy to do this.
* Use the recently added msleep_spin() function to simplify thejhb2006-02-231-56/+41
| | | | | | callout_drain() logic. We no longer need a separate non-spin mutex to do sleep/wakeup with, instead we can now just use the one spin mutex to manage all the callout functionality.
* Oops, missed adding the required include.jhb2005-09-151-0/+1
| | | | Pointy hat to: jhb
* Replace the dont_sleep_in_callout mutex hack (similar to g_x{up,down})jhb2005-09-151-8/+2
| | | | with the disallow sleeping facility.
* Make callout_reset() return a non-zero value if a pending calloutglebius2005-09-081-3/+8
| | | | | | was rescheduled. If there was no pending callout, then return 0. Reviewed by: iedowse, cperciva
* When processing a timeout() callout and returning it to the freeiedowse2005-02-111-1/+2
| | | | | | | | | list, set `curr_callout' to NULL. This ensures that we won't attempt to cancel the current callout if the original callout structure gets recycled while we wait to acquire Giant. This is reported to fix an intermittent syscons problem that was introduced by revision 1.96.
* Add a mechanism for associating a mutex with a callout when theiedowse2005-02-071-15/+106
| | | | | | | | | | | | | | | | | | | | | | callout is first initialised, using a new function callout_init_mtx(). The callout system will acquire this mutex before calling the callout function and release it on return. In addition, the callout system uses the mutex to avoid most of the complications and race conditions inherent in asynchronous timer facilities, so mutex-protected callouts have much simpler semantics. As long as the mutex is held when invoking callout_stop() or callout_reset(), then these functions will guarantee that the callout will be stopped, even if softclock() had already begun to process the callout. Existing Giant-locked callouts will automatically pick up the new race-free semantics. This should close a number of race conditions in the USB code and probably other areas of the kernel too. There should be no change in behaviour for "MP-safe" callouts; these still need to use the techniques mentioned in timeout(9) to avoid race conditions.
* Make "c->c_func = NULL" conditional on CALLOUT_LOCAL_ALLOC in bothcperciva2005-01-191-1/+1
| | | | | | | places where it occurs, not just one. :-) Pointed out by: glebius Pointy had to: cperciva
* Make "c->c_func = NULL" conditional on the CALLOUT_LOCAL_ALLOC flag,cperciva2005-01-191-1/+1
| | | | | | | i.e., only clear c->c_func if the callout c is being used via the old timeout(9) interface. Requested by: glebius
* Clarify the description of the callout_active() macro: It is cleared bycperciva2005-01-191-1/+3
| | | | | callout_stop, callout_drain, and callout_deactivate, but is not automatically cleared when a callout returns.
* Adjust two of my comments to the new world order: Indent protection incperciva2005-01-071-2/+2
| | | | the first column is performed using /**, not /*-.
* Cut a KTR record whenever a callout is invoked. Mark whether it runsrwatson2004-08-061-0/+4
| | | | | with Giant or not, and include the function point so it can be looked up against the kernel symbol table during trace analysis.
* When reseting a pending callout, perform the deregistration incperciva2004-08-061-2/+16
| | | | | | | | callout_reset rather than calling callout_stop. This results in a few lines of code duplication, but it provides a significant performance improvement because it avoids recursing on callout_lock. Requested by: rwatson
* The paper "Hashed Timers and Hierarchical Wheels: Data Structures for thehmp2004-04-251-1/+1
| | | | | | | Efficient Implementation of a Timer Facility" was co-author'ed by T. Lauk, not A. Lauk. Adjust nearby whitespace.
* 1. Remove callout_stop binary compatibility.cperciva2004-04-201-10/+0
| | | | | | | 2. Document that this means that kernel modules must be rebuilt. 3. While I'm here, fix my sorting error in callout.h Requested by: many [1], scottl [2], bde [3]
* Add whitespace before comment blocks. (reported by njl)cperciva2004-04-081-16/+13
| | | | | | | | | Remove spurious whitespace, add indent protection, fix punctuation, remove initialization of static variables to zero, put wakeup_ctr and wakeup_needed in the correct order. (reported by bde) This doesn't fix all the style bugs I introduced, but the remaining style bugs make it easier for me to understand what I did here.
* Introduce a callout_drain() function. This acts in the same manner ascperciva2004-04-061-1/+90
| | | | | | | | | | | callout_stop(), except that if the callout being stopped is currently in progress, it blocks attempts to reset the callout and waits until the callout is completed before it returns. This makes it possible to clean up callout-using code safely, e.g., without potentially freeing memory which is still being used by a callout. Reviewed by: mux, gallatin, rwatson, jhb
* Remove advertising clause from University of California Regent's license,imp2004-04-051-4/+0
| | | | | | per letter dated July 22, 1999. Approved by: core
* Make the DIAGNOSTIC code which complains about long {call|time}out(9)phk2003-12-071-5/+11
| | | | | | functions less noisy: We printf if a new function took longer than the previous record holder, or of the previous record holder took more than twice as long as the current record.
* Rename the debugging mutex "callout_no_sleep" to "dont_sleep_in_callout".phk2003-11-151-4/+4
|
* At the request of several developers, restore the DIAGNOSIC codemckusick2003-11-121-0/+28
| | | | | | | | | | deleted in 1.81. Increase the initial timeout limit to 2ms to eliminate spurious messages of excessive timeouts in the NFS client code. Requested by: Poul-Henning Kamp <phk@phk.freebsd.dk> Requested by: Mike Silbersack <silby@silby.com> Requested by: Sam Leffler <sam@errno.com>
* Get rid of DIAGNOSTIC that gives false positives on slow CPUs.mckusick2003-11-041-28/+0
|
* On ia64 time_t is 64 bit. Explicitly cast tv_sec to long and changemarcel2003-08-231-2/+2
| | | | | | | the corresponding format specifier to %ld in a call to printf() in function softclock(). The printf() is conditional upon DIAGNOSTIC. Found by: LINT
* Don't put callout_lock under #ifdef DIAGNOSTIC despite the fact that itphk2003-06-201-1/+2
| | | | works anyway.
* Crude but efficient:phk2003-06-201-1/+8
| | | | | #ifdef DIAGNOSTIC hold a mutex while calling callout's so that we hear about it if they sleep.
* Use __FBSDID().obrien2003-06-111-1/+3
|
* Add instrumentation which tells us how much work softclock() doesphk2003-06-041-2/+26
| | | | per invocation.
* Under DIAGNOSTIC, only report expensive timeouts if they are more expensivephk2003-02-011-1/+2
| | | | than the last on we reported.
* Fix a format buglet.phk2002-09-051-1/+1
| | | | Spotted by: iedowse
* Under DIAGNOSTIC, complain if a timeout(9) routine took more than 1msec.phk2002-09-041-4/+23
|
* Change callers of mtx_init() to pass in an appropriate lock type name. Injhb2002-04-041-1/+1
| | | | | | | most cases NULL is passed, but in some cases such as network driver locks (which use the MTX_NETWORK_LOCK macro) and UMA zone locks, a name is used. Tested on: i386, alpha, sparc64
* Remove __P.alfred2002-03-191-1/+1
|
* Move most of the kernel submap initialization code, including thedillon2001-08-221-0/+49
| | | | | | | | timeout callwheel and buffer cache, out of the platform specific areas and into the machine independant area. i386 and alpha adjusted here. Other cpus can be fixed piecemeal. Reviewed by: freebsd-smp, jake
* Change callout_stop() to return an integer. If callout_stop() succeeds injhb2001-08-101-2/+3
| | | | | | | | | | removing the callout entry, return 1. If callout_stop() fails to remove the callout entry because it is currently executing or has already been executed, then the function returns 0. The idea was obtained from BSD/OS, however, BSD/OS changed untimeout(), and I've just changed callout_stop() to be more conservative. Obtained from: BSD/OS
* Axe spl's obsoleted by the callout mutex.jhb2001-08-101-26/+4
|
* Catch up to header include changes:jhb2001-03-281-0/+1
| | | | | - <sys/mutex.h> now requires <sys/systm.h> - <sys/mutex.h> and <sys/sx.h> now require <sys/lock.h>
* Change and clean the mutex lock interface.bmilekic2001-02-091-19/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
* Revert the last commit to the callout interface, and add a flag tojlemon2000-11-251-6/+7
| | | | | | | callout_init() indicating whether the callout is safe or not. Update the callers of callout_init() to reflect the new interface. Okayed by: Jake
OpenPOWER on IntegriCloud