summaryrefslogtreecommitdiffstats
path: root/sys/kern/kern_umtx.c
Commit message (Collapse)AuthorAgeFilesLines
* MFC r280792vangyzen2015-10-021-14/+10
| | | | | | | | | | | | | | | | Clean up some cosmetic nits in kern_umtx.c, found during recent work in this area and by the Clang static analyzer. Remove some dead assignments. Fix a typo in a panic string. Use umtx_pi_disown() instead of duplicate code. Use an existing variable instead of curthread. Approved by: kib (mentor until recently) Sponsored by: Dell Inc.
* MFC r279390:kib2015-03-211-31/+31
| | | | Change umtx_lock to be the sleepable mutex.
* MFC r279283:kib2015-03-041-0/+11
| | | | | When failing to claim ownership of a umtx_pi, restore the umutex owner to its previous, unowned state.
* MFC r279282:kib2015-03-041-2/+33
| | | | | When unlocking a contested PI pthread mutex, if the queue of waiters is empty, look up the umtx_pi and disown it if the current thread owns it.
* MFC r277970:kib2015-02-071-2/+46
| | | | | Check for the cycle in the chain of dependency for priority-inheritance mutexes.
* MFC r274478:kib2014-11-201-2/+1
| | | | | Fix assertion, &uc->uc_busy is never zero, the intent is to test the uc_busy value, and not its address.
* Merge the fueword(9) and casueword(9). In particular,kib2014-11-181-218/+328
| | | | | | | | | | | | | | | | | | | | | | | MFC r273783: Add fueword(9) and casueword(9) functions. MFC note: ia64 is handled like arm, with NO_FUEWORD define. MFC r273784: Replace some calls to fuword() by fueword() with proper error checking. MFC r273785: Convert kern_umtx.c to use fueword() and casueword(). MFC note: the sys__umtx_lock and sys__umtx_unlock syscalls are not converted, they are removed from HEAD, and not used. The do_sem2*() family is not yet merged to stable/10, corresponding chunk will be merged after do_sem2* are committed. MFC r273788 (by jkim): Actually install casuword(9) to fix build. MFC r273911: Add type qualifier volatile to the base (userspace) address argument of fuword(9) and suword(9).
* MFC r270345:kib2014-08-291-2/+4
| | | | | In do_lock_pi(), do not override error from umtxq_sleep_pi() when doing suspend check.
* Fix two issues with the spin loops in the umtx(2) implementation.kib2013-06-131-2/+147
| | | | | | | | | | | | | | | | | | | - When looping, check for the pending suspension. Otherwise, other usermode thread which races with the looping one, could try to prevent the process from stopping or exiting. - Add missed checks for the faults from casuword*(). The code is structured in a way which makes the loops exit if the specified address is invalid, since both fuword() and casuword() return -1 on the fault. But if the address is mapped readonly, the typical value read by fuword() is different from -1, while casuword() returns -1. Absent the checks for casuword() faults, this is interpreted as the race with other thread and causes non-interruptible spinning in the kernel. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
* sem: Restart the POSIX sem_* calls after signals with SA_RESTART set.jilles2013-04-191-1/+3
| | | | | | | | | | | | | | | | | | | | | Programs often do not expect an [EINTR] return from sem_wait() and POSIX only allows it if the signal was installed without SA_RESTART. The timeout in sem_timedwait() is absolute so it can be restarted normally. The umtx call can be invoked with a relative timeout and in that case [ERESTART] must be changed to [EINTR]. However, libc does not do this. The old POSIX semaphore implementation did this correctly (before r249566), unlike the new umtx one. It may be desirable to avoid [EINTR] completely, which matches the pthread functions and is explicitly permitted by POSIX. However, the kernel must return [EINTR] at least for signals with SA_RESTART clear, otherwise pthread cancellation will not abort a semaphore wait. In this commit, only restore the 8.x behaviour which is also permitted by POSIX. Discussed with: jhb MFC after: 1 week
* Fix a bug in UMTX_PROFILING:attilio2013-03-211-11/+11
| | | | | | | | | | | | | | | | | | | | UMTX_PROFILING should really analyze the distribution of locks as they index entries in the umtxq_chains hash-table. However, the current implementation does add/dec the length counters for *every* thread insert/removal, measuring at all really userland contention and not the hash distribution. Fix this by correctly add/dec the length counters in the points where it is really needed. Please note that this bug brought us questioning in the past the quality of the umtx hash table distribution. To date with all the benchmarks I could try I was not able to reproduce any issue about the hash distribution on umtx. Sponsored by: EMC / Isilon storage division Reviewed by: jeff, davide MFC after: 2 weeks
* Improve UMTX_PROFILING:attilio2013-03-091-2/+119
| | | | | | | | | | | | | | - Use u_int values for length and max_length values - Add a way to reset the max_length heuristic in order to have the possibility to reuse the mechanism consecutively without rebooting the machine - Add a way to quick display top5 contented buckets in the system for the max_length value. This should give a quick overview on the quality of the hash table distribution. Sponsored by: EMC / Isilon storage division Reviewed by: jeff, davide
* The fields of struct timespec32 should be int32_t and not uint32_t.davide2012-10-271-2/+2
| | | | | | | | Make this change. Reviewed by: bde, davidxu Tested by: pho MFC after: 1 week
* Some style fixes inspired by @bde.davidxu2012-08-111-12/+12
|
* tvtohz will print out an error message if a negative value is givendavidxu2012-08-111-9/+13
| | | | | | to it, avoid this problem by detecting timeout earlier. Reported by: pho
* Fix some style bugs introduced in a previous commit (r233045)davide2012-04-141-8/+8
| | | | | | | Reported by: glebius, jmallet Reviewed by: jmallet Approved by: gnn (mentor) MFC after: 2 days
* In sem_post, the field _has_waiters is no longer used, because somedavidxu2012-04-051-9/+14
| | | | | | | application destroys semaphore after sem_wait returns. Just enter kernel to wake up sleeping threads, only update _has_waiters if it is safe. While here, check if the value exceed SEM_VALUE_MAX and return EOVERFLOW if this is true.
* umtx operation UMTX_OP_MUTEX_WAKE has a side-effect that it accessesdavidxu2012-04-051-2/+82
| | | | | | | | | | | | | | | | | | | | | a mutex after a thread has unlocked it, it event writes data to the mutex memory to clear contention bit, there is a race that other threads can lock it and unlock it, then destroy it, so it should not write data to the mutex memory if there isn't any waiter. The new operation UMTX_OP_MUTEX_WAKE2 try to fix the problem. It requires thread library to clear the lock word entirely, then call the WAKE2 operation to check if there is any waiter in kernel, and try to wake up a thread, if necessary, the contention bit is set again by the operation. This also mitgates the chance that other threads find the contention bit and try to enter kernel to compete with each other to wake up sleeping thread, this is unnecessary. With this change, the mutex owner is no longer holding the mutex until it reaches a point where kernel umtx queue is locked, it releases the mutex as soon as possible. Performance is improved when the mutex is contensted heavily. On Intel i3-2310M, the runtime of a benchmark program is reduced from 26.87 seconds to 2.39 seconds, it even is better than UMTX_OP_MUTEX_WAKE which is deprecated now. http://people.freebsd.org/~davidxu/bench/mutex_perf.c
* Remove stale comments.davidxu2012-03-311-3/+0
|
* Remove trailing semicolon, it is a typo.davidxu2012-03-301-1/+1
|
* Fix COMPAT_FREEBSD32 build.davidxu2012-03-301-3/+3
| | | | Submitted by: Andreas Tobler < andreast at fgznet dot ch >
* Remove trailing space.davidxu2012-03-301-1/+1
|
* Merge umtxq_sleep and umtxq_nanosleep into a single function by usingdavidxu2012-03-301-264/+172
| | | | an abs_timeout structure which describes timeout info.
* Reduce code size by creating common timed sleeping function.davidxu2012-03-291-93/+60
|
* Add rudimentary profiling of the hash table used in the in the umtx code todavide2012-03-161-0/+51
| | | | | | | | hold active lock queues. Reviewed by: attilio Approved by: davidxu, gnn (mentor) MFC after: 3 weeks
* initialize clock ID and flags only when copying timespec, a _umtx_timedavidxu2012-02-291-4/+4
| | | | copy already contains these fields.
* Follow changes made in revision 232144, pass absolute timeout to kernel,davidxu2012-02-271-41/+47
| | | | this eliminates a clock_gettime() syscall.
* In revision 231989, we pass a 16-bit clock ID into kernel, howeverdavidxu2012-02-251-98/+154
| | | | | | | | | | | | according to POSIX document, the clock ID may be dynamically allocated, it unlikely will be in 64K forever. To make it future compatible, we pack all timeout information into a new structure called _umtx_time, and use fourth argument as a size indication, a zero means it is old code using timespec as timeout value, but the new structure also includes flags and a clock ID, so the size argument is different than before, and it is non-zero. With this change, it is possible that a thread can sleep on any supported clock, though current kernel code does not have such a POSIX clock driver system.
* Fix typo.davidxu2012-02-221-1/+1
|
* Use unused fourth argument of umtx_op to pass flags to kernel for operationdavidxu2012-02-221-21/+42
| | | | | | UMTX_OP_WAIT. Upper 16bits is enough to hold a clock id, and lower 16bits is used to pass flags. The change saves a clock_gettime() syscall from libthr.
* Eliminate branch and insert an explicit reader memory barrier to ensuredavidxu2012-01-161-3/+2
| | | | that waiter bit is set before reading semaphore count.
* Add umtx_copyin_timeout() and move parameter checks here.pho2011-12-031-53/+25
| | | | | In collaboration with: kib MFC after: 1 week
* Rename copyin_timeout32 to umtx_copyin_timeout32 and move parameterpho2011-12-031-42/+18
| | | | | | | check here. Include check for negative seconds value. In collaboration with: kib MFC after: 1 week
* Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs.ed2011-11-071-1/+1
| | | | | | The SYSCTL_NODE macro defines a list that stores all child-elements of that node. If there's no SYSCTL_DECL macro anywhere else, there's no reason why it shouldn't be static.
* In order to maximize the re-usability of kernel code in user space thiskmacy2011-09-161-3/+3
| | | | | | | | | | | | | patch modifies makesyscalls.sh to prefix all of the non-compatibility calls (e.g. not linux_, freebsd32_) with sys_ and updates the kernel entry points and all places in the code that use them. It also fixes an additional name space collision between the kernel function psignal and the libc function of the same name by renaming the kernel psignal kern_psignal(). By introducing this change now we will ease future MFCs that change syscalls. Reviewed by: rwatson Approved by: re (bz)
* Expose the umtx_key structure and API to the rest of the kernel.jhb2011-02-231-50/+2
| | | | MFC after: 3 days
* - Follow r216313, the sched_unlend_user_prio is no longer needed, alwaysdavidxu2010-12-291-43/+18
| | | | | | | use sched_lend_user_prio to set lent priority. - Improve pthread priority-inherit mutex, when a contender's priority is lowered, repropagete priorities, this may cause mutex owner's priority to be lowerd, in old code, mutex owner's priority is rise-only.
* Enlarge hash table for new condition variable.davidxu2010-12-231-2/+2
|
* MFp4:davidxu2010-12-221-15/+105
| | | | | | | | | | | | | | | - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
* One of the compat32 functions was copying in a raw timespec, instead ofmdf2010-12-151-2/+1
| | | | | | | | a 32-bit one. This can cause weird timeout issues, as the copying reads garbage from the user. Code by: Deepak Veliath <deepak dot veliath at isilon dot com> MFC after: 1 week
* MFp4:davidxu2010-12-091-7/+9
| | | | | | | | | It is possible a lower priority thread lending priority to higher priority thread, in old code, it is ignored, however the lending should always be recorded, add field td_lend_user_pri to fix the problem, if a thread does not have borrowed priority, its value is PRI_MAX. MFC after: 1 week
* Use atomic instruction to set _has_writer, otherwise there is a racedavidxu2010-11-221-1/+2
| | | | | | causes userland to not wake up a thread sleeping in kernel. MFC after: 3 days
* Only unlock process if a thread is found.davidxu2010-11-151-4/+4
|
* Create a global thread hash table to speed up thread lookup, usedavidxu2010-10-091-8/+3
| | | | | | | | | | rwlock to protect the table. In old code, thread lookup is done with process lock held, to find a thread, kernel has to iterate through process and thread list, this is quite inefficient. With this change, test shows in extreme case performance is dramatically improved. Earlier patch was reviewed by: jhb, julian
* If a thread is removed from umtxq while sleeping, reset error codedavidxu2010-08-251-23/+12
| | | | | to zero, this gives userland a better indication that a thread needn't to be cancelled.
* Use ISO C99 integer types in sys/kern where possible.ed2010-06-211-2/+2
| | | | | | There are only about 100 occurences of the BSD-specific u_int*_t datatypes in sys/kern. The ISO C99 integer types are used here more often.
* Provide groundwork for 32-bit binary compatibility on non-x86 platforms,nwhitehorn2010-03-111-3/+3
| | | | | | | | | for upcoming 64-bit PowerPC and MIPS support. This renames the COMPAT_IA32 option to COMPAT_FREEBSD32, removes some IA32-specific code from MI parts of the kernel and enhances the freebsd32 compatibility code to support big-endian platforms. Reviewed by: kib, jhb
* In function umtxq_insert_queue, use parameter q (shared/exclusive queue)davidxu2010-02-101-1/+1
| | | | | instead of hard coded constant. This does not affect RELENG_8 and previous, because the code only exists in the HEAD.
* Set waiters flag before checking semaphore's counter,davidxu2010-02-081-5/+2
| | | | otherwise we might lose a wakeup. Tested on postgresql database server.
* Fix comments in do_sem_wait().davidxu2010-02-031-2/+1
|
OpenPOWER on IntegriCloud