summaryrefslogtreecommitdiffstats
path: root/sys/amd64/include/atomic.h
Commit message (Collapse)AuthorAgeFilesLines
* MFC r311169,r311898,r312925,r312973,r312975,r313007,r313040,r313080,mjg2017-03-161-0/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | r313254,r313341 amd64: add atomic_fcmpset == sparc64: add atomic_fcmpset == Implement atomic_fcmpset_* for arm and arm64. == Add atomic_fcmpset_*() inlines for powerpc Summary: atomic_fcmpset_*() is analogous to atomic_cmpset(), but saves off the read value from the target memory location into the 'old' pointer in the case of failure. == i386: add atomic_fcmpset == Don't retry a lost reservation in atomic_fcmpset() The desired behavior of atomic_fcmpset_() is to always exit on error. Instead of retrying on lost reservation, leave the retry to the caller, and return == Add atomic_fcmpset_*() inlines for MIPS atomic_fcmpset_*() is analogous to atomic_cmpset(), but saves off the read value from the target memory location into the 'old' pointer. == i386: fixup fcmpset An incorrect output specifier was used which worked with clang by accident, but breaks with the in-tree gcc version. While here plug a whitespace nit. == Implement atomic_fcmpset_*() for RISC-V. == Use 64bit store instruction in atomic_fcmpset_64.
* atomic: Add testandclear on i386/amd64sephe2016-05-161-0/+38
| | | | | | Reviewed by: kib Sponsored by: Microsoft OSTC Differential Revision: https://reviews.freebsd.org/D6381
* Add missing atomic wrapper macro.hselasky2016-01-211-0/+1
| | | | | | Reviewed by: alfred @ Sponsored by: Mellanox Technologies MFC after: 1 week
* Improve comments.kib2015-07-301-4/+4
| | | | | Submitted by: bde MFC after: 2 weeks
* Remove full barrier from the amd64 atomic_load_acq_*(). Strongkib2015-07-281-17/+7
| | | | | | | | | | | | | | | | | ordering semantic of x86 CPUs makes only the compiler barrier neccessary to give the acquire behaviour. Existing implementation ensured sequentially consistent semantic for load_acq, making much stronger guarantee than required by standard's definition of the load acquire. Consumers which depend on the barrier are believed to be identified and already fixed to use proper operations. Noted by: alc (long time ago) Reviewed by: alc, bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
* Add a comment discussing the appropriate use of the atomic_*() functionsalc2015-07-241-0/+19
| | | | | | | | with acquire and release semantics versus the *mb() functions on amd64 processors. Reviewed by: bde (an earlier version), kib Sponsored by: EMC / Isilon Storage Division
* Add the atomic_thread_fence() family of functions with intent tokib2015-07-081-0/+32
| | | | | | | | | | | | | | | | | | | | | | | provide a semantic defined by the C11 fences with corresponding memory_order. atomic_thread_fence_acq() gives r | r, w, where r and w are read and write accesses, and | denotes the fence itself. atomic_thread_fence_rel() is r, w | w. atomic_thread_fence_acq_rel() is the combination of the acquire and release in single operation. Note that reads after the acq+rel fence could be made visible before writes preceeding the fence. atomic_thread_fence_seq_cst() orders all accesses before/after the fence, and the fence itself is globally ordered against other sequentially consistent atomic operations. Reviewed by: alc Discussed with: bde Sponsored by: The FreeBSD Foundation MFC after: 3 weeks
* Add a comment about too strong semantic of atomic_load_acq() on x86.kib2015-06-291-0/+9
| | | | | Submitted by: bde MFC after: 2 weeks
* Remove unneeded data dependency, currently imposed bykib2015-06-281-50/+75
| | | | | | | | | | | | | | | | | | | | | | | atomic_load_acq(9), on it source, for x86. Right now, atomic_load_acq() on x86 is sequentially consistent with other atomics, code ensures this by doing store/load barrier by performing locked nop on the source. Provide separate primitive __storeload_barrier(), which is implemented as the locked nop done on a cpu-private variable, and put __storeload_barrier() before load, to keep seq_cst semantic but avoid introducing false dependency on the no-modification of the source for its later use. Note that seq_cst property of x86 atomic_load_acq() is not documented and not carried by atomics implementations on other architectures, although some kernel code relies on the behaviour. This commit does not intend to change this. Reviewed by: alc Discussed with: bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks
* Remove empty lines before return statements for style consistency.jkim2013-08-211-5/+0
|
* Implement atomic_swap() and atomic_testandset().jkim2013-08-211-15/+57
| | | | Reviewed by: arch, bde, jilles, kib
* - Remove the "a" constraint from main output operand for atomic_cmpset().jkim2013-08-211-10/+10
| | | | - Use "+" modifier for the "expect" because it is also an output (unused).
* Use '+' modifier for a memory operand that is both an input and an output.jkim2013-08-211-23/+16
| | | | | | It was actually done in r86301 but reverted in r150182 because GCC 3.x was not able to handle it for a memory operand. Apparently, this problem was fixed in GCC 4.1+ and several contrib sources already rely on this feature.
* Remove bogus labels. No functional change.jkim2013-08-211-2/+0
|
* Use consistent style. No functional change.jkim2013-08-211-12/+12
|
* Add an unified macro to deny ability from the compiler to reorderattilio2012-10-091-2/+2
| | | | | | | | | | instruction loads/stores at its will. The macro __compiler_membar() is currently supported for both gcc and clang, but kernel compilation will fail otherwise. Reviewed by: bde, kib Discussed with: dim, theraven MFC after: 2 weeks
* Use plain store for atomic_store_rel on x86, instead of implicitlykib2012-06-021-37/+37
| | | | | | | | | locked xchg instruction. IA32 memory model guarantees that store has release semantic, since stores cannot pass loads or stores. Reviewed by: bde, jhb Tested by: pho MFC after: 2 weeks
* Inform a compiler which asm statements in the x86 implementation ofkib2010-12-181-9/+10
| | | | | | | atomics change eflags. Reviewed by: jhb MFC after: 2 weeks
* Rename an argument from "exp" to "expect" since the former makes FlexeLintphk2010-05-201-7/+7
| | | | | | uneasy, in case anybody think it might be exp(3) in libm. This also makes it consistent with other archs.
* atomic_cmpset_barr_* was added in order to cope with compilers willing toattilio2009-10-091-32/+44
| | | | | | | | | | | | | | | | | specify their own version of atomic_cmpset_* which could have been different than the membar version. Right now, however, FreeBSD is bound mostly to GCC-like compilers and it is desired to add new support and compat shim mostly when there is a real necessity, in order to avoid too much compatibility bloats. In this optic, bring back atomic_cmpset_{acq, rel}_* to be the same as atomic_cmpset_* and unwind the atomic_cmpset_barr_* introduction. Requested by: jhb Reviewed by: jhb Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com>
* - All the functions in atomic.h needs to be in "physical" form (likeattilio2009-10-061-46/+29
| | | | | | | | | | | | not defined through macros or similar) in order to be later compiled in the kernel and offer this way the support for modules (and compatibility among the UP case and SMP case). Fix this for the newly introduced atomic_cmpset_barr_* cases by defining and specifying a template. Note that the new DEFINE_CMPSET_GEN() template save more typing on amd64 than the current code. [1] - Fix the style for memory barriers on amd64. [1] Reported by: Paul B. Mahol <onemda at gmail dot com>
* Per their definition, atomic instructions used in conjuction withattilio2009-10-061-46/+67
| | | | | | | | | | | | | | | | | | | | | | | | memory barriers should also ensure that the compiler doesn't reorder paths where they are used. GCC, however, does that aggressively, even in presence of volatile operands. The most reliable way GCC offers for avoid instructions reordering is clobbering "memory" even if that is theoretically an heavy-weight operation, flushing the content of all the registers and forcing reload of them (We could rely, however, on gcc DTRT by just understanding the purpose as this is a well-known pattern for many modern operating-systems). Not all our memory barriers, right now, clobber memory for GCC-like compilers. The most notable cases are IA32 and amd64 where the memory barrier are treacted the same as normal atomic instructions. Fix this by offering the possibility to implement atomic instructions with memory barriers separately from the normal version and implement the GCC-like specific one using memory clobbering. Thanks to Chris Lattner (@apple) for his discussion on llvm specifics. Reported by: jhb Reviewed by: jhb Tested by: rdivacky, Giovanni Trematerra <giovanni dot trematerra at gmail dot com>
* - bump __FreeBSD version to reflect added buf_ring, memory barriers,kmacy2008-11-221-0/+4
| | | | | | | | | | | | | | | | | and ifnet functions - add memory barriers to <machine/atomic.h> - update drivers to only conditionally define their own - add lockless producer / consumer ring buffer - remove ring buffer implementation from cxgb and update its callers - add if_transmit(struct ifnet *ifp, struct mbuf *m) to ifnet to allow drivers to efficiently manage multiple hardware queues (i.e. not serialize all packets through one ifq) - expose if_qflush to allow drivers to flush any driver managed queues This work was supported by Bitgravity Inc. and Chelsio Inc.
* Implement atomic_fetchadd_long() for all architectures and document it.pjd2008-03-161-0/+20
| | | | Reviewed by: attilio, jhb, jeff, kris (as a part of the uidinfo_waitfree.patch)
* Fixed some style bugs (mainly assorted errors in comments, and inconsistentbde2006-12-291-23/+23
| | | | spelling of `result').
* Fixed some style bugs (whitespace only).bde2006-12-291-30/+31
|
* Try harder to garbage-collect the "LOCORE" (really asm) version ofbde2006-12-291-6/+6
| | | | | | | | | | MPLOCKED. The cleaning in rev.1.25 was supposed to have been undone by rev.1.26, but 1.26 could never have actually affected asm files since atomic.h is full of C declarations so including it in asm files would just give syntax errors. The asm MPLOCKED is even less needed than when misplaced definitions of it were first removed, and is now unused in any asm file in the src tree except in anachronismns in sys/i386/i386/support.s.
* Removed gratuitous cosmetic differences with the i386 version. Thisbde2006-12-281-7/+7
| | | | | | | | | | | mainly involves removing all __CC_SUPPORTS___INLINE__ ifdefs. These ifdefs are even less needed for amd64 than for i386, but the i386 atomic.h never had them. The ifdefs here were just an optimization of obsolescent compatibility cruft (__inline) for a null set of compilers. I think null sets of compilers should only be supported in cases where this is more than an optimization, doesn't require extensive ifdefs, and only involves not-so-obsolescent compatibility cruft (plain inline here).
* Avoid an instruction in atomic_cmpset_{int_long)() in most cases.bde2006-12-271-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These functions are used a lot for mutexes, so this reduces the text size of an average kernel by about 0.75%. This wasn't intended to be a significant optimization, but it somehow increased the maximum number of packets per second that can be transmitted by my bge hardware from 320000 to 460000 (this benchmark is CPU-bound and remarkably sensitive to changes in the text section). Details: we would prefer to leave the result of the cmpxchg in %al, but cannot tell gcc that it is there, so we have to convert it to an integer register. We converted to %al, then to %[re]ax, but the latter step is usually wasted since gcc usually only wants the condition code and can recover it from %al just as easily as from %[re]ax. Let gcc promote %al in the few cases where this is needed. Nearby style fixes; - let gcc manage the load of `res', and don't abuse `res' for a copy of `exp' - don't echo `res's name in comments - consistently spell the condition code as 'e' after comparison for equality - don't hard-code %al anywhere except in constraints - for the version that doesn't use cmpxchg, there is no requirement to use %al anywhere, so don't hard-code it in the constraints either. Style non-fix: - for the versions that use cmpxchg, keep using "a" (was %[re]ax, now %al) for the main output operand, although this is not required. The input and output operands that use the "a" constraint are now decoupled, and this makes things clearer except for the reason that the output register is hard-coded. It is now just a hack to tell gcc that the input "a" has been clobbered without increasing the number of operands.
* Add a new atomic_fetchadd() primitive that atomically adds a value to ajhb2005-09-271-0/+21
| | | | | | | | | variable and returns the previous value of the variable. Tested on: i386, alpha, sparc64, arm (cognet) Reviewed by: arch@ Submitted by: cognet (arm) MFC after: 1 week
* Stop using the '+' constraint modifier with inline assembly. The '+'jhb2005-09-151-22/+27
| | | | | | | | | | constraint is actually only allowed for register operands. Instead, use separate input and output memory constraints. Education from: alc Reviewed by: alc Tested on: i386, alpha MFC after: 1 week
* Add aliases for atomic operations on 64-bit integers just like otherjhb2005-08-181-0/+20
| | | | | | 64-bit platforms. MFC after: 1 week
* Like on i386, bypass lock prefix for atomic ops on !SMP kernels.peter2005-07-211-0/+27
|
* Convert the atomic_ptr() operations over to operating on uintptr_tjhb2005-07-151-52/+18
| | | | | | | | | | variables rather than void * variables. This makes it easier and simpler to get asm constraints and volatile keywords correct. MFC after: 3 days Tested on: i386, alpha, sparc64 Compiled on: ia64, powerpc, amd64 Kernel toolchain busted on: arm
* Some cleanups and tweaks to some of the atomic.h files in preparation forjhb2005-07-091-79/+63
| | | | | | | | | | | | | | | further changes and fixes in the future: - Use aliases via macros rather than duplicated inlines wherever possible. - Move all the aliases to the bottom of these files and the inline functions to the top. - Add various comments. - On alpha, drop atomic_{load_acq,store_rel}_{8,char,16,short}(). - On i386 and amd64, don't duplicate the extern declarations for functions in the two non-inline cases (KLD_MODULE and compiler doesn't do inlines), instead, consolidate those two cases. - Some whitespace fixes. Approved by: re (scottl)
* netchild's mega-patch to isolate compiler dependencies into a centraljoerg2005-03-021-11/+15
| | | | | | | | | | | | | | | | place. This moves the dependency on GCC's and other compiler's features into the central sys/cdefs.h file, while the individual source files can then refer to #ifdef __COMPILER_FEATURE_FOO where they by now used to refer to #if __GNUC__ > 3.1415 && __BARC__ <= 42. By now, GCC and ICC (the Intel compiler) have been actively tested on IA32 platforms by netchild. Extension to other compilers is supposed to be possible, of course. Submitted by: netchild Reviewed by: various developers on arch@, some time ago
* Cosmetic and/or trivial sync up with i386.peter2003-11-211-4/+10
| | | | Approved by: re (rwatson)
* Initial landing of SMP support for FreeBSD/amd64.peter2003-11-171-1/+1
| | | | | | | | | | | | | | | | - This is heavily derived from John Baldwin's apic/pci cleanup on i386. - I have completely rewritten or drastically cleaned up some other parts. (in particular, bootstrap) - This is still a WIP. It seems that there are some highly bogus bioses on nVidia nForce3-150 boards. I can't stress how broken these boards are. I have a workaround in mind, but right now the Asus SK8N is broken. The Gigabyte K8NPro (nVidia based) is also mind-numbingly hosed. - Most of my testing has been with SCHED_ULE. SCHED_4BSD works. - the apic and acpi components are 'standard'. - If you have an nVidia nForce3-150 board, you are stuck with 'device atpic' in addition, because they somehow managed to forget to connect the 8254 timer to the apic, even though its in the same silicon! ARGH! This directly violates the ACPI spec.
* Commit MD parts of a loosely functional AMD64 port. This is based onpeter2003-05-011-67/+29
| | | | | | | | | | | | | | | | | | | | | | a heavily stripped down FreeBSD/i386 (brutally stripped down actually) to attempt to get a stable base to start from. There is a lot missing still. Worth noting: - The kernel runs at 1GB in order to cheat with the pmap code. pmap uses a variation of the PAE code in order to avoid having to worry about 4 levels of page tables yet. - It boots in 64 bit "long mode" with a tiny trampoline embedded in the i386 loader. This simplifies locore.s greatly. - There are still quite a few fragments of i386-specific code that have not been translated yet, and some that I cheated and wrote dumb C versions of (bcopy etc). - It has both int 0x80 for syscalls (but using registers for argument passing, as is native on the amd64 ABI), and the 'syscall' instruction for syscalls. int 0x80 preserves all registers, 'syscall' does not. - I have tried to minimize looking at the NetBSD code, except in a couple of places (eg: to find which register they use to replace the trashed %rcx register in the syscall instruction). As a result, there is not a lot of similarity. I did look at NetBSD a few times while debugging to get some ideas about what I might have done wrong in my first attempt.
* Add a knob to turn on and off the CMPXCHG instruction on > i386 IA32 systems.pirzyk2002-10-141-1/+1
| | | | | | | This is most beneficial for vmware client os installs. Reviewed by: jmallet, iedowse, tlambert2@mindspring.com MFC After: never, -STABLE does not currently use this instruction
* Beautify. This has the side effect of improving portability andmarkm2002-07-181-39/+31
| | | | | | making lint work cleaner. Inspired to do this by: jhb
* Clean up the syntax WRT semicolons at the end of function-like-macros, and ↵markm2002-07-171-23/+74
| | | | protect GCCisms from non-GNU compilers and lint.
* Make MPLOCKED work again in asm files and stringify it explicitlybmilekic2002-02-281-4/+4
| | | | | | where necessary. Reviewed by: jake
* Garbage-collect the "LOCORE" version of MPLOCKED.bde2002-02-111-10/+2
|
* Allow the ATOMIC_ASM() macro to pass in the constraints on the V parameterjhb2001-12-181-23/+23
| | | | | | | since the char versions need to use either ax, bx, cx, or dx. Submitted by: Peter Jeremy (mostly) Recommended by: bde
* Use newer constraints for atomic_cmpset().jhb2001-11-121-11/+9
| | | | Requested by: bde
* Use newer constraints for inline assembly for an operand that is both anjhb2001-11-121-21/+21
| | | | | | | input and an output by using the '+' modifier rather than listing the operand in both the input and output sections. Reviwed by: bde
* Allow atomic ops to be somewhat safely used in userland. We always usejhb2001-10-081-3/+7
| | | | | lock prefixes in the userland case so that the binaries will work on both SMP and UP systems.
* RIP <machine/lock.h>.markm2001-02-111-4/+9
| | | | | | | Some things needed bits of <i386/include/lock.h> - cy.c now has its own (only) copy of the COM_(UN)LOCK() macros, and IMASK_(UN)LOCK() has been moved to <i386/include/apic.h> (AKA <machine/apic.h>). Reviewed by: jhb
* - Sort of lie and say that %eax is an output only and not an input for thejhb2001-01-171-2/+2
| | | | | | | | | | | non-386 atomic_load_acq(). %eax is an input since its value is used in the cmpxchg instruction, but we don't care what value it is, so setting it to a specific value is just wasteful. Thus, it is being used without being initialized as the warning stated, but it is ok for it to be used because its value isn't important. Thus, we are only sort of lying when we say it is an output only operand. - Add "cc" to the clobber list for atomic_load_acq() since the cmpxchgl changes ZF.
OpenPOWER on IntegriCloud