summaryrefslogtreecommitdiffstats
path: root/sys/amd64/include/cpufunc.h
Commit message (Collapse)AuthorAgeFilesLines
* MFia64:ps2004-07-301-17/+1
| | | | | Fix -O builds with gcc 3.4 by defining ffs as __builtin_ffs instead of creating an inline function that just calls __builtin_ffs.
* MFi386: move rss() from db_interface.c to cpufunc.hpeter2004-04-071-0/+8
|
* Remove advertising clause from University of California Regent's license,imp2004-04-051-4/+0
| | | | | | per letter dated July 22, 1999 and email from Peter Wemm. Approved by: core, peter
* Don't implement anything in the ffs family in <machine/cpufunc.h>bde2004-03-111-0/+4
| | | | | | | | | | | | | | | in the non-_KERNEL case. This "fixes" applications that include this "kernel-only" header and also include <strings.h> (or get <strings.h> via the default _BSD_VISIBLE pollution in <string.h>. In C++ there was a fatal error: the declaration specifies C linkage but the implementation gives C++ linkage. In C there was only a static/extern mismatch if the headers were included in a certain order order, and a partially redundant declaration for all include orders; gcc emits incomplete or wrong diagnostics for these, but only for compiling with -Wsystem-headers and certain other warning options, so the problem was usually not seen for C. Ports breakage reported by: kris
* MFi386: re-sort non-gcc function prototypes, trim includespeter2004-03-081-44/+30
|
* Fix syntax errors and wrong function prototypes in several MD headerle2004-03-051-3/+3
| | | | | | | | files when using non-GNUC compilers. PR: kern/58515 Submitted by: Stefan Farfeleder <stefan@fafoe.narf.at> Approved by: grog (mentor), obrien
* Re-add debug register functionspeter2004-01-281-2/+129
|
* Add 64 bit bsf*/ffs* routines. Have the ffs() inline use gcc's builtinpeter2003-12-061-1/+40
| | | | because it uses the better cmove instructions to avoid branches.
* Update the graffiti.peter2003-11-081-0/+1
|
* Collect the nastiness for preserving the kernel MSR_GSBASE around thepeter2003-05-151-0/+36
| | | | | | | | | | load_gs() calls into a single place that is less likely to go wrong. Eliminate the per-process context switching of MSR_GSBASE, because it should be constant for a single cpu. Instead, save/restore it during the loading of the new %gs selector for the new process. Approved by: re (amd64/* blanket)
* Add BASIC i386 binary support for the amd64 kernel. This is largelypeter2003-05-141-1/+12
| | | | | | | | | | | | | | | | | | | | | | stolen from the ia64/ia32 code (indeed there was a repocopy), but I've redone the MD parts and added and fixed a few essential syscalls. It is sufficient to run i386 binaries like /bin/ls, /usr/bin/id (dynamic) and p4. The ia64 code has not implemented signal delivery, so I had to do that. Before you say it, yes, this does need to go in a common place. But we're in a freeze at the moment and I didn't want to risk breaking ia64. I will sort this out after the freeze so that the common code is in a common place. On the AMD64 side, this required adding segment selector context switch support and some other support infrastructure. The %fs/%gs etc code is hairy because loading %gs will clobber the kernel's current MSR_GSBASE setting. The segment selectors are not used by the kernel, so they're only changed at context switch time or when changing modes. This still needs to be optimized. Approved by: re (amd64/* blanket)
* Commit MD parts of a loosely functional AMD64 port. This is based onpeter2003-05-011-173/+53
| | | | | | | | | | | | | | | | | | | | | | a heavily stripped down FreeBSD/i386 (brutally stripped down actually) to attempt to get a stable base to start from. There is a lot missing still. Worth noting: - The kernel runs at 1GB in order to cheat with the pmap code. pmap uses a variation of the PAE code in order to avoid having to worry about 4 levels of page tables yet. - It boots in 64 bit "long mode" with a tiny trampoline embedded in the i386 loader. This simplifies locore.s greatly. - There are still quite a few fragments of i386-specific code that have not been translated yet, and some that I cheated and wrote dumb C versions of (bcopy etc). - It has both int 0x80 for syscalls (but using registers for argument passing, as is native on the amd64 ABI), and the 'syscall' instruction for syscalls. int 0x80 preserves all registers, 'syscall' does not. - I have tried to minimize looking at the NetBSD code, except in a couple of places (eg: to find which register they use to replace the trashed %rcx register in the syscall instruction). As a result, there is not a lot of similarity. I did look at NetBSD a few times while debugging to get some ideas about what I might have done wrong in my first attempt.
* Backout my last commit.davidxu2003-04-201-4/+4
| | | | Requested by: bde
* Don't return garbage in high 16 bits.davidxu2003-04-191-4/+4
|
* Create inlines for ltr(sel), lldt(sel), lidt(addr) rather thanpeter2002-09-221-1/+26
| | | | functions that have one instruction.
* Provide in inline function for the (GNUC) assembler "hlt" instruction.markm2002-09-211-0/+7
|
* Move SWTCH_OPTIM_STATS related code out of cpufunc.h. (This sort of statpeter2002-07-211-7/+0
| | | | gathering is not an x86 cpu feature)
* Cast to prevent "signed/unsigned comparison" warnings.markm2002-07-151-2/+2
|
* Revive backed out pmap related changes from Feb 2002. The highlights are:peter2002-07-121-75/+93
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - It actually works this time, honest! - Fine grained TLB shootdowns for SMP on i386. IPI's are very expensive, so try and optimize things where possible. - Introduce ranged shootdowns that can be done as a single IPI. - PG_G support for i386 - Specific-cpu targeted shootdowns. For example, there is no sense in globally purging the TLB cache for where we are stealing a page from the local unshared process on the local cpu. Use pm_active to track this. - Add some instrumentation for the tlb shootdown code. - Rip out SMP code from <machine/cpufunc.h> - Try and fix some very bogus PG_G and PG_PS interactions that were bad enough to cause vm86 bios calls to break. vm86 depended on our existing bugs and this was the cause of the VESA panics last time. - Fix the silly one-line error that caused the 'panic: bad pte' last time. - Fix a couple of other silly one-line errors that should have caused more pain than they did. Some more work is needed: - pmap_{zero,copy}_page[_idle]. These can be done without IPI's if we have a hook in cpu_switch. - The IPI handlers need some cleanup. I have a bogus %ds load that can be avoided. - APTD handling is rather bogus and appears to be a large source of global TLB IPI shootdowns for no really good reason. I see speedups of between 1.5% and ~4% on buildworlds in a while 1 loop. I expect to see a bigger difference when there is significant pageout activity or the system otherwise has memory shortages. I have backed out a few optimizations that I had been using over the last few days in order to be a little more conservative. I'll revisit these again over the next few days as the dust settles. New option: DISABLE_PG_G - In case I missed something.
* Rename pause() to ia32_pause() so it doesn't conflict with the pause()jhb2002-05-221-2/+2
| | | | | function defined in <unistd.h>. I didn't #ifdef _KERNEL it because the mutex implementation in libpthread will probably need this.
* Debug registers aren't selectors, so use saner names for the variables injhb2002-05-221-24/+24
| | | | the inline functions for reading and writing the debug registers.
* - Sort the pause() inline into the appropriate location.jhb2002-05-221-6/+25
| | | | - Add many missing prototypes to the non-GCC section.
* Rename cpu_pause() to pause(). Originally I was going to make this anjhb2002-05-221-1/+1
| | | | | | | | | MI API with empty cpu_pause() functions on other arch's, but this functionality is definitely unique to IA-32, so I decided to leave it as i386-only and wrap it in #ifdef's. I should have dropped the cpu_ prefix when I made that decision. Requested by: bde
* Add an inline function cpu_pause() for the IA32 'pause' instruction.jhb2002-05-211-0/+6
|
* Move do_cpuid into the correct place in this file and makedwmalone2002-04-101-10/+8
| | | | | | | | the indentation more like the other multi-line assembley in this file. Someone who understands gcc constraints could update the constraints for do_cpuid.
* Stage-2 commit of the critical*() code. This re-inlines cpu_critical_enter()dillon2002-04-011-4/+0
| | | | | | | | | | | | | | | | | | | | | and cpu_critical_exit() and moves associated critical prototypes into their own header file, <arch>/<arch>/critical.h, which is only included by the three MI source files that need it. Backout and re-apply improperly comitted syntactical cleanups made to files that were still under active development. Backout improperly comitted program structure changes that moved localized declarations to the top of two procedures. Partially re-apply one of the program structure changes to move 'mask' into an intermediate block rather then in three separate sub-blocks to make the code more readable. Re-integrate bug fixes that Jake made to the sparc64 code. Note: In general, developers should not gratuitously move declarations out of sub-blocks. They are where they are for reasons of structure, grouping, readability, compiler-localizability, and to avoid developer-introduced bugs similar to several found in recent years in the VFS and VM code. Reviewed by: jake
* Compromise for critical*()/cpu_critical*() recommit. Cleanup the interruptdillon2002-03-271-20/+7
| | | | | | | | | | | | | | | | | | | disablement assumptions in kern_fork.c by adding another API call, cpu_critical_fork_exit(). Cleanup the td_savecrit field by moving it from MI to MD. Temporarily move cpu_critical*() from <arch>/include/cpufunc.h to <arch>/<arch>/critical.c (stage-2 will clean this up). Implement interrupt deferral for i386 that allows interrupts to remain enabled inside critical sections. This also fixes an IPI interlock bug, and requires uses of icu_lock to be enclosed in a true interrupt disablement. This is the stage-1 commit. Stage-2 will occur after stage-1 has stabilized, and will move cpu_critical*() into its own header file(s) + other things. This commit may break non-i386 architectures in trivial ways. This should be temporary. Reviewed by: core Approved by: core
* Fixed some style bugs in the removal of __P(()). The main ones werebde2002-03-231-2/+2
| | | | | | not removing tabs before "__P((", and not outdenting continuation lines to preserve non-KNF lining up of code with parentheses. Switch to KNF formatting and/or rewrap the whole prototype in some cases.
* Fix abuses of cpu_critical_{enter,exit} by converting toimp2002-03-211-0/+16
| | | | | | | intr_{disable,restore} as well as providing an implemenation of intr_{disable,restore}. Reviewed by: jake, rwatson, jhb
* Fix minor style(9) violation in de__Pingimp2002-03-201-40/+40
|
* Remove __P.alfred2002-03-201-45/+45
|
* Make it a bit clearer where this file is to be used and where itmarkm2002-02-281-0/+2
| | | | | | should not be. (Comments only) Inspired by: bde
* Back out all the pmap related stuff I've touched over the last few days.peter2002-02-271-169/+75
| | | | | | There is some unresolved badness that has been eluding me, particularly affecting uniprocessor kernels. Turning off PG_G helped (which is a bad sign) but didn't solve it entirely. Userland programs still crashed.
* revert last commit temporarily due to whining on the lists.dillon2002-02-261-1/+1
|
* STAGE-1 of 3 commit - allow (but do not require) interrupts to remaindillon2002-02-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | enabled in critical sections and streamline critical_enter() and critical_exit(). This commit allows an architecture to leave interrupts enabled inside critical sections if it so wishes. Architectures that do not wish to do this are not effected by this change. This commit implements the feature for the I386 architecture and provides a sysctl, debug.critical_mode, which defaults to 1 (use the feature). For now you can turn the sysctl on and off at any time in order to test the architectural changes or track down bugs. This commit is just the first stage. Some areas of the code, specifically the MACHINE_CRITICAL_ENTER #ifdef'd code, is strictly temporary and will be cleaned up in the STAGE-2 commit when the critical_*() functions are moved entirely into MD files. The following changes have been made: * critical_enter() and critical_exit() for I386 now simply increment and decrement curthread->td_critnest. They no longer disable hard interrupts. When critical_exit() decrements the counter to 0 it effectively calls a routine to deal with whatever interrupts were deferred during the time the code was operating in a critical section. Other architectures are unaffected. * fork_exit() has been conditionalized to remove MD assumptions for the new code. Old code will still use the old MD assumptions in regards to hard interrupt disablement. In STAGE-2 this will be turned into a subroutine call into MD code rather then hardcoded in MI code. The new code places the burden of entering the critical section in the trampoline code where it belongs. * I386: interrupts are now enabled while we are in a critical section. The interrupt vector code has been adjusted to deal with the fact. If it detects that we are in a critical section it currently defers the interrupt by adding the appropriate bit to an interrupt mask. * In order to accomplish the deferral, icu_lock is required. This is i386-specific. Thus icu_lock can only be obtained by mainline i386 code while interrupts are hard disabled. This change has been made. * Because interrupts may or may not be hard disabled during a context switch, cpu_switch() can no longer simply assume that PSL_I will be in a consistent state. Therefore, it now saves and restores eflags. * FAST INTERRUPT PROVISION. Fast interrupts are currently deferred. The intention is to eventually allow them to operate either while we are in a critical section or, if we are able to restrict the use of sched_lock, while we are not holding the sched_lock. * ICU and APIC vector assembly for I386 cleaned up. The ICU code has been cleaned up to match the APIC code in regards to format and macro availability. Additionally, the code has been adjusted to deal with deferred interrupts. * Deferred interrupts use a per-cpu boolean int_pending, and masks ipending, spending, and fpending. Being per-cpu variables it is not currently necessary to lock; bus cycles modifying them. Note that the same mechanism will enable preemption to be incorporated as a true software interrupt without having to further hack up the critical nesting code. * Note: the old critical_enter() code in kern/kern_switch.c is currently #ifdef to be compatible with both the old and new methodology. In STAGE-2 it will be moved entirely to MD code. Performance issues: One of the purposes of this commit is to enhance critical section performance, specifically to greatly reduce bus overhead to allow the critical section code to be used to protect per-cpu caches. These caches, such as Jeff's slab allocator work, can potentially operate very quickly making the effective savings of the new critical section code's performance very significant. The second purpose of this commit is to allow architectures to enable certain interrupts while in a critical section. Specifically, the intention is to eventually allow certain FAST interrupts to operate rather then defer. The third purpose of this commit is to begin to clean up the critical_enter()/critical_exit()/cpu_critical_enter()/ cpu_critical_exit() API which currently has serious cross pollution in MI code (in fork_exit() and ast() for example). The fourth purpose of this commit is to provide a framework that allows kernel-preempting software interrupts to be implemented cleanly. This is currently used for two forward interrupts in I386. Other architectures will have the choice of using this infrastructure or building the functionality directly into critical_enter()/ critical_exit(). Finally, this commit is designed to greatly improve the flexibility of various architectures to manage critical section handling, software interrupts, preemption, and other highly integrated architecture-specific details.
* Work-in-progress commit syncing up pmap cleanups that I have been workingpeter2002-02-251-75/+169
| | | | | | | | | | | | | | | | | | | | | | | | | | on for a while: - fine grained TLB shootdown for SMP on i386 - ranged TLB shootdowns.. eg: specify a range of pages to shoot down with a single IPI, since the IPI is very expensive. Adjust some callers that used to trigger this inside tight loops to do a ranged shootdown at the end instead. - PG_G support for SMP on i386 (options ENABLE_PG_G) - defer PG_G activation till after we decide what we are going to do with PSE and the 4MB pages at the start of the kernel. This should solve some rumored strangeness about stale PG_G entries getting stuck underneath the 4MB pages. - add some instrumentation for the fine TLB shootdown - convert some asm instruction wrappers from functions to inlines. gcc seems to do a fair bit better with this. - [temporarily!] pessimize the tlb shootdown IPI handlers. I will fix this again shortly. This has been working fairly well for me for a while, but I have tweaked it again prior to commit since my last major testing round. The only outstanding problem that I know of is PG_G related, which is why there is an option for it (not on by default for SMP). I have seen a world speedups by a few percent (as much as 4 or 5% in one case) but I have *not* accurately measured this - I am a bit sceptical of these numbers.
* Move do_cpuid() from a identcpu.c into cpufunc.h.dwmalone2002-02-121-0/+11
|
* Various assembly fixes mostly in the form of using the "+" modifier forjhb2001-12-181-14/+14
| | | | | | | output operands to mark them as both input and output rather than listing operands twice. Reviewed by: bde
* Modify the critical section API as follows:jhb2001-12-181-4/+7
| | | | | | | | | | | | | | | | | | | - The MD functions critical_enter/exit are renamed to start with a cpu_ prefix. - MI wrapper functions critical_enter/exit maintain a per-thread nesting count and a per-thread critical section saved state set when entering a critical section while at nesting level 0 and restored when exiting to nesting level 0. This moves the saved state out of spin mutexes so that interlocking spin mutexes works properly. - Most low-level MD code that used critical_enter/exit now use cpu_critical_enter/exit. MI code such as device drivers and spin mutexes use the MI wrappers. Note that since the MI wrappers store the state in the current thread, they do not have any return values or arguments. - mtx_intr_enable() is replaced with a constant CRITICAL_FORK which is assigned to curthread->td_savecrit during fork_exit(). Tested on: i386, alpha
* Provide access to the IA32 hardware debug registers from the ddbbsd2001-06-281-1/+64
| | | | | | | kernel debugger. Proper use of these registers allows setting hardware watchpoints for use in kernel debugging. MFC after: 2 weeks
* Back out 1.103. It wasn't approved by the owner of the file andimp2001-04-181-45/+45
| | | | | | introduced style bugs. Submited by: bde
* De __P() while I'm here. Done as a separate commit since it is justimp2001-04-031-45/+45
| | | | | | | | | | stylistic. # Yes, this break K&R, but this file already used so many gcc extensions # keeping K&R support seemed too anachronistic for me. Didn't fix the bug where functions that can only be used in the kernel are exported to userland.
* Make this file C++ safe. It defines many useful functions (inb, outb)imp2001-04-031-0/+4
| | | | | | | that people use from userland in C++ programs. I've had this in my tree for ages and just got bit by it not being in the real tree again. This is a MFC candidate.
* - Add the new critical_t type used to save state inside of criticaljhb2001-03-281-15/+18
| | | | | | | | | sections. - Add implementations of the critical_enter() and critical_exit() functions and remove restore_intr() and save_intr(). - Remove the somewhat bogus disable_intr() and enable_intr() functions on the alpha as the alpha actually uses a priority level and not simple bit flag on the CPU.
* Assembler fixes.markm2000-11-211-4/+4
| | | | | | Fix opcodes that were typed as ".byte 0xNN, 0xMM" when an older assembler could not recognise the newer Pentium instructions. Reviewed by: jhb
* Removed unused include of <machine/lock.h>. The locking interface stoppedbde2000-10-121-4/+0
| | | | being (ab)used here in rev.1.97.
* Replace loadandclear() with atomic_readandclear_int().jhb2000-10-051-11/+0
|
* Major update to the way synchronization is done in the kernel. Highlightsjasone2000-09-071-6/+15
| | | | | | | | | | | | | | | include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
* Fix an __asm operand constraint which broke the -O3 and -O0 builds.bsd2000-02-211-6/+6
| | | | | Submitted by: Seigo Tanimura <tanimura@freebsd.org> Approved by: jkh
* Don't forget to reset the hardware debug registers when a process thatbsd2000-02-201-0/+50
| | | | | | | | | | was using them exits. Don't allow a user process to cause the kernel to take a TRCTRAP on a user space address. Reviewed by: jlemon, sef Approved by: jkh
OpenPOWER on IntegriCloud