summaryrefslogtreecommitdiffstats
path: root/sys/amd64/include/smp.h
Commit message (Collapse)AuthorAgeFilesLines
* Add the !define(COMPILING_LINT)pirzyk2002-10-171-1/+1
| | | | | | pass the pointy hat... Requested by: Juli Mallett <jmallett@FreeBSD.org>
* put an #error directive when SMP and CPU_DISABLE_CMPXCHG are setpirzyk2002-10-171-0/+3
| | | | | | | together. Requested by: Lars Eggart <larse@isi.edu> Enlighted how to do it by: John Baldwin <jhb@freebsd.org>
* Revive backed out pmap related changes from Feb 2002. The highlights are:peter2002-07-121-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - It actually works this time, honest! - Fine grained TLB shootdowns for SMP on i386. IPI's are very expensive, so try and optimize things where possible. - Introduce ranged shootdowns that can be done as a single IPI. - PG_G support for i386 - Specific-cpu targeted shootdowns. For example, there is no sense in globally purging the TLB cache for where we are stealing a page from the local unshared process on the local cpu. Use pm_active to track this. - Add some instrumentation for the tlb shootdown code. - Rip out SMP code from <machine/cpufunc.h> - Try and fix some very bogus PG_G and PG_PS interactions that were bad enough to cause vm86 bios calls to break. vm86 depended on our existing bugs and this was the cause of the VESA panics last time. - Fix the silly one-line error that caused the 'panic: bad pte' last time. - Fix a couple of other silly one-line errors that should have caused more pain than they did. Some more work is needed: - pmap_{zero,copy}_page[_idle]. These can be done without IPI's if we have a hook in cpu_switch. - The IPI handlers need some cleanup. I have a bogus %ds load that can be avoided. - APTD handling is rather bogus and appears to be a large source of global TLB IPI shootdowns for no really good reason. I see speedups of between 1.5% and ~4% on buildworlds in a while 1 loop. I expect to see a bigger difference when there is significant pageout activity or the system otherwise has memory shortages. I have backed out a few optimizations that I had been using over the last few days in order to be a little more conservative. I'll revisit these again over the next few days as the dust settles. New option: DISABLE_PG_G - In case I missed something.
* Fixed some style bugs in the removal of __P(()). The main ones werebde2002-03-231-46/+46
| | | | | | not removing tabs before "__P((", and not outdenting continuation lines to preserve non-KNF lining up of code with parentheses. Switch to KNF formatting and/or rewrap the whole prototype in some cases.
* Remove __P.alfred2002-03-201-46/+46
|
* Back out all the pmap related stuff I've touched over the last few days.peter2002-02-271-2/+1
| | | | | | There is some unresolved badness that has been eluding me, particularly affecting uniprocessor kernels. Turning off PG_G helped (which is a bad sign) but didn't solve it entirely. Userland programs still crashed.
* Work-in-progress commit syncing up pmap cleanups that I have been workingpeter2002-02-251-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | on for a while: - fine grained TLB shootdown for SMP on i386 - ranged TLB shootdowns.. eg: specify a range of pages to shoot down with a single IPI, since the IPI is very expensive. Adjust some callers that used to trigger this inside tight loops to do a ranged shootdown at the end instead. - PG_G support for SMP on i386 (options ENABLE_PG_G) - defer PG_G activation till after we decide what we are going to do with PSE and the 4MB pages at the start of the kernel. This should solve some rumored strangeness about stale PG_G entries getting stuck underneath the 4MB pages. - add some instrumentation for the fine TLB shootdown - convert some asm instruction wrappers from functions to inlines. gcc seems to do a fair bit better with this. - [temporarily!] pessimize the tlb shootdown IPI handlers. I will fix this again shortly. This has been working fairly well for me for a while, but I have tweaked it again prior to commit since my last major testing round. The only outstanding problem that I know of is PG_G related, which is why there is an option for it (not on by default for SMP). I have seen a world speedups by a few percent (as much as 4 or 5% in one case) but I have *not* accurately measured this - I am a bit sceptical of these numbers.
* Small cleanups to the SMP code:jhb2001-12-171-5/+0
| | | | | | | | | | | - Axe inlvtlb_ok as it was completely redundant with smp_active. - Remove references to non-existent variable and non-existent file in i386/include/smp.h. - Don't perform initializations local to each CPU while holding the ap boot lock on i386 while an AP bootstraps itself. - Reorganize the AP startup code some to unify the latter half of the functions to bring an AP up. Eventually this might be broken out into a MI function in subr_smp.c.
* Overhaul of the SMP code. Several portions of the SMP kernel support havejhb2001-04-271-27/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | been made machine independent and various other adjustments have been made to support Alpha SMP. - It splits the per-process portions of hardclock() and statclock() off into hardclock_process() and statclock_process() respectively. hardclock() and statclock() call the *_process() functions for the current process so that UP systems will run as before. For SMP systems, it is simply necessary to ensure that all other processors execute the *_process() functions when the main clock functions are triggered on one CPU by an interrupt. For the alpha 4100, clock interrupts are delievered in a staggered broadcast fashion, so we simply call hardclock/statclock on the boot CPU and call the *_process() functions on the secondaries. For x86, we call statclock and hardclock as usual and then call forward_hardclock/statclock in the MD code to send an IPI to cause the AP's to execute forwared_hardclock/statclock which then call the *_process() functions. - forward_signal() and forward_roundrobin() have been reworked to be MI and to involve less hackery. Now the cpu doing the forward sets any flags, etc. and sends a very simple IPI_AST to the other cpu(s). AST IPIs now just basically return so that they can execute ast() and don't bother with setting the astpending or needresched flags themselves. This also removes the loop in forward_signal() as sched_lock closes the race condition that the loop worked around. - need_resched(), resched_wanted() and clear_resched() have been changed to take a process to act on rather than assuming curproc so that they can be used to implement forward_roundrobin() as described above. - Various other SMP variables have been moved to a MI subr_smp.c and a new header sys/smp.h declares MI SMP variables and API's. The IPI API's from machine/ipl.h have moved to machine/smp.h which is included by sys/smp.h. - The globaldata_register() and globaldata_find() functions as well as the SLIST of globaldata structures has become MI and moved into subr_smp.c. Also, the globaldata list is only available if SMP support is compiled in. Reviewed by: jake, peter Looked over by: eivind
* Remove the BETTER_CLOCK #ifdef's. The code is on by default and is herejhb2001-04-101-2/+0
| | | | | | to stay for the foreseeable future. OK'd by: peter (the idea)
* Defer assignment of low level interrupt handlers for PCI interruptstegge2001-01-281-0/+1
| | | | | described in the MP table until something asks for the interrupt number later on.
* Move io_apic_{read,write} from apic_ipl.s (where they do not belong) intopeter2000-12-061-4/+3
| | | | | | mpapic.c. This gives us the benefit of C type checking. These functions are not called in any critical paths and are not used by the interrupt routines.
* GC unused assembler function apic_eoi()peter2000-12-061-1/+0
|
* Cleanup some leftover lint from the old interrupt system.peter2000-12-041-6/+1
| | | | | | | | Also, while here, run up to 32 interrupt sources on APIC systems. Normalize INTREN/INTRDIS so they are the same on both UP and SMP systems rather than sometimes a macro, and sometimes a function. Reviewed by: jhb, jakeb
* Move MAXCPU from machine/smp.h to machine/param.h to fix breakageps2000-09-231-10/+0
| | | | | with !SMP kernels. Also, replace NCPUS with MAXCPU since they are redundant.
* Remove the NCPU, NAPIC, NBUS, NINTR config options. Make NAPIC,ps2000-09-221-12/+11
| | | | | | NBUS, NINTR dynamic and set NCPU to a maximum of 16 under SMP. Reviewed by: peter
* Make LINT compile.phk2000-09-161-1/+1
|
* Test for both SMP and I386_CPU being set before generating an error.jhb2000-09-071-1/+1
|
* Major update to the way synchronization is done in the kernel. Highlightsjasone2000-09-071-34/+4
| | | | | | | | | | | | | | | include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
* Increase the default NAPIC from 1 to 2 as a bandaid until we allocatemsmith2000-08-181-1/+1
| | | | these dynamically (ie. typically you shouldn't have to set NAPIC at all)
* Be more verbose when changing APIC ID on an IO APIC.tegge2000-08-061-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't allow cpu entries in the MP table to contain APIC IDs out of range. Don't write outside array boundaries if an IO APIC entry in the MP table contains an APIC ID out of range. Assign APIC IDs for all IO APICs according to section 3.6.6 in the Intel MP spec: - If the current APIC ID on an IO APIC doesn't conflict with other IO APICs or CPUs, that APIC ID should be used. The copy of the MP table must be updated if the corresponding APIC ID in the MP table is different. - If the current APIC ID was in conflict with other units, the corresponding APIC ID specified in the MP table is checked for conflict. - If a conflict is still found then fall back to using a new unique ID. The copy of the MP table must be updated. - IDs out of range is considered to be in conflict. During these operations, the IO_TO_ID array cannot be used, since any conflict would have caused information loss. The array is then corrected, since all APIC ID conflicts should have been resolved. PR: 20312, 18919
* Further fixes for multiple-IO-APIC systems from Tor Egge:msmith2000-05-311-0/+1
| | | | | | | | | | | | | | | | | | | | | | | Further experimentation showed that some Dell 2450 machines with the prevention kludge installed still got T_RESERVED traps. CPU interrupt vector 0x7A was observed to be triggered. This might have been the bitwise OR of two different vectors sent from each of the IOAPICs at the same time. IOAPIC #0: 0x68 --> irq 8: RTC timer interrupt IOAPIC #1: 0x32 --> irq 18: scsi host adapter or network interface ---- 0x7a --> T_RESERVED Both IOAPICs had ID 0. Appendix B.3 in the MP spec indicates that the operating system is responsible for assigning unique IDs to the IOAPICs. The enclosed patch programs the IOAPIC IDs according to the IOAPIC entries in the MP table. Submitted by: tegge
* The SMP cleanup commit broke UP compiles. Make UP compiles work again.dillon2000-03-281-1/+18
|
* ISA device drivers use the ISA source interrupt number in locations wheretegge2000-01-041-0/+2
| | | | | | | | | | | | | | the low level interrupt handler number should be used. Change setup_apic_irq_mapping() to allocate low level interrupt handler X (Xintr${X}) for any ISA interrupt X mentioned in the MP table. Remove an assumption in the driver for the system clock (clock.c) that interrupts mentioned in the MP table as delivered to IOAPIC #0 intpin Y is handled by low level interrupt handler Y (Xintr${Y}) but don't assume that low level interrupt handler 0 (Xintr0) is used. Don't allocate two low level interrupt handlers for the system clock. Reviewed by: NOKUBI Hirotaka <hnokubi@yyy.or.jp>
* Change #ifdef KERNEL to #ifdef _KERNEL in the public headers. "KERNEL"peter1999-12-291-2/+2
| | | | | | is an application space macro and the applications are supposed to be free to use it as they please (but cannot). This is consistant with the other BSD's who made this change quite some time ago. More commits to come.
* Fix from Tor so that if we enter the debugger in the tristate going tomjacob1999-09-251-0/+1
| | | | | | SMP (other CPUs stopped but SMP mode not really started). Obtained from:Tor.Egge@fast.no
* $Id$ -> $FreeBSD$peter1999-08-281-1/+1
|
* Implement an all-CPU shootdown-style rendezvous facility. This allowsmsmith1999-07-201-1/+6
| | | | | | | | | | | the caller to specify a function to be guarded between an entry and exit barrier, as well as pre- and post-barrier functions. The primary use for this function is synchronised update of per-cpu private data. The implementation is almost (but not quite) MI; with a better mechanism for masking per-CPU interrupts it could probably be hoisted. Reviewed by: peter (partially)
* Enable vmspace sharing on SMP. Major changes are,luoqi1999-04-281-8/+1
| | | | | | | | | | | | | | | | | - %fs register is added to trapframe and saved/restored upon kernel entry/exit. - Per-cpu pages are no longer mapped at the same virtual address. - Each cpu now has a separate gdt selector table. A new segment selector is added to point to per-cpu pages, per-cpu global variables are now accessed through this new selector (%fs). The selectors in gdt table are rearranged for cache line optimization. - fask_vfork is now on as default for both UP and SMP. - Some aio code cleanup. Reviewed by: Alan Cox <alc@cs.rice.edu> John Dyson <dyson@iquest.net> Julian Elischer <julian@whistel.com> Bruce Evans <bde@zeta.org.au> David Greenman <dg@root.com>
* Maintain a mapping from irq number to (ioapic number, int pin) tuple,tegge1998-09-061-9/+13
| | | | | | | | | | | | and use this when masking/unmasking interrupts. Maintain a mapping from (iopaic number, int pin) tuple to irq number, and use this when configuring devices and programming the ioapics. Previous code assumed that irq number was equal to int pin number, and that the ioapic number was 0. Don't let an AP enter _cpu_switch before all local apics are initialized.
* Add forwarding of roundrobin to other cpus. This gives a more regulartegge1998-05-171-1/+5
| | | | | | | | | | | update of cpu usage as shown by top when one process is cpu bound (no system calls) while the system is otherwise idle (except for top). Don't attempt to switch to the BSP in boot(). If the system was idle when an interrupt caused a panic, this won't work. Instead, switch to the BSP in cpu_reset. Remove some spurious forward_statclock/forward_hardclock warnings.
* Add two workarounds for broken MP tables:tegge1998-04-011-1/+2
| | | | | | | | | | | | | - Attempt to handle PCI devices where the interrupt is an ISA/EISA interrupt according to the mp table. - Attempt to handle multiple IO APIC pins connected to the same PCI or ISA/EISA interrupt source. Print a warning if this happens, since performance is suboptimal. This workaround is only used for PCI devices. With these two workarounds, the -SMP kernel is capable of running on my Asus P/I-P65UP5 motherboard when version 1.4 of the MP table is disabled.
* Declare some variables modified by interrupt handlers as volatile.tegge1998-04-011-3/+3
|
* This mega-commit is meant to fix numerous interrelated problems. Theredyson1998-03-071-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | has been some bitrot and incorrect assumptions in the vfs_bio code. These problems have manifest themselves worse on NFS type filesystems, but can still affect local filesystems under certain circumstances. Most of the problems have involved mmap consistancy, and as a side-effect broke the vfs.ioopt code. This code might have been committed seperately, but almost everything is interrelated. 1) Allow (pmap_object_init_pt) prefaulting of buffer-busy pages that are fully valid. 2) Rather than deactivating erroneously read initial (header) pages in kern_exec, we now free them. 3) Fix the rundown of non-VMIO buffers that are in an inconsistent (missing vp) state. 4) Fix the disassociation of pages from buffers in brelse. The previous code had rotted and was faulty in a couple of important circumstances. 5) Remove a gratuitious buffer wakeup in vfs_vmio_release. 6) Remove a crufty and currently unused cluster mechanism for VBLK files in vfs_bio_awrite. When the code is functional, I'll add back a cleaner version. 7) The page busy count wakeups assocated with the buffer cache usage were incorrectly cleaned up in a previous commit by me. Revert to the original, correct version, but with a cleaner implementation. 8) The cluster read code now tries to keep data associated with buffers more aggressively (without breaking the heuristics) when it is presumed that the read data (buffers) will be soon needed. 9) Change to filesystem lockmgr locks so that they use LK_NOPAUSE. The delay loop waiting is not useful for filesystem locks, due to the length of the time intervals. 10) Correct and clean-up spec_getpages. 11) Implement a fully functional nfs_getpages, nfs_putpages. 12) Fix nfs_write so that modifications are coherent with the NFS data on the server disk (at least as well as NFS seems to allow.) 13) Properly support MS_INVALIDATE on NFS. 14) Properly pass down MS_INVALIDATE to lower levels of the VM code from vm_map_clean. 15) Better support the notion of pages being busy but valid, so that fewer in-transit waits occur. (use p->busy more for pageouts instead of PG_BUSY.) Since the page is fully valid, it is still usable for reads. 16) It is possible (in error) for cached pages to be busy. Make the page allocation code handle that case correctly. (It should probably be a printf or panic, but I want the system to handle coding errors robustly. I'll probably add a printf.) 17) Correct the design and usage of vm_page_sleep. It didn't handle consistancy problems very well, so make the design a little less lofty. After vm_page_sleep, if it ever blocked, it is still important to relookup the page (if the object generation count changed), and verify it's status (always.) 18) In vm_pageout.c, vm_pageout_clean had rotted, so clean that up. 19) Push the page busy for writes and VM_PROT_READ into vm_pageout_flush. 20) Fix vm_pager_put_pages and it's descendents to support an int flag instead of a boolean, so that we can pass down the invalidate bit.
* Remove special handling for resuming clock interrupt when using APIC_IO.tegge1998-03-051-8/+1
| | | | The `generic' vector stubs do the right thing.
* When entering the apic version of slow interrupt handler, leveltegge1998-03-031-1/+4
| | | | | | | | | interrupts are masked, and EOI is sent iff the corresponding ISR bit is set in the local apic. If the CPU cannot obtain the interrupt service lock (currently the global kernel lock) the interrupt is forwarded to the CPU holding that lock. Clock interrupts now have higher priority than other slow interrupts.
* Forward the signal if the process runs on a different CPU. This reducestegge1998-03-031-3/+2
| | | | | | | | the signal handling latency for cpu-bound processes that performs very few system calls. The IPI for forcing an additional software trap is no longer dependent upon BETTER_CLOCK being defined.
* forward_statclock and forward_hardclock are located in mp_machdep.c.tegge1998-03-031-3/+5
|
* Fix page prezeroing for SMP, and fix some potential paging-in-progressdyson1998-02-251-3/+3
| | | | | | hangs. The paging-in-progress diagnosis was a result of Tor Egge's excellent detective work. Submitted by: Partially from Tor Egge.
* Moved the SMP declarations of INTREN() and INTRDIS() to the correct header,bde1998-01-011-3/+1
| | | | i.e., the same header as corresponding non-SMP #defines.
* The improvements to clock statistics by Tor Eggefsmp1997-12-081-1/+8
| | | | | | | Wrappered and enabled by the define BETTER_CLOCK (on by default in smpyests.h) Reviewed by: smp@csn.net Submitted by: Tor Egge <Tor.Egge@idi.ntnu.no>
* General cleanup of the lock pushdown code. They are grouped and enabledfsmp1997-09-071-1/+2
| | | | | | | | | from machine/smptests.h: #define PUSHDOWN_LEVEL_1 #define PUSHDOWN_LEVEL_2 #define PUSHDOWN_LEVEL_3 #define PUSHDOWN_LEVEL_4_NOT
* Another round of lock pushdown.fsmp1997-08-301-1/+9
| | | | | | | | | Add a simplelock to deal with disable_intr()/enable_intr() as used in UP kernel. UP kernel expects that this is enough to guarantee exclusive access to regions of code bracketed by these 2 functions. Add a simplelock to bracket clock accesses in clock.c: clock_lock. Help from: Bruce Evans <bde@zeta.org.au>
* Clean up the SMP AP bootstrap and eliminate the wretched idle procs.peter1997-08-261-2/+1
| | | | | | | | | | | | | | | | | | | | - We now have enough per-cpu idle context, the real idle loop has been revived (cpu's halt now with nothing to do). - Some preliminary support for running some operations outside the global lock (eg: zeroing "free but not yet zeroed pages") is present but appears to cause problems. Off by default. - the smp_active sysctl now behaves differently. It's merely a 'true/false' option. Setting smp_active to zero causes the AP's to halt in the idle loop and stop scheduling processes. - bootstrap is a lot safer. Instead of sharing a statically compiled in stack a number of times (which has caused lots of problems) and then abandoning it, we use the idle context to boot the AP's directly. This should help >2 cpu support since the bootlock stuff was in doubt. - print physical apic id in traps.. helps identify private pages getting out of sync. (You don't want to know how much hair I tore out with this!) More cleanup to follow, this is more of a checkpoint than a 'finished' thing.
* A clean fix for the spl "deadlock before smp_active" problem.fsmp1997-08-241-1/+2
| | | | | | | | Added a new variable, 'bsp_apic_ready', which is set as soon as the bootstrap CPU has initialized its local APIC. Conditionalize the GENSPLR functions to call ss_lock ONLY after bsp_apic_ready is TRUE; This should prevent any problems with races between the time the 1st AP becomes ready and the time smp_active is set.
* Made PEND_INTS default.fsmp1997-08-211-14/+1
| | | | | | | | | | Made NEW_STRATEGY default. Removed misc. old cruft. Centralized simple locks into mp_machdep.c Centralized simple lock macros into param.h More cleanup in the direction of making splxx()/cpl MP-safe.
* Preperation for moving cpl into critical region access.fsmp1997-08-201-2/+8
| | | | | Several new fine-grained locks. Control of new FAST_INTR() methods.
* Removed volatile from arg to simple_lock & friends.fsmp1997-08-181-4/+4
|
* The promised "better fix" for "Trap 9 When Boot SMP" problem.fsmp1997-08-151-2/+3
| | | | | | | | | | We now tsleep() in kthread_init() between start_init() and prepare_usermode() while waiting for ALL the idle_loop() processes to come online. Debugged & tested by: "Thomas D. Dean" <tomdean@ix.netcom.com> Reviewed by: David Greenman <dg@root.com>
* Fixed imen declaration.fsmp1997-07-311-2/+2
| | | | Submitted by: Bruce Evans <bde@zeta.org.au>
OpenPOWER on IntegriCloud