summaryrefslogtreecommitdiffstats
path: root/sys/kern/subr_smp.c
Commit message (Collapse)AuthorAgeFilesLines
* Overhaul of the SMP code. Several portions of the SMP kernel support havejhb2001-04-271-2724/+139
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | been made machine independent and various other adjustments have been made to support Alpha SMP. - It splits the per-process portions of hardclock() and statclock() off into hardclock_process() and statclock_process() respectively. hardclock() and statclock() call the *_process() functions for the current process so that UP systems will run as before. For SMP systems, it is simply necessary to ensure that all other processors execute the *_process() functions when the main clock functions are triggered on one CPU by an interrupt. For the alpha 4100, clock interrupts are delievered in a staggered broadcast fashion, so we simply call hardclock/statclock on the boot CPU and call the *_process() functions on the secondaries. For x86, we call statclock and hardclock as usual and then call forward_hardclock/statclock in the MD code to send an IPI to cause the AP's to execute forwared_hardclock/statclock which then call the *_process() functions. - forward_signal() and forward_roundrobin() have been reworked to be MI and to involve less hackery. Now the cpu doing the forward sets any flags, etc. and sends a very simple IPI_AST to the other cpu(s). AST IPIs now just basically return so that they can execute ast() and don't bother with setting the astpending or needresched flags themselves. This also removes the loop in forward_signal() as sched_lock closes the race condition that the loop worked around. - need_resched(), resched_wanted() and clear_resched() have been changed to take a process to act on rather than assuming curproc so that they can be used to implement forward_roundrobin() as described above. - Various other SMP variables have been moved to a MI subr_smp.c and a new header sys/smp.h declares MI SMP variables and API's. The IPI API's from machine/ipl.h have moved to machine/smp.h which is included by sys/smp.h. - The globaldata_register() and globaldata_find() functions as well as the SLIST of globaldata structures has become MI and moved into subr_smp.c. Also, the globaldata list is only available if SMP support is compiled in. Reviewed by: jake, peter Looked over by: eivind
* Make the ap_boot_mtx mutex static.jhb2001-04-201-1/+1
|
* Blow away the panic mutex in favor of using a single atomic_cmpset() on ajhb2001-04-171-4/+0
| | | | | | panic_cpu shared variable. I used a simple atomic operation here instead of a spin lock as it seemed to be excessive overhead. Also, this can avoid recursive panics if, for example, witness is broken.
* Rename the IPI API from smp_ipi_* to ipi_* since the smp_ prefix is justjhb2001-04-111-14/+14
| | | | | | "redundant noise" and to match the IPI constant namespace (IPI_*). Requested by: bde
* Remove the BETTER_CLOCK #ifdef's. The code is on by default and is herejhb2001-04-101-14/+4
| | | | | | to stay for the foreseeable future. OK'd by: peter (the idea)
* Add an MI API for sending IPI's. I used the same API present on the alphajhb2001-04-101-12/+55
| | | | | | | | because: - it used a better namespace (smp_ipi_* rather than *_ipi), - it used better constant names for the IPI's (IPI_* rather than X*_OFFSET), and - this API also somewhat exists for both alpha and ia64 already.
* Rework the witness code to work with sx locks as well as mutexes.jhb2001-03-281-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Introduce lock classes and lock objects. Each lock class specifies a name and set of flags (or properties) shared by all locks of a given type. Currently there are three lock classes: spin mutexes, sleep mutexes, and sx locks. A lock object specifies properties of an additional lock along with a lock name and all of the extra stuff needed to make witness work with a given lock. This abstract lock stuff is defined in sys/lock.h. The lockmgr constants, types, and prototypes have been moved to sys/lockmgr.h. For temporary backwards compatability, sys/lock.h includes sys/lockmgr.h. - Replace proc->p_spinlocks with a per-CPU list, PCPU(spinlocks), of spin locks held. By making this per-cpu, we do not have to jump through magic hoops to deal with sched_lock changing ownership during context switches. - Replace proc->p_heldmtx, formerly a list of held sleep mutexes, with proc->p_sleeplocks, which is a list of held sleep locks including sleep mutexes and sx locks. - Add helper macros for logging lock events via the KTR_LOCK KTR logging level so that the log messages are consistent. - Add some new flags that can be passed to mtx_init(): - MTX_NOWITNESS - specifies that this lock should be ignored by witness. This is used for the mutex that blocks a sx lock for example. - MTX_QUIET - this is not new, but you can pass this to mtx_init() now and no events will be logged for this lock, so that one doesn't have to change all the individual mtx_lock/unlock() operations. - All lock objects maintain an initialized flag. Use this flag to export a mtx_initialized() macro that can be safely called from drivers. Also, we on longer walk the all_mtx list if MUTEX_DEBUG is defined as witness performs the corresponding checks using the initialized flag. - The lock order reversal messages have been improved to output slightly more accurate file and line numbers.
* Kill the 4MB kernel limit dead. [I hope :-)].peter2001-03-151-3/+9
| | | | | | | | | | | | | | | For UP, we were using $tmp_stk as a stack from the data section. If the kernel text section grew beyond ~3MB, the data section would be pushed beyond the temporary 4MB P==V mapping. This would cause the trampoline up to high memory to fault. The hack workaround I did was to use all of the page table pages that we already have while preparing the initial P==V mapping, instead of just the first one. For SMP, the AP bootstrap process suffered the same sort of problem and got the same treatment. MFC candidate - this breaks on 4.x just the same.. Thanks to: Richard Todd <rmtodd@ichotolot.servalan.com>
* Activate USER_LDT by default. The new thread libraries are going topeter2001-02-231-3/+0
| | | | | | | | depend on this. The linux ABI emulator tries to use it for some linux binaries too. VM86 had a bigger cost than this and it was made default a while ago. Reviewed by: jhb, imp
* Woops, remove an obsolete reference to gd_cpu_lockid.jhb2001-02-091-1/+0
|
* Change and clean the mutex lock interface.bmilekic2001-02-091-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
* Remove some leftovers from the CMAP* stuff in globaldata and thepeter2001-01-301-7/+2
| | | | BSP and AP startup.
* Move the setting of curproc to idleproc up earlier in ap_init(). Thebmilekic2001-01-281-6/+6
| | | | | | | | | | | problem is that a mutex lock, prior to this change, is acquired before the curproc is set to idleproc, so we mess ourselves up by calling the mutex lock routine with curproc == NULL. Moving it up after the aps_ready spin-wait has us hopefully setting it after idleproc is setup. Solved by: jake (the allmighty) :-)
* Defer assignment of low level interrupt handlers for PCI interruptstegge2001-01-281-37/+48
| | | | | described in the MP table until something asks for the interrupt number later on.
* Convert all simplelocks to mutexes and remove the simplelock implementations.jasone2001-01-241-30/+32
|
* - Relocate portions of this file to get it into an order closer to that ofjhb2001-01-241-171/+153
| | | | | | | | | | | | | the alpha mp_machdep.c. - Proc locking. - Catch up to the P_FOO -> PS_FOO proc flags changes. - Stick ap_init()'s prototype with the other prototypes. - Remove the Xforwardirq IPI. - Remove unused simplelocks. - Don't try to psignal() from forward_statclock(), but set the appropriate signal pending flag in p_sflag instead. - Add in KTR_SMP tracepoints for various SMP functions. (Brought over from the alpha port)
* Make intr_nesting_level per-process, rather than per-cpu. Setupjake2001-01-211-1/+3
| | | | | | | | interrupt threads to run with it always >= 1, so that malloc can detect M_WAITOK from "interrupt" context. This is also necessary in order to context switch from sched_ithd() directly. Reviewed By: peter
* Remove APIC_INTR_DIAGNOSTIC - this has been disabled for some time now.peter2001-01-211-25/+0
| | | | Remove some leftovers of removed SMP options.
* Remove the per-cpu pages used for copy and zero-ing pages of memoryjake2001-01-211-8/+0
| | | | | | for SMP; just use the same ones as UP. These weren't used without holding Giant anyway, and the routines that use them would have to be protected from pre-emption to avoid migrating cpus.
* apic_itrace_splz[] is unusedpeter2001-01-191-1/+0
|
* Protect p_stat and p_oncpu with sched_lock in forward_signal().jhb2001-01-181-2/+9
|
* Change return ??? to return -1 in some #if 0'ed code.jake2001-01-121-2/+2
|
* Use PCPU_GET, PCPU_PTR and PCPU_SET to access all per-cpu variablesjake2001-01-101-19/+19
| | | | other then curproc.
* Fix a warning. The type of globaldata.gd_prvspace has changed.jake2001-01-081-1/+2
|
* This is kind of a nasty hack, but it appears to solve the Compaq DL360peter2000-12-061-2/+13
| | | | | | | | SMP problem. Compaq, in their infinite wisdom, forgot to put the IO apic intpin #0 connection to the 8259 PIC into the mptable. This hack is to look and see if intpin #0 has *no* table entry and adds a fake ExtInt entry for the remap routines to use. isa/clock.c will still test the interrupts. This entry is only ever used on an already broken system.
* Cleanup some leftover lint from the old interrupt system.peter2000-12-041-1/+1
| | | | | | | | Also, while here, run up to 32 interrupt sources on APIC systems. Normalize INTREN/INTRDIS so they are the same on both UP and SMP systems rather than sometimes a macro, and sometimes a function. Reviewed by: jhb, jakeb
* Don't wait forever for CPUs to stop or restart. Instead, give up after ajhb2000-11-281-2/+19
| | | | | | | timeout. If DIAGNOSTIC is turned on, then display a message to the console with a map of which CPUs failed to stop or restart. This gives an SMP box at least a fighting chance of getting into DDB if one of the other CPUs has interrupts disabled.
* Catch up to moving headers:jhb2000-10-201-1/+1
| | | | | - machine/ipl.h -> sys/ipl.h - machine/mutex.h -> sys/mutex.h
* - Change fast interrupts on x86 to push a full interrupt frame and tojhb2000-10-061-9/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | return through doreti to handle ast's. This is necessary for the clock interrupts to work properly. - Change the clock interrupts on the x86 to be fast instead of threaded. This is needed because both hardclock() and statclock() need to run in the context of the current process, not in a separate thread context. - Kill the prevproc hack as it is no longer needed. - We really need Giant when we call psignal(), but we don't want to block during the clock interrupt. Instead, use two p_flag's in the proc struct to mark the current process as having a pending SIGVTALRM or a SIGPROF and let them be delivered during ast() when hardclock() has finished running. - Remove CLKF_BASEPRI, which was #ifdef'd out on the x86 anyways. It was broken on the x86 if it was turned on since cpl is gone. It's only use was to bogusly run softclock() directly during hardclock() rather than scheduling an SWI. - Remove the COM_LOCK simplelock and replace it with a clock_lock spin mutex. Since the spin mutex already handles disabling/restoring interrupts appropriately, this also lets us axe all the *_intr() fu. - Back out the hacks in the APIC_IO x86 cpu_initclocks() code to use temporary fast interrupts for the APIC trial. - Add two new process flags P_ALRMPEND and P_PROFPEND to mark the pending signals in hardclock() that are to be delivered in ast(). Submitted by: jakeb (making statclock safe in a fast interrupt) Submitted by: cp (concept of delaying signals until ast())
* Remove the NCPU, NAPIC, NBUS, NINTR config options. Make NAPIC,ps2000-09-221-36/+54
| | | | | | NBUS, NINTR dynamic and set NCPU to a maximum of 16 under SMP. Reviewed by: peter
* Clean up process accounting some more. Unfortunately, it is still notjhb2000-09-121-18/+14
| | | | | quite right on i386 as the CPU who runs statclock() doesn't have a valid clockframe to calculate statistics with.
* When doing statistics for statclock on other CPU's, use the other CPUs'jhb2000-09-111-3/+3
| | | | | | idleproc pointers instead of our own for comparisons. Submitted by: tegge
* Remove an unneeded extern declaration of cp_time.jhb2000-09-081-2/+0
|
* Really fix USER_LDT. (Don't use currentldt as an L-value.)jake2000-09-081-1/+1
|
* Major update to the way synchronization is done in the kernel. Highlightsjasone2000-09-071-26/+62
| | | | | | | | | | | | | | | include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
* Clean up some low level bootstrap code:peter2000-08-111-11/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - stop using the evil 'struct trapframe' argument for mi_startup() (formerly main()). There are much better ways of doing it. - do not use prepare_usermode() - setregs() in execve() will do it all for us as long as the p_md.md_regs pointer is set. (which is now done in machdep.c rather than init_main.c. The Alpha port did it this way all along and is much cleaner). - collect all the magic %cr0 etc register settings into one place and have the AP's call that instead of using magic numbers (!!) that keep changing over and over again. - Make it safe to call kthread_create() earlier, including during the device probe sequence. It doesn't need the callback mechanism that NetBSD's version uses. - kthreads created this way are root-less as they exist before the root filesystem is mounted. init(1) is set up so that it aquires the root pointers prior to running. If other kthreads want filesystem acccess we can make this code more generic. - set all threads start times once we have decided what time it is. - init uses a trampoline rather than the evil prepare_usermode() hack. - kern_descrip.c has a couple of tweaks to deal with forking when there is no rootdir or cwd etc. - adjust the early SYSINIT() sequence so that a few prereqisites are in place. eg: make sure the run queue is initialized before doing forks. With this, the USB code can easily create a kthread to do the device tree discovery. (I have tested it, it works nicely). There are still some open issues before this is truely useful. - tsleep() does not like working before the clock is running. It sort-of tries to spin wait, but it can do more useful things now. - stopping a kthread in kld code at unload time is "interesting" but we have a solution for that. The Alpha code needs no changes for this. It already uses pretty much the same strategies, but a little cleaner.
* Don't skip IOAPIC id conflict detection when only one pci bus is present.tegge2000-08-101-6/+4
| | | | | PR: 20312 Reviewed by: Steve Roome <steve@sse0691.bri.hp.com>
* Be more verbose when changing APIC ID on an IO APIC.tegge2000-08-061-2/+142
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't allow cpu entries in the MP table to contain APIC IDs out of range. Don't write outside array boundaries if an IO APIC entry in the MP table contains an APIC ID out of range. Assign APIC IDs for all IO APICs according to section 3.6.6 in the Intel MP spec: - If the current APIC ID on an IO APIC doesn't conflict with other IO APICs or CPUs, that APIC ID should be used. The copy of the MP table must be updated if the corresponding APIC ID in the MP table is different. - If the current APIC ID was in conflict with other units, the corresponding APIC ID specified in the MP table is checked for conflict. - If a conflict is still found then fall back to using a new unique ID. The copy of the MP table must be updated. - IDs out of range is considered to be in conflict. During these operations, the IO_TO_ID array cannot be used, since any conflict would have caused information loss. The array is then corrected, since all APIC ID conflicts should have been resolved. PR: 20312, 18919
* Further fixes for multiple-IO-APIC systems from Tor Egge:msmith2000-05-311-7/+2
| | | | | | | | | | | | | | | | | | | | | | | Further experimentation showed that some Dell 2450 machines with the prevention kludge installed still got T_RESERVED traps. CPU interrupt vector 0x7A was observed to be triggered. This might have been the bitwise OR of two different vectors sent from each of the IOAPICs at the same time. IOAPIC #0: 0x68 --> irq 8: RTC timer interrupt IOAPIC #1: 0x32 --> irq 18: scsi host adapter or network interface ---- 0x7a --> T_RESERVED Both IOAPICs had ID 0. Appendix B.3 in the MP spec indicates that the operating system is responsible for assigning unique IDs to the IOAPICs. The enclosed patch programs the IOAPIC IDs according to the IOAPIC entries in the MP table. Submitted by: tegge
* Commit major SMP cleanups and move the BGL (big giant lock) in thedillon2000-03-281-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | syscall path inward. A system call may select whether it needs the MP lock or not (the default being that it does need it). A great deal of conditional SMP code for various deadended experiments has been removed. 'cil' and 'cml' have been removed entirely, and the locking around the cpl has been removed. The conditional separately-locked fast-interrupt code has been removed, meaning that interrupts must hold the CPL now (but they pretty much had to anyway). Another reason for doing this is that the original separate-lock for interrupts just doesn't apply to the interrupt thread mechanism being contemplated. Modifications to the cpl may now ONLY occur while holding the MP lock. For example, if an otherwise MP safe syscall needs to mess with the cpl, it must hold the MP lock for the duration and must (as usual) save/restore the cpl in a nested fashion. This is precursor work for the real meat coming later: avoiding having to hold the MP lock for common syscalls and I/O's and interrupt threads. It is expected that the spl mechanisms and new interrupt threading mechanisms will be able to run in tandem, allowing a slow piecemeal transition to occur. This patch should result in a moderate performance improvement due to the considerable amount of code that has been removed from the critical path, especially the simplification of the spl*() calls. The real performance gains will come later. Approved by: jkh Reviewed by: current, bde (exception.s) Some work taken from: luoqi's patch
* Allow SMP systems with an MCA bus to work properly.mdodd2000-01-131-4/+6
| | | | Reviewed by: peter
* Allow SMP && NCPU == 1 to work. From now on, there's no restriction on theluoqi2000-01-071-2/+4
| | | | | value of NCPU relative to the number of cpus physically present, the actual number of cpus utilized will be the smaller of the two.
* ISA device drivers use the ISA source interrupt number in locations wheretegge2000-01-041-6/+64
| | | | | | | | | | | | | | the low level interrupt handler number should be used. Change setup_apic_irq_mapping() to allocate low level interrupt handler X (Xintr${X}) for any ISA interrupt X mentioned in the MP table. Remove an assumption in the driver for the system clock (clock.c) that interrupts mentioned in the MP table as delivered to IOAPIC #0 intpin Y is handled by low level interrupt handler Y (Xintr${Y}) but don't assume that low level interrupt handler 0 (Xintr0) is used. Don't allocate two low level interrupt handlers for the system clock. Reviewed by: NOKUBI Hirotaka <hnokubi@yyy.or.jp>
* Moved scheduling-related code to kern_synch.c so that it is easier to fixbde1999-11-271-8/+1
| | | | | | | | and extend. The new function containing the code is named schedclock() as in NetBSD, but it has slightly different semantics (it already handles incrementation of p->p_cpticks, and it should handle any calling frequency). Agreed with in principle by: dufault
* Eliminate remaining part of incorrect PCI bus numbering sanity check on ↵tegge1999-10-151-7/+0
| | | | systems with more than one PCI bus.
* Zap unneeded #includespeter1999-10-111-2/+0
| | | | Submitted by: phk
* Set up FPU state on the AP.peter1999-09-051-0/+4
| | | | Tested by: phk
* $Id$ -> $FreeBSD$peter1999-08-281-1/+1
|
* Merge the cons.c and cons.h to the best of my ability. alpha may orphk1999-08-091-3/+2
| | | | may not compile, I can't test it.
* Implement an all-CPU shootdown-style rendezvous facility. This allowsmsmith1999-07-201-1/+85
| | | | | | | | | | | the caller to specify a function to be guarded between an entry and exit barrier, as well as pre- and post-barrier functions. The primary use for this function is synchronised update of per-cpu private data. The implementation is almost (but not quite) MI; with a better mechanism for masking per-CPU interrupts it could probably be hoisted. Reviewed by: peter (partially)
OpenPOWER on IntegriCloud