diff options
author | dillon <dillon@FreeBSD.org> | 2002-02-26 17:06:21 +0000 |
---|---|---|
committer | dillon <dillon@FreeBSD.org> | 2002-02-26 17:06:21 +0000 |
commit | 57b097e18ce888ddd81fe5d6bb4145bb08cbe650 (patch) | |
tree | a1eb3a1ea442b6b5ed8c84da1e6a766614f65cfc /sys/i386/isa/apic_vector.s | |
parent | 1db171521516daa282d3ded476eaf5454f2b0a42 (diff) | |
download | FreeBSD-src-57b097e18ce888ddd81fe5d6bb4145bb08cbe650.zip FreeBSD-src-57b097e18ce888ddd81fe5d6bb4145bb08cbe650.tar.gz |
STAGE-1 of 3 commit - allow (but do not require) interrupts to remain
enabled in critical sections and streamline critical_enter() and
critical_exit().
This commit allows an architecture to leave interrupts enabled inside
critical sections if it so wishes. Architectures that do not wish to do
this are not effected by this change.
This commit implements the feature for the I386 architecture and provides
a sysctl, debug.critical_mode, which defaults to 1 (use the feature). For
now you can turn the sysctl on and off at any time in order to test the
architectural changes or track down bugs.
This commit is just the first stage. Some areas of the code, specifically
the MACHINE_CRITICAL_ENTER #ifdef'd code, is strictly temporary and will
be cleaned up in the STAGE-2 commit when the critical_*() functions are
moved entirely into MD files.
The following changes have been made:
* critical_enter() and critical_exit() for I386 now simply increment
and decrement curthread->td_critnest. They no longer disable
hard interrupts. When critical_exit() decrements the counter to
0 it effectively calls a routine to deal with whatever interrupts
were deferred during the time the code was operating in a critical
section.
Other architectures are unaffected.
* fork_exit() has been conditionalized to remove MD assumptions for
the new code. Old code will still use the old MD assumptions
in regards to hard interrupt disablement. In STAGE-2 this will
be turned into a subroutine call into MD code rather then hardcoded
in MI code.
The new code places the burden of entering the critical section
in the trampoline code where it belongs.
* I386: interrupts are now enabled while we are in a critical section.
The interrupt vector code has been adjusted to deal with the fact.
If it detects that we are in a critical section it currently defers
the interrupt by adding the appropriate bit to an interrupt mask.
* In order to accomplish the deferral, icu_lock is required. This
is i386-specific. Thus icu_lock can only be obtained by mainline
i386 code while interrupts are hard disabled. This change has been
made.
* Because interrupts may or may not be hard disabled during a
context switch, cpu_switch() can no longer simply assume that
PSL_I will be in a consistent state. Therefore, it now saves and
restores eflags.
* FAST INTERRUPT PROVISION. Fast interrupts are currently deferred.
The intention is to eventually allow them to operate either while
we are in a critical section or, if we are able to restrict the
use of sched_lock, while we are not holding the sched_lock.
* ICU and APIC vector assembly for I386 cleaned up. The ICU code
has been cleaned up to match the APIC code in regards to format
and macro availability. Additionally, the code has been adjusted
to deal with deferred interrupts.
* Deferred interrupts use a per-cpu boolean int_pending, and
masks ipending, spending, and fpending. Being per-cpu variables
it is not currently necessary to lock; bus cycles modifying them.
Note that the same mechanism will enable preemption to be
incorporated as a true software interrupt without having to
further hack up the critical nesting code.
* Note: the old critical_enter() code in kern/kern_switch.c is
currently #ifdef to be compatible with both the old and new
methodology. In STAGE-2 it will be moved entirely to MD code.
Performance issues:
One of the purposes of this commit is to enhance critical section
performance, specifically to greatly reduce bus overhead to allow
the critical section code to be used to protect per-cpu caches.
These caches, such as Jeff's slab allocator work, can potentially
operate very quickly making the effective savings of the new
critical section code's performance very significant.
The second purpose of this commit is to allow architectures to
enable certain interrupts while in a critical section. Specifically,
the intention is to eventually allow certain FAST interrupts to
operate rather then defer.
The third purpose of this commit is to begin to clean up the
critical_enter()/critical_exit()/cpu_critical_enter()/
cpu_critical_exit() API which currently has serious cross pollution
in MI code (in fork_exit() and ast() for example).
The fourth purpose of this commit is to provide a framework that
allows kernel-preempting software interrupts to be implemented
cleanly. This is currently used for two forward interrupts in I386.
Other architectures will have the choice of using this infrastructure
or building the functionality directly into critical_enter()/
critical_exit().
Finally, this commit is designed to greatly improve the flexibility
of various architectures to manage critical section handling,
software interrupts, preemption, and other highly integrated
architecture-specific details.
Diffstat (limited to 'sys/i386/isa/apic_vector.s')
-rw-r--r-- | sys/i386/isa/apic_vector.s | 194 |
1 files changed, 159 insertions, 35 deletions
diff --git a/sys/i386/isa/apic_vector.s b/sys/i386/isa/apic_vector.s index e3a37e1..0325fbe 100644 --- a/sys/i386/isa/apic_vector.s +++ b/sys/i386/isa/apic_vector.s @@ -19,11 +19,19 @@ #define PUSH_FRAME \ pushl $0 ; /* dummy error code */ \ pushl $0 ; /* dummy trap type */ \ - pushal ; \ + pushal ; /* 8 ints */ \ pushl %ds ; /* save data and extra segments ... */ \ pushl %es ; \ pushl %fs +#define PUSH_DUMMY \ + pushfl ; /* eflags */ \ + pushl %cs ; /* cs */ \ + pushl $0 ; /* dummy eip */ \ + pushl $0 ; /* dummy error code */ \ + pushl $0 ; /* dummy trap type */ \ + subl $11*4,%esp ; + #define POP_FRAME \ popl %fs ; \ popl %es ; \ @@ -31,37 +39,8 @@ popal ; \ addl $4+4,%esp -/* - * Macros for interrupt entry, call to handler, and exit. - */ - -#define FAST_INTR(irq_num, vec_name) \ - .text ; \ - SUPERALIGN_TEXT ; \ -IDTVEC(vec_name) ; \ - PUSH_FRAME ; \ - movl $KDSEL,%eax ; \ - mov %ax,%ds ; \ - mov %ax,%es ; \ - movl $KPSEL,%eax ; \ - mov %ax,%fs ; \ - FAKE_MCOUNT(13*4(%esp)) ; \ - call critical_enter ; \ - movl PCPU(CURTHREAD),%ebx ; \ - incl TD_INTR_NESTING_LEVEL(%ebx) ; \ - pushl intr_unit + (irq_num) * 4 ; \ - call *intr_handler + (irq_num) * 4 ; /* do the work ASAP */ \ - addl $4, %esp ; \ - movl $0, lapic+LA_EOI ; \ - lock ; \ - incl cnt+V_INTR ; /* book-keeping can wait */ \ - movl intr_countp + (irq_num) * 4, %eax ; \ - lock ; \ - incl (%eax) ; \ - decl TD_INTR_NESTING_LEVEL(%ebx) ; \ - call critical_exit ; \ - MEXITCOUNT ; \ - jmp doreti +#define POP_DUMMY \ + addl $16*4,%esp #define IOAPICADDR(irq_num) CNAME(int_to_apicintpin) + 16 * (irq_num) + 8 #define REDIRIDX(irq_num) CNAME(int_to_apicintpin) + 16 * (irq_num) + 12 @@ -114,9 +93,9 @@ IDTVEC(vec_name) ; \ */ #define UNMASK_IRQ(irq_num) \ ICU_LOCK ; /* into critical reg */ \ - testl $IRQ_BIT(irq_num), _apic_imen ; \ + testl $IRQ_BIT(irq_num), apic_imen ; \ je 7f ; /* bit clear, not masked */ \ - andl $~IRQ_BIT(irq_num), _apic_imen ;/* clear mask bit */ \ + andl $~IRQ_BIT(irq_num), apic_imen ;/* clear mask bit */ \ movl IOAPICADDR(irq_num), %ecx ; /* ioapic addr */ \ movl REDIRIDX(irq_num), %eax ; /* get the index */ \ movl %eax, (%ecx) ; /* write the index */ \ @@ -126,6 +105,92 @@ IDTVEC(vec_name) ; \ 7: ; /* already unmasked */ \ ICU_UNLOCK +/* + * Test to see whether we are handling an edge or level triggered INT. + * Level-triggered INTs have to be unmasked. + */ +#define UNMASK_LEVEL_IRQ(irq_num) \ + testl $IRQ_BIT(irq_num), apic_pin_trigger ; \ + jz 9f ; /* edge, don't unmask */ \ + UNMASK_IRQ(irq_num) ; \ +9: + +/* + * Macros for interrupt entry, call to handler, and exit. + */ + +#define FAST_INTR(irq_num, vec_name) \ + .text ; \ + SUPERALIGN_TEXT ; \ +IDTVEC(vec_name) ; \ + PUSH_FRAME ; \ + movl $KDSEL,%eax ; \ + mov %ax,%ds ; \ + mov %ax,%es ; \ + movl $KPSEL,%eax ; \ + mov %ax,%fs ; \ + FAKE_MCOUNT(13*4(%esp)) ; \ + movl PCPU(CURTHREAD),%ebx ; \ + cmpl $0,TD_CRITNEST(%ebx) ; \ + je 1f ; \ +; \ + movl $1,PCPU(INT_PENDING) ; \ + orl $IRQ_BIT(irq_num),PCPU(FPENDING) ; \ + MASK_LEVEL_IRQ(irq_num) ; \ + movl $0, lapic+LA_EOI ; \ + jmp 10f ; \ +1: ; \ + incl TD_CRITNEST(%ebx) ; \ + incl TD_INTR_NESTING_LEVEL(%ebx) ; \ + pushl intr_unit + (irq_num) * 4 ; \ + call *intr_handler + (irq_num) * 4 ; /* do the work ASAP */ \ + addl $4, %esp ; \ + movl $0, lapic+LA_EOI ; \ + lock ; \ + incl cnt+V_INTR ; /* book-keeping can wait */ \ + movl intr_countp + (irq_num) * 4, %eax ; \ + lock ; \ + incl (%eax) ; \ + decl TD_CRITNEST(%ebx) ; \ + cmpl $0,PCPU(INT_PENDING) ; \ + je 2f ; \ +; \ + call unpend ; \ +2: ; \ + decl TD_INTR_NESTING_LEVEL(%ebx) ; \ +10: ; \ + MEXITCOUNT ; \ + jmp doreti + +/* + * Restart a fast interrupt that was held up by a critical section. + * This routine is called from unpend(). unpend() ensures we are + * in a critical section and deals with the interrupt nesting level + * for us. If we previously masked the irq, we have to unmask it. + * + * We have a choice. We can regenerate the irq using the 'int' + * instruction or we can create a dummy frame and call the interrupt + * handler directly. I've chosen to use the dummy-frame method. + */ +#define FAST_UNPEND(irq_num, vec_name) \ + .text ; \ + SUPERALIGN_TEXT ; \ +IDTVEC(vec_name) ; \ +; \ + PUSH_DUMMY ; \ + pushl intr_unit + (irq_num) * 4 ; \ + call *intr_handler + (irq_num) * 4 ; /* do the work ASAP */ \ + addl $4, %esp ; \ + lock ; \ + incl cnt+V_INTR ; /* book-keeping can wait */ \ + movl intr_countp + (irq_num) * 4, %eax ; \ + lock ; \ + incl (%eax) ; \ + UNMASK_LEVEL_IRQ(irq_num) ; \ + POP_DUMMY ; \ + ret ; \ + + /* * Slow, threaded interrupts. * @@ -151,16 +216,27 @@ IDTVEC(vec_name) ; \ ; \ MASK_LEVEL_IRQ(irq_num) ; \ EOI_IRQ(irq_num) ; \ -0: ; \ +; \ movl PCPU(CURTHREAD),%ebx ; \ + cmpl $0,TD_CRITNEST(%ebx) ; \ + je 1f ; \ + movl $1,PCPU(INT_PENDING) ; \ + orl $IRQ_BIT(irq_num),PCPU(IPENDING) ; \ + jmp 10f ; \ +1: ; \ incl TD_INTR_NESTING_LEVEL(%ebx) ; \ ; \ FAKE_MCOUNT(13*4(%esp)) ; /* XXX avoid dbl cnt */ \ + cmpl $0,PCPU(INT_PENDING) ; \ + je 9f ; \ + call unpend ; \ +9: ; \ pushl $irq_num; /* pass the IRQ */ \ call sched_ithd ; \ addl $4, %esp ; /* discard the parameter */ \ ; \ decl TD_INTR_NESTING_LEVEL(%ebx) ; \ +10: ; \ MEXITCOUNT ; \ jmp doreti @@ -304,9 +380,16 @@ Xhardclock: movl $0, lapic+LA_EOI /* End Of Interrupt to APIC */ movl PCPU(CURTHREAD),%ebx + cmpl $0,TD_CRITNEST(%ebx) + je 1f + movl $1,PCPU(INT_PENDING) + orl $1,PCPU(SPENDING); + jmp 10f +1: incl TD_INTR_NESTING_LEVEL(%ebx) call forwarded_hardclock decl TD_INTR_NESTING_LEVEL(%ebx) +10: MEXITCOUNT jmp doreti @@ -328,10 +411,18 @@ Xstatclock: movl $0, lapic+LA_EOI /* End Of Interrupt to APIC */ FAKE_MCOUNT(13*4(%esp)) + movl PCPU(CURTHREAD),%ebx + cmpl $0,TD_CRITNEST(%ebx) + je 1f + movl $1,PCPU(INT_PENDING) + orl $2,PCPU(SPENDING); + jmp 10f +1: incl TD_INTR_NESTING_LEVEL(%ebx) call forwarded_statclock decl TD_INTR_NESTING_LEVEL(%ebx) +10: MEXITCOUNT jmp doreti @@ -494,6 +585,39 @@ MCOUNT_LABEL(bintr) INTR(29,intr29,) INTR(30,intr30,) INTR(31,intr31,) + + FAST_UNPEND(0,fastunpend0) + FAST_UNPEND(1,fastunpend1) + FAST_UNPEND(2,fastunpend2) + FAST_UNPEND(3,fastunpend3) + FAST_UNPEND(4,fastunpend4) + FAST_UNPEND(5,fastunpend5) + FAST_UNPEND(6,fastunpend6) + FAST_UNPEND(7,fastunpend7) + FAST_UNPEND(8,fastunpend8) + FAST_UNPEND(9,fastunpend9) + FAST_UNPEND(10,fastunpend10) + FAST_UNPEND(11,fastunpend11) + FAST_UNPEND(12,fastunpend12) + FAST_UNPEND(13,fastunpend13) + FAST_UNPEND(14,fastunpend14) + FAST_UNPEND(15,fastunpend15) + FAST_UNPEND(16,fastunpend16) + FAST_UNPEND(17,fastunpend17) + FAST_UNPEND(18,fastunpend18) + FAST_UNPEND(19,fastunpend19) + FAST_UNPEND(20,fastunpend20) + FAST_UNPEND(21,fastunpend21) + FAST_UNPEND(22,fastunpend22) + FAST_UNPEND(23,fastunpend23) + FAST_UNPEND(24,fastunpend24) + FAST_UNPEND(25,fastunpend25) + FAST_UNPEND(26,fastunpend26) + FAST_UNPEND(27,fastunpend27) + FAST_UNPEND(28,fastunpend28) + FAST_UNPEND(29,fastunpend29) + FAST_UNPEND(30,fastunpend30) + FAST_UNPEND(31,fastunpend31) MCOUNT_LABEL(eintr) /* |