Merge from projects/mips to head by hand:

sorry for the huge firehose on this commit, it would be too tedious
to do file by file

r201881 | imp | 2010-01-08 20:08:22 -0700 (Fri, 08 Jan 2010) | 3 lines
Rename mips_pcpu_init to mips_pcpu0_init since it applies only to the
BSP.  Provide a missing prototype.

r201880 | neel | 2010-01-08 19:17:14 -0700 (Fri, 08 Jan 2010) | 7 lines
Compute the target of the jump in the 'J' and 'JAL' instructions
correctly. The 256MB segment is formed by taking the top 4 bits
of the address of the instruction in the "branch delay" slot
as opposed to the 'J' or 'JAL' instruction itself.

r201845 | imp | 2010-01-08 15:48:21 -0700 (Fri, 08 Jan 2010) | 2 lines
Centralize initialization of pcpu, and set curthread early...

r201770 | neel | 2010-01-07 22:53:11 -0700 (Thu, 07 Jan 2010) | 4 lines
Add a DDB command "show pcb" to dump out the contents of a thread's PCB.

r201631 | neel | 2010-01-05 23:42:08 -0700 (Tue, 05 Jan 2010) | 5 lines
Remove all CFE-specific code from locore.S. The CFE entrypoint initialization
is now done in platform-specific code.

r201563 | neel | 2010-01-04 23:58:54 -0700 (Mon, 04 Jan 2010) | 6 lines
This change increases the size of the kernel stack for thread0 from
PAGE_SIZE to (2 * PAGE_SIZE). It depends on the memory allocated by
pmap_steal_memory() being aligned to a PAGE_SIZE boundary.

r200656 | imp | 2009-12-17 16:55:49 -0700 (Thu, 17 Dec 2009) | 7 lines
Place holder ptrace mips module.  Not entirely sure what's required
here yet, so I've not connected it to the build.  I think that we'll
need to move something into the processor specific part of the mips
port by requiring mips_cpu_ptrace or platform_cpu_ptrace be provided
by the ports to get/set processor specific registers, ala SSE
registers on x86.

r200342 | imp | 2009-12-09 18:42:44 -0700 (Wed, 09 Dec 2009) | 4 lines
app_descriptor_addr is unused (I know it is referened still).  And
unnecessary since we pass in a3 unmodified to platform_start.
Eliminate it from here and kill one more TARGET_OCTEON in the process.

r199760 | imp | 2009-11-24 10:15:22 -0700 (Tue, 24 Nov 2009) | 2 lines
Add in Cavium's CID.  Report what the unknown CID is.

r199755 | imp | 2009-11-24 09:53:58 -0700 (Tue, 24 Nov 2009) | 5 lines
looks like there's more to this patch than just this one file.  I'll
leave it to neel@ to get all the relevant pieces into the tree.

r199754 | imp | 2009-11-24 09:32:31 -0700 (Tue, 24 Nov 2009) | 6 lines
Include opt_cputype.h for all .c and .S files referencing TARGET_OCTEON.
Spell ld script name right.
# for the most part, we need to enhance infrastructure to obviate the need
# for such an intrusive option.

r199753 | imp | 2009-11-24 09:30:29 -0700 (Tue, 24 Nov 2009) | 3 lines
Remove a comment that's bogus.
Include opt_cputype.h since TARGET_OCTEON moved there.

r199752 | imp | 2009-11-24 09:29:23 -0700 (Tue, 24 Nov 2009) | 4 lines
Make sure kstack0 is page aligned.
# this may have been from neel@ for the sibyte stuff

r199742 | imp | 2009-11-24 01:35:11 -0700 (Tue, 24 Nov 2009) | 8 lines
Move the hard-wiring of the dcache on octeon outside of the if
statement.  When no caches support was added, it looks like
TARGET_OCTEON was bogusly moved inside the if.  Also, include
opt_cputype.h to make TARGET_OCTEON actually active.

# now we die in pmap init somewhere...  Most likely because 32MB of RAM is
# too tight given the load address we're using.

r199741 | imp | 2009-11-24 01:21:48 -0700 (Tue, 24 Nov 2009) | 2 lines
TARGET_OCTEON reqiures opt_cputype.h.

r199736 | imp | 2009-11-24 00:40:38 -0700 (Tue, 24 Nov 2009) | 2 lines
Prefer ANSI spellings of uintXX_t, etc.

r199598 | imp | 2009-11-20 09:30:35 -0700 (Fri, 20 Nov 2009) | 3 lines
Horrible kludge to make octeon32 work.  I think a better way is to
move the generic code into the config files....

r199597 | imp | 2009-11-20 09:27:50 -0700 (Fri, 20 Nov 2009) | 4 lines
cast vaddr to uintptr_t before casting it to a bus_space_handle_t.
# I'm sure this indicates a problem, but I'm not sure what...

r199496 | gonzo | 2009-11-18 15:52:05 -0700 (Wed, 18 Nov 2009) | 5 lines
- Add cpu_init_interrupts function that is supposed to
    prepeare stuff required for spinning out interrupts later
- Add API for managing intrcnt/intrnames arrays
- Some minor style(9) fixes

r199246 | neel | 2009-11-13 02:24:09 -0700 (Fri, 13 Nov 2009) | 10 lines
Make pmap_copy_page() L2-cache friendly by doing the copy through the
cacheable window on physical memory (KSEG0). On the Sibyte processor
going through the uncacheable window (KSEG1) bypasses both L1 and L2
caches so we may end up with stale contents in the L2 cache.

This also makes it consistent with the rest of the function that
uses cacheable mappings to copy pages.

Approved by: imp (mentor)

r198842 | gonzo | 2009-11-02 23:42:55 -0700 (Mon, 02 Nov 2009) | 3 lines
- Handle errors when adding children to nexus. This sittuation
    might occure when there is dublicate of child's entry in hints

r198669 | rrs | 2009-10-30 02:53:11 -0600 (Fri, 30 Oct 2009) | 5 lines
With this commit our friend RMI will now compile. I have
not tested it and the chances of it running yet are about
ZERO.. but it will now compile. The hard part now begins,
 making it run ;-)

r198569 | neel | 2009-10-28 23:18:02 -0600 (Wed, 28 Oct 2009) | 5 lines
Deal with overflow of the COUNT register correctly. The 'cycles_per_hz'
has nothing to do with the rollover.

r198550 | imp | 2009-10-28 11:03:20 -0600 (Wed, 28 Oct 2009) | 3 lines
Remove useless for statement.  i isn't used after it.
Remove needless braces.

r198534 | gonzo | 2009-10-27 21:34:05 -0600 (Tue, 27 Oct 2009) | 8 lines
- Fix busdma sync: dcache invalidation operates on cache line aligned
  addresses and could modify areas of memory that share the same cache
  line at the beginning and at the ending of the buffer. In order to
  prevent a data loss we save these chunks in temporary buffer before
  invalidation and restore them afer it.
Idea suggested by: cognet

r198531 | gonzo | 2009-10-27 18:01:20 -0600 (Tue, 27 Oct 2009) | 3 lines
- Remove bunch of declared but not defined cach-related variables
- Add mips_picache_linesize and mips_pdcache_linesize variables

r198530 | gonzo | 2009-10-27 17:45:48 -0600 (Tue, 27 Oct 2009) | 3 lines
- Replace stubs with actual cache info
- minor style(9) fix

r198355 | neel | 2009-10-21 22:35:32 -0600 (Wed, 21 Oct 2009) | 11 lines
Remove redundant instructions from tlb.S
The "_MTC0 v0, COP_0_TLB_HI" is actually incorrect because v0 has not been
initialized at that point. It worked correctly because we subsequently
did the right thing and initialized TLB_HI correctly.
The "li v0, MIPS_KSEG0_START" is redundant because we do exactly the same
thing 2 instructions down.

r198354 | neel | 2009-10-21 20:51:31 -0600 (Wed, 21 Oct 2009) | 9 lines
Get rid of the hardcoded constants to define cacheable memory:
SDRAM_ADDR_START, SDRAM_ADDR_END and SDRAM_MEM_SIZE
Instead we now keep a copy of the memory regions enumerated by
platform-specific code and use that to determine whether an address
is cacheable or not.

r198310 | gonzo | 2009-10-20 17:13:08 -0600 (Tue, 20 Oct 2009) | 5 lines
- Commit missing part of "bt" fix: store PC register in pcb_context struct
    in cpu_switch and use it in stack_trace function later. pcb_regs contains
    state of the process stored by exception handler and therefor is not
    valid for sleeping processes.

r198264 | neel | 2009-10-19 22:36:08 -0600 (Mon, 19 Oct 2009) | 5 lines
Fix a bug where we would think that the L1 instruction and data cache are
present even though the line size field in the CP0 Config1 register is 0.

r198208 | imp | 2009-10-18 09:21:48 -0600 (Sun, 18 Oct 2009) | 3 lines
Get the PC from the trap frame, since it isn't saved as part of the
pcb regs.

r198205 | imp | 2009-10-18 08:55:55 -0600 (Sun, 18 Oct 2009) | 3 lines
Use correct signature for MipsEmulateBranch.  The other one doesn't
work for 64-bit compiles.

r198182 | gonzo | 2009-10-16 18:22:07 -0600 (Fri, 16 Oct 2009) | 11 lines
- Use PC/RA/SP values as arguments for stacktrace_subr instead of trapframe.
    Context info could be obtained from other sources (see below) no only from
    td_pcb field
- Do not show a0..a3 values unless they're obtained from the stack. These
    are only confirmed values.
- Fix bt command in DDB. Previous implementation used thread's trapframe
    structure as a source info for trace unwinding, but this structure
    is filled only when exception occurs. Valid register values for sleeping
    processes are in pcb_context array. For curthread use pc/sp/ra for current
    frame

r198181 | gonzo | 2009-10-16 16:52:18 -0600 (Fri, 16 Oct 2009) | 2 lines
- Get rid of label_t. It came from NetBSD and was used only in one place

r198066 | gonzo | 2009-10-13 19:43:53 -0600 (Tue, 13 Oct 2009) | 5 lines
- Move stack tracing function to db_trace.c
- Axe unused extern MipsXXX declarations
- Move all declarations for functions in exceptions.S/swtch.S
    from trap.c to respective headers

r197796 | gonzo | 2009-10-05 17:19:51 -0600 (Mon, 05 Oct 2009) | 2 lines
- Revert part of r197685 because this change leads to wrong data in cache.

r197685 | gonzo | 2009-10-01 14:05:36 -0600 (Thu, 01 Oct 2009) | 2 lines
- Sync caches properly when dealing with sf_buf

r197014 | imp | 2009-09-08 21:57:10 -0600 (Tue, 08 Sep 2009) | 2 lines
Ugly hack to get this to compile.  I'm sure there's a better way...

r197013 | imp | 2009-09-08 21:54:55 -0600 (Tue, 08 Sep 2009) | 2 lines
First half of making this 64-bit clean: fix prototypes.

r196988 | gonzo | 2009-09-08 13:15:29 -0600 (Tue, 08 Sep 2009) | 2 lines
- MFC from head@196987

r196313 | imp | 2009-08-17 06:14:40 -0600 (Mon, 17 Aug 2009) | 2 lines
suword64 and csuword64.  Needed by ELF64 stuff...

r196266 | imp | 2009-08-15 16:51:11 -0600 (Sat, 15 Aug 2009) | 5 lines
(1) Fix a few 32/64-bit bugs.
(2) Also, always allocate 2 pages for the stack to optimize TLB usage.
Submitted by:	neel@ (2)

r196265 | imp | 2009-08-15 16:48:09 -0600 (Sat, 15 Aug 2009) | 2 lines
Various 32/64-bit confusion cleanups.

r196264 | imp | 2009-08-15 16:45:46 -0600 (Sat, 15 Aug 2009) | 6 lines
(1) Some CPUs have a range to map I/O cyces on the pci bus.  So allow
them to work by allowding the nexus to assign ports.
(2) Remove some Octeon junk that shouldn't be necessary.
Submitted by:	neel@ (#1) for SB1 port.

r196061 | gonzo | 2009-08-04 11:32:55 -0600 (Tue, 04 Aug 2009) | 2 lines
- Use register_t for registers values

r195984 | gonzo | 2009-07-30 17:48:29 -0600 (Thu, 30 Jul 2009) | 4 lines
- Properly unwind stack for functions with __noreturn__ attribute
Submitted by:	Neelkanth Natu <neelnatu@yahoo.com>

r195983 | gonzo | 2009-07-30 17:29:59 -0600 (Thu, 30 Jul 2009) | 4 lines
- mark map as coherent if requested by flags
- explicitly set memory allocation method in map flags instead
    of duplicating conditions for malloc/contigalloc

r195584 | imp | 2009-07-10 13:09:34 -0600 (Fri, 10 Jul 2009) | 3 lines
Use PTR_* macros for pointers, and not potentially mips64 unsafe
operations.

r195583 | imp | 2009-07-10 13:08:48 -0600 (Fri, 10 Jul 2009) | 2 lines
Use PTR_* macros to deal with pointers.

r195579 | imp | 2009-07-10 13:04:32 -0600 (Fri, 10 Jul 2009) | 2 lines
use ta0-ta3 rather than t4-t7 for n32/n64 goodness.

r195511 | gonzo | 2009-07-09 13:02:17 -0600 (Thu, 09 Jul 2009) | 3 lines
- Ooops, this debug code wasn't supposed to get into
     final commit. My appologises.

r195478 | gonzo | 2009-07-08 16:28:36 -0600 (Wed, 08 Jul 2009) | 5 lines
- Port busdma code from FreeBSD/arm. This is more mature version
    that takes into account all limitation to DMA memory (boundaries,
    alignment) and implements bounce pages.
- Add BUS_DMASYNC_POSTREAD case to bus_dmamap_sync_buf

r195438 | imp | 2009-07-08 00:00:18 -0600 (Wed, 08 Jul 2009) | 2 lines
Turns out this code was right, revert last change.

r195429 | gonzo | 2009-07-07 13:55:09 -0600 (Tue, 07 Jul 2009) | 5 lines
- Move dpcpu initialization to mips_proc0_init. It's
    more appropriate place for it. Besides dpcpu_init
    requires pmap module to be initialized and calling it
    int pmap.c hangs the system

r195399 | imp | 2009-07-06 01:49:24 -0600 (Mon, 06 Jul 2009) | 2 lines
Prefer uintptr_t to int cast here.

r195398 | imp | 2009-07-06 01:48:31 -0600 (Mon, 06 Jul 2009) | 3 lines
Better types for 64-bit compatibility.  Use %p and cast to void * and
prefer uintptr_t to other int-type casts.

r195397 | imp | 2009-07-06 01:47:39 -0600 (Mon, 06 Jul 2009) | 2 lines
No need to force mips32 here.

r195396 | imp | 2009-07-06 01:46:13 -0600 (Mon, 06 Jul 2009) | 3 lines
Pass in the uint64 value, rather than a pointer to it.  that's what
the function expects...

r195395 | imp | 2009-07-06 01:45:02 -0600 (Mon, 06 Jul 2009) | 3 lines
Use ta0 instead of t4 and ta1 instead of t5.  These map to the same
registers on O32 builds, but t4 and t5 don't exist on N32 or N64.

r195394 | imp | 2009-07-06 01:43:50 -0600 (Mon, 06 Jul 2009) | 3 lines
Use better casts for passing the small integer as a pointer here.
Basically, replace int with uintptr_t.

r195393 | imp | 2009-07-06 01:42:54 -0600 (Mon, 06 Jul 2009) | 5 lines
(1) Improvements for SB1.  only allow real memory to be accessed.
(2) make compile n64 by using more-proper casts.
Submitted by:	Neelkanth Natu (1)

r195373 | imp | 2009-07-05 09:23:54 -0600 (Sun, 05 Jul 2009) | 5 lines
(1) Use PTR_LA rather than bare la for N64 goodness (it is dla there)
(2) SB1 needs COHERENT policy, not cached for the config register
Submitted by:	(2) Neelkanth Natu

r195372 | imp | 2009-07-05 09:22:22 -0600 (Sun, 05 Jul 2009) | 3 lines
use "PTR_LA" in preference to a bare la so it translates to dla on
64-bit ABIs.

r195371 | imp | 2009-07-05 09:21:35 -0600 (Sun, 05 Jul 2009) | 6 lines
Now that we define atomic_{load,store}_64 inline in atomic.h, we don't
need to define them here for the !N64 case.
We now define atomic_readandclear_64 in atomic.h, so no need to repeat
it here.

r195364 | imp | 2009-07-05 09:10:07 -0600 (Sun, 05 Jul 2009) | 5 lines
use %p in preference to 0x%08x for printing register_t values.  Cast
them to void * first.  This neatly solves the "how do I print a
register_t" problem because sizeof(void *) is always the same as
sizeof(register_t), afaik.

r195353 | imp | 2009-07-05 00:46:54 -0600 (Sun, 05 Jul 2009) | 6 lines
Publish PAGE_SHIFT to assembler
# we should likely phase out PGSHIFT
Submitted by:	Neelkanth Natu

r195350 | imp | 2009-07-05 00:39:37 -0600 (Sun, 05 Jul 2009) | 7 lines
Switch to ABI agnostic ta0-ta3.  Provide defs for this in the right
places.  Provide n32/n64 register name defintions.  This should have
no effect for the O32 builds that everybody else uses, but should help
make N64 builds possible (lots of other changes are needed for that).
Obtained from:	NetBSD (for the regdef.h changes)

r195334 | imp | 2009-07-03 21:22:34 -0600 (Fri, 03 Jul 2009) | 6 lines
Move from using the lame invalid address I chose when trying to get
Octeon going...  Turns out that you get tlb shutdowns with this...
Use PGSHIFT instead of PAGE_SHIFT.
Submitted by:	Neelkanth Natu

r195147 | gonzo | 2009-06-28 15:01:00 -0600 (Sun, 28 Jun 2009) | 2 lines
- Replace casuword and casuword32 stubs with proper implementation

r195128 | gonzo | 2009-06-27 17:27:41 -0600 (Sat, 27 Jun 2009) | 4 lines
- Add support for handling TLS area address in kernel space.
    From the userland point of view get/set operations are
    performed using sysarch(2) call.

r195127 | gonzo | 2009-06-27 17:01:35 -0600 (Sat, 27 Jun 2009) | 4 lines
- Make cpu_set_upcall_kse conform MIPS ABI. T9 should be
    the same as PC in subroutine entry point
- Preserve interrupt mask

r194938 | gonzo | 2009-06-24 20:15:04 -0600 (Wed, 24 Jun 2009) | 3 lines
- Invalidate cache in pmap_qenter. Fixes corruption of data
    that comes through pipe (may be other bugs)

r194505 | gonzo | 2009-06-19 13:02:40 -0600 (Fri, 19 Jun 2009) | 5 lines
- Keep interrupts mask intact by RESTORE_CPU in MipsKernGenException
    trap() function re-enables interrupts if exception happened with
    interrupts enabled and therefor status register might be modified
    by interrupt filters

r194277 | gonzo | 2009-06-15 20:36:21 -0600 (Mon, 15 Jun 2009) | 2 lines
- Remove debug printfs

r194275 | gonzo | 2009-06-15 19:43:33 -0600 (Mon, 15 Jun 2009) | 2 lines
- Handle KSEG0/KSEG1 addresses for /dev/mem as well. netstat requires it

r193491 | gonzo | 2009-06-05 03:21:03 -0600 (Fri, 05 Jun 2009) | 6 lines
- Status register should be set last in RESTORE_CPU in order
    to prevent race over k0, k1 registers.
- Update interrupts mask in saved status register for
    MipsUserIntr and MipsUserGenException. It might be
    modified by intr filter or ithread.

r192864 | gonzo | 2009-05-26 16:40:12 -0600 (Tue, 26 May 2009) | 4 lines
- Replace CPU_NOFPU and SOFTFLOAT options with CPU_FPU. By default
   we assume that there is no FPU, because majority of SoC does
   not have it.

r192794 | gonzo | 2009-05-26 00:20:50 -0600 (Tue, 26 May 2009) | 5 lines
- Preserve INT_MASK fields in Status register across
    context switches. They should be modified only by
    interrupt setup/teardown and pre_ithread/post_ithread
    functions

r192793 | gonzo | 2009-05-26 00:02:38 -0600 (Tue, 26 May 2009) | 2 lines
- Remove erroneus "break" instruction, it was meant for debug

r192792 | gonzo | 2009-05-26 00:01:17 -0600 (Tue, 26 May 2009) | 2 lines
- Remove now unused NetBSDism intr.h

r192791 | gonzo | 2009-05-25 23:59:05 -0600 (Mon, 25 May 2009) | 7 lines
- Provide proper pre_ithread/post_ithread functions for both
    hard and soft interrupts
- Do not handle masked interrupts
- Do not write Cause register because most bytes are read-only and
    writing the same byte to RW fields are pointless. And in case of
    software interrupt utterly wrong

r192664 | gonzo | 2009-05-23 13:42:23 -0600 (Sat, 23 May 2009) | 4 lines
- cpu_establish_hardintr modifies INT_MASK of Status
    register, so we should use disableintr/restoreintr that
    modifies only IE bit.

r192655 | gonzo | 2009-05-23 12:00:20 -0600 (Sat, 23 May 2009) | 6 lines
- Remove stale comments
- Replace a1 with k1 to while restoring context. a1 was there by mistake,
    interrupts are disabled at this point and it's safe to use k0, k1.
    This code never was reached beacasue current Status register handling
    prevented interrupta from user mode.

r192496 | gonzo | 2009-05-20 17:07:10 -0600 (Wed, 20 May 2009) | 4 lines
- Invalidate caches for respective areain KSEG0 in order
    to prevent further overwriting of KSEG1 data with
    writeback.

r192364 | gonzo | 2009-05-18 20:43:21 -0600 (Mon, 18 May 2009) | 6 lines
- Cleanup ticker initialization code. For some MIPS cpu Counter
    register increments only every second cycle. The only timing
    references for us is Count value. Therefore it's better to convert
    frequencies related to it and use them. Besides cleanup this commit
    fixes twice more then requested sleep interval problem.

r192176 | gonzo | 2009-05-15 20:34:03 -0600 (Fri, 15 May 2009) | 3 lines
- Add informational title for cache info lines to separate
    them from environment variables dump

r192119 | gonzo | 2009-05-14 15:26:07 -0600 (Thu, 14 May 2009) | 3 lines
- Off by one check fix. Check for last address in region
    to fit in KSEG1

r191841 | gonzo | 2009-05-05 20:55:43 -0600 (Tue, 05 May 2009) | 5 lines
- Use index ops in order to avoid TLBMiss exceptions when flushing caches
    on mapping removal
- Writeback all VA for page that is being copied in pmap_copy_page to
    guaranty up-to-date data in SDRAM

r191613 | gonzo | 2009-04-27 20:59:18 -0600 (Mon, 27 Apr 2009) | 4 lines
- When destroying va -> pa mapping writeback all caches or we may endup
    with partial page content in SDRAM
- style(9) fix

r191583 | gonzo | 2009-04-27 12:46:57 -0600 (Mon, 27 Apr 2009) | 5 lines
- Use new spacebus
- Be a bit more verbose on failures
- style(9) fixes
- Use default rid value of 0 instead of MIPS_MEM_RID (0x20)

r191577 | gonzo | 2009-04-27 12:29:59 -0600 (Mon, 27 Apr 2009) | 4 lines
- Use naming convention the same as MIPS spec does: eliminate _sel1 sufix
  and just use selector number. e.g. mips_rd_config_sel1 -> mips_rd_config1
- Add WatchHi/WatchLo accessors for selctors 1..3 (for debug purposes)

r191453 | gonzo | 2009-04-23 23:28:44 -0600 (Thu, 23 Apr 2009) | 4 lines
Fix cut'n'paste code. cfg3 should get the value of selector 3
Spotted by: thompa@

r191452 | gonzo | 2009-04-23 22:18:16 -0600 (Thu, 23 Apr 2009) | 2 lines
- Print supported CPU capabilities during stratup

r191448 | gonzo | 2009-04-23 21:38:51 -0600 (Thu, 23 Apr 2009) | 2 lines
- Fix whitespace to conform style(9)

r191282 | gonzo | 2009-04-19 16:02:14 -0600 (Sun, 19 Apr 2009) | 3 lines
- Make mips_bus_space_generic be of type bus_space_tag_t instead of
    struct bus_space and update all relevant places.

r191084 | gonzo | 2009-04-14 20:28:26 -0600 (Tue, 14 Apr 2009) | 6 lines
Use FreeBSD/arm approach for handling bus space access: space tag is a pointer
to bus_space structure that defines access methods and hence every bus can
define own accessors. Default space is mips_bus_space_generic. It's a simple
interface to physical memory, values are read with regard to host system
byte order.

r191083 | gonzo | 2009-04-14 19:47:52 -0600 (Tue, 14 Apr 2009) | 4 lines
- Cleanout stale #ifdef'ed chunk of code
- Fix whitespaces
- Explicitly undefine  NEXUS_DEBUG flag

r191079 | gonzo | 2009-04-14 16:53:22 -0600 (Tue, 14 Apr 2009) | 2 lines
- Revert changes accidentally killed by merge operation

------------------------------------------------------------------------
r187512 | gonzo | 2009-01-20 22:49:30 -0700 (Tue, 20 Jan 2009) | 4 lines
- Check if maddr/msize hints are there before setting hinted
    resources to device
- Check for irq hint too

r187418 | gonzo | 2009-01-18 19:37:10 -0700 (Sun, 18 Jan 2009) | 4 lines
- Add trampoline stuff for bootloaders that do not support ELF
- Replace arm'ish KERNPHYSADDR/KERNVIRTADDR with
    KERNLOADADDR/TRAMPLOADADDR and clean configs

1 files changed, 779 insertions, 166 deletions

diff --git a/sys/mips/mips/busdma_machdep.c b/sys/mips/mips/busdma_machdep.c
index 13c45b8..5271148 100644
--- a/sys/mips/mips/busdma_machdep.c
+++ b/sys/mips/mips/busdma_machdep.c
@@ -23,50 +23,16 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
+ *  From i386/busdma_machdep.c,v 1.26 2002/04/19 22:58:09 alfred
  */
 
-/*-
- * Copyright (c) 1997, 1998, 2001 The NetBSD Foundation, Inc.
- * All rights reserved.
- *
- * This code is derived from software contributed to The NetBSD Foundation
- * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility,
- * NASA Ames Research Center.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- * 3. All advertising materials mentioning features or use of this software
- *    must display the following acknowledgement:
- *	This product includes software developed by the NetBSD
- *	Foundation, Inc. and its contributors.
- * 4. Neither the name of The NetBSD Foundation nor the names of its
- *    contributors may be used to endorse or promote products derived
- *    from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
- * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
- * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
- * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
- * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
- */
-
-/*	$NetBSD: bus_dma.c,v 1.17 2006/03/01 12:38:11 yamt Exp $	*/
-
 #include <sys/cdefs.h>
 __FBSDID("$FreeBSD$");
 
+/*
+ * MIPS bus dma support routines
+ */
+
 #include <sys/param.h>
 #include <sys/systm.h>
 #include <sys/malloc.h>
@@ -79,6 +45,7 @@ __FBSDID("$FreeBSD$");
 #include <sys/uio.h>
 #include <sys/ktr.h>
 #include <sys/kernel.h>
+#include <sys/sysctl.h>
 
 #include <vm/vm.h>
 #include <vm/vm_page.h>
@@ -88,6 +55,13 @@ __FBSDID("$FreeBSD$");
 #include <machine/bus.h>
 #include <machine/cache.h>
 #include <machine/cpufunc.h>
+#include <machine/md_var.h>
+
+#define MAX_BPAGES 64
+#define BUS_DMA_COULD_BOUNCE	BUS_DMA_BUS3
+#define BUS_DMA_MIN_ALLOC_COMP	BUS_DMA_BUS4
+
+struct bounce_zone;
 
 struct bus_dma_tag {
 	bus_dma_tag_t		parent;
@@ -105,19 +79,59 @@ struct bus_dma_tag {
 	int			map_count;
 	bus_dma_lock_t		*lockfunc;
 	void			*lockfuncarg;
-	/* XXX: machine-dependent fields */
-	vm_offset_t		_physbase;
-	vm_offset_t		_wbase;
-	vm_offset_t		_wsize;
+	struct bounce_zone *bounce_zone;
 };
 
+struct bounce_page {
+	vm_offset_t	vaddr;		/* kva of bounce buffer */
+	vm_offset_t	vaddr_nocache;	/* kva of bounce buffer uncached */
+	bus_addr_t	busaddr;	/* Physical address */
+	vm_offset_t	datavaddr;	/* kva of client data */
+	bus_size_t	datacount;	/* client data count */
+	STAILQ_ENTRY(bounce_page) links;
+};
+
+int busdma_swi_pending;
+
+struct bounce_zone {
+	STAILQ_ENTRY(bounce_zone) links;
+	STAILQ_HEAD(bp_list, bounce_page) bounce_page_list;
+	int		total_bpages;
+	int		free_bpages;
+	int		reserved_bpages;
+	int		active_bpages;
+	int		total_bounced;
+	int		total_deferred;
+	int		map_count;
+	bus_size_t	alignment;
+	bus_addr_t	lowaddr;
+	char		zoneid[8];
+	char		lowaddrid[20];
+	struct sysctl_ctx_list sysctl_tree;
+	struct sysctl_oid *sysctl_tree_top;
+};
+
+static struct mtx bounce_lock;
+static int total_bpages;
+static int busdma_zonecount;
+static STAILQ_HEAD(, bounce_zone) bounce_zone_list;
+
+SYSCTL_NODE(_hw, OID_AUTO, busdma, CTLFLAG_RD, 0, "Busdma parameters");
+SYSCTL_INT(_hw_busdma, OID_AUTO, total_bpages, CTLFLAG_RD, &total_bpages, 0,
+	   "Total bounce pages");
+
 #define DMAMAP_LINEAR		0x1
 #define DMAMAP_MBUF		0x2
 #define DMAMAP_UIO		0x4
-#define DMAMAP_ALLOCATED	0x10
 #define DMAMAP_TYPE_MASK	(DMAMAP_LINEAR|DMAMAP_MBUF|DMAMAP_UIO)
 #define DMAMAP_COHERENT		0x8
+#define DMAMAP_ALLOCATED	0x10
+#define DMAMAP_MALLOCUSED	0x20
+
 struct bus_dmamap {
+	struct bp_list	bpages;
+	int		pagesneeded;
+	int		pagesreserved;
         bus_dma_tag_t	dmat;
 	int		flags;
 	void 		*buffer;
@@ -125,8 +139,15 @@ struct bus_dmamap {
 	void		*allocbuffer;
 	TAILQ_ENTRY(bus_dmamap)	freelist;
 	int		len;
+	STAILQ_ENTRY(bus_dmamap) links;
+	bus_dmamap_callback_t *callback;
+	void		      *callback_arg;
+
 };
 
+static STAILQ_HEAD(, bus_dmamap) bounce_map_waitinglist;
+static STAILQ_HEAD(, bus_dmamap) bounce_map_callbacklist;
+
 static TAILQ_HEAD(,bus_dmamap) dmamap_freelist = 
 	TAILQ_HEAD_INITIALIZER(dmamap_freelist);
 
@@ -137,6 +158,45 @@ static struct mtx busdma_mtx;
 
 MTX_SYSINIT(busdma_mtx, &busdma_mtx, "busdma lock", MTX_DEF);
 
+static void init_bounce_pages(void *dummy);
+static int alloc_bounce_zone(bus_dma_tag_t dmat);
+static int alloc_bounce_pages(bus_dma_tag_t dmat, u_int numpages);
+static int reserve_bounce_pages(bus_dma_tag_t dmat, bus_dmamap_t map,
+				int commit);
+static bus_addr_t add_bounce_page(bus_dma_tag_t dmat, bus_dmamap_t map,
+				   vm_offset_t vaddr, bus_size_t size);
+static void free_bounce_page(bus_dma_tag_t dmat, struct bounce_page *bpage);
+
+/* Default tag, as most drivers provide no parent tag. */
+bus_dma_tag_t mips_root_dma_tag;
+
+/*
+ * Return true if a match is made.
+ *
+ * To find a match walk the chain of bus_dma_tag_t's looking for 'paddr'.
+ *
+ * If paddr is within the bounds of the dma tag then call the filter callback
+ * to check for a match, if there is no filter callback then assume a match.
+ */
+static int
+run_filter(bus_dma_tag_t dmat, bus_addr_t paddr)
+{
+	int retval;
+
+	retval = 0;
+
+	do {
+		if (((paddr > dmat->lowaddr && paddr <= dmat->highaddr)
+		 || ((paddr & (dmat->alignment - 1)) != 0))
+		 && (dmat->filter == NULL
+		  || (*dmat->filter)(dmat->filterarg, paddr) != 0))
+			retval = 1;
+
+		dmat = dmat->parent;		
+	} while (retval == 0 && dmat != NULL);
+	return (retval);
+}
+
 static void
 mips_dmamap_freelist_init(void *dummy)
 {
@@ -157,6 +217,19 @@ bus_dmamap_load_buffer(bus_dma_tag_t dmat, bus_dma_segment_t *segs,
     bus_dmamap_t map, void *buf, bus_size_t buflen, struct pmap *pmap,
     int flags, vm_offset_t *lastaddrp, int *segp);
 
+static __inline int
+_bus_dma_can_bounce(vm_offset_t lowaddr, vm_offset_t highaddr)
+{
+	int i;
+	for (i = 0; phys_avail[i] && phys_avail[i + 1]; i += 2) {
+		if ((lowaddr >= phys_avail[i] && lowaddr <= phys_avail[i + 1])
+		    || (lowaddr < phys_avail[i] && 
+		    highaddr > phys_avail[i]))
+			return (1);
+	}
+	return (0);
+}
+
 /*
  * Convenience function for manipulating driver locks from busdma (during
  * busdma_swi, for example).  Drivers that don't provide their own locks
@@ -213,6 +286,7 @@ _busdma_alloc_dmamap(void)
 			map->flags = DMAMAP_ALLOCATED;
 	} else
 		map->flags = 0;
+	STAILQ_INIT(&map->bpages);
 	return (map);
 }
 
@@ -228,6 +302,11 @@ _busdma_free_dmamap(bus_dmamap_t map)
 	}
 }
 
+/*
+ * Allocate a device specific dma_tag.
+ */
+#define SEG_NB 1024
+
 int
 bus_dma_tag_create(bus_dma_tag_t parent, bus_size_t alignment,
 		   bus_size_t boundary, bus_addr_t lowaddr,
@@ -238,16 +317,12 @@ bus_dma_tag_create(bus_dma_tag_t parent, bus_size_t alignment,
 {
 	bus_dma_tag_t newtag;
 	int error = 0;
-
-	/* Basic sanity checking */
-	if (boundary != 0 && boundary < maxsegsz)
-		maxsegsz = boundary;
-
 	/* Return a NULL tag on failure */
 	*dmat = NULL;
+	if (!parent)
+		parent = mips_root_dma_tag;
 
-	newtag = (bus_dma_tag_t)malloc(sizeof(*newtag), M_DEVBUF,
-	    M_ZERO | M_NOWAIT);
+	newtag = (bus_dma_tag_t)malloc(sizeof(*newtag), M_DEVBUF, M_NOWAIT);
 	if (newtag == NULL) {
 		CTR4(KTR_BUSDMA, "%s returned tag %p tag flags 0x%x error %d",
 		    __func__, newtag, 0, error);
@@ -257,21 +332,16 @@ bus_dma_tag_create(bus_dma_tag_t parent, bus_size_t alignment,
 	newtag->parent = parent;
 	newtag->alignment = alignment;
 	newtag->boundary = boundary;
-	newtag->lowaddr = trunc_page((vm_paddr_t)lowaddr) + (PAGE_SIZE - 1);
-	newtag->highaddr = trunc_page((vm_paddr_t)highaddr) +
-	    (PAGE_SIZE - 1);
+	newtag->lowaddr = trunc_page((vm_offset_t)lowaddr) + (PAGE_SIZE - 1);
+	newtag->highaddr = trunc_page((vm_offset_t)highaddr) + (PAGE_SIZE - 1);
 	newtag->filter = filter;
 	newtag->filterarg = filterarg;
-	newtag->maxsize = maxsize;
-	newtag->nsegments = nsegments;
+        newtag->maxsize = maxsize;
+        newtag->nsegments = nsegments;
 	newtag->maxsegsz = maxsegsz;
 	newtag->flags = flags;
 	newtag->ref_count = 1; /* Count ourself */
 	newtag->map_count = 0;
-	newtag->_wbase = 0;
-	newtag->_physbase = 0;
-	/* XXXMIPS: Should we limit window size to amount of physical memory */
-	newtag->_wsize = MIPS_KSEG1_START - MIPS_KSEG0_START;
 	if (lockfunc != NULL) {
 		newtag->lockfunc = lockfunc;
 		newtag->lockfuncarg = lockfuncarg;
@@ -279,36 +349,68 @@ bus_dma_tag_create(bus_dma_tag_t parent, bus_size_t alignment,
 		newtag->lockfunc = dflt_lock;
 		newtag->lockfuncarg = NULL;
 	}
-
-	/* Take into account any restrictions imposed by our parent tag */
-	if (parent != NULL) {
-		newtag->lowaddr = MIN(parent->lowaddr, newtag->lowaddr);
-		newtag->highaddr = MAX(parent->highaddr, newtag->highaddr);
+        /*
+	 * Take into account any restrictions imposed by our parent tag
+	 */
+        if (parent != NULL) {
+                newtag->lowaddr = min(parent->lowaddr, newtag->lowaddr);
+                newtag->highaddr = max(parent->highaddr, newtag->highaddr);
 		if (newtag->boundary == 0)
 			newtag->boundary = parent->boundary;
 		else if (parent->boundary != 0)
-			newtag->boundary = MIN(parent->boundary,
+                	newtag->boundary = min(parent->boundary,
 					       newtag->boundary);
-		if (newtag->filter == NULL) {
-			/*
-			 * Short circuit looking at our parent directly
-			 * since we have encapsulated all of its information
-			 */
-			newtag->filter = parent->filter;
-			newtag->filterarg = parent->filterarg;
-			newtag->parent = parent->parent;
+		if ((newtag->filter != NULL) ||
+		    ((parent->flags & BUS_DMA_COULD_BOUNCE) != 0))
+			newtag->flags |= BUS_DMA_COULD_BOUNCE;
+                if (newtag->filter == NULL) {
+                        /*
+                         * Short circuit looking at our parent directly
+                         * since we have encapsulated all of its information
+                         */
+                        newtag->filter = parent->filter;
+                        newtag->filterarg = parent->filterarg;
+                        newtag->parent = parent->parent;
 		}
 		if (newtag->parent != NULL)
 			atomic_add_int(&parent->ref_count, 1);
 	}
+	if (_bus_dma_can_bounce(newtag->lowaddr, newtag->highaddr)
+	 || newtag->alignment > 1)
+		newtag->flags |= BUS_DMA_COULD_BOUNCE;
+
+	if (((newtag->flags & BUS_DMA_COULD_BOUNCE) != 0) &&
+	    (flags & BUS_DMA_ALLOCNOW) != 0) {
+		struct bounce_zone *bz;
+
+		/* Must bounce */
+
+		if ((error = alloc_bounce_zone(newtag)) != 0) {
+			free(newtag, M_DEVBUF);
+			return (error);
+		}
+		bz = newtag->bounce_zone;
+
+		if (ptoa(bz->total_bpages) < maxsize) {
+			int pages;
+
+			pages = atop(maxsize) - bz->total_bpages;
 
-	if (error != 0) {
+			/* Add pages to our bounce pool */
+			if (alloc_bounce_pages(newtag, pages) < pages)
+				error = ENOMEM;
+		}
+		/* Performed initial allocation */
+		newtag->flags |= BUS_DMA_MIN_ALLOC_COMP;
+	} else
+		newtag->bounce_zone = NULL;
+	if (error != 0)
 		free(newtag, M_DEVBUF);
-	} else {
+	else
 		*dmat = newtag;
-	}
 	CTR4(KTR_BUSDMA, "%s returned tag %p tag flags 0x%x error %d",
 	    __func__, newtag, (newtag != NULL ? newtag->flags : 0), error);
+
 	return (error);
 }
 
@@ -346,6 +448,7 @@ bus_dma_tag_destroy(bus_dma_tag_t dmat)
         return (0);
 }
 
+#include <sys/kdb.h>
 /*
  * Allocate a handle for mapping from kva/uva/physical
  * address space into bus device space.
@@ -354,9 +457,7 @@ int
 bus_dmamap_create(bus_dma_tag_t dmat, int flags, bus_dmamap_t *mapp)
 {
 	bus_dmamap_t newmap;
-#ifdef KTR
 	int error = 0;
-#endif
 
 	newmap = _busdma_alloc_dmamap();
 	if (newmap == NULL) {
@@ -365,13 +466,64 @@ bus_dmamap_create(bus_dma_tag_t dmat, int flags, bus_dmamap_t *mapp)
 	}
 	*mapp = newmap;
 	newmap->dmat = dmat;
+	newmap->allocbuffer = NULL;
 	dmat->map_count++;
 
+	/*
+	 * Bouncing might be required if the driver asks for an active
+	 * exclusion region, a data alignment that is stricter than 1, and/or
+	 * an active address boundary.
+	 */
+	if (dmat->flags & BUS_DMA_COULD_BOUNCE) {
+
+		/* Must bounce */
+		struct bounce_zone *bz;
+		int maxpages;
+
+		if (dmat->bounce_zone == NULL) {
+			if ((error = alloc_bounce_zone(dmat)) != 0) {
+				_busdma_free_dmamap(newmap);
+				*mapp = NULL;
+				return (error);
+			}
+		}
+		bz = dmat->bounce_zone;
+
+		/* Initialize the new map */
+		STAILQ_INIT(&((*mapp)->bpages));
+
+		/*
+		 * Attempt to add pages to our pool on a per-instance
+		 * basis up to a sane limit.
+		 */
+		maxpages = MAX_BPAGES;
+		if ((dmat->flags & BUS_DMA_MIN_ALLOC_COMP) == 0
+		 || (bz->map_count > 0 && bz->total_bpages < maxpages)) {
+			int pages;
+
+			pages = MAX(atop(dmat->maxsize), 1);
+			pages = MIN(maxpages - bz->total_bpages, pages);
+			pages = MAX(pages, 1);
+			if (alloc_bounce_pages(dmat, pages) < pages)
+				error = ENOMEM;
+
+			if ((dmat->flags & BUS_DMA_MIN_ALLOC_COMP) == 0) {
+				if (error == 0)
+					dmat->flags |= BUS_DMA_MIN_ALLOC_COMP;
+			} else {
+				error = 0;
+			}
+		}
+		bz->map_count++;
+	}
+
+	if (flags & BUS_DMA_COHERENT)
+	    newmap->flags |= DMAMAP_COHERENT;
+
 	CTR4(KTR_BUSDMA, "%s: tag %p tag flags 0x%x error %d",
 	    __func__, dmat, dmat->flags, error);
 
 	return (0);
-
 }
 
 /*
@@ -381,7 +533,15 @@ bus_dmamap_create(bus_dma_tag_t dmat, int flags, bus_dmamap_t *mapp)
 int
 bus_dmamap_destroy(bus_dma_tag_t dmat, bus_dmamap_t map)
 {
+
 	_busdma_free_dmamap(map);
+	if (STAILQ_FIRST(&map->bpages) != NULL) {
+		CTR3(KTR_BUSDMA, "%s: tag %p error %d",
+		    __func__, dmat, EBUSY);
+		return (EBUSY);
+	}
+	if (dmat->bounce_zone)
+		dmat->bounce_zone->map_count--;
         dmat->map_count--;
 	CTR2(KTR_BUSDMA, "%s: tag %p error 0", __func__, dmat);
         return (0);
@@ -416,9 +576,16 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags,
 	dmat->map_count++;
 	*mapp = newmap;
 	newmap->dmat = dmat;
+
+	if (flags & BUS_DMA_COHERENT)
+	    newmap->flags |= DMAMAP_COHERENT;
 	
-        if (dmat->maxsize <= PAGE_SIZE) {
+        if (dmat->maxsize <= PAGE_SIZE &&
+	   (dmat->alignment < dmat->maxsize) &&
+	   !_bus_dma_can_bounce(dmat->lowaddr, dmat->highaddr) && 
+	   !(flags & BUS_DMA_COHERENT)) {
                 *vaddr = malloc(dmat->maxsize, M_DEVBUF, mflags);
+		newmap->flags |= DMAMAP_MALLOCUSED;
         } else {
                 /*
                  * XXX Use Contigmalloc until it is merged into this facility
@@ -440,7 +607,7 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags,
 		   maxphys = dmat->lowaddr;
 		 }
                 *vaddr = contigmalloc(dmat->maxsize, M_DEVBUF, mflags,
-                    0ul, maxphys, dmat->alignment? dmat->alignment : 1ul,
+                    0ul, dmat->lowaddr, dmat->alignment? dmat->alignment : 1ul,
                     dmat->boundary);
         }
         if (*vaddr == NULL) {
@@ -451,6 +618,7 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags,
 		*mapp = NULL;
                 return (ENOMEM);
 	}
+
 	if (flags & BUS_DMA_COHERENT) {
 		void *tmpaddr = (void *)*vaddr;
 
@@ -463,10 +631,10 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags,
 			*vaddr = tmpaddr;
 		} else
 			newmap->origbuffer = newmap->allocbuffer = NULL;
-	} else 
+	} else
 		newmap->origbuffer = newmap->allocbuffer = NULL;
-        return (0);
 
+        return (0);
 }
 
 /*
@@ -481,15 +649,69 @@ bus_dmamem_free(bus_dma_tag_t dmat, void *vaddr, bus_dmamap_t map)
 		    ("Trying to freeing the wrong DMA buffer"));
 		vaddr = map->origbuffer;
 	}
-        if (dmat->maxsize <= PAGE_SIZE)
+
+        if (map->flags & DMAMAP_MALLOCUSED)
 		free(vaddr, M_DEVBUF);
-        else {
+        else
 		contigfree(vaddr, dmat->maxsize, M_DEVBUF);
-	}
+
 	dmat->map_count--;
 	_busdma_free_dmamap(map);
 	CTR3(KTR_BUSDMA, "%s: tag %p flags 0x%x", __func__, dmat, dmat->flags);
+}
+
+static int
+_bus_dmamap_count_pages(bus_dma_tag_t dmat, bus_dmamap_t map, pmap_t pmap,
+    void *buf, bus_size_t buflen, int flags)
+{
+	vm_offset_t vaddr;
+	vm_offset_t vendaddr;
+	bus_addr_t paddr;
+
+	if ((map->pagesneeded == 0)) {
+		CTR3(KTR_BUSDMA, "lowaddr= %d, boundary= %d, alignment= %d",
+		    dmat->lowaddr, dmat->boundary, dmat->alignment);
+		CTR2(KTR_BUSDMA, "map= %p, pagesneeded= %d",
+		    map, map->pagesneeded);
+		/*
+		 * Count the number of bounce pages
+		 * needed in order to complete this transfer
+		 */
+		vaddr = trunc_page((vm_offset_t)buf);
+		vendaddr = (vm_offset_t)buf + buflen;
+
+		while (vaddr < vendaddr) {
+			KASSERT(kernel_pmap == pmap, ("pmap is not kernel pmap"));
+			paddr = pmap_kextract(vaddr);
+			if (((dmat->flags & BUS_DMA_COULD_BOUNCE) != 0) &&
+			    run_filter(dmat, paddr) != 0)
+				map->pagesneeded++;
+			vaddr += PAGE_SIZE;
+		}
+		CTR1(KTR_BUSDMA, "pagesneeded= %d\n", map->pagesneeded);
+	}
+
+	/* Reserve Necessary Bounce Pages */
+	if (map->pagesneeded != 0) {
+		mtx_lock(&bounce_lock);
+		if (flags & BUS_DMA_NOWAIT) {
+			if (reserve_bounce_pages(dmat, map, 0) != 0) {
+				mtx_unlock(&bounce_lock);
+				return (ENOMEM);
+			}
+		} else {
+			if (reserve_bounce_pages(dmat, map, 1) != 0) {
+				/* Queue us for resources */
+				STAILQ_INSERT_TAIL(&bounce_map_waitinglist,
+				    map, links);
+				mtx_unlock(&bounce_lock);
+				return (EINPROGRESS);
+			}
+		}
+		mtx_unlock(&bounce_lock);
+	}
 
+	return (0);
 }
 
 /*
@@ -504,8 +726,7 @@ bus_dmamap_load_buffer(bus_dma_tag_t dmat, bus_dma_segment_t *segs,
     int flags, vm_offset_t *lastaddrp, int *segp)
 {
 	bus_size_t sgsize;
-	bus_size_t bmask;
-	vm_offset_t curaddr, lastaddr;
+	bus_addr_t curaddr, lastaddr, baddr, bmask;
 	vm_offset_t vaddr = (vm_offset_t)buf;
 	int seg;
 	int error = 0;
@@ -513,36 +734,48 @@ bus_dmamap_load_buffer(bus_dma_tag_t dmat, bus_dma_segment_t *segs,
 	lastaddr = *lastaddrp;
 	bmask = ~(dmat->boundary - 1);
 
+	if ((dmat->flags & BUS_DMA_COULD_BOUNCE) != 0) {
+		error = _bus_dmamap_count_pages(dmat, map, pmap, buf, buflen,
+		    flags);
+		if (error)
+			return (error);
+	}
+	CTR3(KTR_BUSDMA, "lowaddr= %d boundary= %d, "
+	    "alignment= %d", dmat->lowaddr, dmat->boundary, dmat->alignment);
+
 	for (seg = *segp; buflen > 0 ; ) {
 		/*
 		 * Get the physical address for this segment.
+		 *
+		 * XXX Don't support checking for coherent mappings
+		 * XXX in user address space.
 		 */
 		KASSERT(kernel_pmap == pmap, ("pmap is not kernel pmap"));
 		curaddr = pmap_kextract(vaddr);
 
 		/*
-		 * If we're beyond the current DMA window, indicate
-		 * that and try to fall back onto something else.
-		 */
-		if (curaddr < dmat->_physbase ||
-		    curaddr >= (dmat->_physbase + dmat->_wsize))
-			return (EINVAL);
-
-		/*
-		 * In a valid DMA range.  Translate the physical
-		 * memory address to an address in the DMA window.
-		 */
-		curaddr = (curaddr - dmat->_physbase) + dmat->_wbase;
-
-
-		/*
 		 * Compute the segment size, and adjust counts.
 		 */
 		sgsize = PAGE_SIZE - ((u_long)curaddr & PAGE_MASK);
+		if (sgsize > dmat->maxsegsz)
+			sgsize = dmat->maxsegsz;
 		if (buflen < sgsize)
 			sgsize = buflen;
 
 		/*
+		 * Make sure we don't cross any boundaries.
+		 */
+		if (dmat->boundary > 0) {
+			baddr = (curaddr + dmat->boundary) & bmask;
+			if (sgsize > (baddr - curaddr))
+				sgsize = (baddr - curaddr);
+		}
+		if (((dmat->flags & BUS_DMA_COULD_BOUNCE) != 0) &&
+		    map->pagesneeded != 0 && run_filter(dmat, curaddr)) {
+			curaddr = add_bounce_page(dmat, map, vaddr, sgsize);
+		}
+
+		/*
 		 * Insert chunk into a segment, coalescing with
 		 * the previous segment if possible.
 		 */
@@ -574,9 +807,8 @@ segdone:
 	 * Did we fit?
 	 */
 	if (buflen != 0)
-		error = EFBIG;
-
-	return error;
+		error = EFBIG; /* XXX better return value here? */
+	return (error);
 }
 
 /*
@@ -597,14 +829,17 @@ bus_dmamap_load(bus_dma_tag_t dmat, bus_dmamap_t map, void *buf,
 
 	KASSERT(dmat != NULL, ("dmatag is NULL"));
 	KASSERT(map != NULL, ("dmamap is NULL"));
+	map->callback = callback;
+	map->callback_arg = callback_arg;
 	map->flags &= ~DMAMAP_TYPE_MASK;
-	map->flags |= DMAMAP_LINEAR|DMAMAP_COHERENT;
+	map->flags |= DMAMAP_LINEAR;
 	map->buffer = buf;
 	map->len = buflen;
 	error = bus_dmamap_load_buffer(dmat,
 	    dm_segments, map, buf, buflen, kernel_pmap,
 	    flags, &lastaddr, &nsegs);
-
+	if (error == EINPROGRESS)
+		return (error);
 	if (error)
 		(*callback)(callback_arg, NULL, 0, error);
 	else
@@ -613,8 +848,7 @@ bus_dmamap_load(bus_dma_tag_t dmat, bus_dmamap_t map, void *buf,
 	CTR5(KTR_BUSDMA, "%s: tag %p tag flags 0x%x error %d nsegs %d",
 	    __func__, dmat, dmat->flags, nsegs + 1, error);
 
-	return (0);
-
+	return (error);
 }
 
 /*
@@ -635,10 +869,9 @@ bus_dmamap_load_mbuf(bus_dma_tag_t dmat, bus_dmamap_t map, struct mbuf *m0,
 	M_ASSERTPKTHDR(m0);
 
 	map->flags &= ~DMAMAP_TYPE_MASK;
-	map->flags |= DMAMAP_MBUF | DMAMAP_COHERENT;
+	map->flags |= DMAMAP_MBUF;
 	map->buffer = m0;
 	map->len = 0;
-
 	if (m0->m_pkthdr.len <= dmat->maxsize) {
 		vm_offset_t lastaddr = 0;
 		struct mbuf *m;
@@ -676,16 +909,14 @@ bus_dmamap_load_mbuf_sg(bus_dma_tag_t dmat, bus_dmamap_t map,
 			int flags)
 {
 	int error = 0;
-
 	M_ASSERTPKTHDR(m0);
 
 	flags |= BUS_DMA_NOWAIT;
 	*nsegs = -1;
 	map->flags &= ~DMAMAP_TYPE_MASK;
-	map->flags |= DMAMAP_MBUF | DMAMAP_COHERENT;
-	map->buffer = m0;
+	map->flags |= DMAMAP_MBUF;
+	map->buffer = m0;			
 	map->len = 0;
-
 	if (m0->m_pkthdr.len <= dmat->maxsize) {
 		vm_offset_t lastaddr = 0;
 		struct mbuf *m;
@@ -693,8 +924,9 @@ bus_dmamap_load_mbuf_sg(bus_dma_tag_t dmat, bus_dmamap_t map,
 		for (m = m0; m != NULL && error == 0; m = m->m_next) {
 			if (m->m_len > 0) {
 				error = bus_dmamap_load_buffer(dmat, segs, map,
-				    m->m_data, m->m_len, 
-				    kernel_pmap, flags, &lastaddr, nsegs);
+						m->m_data, m->m_len,
+						kernel_pmap, flags, &lastaddr,
+						nsegs);
 				map->len += m->m_len;
 			}
 		}
@@ -702,12 +934,11 @@ bus_dmamap_load_mbuf_sg(bus_dma_tag_t dmat, bus_dmamap_t map,
 		error = EINVAL;
 	}
 
+	/* XXX FIXME: Having to increment nsegs is really annoying */
 	++*nsegs;
 	CTR5(KTR_BUSDMA, "%s: tag %p tag flags 0x%x error %d nsegs %d",
 	    __func__, dmat, dmat->flags, error, *nsegs);
-
 	return (error);
-
 }
 
 /*
@@ -718,9 +949,65 @@ bus_dmamap_load_uio(bus_dma_tag_t dmat, bus_dmamap_t map, struct uio *uio,
     bus_dmamap_callback2_t *callback, void *callback_arg,
     int flags)
 {
+	vm_offset_t lastaddr = 0;
+#ifdef __CC_SUPPORTS_DYNAMIC_ARRAY_INIT
+	bus_dma_segment_t dm_segments[dmat->nsegments];
+#else
+	bus_dma_segment_t dm_segments[BUS_DMAMAP_NSEGS];
+#endif
+	int nsegs, i, error;
+	bus_size_t resid;
+	struct iovec *iov;
+	struct pmap *pmap;
 
-	panic("Unimplemented %s at %s:%d\n", __func__, __FILE__, __LINE__);
-	return (0);
+	resid = uio->uio_resid;
+	iov = uio->uio_iov;
+	map->flags &= ~DMAMAP_TYPE_MASK;
+	map->flags |= DMAMAP_UIO;
+	map->buffer = uio;
+	map->len = 0;
+
+	if (uio->uio_segflg == UIO_USERSPACE) {
+		KASSERT(uio->uio_td != NULL,
+		    ("bus_dmamap_load_uio: USERSPACE but no proc"));
+		/* XXX: pmap = vmspace_pmap(uio->uio_td->td_proc->p_vmspace); */
+		panic("can't do it yet");
+	} else
+		pmap = kernel_pmap;
+
+	error = 0;
+	nsegs = -1;
+	for (i = 0; i < uio->uio_iovcnt && resid != 0 && !error; i++) {
+		/*
+		 * Now at the first iovec to load.  Load each iovec
+		 * until we have exhausted the residual count.
+		 */
+		bus_size_t minlen =
+		    resid < iov[i].iov_len ? resid : iov[i].iov_len;
+		caddr_t addr = (caddr_t) iov[i].iov_base;
+
+		if (minlen > 0) {
+			error = bus_dmamap_load_buffer(dmat, dm_segments, map,
+			    addr, minlen, pmap, flags, &lastaddr, &nsegs);
+
+			map->len += minlen;
+			resid -= minlen;
+		}
+	}
+
+	if (error) {
+		/* 
+		 * force "no valid mappings" on error in callback.
+		 */
+		(*callback)(callback_arg, dm_segments, 0, 0, error);
+	} else {
+		(*callback)(callback_arg, dm_segments, nsegs+1,
+		    uio->uio_resid, error);
+	}
+
+	CTR5(KTR_BUSDMA, "%s: tag %p tag flags 0x%x error %d nsegs %d",
+	    __func__, dmat, dmat->flags, error, nsegs + 1);
+	return (error);
 }
 
 /*
@@ -729,25 +1016,78 @@ bus_dmamap_load_uio(bus_dma_tag_t dmat, bus_dmamap_t map, struct uio *uio,
 void
 _bus_dmamap_unload(bus_dma_tag_t dmat, bus_dmamap_t map)
 {
+	struct bounce_page *bpage;
 
+	map->flags &= ~DMAMAP_TYPE_MASK;
+	while ((bpage = STAILQ_FIRST(&map->bpages)) != NULL) {
+		STAILQ_REMOVE_HEAD(&map->bpages, links);
+		free_bounce_page(dmat, bpage);
+	}
 	return;
 }
 
-static __inline void
+static void
 bus_dmamap_sync_buf(void *buf, int len, bus_dmasync_op_t op)
 {
+	char tmp_cl[mips_pdcache_linesize], tmp_clend[mips_pdcache_linesize];
+	vm_offset_t buf_cl, buf_clend;
+	vm_size_t size_cl, size_clend;
+	int cache_linesize_mask = mips_pdcache_linesize - 1;
+
+	/*
+	 * dcache invalidation operates on cache line aligned addresses
+	 * and could modify areas of memory that share the same cache line
+	 * at the beginning and the ending of the buffer. In order to 
+	 * prevent a data loss we save these chunks in temporary buffer
+	 * before invalidation and restore them afer it
+	 */
+	buf_cl = (vm_offset_t)buf  & ~cache_linesize_mask;
+	size_cl = (vm_offset_t)buf  & cache_linesize_mask;
+	buf_clend = (vm_offset_t)buf + len;
+	size_clend = (mips_pdcache_linesize - 
+	    (buf_clend & cache_linesize_mask)) & cache_linesize_mask;
 
 	switch (op) {
+	case BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE:
+	case BUS_DMASYNC_POSTREAD:
+
+		/* 
+		 * Save buffers that might be modified by invalidation
+		 */
+		if (size_cl)
+			memcpy (tmp_cl, (void*)buf_cl, size_cl);
+		if (size_clend)
+			memcpy (tmp_clend, (void*)buf_clend, size_clend);
+		mips_dcache_inv_range((vm_offset_t)buf, len);
+		/* 
+		 * Restore them
+		 */
+		if (size_cl)
+			memcpy ((void*)buf_cl, tmp_cl, size_cl);
+		if (size_clend)
+			memcpy ((void*)buf_clend, tmp_clend, size_clend);
+		break;
+
 	case BUS_DMASYNC_PREREAD|BUS_DMASYNC_PREWRITE:
 		mips_dcache_wbinv_range((vm_offset_t)buf, len);
 		break;
 
 	case BUS_DMASYNC_PREREAD:
-#if 1
-		mips_dcache_wbinv_range((vm_offset_t)buf, len);
-#else
+		/* 
+		 * Save buffers that might be modified by invalidation
+		 */
+		if (size_cl)
+			memcpy (tmp_cl, (void *)buf_cl, size_cl);
+		if (size_clend)
+			memcpy (tmp_clend, (void *)buf_clend, size_clend);
 		mips_dcache_inv_range((vm_offset_t)buf, len);
-#endif
+		/*
+		 * Restore them
+		 */
+		if (size_cl)
+			memcpy ((void *)buf_cl, tmp_cl, size_cl);
+		if (size_clend)
+			memcpy ((void *)buf_clend, tmp_clend, size_clend);
 		break;
 
 	case BUS_DMASYNC_PREWRITE:
@@ -756,6 +1096,51 @@ bus_dmamap_sync_buf(void *buf, int len, bus_dmasync_op_t op)
 	}
 }
 
+static void
+_bus_dmamap_sync_bp(bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op)
+{
+	struct bounce_page *bpage;
+
+	STAILQ_FOREACH(bpage, &map->bpages, links) {
+		if (op & BUS_DMASYNC_PREWRITE) {
+			bcopy((void *)bpage->datavaddr,
+			    (void *)(bpage->vaddr_nocache != 0 ? 
+				     bpage->vaddr_nocache : bpage->vaddr),
+			    bpage->datacount);
+			if (bpage->vaddr_nocache == 0) {
+				mips_dcache_wb_range(bpage->vaddr,
+				    bpage->datacount);
+			}
+			dmat->bounce_zone->total_bounced++;
+		}
+		if (op & BUS_DMASYNC_POSTREAD) {
+			if (bpage->vaddr_nocache == 0) {
+				mips_dcache_inv_range(bpage->vaddr,
+				    bpage->datacount);
+			}
+			bcopy((void *)(bpage->vaddr_nocache != 0 ? 
+	       		    bpage->vaddr_nocache : bpage->vaddr),
+			    (void *)bpage->datavaddr, bpage->datacount);
+			dmat->bounce_zone->total_bounced++;
+		}
+	}
+}
+
+static __inline int
+_bus_dma_buf_is_in_bp(bus_dmamap_t map, void *buf, int len)
+{
+	struct bounce_page *bpage;
+
+	STAILQ_FOREACH(bpage, &map->bpages, links) {
+		if ((vm_offset_t)buf >= bpage->datavaddr &&
+		    (vm_offset_t)buf + len <= bpage->datavaddr + 
+		    bpage->datacount)
+			return (1);
+	}
+	return (0);
+
+}
+
 void
 _bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op)
 {
@@ -764,51 +1149,23 @@ _bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op)
 	int resid;
 	struct iovec *iov;
 	
-
-	/*
-	 * Mixing PRE and POST operations is not allowed.
-	 */
-	if ((op & (BUS_DMASYNC_PREREAD|BUS_DMASYNC_PREWRITE)) != 0 &&
-	    (op & (BUS_DMASYNC_POSTREAD|BUS_DMASYNC_POSTWRITE)) != 0)
-		panic("_bus_dmamap_sync: mix PRE and POST");
-
-	/*
-	 * Since we're dealing with a virtually-indexed, write-back
-	 * cache, we need to do the following things:
-	 *
-	 *	PREREAD -- Invalidate D-cache.  Note we might have
-	 *	to also write-back here if we have to use an Index
-	 *	op, or if the buffer start/end is not cache-line aligned.
-	 *
-	 *	PREWRITE -- Write-back the D-cache.  If we have to use
-	 *	an Index op, we also have to invalidate.  Note that if
-	 *	we are doing PREREAD|PREWRITE, we can collapse everything
-	 *	into a single op.
-	 *
-	 *	POSTREAD -- Nothing.
-	 *
-	 *	POSTWRITE -- Nothing.
-	 */
-
-	/*
-	 * Flush the write buffer.
-	 * XXX Is this always necessary?
-	 */
-	mips_wbflush();
-
-	op &= (BUS_DMASYNC_PREREAD|BUS_DMASYNC_PREWRITE);
-	if (op == 0)
+	if (op == BUS_DMASYNC_POSTWRITE)
+		return;
+	if (STAILQ_FIRST(&map->bpages))
+		_bus_dmamap_sync_bp(dmat, map, op);
+	if (map->flags & DMAMAP_COHERENT)
 		return;
-
 	CTR3(KTR_BUSDMA, "%s: op %x flags %x", __func__, op, map->flags);
 	switch(map->flags & DMAMAP_TYPE_MASK) {
 	case DMAMAP_LINEAR:
-		bus_dmamap_sync_buf(map->buffer, map->len, op);
+		if (!(_bus_dma_buf_is_in_bp(map, map->buffer, map->len)))
+			bus_dmamap_sync_buf(map->buffer, map->len, op);
 		break;
 	case DMAMAP_MBUF:
 		m = map->buffer;
 		while (m) {
-			if (m->m_len > 0)
+			if (m->m_len > 0 &&
+			    !(_bus_dma_buf_is_in_bp(map, m->m_data, m->m_len)))
 				bus_dmamap_sync_buf(m->m_data, m->m_len, op);
 			m = m->m_next;
 		}
@@ -821,7 +1178,10 @@ _bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op)
 			bus_size_t minlen = resid < iov[i].iov_len ? resid :
 			    iov[i].iov_len;
 			if (minlen > 0) {
-				bus_dmamap_sync_buf(iov[i].iov_base, minlen, op);
+				if (!_bus_dma_buf_is_in_bp(map, iov[i].iov_base,
+				    minlen))
+					bus_dmamap_sync_buf(iov[i].iov_base,
+					    minlen, op);
 				resid -= minlen;
 			}
 		}
@@ -830,3 +1190,256 @@ _bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op)
 		break;
 	}
 }
+
+static void
+init_bounce_pages(void *dummy __unused)
+{
+
+	total_bpages = 0;
+	STAILQ_INIT(&bounce_zone_list);
+	STAILQ_INIT(&bounce_map_waitinglist);
+	STAILQ_INIT(&bounce_map_callbacklist);
+	mtx_init(&bounce_lock, "bounce pages lock", NULL, MTX_DEF);
+}
+SYSINIT(bpages, SI_SUB_LOCK, SI_ORDER_ANY, init_bounce_pages, NULL);
+
+static struct sysctl_ctx_list *
+busdma_sysctl_tree(struct bounce_zone *bz)
+{
+	return (&bz->sysctl_tree);
+}
+
+static struct sysctl_oid *
+busdma_sysctl_tree_top(struct bounce_zone *bz)
+{
+	return (bz->sysctl_tree_top);
+}
+
+static int
+alloc_bounce_zone(bus_dma_tag_t dmat)
+{
+	struct bounce_zone *bz;
+
+	/* Check to see if we already have a suitable zone */
+	STAILQ_FOREACH(bz, &bounce_zone_list, links) {
+		if ((dmat->alignment <= bz->alignment)
+		 && (dmat->lowaddr >= bz->lowaddr)) {
+			dmat->bounce_zone = bz;
+			return (0);
+		}
+	}
+
+	if ((bz = (struct bounce_zone *)malloc(sizeof(*bz), M_DEVBUF,
+	    M_NOWAIT | M_ZERO)) == NULL)
+		return (ENOMEM);
+
+	STAILQ_INIT(&bz->bounce_page_list);
+	bz->free_bpages = 0;
+	bz->reserved_bpages = 0;
+	bz->active_bpages = 0;
+	bz->lowaddr = dmat->lowaddr;
+	bz->alignment = MAX(dmat->alignment, PAGE_SIZE);
+	bz->map_count = 0;
+	snprintf(bz->zoneid, 8, "zone%d", busdma_zonecount);
+	busdma_zonecount++;
+	snprintf(bz->lowaddrid, 18, "%#jx", (uintmax_t)bz->lowaddr);
+	STAILQ_INSERT_TAIL(&bounce_zone_list, bz, links);
+	dmat->bounce_zone = bz;
+
+	sysctl_ctx_init(&bz->sysctl_tree);
+	bz->sysctl_tree_top = SYSCTL_ADD_NODE(&bz->sysctl_tree,
+	    SYSCTL_STATIC_CHILDREN(_hw_busdma), OID_AUTO, bz->zoneid,
+	    CTLFLAG_RD, 0, "");
+	if (bz->sysctl_tree_top == NULL) {
+		sysctl_ctx_free(&bz->sysctl_tree);
+		return (0);	/* XXX error code? */
+	}
+
+	SYSCTL_ADD_INT(busdma_sysctl_tree(bz),
+	    SYSCTL_CHILDREN(busdma_sysctl_tree_top(bz)), OID_AUTO,
+	    "total_bpages", CTLFLAG_RD, &bz->total_bpages, 0,
+	    "Total bounce pages");
+	SYSCTL_ADD_INT(busdma_sysctl_tree(bz),
+	    SYSCTL_CHILDREN(busdma_sysctl_tree_top(bz)), OID_AUTO,
+	    "free_bpages", CTLFLAG_RD, &bz->free_bpages, 0,
+	    "Free bounce pages");
+	SYSCTL_ADD_INT(busdma_sysctl_tree(bz),
+	    SYSCTL_CHILDREN(busdma_sysctl_tree_top(bz)), OID_AUTO,
+	    "reserved_bpages", CTLFLAG_RD, &bz->reserved_bpages, 0,
+	    "Reserved bounce pages");
+	SYSCTL_ADD_INT(busdma_sysctl_tree(bz),
+	    SYSCTL_CHILDREN(busdma_sysctl_tree_top(bz)), OID_AUTO,
+	    "active_bpages", CTLFLAG_RD, &bz->active_bpages, 0,
+	    "Active bounce pages");
+	SYSCTL_ADD_INT(busdma_sysctl_tree(bz),
+	    SYSCTL_CHILDREN(busdma_sysctl_tree_top(bz)), OID_AUTO,
+	    "total_bounced", CTLFLAG_RD, &bz->total_bounced, 0,
+	    "Total bounce requests");
+	SYSCTL_ADD_INT(busdma_sysctl_tree(bz),
+	    SYSCTL_CHILDREN(busdma_sysctl_tree_top(bz)), OID_AUTO,
+	    "total_deferred", CTLFLAG_RD, &bz->total_deferred, 0,
+	    "Total bounce requests that were deferred");
+	SYSCTL_ADD_STRING(busdma_sysctl_tree(bz),
+	    SYSCTL_CHILDREN(busdma_sysctl_tree_top(bz)), OID_AUTO,
+	    "lowaddr", CTLFLAG_RD, bz->lowaddrid, 0, "");
+	SYSCTL_ADD_INT(busdma_sysctl_tree(bz),
+	    SYSCTL_CHILDREN(busdma_sysctl_tree_top(bz)), OID_AUTO,
+	    "alignment", CTLFLAG_RD, &bz->alignment, 0, "");
+
+	return (0);
+}
+
+static int
+alloc_bounce_pages(bus_dma_tag_t dmat, u_int numpages)
+{
+	struct bounce_zone *bz;
+	int count;
+
+	bz = dmat->bounce_zone;
+	count = 0;
+	while (numpages > 0) {
+		struct bounce_page *bpage;
+
+		bpage = (struct bounce_page *)malloc(sizeof(*bpage), M_DEVBUF,
+						     M_NOWAIT | M_ZERO);
+
+		if (bpage == NULL)
+			break;
+		bpage->vaddr = (vm_offset_t)contigmalloc(PAGE_SIZE, M_DEVBUF,
+							 M_NOWAIT, 0ul,
+							 bz->lowaddr,
+							 PAGE_SIZE,
+							 0);
+		if (bpage->vaddr == 0) {
+			free(bpage, M_DEVBUF);
+			break;
+		}
+		bpage->busaddr = pmap_kextract(bpage->vaddr);
+		bpage->vaddr_nocache = 
+		    (vm_offset_t)MIPS_PHYS_TO_KSEG1(bpage->busaddr);
+		mtx_lock(&bounce_lock);
+		STAILQ_INSERT_TAIL(&bz->bounce_page_list, bpage, links);
+		total_bpages++;
+		bz->total_bpages++;
+		bz->free_bpages++;
+		mtx_unlock(&bounce_lock);
+		count++;
+		numpages--;
+	}
+	return (count);
+}
+
+static int
+reserve_bounce_pages(bus_dma_tag_t dmat, bus_dmamap_t map, int commit)
+{
+	struct bounce_zone *bz;
+	int pages;
+
+	mtx_assert(&bounce_lock, MA_OWNED);
+	bz = dmat->bounce_zone;
+	pages = MIN(bz->free_bpages, map->pagesneeded - map->pagesreserved);
+	if (commit == 0 && map->pagesneeded > (map->pagesreserved + pages))
+		return (map->pagesneeded - (map->pagesreserved + pages));
+	bz->free_bpages -= pages;
+	bz->reserved_bpages += pages;
+	map->pagesreserved += pages;
+	pages = map->pagesneeded - map->pagesreserved;
+
+	return (pages);
+}
+
+static bus_addr_t
+add_bounce_page(bus_dma_tag_t dmat, bus_dmamap_t map, vm_offset_t vaddr,
+		bus_size_t size)
+{
+	struct bounce_zone *bz;
+	struct bounce_page *bpage;
+
+	KASSERT(dmat->bounce_zone != NULL, ("no bounce zone in dma tag"));
+	KASSERT(map != NULL, ("add_bounce_page: bad map %p", map));
+
+	bz = dmat->bounce_zone;
+	if (map->pagesneeded == 0)
+		panic("add_bounce_page: map doesn't need any pages");
+	map->pagesneeded--;
+
+	if (map->pagesreserved == 0)
+		panic("add_bounce_page: map doesn't need any pages");
+	map->pagesreserved--;
+
+	mtx_lock(&bounce_lock);
+	bpage = STAILQ_FIRST(&bz->bounce_page_list);
+	if (bpage == NULL)
+		panic("add_bounce_page: free page list is empty");
+
+	STAILQ_REMOVE_HEAD(&bz->bounce_page_list, links);
+	bz->reserved_bpages--;
+	bz->active_bpages++;
+	mtx_unlock(&bounce_lock);
+
+	if (dmat->flags & BUS_DMA_KEEP_PG_OFFSET) {
+		/* Page offset needs to be preserved. */
+		bpage->vaddr |= vaddr & PAGE_MASK;
+		bpage->busaddr |= vaddr & PAGE_MASK;
+	}
+	bpage->datavaddr = vaddr;
+	bpage->datacount = size;
+	STAILQ_INSERT_TAIL(&(map->bpages), bpage, links);
+	return (bpage->busaddr);
+}
+
+static void
+free_bounce_page(bus_dma_tag_t dmat, struct bounce_page *bpage)
+{
+	struct bus_dmamap *map;
+	struct bounce_zone *bz;
+
+	bz = dmat->bounce_zone;
+	bpage->datavaddr = 0;
+	bpage->datacount = 0;
+	if (dmat->flags & BUS_DMA_KEEP_PG_OFFSET) {
+		/*
+		 * Reset the bounce page to start at offset 0.  Other uses
+		 * of this bounce page may need to store a full page of
+		 * data and/or assume it starts on a page boundary.
+		 */
+		bpage->vaddr &= ~PAGE_MASK;
+		bpage->busaddr &= ~PAGE_MASK;
+	}
+
+	mtx_lock(&bounce_lock);
+	STAILQ_INSERT_HEAD(&bz->bounce_page_list, bpage, links);
+	bz->free_bpages++;
+	bz->active_bpages--;
+	if ((map = STAILQ_FIRST(&bounce_map_waitinglist)) != NULL) {
+		if (reserve_bounce_pages(map->dmat, map, 1) == 0) {
+			STAILQ_REMOVE_HEAD(&bounce_map_waitinglist, links);
+			STAILQ_INSERT_TAIL(&bounce_map_callbacklist,
+					   map, links);
+			busdma_swi_pending = 1;
+			bz->total_deferred++;
+			swi_sched(vm_ih, 0);
+		}
+	}
+	mtx_unlock(&bounce_lock);
+}
+
+void
+busdma_swi(void)
+{
+	bus_dma_tag_t dmat;
+	struct bus_dmamap *map;
+
+	mtx_lock(&bounce_lock);
+	while ((map = STAILQ_FIRST(&bounce_map_callbacklist)) != NULL) {
+		STAILQ_REMOVE_HEAD(&bounce_map_callbacklist, links);
+		mtx_unlock(&bounce_lock);
+		dmat = map->dmat;
+		(dmat->lockfunc)(dmat->lockfuncarg, BUS_DMA_LOCK);
+		bus_dmamap_load(map->dmat, map, map->buffer, map->len,
+		    map->callback, map->callback_arg, /*flags*/0);
+		(dmat->lockfunc)(dmat->lockfuncarg, BUS_DMA_UNLOCK);
+		mtx_lock(&bounce_lock);
+	}
+	mtx_unlock(&bounce_lock);
+}