summaryrefslogtreecommitdiffstats
path: root/sys/vm/vm_contig.c
Commit message (Collapse)AuthorAgeFilesLines
* Move what remains of vm/vm_contig.c into vm/vm_pageout.c, where similaralc2012-07-181-231/+0
| | | | | | code resides. Rename vm_contig_grow_cache() to vm_pageout_grow_cache(). Reviewed by: kib
* Various improvements to vm_contig_grow_cache(). Most notably, even whenalc2012-07-161-22/+31
| | | | | | | | | | it can't sleep, it can still move clean pages from the inactive queue to the cache. Also, when a page is cached, there is no need to restart the scan. The "next" page pointer held by vm_contig_launder() is still valid. Finally, add a comment summarizing what vm_contig_grow_cache() does based upon the value of "tries". MFC after: 3 weeks
* If vm_contig_grow_cache() is allowed to sleep, then invoke the vm_lowmemalc2012-07-141-0/+15
| | | | handlers.
* Move kmem_alloc_{attr,contig}() to vm/vm_kern.c, where similarly namedalc2012-07-141-144/+0
| | | | functions reside. Correct the comment describing kmem_alloc_contig().
* In vm_object_page_clean(), do not clean OBJ_MIGHTBEDIRTY object flagkib2012-03-171-1/+2
| | | | | | | | | | | | | | | | | | if the filesystem performed short write and we are skipping the page due to this. Propogate write error from the pager back to the callers of vm_pageout_flush(). Report the failure to write a page from the requested range as the FALSE return value from vm_object_page_clean(), and propagate it back to msync(2) to return EIO to usermode. While there, convert the clearobjflags variable in the vm_object_page_clean() and arguments of the helper functions to boolean. PR: kern/165927 Reviewed by: alc MFC after: 2 weeks
* exclude kmem_alloc'ed ARC data buffers from kernel minidumps on amd64kmacy2012-01-271-0/+2
| | | | | | | | excluding other allocations including UMA now entails the addition of a single flag to kmem_alloc or uma zone create Reviewed by: alc, avg MFC after: 2 weeks
* Refactor the code that performs physically contiguous memory allocation,alc2011-11-161-67/+52
| | | | | | | | | | | | | | | | | | | | | | | | yielding a new public interface, vm_page_alloc_contig(). This new function addresses some of the limitations of the current interfaces, contigmalloc() and kmem_alloc_contig(). For example, the physically contiguous memory that is allocated with those interfaces can only be allocated to the kernel vm object and must be mapped into the kernel virtual address space. It also provides functionality that vm_phys_alloc_contig() doesn't, such as wiring the returned pages. Moreover, unlike that function, it respects the low water marks on the paging queues and wakes up the page daemon when necessary. That said, at present, this new function can't be applied to all types of vm objects. However, that restriction will be eliminated in the coming weeks. From a design standpoint, this change also addresses an inconsistency between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions. Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew about vnodes and reservations. Now, vm_page_alloc_contig() is responsible for these things. Reviewed by: kib Discussed with: jhb
* Simplify the implementation of the failure case in kmem_alloc_attr().alc2011-11-041-8/+7
|
* Eliminate vm_phys_bootstrap_alloc(). It was a failed attempt atalc2011-10-301-1/+2
| | | | | | | | | | | | | | | | | eliminating duplicated code in the various pmap implementations. Micro-optimize vm_phys_free_pages(). Introduce vm_phys_free_contig(). It is fast routine for freeing an arbitrary number of physically contiguous pages. In particular, it doesn't require the number of pages to be a power of two. Use "u_long" instead of "unsigned long". Bruce Evans (bde@) has convinced me that the "boundary" parameters to kmem_alloc_contig(), vm_phys_alloc_contig(), and vm_reserv_reclaim_contig() should be of type "vm_paddr_t" and not "u_long". Make this change.
* Use "u_long" instead of "unsigned long".alc2011-10-281-3/+2
|
* contigmalloc(9) and contigfree(9) are now implemented in terms of otheralc2011-10-271-28/+0
| | | | | | more general VM system interfaces. So, their implementation can now reside in kern_malloc.c alongside the other functions that are declared in malloc.h.
* Fix an error in kmem_alloc_attr(). Unless "tries" is updated,alc2011-08-071-0/+1
| | | | | | | kmem_alloc_attr() could get stuck in a loop. Approved by: re (kib) MFC after: 3 days
* Remove unneeded includes of <sys/linker_set.h>. Other headers that usejhb2011-01-111-1/+0
| | | | | | it internally contain nested includes. Reviewed by: bde
* There is no point in vm_contig_launder{,_page}() flushing held pages,alc2010-12-291-5/+4
| | | | | | | | | instead skip over them. As long as a page is held, it can't be reclaimed by contigmalloc(M_WAITOK). Moreover, a held page may be undergoing modification, e.g., vmapbuf(), so even if the hold were released before the completion of contigmalloc(), the page might have to be flushed again. MFC after: 3 weeks
* vm_pageout_flush() might cache the pages that finished write to thekib2010-11-181-1/+1
| | | | | | | | | | | | | | | | | | | backing storage. Such pages might be then reused, racing with the assert in vm_object_page_collect_flush() that verified that dirty pages from the run (most likely, pages with VM_PAGER_AGAIN status) are write-protected still. In fact, the page indexes for the pages that were removed from the object page list should be ignored by vm_object_page_clean(). Return the length of successfully written run from vm_pageout_flush(), that is, the count of pages between requested page and first page after requested with status VM_PAGER_AGAIN. Supply the requested page index in the array to vm_pageout_flush(). Use the returned run length to forward the index of next page to clean in vm_object_page_clean(). Reported by: avg Reviewed by: alc MFC after: 1 week
* With the demise of page coloring, the page queue macros no longer serve anyalc2010-07-021-1/+1
| | | | | | useful purpose. Eliminate them. Reviewed by: kib
* Make vm_contig_grow_cache() extern, and use it when vm_phys_alloc_contig()jchandra2010-06-041-9/+12
| | | | | | | | | | | | | | | | | | | | fails to allocate MIPS page table pages. The current usage of VM_WAIT in case of vm_phys_alloc_contig() failure is not correct, because: "There is no guarantee that any of the available free (or cached) pages after the VM_WAIT will fall within the range of suitable physical addresses. Every time this function sleeps and a single page is freed (or cached) by someone else, this function will be reawakened. With a little bad luck, you could spin indefinitely." We also add low and high parameters to vm_contig_grow_cache() and vm_contig_launder() so that we restrict vm_contig_launder() to the range of pages we are interested in. Reported by: alc Reviewed by: alc Approved by: rrs (mentor)
* Do not leak vm page lock in vm_contig_launder(), vm_pageout_page_lock()kib2010-06-031-1/+3
| | | | | | | always returns with the page locked. Submitted by: alc Pointy hat to: kib
* Continue cleaning the queue instead of moving to the next queue orkib2010-05-101-4/+2
| | | | | | | bailing out if acquisition of page lock caused page position in the queue to change. Pointed out by: alc
* Add a helper function vm_pageout_page_lock(), similar to tegge'kib2010-05-061-2/+4
| | | | | | | | | | | | | vm_pageout_fallback_object_lock(), to obtain the page lock while having page queue lock locked, and still maintain the page position in a queue. Use the helper to lock the page in the pageout daemon and contig launder iterators instead of skipping the page if its lock is contested. Skipping locked pages easily causes pagedaemon or launder to not make a progress with page cleaning. Proposed and reviewed by: alc
* The pages allocated by kmem_alloc_attr() and kmem_malloc() are unmanaged.alc2010-05-031-4/+0
| | | | | Consequently, neither the page lock nor the page queues lock is needed to unwire and free them.
* This change addresses the race condition that was introduced by the previousalc2010-05-021-14/+16
| | | | | | | | | | revision, r207450, to this file. Specifically, between dropping the page queues lock in vm_contig_launder() and reacquiring it in vm_contig_launder_page(), the page may be removed from the active or inactive queue. It could be wired, freed, cached, etc. None of which vm_contig_launder_page() is prepared for. Reviewed by: kib, kmacy
* - acquire the page lock in vm_contig_launder_page before checking page fieldskmacy2010-04-301-6/+15
| | | | - release page queue lock before calling vm_pageout_flush
* On Alan's advice, rather than do a wholesale conversion on a singlekmacy2010-04-301-0/+2
| | | | | | | | | | | | architecture from page queue lock to a hashed array of page locks (based on a patch by Jeff Roberson), I've implemented page lock support in the MI code and have only moved vm_page's hold_count out from under page queue mutex to page lock. This changes pmap_extract_and_hold on all pmaps. Supported by: Bitgravity Inc. Discussed with: alc, jeffr, and kib
* Introduce the function kmem_alloc_attr(), which allocates kernel virtualalc2010-04-091-18/+97
| | | | | | | | | | | | | | | | memory with the specified physical attributes. In particular, like kmem_alloc_contig(), the caller can specify the physical address range from which the physical pages are allocated and the memory attributes (i.e., cache behavior) for these physical pages. However, in contrast to kmem_alloc_contig() or contigmalloc(), the physical pages that are allocated by kmem_alloc_attr() are not necessarily physically contiguous. This function is needed by DRM and VirtualBox. Correct an error in the prototype for kmem_malloc(). The third argument had the wrong type. Tested by: rnoland MFC after: 3 days
* Add support to the virtual memory system for configuring machine-alc2009-07-121-6/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dependent memory attributes: Rename vm_cache_mode_t to vm_memattr_t. The new name reflects the fact that there are machine-dependent memory attributes that have nothing to do with controlling the cache's behavior. Introduce vm_object_set_memattr() for setting the default memory attributes that will be given to an object's pages. Introduce and use pmap_page_{get,set}_memattr() for getting and setting a page's machine-dependent memory attributes. Add full support for these functions on amd64 and i386 and stubs for them on the other architectures. The function pmap_page_set_memattr() is also responsible for any other machine-dependent aspects of changing a page's memory attributes, such as flushing the cache or updating the direct map. The uses include kmem_alloc_contig(), vm_page_alloc(), and the device pager: kmem_alloc_contig() can now be used to allocate kernel memory with non-default memory attributes on amd64 and i386. vm_page_alloc() and the device pager will set the memory attributes for the real or fictitious page according to the object's default memory attributes. Update the various pmap functions on amd64 and i386 that map pages to incorporate each page's memory attributes in the mapping. Notes: (1) Inherent to this design are safety features that prevent the specification of inconsistent memory attributes by different mappings on amd64 and i386. In addition, the device pager provides a warning when a device driver creates a fictitious page with memory attributes that are inconsistent with the real page that the fictitious page is an alias for. (2) Storing the machine-dependent memory attributes for amd64 and i386 as a dedicated "int" in "struct md_page" represents a compromise between space efficiency and the ease of MFCing these changes to RELENG_7. In collaboration with: jhb Approved by: re (kib)
* This change is the next step in implementing the cache control functionalityalc2009-06-261-3/+5
| | | | | | | | | | | required by video card drivers. Specifically, this change introduces vm_cache_mode_t with an appropriate VM_CACHE_DEFAULT definition on all architectures. In addition, this changes adds a vm_cache_mode_t parameter to kmem_alloc_contig() and vm_phys_alloc_contig(). These will be the interfaces for allocating mapped kernel memory and physical memory, respectively, with non-default cache modes. In collaboration with: jhb
* Refactor contigmalloc() into two functions: a simple front-end that dealsalc2009-06-171-8/+19
| | | | | | | | | | | | | with the malloc tag and calls a new back-end, kmem_alloc_contig(), that allocates the pages and maps them. The motivations for this change are two-fold: (1) A cache mode parameter will be added to kmem_alloc_contig(). In other words, kmem_alloc_contig() will be extended to support the allocation of memory with caller-specified caching. (2) The UMA allocation function that is used by the two jumbo frames zones can use kmem_alloc_contig() in place of contigmalloc() and thereby avoid having free jumbo frames held by the zone counted as live malloc()ed memory.
* Pass the size of the mapping to contigmapping() as a "vm_size_t" ratheralc2009-06-171-16/+13
| | | | than a "vm_pindex_t". A "vm_size_t" is more convenient for it to use.
* Make the maintenance of a page's valid bits by contigmalloc() more likealc2009-06-171-0/+1
| | | | | | kmem_alloc() and kmem_malloc(). Specifically, defer the setting of the page's valid bits until contigmapping() when the mapping is known to be successful.
* Simplify contigfree().alc2009-06-051-3/+1
|
* - back out direct map hackkmacy2009-05-191-6/+1
| | | | - it is no longer needed
* apply band-aid to x86_64 systems with more physical memory than kmem by ↵kmacy2009-05-161-1/+6
| | | | allocating from the direct map
* VOP_LOCK1() (and so VOP_LOCK()) and VOP_UNLOCK() are only used inattilio2008-01-131-1/+1
| | | | | | | | | | | conjuction with 'thread' argument passing which is always curthread. Remove the unuseful extra-argument and pass explicitly curthread to lower layer functions, when necessary. KPI results broken by this change, which should affect several ports, so version bumping and manpage update will be further committed. Tested by: kris, pho, Diego Sardina <siarodx at gmail dot com>
* vn_lock() is currently only used with the 'curthread' passed as argument.attilio2008-01-101-1/+1
| | | | | | | | | | | | | | | | Remove this argument and pass curthread directly to underlying VOP_LOCK1() VFS method. This modify makes the code cleaner and in particular remove an annoying dependence helping next lockmgr() cleanup. KPI results, obviously, changed. Manpage and FreeBSD_version will be updated through further commits. As a side note, would be valuable to say that next commits will address a similar cleanup about VFS methods, in particular vop_lock1 and vop_unlock. Tested by: Diego Sardina <siarodx at gmail dot com>, Andrea Di Pasquale <whyx dot it at gmail dot com>
* Make contigmalloc(9)'s page laundering more robust. Specifically, usealc2007-11-251-3/+6
| | | | | | | | | vm_pageout_fallback_object_lock() in vm_contig_launder_page() to better handle a lock-ordering problem. Consequently, trylock's failure on the page's containing object no longer implies that the page cannot be laundered. MFC after: 6 weeks
* Tidy up: Add comments. Eliminate the pointlessalc2007-11-251-19/+23
| | | | | | | malloc_type_allocated(..., 0) calls that occur when contigmalloc() has failed. Eliminate the acquisition and release of the page queues lock from vm_page_release_contig(). Rename contigmalloc2() to contigmapping(), reflecting what it does.
* Change the management of cached pages (PQ_CACHE) in two fundamentalalc2007-09-251-10/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ways: (1) Cached pages are no longer kept in the object's resident page splay tree and memq. Instead, they are kept in a separate per-object splay tree of cached pages. However, access to this new per-object splay tree is synchronized by the _free_ page queues lock, not to be confused with the heavily contended page queues lock. Consequently, a cached page can be reclaimed by vm_page_alloc(9) without acquiring the object's lock or the page queues lock. This solves a problem independently reported by tegge@ and Isilon. Specifically, they observed the page daemon consuming a great deal of CPU time because of pages bouncing back and forth between the cache queue (PQ_CACHE) and the inactive queue (PQ_INACTIVE). The source of this problem turned out to be a deadlock avoidance strategy employed when selecting a cached page to reclaim in vm_page_select_cache(). However, the root cause was really that reclaiming a cached page required the acquisition of an object lock while the page queues lock was already held. Thus, this change addresses the problem at its root, by eliminating the need to acquire the object's lock. Moreover, keeping cached pages in the object's primary splay tree and memq was, in effect, optimizing for the uncommon case. Cached pages are reclaimed far, far more often than they are reactivated. Instead, this change makes reclamation cheaper, especially in terms of synchronization overhead, and reactivation more expensive, because reactivated pages will have to be reentered into the object's primary splay tree and memq. (2) Cached pages are now stored alongside free pages in the physical memory allocator's buddy queues, increasing the likelihood that large allocations of contiguous physical memory (i.e., superpages) will succeed. Finally, as a result of this change long-standing restrictions on when and where a cached page can be reclaimed and returned by vm_page_alloc(9) are eliminated. Specifically, calls to vm_page_alloc(9) specifying VM_ALLOC_INTERRUPT can now reclaim and return a formerly cached page. Consequently, a call to malloc(9) specifying M_NOWAIT is less likely to fail. Discussed with: many over the course of the summer, including jeff@, Justin Husted @ Isilon, peter@, tegge@ Tested by: an earlier version by kris@ Approved by: re (kensmith)
* Enable the new physical memory allocator.alc2007-06-161-374/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allocator uses a binary buddy system with a twist. First and foremost, this allocator is required to support the implementation of superpages. As a side effect, it enables a more robust implementation of contigmalloc(9). Moreover, this reimplementation of contigmalloc(9) eliminates the acquisition of Giant by contigmalloc(..., M_NOWAIT, ...). The twist is that this allocator tries to reduce the number of TLB misses incurred by accesses through a direct map to small, UMA-managed objects and page table pages. Roughly speaking, the physical pages that are allocated for such purposes are clustered together in the physical address space. The performance benefits vary. In the most extreme case, a uniprocessor kernel running on an Opteron, I measured an 18% reduction in system time during a buildworld. This allocator does not implement page coloring. The reason is that superpages have much the same effect. The contiguous physical memory allocation necessary for a superpage is inherently colored. Finally, the one caveat is that this allocator does not effectively support prezeroed pages. I hope this is temporary. On i386, this is a slight pessimization. However, on amd64, the beneficial effects of the direct-map optimization outweigh the ill effects. I speculate that this is true in general of machines with a direct map. Approved by: re
* Conditionally acquire Giant in vm_contig_launder_page().alc2007-06-111-0/+4
|
* Revert VMCNT_* operations introduction.attilio2007-05-311-2/+2
| | | | | | | | Probabilly, a general approach is not the better solution here, so we should solve the sched_lock protection problems separately. Requested by: alc Approved by: jeff (mentor)
* - define and use VMCNT_{GET,SET,ADD,SUB,PTR} macros for manipulatingjeff2007-05-181-2/+2
| | | | | | | | vmcnts. This can be used to abstract away pcpu details but also changes to use atomics for all counters now. This means sched lock is no longer responsible for protecting counts in the switch routines. Contributed by: Attilio Rao <attilio@FreeBSD.org>
* Correct contigmalloc2()'s implementation of M_ZERO. Specifically,alc2007-04-191-1/+1
| | | | | | | | | contigmalloc2() was always testing the first physical page for PG_ZERO, not the current page of interest. Submitted by: Michael Plass PR: 81301 MFC after: 1 week
* Change the free page queue lock from a spin mutex to a default (blocking)alc2007-02-051-7/+7
| | | | | mutex. With the demise of Alpha support, there is no longer a reason for it to be a spin mutex.
* Ensure that the page's oflags field is initialized by contigmalloc().alc2006-11-081-0/+1
|
* Replace PG_BUSY with VPO_BUSY. In other words, changes to the page'salc2006-10-221-2/+2
| | | | | busy flag, i.e., VPO_BUSY, are now synchronized by the per-vm object lock instead of the global page queues lock.
* sun4v requires TSBs (translation storage buffers) to be contiguous and bekmacy2006-10-121-4/+6
| | | | | | | | size aligned requiring heavy usage of vm_page_alloc_contig This change makes vm_page_alloc_contig SMP safe Approved by: scottl (acting as backup for mentor rwatson)
* Make vm_page_release_contig() static.alc2006-09-031-1/+1
|
* Prevent a call to contigmalloc() that asks for more physical memory thanalc2006-08-261-1/+1
| | | | | | | | the machine has from causing a panic. Submitted by: Michael Plass PR: 101668 MFC after: 3 days
* Ignore dirty pages owned by "dead" objects.tegge2006-03-081-0/+4
|
OpenPOWER on IntegriCloud