summaryrefslogtreecommitdiffstats
path: root/include/linux/percpu.h
Commit message (Collapse)AuthorAgeFilesLines
...
* percpu: use negative for auto for pcpu_setup_first_chunk() argumentsTejun Heo2009-03-061-2/+3
| | | | | | | | | | | | | | | Impact: argument semantic cleanup In pcpu_setup_first_chunk(), zero @unit_size and @dyn_size meant auto-sizing. It's okay for @unit_size as 0 doesn't make sense but 0 dynamic reserve size is valid. Alos, if arch @dyn_size is calculated from other parameters, it might end up passing in 0 @dyn_size and malfunction when the size is automatically adjusted. This patch makes both @unit_size and @dyn_size ssize_t and use -1 for auto sizing. Signed-off-by: Tejun Heo <tj@kernel.org>
* percpu: cosmetic renames in pcpu_setup_first_chunk()Tejun Heo2009-03-061-1/+1
| | | | | | | | | | | | | Impact: cosmetic, preparation for future changes Make the following renames in pcpur_setup_first_chunk() in preparation for future changes. * s/free_size/dyn_size/ * s/static_vm/first_vm/ * s/static_chunk/schunk/ Signed-off-by: Tejun Heo <tj@kernel.org>
* percpu: clean up percpu constantsTejun Heo2009-03-061-11/+13
| | | | | | | | | | | | | | | | | | Impact: cleaup Make the following cleanups. * There isn't much arch-specific about PERCPU_MODULE_RESERVE. Always define it whether arch overrides PERCPU_ENOUGH_ROOM or not. * blackfin overrides PERCPU_ENOUGH_ROOM to align static area size. Do it by default. * percpu allocation sizes doesn't have much to do with the page size. Don't use PAGE_SHIFT in their definition. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Bryan Wu <cooloney@kernel.org>
* percpu: fix too low alignment restriction on UPTejun Heo2009-02-261-1/+1
| | | | | | | | | | UP __alloc_percpu() triggered WARN_ON_ONCE() if the requested alignment is larger than that of unsigned long long, which is too small for all the cacheline aligned allocations. Bump it up to SMP_CACHE_BYTES which kmalloc allocations generally guarantee. Signed-off-by: Tejun Heo <tj@kernel.org> Reported-by: Ingo Molnar <mingo@elte.hu>
* alloc_percpu: fix UP buildIngo Molnar2009-02-251-1/+1
| | | | | | | | | | | | | | | | | Impact: build fix the !SMP branch had a 'gfp' leftover: include/linux/percpu.h: In function '__alloc_percpu': include/linux/percpu.h:160: error: 'gfp' undeclared (first use in this function) include/linux/percpu.h:160: error: (Each undeclared identifier is reported only once include/linux/percpu.h:160: error: for each function it appears in.) Use GFP_KERNEL like the SMP version does. Cc: Rusty Russell <rusty@rustcorp.com.au> Cc: Tejun Heo <tj@kernel.org> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* percpu: give more latitude to arch specific first chunk initializationTejun Heo2009-02-241-2/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | Impact: more latitude for first percpu chunk allocation The first percpu chunk serves the kernel static percpu area and may or may not contain extra room for further dynamic allocation. Initialization of the first chunk needs to be done before normal memory allocation service is up, so it has its own init path - pcpu_setup_static(). It seems archs need more latitude while initializing the first chunk for example to take advantage of large page mapping. This patch makes the following changes to allow this. * Define PERCPU_DYNAMIC_RESERVE to give arch hint about how much space to reserve in the first chunk for further dynamic allocation. * Rename pcpu_setup_static() to pcpu_setup_first_chunk(). * Make pcpu_setup_first_chunk() much more flexible by fetching page pointer by callback and adding optional @unit_size, @free_size and @base_addr arguments which allow archs to selectively part of chunk initialization to their likings. Signed-off-by: Tejun Heo <tj@kernel.org>
* percpu: implement new dynamic percpu allocatorTejun Heo2009-02-201-4/+18
| | | | | | | | | | | | | | Impact: new scalable dynamic percpu allocator which allows dynamic percpu areas to be accessed the same way as static ones Implement scalable dynamic percpu allocator which can be used for both static and dynamic percpu areas. This will allow static and dynamic areas to share faster direct access methods. This feature is optional and enabled only when CONFIG_HAVE_DYNAMIC_PER_CPU_AREA is defined by arch. Please read comment on top of mm/percpu.c for details. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Andrew Morton <akpm@linux-foundation.org>
* percpu: kill percpu_alloc() and friendsTejun Heo2009-02-201-25/+22
| | | | | | | | | | | | | | | | Impact: kill unused functions percpu_alloc() and its friends never saw much action. It was supposed to replace the cpu-mask unaware __alloc_percpu() but it never happened and in fact __percpu_alloc_mask() itself never really grew proper up/down handling interface either (no exported interface for populate/depopulate). percpu allocation is about to go through major reimplementation and there's no reason to carry this unused interface around. Replace it with __alloc_percpu() and free_percpu(). Signed-off-by: Tejun Heo <tj@kernel.org>
* alloc_percpu: add align argument to __alloc_percpu.Rusty Russell2009-02-201-2/+3
| | | | | | | | | | | This prepares for a real __alloc_percpu, by adding an alignment argument. Only one place uses __alloc_percpu directly, and that's for a string. tj: af_inet also uses __alloc_percpu(), update it. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Cc: Christoph Lameter <cl@linux-foundation.org> Cc: Jens Axboe <axboe@kernel.dk>
* alloc_percpu: change percpu_ptr to per_cpu_ptrRusty Russell2009-02-201-12/+11
| | | | | | | | | | | | | | | | Impact: cleanup There are two allocated per-cpu accessor macros with almost identical spelling. The original and far more popular is per_cpu_ptr (44 files), so change over the other 4 files. tj: kill percpu_ptr() and update UP too Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Cc: mingo@redhat.com Cc: lenb@kernel.org Cc: cpufreq@vger.kernel.org Signed-off-by: Tejun Heo <tj@kernel.org>
* percpu: make PER_CPU_BASE_SECTION overridable by archesBrian Gerst2009-02-091-1/+7
| | | | | | | | | | | Impact: bug fix IA-64 needs to put percpu data in the seperate section even on UP. Fixes regression caused by "percpu: refactor percpu.h" Signed-off-by: Brian Gerst <brgerst@gmail.com> Acked-by: Tony Luck <tony.luck@intel.com> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* percpu: refactor percpu.hBrian Gerst2009-01-201-18/+23
| | | | | | | | | | Impact: cleanup Refactor the DEFINE_PER_CPU_* macros and add .data.percpu.first section. Signed-off-by: Brian Gerst <brgerst@gmail.com> Signed-off-by: Tejun Heo <tj@kernel.org>
* Merge commit '63cc8c75156462d4b42cbdd76c293b7eee7ddbfe':Ingo Molnar2008-09-051-0/+7
|\ | | | | | | | | | | | | | | | | | | | | "percpu: introduce DEFINE_PER_CPU_PAGE_ALIGNED() macro" into x86/core Conflicts: arch/x86/kernel/cpu/common.c Signed-off-by: Ingo Molnar <mingo@elte.hu>
| * percpu: introduce DEFINE_PER_CPU_PAGE_ALIGNED() macroEric Dumazet2008-05-251-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While examining holes in percpu section I found this : c05f5000 D per_cpu__current_task c05f5000 D __per_cpu_start c05f5004 D per_cpu__cpu_number c05f5008 D per_cpu__irq_regs c05f500c d per_cpu__cpu_devices c05f5040 D per_cpu__cyc2ns <Big Hole of about 4000 bytes> c05f6000 d per_cpu__cpuid4_info c05f6004 d per_cpu__cache_kobject c05f6008 d per_cpu__index_kobject <Big Hole of about 4000 bytes> c05f7000 D per_cpu__gdt_page This is because gdt_page is a percpu variable, defined with a page alignement, and linker is doing its job, two times because of .o nesting in the build process. I introduced a new macro DEFINE_PER_CPU_PAGE_ALIGNED() to avoid wasting this space. All page aligned variables (only one at this time) are put in a separate subsection .data.percpu.page_aligned, at the very begining of percpu zone. Before patch , on a x86_32 machine : .data.percpu 30232 3227471872 .data.percpu 22168 3227471872 Thats 8064 bytes saved for each CPU. Signed-off-by: Eric Dumazet <dada1@cosmosbay.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
* | mm/allocpercpu.c: make 4 functions staticAdrian Bunk2008-07-261-29/+0
|/ | | | | | | | | | | | | This patch makes the following needlessly global functions static: - percpu_depopulate() - __percpu_depopulate_mask() - percpu_populate() - __percpu_populate_mask() Signed-off-by: Adrian Bunk <bunk@kernel.org> Acked-by: Christoph Lameter <cl@linux-foundation.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* per_cpu: fix DEFINE_PER_CPU_SHARED_ALIGNED for modulesEric Dumazet2008-05-141-1/+7
| | | | | | | | | | | | | | | | | | | | Current module loader lookups ".data.percpu" ELF section to perform per_cpu relocation. But DEFINE_PER_CPU_SHARED_ALIGNED() uses another section (".data.percpu.shared_aligned"), currently only handled in vmlinux.lds, not by module loader. To correct this problem, instead of adding logic into module loader, or using at build time a module.lds file for all arches to group ".data.percpu.shared_aligned" into ".data.percpu", just use ".data.percpu" for modules. Alignment requirements are correctly handled by ld and module loader. Signed-off-by: Eric Dumazet <dada1@cosmosbay.com> Cc: Rusty Russell <rusty@rustcorp.com.au> Cc: Fenghua Yu <fenghua.yu@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* Remove superfluous include of string.h from percpu.hRobert P. J. Day2008-04-291-1/+0
| | | | | | | | | There's nothing in percpu.h that requires an explicit inclusion of string.h. Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* PERCPU : __percpu_alloc_mask() can dynamically size percpu_data storageEric Dumazet2008-02-061-1/+1
| | | | | | | | | | | Instead of allocating a fix sized array of NR_CPUS pointers for percpu_data, we can use nr_cpu_ids, which is generally < NR_CPUS. Signed-off-by: Eric Dumazet <dada1@cosmosbay.com> Cc: Christoph Lameter <clameter@sgi.com> Cc: "David S. Miller" <davem@davemloft.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* ia64: use generic percputravis@sgi.com2008-01-301-4/+0
| | | | | | | | | | ia64 has a special processor specific mapping that can be used to locate the offset for the current per cpu area. Cc: linux-ia64@vger.kernel.org Signed-off-by: Mike Travis <travis@sgi.com> Acked-by: Tony Luck <tony.luck@intel.com> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* percpu: move arch XX_PER_CPU_XX definitions into linux/percpu.htravis@sgi.com2008-01-301-0/+24
| | | | | | | | | | | | | | | | | | | - Special consideration for IA64: Add the ability to specify arch specific per cpu flags - remove .data.percpu attribute from DEFINE_PER_CPU for non-smp case. The arch definitions are all the same. So move them into linux/percpu.h. We cannot move DECLARE_PER_CPU since some include files just include asm/percpu.h to avoid include recursion problems. Cc: Rusty Russell <rusty@rustcorp.com.au> Cc: Andi Kleen <ak@suse.de> Signed-off-by: Christoph Lameter <clameter@sgi.com> Signed-off-by: Mike Travis <travis@sgi.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Ingo Molnar <mingo@elte.hu>
* Remove unnecessary includes of spinlock.h under include/linuxRobert P. J. Day2007-07-161-1/+1
| | | | | | | | | | Remove the obviously unnecessary includes of <linux/spinlock.h> under the include/linux/ directory, and fix the couple errors that are introduced as a result of that. Signed-off-by: Robert P. J. Day <rpjday@mindspring.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* [PATCH] x86-64: Account for module percpu space separately from kernel percpuJeremy Fitzhardinge2007-05-021-1/+8
| | | | | | | | | | | | | | Rather than using a single constant PERCPU_ENOUGH_ROOM, compute it as the sum of kernel_percpu + PERCPU_MODULE_RESERVE. This is now common to all architectures; if an architecture wants to set PERCPU_ENOUGH_ROOM to something special, then it may do so (ia64 is the only one which does). Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com> Signed-off-by: Andi Kleen <ak@suse.de> Cc: Rusty Russell <rusty@rustcorp.com.au> Cc: Eric W. Biederman <ebiederm@xmission.com> Cc: Andi Kleen <ak@suse.de>
* [PATCH] Fix typo in "syntax error if percpu macros are incorrectly used" patchJan Blunck2006-10-061-1/+1
| | | | | | | | | | | Trivial typo fix in the "syntax error if percpu macros are incorrectly used" patch. I misspelled "identifier" in all places. D'Oh! Thanks to Dirk Mueller to point this out. Signed-off-by: Jan Blunck <jblunck@suse.de> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] __percpu_alloc_mask() has to be __always_inline in UP caseAl Viro2006-09-291-1/+1
| | | | | | | | | | | | ... or we'll end up with cpu_online_map being evaluated on UP. In modules. cpumask.h is very careful to avoid that, and for a very good reason. So should we... PS: yes, it really triggers (on alpha). Signed-off-by: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] CPU hotplug compatible alloc_percpu()Martin Peschke2006-09-261-19/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch splits alloc_percpu() up into two phases. Likewise for free_percpu(). This allows clients to limit initial allocations to online cpu's, and to populate or depopulate per-cpu data at run time as needed: struct my_struct *obj; /* initial allocation for online cpu's */ obj = percpu_alloc(sizeof(struct my_struct), GFP_KERNEL); ... /* populate per-cpu data for cpu coming online */ ptr = percpu_populate(obj, sizeof(struct my_struct), GFP_KERNEL, cpu); ... /* access per-cpu object */ ptr = percpu_ptr(obj, smp_processor_id()); ... /* depopulate per-cpu data for cpu going offline */ percpu_depopulate(obj, cpu); ... /* final removal */ percpu_free(obj); Signed-off-by: Martin Peschke <mp3@de.ibm.com> Cc: Paul Jackson <pj@sgi.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] trigger a syntax error if percpu macros are incorrectly usedJan Blunck2006-09-261-2/+8
| | | | | | | | | | | | get_cpu_var()/per_cpu()/__get_cpu_var() arguments must be simple identifiers. Otherwise the arch dependent implementations might break. This patch enforces the correct usage of the macros by producing a syntax error if the variable is not a simple identifier. Signed-off-by: Jan Blunck <jblunck@suse.de> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] remove unused blkp field in percpu_dataEric Dumazet2006-01-081-1/+0
| | | | | | | | | | | | | | | I found that blkp field was not used in kernel tree. As most of the times NR_CPUS is a power of two and kmalloc() memory blocks too, this extra field basically doubles the memory space allocated in __alloc_percpu() to store the 'struct percpu_data' (for example, if NR_CPUS=8 on i386, kmalloc(4*8+4) returns a 64 bytes block instead of a 32 bytes block after this patch) Signed-off-by: Eric Dumazet <dada1@cosmosbay.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] slab: remove unused align parameter from alloc_percpuPekka Enberg2006-01-081-4/+3
| | | | | | | | | | | | __alloc_percpu and alloc_percpu both take an 'align' argument which is completely ignored. snmp6_mib_init() in net/ipv6/af_inet6.c attempts to use it, but it will be ignored. Therefore, remove the 'align' argument and fixup the lone caller. Signed-off-by: Matthew Dobson <colpatch@us.ibm.com> Acked-by: Manfred Spraul <manfred@colorfullife.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] Shut up per_cpu_ptr() on UPPaul Mundt2005-11-131-1/+1
| | | | | | | | | | | | | Currently per_cpu_ptr() doesn't really do anything with 'cpu' in the UP case. This is problematic in the cases where this is the only place the variable is referenced: CC kernel/workqueue.o kernel/workqueue.c: In function `current_is_keventd': kernel/workqueue.c:460: warning: unused variable `cpu' Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* Linux-2.6.12-rc2v2.6.12-rc2Linus Torvalds2005-04-161-0/+61
Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
OpenPOWER on IntegriCloud