summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* cgroup: reorganize css_task_iter functionsTejun Heo2015-10-151-21/+28
| | | | | | | | | | | | | | * Rename css_advance_task_iter() to css_task_iter_advance_css_set() and make it clear it->task_pos too at the end of the iteration. * Factor out css_task_iter_advance() from css_task_iter_next(). The new function whines if called on a terminated iterator. Except for the termination check, this is pure reorganization and doesn't introduce any behavior changes. This will help the planned locking update for css_task_iter. Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: factor out css_set_move_task()Tejun Heo2015-10-151-48/+56
| | | | | | | | | | | | | | | | | A task is associated and disassociated with its css_set in three places - during migration, after a new task is created and when a task exits. The first is handled by cgroup_task_migrate() and the latter two are open-coded. These are similar operations and spreading them over multiple places makes it harder to follow and update. This patch collects all task css_set [dis]association operations into css_set_move_task(). While css_set_move_task() may check whether populated state needs to be updated when not strictly necessary, the behavior is essentially equivalent before and after this patch. Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: keep css_set and task lists in chronological orderTejun Heo2015-10-151-6/+5
| | | | | | | | | | | | | css task iteration will be updated to not leak cgroup internal locking to iterator users. In preparation, update css_set and task lists to be in chronological order. For tasks, as migration path is already using list_splice_tail_init(), only cgroup_enable_task_cg_lists() and cgroup_post_fork() need updating. For css_sets, link_css_set() is the only place which needs to be updated. Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: make cgroup_destroy_locked() test cgroup_is_populated()Tejun Heo2015-10-151-6/+5
| | | | | | | | | | | | | | | | | | | | | cgroup_destroy_locked() currently tests whether any css_sets are associated to reject removal if the cgroup contains tasks. This works because a css_set's refcnt converges with the number of tasks linked to it and thus there's no css_set linked to a cgroup if it doesn't have any live tasks. To help tracking resource usage of zombie tasks, putting the ref of css_set will be separated from disassociating the task from the css_set which means that a cgroup may have css_sets linked to it even when it doesn't have any live tasks. This patch updates cgroup_destroy_locked() so that it tests cgroup_is_populated(), which counts the number of populated css_sets, instead of whether cgrp->cset_links is empty to determine whether the cgroup is populated or not. This ensures that rmdirs won't be incorrectly rejected for cgroups which only contain zombie tasks. Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: make css_sets pin the associated cgroupsTejun Heo2015-10-151-4/+6
| | | | | | | | | | | | | | | | Currently, css_sets don't pin the associated cgroups. This is okay as a cgroup with css_sets associated are not allowed to be removed; however, to help resource tracking for zombie tasks, this is scheduled to change such that a cgroup can be removed even when it has css_sets associated as long as none of them are populated. To ensure that a cgroup doesn't go away while css_sets are still associated with it, make each associated css_set hold a reference on the cgroup if non-root. v2: Root cgroups are special and shouldn't be ref'd by css_sets. Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: relocate cgroup_[try]get/put()Tejun Heo2015-10-151-16/+16
| | | | | | | Relocate cgroup_get(), cgroup_tryget() and cgroup_put() upwards. This is pure code reorganization to prepare for future changes. Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: move check_for_release() invocationTejun Heo2015-10-151-7/+1
| | | | | | | | | | | | | To trigger release agent when the last task leaves the cgroup, check_for_release() is called from put_css_set_locked(); however, css_set being unlinked is being decoupled from task leaving the cgroup and the correct condition to test is cgroup->nr_populated dropping to zero which check_for_release() is already updated to test. This patch moves check_for_release() invocation from put_css_set_locked() to cgroup_update_populated(). Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: replace cgroup_has_tasks() with cgroup_is_populated()Tejun Heo2015-10-154-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, cgroup_has_tasks() tests whether the target cgroup has any css_set linked to it. This works because a css_set's refcnt converges with the number of tasks linked to it and thus there's no css_set linked to a cgroup if it doesn't have any live tasks. To help tracking resource usage of zombie tasks, putting the ref of css_set will be separated from disassociating the task from the css_set which means that a cgroup may have css_sets linked to it even when it doesn't have any live tasks. This patch replaces cgroup_has_tasks() with cgroup_is_populated() which tests cgroup->nr_populated instead which locally counts the number of populated css_sets. Unlike cgroup_has_tasks(), cgroup_is_populated() is recursive - if any of the descendants is populated, the cgroup is populated too. While this changes the meaning of the test, all the existing users are okay with the change. While at it, replace the open-coded ->populated_cnt test in cgroup_events_show() with cgroup_is_populated(). Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Li Zefan <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org>
* cgroup: make cgroup->nr_populated count the number of populated css_setsTejun Heo2015-10-152-17/+56
| | | | | | | | | | | | | | | | | | | Currently, cgroup->nr_populated counts whether the cgroup has any css_sets linked to it and the number of children which has non-zero ->nr_populated. This works because a css_set's refcnt converges with the number of tasks linked to it and thus there's no css_set linked to a cgroup if it doesn't have any live tasks. To help tracking resource usage of zombie tasks, putting the ref of css_set will be separated from disassociating the task from the css_set which means that a cgroup may have css_sets linked to it even when it doesn't have any live tasks. This patch updates cgroup->nr_populated so that for the cgroup itself it counts the number of css_sets which have tasks associated with them so that empty css_sets don't skew the populated test. Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: remove an unused parameter from cgroup_task_migrate()Tejun Heo2015-10-151-5/+2
| | | | | | cgroup_task_migrate() no longer uses @old_cgrp. Remove it. Signed-off-by: Tejun Heo <tj@kernel.org>
* cgroup: fix too early usage of static_branch_disable()Tejun Heo2015-09-251-6/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 49d1dc4b8179 ("cgroup: implement static_key based cgroup_subsys_enabled() and cgroup_subsys_on_dfl()") converted cgroup enabled test to use static_key; however, cgroup_disable() is called before static_key subsystem itself is initialized and thus leads to the following warning when "cgroup_disable=" parameter is specified. WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:99 static_key_slow_dec+0x44/0x60() static_key_slow_dec used before call to jump_label_init ... Call Trace: [<ffffffff813b18c2>] dump_stack+0x44/0x62 [<ffffffff8108dd52>] warn_slowpath_common+0x82/0xc0 [<ffffffff8108ddec>] warn_slowpath_fmt+0x5c/0x80 [<ffffffff8119c054>] static_key_slow_dec+0x44/0x60 [<ffffffff81d826b6>] cgroup_disable+0xaf/0xd6 [<ffffffff81d5f9de>] unknown_bootoption+0x8c/0x194 [<ffffffff810b0c03>] parse_args+0x273/0x4a0 [<ffffffff81d5fd67>] start_kernel+0x205/0x4b8 ... Fix it by making cgroup_disable() to record the subsystems to disable in cgroup_disable_mask and moving the actual application to cgroup_init() which is late enough and where the enabled state is first used. Signed-off-by: Tejun Heo <tj@kernel.org> Reported-by: Andrey Wagin <avagin@gmail.com> Link: http://lkml.kernel.org/g/CANaxB-yFuS4SA2znSvcKrO9L_CbHciHYW+o9bN8sZJ8eR9FxYA@mail.gmail.com Fixes: 49d1dc4b81797f88270832b11e9f73809e7e7209
* cgroup: Merge branch 'for-4.3-fixes' into for-4.4Tejun Heo2015-09-241-2/+8
|\ | | | | | | | | | | | | | | | | | | | | Pull to receive 9badce000e2c ("cgroup, writeback: don't enable cgroup writeback on traditional hierarchies"). The commit adds cgroup_on_dfl() usages in include/linux/backing-dev.h thus causing a silent conflict with 9e10a130d9b6 ("cgroup: replace cgroup_on_dfl() tests in controllers with cgroup_subsys_on_dfl()"). The conflict is fixed during this merge. Signed-off-by: Tejun Heo <tj@kernel.org>
| * cgroup, writeback: don't enable cgroup writeback on traditional hierarchiesTejun Heo2015-09-241-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | inode_cgwb_enabled() gates cgroup writeback support. If it returns true, each inode is attached to the corresponding memory domain which gets mapped to io domain. It currently only tests whether the filesystem and bdi support cgroup writeback; however, cgroup writeback support doesn't work on traditional hierarchies and thus it should also test whether memcg and iocg are on the default hierarchy. This caused traditional hierarchy setups to hit the cgroup writeback path inadvertently and ended up creating separate writeback domains for each memcg and mapping them all to the root iocg uncovering a couple issues in the cgroup writeback path. cgroup writeback was never meant to be enabled on traditional hierarchies. Make inode_cgwb_enabled() test whether both memcg and iocg are on the default hierarchy. Signed-off-by: Tejun Heo <tj@kernel.org> Reported-by: Artem Bityutskiy <dedekind1@gmail.com> Reported-by: Dexuan Cui <decui@microsoft.com> Link: http://lkml.kernel.org/g/1443012552.19983.209.camel@gmail.com Link: http://lkml.kernel.org/g/f30d4a6aa8a546ff88f73021d026a453@SIXPR30MB031.064d.mgd.msft.net
* | cgroup: make cgroup_update_dfl_csses() migrate all target processes atomicallyTejun Heo2015-09-221-36/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | cgroup_update_dfl_csses() is responsible for migrating processes when controllers are enabled or disabled on the default hierarchy. As the css association changes for all the processes in the affected cgroups, this involves migrating multiple processes. Up until now, it was implemented by migrating process-by-process until the source css_sets are empty; however, this means that if a process fails to migrate after some succeed before it, the recovery is very tricky. This was considered okay as subsystems weren't allowed to reject process migration on the default hierarchy; unfortunately, enforcing this policy turned out to be problematic for certain types of resources - realtime slices for now. As such, the default hierarchy is gonna allow restricted failures during migration and to support that this patch makes cgroup_update_dfl_csses() migrate all target processes atomically rather than one-by-one. The preceding patches made subsystems ready for multi-process migration and factored out taskset operations making this almost trivial. All tasks of the target processes are put in the same taskset and the migration operations are performed once which either fails or succeeds for all. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Zefan Li <lizefan@huawei.com>
* | cgroup: separate out taskset operations from cgroup_migrate()Tejun Heo2015-09-221-86/+125
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, cgroup_migreate() implements large part of the migration logic inline including building the target taskset and actually migrating them. This patch separates out the following taskset operations. CGROUP_TASKSET_INIT() : taskset initializer cgroup_taskset_add() : add a task to a taskset cgroup_taskset_migrate() : migrate a taskset to the destination cgroup This will be used to implement atomic multi-process migration in cgroup_update_dfl_csses(). This is pure reorganization which doesn't introduce any functional changes. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Zefan Li <lizefan@huawei.com>
* | cgroup: reorder cgroup_migrate()'s parametersTejun Heo2015-09-221-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | cgroup_migrate() has the destination cgroup as the first parameter while cgroup_task_migrate() has the destination cset as the last. Another migration function is scheduled to be added which can make the discrepancy further stand out. Let's reorder cgroup_migrate()'s parameters so that the destination cgroup is the last. This doesn't cause any functional difference. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Zefan Li <lizefan@huawei.com>
* | cgroup, memcg, cpuset: implement cgroup_taskset_for_each_leader()Tejun Heo2015-09-224-18/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It wasn't explicitly documented but, when a process is being migrated, cpuset and memcg depend on cgroup_taskset_first() returning the threadgroup leader; however, this approach is somewhat ghetto and would no longer work for the planned multi-process migration. This patch introduces explicit cgroup_taskset_for_each_leader() which iterates over only the threadgroup leaders and replaces cgroup_taskset_first() usages for accessing the leader with it. This prepares both memcg and cpuset for multi-process migration. This patch also updates the documentation for cgroup_taskset_for_each() to clarify the iteration rules and removes comments mentioning task ordering in tasksets. v2: A previous patch which added threadgroup leader test was dropped. Patch updated accordingly. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Zefan Li <lizefan@huawei.com> Acked-by: Michal Hocko <mhocko@suse.cz> Cc: Johannes Weiner <hannes@cmpxchg.org>
* | cpuset: migrate memory only for threadgroup leadersTejun Heo2015-09-221-18/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If memory_migrate flag is set, cpuset migrates memory according to the destnation css's nodemask. The current implementation migrates memory whenever any thread of a process is migrated making the behavior somewhat arbitrary. Let's tie memory operations to the threadgroup leader so that memory is migrated only when the leader is migrated. While this is a behavior change, given the inherent fuziness, this change is not too likely to be noticed and allows us to clearly define who owns the memory (always the leader) and helps the planned atomic multi-process migration. Note that we're currently migrating memory in migration path proper while holding all the locks. In the long term, this should be moved out to an async work item. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Zefan Li <lizefan@huawei.com>
* | memcg: generate file modified notifications on "memory.events"Tejun Heo2015-09-212-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | cgroup core only recently grew generic notification support. Wire up "memory.events" so that it triggers a file modified event whenever its content changes. v2: Refreshed on top of mem_cgroup relocation. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Michal Hocko <mhocko@kernel.org> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Li Zefan <lizefan@huawei.com>
* | cgroup: generalize obtaining the handles of and notifying cgroup filesTejun Heo2015-09-183-9/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | cgroup core handles creations and removals of cgroup interface files as described by cftypes. There are cases where the handle for a given file instance is necessary, for example, to generate a file modified event. Currently, this is handled by explicitly matching the callback method pointer and storing the file handle manually in cgroup_add_file(). While this simple approach works for cgroup core files, it can't for controller interface files. This patch generalizes cgroup interface file handle handling. struct cgroup_file is defined and each cftype can optionally tell cgroup core to store the file handle by setting ->file_offset. A file handle remains accessible as long as the containing css is accessible. Both "cgroup.procs" and "cgroup.events" are converted to use the new generic mechanism instead of hooking directly into cgroup_add_file(). Also, cgroup_file_notify() which takes a struct cgroup_file and generates a file modified event on it is added and replaces explicit kernfs_notify() invocations. This generalizes cgroup file handle handling and allows controllers to generate file modified notifications. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Li Zefan <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org>
* | cgroup: restructure file creation / removal handlingTejun Heo2015-09-181-67/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The file creation / removal path has always been a bit icky and the planned notification update requires css during file creation. Restructure as follows. * cgroup_addrm_files() now takes both @css and @cgrp and is only called directly by other file handling functions. * cgroup_populate/clear_dir() are replaced with css_populate/clear_dir() taking @css and @cgrp_override. @cgrp_override is used only when files needs to be created on / removed from a cgroup which isn't attached to @css which happens during subsystem rebinds. Subsystem loops are moved to the callers. * cgroup_add_file() now takes both @css and @cgrp. @css isn't used yet but will be used by the planned notification update. This patch doens't cause any behavior changes. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Li Zefan <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org>
* | cgroup: cosmetic updates to rebind_subsystems()Tejun Heo2015-09-181-16/+15
| | | | | | | | | | | | | | | | | | | | | | * Use local variables @scgrp and @dcgrp for @src_root->cgrp and @dst_root->cgrp respectively. * Use initializers to set @src_root and @css in the inner bind loop. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Li Zefan <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org>
* | cgroup: make cgroup_addrm_files() clean up after itself on failuresTejun Heo2015-09-181-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After a file creation failure, cgroup_addrm_files() it didn't remove the files which had already been created. When cgroup_populate_dir() is the caller, this is fine as the caller performs cleanup; however, for other callers, this may leave unactivated dangling files behind. As kernfs directory removals are recursive, this doesn't lead to permanent memory leak but it can, for example, fail future attempts to create those files again. There's no point in keeping around this sort of subtlety and it gets in the way of planned updates to file handling. This patch makes cgroup_addrm_files() clean up after itself on failures. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Li Zefan <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org>
* | cgroup: relocate cgroup_populate_dir()Tejun Heo2015-09-181-32/+31
| | | | | | | | | | | | | | | | | | Move it upwards so that it's right below cgroup_clear_dir() and the forward declaration is unnecessary. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Li Zefan <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org>
* | cgroup: replace cftype->mode with CFTYPE_WORLD_WRITABLETejun Heo2015-09-184-25/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | cftype->mode allows controllers to give arbitrary permissions to interface knobs. Except for "cgroup.event_control", the existing uses are spurious. * Some explicitly specify S_IRUGO | S_IWUSR even though that's the default. * "cpuset.memory_pressure" specifies S_IRUGO while also setting a write callback which returns -EACCES. All it needs to do is simply not setting a write callback. "cgroup.event_control" uses cftype->mode to make the file world-writable. It's a misdesigned interface and we don't want controllers to be tweaking interface file permissions in general. This patch removes cftype->mode and all its spurious uses and implements CFTYPE_WORLD_WRITABLE for "cgroup.event_control" which is marked as compatibility-only. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Li Zefan <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org>
* | cgroup: replace "cgroup.populated" with "cgroup.events"Tejun Heo2015-09-183-14/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | memcg already uses "memory.events" for event reporting and other controllers may need event reporting too. Let's standardize on "$SUBSYS.events" interface file for reporting events which don't happen too frequently and thus can share event notification. "cgroup.populated" is replaced with "populated" field in "cgroup.events" and documentation is updated accordingly. Signed-off-by: Tejun Heo <tj@kernel.org> Cc: Li Zefan <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org>
* | cgroup: replace cgroup_on_dfl() tests in controllers with cgroup_subsys_on_dfl()Tejun Heo2015-09-186-73/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | cgroup_on_dfl() tests whether the cgroup's root is the default hierarchy; however, an individual controller is only interested in whether the controller is attached to the default hierarchy and never tests a cgroup which doesn't belong to the hierarchy that the controller is attached to. This patch replaces cgroup_on_dfl() tests in controllers with faster static_key based cgroup_subsys_on_dfl(). This leaves cgroup core as the only user of cgroup_on_dfl() and the function is moved from the header file to cgroup.c. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Zefan Li <lizefan@huawei.com> Cc: Vivek Goyal <vgoyal@redhat.com> Cc: Jens Axboe <axboe@kernel.dk> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org>
* | cgroup: replace cgroup_subsys->disabled tests with cgroup_subsys_enabled()Tejun Heo2015-09-184-14/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace cgroup_subsys->disabled tests in controllers with cgroup_subsys_enabled(). cgroup_subsys_enabled() requires literal subsys name as its parameter and thus can't be used for cgroup core which iterates through controllers. For cgroup core, introduce and use cgroup_ssid_enabled() which uses slower static_key_enabled() test and can be indexed by subsys ID. This leaves cgroup_subsys->disabled unused. Removed. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Zefan Li <lizefan@huawei.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org>
* | cgroup: implement static_key based cgroup_subsys_enabled() and ↵Tejun Heo2015-09-182-1/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | cgroup_subsys_on_dfl() Whether a subsys is enabled and attached to the default hierarchy seldom changes and may be tested in the hot paths. This patch implements static_key based cgroup_subsys_enabled() and cgroup_subsys_on_dfl() tests. The following patches will update the users and remove duplicate mechanisms. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Zefan Li <lizefan@huawei.com>
* | jump_label: make static_key_enabled() work on static_key_true/false types tooTejun Heo2015-09-181-7/+11
| | | | | | | | | | | | | | | | | | | | | | static_key_enabled() can be used on struct static_key but not on its wrapper types static_key_true and static_key_false. The function is useful for debugging and management of static keys. Update it so that it can be used for the wrapper types too. Signed-off-by: Tejun Heo <tj@kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Andrew Morton <akpm@linux-foundation.org>
* | cgroup: simplify threadgroup lockingTejun Heo2015-09-161-33/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Note: This commit was originally committed as b5ba75b5fc0e but got reverted by f9f9e7b77614 due to the performance regression from the percpu_rwsem write down/up operations added to cgroup task migration path. percpu_rwsem changes which alleviate the performance issue are pending for v4.4-rc1 merge window. Re-apply. Now that threadgroup locking is made global, code paths around it can be simplified. * lock-verify-unlock-retry dancing removed from __cgroup_procs_write(). * Race protection against de_thread() removed from cgroup_update_dfl_csses(). Signed-off-by: Tejun Heo <tj@kernel.org> Link: http://lkml.kernel.org/g/55F8097A.7000206@de.ibm.com
* | sched, cgroup: replace signal_struct->group_rwsem with a global percpu_rwsemTejun Heo2015-09-165-83/+45
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Note: This commit was originally committed as d59cfc09c32a but got reverted by 0c986253b939 due to the performance regression from the percpu_rwsem write down/up operations added to cgroup task migration path. percpu_rwsem changes which alleviate the performance issue are pending for v4.4-rc1 merge window. Re-apply. The cgroup side of threadgroup locking uses signal_struct->group_rwsem to synchronize against threadgroup changes. This per-process rwsem adds small overhead to thread creation, exit and exec paths, forces cgroup code paths to do lock-verify-unlock-retry dance in a couple places and makes it impossible to atomically perform operations across multiple processes. This patch replaces signal_struct->group_rwsem with a global percpu_rwsem cgroup_threadgroup_rwsem which is cheaper on the reader side and contained in cgroups proper. This patch converts one-to-one. This does make writer side heavier and lower the granularity; however, cgroup process migration is a fairly cold path, we do want to optimize thread operations over it and cgroup migration operations don't take enough time for the lower granularity to matter. Signed-off-by: Tejun Heo <tj@kernel.org> Link: http://lkml.kernel.org/g/55F8097A.7000206@de.ibm.com Cc: Ingo Molnar <mingo@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org>
* Revert "sched, cgroup: replace signal_struct->group_rwsem with a global ↵Tejun Heo2015-09-165-45/+83
| | | | | | | | | | | | | | | | | | | | | | | | | | | | percpu_rwsem" This reverts commit d59cfc09c32a2ae31f1c3bc2983a0cd79afb3f14. d59cfc09c32a ("sched, cgroup: replace signal_struct->group_rwsem with a global percpu_rwsem") and b5ba75b5fc0e ("cgroup: simplify threadgroup locking") changed how cgroup synchronizes against task fork and exits so that it uses global percpu_rwsem instead of per-process rwsem; unfortunately, the write [un]lock paths of percpu_rwsem always involve synchronize_rcu_expedited() which turned out to be too expensive. Improvements for percpu_rwsem are scheduled to be merged in the coming v4.4-rc1 merge window which alleviates this issue. For now, revert the two commits to restore per-process rwsem. They will be re-applied for the v4.4-rc1 merge window. Signed-off-by: Tejun Heo <tj@kernel.org> Link: http://lkml.kernel.org/g/55F8097A.7000206@de.ibm.com Reported-by: Christian Borntraeger <borntraeger@de.ibm.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: stable@vger.kernel.org # v4.2+
* Revert "cgroup: simplify threadgroup locking"Tejun Heo2015-09-161-12/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit b5ba75b5fc0e8404e2c50cb68f39bb6a53fc916f. d59cfc09c32a ("sched, cgroup: replace signal_struct->group_rwsem with a global percpu_rwsem") and b5ba75b5fc0e ("cgroup: simplify threadgroup locking") changed how cgroup synchronizes against task fork and exits so that it uses global percpu_rwsem instead of per-process rwsem; unfortunately, the write [un]lock paths of percpu_rwsem always involve synchronize_rcu_expedited() which turned out to be too expensive. Improvements for percpu_rwsem are scheduled to be merged in the coming v4.4-rc1 merge window which alleviates this issue. For now, revert the two commits to restore per-process rwsem. They will be re-applied for the v4.4-rc1 merge window. Signed-off-by: Tejun Heo <tj@kernel.org> Link: http://lkml.kernel.org/g/55F8097A.7000206@de.ibm.com Reported-by: Christian Borntraeger <borntraeger@de.ibm.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: stable@vger.kernel.org # v4.2+
* Linux 4.3-rc1v4.3-rc1Linus Torvalds2015-09-121-2/+2
|
* Merge tag 'cris-for-4.3' of ↵Linus Torvalds2015-09-1250-421/+220
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/jesper/cris Pull CRIS updates from Jesper Nilsson: "Mostly removal of old cruft of which we can use a generic version, or fixes for code not commonly run in the cris port, but also additions to enable some good debug" * tag 'cris-for-4.3' of git://git.kernel.org/pub/scm/linux/kernel/git/jesper/cris: (25 commits) CRISv10: delete unused lib/dmacopy.c CRISv10: delete unused lib/old_checksum.c CRIS: fix switch_mm() lockdep splat CRISv32: enable LOCKDEP_SUPPORT CRIS: add STACKTRACE_SUPPORT CRISv32: annotate irq enable in idle loop CRISv32: add support for irqflags tracing CRIS: UAPI: use generic types.h CRIS: UAPI: use generic shmbuf.h CRIS: UAPI: use generic msgbuf.h CRIS: UAPI: use generic socket.h CRIS: UAPI: use generic sembuf.h CRIS: UAPI: use generic sockios.h CRIS: UAPI: use generic auxvec.h CRIS: UAPI: use generic headers via Kbuild CRIS: UAPI: fix elf.h export CRIS: don't make asm/elf.h depend on asm/user.h CRIS: UAPI: fix ptrace.h CRISv32: Squash compile warnings for axisflashmap CRISv32: Add GPIO driver to the default configs ...
| * CRISv10: delete unused lib/dmacopy.cRabin Vincent2015-09-051-42/+0
| | | | | | | | | | | | | | This file is never built. Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jespern@axis.com>
| * CRISv10: delete unused lib/old_checksum.cRabin Vincent2015-09-051-86/+0
| | | | | | | | | | | | | | This file is never built. Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: fix switch_mm() lockdep splatRabin Vincent2015-09-051-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With lockdep support implemented on CRISv32, we get the following splat. switch_mm() can be called both from the scheduler() (with interrupts disabled) and from flush_old_exec (via activate_mm()), with interrupts enabled. Fix it by disabling interrupts in activate_mm(), similar to powerpc and hexagon. t====================================================== [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ] 3.19.0-08802-g20bc9f1-dirty #323 Not tainted ------------------------------------------------------ init/1 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire: (mmu_context_lock){+.+...}, at: [<c0009290>] switch_mm+0x22/0xc6 and this task is already holding: (&rq->lock){-.-.-.}, at: [<c01a0756>] __schedule+0x5e/0x648 which would create a new lock dependency: (&rq->lock){-.-.-.} -> (mmu_context_lock){+.+...} but this new dependency connects a HARDIRQ-irq-safe lock: (&rq->lock){-.-.-.} ... which became HARDIRQ-irq-safe at: [<c002b03c>] scheduler_tick+0x28/0x5e [<c0007c6c>] timer_interrupt+0x4e/0x6a [<c0043ac4>] handle_irq_event_percpu+0x54/0x13c [<c004343c>] generic_handle_irq+0x2a/0x36 to a HARDIRQ-irq-unsafe lock: (mmu_context_lock){+.+...} ... which became HARDIRQ-irq-unsafe at: ... [<c0039e60>] __lock_acquire+0x8f8/0x1d9c [<c0009290>] switch_mm+0x22/0xc6 [<c009c260>] flush_old_exec+0x500/0x5d4 [<c00da4c6>] load_elf_phdrs+0x7a/0x84 [<c00dbdb0>] load_elf_binary+0x21c/0x13b4 [<c009cdb6>] do_execve+0x22/0x2c [<c001dcf2>] ____call_usermodehelper+0x0/0x154 [<c000581e>] ret_from_kernel_thread+0xe/0x14 other info that might help us debug this: Possible interrupt unsafe locking scenario: CPU0 CPU1 ---- ---- lock(mmu_context_lock); local_irq_disable(); lock(&rq->lock); lock(mmu_context_lock); <Interrupt> lock(&rq->lock); *** DEADLOCK *** 1 lock held by init/1: #0: (&rq->lock){-.-.-.}, at: [<c01a0756>] __schedule+0x5e/0x648 Call Trace: [<c019fe9e>] printk+0x0/0x4e [<c00368f8>] print_shortest_lock_dependencies+0x0/0x15c [<c0048628>] print_stack_trace+0x0/0x88 [<c0038912>] __lock_is_held+0x3e/0x5e [<c003b894>] lock_acquire+0x8a/0xcc [<c01a50c4>] _raw_spin_lock+0x44/0x7a [<c0009290>] switch_mm+0x22/0xc6 [<c01a06f8>] __schedule+0x0/0x648 [<c01a0d76>] schedule+0x36/0x7c [<c0037d04>] trace_hardirqs_on+0x0/0x1e [<c0004e18>] do_work_pending+0x30/0xd4 [<c000591a>] _work_pending+0xe/0x12 Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRISv32: enable LOCKDEP_SUPPORTRabin Vincent2015-09-051-0/+4
| | | | | | | | | | | | | | | | Now that we have stack tracing and irq flags tracing support, we can also enable lockdep support Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: add STACKTRACE_SUPPORTRabin Vincent2015-09-054-0/+88
| | | | | | | | | | | | | | | | | | | | Add stacktrace support, which is required for lockdep and tracing. The stack tracing simply looks at all kernel text symbols found on the stack, similar to the trap stack dumping code, which can also be converted to use this. Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRISv32: annotate irq enable in idle loopRabin Vincent2015-09-051-2/+2
| | | | | | | | | | | | | | | | Use a call to local_irq_enable() instead of incline asm so that the irqsoff latency tracer knows that interrupts are enabled here. Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRISv32: add support for irqflags tracingRabin Vincent2015-09-053-1/+20
| | | | | | | | | | | | | | | | Add support irqflags tracing, which is required for things like lockdep and ftrace. Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: UAPI: use generic types.hRabin Vincent2015-09-053-13/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CRIS' types.h is functionally identical to the asm-generic version. Effective diff: +#ifndef _ASM_GENERIC_TYPES_H +#define _ASM_GENERIC_TYPES_H + #include <asm-generic/int-ll64.h> + +#endif Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: UAPI: use generic shmbuf.hRabin Vincent2015-09-052-42/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CRIS' shmbuf.h is equivalent to the asm-generic verison. Effective diff: -#ifndef _CRIS_SHMBUF_H -#define _CRIS_SHMBUF_H +#ifndef __ASM_GENERIC_SHMBUF_H +#define __ASM_GENERIC_SHMBUF_H + +#include <asm/bitsperlong.h> struct ipc64_perm shm_perm; size_t shm_segsz; __kernel_time_t shm_atime; +#if __BITS_PER_LONG != 64 unsigned long __unused1; +#endif __kernel_time_t shm_dtime; +#if __BITS_PER_LONG != 64 unsigned long __unused2; +#endif __kernel_time_t shm_ctime; +#if __BITS_PER_LONG != 64 unsigned long __unused3; +#endif __kernel_pid_t shm_cpid; __kernel_pid_t shm_lpid; - unsigned long shm_nattch; - unsigned long __unused4; - unsigned long __unused5; + __kernel_ulong_t shm_nattch; + __kernel_ulong_t __unused4; + __kernel_ulong_t __unused5; }; struct shminfo64 { - unsigned long shmmax; - unsigned long shmmin; - unsigned long shmmni; - unsigned long shmseg; - unsigned long shmall; - unsigned long __unused1; - unsigned long __unused2; - unsigned long __unused3; - unsigned long __unused4; + __kernel_ulong_t shmmax; + __kernel_ulong_t shmmin; + __kernel_ulong_t shmmni; + __kernel_ulong_t shmseg; + __kernel_ulong_t shmall; + __kernel_ulong_t __unused1; + __kernel_ulong_t __unused2; + __kernel_ulong_t __unused3; + __kernel_ulong_t __unused4; }; #endif Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: UAPI: use generic msgbuf.hRabin Vincent2015-09-052-33/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CRIS' msgbuf.h is equivalent to the asm-generic version. Effective diff: -#ifndef _CRIS_MSGBUF_H -#define _CRIS_MSGBUF_H - - +#ifndef __ASM_GENERIC_MSGBUF_H +#define __ASM_GENERIC_MSGBUF_H +#include <asm/bitsperlong.h> struct msqid64_ds { struct ipc64_perm msg_perm; __kernel_time_t msg_stime; +#if __BITS_PER_LONG != 64 unsigned long __unused1; +#endif __kernel_time_t msg_rtime; +#if __BITS_PER_LONG != 64 unsigned long __unused2; +#endif __kernel_time_t msg_ctime; +#if __BITS_PER_LONG != 64 unsigned long __unused3; - unsigned long msg_cbytes; - unsigned long msg_qnum; - unsigned long msg_qbytes; +#endif + __kernel_ulong_t msg_cbytes; + __kernel_ulong_t msg_qnum; + __kernel_ulong_t msg_qbytes; __kernel_pid_t msg_lspid; __kernel_pid_t msg_lrpid; - unsigned long __unused4; - unsigned long __unused5; + __kernel_ulong_t __unused4; + __kernel_ulong_t __unused5; }; #endif Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: UAPI: use generic socket.hRabin Vincent2015-09-052-92/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CRIS' socket.h is equivalent to the asm-generic version. Effective diff: -#ifndef _ASM_SOCKET_H -#define _ASM_SOCKET_H - - +#ifndef __ASM_GENERIC_SOCKET_H +#define __ASM_GENERIC_SOCKET_H #include <asm/sockios.h> #define SO_LINGER 13 #define SO_BSDCOMPAT 14 #define SO_REUSEPORT 15 +#ifndef SO_PASSCRED #define SO_PASSCRED 16 #define SO_PEERCRED 17 #define SO_RCVLOWAT 18 #define SO_SNDLOWAT 19 #define SO_RCVTIMEO 20 #define SO_SNDTIMEO 21 +#endif #define SO_SECURITY_AUTHENTICATION 22 Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: UAPI: use generic sembuf.hRabin Vincent2015-09-052-25/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CRIS's sembuf.h is equivalent to the asm-generic version. Effective diff: -#ifndef _CRIS_SEMBUF_H -#define _CRIS_SEMBUF_H +#ifndef __ASM_GENERIC_SEMBUF_H +#define __ASM_GENERIC_SEMBUF_H +#include <asm/bitsperlong.h> struct semid64_ds { struct ipc64_perm sem_perm; __kernel_time_t sem_otime; +#if __BITS_PER_LONG != 64 unsigned long __unused1; +#endif __kernel_time_t sem_ctime; +#if __BITS_PER_LONG != 64 unsigned long __unused2; +#endif unsigned long sem_nsems; unsigned long __unused3; unsigned long __unused4; Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: UAPI: use generic sockios.hRabin Vincent2015-09-052-13/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | CRIS' sockios.h is equivalent to the asm-generic version. Effective diff: -#ifndef __ARCH_CRIS_SOCKIOS__ -#define __ARCH_CRIS_SOCKIOS__ +#ifndef __ASM_GENERIC_SOCKIOS_H +#define __ASM_GENERIC_SOCKIOS_H Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
| * CRIS: UAPI: use generic auxvec.hRabin Vincent2015-09-052-4/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CRIS's auxvec.h is empty just like the asm-generic version. Effective diff: -#ifndef __ASMCRIS_AUXVEC_H -#define __ASMCRIS_AUXVEC_H +#ifndef __ASM_GENERIC_AUXVEC_H +#define __ASM_GENERIC_AUXVEC_H + Signed-off-by: Rabin Vincent <rabin@rab.in> Signed-off-by: Jesper Nilsson <jesper.nilsson@axis.com>
OpenPOWER on IntegriCloud