diff options
Diffstat (limited to 'Documentation/cgroups')
| -rw-r--r-- | Documentation/cgroups/00-INDEX | 18 | ||||
| -rw-r--r-- | Documentation/cgroups/cgroups.txt | 42 | ||||
| -rw-r--r-- | Documentation/cgroups/cpusets.txt | 77 | ||||
| -rw-r--r-- | Documentation/cgroups/devices.txt | 2 | ||||
| -rw-r--r-- | Documentation/cgroups/memcg_test.txt | 22 | ||||
| -rw-r--r-- | Documentation/cgroups/memory.txt | 2 |
6 files changed, 112 insertions, 51 deletions
diff --git a/Documentation/cgroups/00-INDEX b/Documentation/cgroups/00-INDEX new file mode 100644 index 0000000..3f58fa3 --- /dev/null +++ b/Documentation/cgroups/00-INDEX @@ -0,0 +1,18 @@ +00-INDEX + - this file +cgroups.txt + - Control Groups definition, implementation details, examples and API. +cpuacct.txt + - CPU Accounting Controller; account CPU usage for groups of tasks. +cpusets.txt + - documents the cpusets feature; assign CPUs and Mem to a set of tasks. +devices.txt + - Device Whitelist Controller; description, interface and security. +freezer-subsystem.txt + - checkpointing; rationale to not use signals, interface. +memcg_test.txt + - Memory Resource Controller; implementation details. +memory.txt + - Memory Resource Controller; design, accounting, interface, testing. +resource_counter.txt + - Resource Counter API. diff --git a/Documentation/cgroups/cgroups.txt b/Documentation/cgroups/cgroups.txt index d9e5d6f..6eb1a97 100644 --- a/Documentation/cgroups/cgroups.txt +++ b/Documentation/cgroups/cgroups.txt @@ -56,7 +56,7 @@ hierarchy, and a set of subsystems; each subsystem has system-specific state attached to each cgroup in the hierarchy. Each hierarchy has an instance of the cgroup virtual filesystem associated with it. -At any one time there may be multiple active hierachies of task +At any one time there may be multiple active hierarchies of task cgroups. Each hierarchy is a partition of all tasks in the system. User level code may create and destroy cgroups by name in an @@ -124,10 +124,10 @@ following lines: / \ Prof (15%) students (5%) -Browsers like firefox/lynx go into the WWW network class, while (k)nfsd go +Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd go into NFS network class. -At the same time firefox/lynx will share an appropriate CPU/Memory class +At the same time Firefox/Lynx will share an appropriate CPU/Memory class depending on who launched it (prof/student). With the ability to classify tasks differently for different resources @@ -252,10 +252,8 @@ cgroup file system directories. When a task is moved from one cgroup to another, it gets a new css_set pointer - if there's an already existing css_set with the desired collection of cgroups then that group is reused, else a new -css_set is allocated. Note that the current implementation uses a -linear search to locate an appropriate existing css_set, so isn't -very efficient. A future version will use a hash table for better -performance. +css_set is allocated. The appropriate existing css_set is located by +looking into a hash table. To allow access from a cgroup to the css_sets (and hence tasks) that comprise it, a set of cg_cgroup_link objects form a lattice; @@ -327,7 +325,7 @@ and then start a subshell 'sh' in that cgroup: Creating, modifying, using the cgroups can be done through the cgroup virtual filesystem. -To mount a cgroup hierarchy will all available subsystems, type: +To mount a cgroup hierarchy with all available subsystems, type: # mount -t cgroup xxx /dev/cgroup The "xxx" is not interpreted by the cgroup code, but will appear in @@ -335,12 +333,23 @@ The "xxx" is not interpreted by the cgroup code, but will appear in To mount a cgroup hierarchy with just the cpuset and numtasks subsystems, type: -# mount -t cgroup -o cpuset,numtasks hier1 /dev/cgroup +# mount -t cgroup -o cpuset,memory hier1 /dev/cgroup To change the set of subsystems bound to a mounted hierarchy, just remount with different options: +# mount -o remount,cpuset,ns hier1 /dev/cgroup -# mount -o remount,cpuset,ns /dev/cgroup +Now memory is removed from the hierarchy and ns is added. + +Note this will add ns to the hierarchy but won't remove memory or +cpuset, because the new options are appended to the old ones: +# mount -o remount,ns /dev/cgroup + +To Specify a hierarchy's release_agent: +# mount -t cgroup -o cpuset,release_agent="/sbin/cpuset_release_agent" \ + xxx /dev/cgroup + +Note that specifying 'release_agent' more than once will return failure. Note that changing the set of subsystems is currently only supported when the hierarchy consists of a single (root) cgroup. Supporting @@ -351,6 +360,11 @@ Then under /dev/cgroup you can find a tree that corresponds to the tree of the cgroups in the system. For instance, /dev/cgroup is the cgroup that holds the whole system. +If you want to change the value of release_agent: +# echo "/sbin/new_release_agent" > /dev/cgroup/release_agent + +It can also be changed via remount. + If you want to create a new cgroup under /dev/cgroup: # cd /dev/cgroup # mkdir my_cgroup @@ -478,11 +492,13 @@ cgroup->parent is still valid. (Note - can also be called for a newly-created cgroup if an error occurs after this subsystem's create() method has been called for the new cgroup). -void pre_destroy(struct cgroup_subsys *ss, struct cgroup *cgrp); +int pre_destroy(struct cgroup_subsys *ss, struct cgroup *cgrp); Called before checking the reference count on each subsystem. This may be useful for subsystems which have some extra references even if -there are not tasks in the cgroup. +there are not tasks in the cgroup. If pre_destroy() returns error code, +rmdir() will fail with it. From this behavior, pre_destroy() can be +called multiple times against a cgroup. int can_attach(struct cgroup_subsys *ss, struct cgroup *cgrp, struct task_struct *task) @@ -523,7 +539,7 @@ always handled well. void post_clone(struct cgroup_subsys *ss, struct cgroup *cgrp) (cgroup_mutex held by caller) -Called at the end of cgroup_clone() to do any paramater +Called at the end of cgroup_clone() to do any parameter initialization which might be required before a task could attach. For example in cpusets, no task may attach before 'cpus' and 'mems' are set up. diff --git a/Documentation/cgroups/cpusets.txt b/Documentation/cgroups/cpusets.txt index 5c86c25..f9ca389 100644 --- a/Documentation/cgroups/cpusets.txt +++ b/Documentation/cgroups/cpusets.txt @@ -131,7 +131,7 @@ Cpusets extends these two mechanisms as follows: - The hierarchy of cpusets can be mounted at /dev/cpuset, for browsing and manipulation from user space. - A cpuset may be marked exclusive, which ensures that no other - cpuset (except direct ancestors and descendents) may contain + cpuset (except direct ancestors and descendants) may contain any overlapping CPUs or Memory Nodes. - You can list all the tasks (by pid) attached to any cpuset. @@ -142,7 +142,7 @@ into the rest of the kernel, none in performance critical paths: - in fork and exit, to attach and detach a task from its cpuset. - in sched_setaffinity, to mask the requested CPUs by what's allowed in that tasks cpuset. - - in sched.c migrate_all_tasks(), to keep migrating tasks within + - in sched.c migrate_live_tasks(), to keep migrating tasks within the CPUs allowed by their cpuset, if possible. - in the mbind and set_mempolicy system calls, to mask the requested Memory Nodes by what's allowed in that tasks cpuset. @@ -175,6 +175,10 @@ files describing that cpuset: - mem_exclusive flag: is memory placement exclusive? - mem_hardwall flag: is memory allocation hardwalled - memory_pressure: measure of how much paging pressure in cpuset + - memory_spread_page flag: if set, spread page cache evenly on allowed nodes + - memory_spread_slab flag: if set, spread slab cache evenly on allowed nodes + - sched_load_balance flag: if set, load balance within CPUs on that cpuset + - sched_relax_domain_level: the searching range when migrating tasks In addition, the root cpuset only has the following file: - memory_pressure_enabled flag: compute memory_pressure? @@ -222,7 +226,7 @@ nodes with memory--using the cpuset_track_online_nodes() hook. -------------------------------- If a cpuset is cpu or mem exclusive, no other cpuset, other than -a direct ancestor or descendent, may share any of the same CPUs or +a direct ancestor or descendant, may share any of the same CPUs or Memory Nodes. A cpuset that is mem_exclusive *or* mem_hardwall is "hardwalled", @@ -252,7 +256,7 @@ is causing. This is useful both on tightly managed systems running a wide mix of submitted jobs, which may choose to terminate or re-prioritize jobs that -are trying to use more memory than allowed on the nodes assigned them, +are trying to use more memory than allowed on the nodes assigned to them, and with tightly coupled, long running, massively parallel scientific computing jobs that will dramatically fail to meet required performance goals if they start to use more memory than allowed to them. @@ -378,7 +382,7 @@ as cpusets and sched_setaffinity. The algorithmic cost of load balancing and its impact on key shared kernel data structures such as the task list increases more than linearly with the number of CPUs being balanced. So the scheduler -has support to partition the systems CPUs into a number of sched +has support to partition the systems CPUs into a number of sched domains such that it only load balances within each sched domain. Each sched domain covers some subset of the CPUs in the system; no two sched domains overlap; some CPUs might not be in any sched @@ -423,7 +427,7 @@ child cpusets have this flag enabled. When doing this, you don't usually want to leave any unpinned tasks in the top cpuset that might use non-trivial amounts of CPU, as such tasks may be artificially constrained to some subset of CPUs, depending on -the particulars of this flag setting in descendent cpusets. Even if +the particulars of this flag setting in descendant cpusets. Even if such a task could use spare CPU cycles in some other CPUs, the kernel scheduler might not consider the possibility of load balancing that task to that underused CPU. @@ -485,17 +489,22 @@ of CPUs allowed to a cpuset having 'sched_load_balance' enabled. The internal kernel cpuset to scheduler interface passes from the cpuset code to the scheduler code a partition of the load balanced CPUs in the system. This partition is a set of subsets (represented -as an array of cpumask_t) of CPUs, pairwise disjoint, that cover all -the CPUs that must be load balanced. - -Whenever the 'sched_load_balance' flag changes, or CPUs come or go -from a cpuset with this flag enabled, or a cpuset with this flag -enabled is removed, the cpuset code builds a new such partition and -passes it to the scheduler sched domain setup code, to have the sched -domains rebuilt as necessary. +as an array of struct cpumask) of CPUs, pairwise disjoint, that cover +all the CPUs that must be load balanced. + +The cpuset code builds a new such partition and passes it to the +scheduler sched domain setup code, to have the sched domains rebuilt +as necessary, whenever: + - the 'sched_load_balance' flag of a cpuset with non-empty CPUs changes, + - or CPUs come or go from a cpuset with this flag enabled, + - or 'sched_relax_domain_level' value of a cpuset with non-empty CPUs + and with this flag enabled changes, + - or a cpuset with non-empty CPUs and with this flag enabled is removed, + - or a cpu is offlined/onlined. This partition exactly defines what sched domains the scheduler should -setup - one sched domain for each element (cpumask_t) in the partition. +setup - one sched domain for each element (struct cpumask) in the +partition. The scheduler remembers the currently active sched domain partitions. When the scheduler routine partition_sched_domains() is invoked from @@ -522,9 +531,9 @@ be idle. Of course it takes some searching cost to find movable tasks and/or idle CPUs, the scheduler might not search all CPUs in the domain -everytime. In fact, in some architectures, the searching ranges on +every time. In fact, in some architectures, the searching ranges on events are limited in the same socket or node where the CPU locates, -while the load balance on tick searchs all. +while the load balance on tick searches all. For example, assume CPU Z is relatively far from CPU X. Even if CPU Z is idle while CPU X and the siblings are busy, scheduler can't migrate @@ -559,7 +568,7 @@ domain, the largest value among those is used. Be careful, if one requests 0 and others are -1 then 0 is used. Note that modifying this file will have both good and bad effects, -and whether it is acceptable or not will be depend on your situation. +and whether it is acceptable or not depends on your situation. Don't modify this file if you are not sure. If your situation is: @@ -592,7 +601,7 @@ its new cpuset, then the task will continue to use whatever subset of MPOL_BIND nodes are still allowed in the new cpuset. If the task was using MPOL_BIND and now none of its MPOL_BIND nodes are allowed in the new cpuset, then the task will be essentially treated as if it -was MPOL_BIND bound to the new cpuset (even though its numa placement, +was MPOL_BIND bound to the new cpuset (even though its NUMA placement, as queried by get_mempolicy(), doesn't change). If a task is moved from one cpuset to another, then the kernel will adjust the tasks memory placement, as above, the next time that the kernel attempts @@ -600,19 +609,15 @@ to allocate a page of memory for that task. If a cpuset has its 'cpus' modified, then each task in that cpuset will have its allowed CPU placement changed immediately. Similarly, -if a tasks pid is written to a cpusets 'tasks' file, in either its -current cpuset or another cpuset, then its allowed CPU placement is -changed immediately. If such a task had been bound to some subset -of its cpuset using the sched_setaffinity() call, the task will be -allowed to run on any CPU allowed in its new cpuset, negating the -affect of the prior sched_setaffinity() call. +if a tasks pid is written to another cpusets 'tasks' file, then its +allowed CPU placement is changed immediately. If such a task had been +bound to some subset of its cpuset using the sched_setaffinity() call, +the task will be allowed to run on any CPU allowed in its new cpuset, +negating the effect of the prior sched_setaffinity() call. In summary, the memory placement of a task whose cpuset is changed is updated by the kernel, on the next allocation of a page for that task, -but the processor placement is not updated, until that tasks pid is -rewritten to the 'tasks' file of its cpuset. This is done to avoid -impacting the scheduler code in the kernel with a check for changes -in a tasks processor placement. +and the processor placement is updated immediately. Normally, once a page is allocated (given a physical page of main memory) then that page stays on whatever node it @@ -681,10 +686,14 @@ and then start a subshell 'sh' in that cpuset: # The next line should display '/Charlie' cat /proc/self/cpuset -In the future, a C library interface to cpusets will likely be -available. For now, the only way to query or modify cpusets is -via the cpuset file system, using the various cd, mkdir, echo, cat, -rmdir commands from the shell, or their equivalent from C. +There are ways to query or modify cpusets: + - via the cpuset file system directly, using the various cd, mkdir, echo, + cat, rmdir commands from the shell, or their equivalent from C. + - via the C library libcpuset. + - via the C library libcgroup. + (http://sourceforge.net/proects/libcg/) + - via the python application cset. + (http://developer.novell.com/wiki/index.php/Cpuset) The sched_setaffinity calls can also be done at the shell prompt using SGI's runon or Robert Love's taskset. The mbind and set_mempolicy @@ -756,7 +765,7 @@ mount -t cpuset X /dev/cpuset is equivalent to -mount -t cgroup -ocpuset X /dev/cpuset +mount -t cgroup -ocpuset,noprefix X /dev/cpuset echo "/sbin/cpuset_release_agent" > /dev/cpuset/release_agent 2.2 Adding/removing cpus diff --git a/Documentation/cgroups/devices.txt b/Documentation/cgroups/devices.txt index 7cc6e6a..57ca4c8 100644 --- a/Documentation/cgroups/devices.txt +++ b/Documentation/cgroups/devices.txt @@ -42,7 +42,7 @@ suffice, but we can decide the best way to adequately restrict movement as people get some experience with this. We may just want to require CAP_SYS_ADMIN, which at least is a separate bit from CAP_MKNOD. We may want to just refuse moving to a cgroup which -isn't a descendent of the current one. Or we may want to use +isn't a descendant of the current one. Or we may want to use CAP_MAC_ADMIN, since we really are trying to lock down root. CAP_SYS_ADMIN is needed to modify the whitelist or move another diff --git a/Documentation/cgroups/memcg_test.txt b/Documentation/cgroups/memcg_test.txt index 523a9c1..72db89e 100644 --- a/Documentation/cgroups/memcg_test.txt +++ b/Documentation/cgroups/memcg_test.txt @@ -1,5 +1,5 @@ Memory Resource Controller(Memcg) Implementation Memo. -Last Updated: 2009/1/19 +Last Updated: 2009/1/20 Base Kernel Version: based on 2.6.29-rc2. Because VM is getting complex (one of reasons is memcg...), memcg's behavior @@ -356,7 +356,25 @@ Under below explanation, we assume CONFIG_MEM_RES_CTRL_SWAP=y. (Shell-B) # move all tasks in /cgroup/test to /cgroup # /sbin/swapoff -a - # rmdir /test/cgroup + # rmdir /cgroup/test # kill malloc task. Of course, tmpfs v.s. swapoff test should be tested, too. + + 9.8 OOM-Killer + Out-of-memory caused by memcg's limit will kill tasks under + the memcg. When hierarchy is used, a task under hierarchy + will be killed by the kernel. + In this case, panic_on_oom shouldn't be invoked and tasks + in other groups shouldn't be killed. + + It's not difficult to cause OOM under memcg as following. + Case A) when you can swapoff + #swapoff -a + #echo 50M > /memory.limit_in_bytes + run 51M of malloc + + Case B) when you use mem+swap limitation. + #echo 50M > memory.limit_in_bytes + #echo 50M > memory.memsw.limit_in_bytes + run 51M of malloc diff --git a/Documentation/cgroups/memory.txt b/Documentation/cgroups/memory.txt index e150196..a98a7fe 100644 --- a/Documentation/cgroups/memory.txt +++ b/Documentation/cgroups/memory.txt @@ -302,7 +302,7 @@ will be charged as a new owner of it. unevictable - # of pages cannot be reclaimed.(mlocked etc) Below is depend on CONFIG_DEBUG_VM. - inactive_ratio - VM inernal parameter. (see mm/page_alloc.c) + inactive_ratio - VM internal parameter. (see mm/page_alloc.c) recent_rotated_anon - VM internal parameter. (see mm/vmscan.c) recent_rotated_file - VM internal parameter. (see mm/vmscan.c) recent_scanned_anon - VM internal parameter. (see mm/vmscan.c) |
