diff options
Diffstat (limited to 'Documentation/admin-guide/cgroup-v2.rst')
| -rw-r--r-- | Documentation/admin-guide/cgroup-v2.rst | 453 |
1 files changed, 340 insertions, 113 deletions
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index bf842b80bde9..17e6e9565156 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -210,6 +210,35 @@ cgroup v2 currently supports the following mount options. relying on the original semantics (e.g. specifying bogusly high 'bypass' protection values at higher tree levels). + memory_hugetlb_accounting + Count HugeTLB memory usage towards the cgroup's overall + memory usage for the memory controller (for the purpose of + statistics reporting and memory protetion). This is a new + behavior that could regress existing setups, so it must be + explicitly opted in with this mount option. + + A few caveats to keep in mind: + + * There is no HugeTLB pool management involved in the memory + controller. The pre-allocated pool does not belong to anyone. + Specifically, when a new HugeTLB folio is allocated to + the pool, it is not accounted for from the perspective of the + memory controller. It is only charged to a cgroup when it is + actually used (for e.g at page fault time). Host memory + overcommit management has to consider this when configuring + hard limits. In general, HugeTLB pool management should be + done via other mechanisms (such as the HugeTLB controller). + * Failure to charge a HugeTLB folio to the memory controller + results in SIGBUS. This could happen even if the HugeTLB pool + still has pages available (but the cgroup limit is hit and + reclaim attempt fails). + * Charging HugeTLB memory towards the memory controller affects + memory protection and reclaim dynamics. Any userspace tuning + (of low, min limits for e.g) needs to take this into account. + * HugeTLB pages utilized while this option is not selected + will not be tracked by the memory controller (even if cgroup + v2 is remounted later on). + Organizing Processes and Threads -------------------------------- @@ -364,6 +393,13 @@ constraint, a threaded controller must be able to handle competition between threads in a non-leaf cgroup and its child cgroups. Each threaded controller defines how such competitions are handled. +Currently, the following controllers are threaded and can be enabled +in a threaded cgroup:: + +- cpu +- cpuset +- perf_event +- pids [Un]populated Notification -------------------------- @@ -619,10 +655,12 @@ process migrations. and is an example of this type. +.. _cgroupv2-limits-distributor: + Limits ------ -A child can only consume upto the configured amount of the resource. +A child can only consume up to the configured amount of the resource. Limits can be over-committed - the sum of the limits of children can exceed the amount of resource available to the parent. @@ -635,15 +673,16 @@ process migrations. "io.max" limits the maximum BPS and/or IOPS that a cgroup can consume on an IO device and is an example of this type. +.. _cgroupv2-protections-distributor: Protections ----------- -A cgroup is protected upto the configured amount of the resource +A cgroup is protected up to the configured amount of the resource as long as the usages of all its ancestors are under their protected levels. Protections can be hard guarantees or best effort soft boundaries. Protections can also be over-committed in which case -only upto the amount available to the parent is protected among +only up to the amount available to the parent is protected among children. Protections are in the range [0, max] and defaults to 0, which is @@ -976,6 +1015,29 @@ All cgroup core files are prefixed with "cgroup." killing cgroups is a process directed operation, i.e. it affects the whole thread-group. + cgroup.pressure + A read-write single value file that allowed values are "0" and "1". + The default is "1". + + Writing "0" to the file will disable the cgroup PSI accounting. + Writing "1" to the file will re-enable the cgroup PSI accounting. + + This control attribute is not hierarchical, so disable or enable PSI + accounting in a cgroup does not affect PSI accounting in descendants + and doesn't need pass enablement via ancestors from root. + + The reason this control attribute exists is that PSI accounts stalls for + each cgroup separately and aggregates it at each level of the hierarchy. + This may cause non-negligible overhead for some workloads when under + deep level of the hierarchy, in which case this control attribute can + be used to disable PSI accounting in the non-leaf cgroups. + + irq.pressure + A read-write nested-keyed file. + + Shows pressure stall information for IRQ/SOFTIRQ. See + :ref:`Documentation/accounting/psi.rst <psi>` for details. + Controllers =========== @@ -1019,7 +1081,7 @@ All time durations are in microseconds. - user_usec - system_usec - and the following three when the controller is enabled: + and the following five when the controller is enabled: - nr_periods - nr_throttled @@ -1031,7 +1093,11 @@ All time durations are in microseconds. A read-write single value file which exists on non-root cgroups. The default is "100". - The weight in the range [1, 10000]. + For non idle groups (cpu.idle = 0), the weight is in the + range [1, 10000]. + + If the cgroup has been configured to be SCHED_IDLE (cpu.idle = 1), + then the weight will show as a 0. cpu.weight.nice A read-write single value file which exists on non-root @@ -1053,7 +1119,7 @@ All time durations are in microseconds. $MAX $PERIOD - which indicates that the group may consume upto $MAX in each + which indicates that the group may consume up to $MAX in each $PERIOD duration. "max" for $MAX indicates no limit. If only one number is written, $MAX is updated. @@ -1095,6 +1161,16 @@ All time durations are in microseconds. values similar to the sched_setattr(2). This maximum utilization value is used to clamp the task specific maximum utilization clamp. + cpu.idle + A read-write single value file which exists on non-root cgroups. + The default is 0. + + This is the cgroup analog of the per-task SCHED_IDLE sched policy. + Setting this value to a 1 will make the scheduling policy of the + cgroup SCHED_IDLE. The threads inside the cgroup will retain their + own relative priorities, but the cgroup itself will be treated as + very low priority relative to its peers. + Memory @@ -1187,23 +1263,25 @@ PAGE_SIZE multiple when read back. A read-write single value file which exists on non-root cgroups. The default is "max". - Memory usage throttle limit. This is the main mechanism to - control memory usage of a cgroup. If a cgroup's usage goes + Memory usage throttle limit. If a cgroup's usage goes over the high boundary, the processes of the cgroup are throttled and put under heavy reclaim pressure. Going over the high limit never invokes the OOM killer and - under extreme conditions the limit may be breached. + under extreme conditions the limit may be breached. The high + limit should be used in scenarios where an external process + monitors the limited cgroup to alleviate heavy reclaim + pressure. memory.max A read-write single value file which exists on non-root cgroups. The default is "max". - Memory usage hard limit. This is the final protection - mechanism. If a cgroup's memory usage reaches this limit and - can't be reduced, the OOM killer is invoked in the cgroup. - Under certain circumstances, the usage may go over the limit - temporarily. + Memory usage hard limit. This is the main mechanism to limit + memory usage of a cgroup. If a cgroup's memory usage reaches + this limit and can't be reduced, the OOM killer is invoked in + the cgroup. Under certain circumstances, the usage may go + over the limit temporarily. In default configuration regular 0-order allocations always succeed unless OOM killer chooses current task as a victim. @@ -1212,10 +1290,6 @@ PAGE_SIZE multiple when read back. Caller could retry them differently, return into userspace as -ENOMEM or silently ignore in cases like disk readahead. - This is the ultimate protection mechanism. As long as the - high limit is used and monitored properly, this limit's - utility is limited to providing the final safety net. - memory.reclaim A write-only nested-keyed file which exists for all cgroups. @@ -1237,6 +1311,13 @@ PAGE_SIZE multiple when read back. the target cgroup. If less bytes are reclaimed than the specified amount, -EAGAIN is returned. + Please note that the proactive reclaim (triggered by this + interface) is not meant to indicate memory pressure on the + memory cgroup. Therefore socket memory balancing triggered by + the memory reclaim normally is not exercised in this case. + This means that the networking layer will not adapt based on + reclaim induced by memory.reclaim. + memory.peak A read-only single value file which exists on non-root cgroups. @@ -1348,6 +1429,11 @@ PAGE_SIZE multiple when read back. pagetables Amount of memory allocated for page tables. + sec_pagetables + Amount of memory allocated for secondary page tables, + this currently includes KVM mmu allocations on x86 + and arm64. + percpu (npn) Amount of memory used for storing per-cpu kernel data structures. @@ -1441,6 +1527,30 @@ PAGE_SIZE multiple when read back. workingset_nodereclaim Number of times a shadow node has been reclaimed + pgscan (npn) + Amount of scanned pages (in an inactive LRU list) + + pgsteal (npn) + Amount of reclaimed pages + + pgscan_kswapd (npn) + Amount of scanned pages by kswapd (in an inactive LRU list) + + pgscan_direct (npn) + Amount of scanned pages directly (in an inactive LRU list) + + pgscan_khugepaged (npn) + Amount of scanned pages by khugepaged (in an inactive LRU list) + + pgsteal_kswapd (npn) + Amount of reclaimed pages by kswapd + + pgsteal_direct (npn) + Amount of reclaimed pages directly + + pgsteal_khugepaged (npn) + Amount of reclaimed pages by khugepaged + pgfault (npn) Total number of page faults incurred @@ -1450,12 +1560,6 @@ PAGE_SIZE multiple when read back. pgrefill (npn) Amount of scanned pages (in an active LRU list) - pgscan (npn) - Amount of scanned pages (in an inactive LRU list) - - pgsteal (npn) - Amount of reclaimed pages - pgactivate (npn) Amount of pages moved to the active LRU list @@ -1478,6 +1582,15 @@ PAGE_SIZE multiple when read back. collapsing an existing range of pages. This counter is not present when CONFIG_TRANSPARENT_HUGEPAGE is not set. + thp_swpout (npn) + Number of transparent hugepages which are swapout in one piece + without splitting. + + thp_swpout_fallback (npn) + Number of transparent hugepages which were split before swapout. + Usually because failed to allocate some continuous swap space + for the huge page. + memory.numa_stat A read-only nested-keyed file which exists on non-root cgroups. @@ -1526,6 +1639,13 @@ PAGE_SIZE multiple when read back. Healthy workloads are not expected to reach this limit. + memory.swap.peak + A read-only single value file which exists on non-root + cgroups. + + The max swap usage recorded for the cgroup and its + descendants since the creation of the cgroup. + memory.swap.max A read-write single value file which exists on non-root cgroups. The default is "max". @@ -1573,6 +1693,21 @@ PAGE_SIZE multiple when read back. limit, it will refuse to take any more stores before existing entries fault back in or are written out to disk. + memory.zswap.writeback + A read-write single value file. The default value is "1". The + initial value of the root cgroup is 1, and when a new cgroup is + created, it inherits the current value of its parent. + + When this is set to 0, all swapping attempts to swapping devices + are disabled. This included both zswap writebacks, and swapping due + to zswap store failures. If the zswap store failures are recurring + (for e.g if the pages are incompressible), users can observe + reclaim inefficiency after disabling writeback (because the same + pages might be rejected again and again). + + Note that this is subtly different from setting memory.swap.max to + 0, as it still allows for pages to be written to the zswap pool. + memory.pressure A read-only nested-keyed file. @@ -1962,37 +2097,41 @@ IO Priority ~~~~~~~~~~~ A single attribute controls the behavior of the I/O priority cgroup policy, -namely the blkio.prio.class attribute. The following values are accepted for +namely the io.prio.class attribute. The following values are accepted for that attribute: no-change Do not modify the I/O priority class. - none-to-rt - For requests that do not have an I/O priority class (NONE), - change the I/O priority class into RT. Do not modify - the I/O priority class of other requests. + promote-to-rt + For requests that have a non-RT I/O priority class, change it into RT. + Also change the priority level of these requests to 4. Do not modify + the I/O priority of requests that have priority class RT. restrict-to-be For requests that do not have an I/O priority class or that have I/O - priority class RT, change it into BE. Do not modify the I/O priority - class of requests that have priority class IDLE. + priority class RT, change it into BE. Also change the priority level + of these requests to 0. Do not modify the I/O priority class of + requests that have priority class IDLE. idle Change the I/O priority class of all requests into IDLE, the lowest I/O priority class. + none-to-rt + Deprecated. Just an alias for promote-to-rt. + The following numerical values are associated with the I/O priority policies: -+-------------+---+ -| no-change | 0 | -+-------------+---+ -| none-to-rt | 1 | -+-------------+---+ -| rt-to-be | 2 | -+-------------+---+ -| all-to-idle | 3 | -+-------------+---+ ++----------------+---+ +| no-change | 0 | ++----------------+---+ +| promote-to-rt | 1 | ++----------------+---+ +| restrict-to-be | 2 | ++----------------+---+ +| idle | 3 | ++----------------+---+ The numerical value that corresponds to each I/O priority class is as follows: @@ -2008,9 +2147,13 @@ The numerical value that corresponds to each I/O priority class is as follows: The algorithm to set the I/O priority class for a request is as follows: -- Translate the I/O priority class policy into a number. -- Change the request I/O priority class into the maximum of the I/O priority - class policy number and the numerical I/O priority class. +- If I/O priority class policy is promote-to-rt, change the request I/O + priority class to IOPRIO_CLASS_RT and change the request I/O priority + level to 4. +- If I/O priority class policy is not promote-to-rt, translate the I/O priority + class policy into a number, then change the request I/O priority class + into the maximum of the I/O priority class policy number and the numerical + I/O priority class. PID --- @@ -2159,6 +2302,56 @@ Cpuset Interface Files Its value will be affected by memory nodes hotplug events. + cpuset.cpus.exclusive + A read-write multiple values file which exists on non-root + cpuset-enabled cgroups. + + It lists all the exclusive CPUs that are allowed to be used + to create a new cpuset partition. Its value is not used + unless the cgroup becomes a valid partition root. See the + "cpuset.cpus.partition" section below for a description of what + a cpuset partition is. + + When the cgroup becomes a partition root, the actual exclusive + CPUs that are allocated to that partition are listed in + "cpuset.cpus.exclusive.effective" which may be different + from "cpuset.cpus.exclusive". If "cpuset.cpus.exclusive" + has previously been set, "cpuset.cpus.exclusive.effective" + is always a subset of it. + + Users can manually set it to a value that is different from + "cpuset.cpus". The only constraint in setting it is that the + list of CPUs must be exclusive with respect to its sibling. + + For a parent cgroup, any one of its exclusive CPUs can only + be distributed to at most one of its child cgroups. Having an + exclusive CPU appearing in two or more of its child cgroups is + not allowed (the exclusivity rule). A value that violates the + exclusivity rule will be rejected with a write error. + + The root cgroup is a partition root and all its available CPUs + are in its exclusive CPU set. + + cpuset.cpus.exclusive.effective + A read-only multiple values file which exists on all non-root + cpuset-enabled cgroups. + + This file shows the effective set of exclusive CPUs that + can be used to create a partition root. The content of this + file will always be a subset of "cpuset.cpus" and its parent's + "cpuset.cpus.exclusive.effective" if its parent is not the root + cgroup. It will also be a subset of "cpuset.cpus.exclusive" + if it is set. If "cpuset.cpus.exclusive" is not set, it is + treated to have an implicit value of "cpuset.cpus" in the + formation of local partition. + + cpuset.cpus.isolated + A read-only and root cgroup only multiple values file. + + This file shows the set of all isolated CPUs used in existing + isolated partitions. It will be empty if no isolated partition + is created. + cpuset.cpus.partition A read-write single value file which exists on non-root cpuset-enabled cgroups. This flag is owned by the parent cgroup @@ -2166,75 +2359,109 @@ Cpuset Interface Files It accepts only the following input values when written to. - ======== ================================ - "root" a partition root - "member" a non-root member of a partition - ======== ================================ - - When set to be a partition root, the current cgroup is the - root of a new partition or scheduling domain that comprises - itself and all its descendants except those that are separate - partition roots themselves and their descendants. The root - cgroup is always a partition root. - - There are constraints on where a partition root can be set. - It can only be set in a cgroup if all the following conditions - are true. - - 1) The "cpuset.cpus" is not empty and the list of CPUs are - exclusive, i.e. they are not shared by any of its siblings. - 2) The parent cgroup is a partition root. - 3) The "cpuset.cpus" is also a proper subset of the parent's - "cpuset.cpus.effective". - 4) There is no child cgroups with cpuset enabled. This is for - eliminating corner cases that have to be handled if such a - condition is allowed. - - Setting it to partition root will take the CPUs away from the - effective CPUs of the parent cgroup. Once it is set, this - file cannot be reverted back to "member" if there are any child - cgroups with cpuset enabled. - - A parent partition cannot distribute all its CPUs to its - child partitions. There must be at least one cpu left in the - parent partition. - - Once becoming a partition root, changes to "cpuset.cpus" is - generally allowed as long as the first condition above is true, - the change will not take away all the CPUs from the parent - partition and the new "cpuset.cpus" value is a superset of its - children's "cpuset.cpus" values. - - Sometimes, external factors like changes to ancestors' - "cpuset.cpus" or cpu hotplug can cause the state of the partition - root to change. On read, the "cpuset.sched.partition" file - can show the following values. - - ============== ============================== - "member" Non-root member of a partition - "root" Partition root - "root invalid" Invalid partition root - ============== ============================== - - It is a partition root if the first 2 partition root conditions - above are true and at least one CPU from "cpuset.cpus" is - granted by the parent cgroup. - - A partition root can become invalid if none of CPUs requested - in "cpuset.cpus" can be granted by the parent cgroup or the - parent cgroup is no longer a partition root itself. In this - case, it is not a real partition even though the restriction - of the first partition root condition above will still apply. - The cpu affinity of all the tasks in the cgroup will then be - associated with CPUs in the nearest ancestor partition. - - An invalid partition root can be transitioned back to a - real partition root if at least one of the requested CPUs - can now be granted by its parent. In this case, the cpu - affinity of all the tasks in the formerly invalid partition - will be associated to the CPUs of the newly formed partition. - Changing the partition state of an invalid partition root to - "member" is always allowed even if child cpusets are present. + ========== ===================================== + "member" Non-root member of a partition + "root" Partition root + "isolated" Partition root without load balancing + ========== ===================================== + + A cpuset partition is a collection of cpuset-enabled cgroups with + a partition root at the top of the hierarchy and its descendants + except those that are separate partition roots themselves and + their descendants. A partition has exclusive access to the + set of exclusive CPUs allocated to it. Other cgroups outside + of that partition cannot use any CPUs in that set. + + There are two types of partitions - local and remote. A local + partition is one whose parent cgroup is also a valid partition + root. A remote partition is one whose parent cgroup is not a + valid partition root itself. Writing to "cpuset.cpus.exclusive" + is optional for the creation of a local partition as its + "cpuset.cpus.exclusive" file will assume an implicit value that + is the same as "cpuset.cpus" if it is not set. Writing the + proper "cpuset.cpus.exclusive" values down the cgroup hierarchy + before the target partition root is mandatory for the creation + of a remote partition. + + Currently, a remote partition cannot be created under a local + partition. All the ancestors of a remote partition root except + the root cgroup cannot be a partition root. + + The root cgroup is always a partition root and its state cannot + be changed. All other non-root cgroups start out as "member". + + When set to "root", the current cgroup is the root of a new + partition or scheduling domain. The set of exclusive CPUs is + determined by the value of its "cpuset.cpus.exclusive.effective". + + When set to "isolated", the CPUs in that partition will be in + an isolated state without any load balancing from the scheduler + and excluded from the unbound workqueues. Tasks placed in such + a partition with multiple CPUs should be carefully distributed + and bound to each of the individual CPUs for optimal performance. + + A partition root ("root" or "isolated") can be in one of the + two possible states - valid or invalid. An invalid partition + root is in a degraded state where some state information may + be retained, but behaves more like a "member". + + All possible state transitions among "member", "root" and + "isolated" are allowed. + + On read, the "cpuset.cpus.partition" file can show the following + values. + + ============================= ===================================== + "member" Non-root member of a partition + "root" Partition root + "isolated" Partition root without load balancing + "root invalid (<reason>)" Invalid partition root + "isolated invalid (<reason>)" Invalid isolated partition root + ============================= ===================================== + + In the case of an invalid partition root, a descriptive string on + why the partition is invalid is included within parentheses. + + For a local partition root to be valid, the following conditions + must be met. + + 1) The parent cgroup is a valid partition root. + 2) The "cpuset.cpus.exclusive.effective" file cannot be empty, + though it may contain offline CPUs. + 3) The "cpuset.cpus.effective" cannot be empty unless there is + no task associated with this partition. + + For a remote partition root to be valid, all the above conditions + except the first one must be met. + + External events like hotplug or changes to "cpuset.cpus" or + "cpuset.cpus.exclusive" can cause a valid partition root to + become invalid and vice versa. Note that a task cannot be + moved to a cgroup with empty "cpuset.cpus.effective". + + A valid non-root parent partition may distribute out all its CPUs + to its child local partitions when there is no task associated + with it. + + Care must be taken to change a valid partition root to "member" + as all its child local partitions, if present, will become + invalid causing disruption to tasks running in those child + partitions. These inactivated partitions could be recovered if + their parent is switched back to a partition root with a proper + value in "cpuset.cpus" or "cpuset.cpus.exclusive". + + Poll and inotify events are triggered whenever the state of + "cpuset.cpus.partition" changes. That includes changes caused + by write to "cpuset.cpus.partition", cpu hotplug or other + changes that modify the validity status of the partition. + This will allow user space agents to monitor unexpected changes + to "cpuset.cpus.partition" without the need to do continuous + polling. + + A user can pre-configure certain CPUs to an isolated state + with load balancing disabled at boot time with the "isolcpus" + kernel boot command line option. If those CPUs are to be put + into a partition, they have to be used in an isolated partition. Device controller @@ -2365,7 +2592,7 @@ Miscellaneous controller provides 3 interface files. If two misc resources (res_ res_b 10 misc.current - A read-only flat-keyed file shown in the non-root cgroups. It shows + A read-only flat-keyed file shown in the all cgroups. It shows the current usage of the resources in the cgroup and its children.:: $ cat misc.current |