Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 1 | ================ |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 2 | Delay accounting |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 3 | ================ |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 4 | |
| 5 | Tasks encounter delays in execution when they wait |
| 6 | for some kernel resource to become available e.g. a |
| 7 | runnable task may wait for a free CPU to run on. |
| 8 | |
| 9 | The per-task delay accounting functionality measures |
| 10 | the delays experienced by a task while |
| 11 | |
| 12 | a) waiting for a CPU (while being runnable) |
| 13 | b) completion of synchronous block I/O initiated by the task |
| 14 | c) swapping in pages |
Keika Kobayashi | 9b0975a | 2008-07-25 01:48:54 -0700 | [diff] [blame] | 15 | d) memory reclaim |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 16 | |
| 17 | and makes these statistics available to userspace through |
| 18 | the taskstats interface. |
| 19 | |
| 20 | Such delays provide feedback for setting a task's cpu priority, |
| 21 | io priority and rss limit values appropriately. Long delays for |
| 22 | important tasks could be a trigger for raising its corresponding priority. |
| 23 | |
| 24 | The functionality, through its use of the taskstats interface, also provides |
| 25 | delay statistics aggregated for all tasks (or threads) belonging to a |
| 26 | thread group (corresponding to a traditional Unix process). This is a commonly |
| 27 | needed aggregation that is more efficiently done by the kernel. |
| 28 | |
| 29 | Userspace utilities, particularly resource management applications, can also |
| 30 | aggregate delay statistics into arbitrary groups. To enable this, delay |
| 31 | statistics of a task are available both during its lifetime as well as on its |
| 32 | exit, ensuring continuous and complete monitoring can be done. |
| 33 | |
| 34 | |
| 35 | Interface |
| 36 | --------- |
| 37 | |
| 38 | Delay accounting uses the taskstats interface which is described |
| 39 | in detail in a separate document in this directory. Taskstats returns a |
| 40 | generic data structure to userspace corresponding to per-pid and per-tgid |
| 41 | statistics. The delay accounting functionality populates specific fields of |
| 42 | this structure. See |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 43 | |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 44 | include/linux/taskstats.h |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 45 | |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 46 | for a description of the fields pertaining to delay accounting. |
| 47 | It will generally be in the form of counters returning the cumulative |
Keika Kobayashi | 9b0975a | 2008-07-25 01:48:54 -0700 | [diff] [blame] | 48 | delay seen for cpu, sync block I/O, swapin, memory reclaim etc. |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 49 | |
| 50 | Taking the difference of two successive readings of a given |
| 51 | counter (say cpu_delay_total) for a task will give the delay |
| 52 | experienced by the task waiting for the corresponding resource |
| 53 | in that interval. |
| 54 | |
Shailabh Nagar | ad4ecbc | 2006-07-14 00:24:44 -0700 | [diff] [blame] | 55 | When a task exits, records containing the per-task statistics |
| 56 | are sent to userspace without requiring a command. If it is the last exiting |
| 57 | task of a thread group, the per-tgid statistics are also sent. More details |
| 58 | are given in the taskstats interface description. |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 59 | |
Shuah Khan | d522b2c | 2016-09-21 16:19:35 -0600 | [diff] [blame] | 60 | The getdelays.c userspace utility in tools/accounting directory allows simple |
| 61 | commands to be run and the corresponding delay statistics to be displayed. It |
| 62 | also serves as an example of using the taskstats interface. |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 63 | |
| 64 | Usage |
| 65 | ----- |
| 66 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 67 | Compile the kernel with:: |
| 68 | |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 69 | CONFIG_TASK_DELAY_ACCT=y |
| 70 | CONFIG_TASKSTATS=y |
| 71 | |
Peter Zijlstra | e4042ad | 2021-05-04 22:43:32 +0200 | [diff] [blame] | 72 | Delay accounting is disabled by default at boot up. |
| 73 | To enable, add:: |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 74 | |
Peter Zijlstra | e4042ad | 2021-05-04 22:43:32 +0200 | [diff] [blame] | 75 | delayacct |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 76 | |
Peter Zijlstra | 0cd7c74 | 2021-05-10 14:01:00 +0200 | [diff] [blame] | 77 | to the kernel boot options. The rest of the instructions below assume this has |
| 78 | been done. Alternatively, use sysctl kernel.task_delayacct to switch the state |
| 79 | at runtime. Note however that only tasks started after enabling it will have |
| 80 | delayacct information. |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 81 | |
Shailabh Nagar | 163ecdf | 2006-07-30 03:03:11 -0700 | [diff] [blame] | 82 | After the system has booted up, use a utility |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 83 | similar to getdelays.c to access the delays |
| 84 | seen by a given task or a task group (tgid). |
| 85 | The utility also allows a given command to be |
| 86 | executed and the corresponding delays to be |
| 87 | seen. |
| 88 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 89 | General format of the getdelays command:: |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 90 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 91 | getdelays [-t tgid] [-p pid] [-c cmd...] |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 92 | |
| 93 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 94 | Get delays, since system boot, for pid 10:: |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 95 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 96 | # ./getdelays -p 10 |
| 97 | (output similar to next case) |
| 98 | |
| 99 | Get sum of delays, since system boot, for all pids with tgid 5:: |
| 100 | |
| 101 | # ./getdelays -t 5 |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 102 | |
| 103 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 104 | CPU count real total virtual total delay total |
| 105 | 7876 92005750 100000000 24001500 |
| 106 | IO count delay total |
| 107 | 0 0 |
| 108 | SWAP count delay total |
| 109 | 0 0 |
| 110 | RECLAIM count delay total |
| 111 | 0 0 |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 112 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 113 | Get delays seen in executing a given simple command:: |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 114 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 115 | # ./getdelays -c ls / |
| 116 | |
| 117 | bin data1 data3 data5 dev home media opt root srv sys usr |
| 118 | boot data2 data4 data6 etc lib mnt proc sbin subdomain tmp var |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 119 | |
| 120 | |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 121 | CPU count real total virtual total delay total |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 122 | 6 4000250 4000000 0 |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 123 | IO count delay total |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 124 | 0 0 |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 125 | SWAP count delay total |
Keika Kobayashi | 9b0975a | 2008-07-25 01:48:54 -0700 | [diff] [blame] | 126 | 0 0 |
Mauro Carvalho Chehab | c312355 | 2019-04-17 05:46:08 -0300 | [diff] [blame] | 127 | RECLAIM count delay total |
Shailabh Nagar | a3baf64 | 2006-07-14 00:24:42 -0700 | [diff] [blame] | 128 | 0 0 |