blob: 1b8b46deeb299abc24e6a60c09dd71105d219800 [file] [log] [blame]
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -03001================
Shailabh Nagara3baf642006-07-14 00:24:42 -07002Delay accounting
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -03003================
Shailabh Nagara3baf642006-07-14 00:24:42 -07004
5Tasks encounter delays in execution when they wait
6for some kernel resource to become available e.g. a
7runnable task may wait for a free CPU to run on.
8
9The per-task delay accounting functionality measures
10the delays experienced by a task while
11
12a) waiting for a CPU (while being runnable)
13b) completion of synchronous block I/O initiated by the task
14c) swapping in pages
Keika Kobayashi9b0975a2008-07-25 01:48:54 -070015d) memory reclaim
Shailabh Nagara3baf642006-07-14 00:24:42 -070016
17and makes these statistics available to userspace through
18the taskstats interface.
19
20Such delays provide feedback for setting a task's cpu priority,
21io priority and rss limit values appropriately. Long delays for
22important tasks could be a trigger for raising its corresponding priority.
23
24The functionality, through its use of the taskstats interface, also provides
25delay statistics aggregated for all tasks (or threads) belonging to a
26thread group (corresponding to a traditional Unix process). This is a commonly
27needed aggregation that is more efficiently done by the kernel.
28
29Userspace utilities, particularly resource management applications, can also
30aggregate delay statistics into arbitrary groups. To enable this, delay
31statistics of a task are available both during its lifetime as well as on its
32exit, ensuring continuous and complete monitoring can be done.
33
34
35Interface
36---------
37
38Delay accounting uses the taskstats interface which is described
39in detail in a separate document in this directory. Taskstats returns a
40generic data structure to userspace corresponding to per-pid and per-tgid
41statistics. The delay accounting functionality populates specific fields of
42this structure. See
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030043
Shailabh Nagara3baf642006-07-14 00:24:42 -070044 include/linux/taskstats.h
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030045
Shailabh Nagara3baf642006-07-14 00:24:42 -070046for a description of the fields pertaining to delay accounting.
47It will generally be in the form of counters returning the cumulative
Keika Kobayashi9b0975a2008-07-25 01:48:54 -070048delay seen for cpu, sync block I/O, swapin, memory reclaim etc.
Shailabh Nagara3baf642006-07-14 00:24:42 -070049
50Taking the difference of two successive readings of a given
51counter (say cpu_delay_total) for a task will give the delay
52experienced by the task waiting for the corresponding resource
53in that interval.
54
Shailabh Nagarad4ecbc2006-07-14 00:24:44 -070055When a task exits, records containing the per-task statistics
56are sent to userspace without requiring a command. If it is the last exiting
57task of a thread group, the per-tgid statistics are also sent. More details
58are given in the taskstats interface description.
Shailabh Nagara3baf642006-07-14 00:24:42 -070059
Shuah Khand522b2c2016-09-21 16:19:35 -060060The getdelays.c userspace utility in tools/accounting directory allows simple
61commands to be run and the corresponding delay statistics to be displayed. It
62also serves as an example of using the taskstats interface.
Shailabh Nagara3baf642006-07-14 00:24:42 -070063
64Usage
65-----
66
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030067Compile the kernel with::
68
Shailabh Nagara3baf642006-07-14 00:24:42 -070069 CONFIG_TASK_DELAY_ACCT=y
70 CONFIG_TASKSTATS=y
71
Peter Zijlstrae4042ad2021-05-04 22:43:32 +020072Delay accounting is disabled by default at boot up.
73To enable, add::
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030074
Peter Zijlstrae4042ad2021-05-04 22:43:32 +020075 delayacct
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030076
Peter Zijlstra0cd7c742021-05-10 14:01:00 +020077to the kernel boot options. The rest of the instructions below assume this has
78been done. Alternatively, use sysctl kernel.task_delayacct to switch the state
79at runtime. Note however that only tasks started after enabling it will have
80delayacct information.
Shailabh Nagara3baf642006-07-14 00:24:42 -070081
Shailabh Nagar163ecdf2006-07-30 03:03:11 -070082After the system has booted up, use a utility
Shailabh Nagara3baf642006-07-14 00:24:42 -070083similar to getdelays.c to access the delays
84seen by a given task or a task group (tgid).
85The utility also allows a given command to be
86executed and the corresponding delays to be
87seen.
88
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030089General format of the getdelays command::
Shailabh Nagara3baf642006-07-14 00:24:42 -070090
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030091 getdelays [-t tgid] [-p pid] [-c cmd...]
Shailabh Nagara3baf642006-07-14 00:24:42 -070092
93
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030094Get delays, since system boot, for pid 10::
Shailabh Nagara3baf642006-07-14 00:24:42 -070095
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -030096 # ./getdelays -p 10
97 (output similar to next case)
98
99Get sum of delays, since system boot, for all pids with tgid 5::
100
101 # ./getdelays -t 5
Shailabh Nagara3baf642006-07-14 00:24:42 -0700102
103
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -0300104 CPU count real total virtual total delay total
105 7876 92005750 100000000 24001500
106 IO count delay total
107 0 0
108 SWAP count delay total
109 0 0
110 RECLAIM count delay total
111 0 0
Shailabh Nagara3baf642006-07-14 00:24:42 -0700112
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -0300113Get delays seen in executing a given simple command::
Shailabh Nagara3baf642006-07-14 00:24:42 -0700114
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -0300115 # ./getdelays -c ls /
116
117 bin data1 data3 data5 dev home media opt root srv sys usr
118 boot data2 data4 data6 etc lib mnt proc sbin subdomain tmp var
Shailabh Nagara3baf642006-07-14 00:24:42 -0700119
120
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -0300121 CPU count real total virtual total delay total
Shailabh Nagara3baf642006-07-14 00:24:42 -0700122 6 4000250 4000000 0
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -0300123 IO count delay total
Shailabh Nagara3baf642006-07-14 00:24:42 -0700124 0 0
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -0300125 SWAP count delay total
Keika Kobayashi9b0975a2008-07-25 01:48:54 -0700126 0 0
Mauro Carvalho Chehabc3123552019-04-17 05:46:08 -0300127 RECLAIM count delay total
Shailabh Nagara3baf642006-07-14 00:24:42 -0700128 0 0