blob: 59c2a64efb21695f8492686c5d80fae1102fe9f4 [file] [log] [blame]
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -03001==============================
Steven Rostedta6537be2006-06-27 02:54:54 -07002RT-mutex implementation design
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -03003==============================
4
5Copyright (c) 2006 Steven Rostedt
6
7Licensed under the GNU Free Documentation License, Version 1.2
8
Steven Rostedta6537be2006-06-27 02:54:54 -07009
10This document tries to describe the design of the rtmutex.c implementation.
11It doesn't describe the reasons why rtmutex.c exists. For that please see
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -030012Documentation/locking/rt-mutex.rst. Although this document does explain problems
Steven Rostedta6537be2006-06-27 02:54:54 -070013that happen without this code, but that is in the concept to understand
14what the code actually is doing.
15
16The goal of this document is to help others understand the priority
17inheritance (PI) algorithm that is used, as well as reasons for the
18decisions that were made to implement PI in the manner that was done.
19
20
21Unbounded Priority Inversion
22----------------------------
23
24Priority inversion is when a lower priority process executes while a higher
25priority process wants to run. This happens for several reasons, and
26most of the time it can't be helped. Anytime a high priority process wants
27to use a resource that a lower priority process has (a mutex for example),
28the high priority process must wait until the lower priority process is done
29with the resource. This is a priority inversion. What we want to prevent
30is something called unbounded priority inversion. That is when the high
31priority process is prevented from running by a lower priority process for
32an undetermined amount of time.
33
Xishi Qiuc79a8d82013-11-06 13:18:21 -080034The classic example of unbounded priority inversion is where you have three
Steven Rostedta6537be2006-06-27 02:54:54 -070035processes, let's call them processes A, B, and C, where A is the highest
36priority process, C is the lowest, and B is in between. A tries to grab a lock
37that C owns and must wait and lets C run to release the lock. But in the
38meantime, B executes, and since B is of a higher priority than C, it preempts C,
39but by doing so, it is in fact preempting A which is a higher priority process.
40Now there's no way of knowing how long A will be sleeping waiting for C
41to release the lock, because for all we know, B is a CPU hog and will
42never give C a chance to release the lock. This is called unbounded priority
43inversion.
44
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -030045Here's a little ASCII art to show the problem::
Steven Rostedta6537be2006-06-27 02:54:54 -070046
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -030047 grab lock L1 (owned by C)
48 |
49 A ---+
50 C preempted by B
51 |
52 C +----+
Steven Rostedta6537be2006-06-27 02:54:54 -070053
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -030054 B +-------->
55 B now keeps A from running.
Steven Rostedta6537be2006-06-27 02:54:54 -070056
57
58Priority Inheritance (PI)
59-------------------------
60
61There are several ways to solve this issue, but other ways are out of scope
62for this document. Here we only discuss PI.
63
64PI is where a process inherits the priority of another process if the other
65process blocks on a lock owned by the current process. To make this easier
66to understand, let's use the previous example, with processes A, B, and C again.
67
68This time, when A blocks on the lock owned by C, C would inherit the priority
69of A. So now if B becomes runnable, it would not preempt C, since C now has
70the high priority of A. As soon as C releases the lock, it loses its
71inherited priority, and A then can continue with the resource that C had.
72
73Terminology
74-----------
75
76Here I explain some terminology that is used in this document to help describe
77the design that is used to implement PI.
78
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -030079PI chain
80 - The PI chain is an ordered series of locks and processes that cause
Steven Rostedta6537be2006-06-27 02:54:54 -070081 processes to inherit priorities from a previous process that is
82 blocked on one of its locks. This is described in more detail
83 later in this document.
84
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -030085mutex
86 - In this document, to differentiate from locks that implement
Steven Rostedta6537be2006-06-27 02:54:54 -070087 PI and spin locks that are used in the PI code, from now on
88 the PI locks will be called a mutex.
89
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -030090lock
91 - In this document from now on, I will use the term lock when
Steven Rostedta6537be2006-06-27 02:54:54 -070092 referring to spin locks that are used to protect parts of the PI
93 algorithm. These locks disable preemption for UP (when
94 CONFIG_PREEMPT is enabled) and on SMP prevents multiple CPUs from
95 entering critical sections simultaneously.
96
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -030097spin lock
98 - Same as lock above.
Steven Rostedta6537be2006-06-27 02:54:54 -070099
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300100waiter
101 - A waiter is a struct that is stored on the stack of a blocked
Steven Rostedta6537be2006-06-27 02:54:54 -0700102 process. Since the scope of the waiter is within the code for
103 a process being blocked on the mutex, it is fine to allocate
104 the waiter on the process's stack (local variable). This
105 structure holds a pointer to the task, as well as the mutex that
Alex Shif1824df2017-07-31 09:50:53 +0800106 the task is blocked on. It also has rbtree node structures to
107 place the task in the waiters rbtree of a mutex as well as the
108 pi_waiters rbtree of a mutex owner task (described below).
Steven Rostedta6537be2006-06-27 02:54:54 -0700109
110 waiter is sometimes used in reference to the task that is waiting
111 on a mutex. This is the same as waiter->task.
112
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300113waiters
114 - A list of processes that are blocked on a mutex.
Steven Rostedta6537be2006-06-27 02:54:54 -0700115
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300116top waiter
117 - The highest priority process waiting on a specific mutex.
Steven Rostedta6537be2006-06-27 02:54:54 -0700118
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300119top pi waiter
120 - The highest priority process waiting on one of the mutexes
Steven Rostedta6537be2006-06-27 02:54:54 -0700121 that a specific process owns.
122
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300123Note:
124 task and process are used interchangeably in this document, mostly to
Steven Rostedta6537be2006-06-27 02:54:54 -0700125 differentiate between two processes that are being described together.
126
127
128PI chain
129--------
130
131The PI chain is a list of processes and mutexes that may cause priority
132inheritance to take place. Multiple chains may converge, but a chain
133would never diverge, since a process can't be blocked on more than one
134mutex at a time.
135
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300136Example::
Steven Rostedta6537be2006-06-27 02:54:54 -0700137
138 Process: A, B, C, D, E
139 Mutexes: L1, L2, L3, L4
140
141 A owns: L1
142 B blocked on L1
143 B owns L2
144 C blocked on L2
145 C owns L3
146 D blocked on L3
147 D owns L4
148 E blocked on L4
149
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300150The chain would be::
Steven Rostedta6537be2006-06-27 02:54:54 -0700151
152 E->L4->D->L3->C->L2->B->L1->A
153
154To show where two chains merge, we could add another process F and
155another mutex L5 where B owns L5 and F is blocked on mutex L5.
156
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300157The chain for F would be::
Steven Rostedta6537be2006-06-27 02:54:54 -0700158
159 F->L5->B->L1->A
160
161Since a process may own more than one mutex, but never be blocked on more than
162one, the chains merge.
163
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300164Here we show both chains::
Steven Rostedta6537be2006-06-27 02:54:54 -0700165
166 E->L4->D->L3->C->L2-+
167 |
168 +->B->L1->A
169 |
170 F->L5-+
171
172For PI to work, the processes at the right end of these chains (or we may
173also call it the Top of the chain) must be equal to or higher in priority
174than the processes to the left or below in the chain.
175
176Also since a mutex may have more than one process blocked on it, we can
177have multiple chains merge at mutexes. If we add another process G that is
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300178blocked on mutex L2::
Steven Rostedta6537be2006-06-27 02:54:54 -0700179
180 G->L2->B->L1->A
181
182And once again, to show how this can grow I will show the merging chains
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300183again::
Steven Rostedta6537be2006-06-27 02:54:54 -0700184
185 E->L4->D->L3->C-+
186 +->L2-+
187 | |
188 G-+ +->B->L1->A
189 |
190 F->L5-+
191
Alex Shif1824df2017-07-31 09:50:53 +0800192If process G has the highest priority in the chain, then all the tasks up
193the chain (A and B in this example), must have their priorities increased
194to that of G.
Steven Rostedta6537be2006-06-27 02:54:54 -0700195
Alex Shif1824df2017-07-31 09:50:53 +0800196Mutex Waiters Tree
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300197------------------
Steven Rostedta6537be2006-06-27 02:54:54 -0700198
Alex Shif1824df2017-07-31 09:50:53 +0800199Every mutex keeps track of all the waiters that are blocked on itself. The
200mutex has a rbtree to store these waiters by priority. This tree is protected
201by a spin lock that is located in the struct of the mutex. This lock is called
202wait_lock.
Steven Rostedta6537be2006-06-27 02:54:54 -0700203
204
Alex Shif1824df2017-07-31 09:50:53 +0800205Task PI Tree
Steven Rostedta6537be2006-06-27 02:54:54 -0700206------------
207
Alex Shif1824df2017-07-31 09:50:53 +0800208To keep track of the PI chains, each process has its own PI rbtree. This is
209a tree of all top waiters of the mutexes that are owned by the process.
210Note that this tree only holds the top waiters and not all waiters that are
Steven Rostedta6537be2006-06-27 02:54:54 -0700211blocked on mutexes owned by the process.
212
Alex Shif1824df2017-07-31 09:50:53 +0800213The top of the task's PI tree is always the highest priority task that
Steven Rostedta6537be2006-06-27 02:54:54 -0700214is waiting on a mutex that is owned by the task. So if the task has
215inherited a priority, it will always be the priority of the task that is
Alex Shif1824df2017-07-31 09:50:53 +0800216at the top of this tree.
Steven Rostedta6537be2006-06-27 02:54:54 -0700217
Alex Shif1824df2017-07-31 09:50:53 +0800218This tree is stored in the task structure of a process as a rbtree called
219pi_waiters. It is protected by a spin lock also in the task structure,
Steven Rostedta6537be2006-06-27 02:54:54 -0700220called pi_lock. This lock may also be taken in interrupt context, so when
221locking the pi_lock, interrupts must be disabled.
222
223
224Depth of the PI Chain
225---------------------
226
227The maximum depth of the PI chain is not dynamic, and could actually be
228defined. But is very complex to figure it out, since it depends on all
229the nesting of mutexes. Let's look at the example where we have 3 mutexes,
230L1, L2, and L3, and four separate functions func1, func2, func3 and func4.
231The following shows a locking order of L1->L2->L3, but may not actually
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300232be directly nested that way::
Steven Rostedta6537be2006-06-27 02:54:54 -0700233
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300234 void func1(void)
235 {
Steven Rostedta6537be2006-06-27 02:54:54 -0700236 mutex_lock(L1);
237
238 /* do anything */
239
240 mutex_unlock(L1);
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300241 }
Steven Rostedta6537be2006-06-27 02:54:54 -0700242
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300243 void func2(void)
244 {
Steven Rostedta6537be2006-06-27 02:54:54 -0700245 mutex_lock(L1);
246 mutex_lock(L2);
247
248 /* do something */
249
250 mutex_unlock(L2);
251 mutex_unlock(L1);
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300252 }
Steven Rostedta6537be2006-06-27 02:54:54 -0700253
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300254 void func3(void)
255 {
Steven Rostedta6537be2006-06-27 02:54:54 -0700256 mutex_lock(L2);
257 mutex_lock(L3);
258
259 /* do something else */
260
261 mutex_unlock(L3);
262 mutex_unlock(L2);
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300263 }
Steven Rostedta6537be2006-06-27 02:54:54 -0700264
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300265 void func4(void)
266 {
Steven Rostedta6537be2006-06-27 02:54:54 -0700267 mutex_lock(L3);
268
269 /* do something again */
270
271 mutex_unlock(L3);
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300272 }
Steven Rostedta6537be2006-06-27 02:54:54 -0700273
274Now we add 4 processes that run each of these functions separately.
275Processes A, B, C, and D which run functions func1, func2, func3 and func4
276respectively, and such that D runs first and A last. With D being preempted
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300277in func4 in the "do something again" area, we have a locking that follows::
Steven Rostedta6537be2006-06-27 02:54:54 -0700278
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300279 D owns L3
280 C blocked on L3
281 C owns L2
282 B blocked on L2
283 B owns L1
284 A blocked on L1
Steven Rostedta6537be2006-06-27 02:54:54 -0700285
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300286 And thus we have the chain A->L1->B->L2->C->L3->D.
Steven Rostedta6537be2006-06-27 02:54:54 -0700287
288This gives us a PI depth of 4 (four processes), but looking at any of the
289functions individually, it seems as though they only have at most a locking
290depth of two. So, although the locking depth is defined at compile time,
291it still is very difficult to find the possibilities of that depth.
292
293Now since mutexes can be defined by user-land applications, we don't want a DOS
294type of application that nests large amounts of mutexes to create a large
295PI chain, and have the code holding spin locks while looking at a large
296amount of data. So to prevent this, the implementation not only implements
297a maximum lock depth, but also only holds at most two different locks at a
298time, as it walks the PI chain. More about this below.
299
300
301Mutex owner and flags
302---------------------
303
304The mutex structure contains a pointer to the owner of the mutex. If the
305mutex is not owned, this owner is set to NULL. Since all architectures
Alex Shif1824df2017-07-31 09:50:53 +0800306have the task structure on at least a two byte alignment (and if this is
307not true, the rtmutex.c code will be broken!), this allows for the least
308significant bit to be used as a flag. Bit 0 is used as the "Has Waiters"
309flag. It's set whenever there are waiters on a mutex.
Steven Rostedta6537be2006-06-27 02:54:54 -0700310
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300311See Documentation/locking/rt-mutex.rst for further details.
Steven Rostedta6537be2006-06-27 02:54:54 -0700312
313cmpxchg Tricks
314--------------
315
316Some architectures implement an atomic cmpxchg (Compare and Exchange). This
317is used (when applicable) to keep the fast path of grabbing and releasing
318mutexes short.
319
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300320cmpxchg is basically the following function performed atomically::
Steven Rostedta6537be2006-06-27 02:54:54 -0700321
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300322 unsigned long _cmpxchg(unsigned long *A, unsigned long *B, unsigned long *C)
323 {
Jan Altenberg9ba0bdf2006-09-30 23:28:08 -0700324 unsigned long T = *A;
325 if (*A == *B) {
326 *A = *C;
327 }
328 return T;
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300329 }
330 #define cmpxchg(a,b,c) _cmpxchg(&a,&b,&c)
Steven Rostedta6537be2006-06-27 02:54:54 -0700331
332This is really nice to have, since it allows you to only update a variable
333if the variable is what you expect it to be. You know if it succeeded if
334the return value (the old value of A) is equal to B.
335
336The macro rt_mutex_cmpxchg is used to try to lock and unlock mutexes. If
337the architecture does not support CMPXCHG, then this macro is simply set
338to fail every time. But if CMPXCHG is supported, then this will
339help out extremely to keep the fast path short.
340
341The use of rt_mutex_cmpxchg with the flags in the owner field help optimize
342the system for architectures that support it. This will also be explained
343later in this document.
344
345
346Priority adjustments
347--------------------
348
349The implementation of the PI code in rtmutex.c has several places that a
Alex Shif1824df2017-07-31 09:50:53 +0800350process must adjust its priority. With the help of the pi_waiters of a
Steven Rostedta6537be2006-06-27 02:54:54 -0700351process this is rather easy to know what needs to be adjusted.
352
Alex Shif1824df2017-07-31 09:50:53 +0800353The functions implementing the task adjustments are rt_mutex_adjust_prio
354and rt_mutex_setprio. rt_mutex_setprio is only used in rt_mutex_adjust_prio.
Steven Rostedta6537be2006-06-27 02:54:54 -0700355
Alex Shif1824df2017-07-31 09:50:53 +0800356rt_mutex_adjust_prio examines the priority of the task, and the highest
357priority process that is waiting any of mutexes owned by the task. Since
358the pi_waiters of a task holds an order by priority of all the top waiters
359of all the mutexes that the task owns, we simply need to compare the top
360pi waiter to its own normal/deadline priority and take the higher one.
361Then rt_mutex_setprio is called to adjust the priority of the task to the
362new priority. Note that rt_mutex_setprio is defined in kernel/sched/core.c
363to implement the actual change in priority.
Steven Rostedta6537be2006-06-27 02:54:54 -0700364
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300365Note:
366 For the "prio" field in task_struct, the lower the number, the
Alex Shif1824df2017-07-31 09:50:53 +0800367 higher the priority. A "prio" of 5 is of higher priority than a
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300368 "prio" of 10.
Steven Rostedta6537be2006-06-27 02:54:54 -0700369
Alex Shif1824df2017-07-31 09:50:53 +0800370It is interesting to note that rt_mutex_adjust_prio can either increase
Steven Rostedta6537be2006-06-27 02:54:54 -0700371or decrease the priority of the task. In the case that a higher priority
Alex Shif1824df2017-07-31 09:50:53 +0800372process has just blocked on a mutex owned by the task, rt_mutex_adjust_prio
Steven Rostedta6537be2006-06-27 02:54:54 -0700373would increase/boost the task's priority. But if a higher priority task
374were for some reason to leave the mutex (timeout or signal), this same function
Alex Shif1824df2017-07-31 09:50:53 +0800375would decrease/unboost the priority of the task. That is because the pi_waiters
Steven Rostedta6537be2006-06-27 02:54:54 -0700376always contains the highest priority task that is waiting on a mutex owned
377by the task, so we only need to compare the priority of that top pi waiter
378to the normal priority of the given task.
379
380
381High level overview of the PI chain walk
382----------------------------------------
383
384The PI chain walk is implemented by the function rt_mutex_adjust_prio_chain.
385
386The implementation has gone through several iterations, and has ended up
387with what we believe is the best. It walks the PI chain by only grabbing
388at most two locks at a time, and is very efficient.
389
390The rt_mutex_adjust_prio_chain can be used either to boost or lower process
391priorities.
392
393rt_mutex_adjust_prio_chain is called with a task to be checked for PI
394(de)boosting (the owner of a mutex that a process is blocking on), a flag to
Alex Shif1824df2017-07-31 09:50:53 +0800395check for deadlocking, the mutex that the task owns, a pointer to a waiter
Steven Rostedta6537be2006-06-27 02:54:54 -0700396that is the process's waiter struct that is blocked on the mutex (although this
Alex Shif1824df2017-07-31 09:50:53 +0800397parameter may be NULL for deboosting), a pointer to the mutex on which the task
398is blocked, and a top_task as the top waiter of the mutex.
Steven Rostedta6537be2006-06-27 02:54:54 -0700399
400For this explanation, I will not mention deadlock detection. This explanation
401will try to stay at a high level.
402
403When this function is called, there are no locks held. That also means
404that the state of the owner and lock can change when entered into this function.
405
406Before this function is called, the task has already had rt_mutex_adjust_prio
407performed on it. This means that the task is set to the priority that it
Alex Shif1824df2017-07-31 09:50:53 +0800408should be at, but the rbtree nodes of the task's waiter have not been updated
409with the new priorities, and this task may not be in the proper locations
410in the pi_waiters and waiters trees that the task is blocked on. This function
Steven Rostedta6537be2006-06-27 02:54:54 -0700411solves all that.
412
Alex Shif1824df2017-07-31 09:50:53 +0800413The main operation of this function is summarized by Thomas Gleixner in
414rtmutex.c. See the 'Chain walk basics and protection scope' comment for further
415details.
Steven Rostedta6537be2006-06-27 02:54:54 -0700416
417Taking of a mutex (The walk through)
418------------------------------------
419
420OK, now let's take a look at the detailed walk through of what happens when
421taking a mutex.
422
423The first thing that is tried is the fast taking of the mutex. This is
424done when we have CMPXCHG enabled (otherwise the fast taking automatically
425fails). Only when the owner field of the mutex is NULL can the lock be
426taken with the CMPXCHG and nothing else needs to be done.
427
Alex Shif1824df2017-07-31 09:50:53 +0800428If there is contention on the lock, we go about the slow path
429(rt_mutex_slowlock).
Steven Rostedta6537be2006-06-27 02:54:54 -0700430
431The slow path function is where the task's waiter structure is created on
432the stack. This is because the waiter structure is only needed for the
433scope of this function. The waiter structure holds the nodes to store
Alex Shif1824df2017-07-31 09:50:53 +0800434the task on the waiters tree of the mutex, and if need be, the pi_waiters
435tree of the owner.
Steven Rostedta6537be2006-06-27 02:54:54 -0700436
437The wait_lock of the mutex is taken since the slow path of unlocking the
438mutex also takes this lock.
439
440We then call try_to_take_rt_mutex. This is where the architecture that
441does not implement CMPXCHG would always grab the lock (if there's no
442contention).
443
444try_to_take_rt_mutex is used every time the task tries to grab a mutex in the
445slow path. The first thing that is done here is an atomic setting of
Alex Shif1824df2017-07-31 09:50:53 +0800446the "Has Waiters" flag of the mutex's owner field. By setting this flag
447now, the current owner of the mutex being contended for can't release the mutex
448without going into the slow unlock path, and it would then need to grab the
449wait_lock, which this code currently holds. So setting the "Has Waiters" flag
450forces the current owner to synchronize with this code.
Steven Rostedta6537be2006-06-27 02:54:54 -0700451
Alex Shif1824df2017-07-31 09:50:53 +0800452The lock is taken if the following are true:
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300453
Alex Shif1824df2017-07-31 09:50:53 +0800454 1) The lock has no owner
455 2) The current task is the highest priority against all other
456 waiters of the lock
Steven Rostedta6537be2006-06-27 02:54:54 -0700457
Alex Shif1824df2017-07-31 09:50:53 +0800458If the task succeeds to acquire the lock, then the task is set as the
459owner of the lock, and if the lock still has waiters, the top_waiter
460(highest priority task waiting on the lock) is added to this task's
461pi_waiters tree.
Steven Rostedta6537be2006-06-27 02:54:54 -0700462
Alex Shif1824df2017-07-31 09:50:53 +0800463If the lock is not taken by try_to_take_rt_mutex(), then the
464task_blocks_on_rt_mutex() function is called. This will add the task to
465the lock's waiter tree and propagate the pi chain of the lock as well
466as the lock's owner's pi_waiters tree. This is described in the next
467section.
Steven Rostedta6537be2006-06-27 02:54:54 -0700468
469Task blocks on mutex
470--------------------
471
472The accounting of a mutex and process is done with the waiter structure of
473the process. The "task" field is set to the process, and the "lock" field
Alex Shif1824df2017-07-31 09:50:53 +0800474to the mutex. The rbtree node of waiter are initialized to the processes
475current priority.
Steven Rostedta6537be2006-06-27 02:54:54 -0700476
477Since the wait_lock was taken at the entry of the slow lock, we can safely
Alex Shif1824df2017-07-31 09:50:53 +0800478add the waiter to the task waiter tree. If the current process is the
479highest priority process currently waiting on this mutex, then we remove the
480previous top waiter process (if it exists) from the pi_waiters of the owner,
481and add the current process to that tree. Since the pi_waiter of the owner
Steven Rostedta6537be2006-06-27 02:54:54 -0700482has changed, we call rt_mutex_adjust_prio on the owner to see if the owner
483should adjust its priority accordingly.
484
Alex Shif1824df2017-07-31 09:50:53 +0800485If the owner is also blocked on a lock, and had its pi_waiters changed
Steven Rostedta6537be2006-06-27 02:54:54 -0700486(or deadlock checking is on), we unlock the wait_lock of the mutex and go ahead
487and run rt_mutex_adjust_prio_chain on the owner, as described earlier.
488
489Now all locks are released, and if the current process is still blocked on a
490mutex (waiter "task" field is not NULL), then we go to sleep (call schedule).
491
492Waking up in the loop
493---------------------
494
Alex Shif1824df2017-07-31 09:50:53 +0800495The task can then wake up for a couple of reasons:
496 1) The previous lock owner released the lock, and the task now is top_waiter
497 2) we received a signal or timeout
Steven Rostedta6537be2006-06-27 02:54:54 -0700498
Alex Shif1824df2017-07-31 09:50:53 +0800499In both cases, the task will try again to acquire the lock. If it
500does, then it will take itself off the waiters tree and set itself back
501to the TASK_RUNNING state.
Steven Rostedta6537be2006-06-27 02:54:54 -0700502
Alex Shif1824df2017-07-31 09:50:53 +0800503In first case, if the lock was acquired by another task before this task
504could get the lock, then it will go back to sleep and wait to be woken again.
Steven Rostedta6537be2006-06-27 02:54:54 -0700505
Alex Shif1824df2017-07-31 09:50:53 +0800506The second case is only applicable for tasks that are grabbing a mutex
507that can wake up before getting the lock, either due to a signal or
508a timeout (i.e. rt_mutex_timed_futex_lock()). When woken, it will try to
509take the lock again, if it succeeds, then the task will return with the
510lock held, otherwise it will return with -EINTR if the task was woken
511by a signal, or -ETIMEDOUT if it timed out.
Steven Rostedta6537be2006-06-27 02:54:54 -0700512
513
514Unlocking the Mutex
515-------------------
516
517The unlocking of a mutex also has a fast path for those architectures with
518CMPXCHG. Since the taking of a mutex on contention always sets the
519"Has Waiters" flag of the mutex's owner, we use this to know if we need to
520take the slow path when unlocking the mutex. If the mutex doesn't have any
521waiters, the owner field of the mutex would equal the current process and
522the mutex can be unlocked by just replacing the owner field with NULL.
523
524If the owner field has the "Has Waiters" bit set (or CMPXCHG is not available),
525the slow unlock path is taken.
526
527The first thing done in the slow unlock path is to take the wait_lock of the
528mutex. This synchronizes the locking and unlocking of the mutex.
529
530A check is made to see if the mutex has waiters or not. On architectures that
531do not have CMPXCHG, this is the location that the owner of the mutex will
532determine if a waiter needs to be awoken or not. On architectures that
533do have CMPXCHG, that check is done in the fast path, but it is still needed
534in the slow path too. If a waiter of a mutex woke up because of a signal
535or timeout between the time the owner failed the fast path CMPXCHG check and
536the grabbing of the wait_lock, the mutex may not have any waiters, thus the
Jan Altenberg9ba0bdf2006-09-30 23:28:08 -0700537owner still needs to make this check. If there are no waiters then the mutex
Steven Rostedta6537be2006-06-27 02:54:54 -0700538owner field is set to NULL, the wait_lock is released and nothing more is
539needed.
540
Alex Shif1824df2017-07-31 09:50:53 +0800541If there are waiters, then we need to wake one up.
Steven Rostedta6537be2006-06-27 02:54:54 -0700542
543On the wake up code, the pi_lock of the current owner is taken. The top
Alex Shif1824df2017-07-31 09:50:53 +0800544waiter of the lock is found and removed from the waiters tree of the mutex
545as well as the pi_waiters tree of the current owner. The "Has Waiters" bit is
546marked to prevent lower priority tasks from stealing the lock.
Steven Rostedta6537be2006-06-27 02:54:54 -0700547
548Finally we unlock the pi_lock of the pending owner and wake it up.
549
550
551Contact
552-------
553
554For updates on this document, please email Steven Rostedt <rostedt@goodmis.org>
555
556
557Credits
558-------
559
560Author: Steven Rostedt <rostedt@goodmis.org>
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300561
Alex Shif1824df2017-07-31 09:50:53 +0800562Updated: Alex Shi <alex.shi@linaro.org> - 7/6/2017
Steven Rostedta6537be2006-06-27 02:54:54 -0700563
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300564Original Reviewers:
565 Ingo Molnar, Thomas Gleixner, Thomas Duetsch, and
Alex Shif1824df2017-07-31 09:50:53 +0800566 Randy Dunlap
Mauro Carvalho Chehab387b1462019-04-10 08:32:41 -0300567
Alex Shif1824df2017-07-31 09:50:53 +0800568Update (7/6/2017) Reviewers: Steven Rostedt and Sebastian Siewior
Steven Rostedta6537be2006-06-27 02:54:54 -0700569
570Updates
571-------
572
573This document was originally written for 2.6.17-rc3-mm1
Alex Shif1824df2017-07-31 09:50:53 +0800574was updated on 4.12