blob: 895285c037c727554465173e45f9fc465c0ccc3d [file] [log] [blame]
Jonathan Corbet5f096272016-08-08 15:55:49 -06001.. highlight:: none
2
Jan Kiszkabda1a922015-02-17 13:47:32 -08003Debugging kernel and modules via gdb
4====================================
5
6The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
7interfaces allow to debug the Linux kernel and its modules during runtime
8using gdb. Gdb comes with a powerful scripting interface for python. The
9kernel provides a collection of helper scripts that can simplify typical
10kernel debugging steps. This is a short tutorial about how to enable and use
11them. It focuses on QEMU/KVM virtual machines as target, but the examples can
12be transferred to the other gdb stubs as well.
13
14
15Requirements
16------------
17
Jonathan Corbet5f096272016-08-08 15:55:49 -060018- gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
19 for distributions)
Jan Kiszkabda1a922015-02-17 13:47:32 -080020
21
22Setup
23-----
24
Jonathan Corbet5f096272016-08-08 15:55:49 -060025- Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
26 www.qemu.org for more details). For cross-development,
Alexander A. Klimov93431e02020-05-26 08:05:44 +020027 https://landley.net/aboriginal/bin keeps a pool of machine images and
Jonathan Corbet5f096272016-08-08 15:55:49 -060028 toolchains that can be helpful to start from.
Jan Kiszkabda1a922015-02-17 13:47:32 -080029
Jonathan Corbet5f096272016-08-08 15:55:49 -060030- Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
31 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
32 CONFIG_FRAME_POINTER, keep it enabled.
Jan Kiszkabda1a922015-02-17 13:47:32 -080033
Zhouyi Zhoue604f1c2017-07-07 16:51:45 +080034- Install that kernel on the guest, turn off KASLR if necessary by adding
35 "nokaslr" to the kernel command line.
Jonathan Corbet5f096272016-08-08 15:55:49 -060036 Alternatively, QEMU allows to boot the kernel directly using -kernel,
37 -append, -initrd command line switches. This is generally only useful if
38 you do not depend on modules. See QEMU documentation for more details on
Zhouyi Zhoue604f1c2017-07-07 16:51:45 +080039 this mode. In this case, you should build the kernel with
40 CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.
Jan Kiszkabda1a922015-02-17 13:47:32 -080041
Jakob Koschel6b219432023-01-20 00:23:20 +010042- Build the gdb scripts (required on kernels v5.1 and above)::
43
44 make scripts_gdb
45
Jonathan Corbet5f096272016-08-08 15:55:49 -060046- Enable the gdb stub of QEMU/KVM, either
Jan Kiszkabda1a922015-02-17 13:47:32 -080047
Jan Kiszkabda1a922015-02-17 13:47:32 -080048 - at VM startup time by appending "-s" to the QEMU command line
Jonathan Corbet5f096272016-08-08 15:55:49 -060049
50 or
51
Jan Kiszkabda1a922015-02-17 13:47:32 -080052 - during runtime by issuing "gdbserver" from the QEMU monitor
53 console
54
Jonathan Corbet5f096272016-08-08 15:55:49 -060055- cd /path/to/linux-build
Jan Kiszkabda1a922015-02-17 13:47:32 -080056
Jonathan Corbet5f096272016-08-08 15:55:49 -060057- Start gdb: gdb vmlinux
Jan Kiszkabda1a922015-02-17 13:47:32 -080058
Jonathan Corbet5f096272016-08-08 15:55:49 -060059 Note: Some distros may restrict auto-loading of gdb scripts to known safe
60 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
Jan Kiszkabda1a922015-02-17 13:47:32 -080061
62 add-auto-load-safe-path /path/to/linux-build
63
Jonathan Corbet5f096272016-08-08 15:55:49 -060064 to ~/.gdbinit. See gdb help for more details.
Jan Kiszkabda1a922015-02-17 13:47:32 -080065
Jonathan Corbet5f096272016-08-08 15:55:49 -060066- Attach to the booted guest::
67
Jan Kiszkabda1a922015-02-17 13:47:32 -080068 (gdb) target remote :1234
69
70
71Examples of using the Linux-provided gdb helpers
72------------------------------------------------
73
Jonathan Corbet5f096272016-08-08 15:55:49 -060074- Load module (and main kernel) symbols::
75
Jan Kiszkabda1a922015-02-17 13:47:32 -080076 (gdb) lx-symbols
77 loading vmlinux
78 scanning for modules in /home/user/linux/build
79 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
80 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
81 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
82 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
83 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
84 ...
85 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
86
Jonathan Corbet5f096272016-08-08 15:55:49 -060087- Set a breakpoint on some not yet loaded module function, e.g.::
88
Jan Kiszkabda1a922015-02-17 13:47:32 -080089 (gdb) b btrfs_init_sysfs
90 Function "btrfs_init_sysfs" not defined.
91 Make breakpoint pending on future shared library load? (y or [n]) y
92 Breakpoint 1 (btrfs_init_sysfs) pending.
93
Jonathan Corbet5f096272016-08-08 15:55:49 -060094- Continue the target::
95
Jan Kiszkabda1a922015-02-17 13:47:32 -080096 (gdb) c
97
Jonathan Corbet5f096272016-08-08 15:55:49 -060098- Load the module on the target and watch the symbols being loaded as well as
99 the breakpoint hit::
100
Jan Kiszkabda1a922015-02-17 13:47:32 -0800101 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
102 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
103 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
104 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
105
106 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
107 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
108
Jonathan Corbet5f096272016-08-08 15:55:49 -0600109- Dump the log buffer of the target kernel::
110
Jan Kiszkabda1a922015-02-17 13:47:32 -0800111 (gdb) lx-dmesg
112 [ 0.000000] Initializing cgroup subsys cpuset
113 [ 0.000000] Initializing cgroup subsys cpu
114 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
115 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
116 [ 0.000000] e820: BIOS-provided physical RAM map:
117 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
118 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
119 ....
120
Barry Song526940e2021-05-06 18:05:09 -0700121- Examine fields of the current task struct(supported by x86 and arm64 only)::
Jonathan Corbet5f096272016-08-08 15:55:49 -0600122
Jan Kiszkabda1a922015-02-17 13:47:32 -0800123 (gdb) p $lx_current().pid
124 $1 = 4998
125 (gdb) p $lx_current().comm
126 $2 = "modprobe\000\000\000\000\000\000\000"
127
Jonathan Corbet5f096272016-08-08 15:55:49 -0600128- Make use of the per-cpu function for the current or a specified CPU::
129
Jan Kiszkabda1a922015-02-17 13:47:32 -0800130 (gdb) p $lx_per_cpu("runqueues").nr_running
131 $3 = 1
132 (gdb) p $lx_per_cpu("runqueues", 2).nr_running
133 $4 = 0
134
Jonathan Corbet5f096272016-08-08 15:55:49 -0600135- Dig into hrtimers using the container_of helper::
136
Jan Kiszkabda1a922015-02-17 13:47:32 -0800137 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
138 (gdb) p *$container_of($next, "struct hrtimer", "node")
139 $5 = {
140 node = {
141 node = {
142 __rb_parent_color = 18446612133355256072,
143 rb_right = 0x0 <irq_stack_union>,
144 rb_left = 0x0 <irq_stack_union>
145 },
146 expires = {
147 tv64 = 1835268000000
148 }
149 },
150 _softexpires = {
151 tv64 = 1835268000000
152 },
153 function = 0xffffffff81078232 <tick_sched_timer>,
154 base = 0xffff88003fd0d6f0,
155 state = 1,
156 start_pid = 0,
157 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
158 start_comm = "swapper/2\000\000\000\000\000\000"
159 }
160
Jan Kiszkabda1a922015-02-17 13:47:32 -0800161
162List of commands and functions
163------------------------------
164
165The number of commands and convenience functions may evolve over the time,
Jonathan Corbet5f096272016-08-08 15:55:49 -0600166this is just a snapshot of the initial version::
Jan Kiszkabda1a922015-02-17 13:47:32 -0800167
168 (gdb) apropos lx
169 function lx_current -- Return current task
170 function lx_module -- Find module by name and return the module variable
171 function lx_per_cpu -- Return per-cpu variable
172 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
173 function lx_thread_info -- Calculate Linux thread_info from task variable
174 lx-dmesg -- Print Linux kernel log buffer
175 lx-lsmod -- List currently loaded modules
176 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
177
178Detailed help can be obtained via "help <command-name>" for commands and "help
179function <function-name>" for convenience functions.