Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 1 | .. highlight:: none |
| 2 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 3 | Debugging kernel and modules via gdb |
| 4 | ==================================== |
| 5 | |
| 6 | The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware |
| 7 | interfaces allow to debug the Linux kernel and its modules during runtime |
| 8 | using gdb. Gdb comes with a powerful scripting interface for python. The |
| 9 | kernel provides a collection of helper scripts that can simplify typical |
| 10 | kernel debugging steps. This is a short tutorial about how to enable and use |
| 11 | them. It focuses on QEMU/KVM virtual machines as target, but the examples can |
| 12 | be transferred to the other gdb stubs as well. |
| 13 | |
| 14 | |
| 15 | Requirements |
| 16 | ------------ |
| 17 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 18 | - gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true |
| 19 | for distributions) |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 20 | |
| 21 | |
| 22 | Setup |
| 23 | ----- |
| 24 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 25 | - Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and |
| 26 | www.qemu.org for more details). For cross-development, |
Alexander A. Klimov | 93431e0 | 2020-05-26 08:05:44 +0200 | [diff] [blame] | 27 | https://landley.net/aboriginal/bin keeps a pool of machine images and |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 28 | toolchains that can be helpful to start from. |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 29 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 30 | - Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave |
| 31 | CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports |
| 32 | CONFIG_FRAME_POINTER, keep it enabled. |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 33 | |
Zhouyi Zhou | e604f1c | 2017-07-07 16:51:45 +0800 | [diff] [blame] | 34 | - Install that kernel on the guest, turn off KASLR if necessary by adding |
| 35 | "nokaslr" to the kernel command line. |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 36 | Alternatively, QEMU allows to boot the kernel directly using -kernel, |
| 37 | -append, -initrd command line switches. This is generally only useful if |
| 38 | you do not depend on modules. See QEMU documentation for more details on |
Zhouyi Zhou | e604f1c | 2017-07-07 16:51:45 +0800 | [diff] [blame] | 39 | this mode. In this case, you should build the kernel with |
| 40 | CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR. |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 41 | |
Jakob Koschel | 6b21943 | 2023-01-20 00:23:20 +0100 | [diff] [blame] | 42 | - Build the gdb scripts (required on kernels v5.1 and above):: |
| 43 | |
| 44 | make scripts_gdb |
| 45 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 46 | - Enable the gdb stub of QEMU/KVM, either |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 47 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 48 | - at VM startup time by appending "-s" to the QEMU command line |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 49 | |
| 50 | or |
| 51 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 52 | - during runtime by issuing "gdbserver" from the QEMU monitor |
| 53 | console |
| 54 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 55 | - cd /path/to/linux-build |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 56 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 57 | - Start gdb: gdb vmlinux |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 58 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 59 | Note: Some distros may restrict auto-loading of gdb scripts to known safe |
| 60 | directories. In case gdb reports to refuse loading vmlinux-gdb.py, add:: |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 61 | |
| 62 | add-auto-load-safe-path /path/to/linux-build |
| 63 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 64 | to ~/.gdbinit. See gdb help for more details. |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 65 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 66 | - Attach to the booted guest:: |
| 67 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 68 | (gdb) target remote :1234 |
| 69 | |
| 70 | |
| 71 | Examples of using the Linux-provided gdb helpers |
| 72 | ------------------------------------------------ |
| 73 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 74 | - Load module (and main kernel) symbols:: |
| 75 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 76 | (gdb) lx-symbols |
| 77 | loading vmlinux |
| 78 | scanning for modules in /home/user/linux/build |
| 79 | loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko |
| 80 | loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko |
| 81 | loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko |
| 82 | loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko |
| 83 | loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko |
| 84 | ... |
| 85 | loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko |
| 86 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 87 | - Set a breakpoint on some not yet loaded module function, e.g.:: |
| 88 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 89 | (gdb) b btrfs_init_sysfs |
| 90 | Function "btrfs_init_sysfs" not defined. |
| 91 | Make breakpoint pending on future shared library load? (y or [n]) y |
| 92 | Breakpoint 1 (btrfs_init_sysfs) pending. |
| 93 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 94 | - Continue the target:: |
| 95 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 96 | (gdb) c |
| 97 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 98 | - Load the module on the target and watch the symbols being loaded as well as |
| 99 | the breakpoint hit:: |
| 100 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 101 | loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko |
| 102 | loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko |
| 103 | loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko |
| 104 | loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko |
| 105 | |
| 106 | Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36 |
| 107 | 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj); |
| 108 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 109 | - Dump the log buffer of the target kernel:: |
| 110 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 111 | (gdb) lx-dmesg |
| 112 | [ 0.000000] Initializing cgroup subsys cpuset |
| 113 | [ 0.000000] Initializing cgroup subsys cpu |
| 114 | [ 0.000000] Linux version 3.8.0-rc4-dbg+ (... |
| 115 | [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314 |
| 116 | [ 0.000000] e820: BIOS-provided physical RAM map: |
| 117 | [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable |
| 118 | [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved |
| 119 | .... |
| 120 | |
Barry Song | 526940e | 2021-05-06 18:05:09 -0700 | [diff] [blame] | 121 | - Examine fields of the current task struct(supported by x86 and arm64 only):: |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 122 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 123 | (gdb) p $lx_current().pid |
| 124 | $1 = 4998 |
| 125 | (gdb) p $lx_current().comm |
| 126 | $2 = "modprobe\000\000\000\000\000\000\000" |
| 127 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 128 | - Make use of the per-cpu function for the current or a specified CPU:: |
| 129 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 130 | (gdb) p $lx_per_cpu("runqueues").nr_running |
| 131 | $3 = 1 |
| 132 | (gdb) p $lx_per_cpu("runqueues", 2).nr_running |
| 133 | $4 = 0 |
| 134 | |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 135 | - Dig into hrtimers using the container_of helper:: |
| 136 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 137 | (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next |
| 138 | (gdb) p *$container_of($next, "struct hrtimer", "node") |
| 139 | $5 = { |
| 140 | node = { |
| 141 | node = { |
| 142 | __rb_parent_color = 18446612133355256072, |
| 143 | rb_right = 0x0 <irq_stack_union>, |
| 144 | rb_left = 0x0 <irq_stack_union> |
| 145 | }, |
| 146 | expires = { |
| 147 | tv64 = 1835268000000 |
| 148 | } |
| 149 | }, |
| 150 | _softexpires = { |
| 151 | tv64 = 1835268000000 |
| 152 | }, |
| 153 | function = 0xffffffff81078232 <tick_sched_timer>, |
| 154 | base = 0xffff88003fd0d6f0, |
| 155 | state = 1, |
| 156 | start_pid = 0, |
| 157 | start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>, |
| 158 | start_comm = "swapper/2\000\000\000\000\000\000" |
| 159 | } |
| 160 | |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 161 | |
| 162 | List of commands and functions |
| 163 | ------------------------------ |
| 164 | |
| 165 | The number of commands and convenience functions may evolve over the time, |
Jonathan Corbet | 5f09627 | 2016-08-08 15:55:49 -0600 | [diff] [blame] | 166 | this is just a snapshot of the initial version:: |
Jan Kiszka | bda1a92 | 2015-02-17 13:47:32 -0800 | [diff] [blame] | 167 | |
| 168 | (gdb) apropos lx |
| 169 | function lx_current -- Return current task |
| 170 | function lx_module -- Find module by name and return the module variable |
| 171 | function lx_per_cpu -- Return per-cpu variable |
| 172 | function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable |
| 173 | function lx_thread_info -- Calculate Linux thread_info from task variable |
| 174 | lx-dmesg -- Print Linux kernel log buffer |
| 175 | lx-lsmod -- List currently loaded modules |
| 176 | lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules |
| 177 | |
| 178 | Detailed help can be obtained via "help <command-name>" for commands and "help |
| 179 | function <function-name>" for convenience functions. |