Introduction

Recently I’ve been working on some kernel job, and debugging the kernel makes me exhausted. I referenced some blogs and tried to make it easier to debug the kernel. Now I can debug the kernel like a userland program with the gui interface of vscode. I’d include the blogs or other materials I referenced as much as I could. If you know the original entry, you may contact me.

Prerequisites

I assume you know how to use gdb to debug programs, and the tools we need include:

kvm-virtual machine click here to see how to install kvm virtual machine on ubuntu
gdb
vscode (optional)

Debugging

Some tutorials may use the debug extension of ubuntu, but it’s not necessary if you compile the kernel yourself. The key is to disable the aslr of the kernel and enable the debugging of qemu.

compiling your kernel

Notion 1: There are lots of tutorials about how to compile the kernel and install it. Just don’t forget to enable the debug info when you are setting .config file. Besides, you need to put the same kernel source code on both guest and host machine.

Notion 2: Remember to compile the kernel in the guest virtual machine and don’t forget to send a vmlinux binary file (in /your/linux_source_code_guest directory) to the host machine (send to /your/linux_source_code_host directory).

Notion 3: When compiling the code, the finalized kernel may be too large for /boot directory, then you need to strip some of the drivers to reduce the installed kernel size. Please see this.

enable debugging of qemu

I referenced this blog to enable the debugging of qemu, but I changed the debugging port to 12345. You may change the port as you like because if multiple virtual machines have “-s” option, you can only start one of them.

1	virsh edit "$GUESTNAME"

Replace the first line:

1	<domain type='kvm'>

with:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
    <qemu:commandline>
        <qemu:arg value='-gdb'/>
        <qemu:arg value='tcp::12345'/>
    </qemu:commandline>

disable the kernel aslr

You must turn it off, otherwise gdb cannot find your code. Most of the tutorials use qemu and mount an initramfs, so they just add a nokaslr in the command. When I need to debug the whole system, I need a complete virtual machine. To disable kernel aslr, I insert this in /etc/default/grub, you may check this for more details:

1	GRUB_CMDLINE_LINUX_DEFAULT="quiet nokaslr"

after that, execute

1	sudo update-grub

and reboot your virtual machine

debugging your kernel

Now use gdb -tui /your/linux_source_code_host/vmlinux to start debugging. When gdb starts, you need to use target remote :12345 to connect to the guest machine. Here 12345 is the port you defined in kvm mentioned above. Then you could set break points so that the kernel would pause as you like.

The notion is that you could only pause at the code that you set as built-in when you compile the kernel. Before the compilation, you need to run make menuconfig and for each part you may choose [ ], [*] or [M]. You could set break points when the corresponding code is set as [*]. To debug the modules, you need to do the following jobs.

debugging your module

First of all, compile your module, which is located in /home/ubuntu_guest/module_path. At the same time, send your module code to the host machine, which may be put at /home/ubuntu_host/module_path.

Since the module is not with the kernel, you need to tell where the module is loaded. After you install your module with insmod or modprobe in the guest virtual machine (Debugged modules must be installed first!), check the corresponding address (in guest machine, rather than host machine) by

1	cat /sys/module/module_name/sections/.text

Don’t forget to replace “module_name” with the name of your module, and you will get the .text address. You may get addresses like 0xffffffffc079c000 for .text.

Then send the compiled kernel module to your host machine (maybe named /home/ubuntu_host/module_path/my_module.ko). Use gdb to debug the kernel as mentioned above.

When gdb starts, use (on host machine):

1	add-symbol-file /home/ubuntu_host/module_path/module.ko 0xffffffffc079c000

to instruct gdb to load the binary file on host machine and the corresponding loaded address on guest machine. You may also need to tell the corresponding directory on host since it’s comiled on the guest machine.

1	set substitute-path /home/ubuntu_guest/module_path /home/ubuntu_host/module_path

Now you can set break points on your module files.

Setting up ide environment in vscode (optional)

With the gui interface of vscode, debugging is less painful. First of all, setting up the launch.json, and remember to replace your debugging port (mentioned above). This launch.json is on the linux directory on the most machine, and don’t forget to copy the vmlinux file from the guest.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "kernel-debug",
            "type": "cppdbg",
            "request": "launch",
            "miDebuggerServerAddress": "127.0.0.1:12345",
            "program": "${workspaceFolder}/vmlinux",
            "stopAtEntry": true,
            "cwd": "${workspaceFolder}",
            "MIMode": "gdb",
        }
    ]
}

Now start the guest virtual machine. If you just tap F5, you starts to debug the kernel, but without modules.

Write a gdb script to tell the address of the kernel module

1 2	set substitute-path /home/ubuntu_guest/module_path /home/ubuntu_host/module_path add-symbol-file /home/ubuntu_host/module_path/module.ko 0xffffffffc079c000

Load the kernel module as mentioned above by typing:

1	-exec source gdb_script

in the debug console of vscode.

Now everything is ready and enjoy it!