02 — GDB for Kernel Debugging

Technical Overview

GDB (GNU Debugger) is the primary debugging tool for compiled programs on Linux and is extensible to kernel debugging through three mechanisms: KGDB (the in-kernel GDB stub for live kernel debugging), crash dump analysis (offline analysis of vmcore with GDB), and QEMU GDB stub (the easiest path for kernel development). Understanding GDB's fundamentals — breakpoints, watchpoints, backtraces, memory inspection, frame navigation — is prerequisite to using it in any of these contexts. The GDB Python API further enables custom scripts for kernel data structure navigation, making it practical to navigate the kernel's complex linked lists, trees, and embedded structs.

---

Prerequisites

• Familiarity with C and assembly (GDB exposes both)
• Understanding of kernel data structures (task_struct, mm_struct, etc.)
• Comfort with Linux build system (compiling a kernel or module with debug symbols)
• For KGDB: access to a serial console or network connection between host and target
• For QEMU debugging: QEMU installation and basic VM management

---

Core Content

GDB Fundamentals

# Start GDB with a binary
gdb ./myprogram

# Load a coredump for a user-space application
gdb ./myprogram /var/crash/core.12345

# Attach to a running process
gdb -p 12345
# or inside gdb:
(gdb) attach 12345

# Basic commands:
(gdb) run arg1 arg2      # start the program
(gdb) continue           # continue after a break (alias: c)
(gdb) step               # single-step, entering functions (alias: s)
(gdb) next               # single-step, NOT entering functions (alias: n)
(gdb) stepi              # single machine instruction step
(gdb) nexti              # single machine instruction, don't enter calls
(gdb) finish             # run until current function returns
(gdb) quit               # exit GDB

Breakpoints:

(gdb) break main                    # break at function entry
(gdb) break myfile.c:145            # break at specific line
(gdb) break *0xffffffffa5e4c1f3     # break at raw address
(gdb) break my_driver_write if len > 1024  # conditional break
(gdb) info breakpoints              # list all breakpoints
(gdb) delete 3                      # delete breakpoint #3
(gdb) disable 2                     # disable without deleting
(gdb) enable 2                      # re-enable

# Hardware breakpoints (limited to 4 on x86-64, survive memory write)
(gdb) hbreak *0xffffffffc0a23178   # hardware breakpoint at address

Watchpoints (break when memory is accessed/modified):

(gdb) watch my_variable             # break on write to variable
(gdb) rwatch my_variable            # break on read
(gdb) awatch my_variable            # break on read or write
(gdb) watch -l *((int*)0xffff9a7b82f3d800)  # watch specific address

Backtraces and frame navigation:

(gdb) backtrace                     # print call stack (alias: bt)
(gdb) backtrace full               # bt with all local variables
(gdb) frame 3                      # switch to frame #3
(gdb) info frame                   # details about current frame
(gdb) info locals                  # local variables in current frame
(gdb) info args                    # function arguments in current frame
(gdb) up                           # go to calling frame
(gdb) down                         # go to called frame

Memory inspection:

(gdb) print my_struct->field        # print value of expression
(gdb) print *ptr                    # dereference pointer
(gdb) print/x my_var               # print in hex
(gdb) print/d my_var               # print as decimal
(gdb) print/t my_var               # print in binary
(gdb) x/10gx 0xffff9a7b82f3d800   # examine 10 8-byte words in hex at address
(gdb) x/10i $rip                   # examine 10 instructions at RIP
(gdb) x/s 0xffff9a7b82001000       # examine as string
(gdb) info registers               # all register values
(gdb) print $rip                   # instruction pointer
(gdb) print $rsp                   # stack pointer

KGDB: In-Kernel GDB Stub

KGDB provides a GDB stub inside the Linux kernel, allowing a host GDB to connect to and debug a target kernel via serial port or network (KGDBOE — KGDB over Ethernet).

Setup on the target (kernel being debugged):

# Method 1: Kernel command line (GRUB)
# Add to /etc/default/grub:
GRUB_CMDLINE_LINUX="kgdboc=ttyS0,115200 kgdbwait"
#                          ^serial port  ^baud    ^wait for GDB before booting

# Method 2: Dynamic setup (no reboot required, only on ttyS0)
echo "ttyS0,115200" > /sys/module/kgdboc/parameters/kgdboc

# Trigger a breakpoint to enter KGDB (drop to GDB stub)
# Method A: Magic SysRq
echo g > /proc/sysrq-trigger   # kernel enters KGDB debug mode

# Method B: from kernel panic (if kgdbwait is set, kernel waits for GDB on boot)

# Method C: from kernel code
# Add BREAKPOINT() macro to kernel source at desired location

Connecting from the host (running GDB):

# On the host machine (different machine connected via serial cable)
gdb vmlinux   # vmlinux WITH debug symbols (not stripped)

# Connect to target via serial
(gdb) target remote /dev/ttyS0
# or via network (KGDBOE):
(gdb) target remote 192.168.1.100:7000

# Load kernel symbols
(gdb) lx-symbols   # if using lx-* GDB scripts (see below)

# Now standard GDB commands work
(gdb) bt           # backtrace of current execution
(gdb) info threads # list all kernel threads
(gdb) thread 5     # switch to kernel thread 5
(gdb) continue     # resume kernel execution
(gdb) break schedule  # set breakpoint on kernel scheduler

KGDB limitations: - Cannot debug CPU-0 exception handlers (KGDB uses CPU-0 for its communication) - Requires a physical serial connection or network between host and target - The kernel is frozen while in KGDB — all CPUs stop. Not suitable for timing-sensitive debugging. - SMP debugging: only the CPU that hit the breakpoint is stopped initially; other CPUs are stopped via NMI when KGDB is entered.

QEMU + GDB: The Development Standard

QEMU's -gdb flag exposes a GDB stub that makes kernel debugging trivial without physical hardware:

# Start QEMU with GDB stub (paused, waiting for GDB connection)
qemu-system-x86_64 \
  -kernel /path/to/arch/x86/boot/bzImage \
  -initrd /path/to/initramfs.cpio.gz \
  -append "root=/dev/sda console=ttyS0 nokaslr" \
  -m 2G \
  -nographic \
  -gdb tcp::1234 \
  -S   # Start paused, waiting for GDB
  # -S means QEMU waits for GDB before executing the first instruction

# In another terminal: connect GDB to QEMU
gdb vmlinux
(gdb) target remote localhost:1234

# Optional: disable ASLR for easier address matching
# Already done with nokaslr in kernel command line above

# Set a breakpoint BEFORE resuming
(gdb) break start_kernel   # break at kernel entry point
(gdb) continue             # QEMU now runs until start_kernel

# Step through kernel initialization
(gdb) next
(gdb) step
(gdb) bt

# Debug a specific subsystem by setting breakpoints
(gdb) break tcp_connect    # break when any process calls tcp_connect
(gdb) continue
# Now in a different terminal on the QEMU VM, run: curl http://example.com
# GDB will break when the connection is made
(gdb) info args            # see what socket was passed

GDB Scripts for Kernel Debugging

The Linux kernel includes GDB helper scripts (scripts/gdb/) that provide kernel-specific commands. These ship with most distributions as python-gdb extensions.

# Load kernel GDB scripts
(gdb) source /usr/share/gdb/auto-load/vmlinux-gdb.py
# or
(gdb) add-auto-load-safe-path /path/to/linux-source

# After loading, kernel-specific commands become available:

# lx-ps: list all processes (like 'ps')
(gdb) lx-ps
     pid       task            comm
       0  0xffffffff82212780  swapper/0
       1  0xffff9a7b80010000  systemd
    1234  0xffff9a7b82f3d800  myprocess

# lx-dmesg: print kernel message buffer
(gdb) lx-dmesg

# lx-lsmod: list loaded modules
(gdb) lx-lsmod

# lx-symbols: reload all module symbols (after loading a new module)
(gdb) lx-symbols

# lx-fdtdump: dump flattened device tree (for embedded kernels)
(gdb) lx-fdtdump

Writing custom GDB Python scripts for kernel structures:

# ~/.gdbinit or loaded via: source /path/to/script.py
import gdb

class LxListForEach(gdb.Command):
    """Iterate a Linux kernel list_head and print each element."""

    def __init__(self):
        super().__init__("lx-list-for-each", gdb.COMMAND_USER)

    def invoke(self, args, from_tty):
        # Parse args: lx-list-for-each list_head_addr struct_type offset_field
        argv = gdb.string_to_argv(args)
        head_addr = int(argv[0], 16)
        struct_type = argv[1]

        # Get list_head type
        list_type = gdb.lookup_type("struct list_head")

        # Walk the list
        head = gdb.Value(head_addr).cast(list_type.pointer())
        curr = head['next']
        while int(curr) != int(head):
            # Compute container_of: curr - offsetof(struct_type, list_head_field)
            struct_ptr = curr.cast(gdb.lookup_type(struct_type).pointer())
            print(struct_ptr.dereference())
            curr = curr['next']

LxListForEach()

# Using the custom command to walk task list
(gdb) lx-list-for-each 0xffffffff82212780 "struct task_struct"

# Navigate struct manually
(gdb) set $task = (struct task_struct*)0xffff9a7b82f3d800
(gdb) print $task->comm        # process name
(gdb) print $task->pid         # PID
(gdb) print $task->state       # process state
(gdb) print $task->mm->start_brk  # heap start address

Debugging Kernel Modules with GDB

Modules loaded into a running kernel are at dynamic addresses. GDB needs these addresses to resolve symbols.

# Find module load address
cat /proc/modules | grep mydriver
# Output: mydriver 16384 0 - Live 0xffffffffc0a23000

# Or from /sys/module/
cat /sys/module/mydriver/sections/.text
# Output: 0xffffffffc0a23000

# In GDB (crash dump or KGDB):
(gdb) add-symbol-file /lib/modules/5.15.0-76/kernel/drivers/mydriver/mydriver.ko \
         0xffffffffc0a23000
# -s .text 0xXXX (if text section is different from module base)

# Now module symbols are resolved
(gdb) break my_driver_write   # sets breakpoint in module
(gdb) dis my_driver_write     # disassemble module function

GDB for vmcore Analysis

GDB can analyze vmcore crash dumps directly (without the crash tool), though crash is generally more ergonomic:

gdb /usr/lib/debug/boot/vmlinux-5.15.0-76-generic /var/crash/vmcore

# Get backtrace (note: for crash dumps, GDB doesn't know about threads without crash)
(gdb) bt   # backtrace of the "active" context at crash

# Read a kernel variable
(gdb) print init_task.comm   # the initial task's command name

# Read memory at a kernel address
(gdb) x/10gx 0xffff9a7b82f3d800

# Disassemble a function at crash time
(gdb) disas my_driver_write

# Find the panicking instruction
(gdb) x/5i ($pc - 8)   # 5 instructions around the program counter

GDB Python API for Custom Analysis

GDB's Python API enables powerful custom analysis scripts, especially useful for kernel data structure navigation:

# Script to find all processes in uninterruptible sleep (D state)
import gdb

def find_d_state_processes():
    """Find all processes in D (uninterruptible sleep) state."""
    TASK_UNINTERRUPTIBLE = 2  # from include/linux/sched.h

    # Get the init_task symbol
    init_task = gdb.parse_and_eval("init_task")

    # Walk the tasks list using the tasks.next/prev linked list
    tasks = gdb.lookup_type("struct task_struct")

    # container_of implementation
    task_ptr = init_task.address
    result = []

    while True:
        task = task_ptr.dereference()
        state = int(task["state"])
        if state & TASK_UNINTERRUPTIBLE:
            pid = int(task["pid"])
            comm = task["comm"].string()
            result.append((pid, comm))

        next_ptr = task["tasks"]["next"]
        # Compute container_of: task_ptr = next_ptr - offsetof(task_struct, tasks)
        offset = gdb.parse_and_eval("(int)&((struct task_struct*)0)->tasks")
        task_ptr = (next_ptr.cast(gdb.lookup_type("char").pointer()) - int(offset)).cast(
            tasks.pointer()
        )

        if int(task_ptr) == int(init_task.address):
            break

    for pid, comm in result:
        print(f"PID {pid}: {comm} (D state)")

find_d_state_processes()

---

Historical Context

GDB was created by Richard Stallman in 1986 as part of the GNU project. Its architecture — a remote protocol (RSP) for communicating with a debug stub on the target — was designed to support both local and remote debugging from the start. The Remote Serial Protocol (RSP) is still the wire protocol KGDB uses.

KGDB was first implemented around 2001 (various independent implementations existed) and merged into the mainline Linux kernel in 2.6.26 (2008). Before mainline inclusion, kernel debugging required out-of-tree patches or emulators.

QEMU's GDB stub (present since QEMU 0.1) made kernel debugging practical for kernel developers who lack access to physical serial connections. The combination of QEMU + GDB + virtio devices is now the standard kernel development environment for most kernel contributors.

The GDB Python API was added in GDB 7.0 (2009), enabling the kernel GDB scripts (lx-ps, lx-dmesg etc.) that ship with the kernel source since Linux 4.0 (2015).

---

Production Examples

# Full workflow: debug a kernel module crash in QEMU

# 1. Compile kernel with debug symbols
make menuconfig   # Enable: CONFIG_DEBUG_INFO=y, CONFIG_KGDB=y, CONFIG_FRAME_POINTER=y
make -j$(nproc)

# 2. Start QEMU
qemu-system-x86_64 -kernel arch/x86/boot/bzImage \
  -hda /path/to/rootfs.qcow2 \
  -append "root=/dev/sda rw console=ttyS0 nokaslr" \
  -nographic -m 2G -gdb tcp::1234 -S &

# 3. Connect GDB
gdb vmlinux
(gdb) target remote :1234
(gdb) lx-symbols
(gdb) break mymodule_init   # break when module loads
(gdb) continue

# In QEMU VM shell:
insmod mymodule.ko           # triggers the breakpoint

# In GDB:
(gdb) info registers
(gdb) bt
(gdb) step                   # step through module initialization

---

Debugging Notes

"Remote connection closed" when connecting to KGDB: The target kernel may not be in KGDB mode. Verify with echo g > /proc/sysrq-trigger on the target. Also verify baud rate matches on both ends.

Breakpoint not hitting in kernel module: Module was compiled without debug symbols, or symbols not loaded. Run add-symbol-file with the correct .text address.

GDB showing wrong source lines: Kernel built without -O0 — optimizations inline and reorder code, making source-line correspondence inaccurate. For debugging, disable optimization with Makefile change (CFLAGS_myfile.o := -O0) for the specific file.

"Cannot access memory at address X": In KGDB, the target CPU is paused. The address may be in a CPU-local variable (per-CPU, accessible only from that CPU) or in a memory-mapped hardware register that requires CPU context to read.

---

Security Implications

• KGDB with kgdboc=ttyS0 gives anyone with physical serial port access full kernel debug capability — reading arbitrary kernel memory, setting breakpoints, patching memory. Never enable in production without physical port security.
• KGDBOE (KGDB over Ethernet) is even more dangerous — it allows remote kernel debugging. Never expose KGDBOE to untrusted networks. If used, restrict to a dedicated management network with firewall rules.
• gdb -p PID on a running process allows reading that process's memory. This requires ptrace capability (CAP_SYS_PTRACE). Restrict with: echo 1 > /proc/sys/kernel/yama/ptrace_scope (prevents non-parent ptrace).
• vmlinux with debug symbols contains all kernel addresses — essentially a roadmap for exploitation. Protect vmlinux files (only accessible to root).

---

Performance Implications

• KGDB freezes all CPUs while a breakpoint is active. Production impact: complete system stall. Never use in production.
• QEMU GDB stub adds minimal overhead when not connected. With GDB connected but running freely (continue): ~5% overhead from QEMU's trap checking.
• Symbol file loading (add-symbol-file) for large kernel modules can take seconds. For the full vmlinux: tens of seconds.

---

Failure Modes and Real Incidents

KGDB deadlock during scheduler debugging: Setting a breakpoint in schedule() caused a deadlock because entering KGDB itself requires scheduling. Fix: use hardware breakpoints (hbreak) in scheduler code, and ensure you understand which kernel locks are held at the breakpoint location.

GDB crash analysis mismatch: A vmcore was captured from kernel version 5.15.0-76 but analyzed with vmlinux from 5.15.0-75. GDB loaded symbols but struct offsets were different (the struct layouts had changed between minor versions). All print commands returned garbage. Fix: always use the exact matching vmlinux for crash analysis; store vmlinux alongside vmcore.

---

Modern Usage

• GDB 14.x (2023): improved Python API, better TUI (text user interface) mode, improved DWARF parsing for modern compiler outputs.
• VSCode + GDB: The VSCode C/C++ extension wraps GDB with a graphical interface, supporting KGDB remote debugging with a launch.json configuration.
• GDB Dashboard: a Python-based GDB extension that provides a rich TUI showing registers, stack, disassembly, and source simultaneously.

---

Future Directions

• GDB and eBPF uprobes: using eBPF uprobes as software breakpoints for user-space code, enabling tracing without GDB's overhead.
• LLDB for kernel debugging: LLDB (LLVM's debugger) is gaining kernel debugging capabilities. Some architectures (ARM64 with LLDB) have better support than GDB.
• Live patching + GDB: the interaction between kpatch/livepatch and GDB symbols needs careful handling — patched functions have new addresses.

---

Exercises

1. GDB userspace warmup: Compile a simple C program with a segfault (NULL dereference). Run it under GDB, catch the SIGSEGV, print the backtrace, inspect register values, and identify the exact line causing the fault using list and frame commands.

2. QEMU + GDB kernel debugging: Set up a QEMU VM with a custom-compiled kernel (debug symbols enabled). Connect GDB. Set a breakpoint on do_sys_open (or do_sys_openat2 in newer kernels). Run cat /etc/hostname in the QEMU VM and catch the break. Print the filename argument and all local variables.

3. Module symbol loading: Write a minimal kernel module that panics when its /proc file is read. Compile it with debug symbols. Load it in a QEMU VM, trigger the panic, and capture the vmcore. Use GDB + add-symbol-file to decode the exact source line of the panic.

4. GDB Python script: Write a GDB Python script that walks the Linux kernel's task_struct list and prints: PID, command name, process state (TASK_RUNNING=0, TASK_INTERRUPTIBLE=1, etc.), and CPU affinity. Test it on a QEMU vmcore or live KGDB session.

5. Kernel data structure inspection: In GDB with a vmcore or KGDB session, find the tcp_sock struct for an established TCP connection: (a) use lx-ps to find a process with open sockets, (b) use files (via custom script or crash tool) to find the socket FD, (c) walk the socket struct to find the remote IP address and port.

---

References

• GDB Manual: https://sourceware.org/gdb/current/onlinedocs/gdb/
• Linux Kernel Documentation: Documentation/dev-tools/kgdb.rst
• Linux Kernel GDB Scripts: Documentation/dev-tools/gdb-kernel-debugging.rst
• Matz, Michael. "Debugging with GDB." Red Hat Developer Guide.
• QEMU GDB documentation: https://www.qemu.org/docs/master/system/gdb.html
• Love, Robert. Linux Kernel Development. 3rd ed. Addison-Wesley, 2010. Chapter 18 (Debugging).
• Corbet, Jonathan et al. Linux Device Drivers. 3rd ed. O'Reilly, 2005. Chapter 4 (Debugging Techniques).