Skip to content

05 — Debug Info and DWARF

Overview

When a compiler transforms source code into machine code, it discards almost everything a human cares about: variable names, type definitions, source file positions, the mapping between a line of C and the five instructions it became. Debug information is the metadata that restores this mapping — the bridge between the binary a machine executes and the source a programmer reads. DWARF (Debugging With Attributed Record Formats) is the standard encoding of this metadata, used on Linux, macOS, BSD, and increasingly on other platforms. Understanding DWARF reveals how debuggers work, how stack unwinding is implemented, how sanitizers find bugs across function call boundaries, and why your -g build is ten times larger than your -O2 build.

Prerequisites

  • Understanding of ELF binary format and sections (see 30-compilers-and-linkers/03-linkers-and-loaders.md)
  • Familiarity with the compilation pipeline (see 30-compilers-and-linkers/01-compilation-pipeline.md)
  • Basic understanding of x86-64 calling conventions (frame pointer, stack layout)
  • Some exposure to GDB or LLDB usage

Historical Context

Why Machine Code Needs Metadata

Consider a simple C function:

int add(int a, int b) {
    int result = a + b;
    return result;
}

After compilation with -O0 (no optimization):

add:
    push   rbp
    mov    rbp, rsp
    mov    DWORD PTR [rbp-0x4], edi   ; store 'a' at rbp-4
    mov    DWORD PTR [rbp-0x8], esi   ; store 'b' at rbp-8
    mov    eax, DWORD PTR [rbp-0x4]
    add    eax, DWORD PTR [rbp-0x8]
    mov    DWORD PTR [rbp-0xc], eax   ; store 'result' at rbp-12
    mov    eax, DWORD PTR [rbp-0xc]
    pop    rbp
    ret

The machine code has no variable names, no type information, no source line numbers. A debugger wanting to show p result must know: "when execution is at this address, the variable named result of type int is located at memory address rbp - 12." This knowledge is DWARF.

DWARF History

DWARF (the acronym is humorous — it was created alongside the ELF format) was first specified in the late 1980s for UNIX System V. DWARF2 (1993) became the widespread standard, followed by DWARF3 (2005), DWARF4 (2010), and DWARF5 (2017). DWARF5 is the current standard, though most toolchains still default to DWARF4 for compatibility.

Apple maintains a DWARF variant (with extensions) in .dSYM bundles on macOS. Microsoft uses PDB (Program Database) format on Windows, though LLVM/Clang on Windows can also emit DWARF.

DWARF Structure: DIEs

The core DWARF data structure is the DIE (Debugging Information Entry). A DIE is a node in a tree. It has:

  • A tag (DW_TAG_*): the kind of entity it represents
  • A set of attributes (DW_AT_*): key-value pairs describing the entity
  • Children: nested DIEs

The tree structure mirrors the lexical structure of the source code:

DWARF DIE Tree (for a simple C compilation unit):

DW_TAG_compile_unit
  DW_AT_name: "add.c"
  DW_AT_language: DW_LANG_C99
  DW_AT_comp_dir: "/home/user/src"
  DW_AT_low_pc: 0x401000   (start of code for this CU)
  DW_AT_high_pc: 0x401050  (end of code for this CU)
  |
  +-- DW_TAG_base_type
  |     DW_AT_name: "int"
  |     DW_AT_encoding: DW_ATE_signed
  |     DW_AT_byte_size: 4
  |
  +-- DW_TAG_subprogram
        DW_AT_name: "add"
        DW_AT_decl_file: 1
        DW_AT_decl_line: 1
        DW_AT_type: <ref to DW_TAG_base_type "int">
        DW_AT_low_pc: 0x401000
        DW_AT_high_pc: 0x401020
        DW_AT_frame_base: DW_OP_reg6 (rbp)
        |
        +-- DW_TAG_formal_parameter
        |     DW_AT_name: "a"
        |     DW_AT_type: <ref to int>
        |     DW_AT_location: DW_OP_fbreg -4
        |
        +-- DW_TAG_formal_parameter
        |     DW_AT_name: "b"
        |     DW_AT_type: <ref to int>
        |     DW_AT_location: DW_OP_fbreg -8
        |
        +-- DW_TAG_variable
              DW_AT_name: "result"
              DW_AT_type: <ref to int>
              DW_AT_location: DW_OP_fbreg -12

The tag taxonomy is rich: DW_TAG_class_type, DW_TAG_typedef, DW_TAG_array_type, DW_TAG_pointer_type, DW_TAG_structure_type, DW_TAG_enumeration_type, DW_TAG_namespace, DW_TAG_inlined_subroutine, DW_TAG_lexical_block. DWARF can represent essentially the entire type system of C++, Rust, Go, or Ada.

ELF Sections Carrying DWARF

DWARF information is stored in ELF sections. The naming convention: .debug_* sections in unlinked objects, potentially moved to a separate .dwo file with split DWARF.

ELF Section         DWARF Content
-------------------------------------------------------------
.debug_info         DIE trees (the main DWARF payload)
.debug_abbrev       Abbreviation tables (compress .debug_info)
.debug_line         Line number information (PC → source line map)
.debug_frame        Call Frame Information (stack unwinding tables)
.eh_frame           Exception Handling frame info (C++ unwinding, GCC ABI)
.debug_loc          Variable location lists (locations that change with PC)
.debug_loclists     (DWARF5 replacement for .debug_loc)
.debug_ranges       PC range lists for DIEs covering non-contiguous code
.debug_rnglists     (DWARF5 replacement for .debug_ranges)
.debug_str          String table (deduplicated strings from DIEs)
.debug_types        Type DIEs (DWARF4 type units, reduce duplication)
.debug_pubnames     Global name lookup table (legacy)
.debug_aranges      Address-to-CU lookup table (used by debuggers for fast lookup)

The .debug_abbrev section is a compression mechanism: instead of storing the full attribute name in every DIE, an abbreviation table assigns short codes. A DIE stores its abbreviation code, and the debugger looks up the corresponding tag + attribute list.

Line Number Information (.debug_line)

The line number program is a compact encoding of the mapping: program counter → (source file, line, column). It is specified as a state machine with a small instruction set that updates a tuple of registers.

Line number state machine registers:
  address:    current program counter
  file:       index into file table
  line:       source line number (1-based)
  column:     source column (0 = unknown)
  is_stmt:    true if good breakpoint location
  basic_block: true at start of basic block
  end_sequence: true at end of instruction sequence

Opcodes (examples):
  DW_LNS_advance_pc:    increment address
  DW_LNS_advance_line:  increment line counter
  DW_LNS_set_file:      change current file
  DW_LNS_copy:          emit a row to the matrix (record current state)
  DW_LNE_end_sequence:  mark end, reset state

Special opcodes (single byte):
  Encode small simultaneous address + line advances in one byte.
  Example opcode 0x48:
    line_advance = (0x48 - opcode_base) % line_range + line_base
    addr_advance = (0x48 - opcode_base) / line_range * min_insn_length

The result of running the state machine is the line number matrix: a sorted table of rows, each mapping a PC range to a source location. Debuggers binary-search this table to answer "what source line is executing at address 0x401013?"

Call Frame Information (CFI)

CFI answers the question: "given execution is at address X, how do I unwind the call stack?" This is needed by:

  • Debuggers: to produce a backtrace when stopped at a breakpoint
  • C++ exception handling: to find exception handlers on the stack (__cxa_throw walks frames)
  • Sanitizers: AddressSanitizer and others use stack unwinding for error reports
  • Profilers: Linux perf, pprof use CFI for frame-pointer-free stack sampling

CFI encodes CFA (Canonical Frame Address) and register rules for each PC:

CFI example for the add() function:

At PC = add (function prologue start):
  CFA = rsp + 8        (return address pushed by call)
  rip = CFA - 8        (saved return address is at CFA-8)

After push rbp:
  CFA = rsp + 16
  rbp = CFA - 16       (saved rbp is now at CFA-16)

After mov rbp, rsp:
  CFA = rbp + 16       (frame pointer set — CFA now derived from rbp)

At any interior PC:
  Unwind: caller's rsp = CFA, caller's rip = [CFA-8], caller's rbp = [CFA-16]

CFI is stored in .debug_frame (DWARF) and .eh_frame (GNU/GCC ABI, used for exception handling even in non-debug builds). The .eh_frame section is present in production binaries; .debug_frame may be stripped.

The CFI encoding is similar to the line number state machine: a sequence of instructions that build an unwind table. DW_CFA_def_cfa, DW_CFA_offset, DW_CFA_register, DW_CFA_advance_loc are common operations.

DWARF Expressions: Variable Location

For optimized code, a variable's location may not be a simple rbp-12. The optimizer may:

  • Keep a variable in a register (never spill to memory)
  • Spill it at some points but keep it in a register at others
  • Compute it from other live values (no dedicated storage)
  • Eliminate it entirely if it is constant-folded

DWARF expressions (DW_OP_* opcodes) form a Turing-complete stack machine for describing variable locations:

DW_OP_reg0 through DW_OP_reg31:
  "The variable is in register N right now."

DW_OP_fbreg offset:
  "The variable is at [frame_base + offset] in memory."

DW_OP_addr address:
  "The variable is at this absolute address (static/global)."

DW_OP_stack_value:
  "The variable's value IS the top of the DWARF stack
   (not a memory address — value computed from other values)."

Complex example (variable held in two registers, reconstructed):
  DW_OP_reg3       ; push high 32 bits from rdx
  DW_OP_const4u 32
  DW_OP_shl        ; shift left 32
  DW_OP_reg0       ; push low 32 bits from rax
  DW_OP_or         ; combine into 64-bit value
  DW_OP_stack_value

When the variable's location changes as execution progresses through a function (register allocated differently at different points), a location list (.debug_loclists) provides a sequence of [PC range, location expression] pairs. The debugger selects the correct entry for the current PC.

ORC: Linux Kernel's DWARF Replacement

The Linux kernel does not use DWARF for its own stack unwinding (kernel oops reports, perf). Generating DWARF for a kernel module and unwinding from an interrupt handler using DWARF is reliable but slow — parsing DWARF tables on the oops path takes milliseconds.

ORC (Oops Rewind Capability), introduced in Linux 4.14 (2017) by Josh Poimboeuf, replaces kernel DWARF unwinding with a simpler, faster table format:

ORC entry (8 bytes per PC):
  sp_offset: 16-bit  (how to compute SP of previous frame from current SP/BP)
  bp_offset: 16-bit  (how to compute saved BP)
  sp_reg:    2-bit   (which register SP is relative to: SP or BP)
  bp_reg:    2-bit   (which register BP is relative to)
  type:      2-bit   (CALL, REGS, REGS_PARTIAL — frame type)

ORC unwind tables are generated at kernel build time by objtool, which analyzes the object files directly. The result is a sorted, binary-searchable table that unwinds one frame in ~100 nanoseconds — 100x faster than DWARF unwinding. ORC is now the only kernel stack unwinding mechanism; frame pointer unwinding (CONFIG_FRAME_POINTER) is a separate option and is slower.

Split DWARF (.dwo Files)

Debug info is large. For a Chromium build with debug symbols, .debug_info can reach tens of gigabytes. This bloats link times: the linker must process all .debug_info sections when producing the final binary.

Split DWARF (DWARF4 extension, proposed by Cary Coutant; DWARF5 standard) separates type and most DIE information into .dwo (DWARF object) files that are not fed to the linker. The linked binary contains only small .debug_info stubs that reference the .dwo files by build ID.

Without split DWARF:
  foo.o (large .debug_info) ---+
  bar.o (large .debug_info) ---+--> linker --> a.out (huge, slow link)
  baz.o (large .debug_info) ---+

With split DWARF:
  foo.o (tiny .debug_info stub) ---+
  foo.dwo (full debug info)        |
  bar.o (stub) ---+--> linker --> a.out (small, fast link)
  bar.dwo          |
  baz.o (stub) ---+
  baz.dwo

GDB/LLDB reads a.out, finds .dwo references, loads .dwo files on demand.

Compile with: clang -gsplit-dwarf or gcc -gsplit-dwarf. Pair with -fdebug-types-section to further deduplicate type information. Split DWARF can reduce linker memory usage 5–10x for large C++ projects.

debuginfod: Remote Debug Info Server

Stripped production binaries have no debug info. When a crash dump is captured and analyzed on a developer's machine, the debugger cannot resolve symbols or source lines without the matching debug info.

debuginfod (elfutils project, 2019) is an HTTP server that indexes debug info and source files by build ID, serving them on demand to debuggers:

Workflow:
  1. CI/CD builds binary, strips it, uploads to production
  2. CI/CD also uploads .debug file (unstripped) to debuginfod server
  3. Production binary crashes → core dump collected

  On developer machine:
  4. gdb -c core ./stripped-binary
  5. GDB reads build ID from binary
  6. GDB queries debuginfod: GET /buildid/<id>/debuginfo
  7. debuginfod returns debug info from index
  8. GDB loads debug info → full source-level backtrace

  DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org" gdb ...

Fedora, Ubuntu, Debian, Arch Linux, and OpenSUSE operate public debuginfod servers. Fedora's server provides debug info for all packages. Setting DEBUGINFOD_URLS in your environment makes GDB and LLDB transparently fetch missing debug info.

Production Examples

GDB with optimized code: When debugging a -O2 binary, variables are often marked <optimized out> — their DWARF location expression evaluated to no valid location at the current PC. This is correct behavior. To debug optimized code effectively, add -fno-omit-frame-pointer and -O1 -g rather than -O2 -g.

Linux kernel crash dump analysis: crash utility reads kernel vmcore files with the DWARF (or ORC) debug info from vmlinux to produce symbolic stack traces. eu-stack -p <pid> uses libdw (elfutils) to walk DWARF CFI for a running process.

Go runtime: The Go toolchain emits DWARF for Go programs. dlv (Delve debugger) uses Go DWARF to support goroutine-aware debugging, including showing goroutine stacks and switching between goroutines in the debugger.

Rust and DWARF: Rust emits rich DWARF including type information for generics, closures, and trait objects. LLDB uses this to pretty-print Vec<T>, Option<T>, and Result<T, E> without custom formatters.

Debugging Notes

  • Verify debug info is present: readelf -S binary | grep debug lists DWARF sections. llvm-dwarfdump binary dumps the DWARF tree in human-readable WAT-like format. readelf --debug-dump=info binary is the GNU equivalent.
  • Missing frame info: If bt in GDB shows ?? frames, the binary lacks .eh_frame for those functions. Compile with -fasynchronous-unwind-tables (default on x86-64 Linux) to ensure frame tables are always emitted.
  • DWARF version mismatch: Mixing DWARF4 and DWARF5 object files in one binary is generally safe (linker handles both), but some older debuggers only support up to DWARF4. Check with llvm-dwarfdump --verify binary.
  • Stripped binary, separate debug file: Ubuntu ships foo and foo-dbgsym packages. The debug info is in /usr/lib/debug/usr/bin/foo.debug. GDB automatically loads it via build-ID lookup if the debug package is installed, or if debuginfod is configured.
  • Core dump analysis: gdb binary core requires the binary at the same path as when it crashed, or set set sysroot /path/to/sysroot. The build ID embedded in the core file must match the debug info.

Security Implications

  • Debug info as intelligence: DWARF embedded in production binaries leaks function names, variable names, type layouts, and source file paths — useful for reverse engineers. Strip production binaries (strip binary) and deploy debug info separately.
  • Symbol server trust: A compromised debuginfod server could serve debug info that misleads developers — showing incorrect source lines, hiding malicious code. Verify debuginfod server identity with TLS and use build-ID verification.
  • CFI as attack surface: .eh_frame is parsed by libgcc_s and libunwind during exception unwinding. Bugs in DWARF unwinding (e.g., corrupted .eh_frame) can crash programs or be exploited via malicious ELF files. Keep elfutils and libunwind updated.
  • Type confusion via DWARF: Debuggers that trust DWARF unconditionally could be misled by a malicious binary with crafted DWARF to display incorrect memory layouts. gdb's scripting interface combined with crafted DWARF has been a CTF technique.

Performance Implications

  • Debug info build time: A full DWARF debug build takes 2–5x longer to link than a release build due to debug section processing. Use split DWARF (-gsplit-dwarf) and parallel linking (lld) to mitigate.
  • Binary size: A debug build of a medium C++ project (100K lines) may produce 500 MB of .debug_info. Use -gz (DWARF section compression with zlib) to reduce on-disk size 3–5x. Link with lld --compress-debug-sections=zlib.
  • Debugger startup time: GDB loading a large binary eagerly reads and indexes DWARF. For multi-GB debug info, startup can take 30–60 seconds. gdb -readnow forces eager load; gdb -readnever disables it (backtrace-only mode). LLDB's .index cache (lldb-index-store) accelerates repeated debugging sessions.
  • Stack unwinding cost: libunwind using DWARF CFI takes ~1–5μs per frame. ORC in the kernel takes ~100ns per frame. For performance-sensitive profilers (sampling at 10,000 Hz), this overhead is significant. Frame-pointer-based unwinding (~50ns per frame) is faster but requires -fno-omit-frame-pointer everywhere.

Failure Modes

  • Inlining breaks variable visibility: When a function is inlined, its DIE becomes a DW_TAG_inlined_subroutine child of the caller's DW_TAG_subprogram. Variables inlined functions may be invisible in the debugger or show the inlined call site as the source location.
  • PC/line mismatch after patching: If a production binary is hot-patched (live patching) without updating debug info, debugger source lines will be wrong. Always deploy updated debug info when live-patching.
  • LTO and DWARF: Link-Time Optimization (LTO) restructures code across translation units. Standard DWARF DIEs reference compiler-unit-local offsets. LTO-aware DWARF generation (-flto -fno-lto-partition) is required for accurate debug info in LTO builds.
  • Async stack traces: For coroutine-based runtimes (Rust async, Go goroutines, C++20 coroutines), DWARF has no standard way to represent the virtual call stack. Each runtime implements custom unwinding logic (Go's runtime.Stack(), Rust's async-backtrace crate).

Modern Usage (2024–2025)

  • DWARF5 adoption: LLVM 14+ and GCC 11+ default to DWARF5, which brings more compact encodings (DW_FORM_strx, DW_FORM_addrx), improved location list format (.debug_loclists), and better support for Rust/C++ generics.
  • BOLT and debug info: Facebook's BOLT (Binary Optimization and Layout Tool) reorders functions in a binary for better I-cache behavior. It updates DWARF debug info to remain accurate after code relayout.
  • Sanitizer DWARF integration: AddressSanitizer, UBSan, and MemorySanitizer use DWARF to print accurate source locations when reporting errors. DWARF5 location list improvements make sanitizer reports more accurate in optimized builds.
  • Continuous profiling with DWARF: Tools like Polar Signals (parca), Datadog Continuous Profiler, and Google Cloud Profiler use DWARF CFI for in-process stack sampling, producing symbolized flame graphs without system overhead.

Future Directions

  • DWARF and Wasm: WASM modules can embed DWARF debug info (.debug_* custom sections). Chrome DevTools uses this for source-level WASM debugging. The WASM debug info specification (DWARF extensions for WASM) is being standardized.
  • Compressed debug info (DWARF6): The DWARF committee is working on DWARF6, which will include improved compression, better support for modern language features (C++23, Rust 2024 edition), and more compact address encoding.
  • Machine-readable CTDB (Compact Type Database): CTF (Compact Type Format, originally from Sun/DTrace) and BTF (BPF Type Format, Linux kernel) are DWARF alternatives for specific use cases (kernel type info, eBPF verifier). They are simpler and faster to parse but less expressive.
  • AI-assisted debug info recovery: For stripped binaries, ML models (trained on source/binary pairs) can recover approximate variable names and types — a form of reverse engineering guided by DWARF patterns.

Exercises

  1. DWARF inspection: Compile a C program with gcc -g -O0. Run llvm-dwarfdump --debug-info a.out | head -200. Identify the DW_TAG_compile_unit, a DW_TAG_subprogram, and a DW_TAG_variable. Find the variable's location expression.

  2. Line number table: Run llvm-dwarfdump --debug-line a.out. Find the row in the line number matrix corresponding to a specific source line. Set a breakpoint in GDB at that line and verify it matches the address in the table.

  3. CFI unwinding: Compile with -O2 -g -fno-omit-frame-pointer and without frame pointer. Compare the .eh_frame sections with llvm-dwarfdump --debug-frame. In GDB, trigger a crash and compare backtraces between the two builds.

  4. Split DWARF: Build a medium C++ project with and without -gsplit-dwarf. Compare link times and final binary sizes. Verify that GDB can still produce full backtraces with the .dwo files present but not when they are deleted.

  5. debuginfod client: Set DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org. Run gdb /usr/bin/bash core (generate a core with kill -ABRT $$ in bash). Observe GDB automatically fetching debug info from the remote server. Record the round-trip time.

References

  • DWARF Debugging Standard, Version 5. http://dwarfstd.org/dwarf5std.html
  • Poimboeuf, J. (2017). "ORC Unwinder." LWN.net. https://lwn.net/Articles/728339/
  • Coutant, C. (2012). "Split DWARF." GNU Binutils mailing list.
  • Eager, M. (2012). "Introduction to the DWARF Debugging Format." http://dwarfstd.org/doc/Debugging%20using%20DWARF-2012.pdf
  • elfutils/debuginfod documentation. https://sourceware.org/elfutils/Debuginfod.html
  • llvm-dwarfdump documentation. https://llvm.org/docs/CommandGuide/llvm-dwarfdump.html
  • Matz, M., et al. "System V ABI AMD64 Supplement." (defines .eh_frame usage)
  • GDB Internals: DWARF2 Reader. https://sourceware.org/gdb/wiki/Internals