Skip to content

01 — Exploit Mitigations

Technical Overview

Exploit mitigations are hardware and software mechanisms that raise the cost of converting a vulnerability into a working exploit. No mitigation is absolute—every mitigation has been bypassed—but the defense-in-depth principle holds: requiring an attacker to chain multiple bypass techniques dramatically increases the skill barrier and reduces the number of capable attackers.

The modern mitigation stack is the product of a three-decade arms race. Each major class of exploit (stack overflows, heap overflows, code reuse attacks) spawned countermeasures that spawned new exploit classes. Understanding the design rationale, implementation details, and bypass history of each mitigation is essential for kernel security engineering.

Prerequisites

  • x86-64 assembly and calling conventions.
  • ELF binary format (sections, segments, relocations).
  • Linux virtual memory layout.
  • GCC/Clang compilation pipeline.
  • Basic exploit development concepts (stack overflow, ROP).

Core Content

Defense-in-Depth Architecture

No single mitigation stops a skilled attacker. The goal is layered defense:

Application logic (input validation)
    │
    ▼
FORTIFY_SOURCE (detect buffer overflows in standard library calls)
    │
    ▼
Stack Canaries (detect stack buffer overflow before return)
    │
    ▼
ASLR + PIE (randomize addresses, make hardcoded ROP gadgets fail)
    │
    ▼
NX/DEP (non-executable stack/heap, prevent shellcode execution)
    │
    ▼
RELRO + BIND_NOW (make GOT read-only, prevent GOT overwrite)
    │
    ▼
CFI (Control Flow Integrity, prevent arbitrary ROP)
    │
    ▼
SMEP/SMAP (prevent kernel from executing/reading user memory)
    │
    ▼
KPTI (separate kernel and user page tables, mitigate Meltdown)

Mitigation Timeline Diagram

Year  Mitigation               Where Deployed
────  ─────────────────────────────────────────────────────────
1998  StackGuard (stack canary) Linux (ported to GCC)
2003  PIE (Position Independent Exe) GCC -fPIE
2004  NX / DEP                 Intel EM64T, AMD64 (hardware NX bit)
2005  ASLR                     Linux kernel 2.6.12
2006  RELRO (read-only GOT)    GNU binutils, GCC -Wl,-z,relro
2009  ASLR for x86-64 (64-bit) Linux 2.6.25 (stronger randomization)
2012  SMEP                     Intel Ivy Bridge (CR4.SMEP bit)
2014  SMAP                     Intel Broadwell (CR4.SMAP bit)
2015  CFI (Control Flow Integrity) Clang (coarse-grained CFI)
2018  KPTI (Kernel Page Table Isolation) Linux 4.15 (Meltdown mitigation)
2020  Shadow Stack (CET)       Intel Tiger Lake (hardware shadow stack)
2021  Fine-grained CFI (KCFI)  Linux 6.1 (kernel CFI)

Stack Canaries (GCC Stack Smashing Protector — SSP)

Mechanism: GCC inserts a random "canary" value between local variables and the saved return address. Before the function returns, it checks that the canary is unchanged. If it has changed (indicating a stack buffer overflow that overwrote it), the program calls __stack_chk_fail() and aborts.

On x86-64 Linux, the canary value is stored in Thread Control Block at fs:[0x28]:

// Compiler-generated prologue (conceptually):
void vulnerable(char *input) {
    uint64_t canary = *(uint64_t *)fs_base[0x28]; // read from TCB
    char buf[64];
    // ... function body ...
    if (*(uint64_t *)fs_base[0x28] != canary)
        __stack_chk_fail();  // abort
}

x86-64 assembly:

// Prologue
mov    rax, QWORD PTR fs:0x28
mov    QWORD PTR [rbp-0x8], rax  ; save canary
xor    eax, eax

// Epilogue
mov    rax, QWORD PTR [rbp-0x8]  ; reload canary
xor    rax, QWORD PTR fs:0x28    ; compare
jne    __stack_chk_fail           ; abort if different

Types of canaries: - Terminator canary: 0x00, 0x0a, 0x0d, 0xff — null bytes and newlines stop many string copy operations. - Random canary: random 64-bit value from /dev/urandom at program start; stored in fs:[0x28]. - Random XOR canary: random value XORed with the return address — further increases entropy.

Bypass methods: - Format string leak: use printf("%lx %lx...") to leak the canary value from the stack. - Adjacent variable overwrite: overflow into a pointer variable before the canary to achieve arbitrary write without touching the canary. - Brute force on 32-bit: only 256 values for the 2nd byte (first byte is always null); feasible via forking processes.

Enable (default on most distributions):

gcc -fstack-protector-strong -o prog prog.c  # protect functions with arrays > 8 bytes
gcc -fstack-protector-all    -o prog prog.c  # protect all functions

ASLR (Address Space Layout Randomization)

Mechanism: randomize the base addresses of: - Stack: randomized each execution. - Heap (mmap): randomized each execution. - Shared libraries: randomized each execution. - Executable: randomized if compiled as PIE.

Linux ASLR entropy:

Address Space Component    64-bit entropy    32-bit entropy
─────────────────────────  ───────────────   ───────────────
Stack base                 ~30 bits          ~8 bits
mmap base (heap, libs)     ~28 bits          ~8 bits
Executable base (PIE)      ~28 bits          ~8 bits

On 64-bit systems, 28–30 bits = 2^28 = 268 million possible positions. Brute force is infeasible. On 32-bit systems, 2^8 = 256 positions—trivially brute-forceable in seconds by a forking process.

Configure:

# Check current ASLR setting
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled
# 1 = randomize stack, mmap, vDSO (not heap)
# 2 = full (stack, mmap, heap, vDSO) — recommended

sysctl -w kernel.randomize_va_space=2

PIE (Position Independent Executable): required for ASLR to randomize the executable itself. Without PIE, the text segment is at a fixed address, providing reliable ROP gadgets.

gcc -fPIE -pie -o prog prog.c
# Verify PIE:
readelf -h prog | grep "Type"
# DYN = PIE; EXEC = not PIE

ASLR bypass methods: - Information leak: use a format string bug or heap spray to leak an address, then compute gadget locations at runtime. - Heap spray: fill the heap with shellcode/ROP chains so that jumping to any of a large range of addresses hits the payload. - Brute force (32-bit): spawn child processes until one doesn't crash. - Return to vDSO: the vDSO is mapped at a consistent offset on some kernel versions; its system call trampolines serve as ROP gadgets.

NX / DEP (No-Execute / Data Execution Prevention)

Mechanism: the CPU's NX bit (bit 63 of a PTE) marks pages as non-executable. The kernel sets NX on data pages (stack, heap). Any attempt to execute code from these pages causes a page fault, which the kernel converts to SIGSEGV.

Hardware: AMD introduced the NX bit in AMD64 (2003); Intel added it to EM64T/x86-64 (2004). The EFER.NXE MSR must be set to enable NX enforcement.

Verify NX is enabled:

# Check CPU capability
grep nx /proc/cpuinfo

# Check binary NX settings
readelf -l prog | grep GNU_STACK
# RW  (without E flag) = stack is non-executable

Bypass methods: - Return-to-libc (ret2libc): instead of shellcode, return to system("/bin/sh") in libc. The code is in a legitimate executable page. - ROP (Return-Oriented Programming): chain "gadgets"—short instruction sequences ending in ret—from existing executable code. Turing-complete computation possible with sufficient gadgets.

RELRO (RELocation Read-Only)

Mechanism: The GOT (Global Offset Table) stores resolved dynamic library function addresses. An attacker who can write to the GOT can redirect calls to arbitrary addresses. RELRO makes the GOT read-only after the dynamic linker resolves all symbols.

Two levels: - Partial RELRO: reorder ELF sections to reduce risk; GOT .got.plt remains writable. - Full RELRO: force BIND_NOW (resolve all symbols at startup), then mprotect the entire GOT read-only.

# Enable full RELRO
gcc -Wl,-z,relro,-z,now -o prog prog.c

# Verify
readelf -d prog | grep BIND_NOW
checksec --file=prog  # shows RELRO status

Performance cost of full RELRO: all dynamic symbols resolved at startup. For programs with hundreds of dynamic symbol imports, this adds milliseconds to startup time. Long-running servers are unaffected.

SMEP (Supervisor Mode Execution Prevention)

Mechanism: prevents the CPU's ring 0 (kernel) from executing code from ring 3 (user space) pages. Enforced by CR4.SMEP bit (Intel Ivy Bridge, 2012).

Prevents: ret2user attacks where the kernel is tricked into executing shellcode placed in user memory.

// Example ret2user primitive (pre-SMEP):
// Attacker places shellcode in user space:
char shellcode[] = "\x31\xc0\x50\x68\x2f\x2f\x73\x68...";

// Kernel vulnerability redirects execution to user address:
(void(*)(void))(user_shellcode_addr)();
// Without SMEP: executes. With SMEP: #PF (fault) → panic/crash.

Check SMEP status:

cat /proc/cpuinfo | grep smep
# Or from kernel:
grep SMEP /boot/config-$(uname -r)

Bypass methods: - Native kernel gadgets: redirect to existing kernel code instead of user shellcode (ROP). - CR4 overwrite: if attacker can write to CR4 (requires other primitives), clear SMEP bit. Mitigated by PINNED_CR4 in Linux 5.3 (CR4 bits cannot be cleared once set).

SMAP (Supervisor Mode Access Prevention)

Mechanism: prevents ring 0 from reading or writing ring 3 memory except within explicit STAC/CLAC instruction windows. Enforced by CR4.SMAP bit (Intel Broadwell, 2014).

This prevents an entire class of kernel exploits that rely on the kernel reading attacker-controlled user memory as data (fake kernel structures, fake file operations structs, etc.).

The kernel uses copy_from_user() and copy_to_user() which internally bracket the access with STAC/CLAC:

// Kernel implementation of copy_from_user (conceptual):
static inline void stac(void) {
    asm volatile("stac" ::: "memory");  // Set AC flag: allow user access
}
static inline void clac(void) {
    asm volatile("clac" ::: "memory"); // Clear AC flag: block user access
}

long copy_from_user(void *to, const void __user *from, unsigned long n) {
    stac();
    // ... memcpy ...
    clac();
    return 0;
}

Bypass methods: - STAC-bracketed gadgets: if the attacker can pivot execution to a kernel function that is already inside a STAC/CLAC window, SMAP is bypassed. - Kernel data-only attacks: modify kernel data structures without executing user code (harder with SMEP/SMAP but still possible with write primitives).

FORTIFY_SOURCE

Compile-time and runtime checks for buffer overflows in standard C library functions.

gcc -D_FORTIFY_SOURCE=2 -O2 -o prog prog.c

Effects: replaces strcpy, sprintf, memcpy, etc. with checked versions that know the destination buffer size. If the copy would overflow, abort.

char buf[10];
strcpy(buf, "very long string");  // FORTIFY detects overflow at compile time or runtime

Intel CET (Control-flow Enforcement Technology): Shadow Stack

Intel CET (Tiger Lake, 2020) implements hardware shadow stack: a separate, protected stack that stores only return addresses. On function call, the hardware pushes the return address to both the normal stack and the shadow stack. On ret, the CPU verifies they match.

Normal stack:    [...local vars...][canary][rbp][ret_addr]
Shadow stack:    [ret_addr]  (hardware-protected, not accessible by ROP)

ROP attack:
  Overwrites ret_addr on normal stack → CET compares with shadow stack → MISMATCH → fault

Linux kernel CET support: CONFIG_X86_USER_SHADOW_STACK (Linux 6.6+).

Historical Context

Stack smashing protection was first described by Solar Designer (1997) and implemented in StackGuard by Crispin Cowan et al. (Usenix Security 1998). GCC integrated SSP as -fstack-protector in version 4.1 (2006).

PaX (patch for Linux, 2000) introduced ASLR, PAGEEXEC (non-executable pages before hardware NX), and mprotect() restrictions—years before mainline Linux adopted these features. PaX and its descendant Grsecurity remain more hardened than mainline Linux.

The NX bit and ASLR forced attackers to develop ROP (Return-Oriented Programming), first systematized by Hovav Shacham (CCS 2007). ROP chains together "gadgets" from existing executable code, bypassing NX. SMEP forced ROP to be kernel-only; KPTI protected against Meltdown. The arms race continues.

Production Examples

CVE-2016-5195 (Dirty COW): a race condition in copy-on-write handling in Linux's memory management allowed writing to read-only memory-mapped files, gaining root privilege. Stack canaries and ASLR did not protect against this data-corruption primitive. SELinux restricted the impact significantly—a system with tight SELinux policy could not be fully exploited even with the write primitive. Patched in Linux 4.8.3 (October 2016).

CVE-2017-1000112 (UDP MSG_MORE UAF): a use-after-free in the UDP socket path. Despite SMEP and SMAP, the exploit worked by manipulating kernel heap structures without executing user code (kernel data-only attack). Demonstrates that SMEP/SMAP alone are insufficient—they prevent specific attack primitives, not all exploits.

Debugging Notes

# checksec: audit binary mitigation coverage
apt install checksec
checksec --file=./mybinary
# Shows: RELRO, STACK CANARY, NX, PIE, RPATH, RUNPATH, Symbols, FORTIFY

# Verify ASLR is enabled
cat /proc/sys/kernel/randomize_va_space  # should be 2

# Check kernel config for all mitigations
grep -E "SMEP|SMAP|KPTI|RETPOLINE|CFI" /boot/config-$(uname -r)

# ldd randomization check (ASLR for libraries)
ldd /bin/ls; ldd /bin/ls  # addresses should differ between runs

Security Implications

Every mitigation has a bypass. The value is not in eliminating exploits but in: 1. Raising the skill bar (only sophisticated attackers can bypass multiple mitigations). 2. Buying time for patch deployment. 3. Converting memory-safety bugs into crashes (canaries, shadow stack) rather than silent exploits.

Organizations should treat mitigation coverage as a baseline hygiene metric: all production binaries should have ASLR+PIE+NX+canaries+full RELRO enabled.

Performance Implications

Mitigation Overhead
Stack canary ~1% (one load + comparison per function return)
ASLR < 0.1% (address randomization at exec)
PIE 1–5% (PIC code is slightly less efficient on x86-32; negligible on x86-64)
Full RELRO Startup time increase (symbol binding); runtime: 0%
SMEP/SMAP < 0.1% (CR4 check is hardware, not software)
KPTI 3–30% (syscall-intensive workloads, I/O heavy)
Shadow Stack (CET) ~2% (extra shadow stack write per call/ret)
FORTIFY_SOURCE < 1% (bounds checks at libc calls)

KPTI is the most expensive mitigation, particularly for workloads doing many small syscalls (Redis, Nginx with small requests). On Intel CPUs with PCID (Process Context IDs, Haswell+), KPTI overhead is halved.

Failure Modes and Real Incidents

Canary leak via format string (CVE-2012-0207): A format string vulnerability in a network daemon allowed printing the stack canary value. The attacker then used a second vulnerability to overflow the stack with the correct canary. Lesson: canaries are not sufficient—any information leak defeats them.

32-bit ASLR brute force (2014, nginx + PHP-FPM): Certain nginx/PHP configurations used fork() to serve requests, inheriting the parent's address space layout. The child's ASLR did not re-randomize. Combined with 32-bit address space (8 bits entropy), attackers brute-forced the stack address in < 1 second by sending thousands of requests. Fix: use execve instead of fork for security-critical context changes, or CLONE_VM | CLONE_VFORK alternatives.

Modern Usage

On modern Linux distributions, the default mitigation stack for user-space binaries is: - Fedora 38+: full RELRO, PIE, stack canary, FORTIFY_SOURCE=3, frame pointers. - Ubuntu 22.04+: partial RELRO (full optional), PIE, stack canary, FORTIFY_SOURCE=2. - Debian 12+: full RELRO, PIE, stack canary, FORTIFY_SOURCE=2.

Kernel mitigations via /proc/sys/kernel/: - kptr_restrict=2: hide kernel pointers. - dmesg_restrict=1: restrict dmesg to root. - perf_event_paranoid=3: block all perf events from unprivileged users.

Future Directions

  • Memory safety languages: Rust's ownership model eliminates entire classes of memory safety bugs that mitigations try to contain. The Linux kernel is adopting Rust (Rust for Linux, merged 6.1).
  • Hardware CET wide adoption: Intel CET shadow stack is now available on most new x86 CPUs; kernel and glibc support is maturing.
  • MTE (Memory Tagging Extension): ARM's hardware memory tagging (ARMv8.5-A) detects use-after-free and buffer overflows with ~1% overhead. Android 13+ uses MTE on Pixel 8 hardware.

Exercises

  1. Compile a stack-smashing vulnerable program (gets(buf)) without (-fno-stack-protector) and with (-fstack-protector-strong) stack canary protection. Attempt to overflow the buffer in both. Observe the difference in behavior (crash vs. abort with backtrace).

  2. Disable ASLR (echo 0 > /proc/sys/kernel/randomize_va_space), compile a PIE binary, and note the fixed load address. Re-enable ASLR and observe the randomized addresses across executions. Use ldd to compare shared library addresses.

  3. Examine a system binary with checksec. Find one binary that lacks a mitigation (perhaps a legacy or third-party tool). Document what exploitation primitive would be enabled by the missing mitigation.

  4. Write a simple ROP chain exploit against a NX+ASLR+PIE binary using a format string information leak to defeat ASLR, then use gadgets from a known library offset. Describe each step.

  5. Review the KPTI implementation in the Linux kernel (arch/x86/mm/kaiser.c or pti.c). Explain how the trampoline page works and why switching page tables on every syscall entry/exit is necessary.

References

  • Cowan, C. et al. "StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks." Usenix Security, 1998.
  • Shacham, H. "The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86)." CCS 2007.
  • Team, PaX. PaX Address Space Layout Randomization. https://pax.grsecurity.net/docs/aslr.txt (2001).
  • Gruss, D. et al. "KASLR is Dead: Long Live KASLR." ESSoS 2017.
  • Intel CET specification: https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf
  • checksec.sh: https://github.com/slimm609/checksec.sh
  • Linux kernel self-protection project: https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project