Skip to content

05 — Exploit Mitigations and Their Bypasses

Overview

Modern kernel and application exploit mitigations represent a decades-long arms race between defenders and attackers. Each mitigation raises the cost of exploitation; each bypass technique restores attacker capability. Understanding this cat-and-mouse history is essential for security engineers who need to reason about actual residual risk — not just "mitigation X is deployed" but "what does an attacker need to do to bypass X, and how hard is that?"

This document traces each major mitigation, its intended protection, the bypass techniques that were developed, and the subsequent counter-measures that made bypasses harder. The story is one of stacked defenses where no single mitigation is sufficient but the combination significantly raises the cost of reliable exploitation.

Prerequisites

  • x86-64 memory model: stack, heap, code segments
  • Return-oriented programming (ROP) concepts
  • Kernel virtual memory layout (see 11-memory-management/)
  • Basic understanding of kernel exploit classes (see 27-kernel-exploits/01-kernel-exploit-classes.md)
  • Familiarity with glibc and ELF binary structure

Mitigation Timeline

Year  Mitigation              Level      Primary Target
----  ----------------------  ---------  ---------------------------
1998  StackGuard (canaries)   Compiler   Stack buffer overflow ret overwrite
2001  Non-Executable stack    OS/HW      Direct shellcode on stack
2003  Stack canaries in GCC   Compiler   Stack buffer overflow (widespread)
2004  PaX ASLR (kernel patch) Kernel     Fixed-address exploitation
2004  OpenBSD W^X             OS         Combined execute + write
2005  Windows DEP (HW NX)     OS/HW      Shellcode injection
2007  Windows Vista ASLR      OS         Fixed-address exploitation
2009  SMEP (Supervisor Mode   HW         Kernel ret2user attacks
       Execute Prevention)
2011  SMAP (Supervisor Mode   HW         Kernel access to user memory
       Access Prevention)
2012  Linux ASLR improvements OS         Predictable ASLR entropy
2014  Linux KASLR             Kernel     Fixed kernel virtual addresses
2016  Linux SMEP/SMAP enforce Kernel     Userspace gadgets from kernel
2017  CFI (Shadow Call Stack  Compiler   ROP / control flow hijack
       & type-based CFI)
2017  Retpoline               Compiler   Spectre variant 2
2020  KCFI (Kernel CFI)       Kernel     Kernel ROP/function ptr hijack
2022  Intel CET (IBT + SS)    HW         ROP + indirect call targets
2023  Linux KCFI upstream     Kernel     Kernel indirect call hijack

Stack Canaries

The Mitigation

Stack canaries (StackGuard, 1998; GCC -fstack-protector, 2003) place a random value (the canary) on the stack between local variables and the saved return address. On function return, the canary is checked against the stored value; if it has been modified, the program terminates with __stack_chk_fail.

Stack frame layout WITH canary:
+-------------------+
| Return Address    |  <- target of stack buffer overflow
+-------------------+
| Canary Value      |  <- random, checked on return
+-------------------+
| Saved RBP         |
+-------------------+
| Local variables   |
|   buf[64]         |  <- buffer being overflowed
+-------------------+
| ...               |

Bypass 1: Information Leak to Read the Canary

The canary is only effective if the attacker cannot learn its value before overwriting. Information leak vulnerabilities (format string bugs, out-of-bounds reads, heartbleed-style over-reads) can disclose the canary value, allowing the attacker to write it correctly during overflow:

Attack flow:
1. Use format string vuln: printf(user_input) leaks stack values
   -> Read canary value from stack
2. Use buffer overflow: overwrite buffer + correct canary + fake return address
3. Stack check passes (correct canary written), code hijacked

/proc/PID/mem on Linux allows a process to read its own memory (or a debugger to read another process's memory). On a system where an attacker can read arbitrary process memory, canary bypass is trivial.

Bypass 2: Overwrite Vtable, Not Return Address

Function pointer overflows and heap overflows can corrupt C++ vtable pointers without touching the stack canary at all. The canary only protects the return address on the stack frame where the canary lives; heap allocations have no canary.

Counter-measure

Per-thread canaries (modern Linux libc randomizes per-thread) and position-independent canaries reduce predictability. Shadow stacks (see below) protect return addresses independently of canary value.

ASLR: Address Space Layout Randomization

The Mitigation

ASLR randomizes the base addresses of the stack, heap, and shared libraries (and in kernel, the kernel itself via KASLR) on each process execution. An attacker cannot hardcode addresses because they change each run.

Without ASLR:
  libc always at: 0xb7e00000
  stack always at: 0xbffff000
  heap always at: 0x08048000+

With ASLR (32-bit, ~16 bits entropy for stack):
  run 1: libc=0xb6f23000, stack=0xbf34a000
  run 2: libc=0xb7a11000, stack=0xbfce0000
  run 3: ...

Bypass 1: 32-bit Brute Force

32-bit ASLR entropy is approximately 16 bits for the stack and library randomization (due to alignment constraints). An attacker can simply retry the exploit ~65,536 times. A local attacker can fork-and-retry thousands of times per second. For a network service that crashes but restarts, automated brute-forcing across thousands of requests is practical:

P(success per try) = 1/65536
Expected tries = 65536
At 1000 tries/sec = ~65 seconds to compromise

64-bit ASLR has significantly higher entropy (28-48 bits depending on configuration), making brute force infeasible.

Bypass 2: Information Leak (Most Common)

Nearly all modern ASLR bypasses use a memory disclosure vulnerability to leak an address, compute the base from the known offset, then use the disclosed address in the second stage:

Stage 1: Trigger info leak
  - Format string: "%p %p %p %p" leaks stack/libc pointer
  - Heap overflow into adjacent string buffer that gets printed
  - UAF reading freed memory that was previously a pointer
  - Kernel: /proc/kallsyms readable (pre-hardening)
  - Kernel: kernel pointer in dmesg (pre-hardening)
  - Kernel: timer_list in /proc/timer_list leaked function pointers

Stage 2: Compute base
  leaked_ptr - known_offset = base_address

Stage 3: Exploit with computed addresses
  ROP chain uses computed addresses

Bypass 3: Heap Spraying

Spray the heap with many copies of a NOP sled + shellcode. If ASLR entropy is low enough that guessing a heap address is within brute-force range, or if the attacker can trigger many allocations to fill the address space, they can jump to a likely address. More effective against JavaScript JIT heap sprays where the attacker controls large allocations.

Counter-measure: KASLR

Kernel ASLR (Linux 3.14+, Windows Vista+) randomizes the kernel's virtual address. The kernel is no longer at a predictable address. Bypasses require a kernel address leak — which was historically easy:

  • /proc/kallsyms: Before hardening, readable by any user. Lists all kernel symbol addresses. Now requires CAP_SYSLOG or kptr_restrict=0.
  • dmesg: Kernel log often contains pointer-valued addresses. dmesg_restrict=1 and kptr_restrict=2 hide these.
  • Side channels: Speculative execution (Spectre) can leak kernel addresses through cache timing without requiring readable files.
  • Timing attacks on slab allocators: Allocation timing differences reveal heap layout information.

NX / DEP: Non-Executable Memory

The Mitigation

Hardware NX (No eXecute) bit on x86-64 marks memory pages as non-executable. The CPU raises a fault if the instruction pointer enters an NX page. Applied to stack, heap, and data segments — all writeable memory is non-executable.

Bypass: Return-to-libc and ROP

Since data pages cannot be executed, attackers chain existing executable code. Return-to-libc (2000): overwrite return address with system() address, place /bin/sh as argument. The attacker executes existing library code, not injected shellcode.

Return-Oriented Programming (ROP, Shacham 2007) generalizes this: chain sequences of existing instructions ending in ret (called "gadgets") to build arbitrary computation using only existing code in the binary and loaded libraries.

ROP chain on stack:
+------------------+
| gadget3_addr     |  <- ret pops this, jumps to gadget3
+------------------+
| gadget3_arg      |
+------------------+
| gadget2_addr     |  <- ret pops this, jumps to gadget2
+------------------+
| gadget2_arg      |
+------------------+
| gadget1_addr     |  <- overwritten return address
+------------------+
| ...local vars... |

Gadget example:
  0x401234: pop rdi ; ret   <- set first argument register
  0x401238: ...

ROP chains are Turing-complete: any computation can be expressed as a sequence of gadgets. Defeating NX entirely for exploitation purposes.

SMEP: Supervisor Mode Execute Prevention

The Mitigation

SMEP (Intel 2011, AMD 2012; Linux kernel ~3.x) prevents the kernel (ring 0) from executing code in userspace pages. The CR4.SMEP bit, when set, causes a fault if the instruction pointer points to a user-space address while CPL=0.

Before SMEP: the classic "ret2user" attack: write shellcode in userspace memory, use a kernel vulnerability to redirect kernel execution to userspace — easy because kernel and user share the virtual address space.

ret2user attack (pre-SMEP):
  1. Place shellcode at known user virtual address (e.g., mmap(0x10000, ...))
  2. Exploit kernel bug to overwrite a kernel function pointer
  3. Kernel function pointer now points to 0x10000 (user space)
  4. Kernel executes attacker's shellcode in ring 0
  5. commit_creds(prepare_kernel_cred(NULL)) -> root

Bypass: Kernel-Only ROP

With SMEP, attackers must build their entire ROP chain from kernel-space gadgets only. Userspace addresses are unusable. The gadget search space is still large (the kernel is large), but requires kernel address knowledge (KASLR bypass) and significantly more sophisticated gadget chains.

Bypass: Disable SMEP via CR4 Modification

A privileged kernel-mode gadget that writes to CR4 can clear the SMEP bit:

ROP gadget: mov cr4, rax ; ret
Payload: set rax to CR4 value with bit 20 (SMEP) cleared
Result: SMEP disabled, ret2user works again

Counter-measure: Linux pins CR4 bits (since ~4.x). The kernel sets "pinned" bits that cannot be cleared by ROP gadgets. Writing an invalid CR4 value triggers a fault.

SMAP: Supervisor Mode Access Prevention

The Mitigation

SMAP (Intel 2014, AMD 2014) prevents the kernel from reading or writing user-space memory unless the EFLAGS.AC (Alignment Check) flag is set. The kernel must explicitly set AC before accessing user memory (stac instruction) and clear it afterward (clac). This prevents kernel exploit code from using user-controlled data directly.

Before SMAP: an attacker could place a fake kernel structure in userspace, then trick a kernel code path into using that structure — the kernel would read/write from the user-provided address.

Bypass: copy_from_user Gadget

The kernel legitimately accesses user memory via copy_from_user(), which sets EFLAGS.AC before the access and clears it after. A kernel exploit that can redirect execution into the copy_from_user path while controlling its arguments achieves user-to-kernel copy with SMAP enabled.

copy_from_user attack with SMAP:
  1. Exploit kernel bug — redirect execution
  2. Jump into copy_from_user code (between stac and clac)
  3. Control the src argument to copy_from_user
  4. Kernel copies attacker's data into kernel buffer
  5. Attacker achieves desired kernel data write

This requires a kernel information leak (to find copy_from_user's address), kernel code execution, and careful timing. Much harder than pre-SMAP exploitation.

CFI: Control Flow Integrity

The Mitigation

CFI restricts valid targets for indirect calls (function pointers, virtual dispatch) and returns (shadow stacks). Two sub-types:

Forward-edge CFI (KCFI, Linux 6.1+): Each indirect call site has a type signature. Before the call instruction, a check verifies the callee's signature matches the expected type. A mismatched function pointer (e.g., an attacker replacing a callback pointer with a gadget address) is rejected.

Without CFI:
  call [rax]   <- rax can point anywhere

With KCFI:
  movabs rcx, <expected_hash>  ; load expected type hash
  cmpl   -4(%rax), %ecx        ; compare with callee's type annotation
  jne    fail                  ; trap if wrong type
  call   *%rax                 ; proceed

Backward-edge CFI (Shadow Stack / Intel CET): Return addresses are stored in a separate shadow stack that is not writable by normal code. On function return, the CPU compares the stack return address against the shadow stack; mismatch causes a fault. Defeats stack smashing (return address corruption) and ROP chains (which corrupt the stack).

Bypass: Type-Compatible Gadgets

KCFI checks that the callee has the correct type signature — but "correct type" is determined by the function prototype. An attacker can search for type-compatible gadgets: kernel functions whose type signature matches what the exploit site expects, even if they are not the intended callee.

Attack site expects: void (*fn)(struct request *req)
Attacker needs: code execution with control of one pointer arg

Search for all kernel functions with signature:
  void f(struct request *)

Find one that:
  - Is reachable (exported or known address)
  - Has useful side effects (e.g., writes to attacker-controlled field)
  - Or has additional type confusion potential

The attack surface depends on type granularity. Coarse-grained CFI (all functions with same arity) is easily bypassed. Fine-grained CFI (hash of full parameter types) is much harder.

Counter-measure: Fine-Grained CFI + Shadow Stack

Fine-grained CFI (type hash that includes parameter types, return type, calling convention) dramatically reduces valid gadget sets. Combined with a hardware shadow stack (Intel CET, available in hardware since 2020, Linux support landing in 6.x) for return addresses, defeating both forward-edge and backward-edge CFI simultaneously becomes extremely difficult.

KASLR Bypasses: Side Channels

Speculative execution vulnerabilities (disclosed January 2018) fundamentally undermined KASLR:

Spectre Variant 2 (branch target injection): The CPU's indirect branch predictor can be trained to speculatively execute attacker-chosen code. By training the predictor from userspace to predict a gadget in the kernel, attacker-controlled speculative execution in kernel context leaks kernel memory via cache side channels, disclosing kernel virtual addresses.

Meltdown (CVE-2017-5754): On unpatched hardware, the kernel's virtual address space is mapped into the process's page tables. Speculative execution briefly accesses kernel memory (before the permission check raises an exception). The access leaves cache residue — a side channel that discloses kernel memory content including addresses.

Counter-measure: KPTI (Kernel Page Table Isolation) — the kernel removes its own mappings from the page table when executing in userspace. Only a minimal trampoline is mapped. This defeats Meltdown but has 5–30% overhead on syscall-heavy workloads (system call requires CR3 switch).

Mitigation Stack Summary

Mitigation    Protects Against         Bypass                  Counter
-----------   ----------------------   -------------------     ---------
Canary        Stack ret addr overwrite  Info leak reads it      Shadow stack
NX/DEP        Shellcode exec            ROP chains              CFI
ASLR          Fixed addresses           Info leak + compute     More entropy
KASLR         Fixed kernel addresses    Side channels, leaks    KPTI, kptr_restrict
SMEP          ret2user                  Kernel-only ROP         CR4 pinning
SMAP          Fake kernel structs        copy_from_user gadget   Careful coding
CFI (fwd)     Indirect call hijack      Type-compatible gadgets Fine-grained types
Shadow Stack  ROP / ret addr overwrite  (active research)       Hardware CET
KPTI          Meltdown side channel     Spectre variants        Microcode

Security Implications

No single mitigation is sufficient. The point of defense-in-depth is to require an attacker to chain multiple bypasses, each of which requires a specific primitive (info leak, controlled write, gadget chain). A well-hardened system requires:

  1. An initial memory safety violation (UAF, buffer overflow, type confusion)
  2. An information leak to defeat ASLR/KASLR
  3. A write primitive to overwrite a target
  4. SMEP/CFI bypass techniques
  5. Privilege escalation primitive (commit_creds call or equivalent)

Each step can fail, be detected, or be blocked by a mitigation. The combination makes reliable exploitation difficult and expensive.

Performance Implications

Mitigations have measurable costs:

Mitigation Overhead (typical)
Stack canary < 1% (one compare per return)
ASLR < 0.1% (setup cost only)
KPTI 5–30% (syscall-heavy loads)
Retpoline 2–15% (branch-heavy kernel)
CFI checks 1–3% (indirect calls)
Shadow stack 1–3% (HW CET) / 10–20% SW
seccomp BPF ~1% (per syscall filter)

The combination of KPTI + retpoline post-Spectre/Meltdown caused significant controversy because the overhead was paid by all users and workloads to mitigate vulnerabilities that primarily affected cloud multi-tenant environments.

Failure Modes

  • Incomplete ASLR: Statically-linked binaries or non-PIE executables have fixed text segment addresses even with ASLR. The attacker uses the fixed binary as a gadget source even when libraries are randomized.
  • CFI with too-coarse granularity: Early CFI implementations checked only that a function pointer pointed to "the start of any function" — this blocked many attacks but left significant gadget space. Fine-grained CFI is required for real protection.
  • KPTI performance regression without necessity: Many workloads that are not multi-tenant cloud VMs pay the KPTI overhead unnecessarily. The IBRS_ALL microcode feature (partial mitigation) allows KPTI to be disabled on systems where Meltdown is not a threat model.

Exercises

  1. Compile a simple C program with and without -fstack-protector-all. Use GDB to examine the stack frame in both cases, identifying the canary value. Overflow the buffer exactly to the canary, read its value, then perform a controlled overflow that preserves the canary.
  2. Write a 32-bit ROP chain (no ASLR, use ldd to find libc base): call system("/bin/sh") by chaining a pop rdi; ret gadget (to set the argument) with system function address. Use ROPgadget tool to find gadgets.
  3. Enable KASLR and KPTR_RESTRICT on a Linux VM. Attempt to read kernel addresses from /proc/kallsyms, dmesg, and /proc/timer_list as both root and a regular user. Document which sources leak at which privilege level.
  4. Use checksec on 5 programs on your system to inventory which protections each has (PIE, canary, NX, RELRO). Find at least one program that is missing a protection and explain why it might be intentionally excluded.
  5. Study the Spectre v1 proof-of-concept. Implement the classic "bounds check bypass" variant that reads beyond an array boundary via speculative execution and observe cache timing differences. Verify the mitigation (array_index_nospec in kernel) blocks the side channel.
  6. Research one real-world kernel CVE from 2022–2024 that required bypassing at least two of: KASLR, SMEP, CFI. Document: (a) the initial vulnerability class, (b) how each mitigation was bypassed, (c) which mitigation, if properly configured, would have blocked the exploit.

References

  • Shacham, H. (2007). "The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls." ACM CCS 2007.
  • Solar Designer. (1997). "Getting around non-executable stack (and fix)." Bugtraq.
  • Abadi, M. et al. (2005). "Control-Flow Integrity." ACM CCS 2005.
  • Kocher, P. et al. (2019). "Spectre Attacks: Exploiting Speculative Execution." IEEE S&P 2019.
  • Lipp, M. et al. (2018). "Meltdown." USENIX Security 2018.
  • PaX Team. "PaX address space layout randomization." https://pax.grsecurity.net/
  • Corbet, J. "Linux KASLR: https://lwn.net/Articles/569635/
  • Intel CET specification: https://software.intel.com/content/www/us/en/develop/articles/technical-look-control-flow-enforcement-technology.html
  • Project Zero: "In-the-wild kernel exploits 2022." https://googleprojectzero.blogspot.com/
  • Grsecurity / PaX project (historical bypasses and mitigations): https://grsecurity.net/
  • Saelo (Samuel Groß). Various macOS/XNU exploit writeups. https://github.com/saelo/