Section 26: Security — Overview
Section Purpose and Scope
This section examines systems security from an engineering perspective: the kernel attack surface, hardware security mechanisms, mandatory access control frameworks, microarchitectural side channels, and supply chain threats. It builds a threat-model-driven understanding of what protections exist, what they actually enforce versus what they assume, and what attacker capabilities each mitigation defeats. This is not a checklist of CVEs — it is an architectural analysis of the trust boundaries in a modern Linux system and the layered defenses that protect them.
Prerequisites
- Section 03: Kernel Fundamentals (kernel/user separation, system calls)
- Section 06: CPU Architecture (rings, virtual memory, hardware security features)
- Section 07: Process Management (credentials, capabilities, /proc security)
- Section 11: Memory Management (virtual address space, ASLR, page permissions)
- Section 20: Containers (namespace isolation, seccomp, capabilities)
- Section 24: Debugging (exploit research requires debugging skills)
Learning Objectives
- Construct a threat model for a Linux system including kernel, hardware, and supply chain vectors.
- Explain ASLR, PIE, stack canaries, DEP/NX, and SMEP/SMAP as mitigations and their limitations.
- Describe KPTI and its relationship to Meltdown; explain why it exists.
- Explain CFI (forward-edge and backward-edge) and its relationship to ROP mitigations.
- Compare SELinux, AppArmor, and TOMOYO as MAC implementations.
- Describe Intel SGX and AMD SEV as confidential computing primitives.
- Articulate Spectre's threat model and why it remains fundamentally unmitigated in software.
- Analyze the Linux capability model and identify privilege escalation paths.
Architecture Overview
Defense-in-Depth Model (Linux x86-64):
┌─────────────────────────────────────────────────────────────────┐
│ Hardware Layer │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
│ │ SMEP/SMAP │ │ NX/XD bit │ │ CET (Shadow Stack + │ │
│ │ (no exec │ │ (no exec data│ │ IBT - Indirect Branch │ │
│ │ user pages │ │ pages) │ │ Tracking) │ │
│ │ in kernel) │ │ │ │ │ │
│ └──────────────┘ └──────────────┘ └────────────────────────┘ │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
│ │ IOMMU │ │ TPM │ │ Intel SGX / AMD SEV │ │
│ │ (DMA │ │ (measured │ │ (hardware-encrypted │ │
│ │ isolation) │ │ boot, PCR) │ │ enclaves/VMs) │ │
│ └──────────────┘ └──────────────┘ └────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ Kernel Layer │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
│ │ KPTI │ │ KASLR │ │ Kernel CFI │ │
│ │ (page table │ │ (randomize │ │ (Clang CFI for kernel, │ │
│ │ isolation │ │ kernel │ │ KCFI, FineIBT) │ │
│ │ vs Meltdown)│ │ base addr) │ │ │ │
│ └──────────────┘ └──────────────┘ └────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ LSM (Linux Security Modules) framework │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌────────────────┐ │ │
│ │ │ SELinux │ │ AppArmor │ │ TOMOYO │ │ │
│ │ │ (type │ │ (path- │ │ (path-based │ │ │
│ │ │ enforcement)│ │ based MAC) │ │ learning MAC) │ │ │
│ │ └─────────────┘ └─────────────┘ └────────────────┘ │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ seccomp-BPF │ │ Capabilities │ │
│ │ (syscall │ │ (fine-grained│ │
│ │ filter) │ │ privs) │ │
│ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ User Space Layer │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
│ │ ASLR + PIE │ │ Stack Canary │ │ RELRO + Full RELRO │ │
│ │ (randomize │ │ (__stack_chk │ │ (read-only PLT/GOT │ │
│ │ load addrs) │ │ _fail) │ │ after relocation) │ │
│ └──────────────┘ └──────────────┘ └────────────────────────┘ │
│ ┌──────────────┐ ┌────────────────────────────────────────┐ │
│ │ Fortify │ │ Namespaces (as security primitive) │ │
│ │ Source │ │ (user ns, pid ns, net ns isolation) │ │
│ └──────────────┘ └────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Spectre Attack Class (simplified):
┌────────────────────────────────────────────────────────────────┐
│ Attacker trains branch predictor with controlled inputs │
│ ↓ │
│ Victim code executes speculatively past bounds check │
│ (bounds check fails, but speculative access already happened) │
│ ↓ │
│ Speculative access brings secret data into cache │
│ ↓ │
│ Attacker measures cache timing to infer secret value │
│ │
│ Defense: retpoline (v1 partial), IBRS/STIBP (v2), │
│ ssbd (v4), index masking (v1), site-by-site lfence │
└────────────────────────────────────────────────────────────────┘
Key Concepts
- Threat Model: Structured analysis of: what assets need protection, who the adversaries are, what capabilities they have, what their goals are, and what mitigations address which threats. STRIDE, PASTA, attack trees are formalization methods.
- ASLR (Address Space Layout Randomization): Randomizes base addresses of stack, heap, and library mappings. Defeats hardcoded-address exploits. Entropy limited on 32-bit; partial info leaks can break it. Kernel: KASLR randomizes
_textbase. - PIE (Position-Independent Executable): Executable compiled to not assume a fixed load address. Required for ASLR to apply to the main binary. Compiled with
-fPIE -pie. - Stack Canary: Random value placed between local variables and saved return address. Checked on function return. Defeats sequential stack overflows. Does not defeat heap overflows or attacks that skip the canary.
- DEP/NX (Data Execution Prevention / No-Execute): CPU hardware bit marks pages as non-executable. Data pages cannot be executed as code. Defeats shellcode injection. Defeated by ROP.
- SMEP (Supervisor Mode Execution Prevention): CPU prevents kernel from executing code on user-space pages. Defeats ret2usr attacks. Controlled by CR4.SMEP bit.
- SMAP (Supervisor Mode Access Prevention): CPU prevents kernel from reading/writing user-space memory without explicit
stac/clacinstructions. Defeats kernel code that reads user-supplied pointers inadvertently. - KPTI (Kernel Page Table Isolation): Maintains two sets of page tables: one for user mode (minimal kernel mappings), one for kernel mode. Mitigates Meltdown — user-space speculatively cannot read kernel memory because kernel mappings not present in user page tables. Costs ~5-30% syscall overhead.
- CFI (Control Flow Integrity): Restricts indirect branches/calls to a valid set of targets. Forward-edge CFI (restrict indirect calls to functions with matching type signatures). Backward-edge CFI (Intel CET Shadow Stack — hardware-maintained second stack of return addresses).
- Retpoline: Compiler transformation that replaces indirect branches with a
call/pause/lfencesequence to prevent speculative execution to attacker-controlled targets. Mitigates Spectre v2. - SELinux (Security-Enhanced Linux): MAC framework with type enforcement. Every process and file labeled. Policy defines allowed
{process_type, file_type, permission}triples. Mandatory: root cannot override. Developed by NSA. - AppArmor: Path-based MAC framework. Simpler policy language than SELinux. Profiles restrict what files, capabilities, and network operations a program can use. Default on Ubuntu/Debian.
- seccomp: Syscall filter. SECCOMP_MODE_FILTER allows BPF programs to allow/deny/trap syscalls. Default Docker profile is a practical reference.
- TPM (Trusted Platform Module): Hardware security chip. Stores cryptographic keys, performs measured boot (hashes each boot stage into PCRs). PCR values can gate key release — disk encryption key only released if boot measurements match expected values.
- Intel SGX (Software Guard Extensions): Hardware-isolated enclave in application address space. Memory encrypted by CPU — OS and hypervisor cannot read enclave memory. Compromised by microarchitectural side channels (SGAxe, Plundervolt).
- AMD SEV (Secure Encrypted Virtualization): VM memory encrypted per-VM key, managed by AMD SP. Hypervisor cannot read VM memory. SEV-SNP adds integrity protection and attestation.
- Spectre / Meltdown: Microarchitectural side channel attacks leveraging speculative execution and cache timing. Meltdown (CVE-2017-5754): read kernel memory from user space via speculative execution; mitigated by KPTI. Spectre (CVE-2017-5753, 5715): cross-process/cross-privilege info leakage via branch predictor poisoning; fundamentally architectural.
Major Historical Milestones
| Year | Event |
|---|---|
| 1988 | Morris Worm exploits first public stack overflow vulnerability |
| 1996 | Aleph One "Smashing the Stack for Fun and Profit" — formalizes stack exploitation |
| 1998 | StackGuard (stack canaries) introduced |
| 2001 | PaX project implements ASLR and non-executable pages on Linux |
| 2004 | NX bit support in x86 hardware (AMD); ASLR merged into Linux 2.6.12 (2005) |
| 2005 | SELinux merged into Linux 2.6.0 (2003); widely deployed by 2005 |
| 2007 | Return-Oriented Programming (ROP) formalized by Shacham |
| 2008 | AppArmor merged into Linux 2.6.36 (2010) |
| 2012 | SMEP support in Intel Ivy Bridge CPUs; kernel exploitation significantly harder |
| 2015 | Intel SGX launched in Skylake CPUs |
| 2016 | Project Zero documents SMAP bypass techniques; ARM MTE development begins |
| 2018 | Meltdown and Spectre disclosed (Jan 3); KPTI emergency patches |
| 2018 | AMD SEV first generation available |
| 2019 | RIDL, Fallout, ZombieLoad — MDS attacks; microcode mitigations |
| 2020 | CET (Shadow Stack) in Tiger Lake CPUs; Android adopts CFI broadly |
| 2021 | AMD SEV-SNP for VM integrity; supply chain attacks (SolarWinds, xz-utils) |
| 2022 | KCFI (Kernel CFI with Clang) merged; FineIBT for kernel indirect branches |
| 2023 | Downfall (GDS), Inception (Spectre variant) — new microarchitectural vulnerabilities |
Modern Relevance
The Linux security model is a layered system where each mechanism compensates for the limits of others. No single protection is sufficient; the goal is to raise attacker cost at each layer. The microarchitectural side-channel era (post-Spectre) has fundamentally changed security assumptions: code that doesn't execute can still leak data through speculative paths and cache timing.
Supply chain security has emerged as an equal or greater threat vector than kernel exploitation in many environments. The xz-utils backdoor (2024) demonstrated that sophisticated nation-state actors target the software supply chain. SLSA, sigstore, and reproducible builds are the infrastructure response.
Confidential computing (SGX/SEV/TDX) is becoming a requirement for regulated workloads in cloud environments where even the cloud provider is untrusted. The attestation infrastructure to verify enclave/VM identity is still maturing.
File Map
26-security/
├── 00-overview.md ← this file
├── 01-threat-models.md ← STRIDE, attack trees, trust boundaries
├── 02-privilege-escalation.md ← Linux privilege model, escalation paths
├── 03-kernel-attack-surface.md ← syscall surface, driver vulnerabilities, /proc
├── 04-memory-safety.md ← buffer overflows, use-after-free, type confusion
├── 05-aslr-pie.md ← ASLR internals, entropy, info leak mitigations
├── 06-stack-canaries.md ← canary placement, implementation, bypass classes
├── 07-dep-nx.md ← NX hardware bit, W^X policy, ROP as response
├── 08-smep-smap.md ← CR4 bits, kernel enforcement, bypass history
├── 09-kpti.md ← page table isolation, Meltdown connection, cost
├── 10-cfi.md ← forward/backward edge, CET shadow stack, KCFI
├── 11-seccomp.md ← BPF filter design, seccomp-notify, Docker profile
├── 12-selinux.md ← type enforcement, policy, audit2allow workflow
├── 13-apparmor-tomoyo.md ← path-based MAC, profile generation
├── 14-capabilities.md ← capability taxonomy, dropping, ambient caps
├── 15-namespaces-security.md ← user namespace risks, unprivileged container issues
├── 16-tpm-tee.md ← TPM measured boot, PCR sealing, IMA/EVM
├── 17-intel-sgx.md ← enclave model, attestation, side channels
├── 18-amd-sev.md ← SEV/SEV-SNP, VM encryption, attestation
├── 19-supply-chain.md ← SLSA, sigstore, reproducible builds, sbom
└── 20-spectre-meltdown.md ← microarchitectural attacks, mitigation landscape
Cross-References
- Section 06 (CPU Architecture): Hardware security features (SMEP/SMAP/NX/CET) require CPU architecture knowledge
- Section 11 (Memory Management): Virtual address space, page tables — ASLR and KPTI operate here
- Section 20 (Containers): Seccomp, capabilities, user namespaces in container security context
- Section 24 (Debugging): Sanitizers (ASAN, KASAN) find the vulnerabilities that exploits target
- Section 27 (Kernel Exploits): Attack techniques that the defenses in this section counter
- Section 28 (Reliability Engineering): Security incidents require same incident response process as reliability failures
- Section 43 (Formal Verification): Hardware formal verification for security properties; seL4 verified OS kernel