Skip to content

01 — Virtualization Fundamentals

Prerequisites

  • Operating system concepts: privilege rings, system calls, interrupts, page tables
  • CPU architecture: instruction sets, privilege levels (Ring 0 / Ring 3), control registers (CR0, CR3, CR4)
  • Memory management: virtual addresses, physical addresses, page fault mechanics
  • Basic understanding of I/O: port I/O, MMIO, DMA

Historical Context

Virtualization is older than the personal computer. IBM's Cambridge Scientific Center built the CP-40 in 1967 — an experimental time-sharing system that ran multiple isolated "virtual machines" on a single IBM System/360 mainframe. Each user saw a complete logical copy of the machine. The follow-on CP-67 (1967) was the first commercially deployed VM system. IBM formalized the concept in VM/370 (1972), which shipped with CMS (Conversational Monitor System) as the single-user guest OS that ran atop the hypervisor.

The academic foundations were laid by Gerald Popek and Robert Goldberg in their 1974 paper "Formal Requirements for Virtualizable Third Generation Architectures" (CACM 1974). Their theorem defined the precise mathematical conditions under which an ISA can be efficiently virtualized — work that remained purely theoretical for x86 until hardware vendors finally implemented it in 2005–2006.

The x86 platform was famously not virtualizable under the Popek–Goldberg definition. VMware (founded 1998) solved this with binary translation, shipping the first x86 hypervisor in VMware Workstation 1.0 (1999). Xen followed in 2003 from the University of Cambridge with a paravirtualization approach. The Linux kernel gained KVM in 2007 after Avi Kivity's patch was merged, turning any Linux host into a hypervisor by exploiting newly available Intel VT-x hardware extensions.

The Popek–Goldberg Theorem (1974)

Popek and Goldberg classified all instructions of an ISA into three categories:

  • Privileged instructions: trap to the OS when executed in user mode; execute natively in kernel mode
  • Sensitive instructions: either (a) control-sensitive — affect machine configuration (e.g., load CR0, modify interrupt flag) or (b) behavior-sensitive — behave differently depending on privilege level (e.g., POPF on x86 silently drops IF bit in user mode instead of trapping)
  • Innocuous instructions: all other instructions; safe to run at any privilege level

The Theorem

A virtual machine monitor (VMM) can be constructed if the set of sensitive instructions is a subset of the set of privileged instructions.

In plain terms: every instruction that could affect the hypervisor's control must trap when a guest tries to execute it, so the hypervisor can intercept and emulate it safely.

    Ideal (Virtualizable) ISA:
    +------------------------------------+
    |         All Instructions           |
    |  +----------------------------+    |
    |  |  Privileged Instructions   |    |
    |  |  +----------------------+  |    |
    |  |  | Sensitive Instructions|  |    |
    |  |  +----------------------+  |    |
    |  +----------------------------+    |
    +------------------------------------+

    Sensitive ⊆ Privileged  →  VMM possible

    x86 pre-VT-x (NOT virtualizable):
    +------------------------------------+
    |  Privileged   |  Sensitive-but-    |
    |  Instructions |  NOT-privileged    |
    |               |  (e.g. POPF, SGDT)|
    +------------------------------------+
    Sensitive ⊄ Privileged  →  VMM requires tricks

Why x86 Violated the Theorem

x86 had 17 "sensitive but non-privileged" instructions — instructions that behave differently depending on privilege level but do not trap when executed in Ring 3. Classic examples:

  • POPF / PUSHF: reading/writing EFLAGS including the interrupt-enable flag (IF). In Ring 3, the IF bit is silently ignored rather than causing a trap.
  • SGDT, SIDT, SLDT: read the GDT/IDT/LDT descriptor table registers — these reveal the host OS's real descriptor tables, breaking isolation.
  • STR: stores the task register — also leaks privileged state.

VMware solved this with binary translation: scan guest code before execution, rewrite sensitive-but-non-trapping instructions to safe equivalents that do cause a trap. Expensive, but it worked.

Virtualization Concept

A Virtual Machine Monitor (VMM), commonly called a hypervisor, creates the illusion that each guest OS has exclusive access to a complete hardware platform. Three essential properties (Popek–Goldberg):

  1. Fidelity: a program running under the VMM behaves identically to running on bare hardware (except timing)
  2. Safety: the VMM retains complete control of hardware resources
  3. Efficiency: a statistically dominant fraction of instructions execute natively without VMM intervention
Without Virtualization:
+---------------------------+
|   Application             |
+---------------------------+
|   Operating System        |
+---------------------------+
|   Hardware                |
+---------------------------+

With Virtualization:
+----------+  +----------+
| App A    |  | App B    |
+----------+  +----------+
| Guest OS1|  | Guest OS2|
+----------+  +----------+
|    Virtual Hardware       |
+---------------------------+
|      Hypervisor           |
+---------------------------+
|      Physical Hardware    |
+---------------------------+

Hypervisor Types

Type 1 — Bare-Metal Hypervisor

Runs directly on the physical hardware. The hypervisor is the OS from the hardware's perspective. No host OS underneath.

Examples: KVM (with Linux as the co-scheduler), Xen, VMware ESXi, Microsoft Hyper-V, IBM PowerVM

Characteristics: - Lower latency: no host OS scheduling overhead - Better isolation: no host OS attack surface - Full hardware resource control - Requires dedicated machine - KVM is sometimes called "Type 1.5" because it is a Linux kernel module — Linux manages hardware, but KVM gains Ring 0 control via VMX root mode

Type 2 — Hosted Hypervisor

Runs as a process on top of an existing host OS. Hardware access mediated by host OS.

Examples: VMware Workstation, VMware Fusion, VirtualBox, QEMU (in TCG mode), Parallels Desktop

Characteristics: - Easy to install: just an application - Host OS adds overhead and scheduling jitter - Host OS provides device drivers (simpler compatibility) - Higher attack surface: guest escape → host OS → hardware

Type 1 (Bare-Metal):          Type 2 (Hosted):
+--------+  +--------+        +--------+  +--------+
| Guest1 |  | Guest2 |        | Guest1 |  | Guest2 |
+--------+  +--------+        +--------+  +--------+
| Hypervisor (Ring 0) |        | VMware / VirtualBox |
+--------------------+        +--------------------+
|   Hardware         |        |   Host OS (Ring 0)  |
+--------------------+        +--------------------+
                              |   Hardware          |
                              +--------------------+

Hypervisor Comparison Table

Property Type 1 (ESXi, Xen) Type 1.5 (KVM) Type 2 (VirtualBox)
Runs on Bare hardware Linux kernel Host OS
Overhead Very low (~1–3%) Low (~2–5%) Medium (~5–15%)
Guest isolation Strongest Strong Weaker
Device support Needs own drivers Reuses Linux drivers Reuses host OS drivers
Production use Data centers Cloud providers Developer desktops
Key examples ESXi, Hyper-V AWS EC2, GCP, Azure VirtualBox, Fusion
Live migration Yes Yes (QEMU) Limited
Memory overcommit Yes (with balloon) Yes (KSM+balloon) Limited

Virtualization Techniques

Full Virtualization (Trap-and-Emulate)

Guest OS runs unmodified. Privileged instructions from the guest trap into the hypervisor (via hardware extensions), which emulates their effect. The guest never knows it is virtualized.

With hardware assist (Intel VT-x / AMD-V), the CPU operates in two modes: - VMX root mode: hypervisor runs; full hardware access - VMX non-root mode: guest runs; sensitive instructions cause automatic VMEXIT to hypervisor

This is the dominant mode today: KVM + QEMU running unmodified Linux or Windows guests.

Paravirtualization (PV)

Guest OS kernel is modified to be "hypervisor-aware." Instead of executing privileged instructions that would trap, the guest kernel directly calls hypercalls — a hypervisor ABI analogous to system calls but for the guest→hypervisor boundary.

Xen PV was the canonical implementation. The guest kernel is recompiled with Xen-specific hypercalls replacing sensitive instructions. Results in very low overhead (~2–5% vs native) because there are almost no unexpected traps — all transitions are explicit hypercalls.

Drawback: requires maintaining a modified kernel for each supported guest OS. Linux has had Xen PV support built-in since 2.6.23.

Hardware-Assisted Virtualization (HVM)

Intel introduced VMX (Virtual Machine Extensions) in VT-x (2005). AMD introduced AMD-V / SVM (Secure Virtual Machine) in 2006. Both add:

  • New CPU execution mode (VMX non-root) where sensitive instructions cause automatic hardware traps (VMEXITs) to the hypervisor
  • VMCS (Intel) / VMCB (AMD): per-VM/vCPU data structures storing guest and host state
  • Extended Page Tables (Intel EPT) / Nested Page Tables (AMD NPT): hardware two-level address translation

This eliminates the need for binary translation or guest kernel modification. The hardware enforces isolation.

Hardware-Assisted Virtualization Flow:

  Guest (VMX non-root mode, Ring 0/3)
      |
      |  Sensitive instruction (e.g., write CR0, HLT, I/O port)
      |
      v
  VMEXIT triggered by hardware
      |
      v
  Hypervisor (VMX root mode, Ring 0)
      |
      |  Read VMEXIT reason from VMCS
      |  Emulate the instruction effect
      |
      v
  VMENTER — resume guest (VMRESUME)
      |
      v
  Guest continues execution

Key Milestones Timeline

Year Event
1967 IBM CP-40: first VM system (experimental)
1967 IBM CP-67: first deployed VM hypervisor
1972 IBM VM/370: first widely used commercial VM
1974 Popek & Goldberg theorem published (CACM)
1998 VMware founded
1999 VMware Workstation 1.0: first x86 hypervisor (binary translation)
2001 VMware ESX 1.0: first bare-metal x86 hypervisor
2003 Xen 1.0: open-source paravirtualization hypervisor
2005 Intel VT-x released (Vanderpool Technology)
2006 AMD-V released; AWS EC2 beta (Xen-based)
2007 KVM merged into Linux kernel 2.6.20
2008 Xen 3.3: HVM (hardware-assisted) mode
2013 QEMU/KVM becomes default for OpenStack
2018 AWS Nitro (KVM-based), Firecracker open-sourced
2020 Apple M1: Hypervisor.framework, Virtualization.framework

Production Examples

Amazon EC2: Originally Xen (2006), transitioned to Nitro hypervisor (KVM-based) starting 2017. Nitro offloads I/O to dedicated hardware (Nitro cards), leaving the host CPU entirely for guest workloads — nearly zero hypervisor overhead for I/O.

Google Cloud Platform: Uses KVM with custom patches. Live migration is used routinely for host maintenance — VMs migrate transparently with <200ms downtime.

VMware vSphere: ESXi used in the vast majority of enterprise data centers for decades. Full VM lifecycle management, vMotion (live migration), vSAN (virtual storage).

Microsoft Azure: Hyper-V (Type 1, Windows-based hypervisor), with Azure Boost (similar to Nitro) offloading storage and networking.

Security Implications

  • VM escape: guest compromises the hypervisor, gaining access to other VMs or the host. CVE-2015-3456 (VENOM) — QEMU floppy controller buffer overflow allowing VM escape.
  • Side-channel attacks: VMs share physical CPU. Spectre/Meltdown (2018) affected all hypervisors. L1TF (L1 Terminal Fault, CVE-2018-3646) specifically targeted EPT entries, allowing a guest to read host memory via L1 cache timing. Mitigation required flushing L1 cache on VMENTER.
  • Hyperjacking: attacker installs a rogue hypervisor beneath the running OS (Blue Pill attack, 2006, Joanna Rutkowska). Defenses: Secure Boot, TPM-based attestation.
  • Timing attacks: VMs share hardware counters. High-resolution timers must be virtualized carefully to avoid leaking host information.

Performance Implications

  • VMEXIT cost: each trap to the hypervisor costs 1,000–10,000 ns depending on reason. Minimizing VMEXITs is the primary performance goal.
  • Memory overhead: EPT walk adds 1–5% overhead vs native. Shadow page tables were 10–30% overhead.
  • I/O virtualization: emulated devices (e8139 NIC, IDE disk) are the worst performers. VirtIO reduces overhead dramatically. SR-IOV (direct hardware passthrough) eliminates it.
  • Cache effects: VMs sharing a physical CPU share L3 cache. Noisy neighbor problem — one VM doing heavy I/O pollutes cache for others.
  • NUMA: VM vCPUs and memory should be NUMA-local. Cross-NUMA access adds 30–40% memory latency.

Debugging Notes

Determine if running inside a VM:

# Check for hypervisor CPUID bit
dmesg | grep -i "hypervisor\|kvm\|vmware\|xen"

# Check systemd-detect-virt
systemd-detect-virt

# Check DMI
dmidecode -s system-product-name

# Check for KVM paravirt features
cat /proc/cpuinfo | grep hypervisor

Host-side (KVM) debugging:

# Count VMEXITs per reason (per-VM)
cat /sys/kernel/debug/kvm/*/exits

# Watch KVM stats
watch -n1 cat /sys/kernel/debug/kvm/*/exits

# Check vcpu halt stats
cat /sys/kernel/debug/kvm/*/halt_poll_success_ns

Failure Modes

  • Hypervisor crash: all VMs on host lose state simultaneously — catastrophic failure affecting all tenants. Mitigated by redundancy (live migration away before failures), HA clustering.
  • Memory balloon exhaustion: hypervisor over-reclaims guest memory; guest OOM-kills its own processes. Monitor balloon driver metrics.
  • Clock skew: VMs can drift if they do not use paravirtualized clock (kvmclock). NTP alone may be insufficient; use kvm-clock or PTP hardware timestamping.
  • vCPU steal time: guest vCPU scheduled out by host; guest sees high "steal" time in top/vmstat. Indicates host overcommit.

Modern Usage and Future Directions

Confidential computing: AMD SEV (Secure Encrypted Virtualization) and Intel TDX (Trust Domain Extensions) encrypt VM memory in hardware, so even the hypervisor cannot read guest data. Enables trusted execution environments in untrusted cloud infrastructure.

MicroVMs: Firecracker (AWS), Cloud Hypervisor (Intel), crosvm (Google) — stripped-down VMMs for serverless and container workloads. Boot in <200ms, minimal attack surface.

Unikernels: single-address-space OS images (MirageOS, Unikraft) that run directly as VMs — no guest OS overhead, smallest possible attack surface.

WebAssembly runtimes: wasmtime, wasmer as an alternative isolation boundary — lightweight, language-agnostic sandboxing without full VM overhead.

Exercises

  1. On a Linux host with KVM, run kvm-ok and examine /proc/cpuinfo flags for vmx (Intel) or svm (AMD). Explain what each flag enables.
  2. Use QEMU to boot a Linux VM: qemu-system-x86_64 -enable-kvm -m 512 -kernel vmlinuz -append "console=ttyS0" -nographic. Monitor VMEXIT counts via /sys/kernel/debug/kvm/.
  3. Compare boot time of an emulated (no -enable-kvm) vs KVM-accelerated VM. Explain the difference.
  4. Research the VENOM (CVE-2015-3456) vulnerability. Identify what device was vulnerable, what the exploit did, and how it was patched.
  5. Write a one-paragraph explanation of why x86 was not virtualizable pre-VT-x, using POPF as a concrete example.

References

  • Popek, G.J. & Goldberg, R.P. (1974). "Formal Requirements for Virtualizable Third Generation Architectures." Communications of the ACM, 17(7), 412–421.
  • Intel. Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3C: System Programming Guide, Part 3. Chapters 23–33 (VMX).
  • Barham, P. et al. (2003). "Xen and the Art of Virtualization." SOSP 2003.
  • VMware. (2007). "Understanding Full Virtualization, Paravirtualization, and Hardware Assist." VMware White Paper.
  • Adams, K. & Agesen, O. (2006). "A Comparison of Software and Hardware Techniques for x86 Virtualization." ASPLOS 2006.
  • Kivity, A. et al. (2007). "KVM: the Linux Virtual Machine Monitor." Linux Symposium 2007.