Skip to content

Library OS and Unikernels: The Future of Minimal Execution Environments

Overview

The idea of a library operating system—where OS services are compiled directly into the application rather than provided by a separate kernel layer—dates to the MIT Exokernel project of the mid-1990s. After a decade of relative academic dormancy, the idea has returned with renewed urgency, driven by the economics and security requirements of cloud computing, the proliferation of serverless functions, and the maturation of hypervisor technology that makes running single-application virtual machines practical and fast.

Unikernels represent the most radical form of this idea: a single-address-space machine image containing exactly the application code plus only the OS library components it requires, compiled as a single binary, running directly on bare metal or in a thin hypervisor, with no shell, no userspace/kernel distinction, no package manager, and no unnecessary attack surface.

This document surveys the unikernel landscape, examines production deployments, analyzes the genuine trade-offs, and assesses where this approach will and will not succeed.

Prerequisites

  • Understanding of monolithic OS architecture and the kernel/user distinction
  • Familiarity with hypervisors (Type 1: KVM/Xen; Type 2: QEMU)
  • Basic awareness of virtual machine boot process and GRUB/multiboot
  • Understanding of containers (Docker/OCI) for comparison purposes
  • Familiarity with ELF binary format and linking

Historical Context: The Exokernel Insight

MIT's Exokernel (Dawson Engler, Frans Kaashoek, 1995) proposed that the OS should be responsible for only one thing: securely multiplexing hardware resources. All higher-level abstractions—filesystems, network protocols, process scheduling policies—should be implemented in untrusted user-level library OSes (libOSes), allowing applications to customize OS behavior without kernel changes.

The Exokernel itself would export hardware resources directly: disk blocks, network packets, CPU time slices. Applications would use libOS libraries to implement their preferred abstractions on top.

This was radical and impractical in the 1990s when hardware was limited and application packaging was immature. In the 2020s, with: - Cloud hypervisors that boot VMs in milliseconds - Container registries for image distribution - Static compilation and cross-compilation toolchains at scale - NVMe storage and 100GbE networking that saturate general-purpose OS overhead

...the economic case for unikernels has strengthened considerably.

The Unikernel Vision vs. Containers vs. VMs

ISOLATION AND OVERHEAD COMPARISON

Full VM (Linux guest on KVM):
  +-----------------------------------------------+
  | Linux kernel (5+ MB, 30M+ lines) + services   |
  | init, systemd, sshd, cron, etc.               |
  | Application                                    |
  +-----------------------------------------------+
  |           KVM hypervisor (thin)                |
  |              Host Linux kernel                 |
  +-----------------------------------------------+
  Boot time: 2-10 seconds   Memory: 256MB+ base

Container (Docker/OCI on Linux):
  +-----------------------------------------------+
  | Application + libraries (shared kernel)        |
  +-----------------------------------------------+
  |    Linux kernel (shared, namespaced/cgroups)   |
  +-----------------------------------------------+
  Boot time: 100-500ms      Memory: ~10MB+ overhead
  (not a true isolation boundary — shared kernel)

Unikernel (e.g., Nanos, MirageOS):
  +-----------------------------------------------+
  | Application + only required OS libraries       |
  | (no shell, no sshd, no init, no package mgr)  |
  +-----------------------------------------------+
  |         Thin hypervisor (KVM/Xen) or          |
  |              bare metal (ARM/x86)              |
  +-----------------------------------------------+
  Boot time: 1-50ms         Memory: 1-20MB total image
  Strong VM isolation boundary                    

Unikernel (bare metal/embedded):
  +-----------------------------------------------+
  | Application + OS library linked as one binary |
  +-----------------------------------------------+
  |              Hardware directly                 |
  +-----------------------------------------------+
  Boot time: <1ms           Memory: minimal

Major Unikernel Projects

Unikraft

Unikraft is the most mature modular unikernel framework. Started as a research project at NEC Labs Europe, it is now a CNCF Sandbox project.

Core philosophy: build POSIX-compatible unikernels from a component menu. Unikraft uses a Kconfig/Makefile build system (familiar to Linux kernel developers) where you configure which components to include:

  • Memory allocator (buddy, TLSF, or custom)
  • Scheduler (cooperative, preemptive, or custom)
  • Network stack (lwIP, musl sockets, or custom)
  • Filesystem (9pfs for host-mounted FS, ramfs, ext4)
  • Architecture (x86-64, ARM64, RISC-V)
  • Platform (KVM, Xen, linuxu for debugging)

Unikraft provides a compatibility shim for musl libc, allowing many standard Linux applications to run with minimal or no code modification. Projects like Redis, Nginx, SQLite, Python, and Node.js have been ported.

UNIKRAFT BUILD SYSTEM

   Application source
        +
   musl libc shim
        +                   menuconfig:
   [Selected Unikraft       [*] Scheduler: preemptive
    components]             [*] Network: lwIP
        +                   [*] FS: ramfs
   Platform layer           [ ] FS: 9pfs
   (KVM/Xen/bare metal)     [*] Memory: buddy allocator
        |
        v
   Single ELF binary (unikernel image)
        |
        v
   Boot via QEMU/KVM in ~5ms

Performance results from Unikraft benchmarks: - Nginx serving static files: ~4x throughput improvement vs Linux container (due to eliminated context switch overhead) - SQLite: ~2x improvement on read-heavy workloads - Boot time: 5–10ms vs 2–5 seconds for minimal Linux VM

MirageOS

MirageOS is a unikernel framework written in OCaml, developed at the University of Cambridge and funded by Citrix and the EU. It takes a different approach: rather than wrapping existing C libraries, MirageOS rewrites OS components in OCaml with type safety guarantees.

Key properties: - Type-safe entire stack: Network protocols (TCP/IP, TLS, DNS), storage (key-value stores, filesystem), and the application are all written in OCaml - Formal verification potential: OCaml's type system eliminates entire classes of bugs; some components have been formally verified with Coq - Functorization: Network stack, storage, and other components are parameterized by module signatures, allowing easy substitution (e.g., swap the underlying memory allocator or the TLS implementation)

Production deployment: Citrix/XenServer uses MirageOS unikernels for the control plane of Xen hypervisor toolstacks. Docker's certificate infrastructure has used MirageOS-based components.

The TLS implementation (ocaml-tls) has been formally audited and found to be significantly simpler and lower-vulnerability than OpenSSL, despite supporting TLS 1.3.

MIRAGEOS FUNCTOR ARCHITECTURE

  Application (OCaml)
      |
  Cohttp HTTP server
  (parameterized by: NET and TLS modules)
      |
      +-- NET = Tcpip.Stack.Direct (pure OCaml TCP/IP)
      |         OR
      |         Tcpip.Stack.Socket (UNIX sockets for dev)
      |
      +-- TLS = Tls.Make(X509)(Nocrypto)
      |
      v
  MirageOS platform layer (Xen/KVM/Solo5)
      |
      v
  Single binary image

IncludeOS

IncludeOS is a C++ unikernel framework designed for network applications. It compiles a C++ application directly against its OS libraries, producing a bootable image that runs on KVM or bare metal.

Notable characteristics: - Boot time under 1ms on bare metal (measured booting a simple HTTP server on ARM) - Used by several European telecom companies for network function virtualization (NFV) appliances - Extensive support for network protocols: TCP, UDP, DHCP, DNS, HTTP, HTTPS - No process model: single address space, single thread (with optional cooperative threading)

The single-address-space model means memory corruption bugs that would crash only a process in Linux will crash the entire VM. However, since the VM can be restarted in <1ms, this is often acceptable.

Nanos / Ops

Nanos (maintained by NanoVMs) takes the most pragmatic approach: run unmodified Linux ELF binaries as unikernels.

The Nanos kernel implements a Linux-compatible system call interface. When you provide an existing Linux binary (say, a Go HTTP server compiled for Linux/amd64), the ops build tool packages it as a Nanos image. The resulting image runs on KVM/QEMU without a Linux kernel—Nanos provides the syscall layer directly.

NANOS DEPLOYMENT WORKFLOW

  $ go build -o server ./server.go       # normal Linux binary
  $ ops run server                        # packages + boots as unikernel

  ops workflow:
  1. Takes Linux ELF binary
  2. Packages with Nanos kernel image
  3. Creates disk image with config
  4. Boots in QEMU/KVM with ~10-50ms boot time

  At runtime: server calls write() -> Nanos syscall handler -> hardware
  (no Linux kernel involved)

Nanos supports cloud deployment to AWS, GCP, Azure (via image upload), and to on-premise KVM. The target audience is operators running microservices who want VM-level isolation without the overhead of a full Linux guest.

Production users include several fintech and SaaS companies for stateless API services.

Unikernel Security Model

The security argument for unikernels is straightforward:

Attack surface reduction:

Component Linux VM Unikernel
Shell bash/sh None
Package manager apt/yum None
Unused system calls 300+ Typically 30-80
SSH daemon Present by default None
Syslog daemon Present None (logging via hypervisor)
World-writable /tmp Present None or minimal
Loaded kernel modules Dozens None (no module loading)

A Nanos unikernel running an HTTP server implements only the syscalls the Go runtime actually uses. A buffer overflow that achieves code execution cannot spawn a shell (there is no shell to spawn), cannot access a C2 server's download link (there is no package manager), and cannot escalate to root (there is no concept of root—the process is the kernel).

Limitations:

  • No process isolation: If your application has a logic bug that allows memory corruption, the entire VM is compromised. In Linux, a compromised process is (usually) isolated from other processes.
  • No multi-tenancy: Unikernels are one-application-per-VM. If you need to run multiple applications with isolation between them, you need multiple VMs (or a different approach).
  • Harder to debug: No strace, no gdb attach, no /proc. Debugging unikernel issues requires either serial console output, remote debugging via GDB stub, or specialized tooling.

Unikernel for Serverless

The Lambda-style execution model is a natural fit for unikernels:

SERVERLESS COLD START COMPARISON

Traditional Lambda (Linux-based):
  Request -> provision micro-VM (Firecracker) -> boot minimal Linux (~125ms)
           -> load Python runtime (~200ms) -> execute function
  Total cold start: 300-500ms

Unikernel-based (hypothetical Nanos + Go):
  Request -> provision KVM VM -> boot Nanos + Go binary (~15ms)
           -> execute function
  Total cold start: 15-30ms

MirageOS (OCaml):
  Request -> provision Xen VM -> boot MirageOS image (~5ms)
           -> execute function
  Total cold start: 5-10ms

AWS Firecracker (their microVM hypervisor for Lambda) achieves ~125ms Linux boot. Unikernel approaches can reduce this by 5–10x. For short-lived functions where cold start dominates execution time, this is significant.

Projects exploring unikernels in serverless: - Unikraft in CNCF: explicit goal of unikernel-as-function-runtime - Kata Containers + Nanos: experimental integration - IBM's Unikernel Systems work (now largely folded into Unikraft ecosystem)

Challenges

Debugging

The most significant practical challenge of unikernels in production is debugging. When a Nanos or IncludeOS VM crashes: - No core dump from the OS (you can get a VM state dump from KVM) - No /var/log (logs must be externalized via syslog-over-UDP or similar) - No strace capability - No runtime introspection without purpose-built tooling

MirageOS partially addresses this: OCaml's runtime provides backtraces, and the functorized architecture allows running the same binary in a UNIX process context (mirage configure -t unix) for debugging with standard tools.

Unikraft provides a linuxu (Linux userland) platform that runs the unikernel as a regular Linux process for debugging.

Ecosystem Compatibility

Most software was written assuming a Linux POSIX environment. Porting requires: - Replacing all fork()-based patterns (unikernels have no fork) - Eliminating shell scripting in startup - Replacing filesystem operations with in-memory or remote alternatives - Replacing signals with equivalent unikernel primitives

Nanos sidesteps this for statically-compiled Go and Rust binaries (which do not use fork). Python, Ruby, and Node.js require more work.

No Multi-Tenancy

Each unikernel is one application. If you run 100 microservices, you need 100 VMs. The VM overhead per unikernel is low (1-20MB, <50ms boot) but it is not zero. Container-based deployments share the kernel and achieve much higher density per host.

Will Unikernels Win?

The honest assessment is: unikernels will succeed in specific niches, not as a general replacement for Linux.

Likely winning niches: - Network functions (load balancers, firewalls, routers) — IncludeOS, custom Unikraft - Stateless serverless functions — Unikraft, Nanos - Security-critical single-function services (TLS termination, secret management) — MirageOS - Embedded IoT devices with update-over-the-air requirements

Unlikely to displace Linux for: - General-purpose application servers with complex dependencies - Development environments (no shell = painful developer experience) - Stateful services with complex storage requirements - Multi-tenant platforms

Failure Modes

  • Boot failure with no recovery path: A misconfigured unikernel may fail to boot with only a serial console message. Unlike Linux, there is no rescue mode, no single-user mode, no safe boot option by default.
  • Memory leak = crash: Without an OS managing process memory boundaries, a leak eventually exhausts the available memory and crashes the VM. The VM restarts quickly but state is lost.
  • Stale binary distribution: Unikernel images combine application code with OS components. A vulnerability in the embedded TCP/IP stack (e.g., lwIP CVE) requires rebuilding and redeploying all images using it, rather than patching the OS on all hosts.

Exercises

  1. Install ops (Nanos toolchain) and deploy a simple Go HTTP server as a unikernel on your local machine using QEMU. Measure boot time vs a Docker container running the same binary.

  2. Build a Unikraft image for Redis using the Unikraft build system. Compare memory usage (RSS) of the unikernel image vs a standard Redis container.

  3. Read the MirageOS tutorial for building a static website server. Follow it and deploy the result. Then intentionally remove a capability from the unikernel build and observe the compile-time error.

  4. Analyze the attack surface of a Nanos unikernel running a Python Flask application. What syscalls does it expose? Compare to the syscall surface of the same application in a Docker container (strace -c).

  5. Study the IncludeOS boot sequence source code. How does it initialize the network stack without a Linux kernel? Where does the first C++ constructor run?

References

  • Engler, D. et al. "Exokernel: An Operating System Architecture for Application-Level Resource Management" (SOSP 1995)
  • Madhavapeddy, A. et al. "Unikernels: Library Operating Systems for the Cloud" (ASPLOS 2013) — the paper that named the field
  • Kuenzer, S. et al. "Unikraft: Fast, Specialized Unikernels the Easy Way" (EuroSys 2021)
  • Cantrill, B. "Unikernels are unfit for production" (2016, counter-argument) — important critique
  • Nanos/ops documentation: https://nanos.org / https://ops.city
  • MirageOS documentation: https://mirage.io
  • Unikraft project: https://unikraft.org
  • IncludeOS documentation: https://www.includeos.org
  • Olivier, P. et al. "A Binary-Compatible Unikernel" (VEE 2019) — Unikraft binary compat approach
  • AWS Firecracker: "Firecracker: Lightweight Virtualization for Serverless Applications" (NSDI 2020)