Skip to content

02 — Kernel Modules (LKMs)

Technical Overview

A Loadable Kernel Module (LKM) is an object file that can be dynamically inserted into a running kernel to extend its functionality — adding a device driver, a filesystem, a network protocol, or a security module — without requiring a kernel recompile or reboot. This is the fundamental mechanism by which Linux separates the core kernel from the enormous diversity of hardware and software features that might optionally be needed.

The kernel is not a single executable. When you run uname -r and get 6.8.0-50-generic, that string identifies a kernel image stored as /boot/vmlinuz-6.8.0-50-generic plus a directory of modules at /lib/modules/6.8.0-50-generic/. The kernel image contains the core scheduler, memory manager, and VFS layer. The modules directory may contain thousands of .ko (kernel object) files for every supported driver and filesystem. A typical Ubuntu installation ships over 6,000 .ko files; at any given time, perhaps 50-200 are loaded.

Prerequisites

  • Understanding of C compilation (object files, linking, symbols)
  • Basic Linux process model (kernel space vs user space)
  • Familiarity with the Linux driver model (see 01-driver-model.md)
  • Understanding of kernel vs user memory addressing

Module Structure

A minimal module requires two functions and three metadata macros:

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Jane Engineer <jane@example.com>");
MODULE_DESCRIPTION("A minimal example module");
MODULE_VERSION("1.0");

static int __init my_module_init(void)
{
    printk(KERN_INFO "my_module: loaded\n");
    return 0;   /* 0 = success; negative errno = failure */
}

static void __exit my_module_exit(void)
{
    printk(KERN_INFO "my_module: unloaded\n");
}

module_init(my_module_init);
module_exit(my_module_exit);

module_init and module_exit are macros that annotate the functions and place pointers to them in special ELF sections (.init.text and .exit.text). When the module is loaded, init_module() syscall invokes the function registered with module_init. When unloaded, delete_module() syscall invokes the module_exit function.

The __init annotation places code in a section that is freed after initialization completes — for built-in code, this reclaims memory. For modules, the section is present in the .ko file but discarded after the init function returns.

Module Lifecycle Diagram

  Module file: my_module.ko
  ┌─────────────────────────────────────┐
  │  ELF sections:                      │
  │  .text      — runtime code          │
  │  .init.text — __init functions      │
  │  .exit.text — __exit functions      │
  │  .modinfo   — metadata strings      │
  │  __versions — kernel symbol hashes  │
  │  __ksymtab  — exported symbols      │
  └─────────────────────────────────────┘
            │
            │  insmod / modprobe
            ▼
  ┌──────────────────────────────────────────────────────┐
  │   KERNEL: finit_module() / init_module() syscall     │
  │                                                      │
  │   1. Copy .ko into kernel memory                     │
  │   2. Verify: vermagic string (kernel version match)  │
  │   3. Verify: module signature (if CONFIG_MODULE_SIG) │
  │   4. Resolve symbols against kernel symbol table     │
  │      (undefined symbols → fail with ENOEXEC)         │
  │   5. Apply relocations (patch symbol addresses)      │
  │   6. Call module_init function                       │
  │   7. Add to loaded module list (/proc/modules)       │
  └──────────────────────────────────────────────────────┘
            │
            │  module running
            ▼
  ┌─────────────────────────────────────┐
  │   MODULE_STATE_LIVE                 │
  │   - Functions callable              │
  │   - Symbols visible to other mods   │
  │   - Parameters in sysfs             │
  │   - refcount tracks users           │
  └─────────────────────────────────────┘
            │
            │  rmmod
            ▼
  ┌──────────────────────────────────────────────────────┐
  │   KERNEL: delete_module() syscall                    │
  │                                                      │
  │   1. Check refcount == 0 (else EBUSY)                │
  │   2. STATE = GOING                                   │
  │   3. Call module_exit function                       │
  │   4. Remove from /proc/modules                       │
  │   5. Free module memory                              │
  └──────────────────────────────────────────────────────┘

Module Compilation: The Kbuild System

Linux modules are compiled using the kernel's own build system (Kbuild) to ensure they are compiled with exactly the right flags, against the correct kernel headers. You cannot build a module with arbitrary CFLAGS — the vermagic string baked into the module must match the running kernel.

# Minimal external module Makefile
obj-m += my_module.o

# Multi-file module:
# obj-m += my_driver.o
# my_driver-objs := main.o hw.o dma.o

all:
    make -C /lib/modules/$(shell uname -r)/build \
         M=$(PWD) modules

clean:
    make -C /lib/modules/$(shell uname -r)/build \
         M=$(PWD) clean

The -C /lib/modules/.../build flag changes directory to the kernel build directory (a symlink to /usr/src/linux-headers-$(uname -r) on Debian/Ubuntu), which contains the full Kbuild infrastructure. M=$(PWD) tells Kbuild where the external module source is. Kbuild then compiles the module using the exact same compiler flags used to build the kernel itself.

The resulting .ko file contains a vermagic string in its .modinfo section that encodes the kernel version, SMP status, preemption model, and compiler version. The kernel's module loader rejects any .ko whose vermagic does not match the running kernel.

Loading and Unloading

insmod

insmod loads a .ko file directly by calling the finit_module() syscall (Linux 3.8+) with a file descriptor, or the older init_module() with a memory buffer. It does not resolve dependencies — if my_module.ko requires symbols from another_module.ko, insmod will fail with ENOEXEC: Unknown symbol.

insmod /path/to/my_module.ko [param=value ...]

modprobe

modprobe is the high-level module loader. It reads /lib/modules/$(uname -r)/modules.dep to determine dependencies, loads them first, then loads the requested module. On unload (modprobe -r), it removes the module and then removes any dependencies that are no longer needed.

modprobe e1000e                        # load NIC driver and its deps
modprobe -r e1000e                     # unload and remove unused deps
modprobe --show-depends nvme           # show what would be loaded

modules.dep is generated by depmod, which scans all .ko files in the modules directory, resolves their undefined symbols against the kernel's System.map and against other modules' exported symbols, and writes the dependency graph.

lsmod / rmmod

lsmod                                  # show loaded modules (reads /proc/modules)
rmmod my_module                        # unload; fails if refcount > 0

/proc/modules format:

Module                  Size  Used by
my_module              16384  0
nvme                  106496  4
nvme_core             176128  6 nvme

The "Used by" column shows both the reference count and which modules depend on this one.

Module Parameters

Modules can expose configurable parameters via the module_param macro. Parameters can be set at load time and are visible (and sometimes writable) via sysfs at /sys/module/<name>/parameters/.

static int debug_level = 0;
static char *device_name = "mydev0";
static int queue_depth = 32;

module_param(debug_level, int, 0644);     /* name, type, sysfs permissions */
MODULE_PARM_DESC(debug_level, "Debug verbosity (0=none, 3=max)");

module_param(device_name, charp, 0444);   /* read-only from sysfs */
MODULE_PARM_DESC(device_name, "Device name to use");

module_param(queue_depth, int, 0644);
MODULE_PARM_DESC(queue_depth, "I/O queue depth per CPU");

At load time:

modprobe my_module debug_level=2 queue_depth=64

After loading:

cat /sys/module/my_module/parameters/debug_level
# 2
echo 3 > /sys/module/my_module/parameters/debug_level  # if writable (0644)

Supported types: bool, int, uint, long, ulong, short, ushort, charp (char pointer), byte, and arrays via module_param_array.

Kernel Symbol Table

When a module calls a kernel function (e.g., kmalloc, printk, register_chrdev), those symbols must be resolved to their runtime addresses when the module loads. The kernel maintains an exported symbol table — the set of symbols that modules are allowed to use.

/* In a kernel source file or another module */
EXPORT_SYMBOL(my_useful_function);          /* any module can use */
EXPORT_SYMBOL_GPL(my_gpl_only_function);    /* only GPL modules */
EXPORT_SYMBOL_GPL_FUTURE(deprecated_func);  /* deprecated */

EXPORT_SYMBOL_GPL is the kernel's mechanism for enforcing GPL licensing on kernel interfaces. A module with MODULE_LICENSE("Proprietary") cannot link against GPL-only symbols; the module loader will reject it with ENOEXEC. This is the legal mechanism that enforces the kernel's interpretation of the GPL for binary-only drivers.

The complete set of exported symbols is visible in /proc/kallsyms (with CONFIG_KALLSYMS=y). On hardened kernels, /proc/kallsyms may show all symbols as address 0 for non-root users to prevent KASLR bypass.

grep " T " /proc/kallsyms | head      # T = text (code) symbols
grep "register_chrdev" /proc/kallsyms

Module Signing

CONFIG_MODULE_SIG enables module signature enforcement. With this enabled, the kernel will only load modules that are signed by a trusted key. The signing key is embedded in the kernel image at build time (certs/signing_key.pem).

# Sign a module manually
scripts/sign-file sha256 signing_key.pem signing_key.x509 my_module.ko

# Check if module is signed
modinfo my_module.ko | grep sig

# Kernel enforcement levels:
# CONFIG_MODULE_SIG_FORCE=y  → refuse unsigned modules entirely
# CONFIG_MODULE_SIG=y alone  → warn but load unsigned modules

Secure Boot integration: On systems with UEFI Secure Boot enabled, the bootloader (shim + grub) verifies the kernel. The kernel then enforces CONFIG_MODULE_SIG_FORCE because loading an unsigned module would break the Secure Boot chain of trust — an attacker who could load an unsigned module could subvert any kernel security feature.

Distribution kernels (Ubuntu, Fedora, RHEL) ship with pre-signed modules. Third-party modules (NVIDIA, VirtualBox, ZFS) use DKMS (Dynamic Kernel Module Support) to recompile and sign modules for each kernel update using a machine-owner key (MOK) enrolled via mokutil.

Module Security Analysis

A loaded kernel module runs at ring 0 with full kernel privileges. There is no sandboxing, no memory isolation, no capability restriction. A module can:

  • Read and write any physical memory address
  • Install rootkit hooks (overwrite system call table, hook function pointers)
  • Disable security features (clear CR4.WP to make read-only pages writable)
  • Exfiltrate data from any process
  • Bypass SELinux/AppArmor

This makes module loading a critical attack surface. The mitigations are:

  1. Require CAP_SYS_MODULE for init_module/finit_module/delete_module syscalls
  2. Module signing (CONFIG_MODULE_SIG_FORCE) on Secure Boot systems
  3. Lockdown LSM (Linux 5.4): when Secure Boot is active, lockdown=confidentiality prevents writing to /dev/mem, /dev/kmem, hibernation, and (in integrity mode) unsigned module loading
  4. seccomp + container policies: container runtimes block init_module syscall by default

A significant historical attack: if an attacker could write to /proc/sys/kernel/modprobe, they could redirect the modprobe path to an arbitrary executable. This was patched by making the modprobe path read-only under Lockdown mode.

Historical Context

The LKM concept predates Linux. BSD kernels had loadable kernel modules (KLDs in FreeBSD). Solaris had kernel modules (/kernel/drv/). The motivation is practical: a kernel that must include every possible driver would be enormous and unmaintainable.

Early Linux (pre-1.0) had very limited module support. By Linux 2.2, modules were widespread, though the ABI was unstable — modules built for 2.2.x rarely worked on 2.4.x without recompilation.

The 2.6 kernel (2003) introduced the current .ko format (replacing the older .o format), module_param (replacing the fragile MODULE_PARM), and improved symbol versioning (CONFIG_MODVERSIONS) which hashes each exported symbol's type signature and embeds it in the module, allowing some cross-version compatibility.

Controversy around binary-only modules (particularly NVIDIA's) drove the creation of EXPORT_SYMBOL_GPL as a legal/technical enforcement mechanism. This remains contested: NVIDIA shipped a binary blob with a thin GPL-licensed wrapper (nvidia.ko calling into nvidia.bin) until they released open-source kernel modules in 2022 (for Turing and later GPUs).

Production Examples

DKMS for out-of-tree modules: ZFS on Linux (zfs.ko) is not included in the mainline kernel due to CDDL/GPL license incompatibility. Ubuntu ships it via DKMS: when a new kernel is installed, DKMS automatically recompiles and signs ZFS modules for the new kernel version.

NVMe driver as module: On most Linux systems, nvme.ko is loaded from an initramfs (initial RAM filesystem) that is loaded by the bootloader before the root filesystem is mounted. The initramfs contains only the modules necessary to mount the root filesystem. dracut and initramfs-tools assemble this image based on the detected hardware.

eBPF as a safer alternative: Many things that previously required a kernel module (packet filtering, tracing, profiling) can now be done with eBPF programs, which run in a verifier-checked sandbox. The verifier statically analyzes the BPF bytecode to prove it terminates and doesn't perform dangerous memory accesses. This is a safer alternative for observability use cases.

Debugging Notes

# View complete module information
modinfo nvme

# Check kernel log for module loading errors
dmesg | grep -E "(module|insmod|modprobe)" | tail -20

# Show all exported symbols of a loaded module
cat /proc/kallsyms | grep "\[my_module\]"

# Track module load/unload events with ftrace
echo 1 > /sys/kernel/debug/tracing/events/module/module_load/enable
cat /sys/kernel/debug/tracing/trace_pipe

# Dump module parameters
ls /sys/module/e1000e/parameters/

# Check if module loading is disabled (lockdown mode)
cat /sys/kernel/security/lockdown

Common error codes: - ENOEXEC (Exec format error): vermagic mismatch, symbol not found, or bad ELF - EBUSY (Device busy): refcount > 0, module in use - ENOMEM: kernel memory allocation failed - EPERM (Operation not permitted): missing CAP_SYS_MODULE, or Secure Boot lockdown

Performance Implications

Module loading is synchronous and takes O(ms) for simple modules, up to hundreds of ms for complex drivers that initialize hardware. systemd-modules-load.service loads modules listed in /etc/modules-load.d/ at boot, potentially serializing the boot sequence.

The module reference counting path (try_module_get / module_put) is on the hot path for system calls that dispatch through module-provided function pointers (e.g., filesystem operations, socket operations). This is a single atomic increment/decrement, which is cheap on x86 but can cause cacheline bouncing on multi-socket NUMA systems under very high syscall rates.

Failure Modes

  • Module taints the kernel: loading a proprietary or out-of-tree module sets the TAINTED flag in the kernel, visible in panic output and cat /proc/sys/kernel/tainted. This warns that the module may be responsible for instability. Distro support is often declined for tainted kernels.
  • Symbol version mismatch (CONFIG_MODVERSIONS): if a module was compiled against kernel headers where schedule() had one signature, but the running kernel has a different signature, loading is refused even if the version string matches.
  • Double-free in module_exit: if cleanup code frees a resource that was never allocated (e.g., init failed partway through), kernel memory corruption results. KASAN detects this.

Modern Usage

The trend is toward reducing LKM usage: - eBPF replaces custom modules for tracing, filtering, and lightweight protocol handling - io_uring handles many I/O patterns previously needing custom kernel code - Rust modules: Linux 6.1 merged Rust language support. The first real Rust driver (drivers/gpu/nova/, a NVIDIA GSP firmware driver) appeared in 6.11. Rust's borrow checker prevents an entire class of use-after-free and memory safety bugs at compile time.

Future Directions

Module isolation (proposed but not merged): giving each module a separate virtual address range (using VM isolation) so a bug in one driver cannot corrupt another driver's data. Comparable to Driver Isolation in Windows (IsolatedDrivers in WDK).

Kernel livepatch (livepatch.ko): the production alternative to module reload for security fixes. Livepatching replaces individual kernel functions at runtime using ftrace trampolines, allowing CVE patches without rebooting. Red Hat's kpatch and SUSE's kGraft merged as CONFIG_LIVEPATCH in Linux 4.0.

Exercises

  1. Write a module that creates a /proc entry and returns the current system uptime when read. Use proc_create() and seq_file interface.
  2. Add a module_param for a log level and use pr_debug, pr_info, pr_err to print at different levels.
  3. Intentionally cause a module to fail to load by making module_init return -ENOMEM. Observe the kernel log.
  4. Use modprobe --dump-modversions on two modules to compare their symbol hash tables. Try loading a module after manually corrupting its vermagic with a hex editor — observe the rejection.
  5. Write a module that uses EXPORT_SYMBOL_GPL to export a function, then write a second module that calls it. Observe what happens when you mark the second module MODULE_LICENSE("Proprietary").

References

  • Documentation/kbuild/modules.rst — Linux kernel documentation on external modules
  • Documentation/core-api/kernel-api.rst — EXPORT_SYMBOL documentation
  • include/linux/module.h — module macro definitions
  • Linux Device Drivers, 3rd Edition, Chapter 2 — Corbet, Rubini, Kroah-Hartman
  • Jonathan Corbet, "Signed kernel modules" — LWN.net, 2012
  • Greg Kroah-Hartman, "Kernel module signing" — kernel.org
  • Documentation/admin-guide/module-signing.rst
  • DKMS documentation: https://github.com/dell/dkms