Skip to content

Toy Kernel Project: Build a Minimal x86-64 Kernel

A complete project guide for building a minimal x86-64 kernel from scratch. By the end, the kernel boots in QEMU, outputs text to serial and VGA, handles hardware interrupts, manages physical and virtual memory, and can run a userspace process. This project is the best possible complement to reading about kernel internals — you will understand every layer because you built it.

Project Overview

Phase What You Build Milestone
Phase 1: Booting Multiboot2 entry, GDT, long mode, VGA text "Hello from kernel" on screen
Phase 2: Interrupts IDT, PIC, keyboard, timer Keyboard input echoes to screen
Phase 3: Memory Buddy allocator, page tables, kmalloc kmalloc(4096) works correctly
Phase 4: Userspace Ring 3 entry, syscall, simple shell /bin/hello runs in userspace

Estimated time: 3–6 months of evenings/weekends

Reference implementations: - xv6 (MIT): https://github.com/mit-pdos/xv6-riscv (conceptual reference — different arch) - OSDev Wiki: https://wiki.osdev.org (your primary reference for x86 specifics) - Writing an OS in Rust (blog series): https://os.phil-opp.com (excellent Rust alternative)

Toolchain Setup

Required Tools

# Install cross-compiler targeting x86_64 bare metal (no OS, no libc)
sudo apt install gcc-multilib nasm qemu-system-x86_64 grub-pc-bin xorriso

# Build cross-compiler from source (recommended for serious work)
# Using osdev cross-compiler guide: https://wiki.osdev.org/GCC_Cross-Compiler
export TARGET=x86_64-elf
export PREFIX="$HOME/opt/cross"
export PATH="$PREFIX/bin:$PATH"

# After building cross-compiler:
x86_64-elf-gcc --version

Build System Setup

# Top-level Makefile
TARGET   := x86_64-elf
CC       := $(TARGET)-gcc
AS       := nasm
LD       := $(TARGET)-ld
OBJCOPY  := $(TARGET)-objcopy

CFLAGS   := -O0 -g -ffreestanding -fno-stack-protector -mno-red-zone \
            -mno-mmx -mno-sse -mno-sse2 -Wall -Wextra \
            -I src/include

ASFLAGS  := -f elf64

LDFLAGS  := -T src/link.ld -nostdlib

OBJS     := build/boot.o build/kernel_main.o build/vga.o \
            build/gdt.o build/idt.o build/isr.o \
            build/pic.o build/mm.o build/vmm.o build/kmalloc.o

.PHONY: all qemu clean

all: kernel.elf iso/boot/kernel.elf
    grub-mkrescue -o myos.iso iso/

kernel.elf: $(OBJS)
    $(LD) $(LDFLAGS) -o $@ $^

build/%.o: src/%.asm
    $(AS) $(ASFLAGS) -o $@ $<

build/%.o: src/%.c
    $(CC) $(CFLAGS) -c -o $@ $<

qemu: all
    qemu-system-x86_64 -cdrom myos.iso \
        -serial stdio \
        -m 128M \
        -d int,cpu_reset \
        -no-reboot

clean:
    rm -f build/*.o kernel.elf myos.iso

Linker Script (src/link.ld)

ENTRY(_start)
SECTIONS {
    . = 1M;          /* Load at 1 MB physical — standard convention */

    .text ALIGN(4K) : {
        *(.multiboot2)   /* Multiboot header must be in first 32KB */
        *(.text)
    }

    .rodata ALIGN(4K) : { *(.rodata*) }

    .data ALIGN(4K) : { *(.data) }

    .bss ALIGN(4K) : {
        *(COMMON)
        *(.bss)
    }

    /* Symbols for kernel end — used by memory manager */
    kernel_start = LOADADDR(.text);
    kernel_end   = .;
}

Phase 1: Booting

Step 1.1: Multiboot2 Header and Assembly Entry

The bootloader (GRUB) loads your kernel and jumps to _start. Before that, GRUB scans the first 32 KB of your binary for the Multiboot2 magic number.

; src/boot.asm
section .multiboot2
header_start:
    dd 0xe85250d6                          ; Multiboot2 magic
    dd 0                                   ; Architecture: x86 protected mode
    dd header_end - header_start           ; Header length
    dd -(0xe85250d6 + 0 + (header_end - header_start))  ; Checksum

    ; End tag (required)
    dw 0    ; type
    dw 0    ; flags
    dd 8    ; size
header_end:

section .bss
align 16
stack_bottom:
    resb 16384          ; 16 KB initial stack
stack_top:

section .text
global _start
_start:
    ; GRUB left us in 32-bit protected mode
    ; Disable interrupts immediately
    cli

    ; Set up stack
    mov esp, stack_top

    ; Save multiboot info pointer (ebx) before we clobber registers
    push ebx
    push eax    ; magic = 0x36d76289 for Multiboot2

    ; Check for CPUID support, SSE, and set up for 64-bit
    ; (detailed steps: enable PAE, set up page tables, enable long mode)
    call setup_long_mode    ; sets up minimal page tables + enables long mode

    ; Load 64-bit GDT
    lgdt [gdt64.pointer]

    ; Far jump to flush pipeline and enter 64-bit code segment
    jmp gdt64.code:long_mode_start

    ; Never reached
    hlt

; --- 64-bit code begins here ---
bits 64
long_mode_start:
    ; Initialize segment registers to 64-bit data segment
    mov ax, gdt64.data
    mov ss, ax
    mov ds, ax
    mov es, ax

    ; Pop multiboot args and call C kernel entry
    pop rdi    ; arg1 = multiboot magic
    pop rsi    ; arg2 = multiboot info pointer
    call kernel_main

    ; If kernel_main returns, halt
.halt:
    cli
    hlt
    jmp .halt

Step 1.2: GDT Setup

/* src/gdt.c */
#include "gdt.h"

/* 64-bit GDT: null descriptor, code segment, data segment */
struct gdt_entry {
    uint16_t limit_low;
    uint16_t base_low;
    uint8_t  base_mid;
    uint8_t  access;
    uint8_t  granularity;
    uint8_t  base_high;
} __attribute__((packed));

struct gdt_pointer {
    uint16_t limit;
    uint64_t base;
} __attribute__((packed));

static struct gdt_entry gdt[3];
static struct gdt_pointer gdt_ptr;

static void gdt_set_entry(int i, uint32_t base, uint32_t limit,
                          uint8_t access, uint8_t gran)
{
    gdt[i].base_low    = base & 0xFFFF;
    gdt[i].base_mid    = (base >> 16) & 0xFF;
    gdt[i].base_high   = (base >> 24) & 0xFF;
    gdt[i].limit_low   = limit & 0xFFFF;
    gdt[i].granularity = ((limit >> 16) & 0x0F) | (gran & 0xF0);
    gdt[i].access      = access;
}

void gdt_init(void)
{
    gdt_set_entry(0, 0, 0, 0, 0);                  /* null */
    gdt_set_entry(1, 0, 0xFFFFF, 0x9A, 0xA0);      /* 64-bit code: L=1, D=0 */
    gdt_set_entry(2, 0, 0xFFFFF, 0x92, 0xC0);      /* 64-bit data */

    gdt_ptr.limit = sizeof(gdt) - 1;
    gdt_ptr.base  = (uint64_t)&gdt;

    /* Reload GDT and segment registers */
    asm volatile("lgdt %0" : : "m"(gdt_ptr));
}

Step 1.3: VGA Text Output

/* src/vga.c */
#include <stdint.h>
#include <stddef.h>

#define VGA_ADDR  0xB8000
#define VGA_COLS  80
#define VGA_ROWS  25

static uint16_t *const vga = (uint16_t *)VGA_ADDR;
static int col = 0, row = 0;

static inline uint16_t vga_char(char c, uint8_t color) {
    return (uint16_t)c | ((uint16_t)color << 8);
}

void vga_clear(void) {
    for (int i = 0; i < VGA_ROWS * VGA_COLS; i++)
        vga[i] = vga_char(' ', 0x07);
    col = row = 0;
}

void vga_putchar(char c) {
    if (c == '\n') { col = 0; row++; return; }
    if (col >= VGA_COLS) { col = 0; row++; }
    if (row >= VGA_ROWS) { /* TODO: scroll */ row = VGA_ROWS - 1; }
    vga[row * VGA_COLS + col] = vga_char(c, 0x0F);  /* white on black */
    col++;
}

void vga_print(const char *s) {
    while (*s) vga_putchar(*s++);
}

Phase 1 Kernel Main

/* src/kernel_main.c */
#include "vga.h"
#include "gdt.h"
#include "serial.h"

void kernel_main(uint32_t mb_magic, void *mb_info)
{
    vga_clear();
    gdt_init();

    vga_print("Kernel booted!\n");
    vga_print("Multiboot2 magic: ");
    /* print hex magic... */

    serial_init();          /* UART 8250 — for GDB and logging */
    serial_print("Hello from kernel (serial)\n");

    /* Halt */
    for (;;) asm("hlt");
}

Milestone 1 Test:

make qemu
# QEMU window should show: "Kernel booted!"
# stdio should show: "Hello from kernel (serial)"

Phase 2: Interrupts

Step 2.1: IDT Setup

/* src/idt.c — abbreviated */
struct idt_entry {
    uint16_t offset_low;
    uint16_t selector;      /* kernel code segment */
    uint8_t  ist;           /* interrupt stack table — 0 for now */
    uint8_t  type_attr;     /* 0x8E = interrupt gate, ring 0, present */
    uint16_t offset_mid;
    uint32_t offset_high;
    uint32_t zero;
} __attribute__((packed));

#define IDT_SIZE 256
static struct idt_entry idt[IDT_SIZE];

void idt_set_gate(int n, uint64_t handler, uint16_t sel, uint8_t flags)
{
    idt[n].offset_low  = handler & 0xFFFF;
    idt[n].offset_mid  = (handler >> 16) & 0xFFFF;
    idt[n].offset_high = (handler >> 32) & 0xFFFFFFFF;
    idt[n].selector    = sel;
    idt[n].type_attr   = flags;
    idt[n].ist = idt[n].zero = 0;
}

Step 2.2: PIC (8259A) Initialization

/* src/pic.c */
#define PIC1_CMD  0x20
#define PIC1_DATA 0x21
#define PIC2_CMD  0xA0
#define PIC2_DATA 0xA1
#define PIC_EOI   0x20

void pic_init(void)
{
    /* ICW1: start init sequence, edge-triggered, cascade */
    outb(PIC1_CMD, 0x11);
    outb(PIC2_CMD, 0x11);
    /* ICW2: remap to IRQ 32-47 (avoid conflict with CPU exceptions 0-31) */
    outb(PIC1_DATA, 32);    /* IRQ0-7  → INT 32-39 */
    outb(PIC2_DATA, 40);    /* IRQ8-15 → INT 40-47 */
    /* ICW3: cascade identity */
    outb(PIC1_DATA, 0x04);
    outb(PIC2_DATA, 0x02);
    /* ICW4: 8086 mode */
    outb(PIC1_DATA, 0x01);
    outb(PIC2_DATA, 0x01);
    /* Unmask all IRQs */
    outb(PIC1_DATA, 0x00);
    outb(PIC2_DATA, 0x00);
}

void pic_send_eoi(uint8_t irq)
{
    if (irq >= 8) outb(PIC2_CMD, PIC_EOI);
    outb(PIC1_CMD, PIC_EOI);
}

Milestone 2 Test: After loading the IDT and enabling interrupts (sti), pressing a key on the QEMU keyboard should call your keyboard IRQ handler (IRQ1 = INT 33).

Phase 3: Memory Management

Step 3.1: Physical Memory Manager (Bitmap Allocator)

/* src/pmm.c — physical memory manager */
#include <stdint.h>
#include <stddef.h>

#define PAGE_SIZE    4096
#define BITMAP_BITS  (1024 * 1024)  /* manage 4 GB = 1M pages */

static uint8_t bitmap[BITMAP_BITS / 8];
static uint64_t total_pages;
static uint64_t free_pages;

static inline void bitmap_set(uint64_t page) {
    bitmap[page / 8] |= (1 << (page % 8));
}

static inline void bitmap_clear(uint64_t page) {
    bitmap[page / 8] &= ~(1 << (page % 8));
}

static inline int bitmap_test(uint64_t page) {
    return (bitmap[page / 8] >> (page % 8)) & 1;
}

void pmm_init(uint64_t mem_size_bytes, uint64_t kernel_end_phys)
{
    /* Mark all memory as used initially */
    total_pages = mem_size_bytes / PAGE_SIZE;
    for (uint64_t i = 0; i < total_pages / 8; i++)
        bitmap[i] = 0xFF;

    /* Mark available memory as free (parse Multiboot2 memory map) */
    /* For simplicity, mark everything above kernel_end as free */
    uint64_t free_start = (kernel_end_phys + PAGE_SIZE - 1) / PAGE_SIZE;
    for (uint64_t i = free_start; i < total_pages; i++)
        bitmap_clear(i);

    free_pages = total_pages - free_start;
}

uint64_t pmm_alloc_page(void)
{
    for (uint64_t i = 0; i < total_pages; i++) {
        if (!bitmap_test(i)) {
            bitmap_set(i);
            free_pages--;
            return i * PAGE_SIZE;
        }
    }
    return 0;   /* out of memory */
}

void pmm_free_page(uint64_t phys_addr)
{
    uint64_t page = phys_addr / PAGE_SIZE;
    bitmap_clear(page);
    free_pages++;
}

Step 3.2: Virtual Memory (4-Level Page Tables)

/* src/vmm.c — virtual memory manager */
#define PML4_INDEX(va) (((va) >> 39) & 0x1FF)
#define PDPT_INDEX(va) (((va) >> 30) & 0x1FF)
#define PD_INDEX(va)   (((va) >> 21) & 0x1FF)
#define PT_INDEX(va)   (((va) >> 12) & 0x1FF)

#define PAGE_PRESENT  (1ULL << 0)
#define PAGE_WRITABLE (1ULL << 1)
#define PAGE_USER     (1ULL << 2)

typedef uint64_t page_table_t[512];

extern page_table_t pml4;   /* set up in boot.asm */

void vmm_map_page(uint64_t virt, uint64_t phys, uint64_t flags)
{
    uint64_t *pml4e = &pml4[PML4_INDEX(virt)];

    /* Allocate PDPT if not present */
    if (!(*pml4e & PAGE_PRESENT)) {
        uint64_t pdpt_phys = pmm_alloc_page();
        memset((void *)pdpt_phys, 0, PAGE_SIZE);
        *pml4e = pdpt_phys | PAGE_PRESENT | PAGE_WRITABLE;
    }

    /* Walk down: PDPT → PD → PT */
    uint64_t *pdpt = (uint64_t *)(*pml4e & ~0xFFF);
    uint64_t *pdpte = &pdpt[PDPT_INDEX(virt)];
    if (!(*pdpte & PAGE_PRESENT)) {
        uint64_t pd_phys = pmm_alloc_page();
        memset((void *)pd_phys, 0, PAGE_SIZE);
        *pdpte = pd_phys | PAGE_PRESENT | PAGE_WRITABLE;
    }

    uint64_t *pd = (uint64_t *)(*pdpte & ~0xFFF);
    uint64_t *pde = &pd[PD_INDEX(virt)];
    if (!(*pde & PAGE_PRESENT)) {
        uint64_t pt_phys = pmm_alloc_page();
        memset((void *)pt_phys, 0, PAGE_SIZE);
        *pde = pt_phys | PAGE_PRESENT | PAGE_WRITABLE;
    }

    uint64_t *pt = (uint64_t *)(*pde & ~0xFFF);
    pt[PT_INDEX(virt)] = phys | flags | PAGE_PRESENT;

    /* Flush TLB entry */
    asm volatile("invlpg (%0)" : : "r"(virt) : "memory");
}

Milestone 3 Test:

/* In kernel_main, test the allocator */
uint64_t page1 = pmm_alloc_page();
uint64_t page2 = pmm_alloc_page();
vga_print("page1: "); print_hex(page1);
vga_print("page2: "); print_hex(page2);
/* They should be different pages, 4096 apart */
pmm_free_page(page1);
uint64_t page3 = pmm_alloc_page();
/* page3 should equal page1 — reused the freed page */

Phase 4: Userspace

Step 4.1: Enter Ring 3

To jump to ring 3, use iretq with a fabricated interrupt frame:

/* src/userspace.c */
#define USER_CODE_SEG 0x1B   /* selector: GDT index 3, RPL=3 */
#define USER_DATA_SEG 0x23   /* selector: GDT index 4, RPL=3 */
#define RFLAGS_IF     0x200  /* interrupt enable flag */

void enter_userspace(uint64_t rip, uint64_t rsp)
{
    /* iretq pops: RIP, CS, RFLAGS, RSP, SS in that order */
    asm volatile(
        "push %[ss]    \n"  /* SS */
        "push %[rsp]   \n"  /* RSP (user stack) */
        "push %[rfl]   \n"  /* RFLAGS */
        "push %[cs]    \n"  /* CS */
        "push %[rip]   \n"  /* RIP (entry point) */
        "iretq         \n"
        :
        : [ss]  "r"((uint64_t)USER_DATA_SEG),
          [rsp] "r"(rsp),
          [rfl] "r"((uint64_t)RFLAGS_IF),
          [cs]  "r"((uint64_t)USER_CODE_SEG),
          [rip] "r"(rip)
    );
}

Step 4.2: Syscall Handler

/* src/syscall.c */
#define SYSCALL_WRITE  1
#define SYSCALL_EXIT   60

void syscall_handler(uint64_t number, uint64_t arg1, uint64_t arg2, uint64_t arg3)
{
    switch (number) {
    case SYSCALL_WRITE:
        /* arg1 = fd, arg2 = buf, arg3 = len */
        /* For now: write to VGA */
        vga_print((const char *)arg2);
        break;
    case SYSCALL_EXIT:
        vga_print("\n[process exited]\n");
        for (;;) asm("hlt");
        break;
    default:
        vga_print("[unknown syscall]\n");
    }
}

Milestone 4 Test: A userspace program compiled and embedded in the kernel image should execute and print "Hello from userspace!" via the write syscall.

Debugging Workflow

GDB with QEMU

# Start QEMU with GDB stub
qemu-system-x86_64 -cdrom myos.iso -s -S -m 128M -serial stdio

# In another terminal
x86_64-elf-gdb kernel.elf
(gdb) target remote :1234
(gdb) break kernel_main
(gdb) continue
# Execution stops at kernel_main
(gdb) info registers
(gdb) x/10i $rip    # disassemble next 10 instructions
(gdb) x/8gx $rsp    # dump stack

Common Boot Failures

Symptom Cause Debug Step
QEMU shows "Booting from CD-ROM" then blank Multiboot header not found Check header is in first 32KB; verify checksum
Immediate triple fault GDT not set up before long mode jump Use QEMU -d int,cpu_reset to see fault vector
Triple fault after IDT load Handler addresses wrong in IDT entries Dump IDT entries with GDB before lidt
Page fault on first memory access Page tables not covering accessed address Map wider range in boot page tables
General protection fault entering ring 3 Segment descriptor flags wrong Verify DPL=3 in user code/data segments

Milestone Checkpoints

Milestone How to Verify
M1: Boots QEMU window shows "Kernel booted!"
M2: Interrupts Keypress echoes to screen; timer increments counter
M3: Memory pmm_alloc_page returns non-overlapping addresses; vmm_map_page doesn't triple fault
M4: Userspace Embedded user_hello.elf runs and prints via syscall

Suggested Extensions

Extension Skills Developed
Implement fork() Copy page tables, duplicate process state
Implement exec() Parse ELF binary, load segments into fresh address space
Add a simple round-robin scheduler timer_handler switches current_task
Implement read() + keyboard buffer Blocking syscall, interrupt-driven wakeup
Add ext2 read-only filesystem on a disk image Block driver (ATA PIO), inode parsing