Section 46: Labs — Overview

Purpose and Scope

This section is the hands-on laboratory catalog for the systems knowledge archive. Labs are the mechanism by which conceptual understanding is converted into systems intuition. The catalog covers exercises in twelve domains: kernel building and configuration, kernel module writing, device driver development, kernel debugging, performance analysis, security exploitation and defense, systems networking, distributed systems, filesystem implementation, scheduler implementation, hypervisor basics, and eBPF programming. Each lab entry specifies objectives, environment requirements, difficulty rating (1-5), estimated time, prerequisites, and step-by-step setup instructions.

The lab philosophy is that a practitioner who cannot debug a kernel oops, profile a syscall-heavy workload, or trace a TCP connection through the kernel networking stack does not deeply understand those subjects. Reading and lab work are complementary, not substitutable. This catalog is designed to be the practical complement to every conceptual section in the archive.

Prerequisites

Access to a machine capable of running virtual machines (8GB RAM minimum, 16GB recommended)
Linux host environment (Ubuntu 22.04 LTS or Fedora 38+ recommended; macOS with UTM/QEMU works)
Basic comfort with the Linux command line, git, make, and gcc
Section-specific prerequisites noted per lab

Learning Objectives

Upon completing this section, the reader will be able to:

Set up a complete kernel development environment with QEMU, GDB, and symbol loading
Navigate the lab catalog and select appropriate exercises for their current skill level
Build and boot a custom Linux kernel with debug configuration
Write, load, and debug a kernel module without crashing the host system
Use profiling tools (perf, bpftrace, ftrace) to answer performance questions about real workloads
Set up a multi-node distributed system testbed using containers or VMs

Environment Setup Architecture

RECOMMENDED LAB ENVIRONMENT
==============================

  HOST MACHINE
  ┌──────────────────────────────────────────────────────────┐
  │  Ubuntu 22.04 LTS or Fedora 38+                          │
  │                                                          │
  │  ┌────────────────────┐   ┌─────────────────────────┐   │
  │  │   QEMU VM          │   │   Docker / podman       │   │
  │  │   (kernel dev)     │   │   (distributed labs)    │   │
  │  │                    │   │                         │   │
  │  │  virtio-net        │   │  node1  node2  node3    │   │
  │  │  virtio-blk        │   │  (simulated cluster)    │   │
  │  │  9p filesystem     │   └─────────────────────────┘   │
  │  │  (shared source)   │                                  │
  │  └────────────────────┘   ┌─────────────────────────┐   │
  │                           │   QEMU with KVM          │   │
  │  GDB remote stub  <──>    │   (hypervisor labs)      │   │
  │  (kernel debugging)       └─────────────────────────┘   │
  └──────────────────────────────────────────────────────────┘

  ALTERNATIVE: Cloud VM (AWS c5.2xlarge or equivalent)
  ┌──────────────────────────┐
  │  Nested virtualization   │
  │  enabled EC2 instance    │
  │  (metal or c5n.metal)    │
  └──────────────────────────┘

KERNEL DEV WORKFLOW
====================

  git clone kernel ──> make defconfig / menuconfig
          |
          v
  make -j$(nproc) ──> arch/x86/boot/bzImage
          |
          v
  qemu-system-x86_64  ──> boot kernel
  -kernel bzImage          -append "nokaslr console=ttyS0"
  -initrd rootfs.cpio.gz
          |
          v
  GDB: target remote :1234 ──> breakpoints in kernel code
  lx-symbols loadable module debug

Lab Catalog

Category 1: Kernel Building (Difficulty 1-2)

Lab KRN-01: Build and Boot a Minimal Linux Kernel - Objective: Understand kernel configuration and build system; boot a custom kernel under QEMU - Difficulty: 1/5 | Time: 4 hours - Prerequisites: Linux command line, make, gcc - Deliverable: Custom kernel booting to shell prompt under QEMU

Lab KRN-02: Kernel Configuration Deep Dive - Objective: Map menuconfig options to kernel source files; understand Kconfig system - Difficulty: 2/5 | Time: 3 hours - Prerequisites: KRN-01

Lab KRN-03: Cross-Compilation for ARM - Objective: Build Linux kernel for ARM Raspberry Pi; boot under QEMU ARM emulation - Difficulty: 2/5 | Time: 5 hours

Category 2: Kernel Modules (Difficulty 2-3)

Lab MOD-01: Hello World Kernel Module - Objective: Write, build, load, and unload a basic kernel module using init/exit hooks - Difficulty: 2/5 | Time: 2 hours - Prerequisites: KRN-01, C programming

Lab MOD-02: Proc Filesystem Interface - Objective: Create a kernel module that exposes information through /proc - Difficulty: 2/5 | Time: 3 hours - Prerequisites: MOD-01

Lab MOD-03: Kernel Timers and Workqueues - Objective: Implement deferred work; understand bottom-half processing - Difficulty: 3/5 | Time: 4 hours - Prerequisites: MOD-01

Lab MOD-04: Netfilter Hook Module - Objective: Intercept and analyze network packets from kernel module - Difficulty: 3/5 | Time: 5 hours - Prerequisites: MOD-01, basic networking knowledge

Category 3: Device Drivers (Difficulty 3-4)

Lab DRV-01: Character Device Driver - Objective: Implement a character device with read/write/ioctl; register with cdev subsystem - Difficulty: 3/5 | Time: 6 hours - Prerequisites: MOD-01

Lab DRV-02: Platform Device and Device Tree - Objective: Write a platform driver; understand device tree binding - Difficulty: 3/5 | Time: 6 hours - Prerequisites: DRV-01

Lab DRV-03: USB Device Driver - Objective: Write a USB device driver using the USB core; handle URBs - Difficulty: 4/5 | Time: 8 hours - Prerequisites: DRV-01

Lab DRV-04: Virtio Driver Implementation - Objective: Understand virtio ring protocol; implement a minimal virtio driver - Difficulty: 4/5 | Time: 10 hours - Prerequisites: DRV-01, KRN-01

Category 4: Kernel Debugging (Difficulty 3-4)

Lab DBG-01: GDB Kernel Debugging - Objective: Set up GDB remote debugging; set breakpoints in kernel code; inspect kernel data structures - Difficulty: 3/5 | Time: 4 hours - Prerequisites: KRN-01, GDB familiarity

Lab DBG-02: Analyzing a Kernel Oops - Objective: Deliberately trigger a kernel oops; decode the call stack; identify root cause - Difficulty: 3/5 | Time: 3 hours - Prerequisites: MOD-01, DBG-01

Lab DBG-03: KASAN and Memory Error Detection - Objective: Enable KASAN; trigger use-after-free in a module; interpret KASAN report - Difficulty: 3/5 | Time: 4 hours - Prerequisites: MOD-01

Lab DBG-04: ftrace and Function Graph Tracer - Objective: Use ftrace to trace kernel function calls; build a call graph for a system call path - Difficulty: 4/5 | Time: 5 hours - Prerequisites: KRN-01

Category 5: Performance Analysis (Difficulty 3-4)

Lab PERF-01: perf stat and record/report - Objective: Profile a CPU-bound workload; identify hot functions; interpret IPC and cache miss rates - Difficulty: 3/5 | Time: 4 hours - Prerequisites: KRN-01, perf installed

Lab PERF-02: Flame Graph Generation - Objective: Generate on-CPU and off-CPU flame graphs; identify bottlenecks in a multi-threaded workload - Difficulty: 3/5 | Time: 4 hours - Prerequisites: PERF-01

Lab PERF-03: bpftrace for System Tracing - Objective: Write bpftrace one-liners and scripts; trace syscall latency, block I/O, scheduler events - Difficulty: 3/5 | Time: 5 hours - Prerequisites: KRN-01, basic eBPF understanding

Lab PERF-04: NUMA Performance Analysis - Objective: Use numactl and perf to identify NUMA imbalance; measure remote vs local memory access latency - Difficulty: 4/5 | Time: 6 hours - Prerequisites: PERF-01, multi-NUMA system or emulation

Category 6: Security Labs (Difficulty 3-5)

Lab SEC-01: Stack Buffer Overflow Exploitation (No ASLR) - Objective: Exploit a stack buffer overflow in a toy program; understand ret2shellcode - Difficulty: 3/5 | Time: 4 hours | Environment: VM with ASLR/NX disabled

Lab SEC-02: Return-Oriented Programming - Objective: Build a ROP chain against a binary with NX enabled; understand gadget chaining - Difficulty: 4/5 | Time: 8 hours - Prerequisites: SEC-01

Lab SEC-03: Kernel Privilege Escalation - Objective: Exploit a deliberately vulnerable kernel module to achieve privilege escalation; understand SMEP/SMAP - Difficulty: 5/5 | Time: 12 hours - Prerequisites: MOD-01, SEC-02, DBG-01

Lab SEC-04: eBPF Security with seccomp-bpf - Objective: Write a seccomp-bpf filter; restrict system calls for a process; integrate with libseccomp - Difficulty: 3/5 | Time: 4 hours

Category 7: Networking (Difficulty 3-4)

Lab NET-01: Packet Capture and Analysis - Objective: Capture packets with tcpdump/Wireshark; trace a TCP connection establishment; analyze retransmits - Difficulty: 2/5 | Time: 3 hours

Lab NET-02: Raw Socket Programming - Objective: Write a raw socket program that sends/receives Ethernet frames directly - Difficulty: 3/5 | Time: 5 hours

Lab NET-03: XDP Program - Objective: Write an XDP program that drops packets matching a filter at line rate - Difficulty: 4/5 | Time: 6 hours - Prerequisites: eBPF basics, NET-01

Lab NET-04: tc eBPF Traffic Shaping - Objective: Use tc with eBPF to implement traffic classification and rate limiting - Difficulty: 4/5 | Time: 6 hours - Prerequisites: NET-03

Category 8: Distributed Systems (Difficulty 3-5)

Lab DIST-01: Vector Clock Implementation - Objective: Implement vector clocks; demonstrate causal ordering; show scenarios where Lamport clocks fail - Difficulty: 3/5 | Time: 4 hours

Lab DIST-02: Leader Election with Bully Algorithm - Objective: Implement bully algorithm leader election; simulate node failures - Difficulty: 3/5 | Time: 5 hours

Lab DIST-03: Raft Consensus Implementation - Objective: Implement Raft leader election and log replication; pass the MIT 6.824 test suite - Difficulty: 5/5 | Time: 40+ hours - Prerequisites: DIST-01, DIST-02

Category 9: Filesystem Implementation (Difficulty 4-5)

Lab FS-01: FUSE Filesystem - Objective: Implement a simple filesystem using FUSE; support create/read/write/delete - Difficulty: 4/5 | Time: 10 hours

Lab FS-02: Toy Ext2-Compatible Filesystem - Objective: Implement ext2 inode, block group, and directory structures in a file-backed block device - Difficulty: 5/5 | Time: 20+ hours - Prerequisites: FS-01

Category 10: Scheduler Implementation (Difficulty 4-5)

Lab SCHED-01: User-Space Thread Scheduler - Objective: Implement a round-robin scheduler for user-space threads using setjmp/longjmp or ucontext - Difficulty: 4/5 | Time: 8 hours

Lab SCHED-02: Priority Scheduler with Aging - Objective: Extend SCHED-01 with priority levels and aging to prevent starvation - Difficulty: 4/5 | Time: 6 hours - Prerequisites: SCHED-01

File Map

46-labs/
├── 00-overview.md                    ← This file
├── 01-environment-setup.md
├── 02-vm-qemu-setup.md
├── 03-container-lab-environment.md
├── 04-kernel-build-labs.md
├── 05-module-writing-labs.md
├── 06-driver-writing-labs.md
├── 07-debugging-labs.md
├── 08-performance-analysis-labs.md
├── 09-security-labs.md
├── 10-networking-labs.md
├── 11-distributed-systems-labs.md
├── 12-filesystem-labs.md
├── 13-scheduler-labs.md
└── 14-lab-solutions-and-hints.md

Cross-References

Section 45 (Learning Roadmaps): each track references specific labs in this catalog
Section 47 (Projects): labs are prerequisites for the larger project-based exercises
Section 24 (Debugging): debugging techniques applied in DBG-series labs
Section 25 (Performance Engineering): methodology behind PERF-series labs
Section 26 (Security): exploit techniques practiced in SEC-series labs