07 — Rust Async Runtime

Overview

Rust's async/await system is one of the most sophisticated and unusual concurrency models in mainstream programming. It achieves true zero-cost abstraction: an async function compiles to a state machine on the stack with no heap allocation required, no garbage collector involvement, and no runtime thread per task. The resulting code runs at speeds comparable to hand-written state machines while reading like sequential blocking code. However, this power comes with significant complexity — understanding Pin, Waker, the Future trait's poll model, and the distinction between async code and the executor that drives it are prerequisites for writing correct async Rust.

This document covers the mechanics from the ground up, explores Tokio as the dominant production executor, and compares the model against goroutines (Go) and OS threads to illuminate the tradeoffs that govern where each model excels.

Prerequisites

• Rust ownership and borrowing (see 44-rust-and-memory-safety/01-ownership-model.md)
• Understanding of event loops and non-blocking I/O (see 15-networking/04-epoll-kqueue.md)
• Familiarity with state machines and coroutines (see 08-threading-models/03-fibers-and-coroutines.md)
• Basic knowledge of Rust generics and traits

Historical Context

Rust's async story evolved significantly before stabilizing. The original pre-1.0 libgreen provided M:N green threads, but it was removed before Rust 1.0 (2015) because the zero-cost abstraction philosophy conflicted with mandatory runtime overhead. For years, async Rust was handled through callback-based futures in the futures 0.1 crate, which was verbose and difficult to compose.

The async/await syntax was designed through a multi-year RFC process. std::future::Future was stabilized in Rust 1.36 (2019). The async fn and .await syntax was stabilized in Rust 1.39 (November 2019). This relative lateness — compared to C# (2012), Python (2015), JavaScript (2017) — meant Rust's async design incorporated lessons from all previous implementations, resulting in a more correct but more complex model.

The Future Trait

A Future in Rust represents a computation that may not have completed yet. The trait definition:

pub trait Future {
    type Output;
    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

pub enum Poll<T> {
    Ready(T),
    Pending,
}

This is a pull model: the executor calls poll to drive the future forward. The future either completes (Ready(value)) or indicates it is not ready (Pending) and registers a Waker via the Context to be notified when it should be polled again. This is fundamentally different from callbacks (push model) — the executor controls when computation happens.

State Machine Transformation

When you write:

async fn read_and_echo(stream: TcpStream) -> io::Result<()> {
    let mut buf = [0u8; 1024];
    let n = stream.read(&mut buf).await?;
    stream.write_all(&buf[..n]).await?;
    Ok(())
}

The compiler transforms this into an enum (state machine) roughly equivalent to:

enum ReadAndEchoState {
    Start { stream: TcpStream },
    WaitingRead { stream: TcpStream, buf: [u8; 1024], read_future: ReadFuture },
    WaitingWrite { stream: TcpStream, write_future: WriteAllFuture },
    Done,
}

The .await points become state transition boundaries. Each call to poll advances the state machine from one .await to the next, or returns Pending if the underlying I/O is not ready. No heap allocation is required — the entire state machine lives on the caller's stack or in a single allocation when spawned onto an executor.

Why Pin Is Necessary

Self-referential structs are the fundamental problem. Consider an async function that holds both a buffer and a future that borrows from that buffer:

async fn process() {
    let buf = [0u8; 1024];
    let future = some_io_operation(&buf);  // future borrows buf
    future.await;
}

The generated state machine struct holds both buf and future, where future contains a pointer into buf. If this struct is moved in memory (as would happen if you passed it around normally), the internal pointer becomes dangling.

Pin<&mut T> is a guarantee to the compiler that the value behind the pointer will not be moved. A Pin<&mut Future> tells the future: "I promise you will stay at this memory address until you are dropped." This allows the future to safely contain internal references.

Stack / Heap allocation:
+----------------------------------+
|  ReadAndEchoState (pinned here)  |
|  +----------+  +---------------+ |
|  | buf[1024]|<-| read_future   | |
|  |  (data)  |  | (ptr into buf)| |
|  +----------+  +---------------+ |
+----------------------------------+
        ^
        | Pin guarantees this address is stable

For most users, pin!() macro (stable since Rust 1.68) or Box::pin() handles pinning without manual unsafe code. The Unpin auto-trait marks types that can be safely moved even when "pinned."

Waker and Context

When a future returns Pending, it must arrange to be polled again when progress is possible. It does this via the Waker provided in Context:

fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
    if self.data_ready() {
        Poll::Ready(self.read_data())
    } else {
        // Register: when data arrives, call cx.waker().wake()
        self.register_waker(cx.waker().clone());
        Poll::Pending
    }
}

The Waker is provided by the executor. When the I/O event arrives (via epoll/kqueue), the executor's I/O driver calls waker.wake(), which schedules the task for re-polling. This is the async system's equivalent of epoll_wait notification.

Async Executors

A Future does nothing on its own — it must be polled. An executor is responsible for polling futures until they complete. Rust's standard library deliberately does not include an executor; it provides only the Future trait and Waker mechanism, allowing different executors optimized for different workloads.

Tokio: The Production Executor

Tokio is the dominant async runtime for production Rust. It powers applications at AWS (S3 internals), Discord (millions of concurrent WebSocket connections), Cloudflare (edge workers), and Databricks (query processing).

Tokio Architecture

+------------------------------------------------------------------+
|                        USER CODE                                 |
|  async fn main() { tokio::spawn(task1); tokio::spawn(task2); }  |
+------------------------------------------------------------------+
         |                              |
    spawn(future)               I/O operations
         |                         (TcpStream::read etc.)
+--------v-----------+    +----------v-----------+
|   Task Queue       |    |   I/O Driver          |
|   (per-thread      |    |   (epoll fd on Linux  |
|    local + global  |    |    kqueue on macOS    |
|    work-steal      |    |    IOCP on Windows)   |
|    queue)          |    |                       |
+--------+-----------+    +----------+------------+
         |                           |
         |    +----------------------+
         |    | Waker.wake() called
         v    v     when I/O ready
+--------+----+------+
|   Worker Threads   |   (default: num_cpus threads)
|   (OS threads,     |
|    poll futures)   |
+--------------------+
         |
+--------v-----------+
|   Timer Wheel      |   (hierarchical timing wheel)
|   (sleep, timeout) |
+--------------------+
         |
+--------v-----------+
|   Blocking Thread  |   (separate pool for blocking_in_place)
|   Pool             |   (spawn_blocking for sync code)
+--------------------+

Multi-threaded Work-Stealing Scheduler

Tokio's default runtime (tokio::runtime::Builder::new_multi_thread()) creates N worker threads (default: number of logical CPUs). Each worker has a local deque (double-ended queue) of ready tasks. When a worker runs out of local tasks, it steals from the back of another worker's deque — the classic Chase-Lev work-stealing algorithm, also used in Java's ForkJoinPool.

#[tokio::main]
async fn main() {
    let handle = tokio::spawn(async {
        // This future is scheduled on a worker thread
        tokio::time::sleep(Duration::from_millis(100)).await;
        "done"
    });
    println!("{}", handle.await.unwrap());
}

I/O Driver

Tokio registers file descriptors with epoll (Linux), kqueue (macOS/BSD), or IOCP (Windows) via the mio crate. When a TcpStream::read() would block, Tokio registers the fd with the I/O driver and returns Pending. When epoll reports the fd is readable, the I/O driver calls wake() on the associated waker, scheduling the task for re-polling.

// Tokio's TcpStream implements AsyncRead:
async fn read_some(stream: &mut TcpStream) -> Vec<u8> {
    let mut buf = vec![0u8; 4096];
    let n = stream.read(&mut buf).await.unwrap();
    buf.truncate(n);
    buf
}
// ^ Returns Pending if no data available, woken by epoll

Timer Wheel

Tokio implements a hierarchical timing wheel for tokio::time::sleep and timeout. The wheel has multiple levels (milliseconds → seconds → minutes → hours) with O(1) insertion and deletion. At each tick, expired timers call wake() on their associated tasks.

Blocking Thread Pool

Async code must not block — a blocking call in an async task prevents that worker thread from polling other tasks. Tokio provides spawn_blocking to offload blocking operations to a separate thread pool that can grow on demand:

let result = tokio::task::spawn_blocking(|| {
    // This runs on a blocking thread, not a Tokio worker thread
    std::fs::read_to_string("/etc/hosts").unwrap()
}).await.unwrap();

Structured Concurrency with JoinSet

use tokio::task::JoinSet;

async fn fetch_all(urls: Vec<String>) -> Vec<String> {
    let mut set = JoinSet::new();
    for url in urls {
        set.spawn(async move { fetch(url).await });
    }
    let mut results = Vec::new();
    while let Some(result) = set.join_next().await {
        results.push(result.unwrap());
    }
    results
}

JoinSet implements structured concurrency: all spawned tasks are cancelled when the JoinSet is dropped. This prevents task leaks — a common source of bugs in async code where spawned tasks outlive their logical scope.

Async Cancellation

In Rust async, dropping a future cancels it. If you have:

let fut = some_async_fn();
// Drop fut here — the operation is cancelled
drop(fut);

The future's destructor runs, which may drop in-progress state. This is powerful but dangerous: if some_async_fn is partway through writing to a database when dropped, the write is incomplete. This is called the "cancellation safety" problem.

// DANGEROUS: not cancellation safe
async fn write_and_flush(writer: &mut Writer, data: &[u8]) {
    writer.write_all(data).await;  // If cancelled here...
    writer.flush().await;           // ...flush never runs
}

The tokio::select! macro implicitly cancels futures when another branch completes, which requires all branches to be "cancellation safe."

Performance Comparison

Metric	Tokio async task	Go goroutine	OS thread
Memory per task	~100–400 bytes	~2–8 KB (growable)	2–8 MB (fixed)
Creation cost	~100 ns	~300 ns	~10–50 µs
Context switch cost	No OS switch needed	~100 ns	~1–10 µs (OS + TLB)
Max concurrent units	Millions	Millions	~10K (memory limited)
GC pressure	None	GC pauses possible	None
Blocking I/O handling	Explicit (spawn_blocking)	Transparent	Natural
Debug experience	Moderate (async stacks)	Good (goroutine ID)	Good (thread ID)

Other Executors

• async-std: Mirrors the std library API, auto-detects CPU count, simpler API surface than Tokio. Less widely deployed in production.
• smol: Minimalist executor (~1K lines). Good for understanding executor internals. Used in some embedded/WASM contexts.
• embassy: No-std async executor for embedded systems (microcontrollers). Tasks are statically allocated, no heap. Used in firmware on STM32, nRF52, RP2040.
• Single-threaded mini executors: For tutorials: a complete working executor in ~50 lines using thread_local waker and a simple task queue.

Debugging Notes

Async Stack Traces

Async stack traces in Rust are challenging because the call stack does not reflect the logical call chain. A future awaiting another future shows only the immediate poll site. Tokio provides tokio-console — a terminal UI that shows all running tasks, their state, and time spent:

cargo add tokio-console
# In code:
console_subscriber::init();
# Run:
tokio-console

Common Issues

• Blocking in async context: std::thread::sleep, std::fs::read, blocking mutexes (std::sync::Mutex) used inside async tasks block the worker thread. Use tokio::time::sleep, tokio::fs, tokio::sync::Mutex instead.
• Missed .await: A future returned but not awaited does nothing. The Rust compiler emits a warning for unused futures.
• Waker not registered: A custom future that returns Pending without registering the waker will never be re-polled — the task hangs indefinitely.
• Holding a non-Send type across an .await: Rc<T>, raw pointers, and types containing them cannot be sent between threads. The compiler rejects async fns that hold non-Send types across await points when used with a multi-threaded executor.

Security Implications

• Async context propagation: Request-scoped data (auth tokens, tracing spans) must be explicitly threaded through async call chains or stored in task-local storage. Unlike thread-local storage, task-local storage (tokio::task_local!) is correct for async.
• Cancellation and partial writes: Cancelled futures that have partially written to a socket, file, or database can leave inconsistent state. Security-sensitive operations (writing audit logs, completing financial transactions) must be cancellation-safe or protected by a non-cancellable wrapper.
• Executor thread pool exhaustion: spawn_blocking uses a thread pool with a default maximum of 512 threads. If blocking tasks accumulate faster than they complete, new blocking work is queued and async tasks that depend on them deadlock. This can be triggered by slow external services holding blocking threads.

Future Directions

• async fn in traits (stable Rust 1.75+): Previously required async-trait proc macro which heap-allocated every call. Native async fn in traits allows zero-cost async interfaces.
• impl Trait in function return types for async (RPIT): More flexible async API design without boxing.
• Async generators / async iterators: AsyncIterator trait in std will enable for await x in stream {} syntax.
• Better cancellation primitives: The ecosystem is developing conventions and types (like CancellationToken in tokio-util) to make structured cancellation safer.

Exercises

1. Implement a minimal single-threaded async executor from scratch: a Task struct that holds a boxed future, a run queue (VecDeque), and a Waker that re-queues the task when called. Run two cooperative tasks on it.
2. Write a Tokio TCP echo server. Measure maximum concurrent connections with wrk or hping3. Compare memory usage per connection with a thread-per-connection server handling the same load.
3. Implement a future that wraps a raw epoll file descriptor: poll registers the fd with epoll if not ready and returns Pending; the waker is called by the I/O driver. Verify it works with a UDP socket.
4. Demonstrate a cancellation-safety bug: write an async function that increments a counter, writes to a Vec, and flushes — but when cancelled mid-write the counter and Vec are inconsistent. Then fix it by making the operation atomic with respect to cancellation.
5. Use tokio-console to profile a Tokio application under load. Identify the task with the highest total poll time and the task that has been pending the longest.
6. Compare state machine sizes: write 5 async functions with different numbers of .await points and use std::mem::size_of_val on pinned instances to measure the size of each generated state machine.

References

• Async Book (official): https://rust-lang.github.io/async-book/
• Tokio Tutorial: https://tokio.rs/tokio/tutorial
• Withoutboats. "How Rust async is different from other async systems." https://without.boats/blog/
• Withoutboats. "Pin and suffering." https://without.boats/blog/pin/
• Tokio internals blog posts: https://tokio.rs/blog/
• Cliff L. Biffle. "Embassy: async embedded Rust." https://embassy.dev
• Nikos Nikoleris. "Tokio's work-stealing scheduler." (Tokio blog, 2023)
• RFC 2394: async fn in Rust. https://github.com/rust-lang/rfcs/blob/master/text/2394-async_await.md
• Tokio source: https://github.com/tokio-rs/tokio