Skip to content

Priority Inversion

Technical Overview

Priority inversion is one of the most subtle and dangerous failure modes in priority-based real-time systems. It occurs when a high-priority task is effectively blocked not by a direct dependency, but by the accidental scheduling of a medium-priority task that preempts a low-priority task holding a resource the high-priority task needs.

The result is a temporary but sometimes indefinite priority inversion: the high-priority task behaves as if it has lower priority than the medium-priority task, despite the scheduler's intent. In a hard real-time system, this can cause deadline misses. In safety-critical systems, it can cause loss of control. The Mars Pathfinder rover experienced priority inversion on the surface of Mars in 1997, causing repeated system resets — and the story of diagnosing and fixing it remotely via software uplink is one of the most dramatic episodes in the history of real-time computing.

Understanding priority inversion is essential for anyone writing multithreaded code with priority-based scheduling, RT systems, or OS kernels.

Prerequisites

  • 01-scheduling-fundamentals.md (scheduling policy hierarchy)
  • 04-linux-realtime-scheduling.md (SCHED_FIFO, SCHED_RR)
  • Understanding of mutex semantics and blocking
  • Basic POSIX threading concepts (pthread_mutex_t, pthread_attr_t)

The Priority Inversion Scenario

The classic three-task priority inversion scenario requires three tasks at three priority levels and one shared resource (mutex):

Three Tasks:
  H = High priority  (e.g., SCHED_FIFO priority 90)
  M = Medium priority (SCHED_FIFO priority 50)
  L = Low priority    (SCHED_FIFO priority 10)

Shared resource: mutex R

The inversion unfolds in four phases:

Priority Inversion Timeline:

Time  CPU   H (p=90)        M (p=50)        L (p=10)     Comments
───────────────────────────────────────────────────────────────────────
 0           sleeping        sleeping        RUNNING      L starts
 1           sleeping        sleeping        lock(R)✓     L acquires mutex R
 2           sleeping        sleeping        [working]    L in critical section
 3           WOKEN           sleeping        [working]    H wakes up
 4           RUNNING         sleeping        PREEMPTED    H preempts L (H>L)
 5           lock(R)→BLOCK   sleeping        [waiting]    H tries to lock R → BLOCKED!
             [blocked on R]                               H must wait for L
 6           [blocked]       WOKEN           RUNNING      M wakes up
 7           [blocked]       RUNNING         PREEMPTED    M preempts L! (M>L)

                             ← M RUNS ────────────────────────────────

 ...         [blocked]       [running]       [waiting]    M runs indefinitely

   H is BLOCKED waiting for L. L cannot run because M preempts it.
   H effectively has LOWER priority than M, despite H > M > L.
   This is PRIORITY INVERSION.

 N           [blocked]       M blocks/exits  RESUMING     Eventually M blocks
 N+1         [blocked]       sleeping        unlock(R)    L resumes, releases R
 N+2         UNBLOCKED       sleeping        preempted    H finally runs!

The inversion period: H is blocked for the entire duration that M runs — which could be arbitrarily long if M is CPU-bound and never yields. The scheduler has no visibility into this situation: from its perspective, H is blocked (sleeping), M is the highest-priority runnable task, and M correctly runs. The inversion is invisible to the scheduler without additional mechanism.

Why this is dangerous: In a hard real-time system, H may have a deadline 50ms from when it woke. If M runs for 60ms, H misses its deadline. The system has failed its real-time guarantee despite "correct" behavior from every component individually.

Mars Pathfinder: The Famous Case Study (1997)

Background

NASA's Mars Pathfinder rover landed on Mars on July 4, 1997. After landing, the rover and lander began operating normally. However, starting several days into the mission, the system would unexpectedly reset — entering a "safe mode" and halting science operations. The resets occurred repeatedly, threatening the mission.

The VxWorks System Architecture

Pathfinder ran VxWorks, a commercial real-time OS from Wind River Systems. The relevant software components:

Three key tasks (approximate priorities):
  ASI/MET meteorological science task   (LOW priority)
  bus_manager IPC management task       (MEDIUM priority)  
  data_logging task                     (MEDIUM priority)
  image_processing task                 (HIGH priority)

Shared resource: information bus (IPC mechanism, protected by a mutex)
  Used to pass data between tasks.

The Inversion

The meteorological science task (low priority) needed to publish data to the information bus. It acquired the bus mutex, published data, then held the mutex briefly during cleanup. While it held the mutex, the image processing task (high priority) woke up and tried to acquire the bus mutex — and blocked.

Meanwhile, the bus manager task (medium priority) became runnable and preempted the meteorological task. The bus manager task was CPU-intensive and non-blocking — it ran continuously, holding the meteorological task off the CPU.

The image processing task was now effectively blocked behind the bus manager task despite having much higher priority: the classic three-task inversion.

The Watchdog

VxWorks included a watchdog timer that monitored whether high-priority tasks were completing their work within expected deadlines. When the image processing task failed to complete on time due to the inversion, the watchdog concluded that the system was in an inconsistent state and triggered a full system reset — returning the rover to a known-safe state and halting science operations.

Diagnosis

The Pathfinder team at JPL had access to VxWorks' real-time debugging facilities. They could examine task state, mutex ownership, and scheduling history from telemetry. The diagnosis required understanding VxWorks' scheduling behavior, the mutex implementation, and the task interaction — non-trivial under normal circumstances, remarkable given they were debugging a computer 40 million miles away.

Post-mortem task state analysis (reconstructed from JPL records):

Mutex: bus_mutex
  Owner: ASI/MET task (priority 10)
  Waiter: image_processing task (priority 90)

Runqueue:
  bus_manager task (priority 50) — RUNNING
  ASI/MET task (priority 10) — RUNNABLE but preempted by bus_manager
  image_processing task (priority 90) — BLOCKED on bus_mutex

Diagnosis: Classic priority inversion.
  image_processing blocked waiting for bus_mutex held by ASI/MET
  ASI/MET preempted by bus_manager
  Watchdog fires when image_processing misses deadline

The Remote Fix

VxWorks supported enabling priority inheritance via a compile-time flag that was also accessible at runtime through a configuration parameter. The Pathfinder engineers crafted an uplink command sequence that enabled priority inheritance for the bus mutex. The command was transmitted to Mars, loaded into the running VxWorks system, and the problem disappeared immediately.

This remains one of the most impressive real-time debugging case studies in history: diagnosing a race condition and applying a fix to a system on another planet, remotely, within days.

Priority Inheritance: The Standard Solution

Priority inheritance prevents inversion by dynamically boosting the priority of a lock holder when a higher-priority task blocks on that lock:

Priority Inheritance Rules:
1. When task H blocks on mutex held by task L:
   → Boost L's priority to H's priority (temporarily)
   → L runs at H's priority until it releases the mutex

2. When L releases the mutex:
   → L's priority returns to its base priority
   → H is unblocked, runs at its original high priority
   → (or if L holds multiple mutexes with waiting high-priority tasks:
       L retains the highest among them until all are released)

3. Transitivity: if L → waits for → L2 (L blocks on another mutex held by L2):
   → L2's priority is also boosted to H's priority (transitive inheritance)

Priority Inheritance Timeline (same scenario with PI enabled):

Time  CPU   H (p=90)        M (p=50)        L (p=10→90)  Comments
───────────────────────────────────────────────────────────────────────
 0                                           RUNNING      L starts
 1                                           lock(R)✓     L acquires mutex R
 2                                           [working]    L in critical section
 3           WOKEN                           [working]    H wakes up
 4           RUNNING                         PREEMPTED    H preempts L (H>L)
 5           lock(R)→BLOCK                   [waiting]    H tries to lock R
             [blocked on R]                               PI: L boosted to p=90!
 6           [blocked]       WOKEN           RUNNING      M wakes, but L now p=90!
 7           [blocked]       RUNNABLE        RUNNING      L (p=90) > M (p=50) → L runs

 8           [blocked]       RUNNABLE        unlock(R)    L finishes, releases R
             [waiting]                       p restored=10  L priority restored
 9           UNBLOCKED       RUNNABLE        PREEMPTED    H unblocked, H>M>L, H runs
10           RUNNING                                      H completes its work
11                           RUNNING                      M runs (H complete)

With priority inheritance, H's wait for R is bounded by L's actual time to finish its critical section — not by M's arbitrary runtime. The inversion window is eliminated.

Linux's rt_mutex:

rt_mutex (in kernel/locking/rtmutex.c) implements priority inheritance for kernel use:

/* Acquire an rt_mutex (sleeping, with priority inheritance): */
int rt_mutex_lock(struct rt_mutex *lock);
int rt_mutex_lock_interruptible(struct rt_mutex *lock);
int rt_mutex_trylock(struct rt_mutex *lock);

/* Internal structure: */
struct rt_mutex {
    raw_spinlock_t      wait_lock;
    struct rb_root_cached waiters;    /* pi_waiters: sorted by priority */
    struct task_struct *owner;        /* task holding the mutex */
};

The pi_waiters plist (priority-sorted list) in the task struct tracks all mutexes that a task owns, plus the highest-priority waiter for each:

task_struct (Low-priority L):
  pi_waiters: {H waiting for mutex R (priority 90),
               H2 waiting for mutex S (priority 80)}
  prio → max(pi_waiters priorities) = 90
         ← L runs at priority 90 until it releases R and S

When H blocks on a mutex held by L, the kernel: 1. Adds H to mutex->waiters (sorted plist by H's priority) 2. Adds H to L->pi_waiters (the set of tasks waiting for L's locks) 3. Updates L's effective priority to max(L->normal_prio, H->prio) 4. If L is itself blocked waiting for another lock (transitivity): propagates up the chain

POSIX thread mutex with priority inheritance:

pthread_mutexattr_t attr;
pthread_mutex_t mutex;

pthread_mutexattr_init(&attr);
pthread_mutexattr_setprotocol(&attr, PTHREAD_PRIO_INHERIT);
pthread_mutex_init(&mutex, &attr);
/* Now this mutex has PI semantics */

On Linux, PTHREAD_PRIO_INHERIT uses the kernel's rt_mutex under the hood via the futex syscall with FUTEX_LOCK_PI.

Priority Ceiling Protocol: The Alternative

Priority inheritance is reactive: it boosts priorities after an inversion is detected. The Priority Ceiling Protocol (PCP) is proactive: it prevents inversion by design.

Rules of PCP: 1. Each mutex is assigned a "ceiling" = the highest priority of any task that may lock it 2. A task may only acquire a mutex if the task's priority is strictly greater than the ceiling of any currently locked mutex in the system 3. A lock holder inherits the ceiling priority of the mutex for the duration of the lock

Example:
  mutex R: ceiling = 90 (H, M, and L all may lock it)
  mutex S: ceiling = 50 (only M and L may lock it)

  L (priority 10) tries to acquire R:
    Current ceiling of locked mutexes = 0 (none locked)
    L's priority (10) > 0 → ALLOWED
    L acquires R, L's priority boosted to 90 (ceiling of R)

  Now L has priority 90. M (priority 50) tries to acquire anything:
    Current ceiling of locked mutexes = 90 (R is locked at ceiling 90)
    M's priority (50) ≤ 90 → DENIED until L releases R

  H tries to acquire R:
    Current ceiling = 90. H's priority = 90. 90 ≤ 90 → DENIED (strict greater)

    → This prevents H from even being blocked on R while L holds it with ceiling 90!
    → Inversion cannot occur.

PCP vs Priority Inheritance: - PCP requires knowing which tasks may lock which mutexes at system design time (static analysis). Not always possible in dynamic systems. - PCP is pessimistic: a task is blocked even if it would not actually conflict. Reduces parallelism. - PCP provides deadlock freedom as a provable guarantee: under PCP, circular lock chains cannot form. - Priority inheritance is optimistic: tasks are only blocked when an actual conflict occurs. Better parallelism but no deadlock freedom guarantee. - For hard real-time systems where response time bounds are analytically derived, PCP is preferred. For general-purpose OS kernels, priority inheritance is more practical.

Linux supports PCP via PTHREAD_PRIO_PROTECT:

pthread_mutexattr_setprotocol(&attr, PTHREAD_PRIO_PROTECT);
pthread_mutexattr_setprioceiling(&attr, 90);  /* set mutex ceiling */
pthread_mutex_init(&mutex, &attr);

Priority Inheritance in the Linux Kernel (PREEMPT_RT)

With PREEMPT_RT, spinlocks are converted to sleeping rt_mutex objects. This extends priority inheritance to all kernel critical sections, not just explicitly-PI mutexes.

Without PREEMPT_RT: - spin_lock() disables preemption — no task can be scheduled while holding it - But interrupt handlers can run (and they can trigger scheduling events) - A long spinlock critical section is an inversion source for RT tasks

With PREEMPT_RT: - spin_lock()rt_mutex_lock(): preemptible, with PI - An RT task blocked on a spinlock (now rt_mutex) triggers PI on the holder - The holder runs at the RT task's priority until it releases the lock

This makes Linux kernel code safe for RT tasks in the way that previously required dedicated RTOS design.

Detecting Priority Inversion in Production

# Method 1: Observe scheduling delays with perf sched
perf sched record -- sleep 10
perf sched latency --sort max
# Look for SCHED_FIFO tasks with unexpectedly high scheduling latency

# Method 2: ftrace for PI wakeup events
echo 1 > /sys/kernel/debug/tracing/events/lock/contention_begin/enable
echo 1 > /sys/kernel/debug/tracing/events/lock/contention_end/enable
cat /sys/kernel/debug/tracing/trace_pipe | grep -A2 "contention"

# Method 3: RT mutex debugging (CONFIG_RT_MUTEX_TESTER, CONFIG_DEBUG_RT_MUTEXES)
# Enables additional locking assertions and PI chain validation

# Method 4: bpftrace for futex PI blocking
bpftrace -e '
  tracepoint:syscalls:sys_enter_futex
  /args->op == 9 || args->op == 10/ {  /* FUTEX_LOCK_PI, FUTEX_TRYLOCK_PI */
    @start[tid] = nsecs;
    @waiter[tid] = comm;
  }
  tracepoint:syscalls:sys_exit_futex
  /@start[tid]/ {
    $wait = (nsecs - @start[tid]) / 1000;
    if ($wait > 1000) {  /* > 1ms wait on PI mutex */
      printf("PI wait: %s tid=%d waited %lld us\n", @waiter[tid], tid, $wait);
    }
    delete(@start[tid]);
    delete(@waiter[tid]);
  }'

# Method 5: Check pi_waiters chain depth
# In kernel debugger (kgdb, crash) or /proc/[pid]/wchan
cat /proc/[pid]/wchan  # should show futex_wait_requeue_pi if PI-blocked

# Method 6: valgrind --tool=helgrind detects lock ordering violations
# (which often reveal potential PI scenarios)

Real-World Inversion Scenarios Beyond Mars Pathfinder

Linux audio (JACK without PREEMPT_RT): The audio thread runs at SCHED_FIFO priority 70. The kernel spinlock protecting the ALSA ring buffer could be held by a low-priority process context while an interrupt handler ran. If a medium-priority kernel thread then preempted the lock holder, the audio thread would miss its deadline. This motivated the PREEMPT_RT patchset and the CONFIG_PREEMPT model.

Database and filesystem interactions: A database transaction at low priority holds a page lock (file-backed). An urgent data recovery process at high priority needs the same page. A background fsync process at medium priority preempts the transaction. The recovery process experiences priority inversion. Linux's page lock (lock_page()) does not have priority inheritance — a known limitation.

Android Binder IPC: Android's Binder IPC mechanism uses Linux's futex with priority inheritance (FUTEX_WAIT_REQUEUE_PI). When an app's Binder call is handled by a system service thread, the service thread inherits the app's priority. Without this, a low-priority system service handling a call from a high-priority app thread could be preempted by medium-priority apps. Android engineers specifically instrumented Binder for PI after experiencing jank in the UI layer.

VxWorks and aerospace: After Mars Pathfinder, Wind River documented priority inheritance as a recommended practice in VxWorks programming guides. Subsequent NASA missions (Mars Exploration Rovers Spirit and Opportunity, Mars Reconnaissance Orbiter) explicitly enabled priority inheritance on all shared mutexes. The lesson was institutionalized.

Avoiding Priority Inversion by Design

Several design patterns prevent priority inversion without needing PI:

Lock-free data structures: Eliminate mutexes entirely. Ring buffers, read-copy-update (RCU), compare-and-swap queues. If no mutex is held, no inversion is possible. This is the approach used in Linux's RCU — RT tasks can read without blocking, and writers complete without holding locks across scheduling points.

Priority-partitioned resources: Assign each resource an owner priority class. Only tasks of that class access the resource. No cross-priority lock acquisition, no inversion possible. Rigid but safe.

Deadline monotonic assignment: Assign task priorities based on deadlines (shortest deadline = highest priority). Analyze the system statically to prove all tasks meet their deadlines — accounting for inversion-time bounds when using PI or PCP.

Avoid holding locks across blocking points: Design critical sections to never call blocking operations. If a task under a lock might sleep, the lock should use PI semantics. Kernel convention: might_sleep() annotations in critical sections catch this at compile time.

Timeout on lock acquisition: pthread_mutex_timedlock() or rt_mutex_lock_interruptible() — if the lock is not acquired within a deadline, take a fallback path. Not a fix for inversion but limits its impact.

Priority Inversion in Non-Preemptive Systems

Priority inversion requires preemption — in a cooperative scheduler (only yield at explicit points), a medium-priority task cannot preempt a low-priority one mid-execution. However, non-preemptive systems can have analogous problems:

  • A low-priority coroutine holds a resource and makes a long I/O call (cooperative yield). A high-priority coroutine that wants the resource must wait. If medium-priority coroutines keep yielding into runnable state, the high-priority one is starved.
  • In event loops (Node.js, Python asyncio), a slow event handler (low-priority by virtue of being long) delays all subsequent events (including high-priority ones). This is the async equivalent of priority inversion.
  • Solution: Work decomposition — break long operations into small steps, yielding between them. Node.js setImmediate() and nextTick() queues address this.

Security Implications

Priority inversion as denial-of-service: An attacker who can create a low-priority task that holds a lock needed by high-priority system tasks can intentionally cause priority inversion. If the attacker then continuously schedules medium-priority tasks, the inversion can persist indefinitely. This requires unprivileged task creation with specific priority and locking access — not trivially exploitable but a real concern in RTOS environments.

Privilege escalation via PI: If a low-priority unprivileged task can cause a high-privilege task to block on its mutex, the high-privilege task waits for the low-privilege task. In some designs, this means the low-privilege task could control when the high-privilege task runs. This is not privilege escalation in the traditional sense but can be used to time attacks (e.g., force a timing-sensitive cryptographic operation to run at a predictable moment).

PI chain depth attacks: A malicious task could create a deep PI chain (task A waits for B waits for C waits for D...) requiring O(depth) work for every priority propagation. The Linux kernel limits PI chain length (max_lock_depth = 1024) to prevent this DoS vector.

Performance Implications

  • PI mutex acquisition is more expensive than non-PI: must check for and potentially update pi_waiters, potentially walk and update a PI chain. Overhead: ~200-500ns per acquisition vs ~50-100ns for non-PI mutex.
  • PI chain propagation is O(chain_depth) per priority change. Deep inheritance chains (A→B→C→...→Z) all need updating when A's priority changes. In practice, chains are short (1-3 levels); max_lock_depth limits pathological cases.
  • Using PTHREAD_PRIO_INHERIT on every mutex is not free — most applications do not need PI because they don't use mixed priorities. Use PI only for mutexes shared between tasks at different priority levels.
  • RT task throughput under PI contention: a high-priority task that frequently contends with low-priority tasks on PI mutexes will have higher latency than if it used lock-free designs. PI is a correctness tool, not a performance one.

Failure Modes

  • PI not configured: The most common failure mode. PI is available but not enabled (default pthread_mutex_t in glibc does not use PI). System experiences inversion but no obvious crash — just occasional latency spikes. Often diagnosed only after extensive production observation or specific RT analysis.
  • Nested PI chains with priority drops: If H blocks on mutex held by L, L is boosted to H's priority. Then H is cancelled (SIGKILL). L's priority is restored. But if the cancellation is not properly handled (the mutex is never released, the task is killed while holding), L becomes a zombie mutex holder at inflated priority. pthread_mutex_destroy() or PTHREAD_MUTEX_ROBUST_NP (error-checking mutex) addresses this.
  • PI and condition variables: pthread_cond_wait() releases a mutex and sleeps. If the condition variable signal wakes a high-priority waiter, it must re-acquire the mutex. If a low-priority task currently holds it, PI applies. But pthread_cond_broadcast() can wake multiple waiters; the priority order of re-acquisition is not guaranteed to respect PI. Use PTHREAD_PRIO_INHERIT on the associated mutex.
  • Starvation with PI disabled: Without PI, a low-priority task holding a lock against a blocked high-priority task may never run if medium-priority tasks continuously arrive. The high-priority task starves. No mechanism prevents this without PI.

Modern Usage and Future Directions

Android and PI: Android's entire Binder IPC framework is built on priority inheritance. Every IPC call inherits the caller's priority. This is explicitly documented in Android's real-time audio programming guide as a requirement for low-latency audio.

Linux real-time ecosystem: PREEMPT_RT + rt_mutex provides PI throughout the kernel. The full PREEMPT_RT mainlining (nearly complete as of Linux 6.12) means PI is available everywhere kernel code can block. This closes the historical gap where even correctly-written RT user-space code could experience inversion inside the kernel.

Lock-free as the modern alternative: Modern high-performance systems trend toward lock-free and wait-free data structures (atomic operations, RCU, MPSC queues) rather than PI mutexes. In these designs, readers never block and writers complete without holding locks — priority inversion is architecturally impossible. The Linux kernel's use of RCU is a model for this approach.

Formal verification of PI protocols: Tools like TLA+ and SPIN have been used to formally verify priority inheritance and priority ceiling protocol implementations. The UPPAAL model checker has been used to verify real-time task sets with PI for worst-case response time analysis.

Exercises

  1. Demonstrate inversion: Write three POSIX threads at SCHED_FIFO priorities 90, 50, and 10. Use a non-PI pthread_mutex_t. Low-priority thread acquires mutex, then sleeps briefly. High-priority thread wakes and tries to acquire mutex, blocks. Medium-priority thread wakes and runs continuously. Measure how long the high-priority thread waits. This should significantly exceed L's critical section time.
  2. Fix with priority inheritance: Repeat exercise 1 but initialize the mutex with PTHREAD_PRIO_INHERIT. Observe that H now waits only for L's actual critical section, regardless of M's behavior. Measure the difference in H's waiting time.
  3. PI chain: Create a 3-level PI chain: H waits for mutex_A held by M, M waits for mutex_B held by L. Verify with /proc/[pid]/status (VmPeak, priority fields) that M and L both run at H's priority via transitive inheritance.
  4. Reproduce Mars Pathfinder: Implement a simplified version of the Pathfinder scenario using VxWorks-like semantics: a shared "information bus" mutex, an ASI/MET-like low-priority task that holds it for variable duration, a bus-manager-like medium-priority CPU hog, and an image-processing-like high-priority task with a deadline. Demonstrate system "reset" (deadline miss) without PI, then fix with PI.
  5. Lock-free alternative: Implement a producer-consumer queue that the high-priority reader accesses without locking (using std::atomic acquire-release semantics or an SPSC ring buffer). Verify that the reader's latency is independent of producer/writer priority by scheduling the reader at SCHED_FIFO 90 and the writer at SCHED_FIFO 10 with an interfering medium-priority busy task.

References

  • Sha, L., Rajkumar, R., Lehoczky, J.P., "Priority Inheritance Protocols: An Approach to Real-Time Synchronization", IEEE Transactions on Computers, 1990 — the seminal PI paper
  • Baker, T.P., "A Stack-Based Resource Allocation Policy for Realtime Processes", RTSS 1990 — Stack Resource Policy (related to PCP)
  • Jones, M., "What Really Happened on Mars?", The Risks Digest (forwarded from internal JPL analysis), 1997 — the primary source on Mars Pathfinder
  • Reeves, G., "What Really Happened on Mars", NASA JPL internal communication posted publicly — engineer's account of diagnosing and fixing the Mars Pathfinder problem remotely
  • Buttazzo, G., "Hard Real-Time Computing Systems: Predictable Scheduling Algorithms and Applications", Springer — comprehensive coverage of PI, PCP, EDF in RTOS
  • Wind River VxWorks Programmer's Guide — section on task scheduling and resource management
  • Linux Kernel Documentation: Documentation/locking/rt-mutex.rst
  • Linux Kernel Documentation: Documentation/locking/rt-mutex-design.rst
  • Android Open Source Project: "Thread priority and real-time audio" documentation
  • POSIX Realtime Extensions: pthread_mutexattr_setprotocol(3), pthread_mutexattr_setprioceiling(3)
  • Linux source: kernel/locking/rtmutex.c, kernel/locking/rtmutex_api.c
  • Gleixner, T., "Realtime Linux: Spinning the Wheels of Progress", Linux Kernel Summit 2007