Real-Time Networking

Overview

Real-time networking is the discipline of transmitting data between nodes with guaranteed bounded latency and deterministic delivery. Standard Ethernet, TCP/IP, and Wi-Fi are designed for high throughput and best-effort delivery — they tolerate variable latency, retransmissions, and queuing delay because the applications that dominate the internet (web browsing, file transfer, video streaming) are not sensitive to jitter at the sub-millisecond level. Industrial automation, robotics, automotive control, and audio/video production are fundamentally different: a CNC machine that misses a motion command by 500 microseconds cuts the wrong path; an automotive brake-by-wire system that loses synchronization cannot guarantee stopping distance; an audio production network that delivers samples 2ms late causes an audible glitch.

Real-time networking solves this by providing mechanisms for time synchronization, traffic shaping, and frame scheduling that constrain worst-case latency to bounded values. The dominant modern standard for doing this over Ethernet is Time-Sensitive Networking (TSN), a suite of IEEE 802.1 standards developed by the IEEE 802.1 Time-Sensitive Networking Task Group. TSN transforms standard Ethernet into a deterministic fabric while preserving backward compatibility with existing Ethernet infrastructure.

Prerequisites

Understanding of Ethernet frame structure (MAC header, EtherType, payload, FCS)
Familiarity with switched Ethernet (VLANs, 802.1Q tagging, spanning tree)
Basic understanding of queuing theory and priority queuing
Awareness of real-time systems concepts (hard RT vs soft RT, jitter, latency budget)
Understanding of PTP/IEEE 1588 at a conceptual level is helpful

Historical Context

The Industrial Fieldbus Wars

Before Ethernet became viable in industrial settings, dozens of proprietary fieldbuses competed: PROFIBUS, DeviceNet, CAN, Modbus, Foundation Fieldbus, Interbus, and many others. Each was purpose-built for deterministic communication over short distances in electrically noisy factory environments. They were slow (typically 1-12 Mbps), limited in topology, and completely incompatible with each other. A factory using Siemens PLCs and Beckhoff servo drives needed two separate physical networks.

The push to unify on Ethernet began in the late 1990s. PROFINET (Siemens-led, 2003), EtherCAT (Beckhoff, 2003), EtherNet/IP (Rockwell/ODVA, 2001), and Powerlink (B&R, 2001) each added determinism to Ethernet using different approaches — some by using dedicated hardware, some by time-slicing the medium, some by modifying the protocol entirely. These are now called "Industrial Ethernet" protocols. They work but are proprietary and mutually incompatible.

TSN represents the IEEE's attempt to standardize the determinism mechanisms so that any TSN-capable switch and any TSN-capable device can interoperate. The first TSN standards were ratified between 2011 and 2018.

Time-Sensitive Networking (TSN) Standard Suite

TSN is not a single protocol but a collection of IEEE 802.1 amendments, each addressing a specific aspect of deterministic Ethernet:

TSN Standard Suite Overview
==============================

802.1AS  — Time synchronization (generalized PTP)
802.1Qbv — Time-Aware Shaper (scheduled traffic)
802.1Qbu — Frame Preemption (interrupt low-priority frames)
802.3br  — Interspersed Express Traffic (frame preemption at MAC layer)
802.1Qcc — Stream Reservation Protocol (centralized/distributed config)
802.1Qci — Per-Stream Filtering and Policing (ingress protection)
802.1CB  — Frame Replication and Elimination (redundancy)
802.1Qav — Credit-Based Shaper (audio/video bridging - predecessor)

802.1AS: Generalized Precision Time Protocol (gPTP)

Deterministic scheduling requires that all nodes share a common sense of time. Standard NTP achieves millisecond-level synchronization over the internet, which is insufficient. IEEE 1588 Precision Time Protocol (PTP) achieves sub-microsecond synchronization over a single LAN segment but requires hardware timestamping and is complex to configure. 802.1AS is a profile of IEEE 1588 optimized for TSN bridge networks.

Clock Synchronization Mechanism

gPTP Clock Synchronization (Peer-to-Peer Delay Mechanism)
===========================================================

Time Master (grandmaster clock)
    │
    │  Sync message (T1 timestamp)
    ▼
TSN Switch A
    │  Residence time measured, added to correction field
    │  Link delay measured (peer-to-peer delay request/response)
    ▼
TSN Switch B
    │  Residence time measured and accumulated
    ▼
Time Slave (endpoint device)


Message exchange for link delay measurement:

Endpoint A          Switch Port B
    │                    │
    │── Pdelay_Req ──────►│  T1 (A sends), T2 (B receives)
    │                    │
    │◄─ Pdelay_Resp ──────│  T3 (B sends back), T4 (A receives)
    │                    │
    │◄─ Pdelay_Resp_FU ───│  carries T3 value
    │                    │

Link delay = ((T4 - T1) - (T3 - T2)) / 2
Rate ratio (frequency offset) computed from multiple exchanges

gPTP synchronization typically achieves: - 100ns-500ns accuracy across a single switch hop - Under 1µs end-to-end on a well-configured 5-hop network - Sub-100ns with better hardware timestamping units

Hardware Requirements for gPTP

Achieving sub-microsecond accuracy requires hardware timestamping in the NIC or switch MAC — the timestamp must be captured at the precise moment the first bit of the Sync message exits or enters the PHY. Software timestamping introduces jitter of 1-100µs due to interrupt latency and OS scheduling, which defeats the purpose.

802.1Qbv: Time-Aware Shaper (TAS)

802.1Qbv is the core traffic scheduling mechanism of TSN. It controls when each of the eight 802.1Q priority queues is allowed to transmit by operating a Gate Control List (GCL) — a time-indexed schedule of which queues are open and which are closed.

Gate Control List

802.1Qbv Gate Control List Example
=====================================

System has 8 queues (Q0 = lowest priority, Q7 = highest).
Cycle time = 1ms. All times are offsets from cycle start.

Time (µs)   Gate State           Open Queues
---------   ------------------   --------------------------------
0 - 100     10000000 (binary)    Q7 only (time-critical control)
100 - 300   01000000             Q6 only (safety-critical alarms)
300 - 700   00000011             Q0, Q1 (best-effort traffic)
700 - 800   10000000             Q7 only (second control window)
800 - 1000  00000001             Q0 only (background traffic)


         Q7: ▐████░░░░░░░░░░░░░░░░░░▐████░░░░░░░░░░░░░░░
         Q6: ▐░░░░████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▐
         Q1: ▐░░░░░░░░░░░░░░░████████░░░░░░░░░░░░░░░░░░░▐
         Q0: ▐░░░░░░░░░░░░░░░████████░░░░░░░░░░░████████▐
              0      100    300      700    800       1000µs

The GCL repeats every cycle. A 60Hz motion control system uses 1/60s ≈ 16.67ms cycles.
Guaranteed delivery windows for Q7 are 0-100µs and 700-800µs each cycle.
Worst-case latency for Q7 traffic: one full cycle (1ms) + transmission time.

Guard Band

A critical implementation detail: when the GCL transitions from a lower-priority open state to a time-critical open state, a "guard band" (also called a "hold" interval) must close all queues briefly before the critical window opens. This prevents a large best-effort frame from starting transmission 1µs before the critical window, occupying the medium for the entire critical window.

Guard Band Protection
======================

Without guard band (WRONG):
  ...best-effort frame starts at t=98µs (2µs before Q7 window)...
  ▐═══best-effort frame (1500 bytes = ~12µs at 1Gbps)═══▐Q7 frame delayed!

With guard band:
  Gate closes all queues at t=(100 - max_frame_time)µs
  Guard band = 1526 bytes / 1Gbps ≈ 12.2µs
  Close all at t=87.8µs, open Q7 at t=100µs ─► Q7 gets clean slot

Credit-Based Shaper (CBS, 802.1Qav)

A simpler predecessor to TAS is the Credit-Based Shaper (802.1Qav), originally designed for Audio Video Bridging (AVB). CBS assigns a "credit" to a queue that accumulates at a defined rate when the queue is idle and depletes when frames are sent. A queue can only transmit when it has positive credit. This limits bandwidth consumption by a stream to a configured rate while still allowing bursting.

802.1Qbu / 802.3br: Frame Preemption

Frame preemption allows an express (high-priority) frame to interrupt the transmission of a preemptable (low-priority) frame. Without preemption, a 1500-byte frame at 100Mbps takes 120µs to transmit — during which no high-priority frame can be sent. With preemption, the switch hardware can interrupt the low-priority frame after a minimum fragment is transmitted, send the high-priority frame, and then resume the interrupted frame.

Frame Preemption Sequence
===========================

Without preemption:
  Time ──►
  ▐══════════════ Low-pri frame (1500B = 120µs @ 100Mbps) ══════════════▐
                                                                         ▐ Hi-pri delayed 120µs!

With preemption (802.1Qbu):
  ▐══ Low-pri frag ══▐ Hi-pri frame ▐══════ Low-pri resumed ══════════▐
  ┌──────────────────────────────────────────────────────────────────┐
  │ Fragment 1 (min 64B) + mCRC  |  High-priority  |  Fragment 2 + FCS│
  └──────────────────────────────────────────────────────────────────┘
  Hi-pri frame additional latency: ~5µs (fragment time + guard)
  vs 120µs without preemption

Preemption requires support in both the transmitting switch port and the receiving device MAC (802.3br "interspersed express traffic").

EtherCAT: Industrial Fieldbus Over Ethernet

EtherCAT (Ethernet for Control Automation Technology) was developed by Beckhoff Automation and released in 2003. Rather than using standard Ethernet switching, EtherCAT uses a passing/processing model where all slave devices are daisy-chained and process the Ethernet frame as it passes through them at wire speed.

EtherCAT Frame Passing Model
==============================

Master                     Slave 1          Slave 2          Slave 3
   │                          │                │                │
   │──── EtherCAT frame ──────►│                │                │
   │     [HDR][datagram1][dg2] │                │                │
   │                    ┌──────┘                │                │
   │           reads/writes own    ─────────────►│                │
   │           data, frame         reads/writes                   │
   │           continues          own data ──────────────────────►│
   │                                                              │
   │◄── Frame returns to master with all data filled in ──────────┘
   │     Processing latency per slave: ~100-200ns

Key properties:
- One EtherCAT frame can carry data for all 100+ slaves
- Distributed Clocks: slaves sync to 1µs accuracy
- Cycle times: 62.5µs achievable for 100 servo drives
- Masters: standard NIC (not TSN hardware required)

Distributed Clocks

EtherCAT's "Distributed Clocks" mechanism synchronizes all slave clocks to sub-microsecond accuracy. The master sends a frame whose timestamp is captured by each slave as it passes. The master then reads all slave timestamps and computes offsets. Each slave adjusts its local clock to match. This achieves ~1µs synchronization without IEEE 1588 hardware.

PROFINET IRT (Isochronous Real-Time)

PROFINET (Process Field Network) is Siemens' Industrial Ethernet protocol, now standardized in IEC 61158. PROFINET defines three communication classes:

PROFINET Communication Classes
================================

Class         Mechanism              Cycle time    Use case
------------  ---------------------  ------------  ---------------------------
TCP/IP (NRT)  Standard Ethernet      >100ms        Configuration, diagnostics
RT            VLAN priority queue    1-100ms       Standard I/O (most devices)
IRT           TDMA (time-sliced)     250µs-1ms     High-precision motion control

PROFINET IRT uses TDMA (Time Division Multiple Access) — a pre-negotiated schedule where each device transmits in its assigned time slot. This is functionally similar to TSN's TAS but is a proprietary mechanism requiring PROFINET-specific ASICs in switches.

SOME/IP: Automotive Real-Time Communication

SOME/IP (Scalable service-Oriented MiddlwarE over IP) is an automotive middleware protocol developed by BMW and standardized by AUTOSAR. Modern vehicles have 50-100+ ECUs (Electronic Control Units) that must exchange sensor data, commands, and status with latency requirements from 1ms (brake control) to 100ms (infotainment).

SOME/IP sits above UDP/TCP and provides: - Service discovery (find ECUs offering specific services) - Remote procedure calls - Event notifications with serialization - Multicast for broadcast sensor data

For the tightest automotive loops (steer-by-wire, brake-by-wire), TSN is being adopted in addition to SOME/IP to provide the hardware-level scheduling guarantees that the software protocol cannot.

RT-Preempt + TSN on Linux

Linux with the PREEMPT_RT patch provides soft real-time capability (bounded latency in the microsecond range under load). Combined with TSN hardware, this enables Linux-based industrial controllers:

Linux TSN Stack
=================

Application (industrial controller)
    │
POSIX real-time API (SO_TXTIME socket option)
    │
Linux network stack
├── Qdisc: taprio (Time-Aware Priority qdisc — implements 802.1Qbv GCL in software)
├── Qdisc: etf (Earliest TxTime First — launches frames at precise hardware times)
└── tc-flower, tc-cbs (Credit-Based Shaper)
    │
NIC driver with hardware offload (ETF launch time in NIC hardware)
    │
TSN-capable NIC (Intel i210/i225/i226 support partial hardware offload)
    │
TSN switch (NXP SJA1105, Microchip LAN9668, Intel Elkhart Lake built-in)

taprio Qdisc Configuration

# Configure a 1ms cycle with two windows:
# 0-100µs: Q7 open (time-critical)
# 100-1000µs: Q0-Q6 open (best-effort)

tc qdisc replace dev eth0 parent root handle 100 taprio \
    num_tc 8 \
    map 0 1 2 3 4 5 6 7 \
    queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7 \
    base-time 1000000000 \
    sched-entry S 80 100000 \
    sched-entry S 7f 900000 \
    flags 0x2     # hardware offload mode

# ETF qdisc for precise launch time
tc qdisc add dev eth0 parent 100:8 etf \
    clockid CLOCK_TAI \
    delta 50000 \
    offload

Debugging Notes

# Check PHC (PTP Hardware Clock) on NIC
ethtool -T eth0
ls /dev/ptp*

# Synchronize to PTP grandmaster
ptp4l -i eth0 -m -f /etc/linuxptp/automotive-master.cfg

# Check PTP offset (should be <1µs for TSN)
pmc -u -b 0 'GET CURRENT_DATA_SET'

# Monitor taprio schedule
tc -s qdisc show dev eth0

# Check for late frames (ETF reports these)
tc -s qdisc show dev eth0 | grep -A5 etf

# Trace network latency
sudo perf trace -e net:* -p $(pgrep controller) 2>&1 | head -30

# Hardware timestamping test
testptp -d /dev/ptp0 -t  # test time accuracy

Common Pitfalls

TSN GCL schedule must match across all switches in the path; a single non-TSN switch in the path breaks determinism
PTP and the application must use CLOCK_TAI (International Atomic Time), not CLOCK_REALTIME, because TAI has no leap second discontinuities
SO_TXTIME requires the NIC to support ETF hardware offload; without offload, software mode still works but adds ~100µs jitter
CPU frequency scaling (C-states, P-states) adds latency to interrupt handling; must be disabled on RT systems

Security Implications

GCL injection: If an attacker can modify the GCL (gate control list) on a switch, they can starve critical traffic queues, causing control loops to miss their deadlines. TSN configurations must be authenticated.
Time master spoofing: A rogue device claiming to be a better PTP grandmaster (lower clock class/accuracy) can take over time synchronization, injecting a false time base that desynchronizes all scheduled traffic. 802.1AS-2020 adds authentication via a TLV mechanism.
Traffic injection: An attacker on the network can inject frames into high-priority queues if 802.1Qci (per-stream filtering and policing) is not configured, consuming the reserved bandwidth meant for safety-critical traffic.
Side-channel via timing: Deterministic delivery timing leaks information about what is happening in a control system. In military or critical infrastructure contexts, TSN traffic analysis can reveal operational patterns.

Performance Implications

A TSN switch adds approximately 1-5µs residence time to frame delivery (hardware-assisted forwarding with precise scheduling).
Each hop in a TSN path adds one cycle of worst-case latency budget. A system with 5 switch hops and 1ms cycles has a worst-case end-to-end latency of 5ms for the lowest-priority traffic, but guaranteed delivery within one cycle for Q7 (critical) traffic.
EtherCAT achieves lower latency than TSN for direct daisy-chain topologies (sub-100µs round trip for all slaves) but requires ring/line topology.
gPTP synchronization overhead: one Sync + Follow_Up message pair per second per port is negligible on modern switches.

Failure Modes

Grandmaster clock failure: If the PTP grandmaster fails and failover to a backup grandmaster takes longer than the clock holdover time, nodes drift apart and scheduled traffic misses windows. Holdover time depends on local oscillator quality (XO vs TCXO vs OCXO).
Switch non-TSN insertion: A non-TSN switch inserted between two TSN islands breaks the time synchronization chain and the scheduling guarantees. This commonly happens when maintenance staff replace a failed switch with an unmanaged commodity switch.
GCL misconfiguration: A GCL that doesn't leave enough guard band, or whose cycle time doesn't align with the application's period, causes priority inversion — critical frames waiting behind best-effort traffic.
CPU overload on Linux RT + TSN: Even with PREEMPT_RT, a CPU-bound task at RT priority can delay NIC driver interrupt handling, causing frames to miss their ETF launch window and be dropped or delayed.
IEEE 1588 BMCA oscillation: The Best Master Clock Algorithm can oscillate if multiple clocks have similar quality parameters, causing the grandmaster to change repeatedly, re-synchronizing all clocks repeatedly.

Modern Usage (2025)

Industrial automation: TSN is the backbone of Industry 4.0 networks; Siemens, Beckhoff, and Rockwell all ship TSN-capable devices. EtherCAT remains dominant for ultra-low latency servo control.
Automotive: TSN is specified in IEEE 802.1 automotive profiles and is integrated in AUTOSAR Adaptive Platform. BMW, Toyota, and Tesla are deploying TSN in in-vehicle backbone networks.
Professional audio/video: SMPTE ST 2110 (professional media over IP) uses PTP (802.1AS profile) for synchronization; AES67 audio-over-IP uses PTP as well.
5G fronthaul: O-RAN fronthaul between radio units and DU (distributed unit) uses IEEE 1914.3 (eCPRI) with PTP/SyncE for phase alignment, effectively a form of real-time networking over Ethernet.
Robotics: ROS 2 with DDS middleware runs over TSN in next-generation collaborative robot cells.

Future Directions

DetNet (Deterministic Networking): IETF DetNet extends TSN-like guarantees from single-hop Ethernet to multi-hop IP/MPLS networks, enabling end-to-end determinism across WAN segments for industrial IoT.
TSN + 5G integration: 3GPP and IEEE are standardizing how TSN streams can traverse 5G wireless links with preserved timing guarantees, enabling wireless TSN endpoints.
Wireless TSN: 802.11 (Wi-Fi) extensions for deterministic delivery (802.11be / Wi-Fi 7 MLD, 802.11az) aim to bring bounded latency to wireless industrial networks.
CAN over TSN: Automotive systems are gradually migrating from CAN bus to Ethernet + SOME/IP + TSN for bandwidth, with TSN providing the determinism previously guaranteed by CAN's CSMA/CD arbitration.
Open-source TSN stacks: The Linux kernel's taprio/etf qdisc stack, OpenAVB, and Open-TSN are maturing to make TSN deployment accessible without proprietary tools.

Exercises

Set up a two-node Linux system (two VMs or two Raspberry Pi 4 boards with gig Ethernet) and configure ptp4l in master-slave mode. Measure the clock offset over time using phc2sys and ts2phc. What is the achieved synchronization accuracy? What changes when you put a software bridge in the path?
Configure the taprio qdisc on a Linux system with Intel i210 NIC. Create a 1ms GCL with two windows: 200µs for priority 7 and 800µs for everything else. Send streams at both priorities and measure the latency distribution of the high-priority stream using tcpdump with hardware timestamps.
Read the EtherCAT specification introduction (freely available from beckhoff.com). Draw a diagram comparing EtherCAT's distributed processing model to a TSN star topology with a central switch. What are the tradeoffs in terms of latency, topology flexibility, and scalability?
Research the PROFINET IRT slot model. Compare it to TSN's GCL model. What hardware requirements does PROFINET IRT impose on switches vs what TSN 802.1Qbv requires?
Investigate a real TSN deployment (the 5G fronthaul eCPRI specification, or the IEEE 802.1 automotive profile). Document the specific TSN standards used, the cycle time requirements, and the achieved synchronization accuracy in production.

References

IEEE 802.1AS-2020: Timing and Synchronization for Time-Sensitive Applications. IEEE Standards Association.
IEEE 802.1Qbv-2015: Enhancements for Scheduled Traffic. IEEE Standards Association.
IEEE 802.1Qbu-2016: Frame Preemption. IEEE Standards Association.
Farkas, J. et al. (2018). Time-Sensitive Networking: A Survey. IEEE Communications Surveys & Tutorials.
Beckhoff Automation. (2003). EtherCAT Technology Group — The Ethernet Fieldbus. https://www.ethercat.org/
Linux kernel TSN documentation: Documentation/networking/tsn.rst
Loeser, C. & Haertig, H. (2004). Low-Latency Hard Real-Time Communication over Ethernet. ECRTS.
Wisniewski, L. et al. (2020). Challenges of Real-Time Communication in TSN Networks. IEEE ETFA.
AUTOSAR. (2020). Specification of Socket Adaptor SOME/IP. AUTOSAR Classic Platform.
IEEE 1914.3-2018: Radio Over Ethernet Encapsulations and Mappings (eCPRI).