06 — TLS Internals

Technical Overview

TLS (Transport Layer Security) provides confidentiality, integrity, and authentication for data transmitted over TCP. TLS 1.3, standardized in RFC 8446 (2018), eliminated several legacy vulnerabilities and reduced handshake latency from 2 RTTs to 1 RTT through a redesigned protocol flow. Understanding TLS internals is essential for diagnosing handshake failures, performance tuning of HTTPS servers, implementing mTLS service meshes, and understanding security boundaries in microservice architectures.

---

Prerequisites

• Asymmetric cryptography basics (public/private keys, digital signatures)
• Symmetric encryption concepts (AES-GCM, ChaCha20)
• TCP connection lifecycle (see 01-tcp-state-machine.md)
• QUIC protocol familiarity (see 04-quic-protocol.md)
• Familiarity with openssl, curl, nginx

---

Core Content

TLS Purpose: CIA Triad

Property	Mechanism	What it prevents
Confidentiality	Symmetric encryption (AES-256-GCM)	Eavesdropping
Integrity	AEAD MAC (GCM auth tag)	Tampering with data in transit
Authentication	Certificate + digital signature	Impersonation / MITM

TLS does NOT provide: anonymity (SNI is visible), availability (TCP DoS still works), or forward secrecy by itself (only with ephemeral key exchange, which TLS 1.3 mandates).

---

TLS 1.3 Handshake (1-RTT)

Client                                          Server
  |                                               |
  | ─── ClientHello ──────────────────────────>  |
  |   client_random (32 bytes)                    |
  |   cipher_suites: TLS_AES_128_GCM_SHA256,     |
  |                  TLS_AES_256_GCM_SHA384,     |
  |                  TLS_CHACHA20_POLY1305_SHA256 |
  |   extensions:                                 |
  |     key_share: X25519 ephemeral public key    |
  |     supported_versions: TLS 1.3               |
  |     server_name: example.com (SNI)            |
  |     supported_groups: x25519, secp256r1       |
  |     signature_algorithms: ecdsa_secp256r1,    |
  |                           rsa_pss_rsae_sha256 |
  |                                               |
  | <── ServerHello ──────────────────────────── |
  |   server_random (32 bytes)                    |
  |   cipher_suite: TLS_AES_256_GCM_SHA384       |
  |   key_share: X25519 ephemeral public key      |
  |     [Both sides now derive handshake keys     |
  |      from DH shared secret + randoms]         |
  |                                               |
  | <── {Certificate} ────────────────────────── |  encrypted
  |   server certificate chain                    |  with
  | <── {CertificateVerify} ──────────────────── |  handshake
  |   signature over transcript hash             |  keys
  | <── {Finished} ───────────────────────────── |
  |   HMAC over transcript (verifies all msgs)   |
  |                                               |
  |   [Client derives traffic keys]               |
  | ─── {Finished} ───────────────────────────>  |  encrypted
  |   HMAC over transcript                        |  with
  | ─── {Application Data} ───────────────────>  |  application
  |   HTTP request (can be sent with Finished)    |  traffic keys
  |                                               |
  | <── {Application Data} ──────────────────── |
  |   HTTP response                               |

Key exchange: TLS 1.3 mandates forward secrecy — ephemeral Diffie-Hellman (ECDHE: X25519 or P-256). Even if the server's long-term private key is compromised in the future, past sessions cannot be decrypted (no static RSA key exchange, unlike TLS 1.2).

Key derivation: HKDF (HMAC-based Key Derivation Function) with SHA-256 or SHA-384 produces distinct keys for handshake and application traffic, and separate keys for each direction.

---

TLS 1.3 0-RTT Session Resumption

After a completed TLS 1.3 session, the server sends a NewSessionTicket containing a PSK (Pre-Shared Key):

Previous session end:
  Server ─→ Client: NewSessionTicket{
      ticket_lifetime: 604800 (7 days)
      ticket_nonce: ...
      ticket: <encrypted session state>
      extensions: early_data (max_early_data_size)
  }

0-RTT resumed connection:
  Client ─────────────────────────────────────────→ Server
    ClientHello{
        pre_shared_key: [PSK identity from ticket]
        early_data: present  ← signals 0-RTT intent
        key_share: X25519    ← still needed for new keys
    }
    [Early Data / Application Data ──────────────→]
    (encrypted with early_traffic_secret derived from PSK)

  Server ─────────────────────────────────────────→ Client
    ServerHello{ pre_shared_key: selected }
    {EncryptedExtensions{ early_data: accepted }}
    {Finished}

0-RTT replay protection: 0-RTT data has no forward secrecy relative to PSK compromise, and is vulnerable to replay. Servers must either: 1. Maintain a "seen nonces" database to reject replays (server-side state) 2. Restrict 0-RTT to idempotent requests only (application-level) 3. Disable 0-RTT entirely for sensitive endpoints

---

TLS Record Layer

Once the handshake completes, application data is protected by the TLS record layer:

TLS Record (Application Data):
+----+--------+--------+---------------------------+----------+
| CT | Legacy | Length | Encrypted Content         | Auth Tag |
| 23 | 03 03  | 2 bytes| (plaintext type + padding)| 16 bytes |
+----+--------+--------+---------------------------+----------+

CT = 23 (application_data)
Legacy Version = 03 03 (TLS 1.2 for compatibility — actual version in extension)

AEAD (Authenticated Encryption with Associated Data): TLS 1.3 uses only AEAD ciphers: - AES-128-GCM: hardware-accelerated (AES-NI + CLMUL), ~15 GB/s on modern cores - AES-256-GCM: same speed, higher security margin - ChaCha20-Poly1305: software-optimized, preferred on mobile/IoT without AES-NI

The nonce is derived by XOR-ing the sequence number with the IV. Each record has a unique nonce, preventing nonce reuse that would catastrophically break GCM.

Maximum record size: 16KB (TLS_MAX_PLAINTEXT_LENGTH). Breaking data into records introduces per-record overhead (5-byte header + 16-byte auth tag = 21 bytes overhead per 16KB of data ≈ 0.13%).

---

Certificate Chain Validation

When the client receives the server's certificate, it verifies:

1. Signature: the certificate was signed by a CA whose public key the client trusts
2. Chain: the CA certificate was itself signed by a trusted root CA (chain of trust)
3. Validity period: notBefore < now < notAfter
4. Revocation: certificate has not been revoked (via OCSP or CRL)
5. Name matching: server's hostname matches certificate's CN or SAN (Subject Alternative Name)

# Inspect certificate chain
openssl s_client -connect example.com:443 -showcerts 2>/dev/null \
    | openssl x509 -text -noout | grep -A2 'Subject\|Issuer\|Validity'

# Check OCSP stapling
openssl s_client -connect example.com:443 -status 2>/dev/null \
    | grep -A 10 'OCSP response'

# Verify certificate chain manually
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt server.crt

OCSP Stapling: instead of the client querying the OCSP responder (adding 100–300ms latency), the server periodically fetches a signed OCSP response and includes ("staples") it in the TLS handshake. The client verifies the OCSP response (signed by the CA, valid timestamp) without a separate network request.

# nginx OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/ssl/chain.crt;
resolver 8.8.8.8;

---

Session Tickets (Server-Side Resumption)

Session tickets enable stateless TLS resumption. The server encrypts session state with a server-side key and sends it to the client. On reconnect, the client presents the ticket; the server decrypts it and restores the session.

Security considerations: - Ticket keys must be rotated (compromised key → ability to decrypt all sessions using those tickets) - Ticket keys are often shared across a server fleet (all servers can resume tickets from any other server) - STEK (Session Ticket Encryption Key) rotation interval should be shorter than the ticket lifetime

# nginx session ticket key rotation
ssl_session_tickets on;
ssl_session_ticket_key /etc/nginx/ssl/ticket_key_current.key;
ssl_session_ticket_key /etc/nginx/ssl/ticket_key_previous.key;
# Rotate keys daily: key file contains 80 bytes of random data

---

Kernel TLS (kTLS)

kTLS moves TLS record layer processing into the kernel, enabling sendfile() for encrypted file serving:

Without kTLS (HTTPS):             With kTLS (HTTPS):
  file → user buffer               file → kernel page cache
  user buffer → encrypt            kernel TLS module encrypts
  encrypted → socket buffer        encrypted → NIC DMA directly
  socket buffer → NIC DMA          (sendfile-like, no user copy)
  (2 copies + encrypt in user)     (0 user copies)

/* kTLS initialization after TLS handshake */
struct tls_crypto_info_aes_gcm_128 crypto = {
    .info.version = TLS_1_3_VERSION,
    .info.cipher_type = TLS_CIPHER_AES_GCM_128,
    .iv = ...,       /* from HKDF */
    .key = ...,      /* from HKDF */
    .salt = ...,
    .rec_seq = ...,
};
setsockopt(fd, SOL_TLS, TLS_TX, &crypto, sizeof(crypto));
/* Now: write() to the socket auto-encrypts using kTLS */
/* sendfile() now works for HTTPS */

Check kTLS usage:

cat /proc/net/tls_stat
# TlsCurrTxSw = sockets using software kTLS TX
# TlsCurrTxDevice = sockets using NIC-offloaded kTLS TX

---

TLS Offload (NIC-Based)

Mellanox/Nvidia ConnectX-5+ and Intel E800 series NICs support TLS offload: - TLS TX offload: the NIC encrypts packets using hardware AES, at full 100G line rate with near-zero CPU - TLS RX offload: the NIC decrypts received TLS records, delivers plaintext to the host

# Check NIC TLS offload capability
ethtool -k eth0 | grep tls
# tls-hw-tx-offload: on

# Enable via OpenSSL + kTLS socket option
# (automatically used if NIC supports it and kTLS is active)

---

mTLS: Mutual Authentication

In standard TLS, only the server is authenticated. In mTLS (mutual TLS), both parties present certificates, enabling bidirectional authentication — the foundation of zero-trust networking and service meshes.

Standard TLS:                     mTLS:
  Client ← authenticates Server   Client ← authenticates Server
  Client → anonymous               Client → authenticated by certificate

# nginx mTLS: require client certificate
ssl_verify_client on;
ssl_client_certificate /etc/nginx/ca.crt;
# Reject connections from clients without valid cert

Istio service mesh mTLS: Istio automatically provisions TLS certificates for every pod via the SPIFFE/SVID identity system. Envoy sidecar proxies terminate and originate mTLS, transparent to the application. All service-to-service communication is mTLS-encrypted and authenticated by default (PeerAuthentication: STRICT).

---

Historical Context

SSL was developed by Netscape in 1994–1995 for secure web commerce. SSL 3.0 was the first widely deployed version. The IETF standardized TLS 1.0 (essentially SSL 3.1) in RFC 2246 (1999), followed by TLS 1.1 (2006) and TLS 1.2 (2008).

TLS 1.3 was standardized in 2018 after an unusually long (3-year) IETF process. The main fights were over 0-RTT (financial institutions worried about replay attacks), over removing all non-AEAD cipher suites (some operators wanted AES-CBC for hardware compatibility), and over removing the ability to do session resumption without forward secrecy.

The migration from TLS 1.2 to 1.3 was complicated by the discovery that many middleboxes (Palo Alto, Juniper, F5 load balancers) were inspecting and modifying TLS 1.2 handshakes in ways that broke with TLS 1.3's format. The TLS 1.3 spec includes a version negotiation mechanism and version greasing specifically to detect and work around these middleboxes.

---

Production Examples

High-performance HTTPS server tuning:

server {
    listen 443 ssl;
    ssl_protocols TLSv1.3;                          # 1.3 only
    ssl_ciphers TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256;
    ssl_prefer_server_ciphers off;                  # let client choose
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;               # 50MB session cache
    ssl_session_tickets on;
    ssl_stapling on;
    ssl_stapling_verify on;

    # Enable kTLS (OpenSSL 3.0+, kernel 5.2+)
    ssl_conf_command Options KTLS;
    sendfile on;

    # HTTP/3 support
    listen 443 quic reuseport;
    http3 on;
    add_header Alt-Svc 'h3=":443"; ma=86400';
}

Certificate debugging:

# Full TLS handshake debug
openssl s_client -connect example.com:443 -tlsextdebug -msg 2>&1 | head -50

# Check cipher negotiated
curl -vv --tlsv1.3 https://example.com 2>&1 | grep 'TLSv\|cipher\|cert'

# Certificate expiry check
echo | openssl s_client -connect example.com:443 2>/dev/null \
    | openssl x509 -noout -dates

---

Debugging Notes

# TLS handshake failure diagnosis
openssl s_client -connect host:443 -debug 2>&1 | grep -E 'alert|error|DONE'

# Common errors:
# "no peer certificate available" → server not sending cert (wrong port?)
# "alert handshake failure" → cipher suite mismatch or cert validation failure
# "alert unknown ca" → client doesn't trust the CA
# "certificate verify failed" → hostname mismatch or expired cert

# Check which TLS version was negotiated
nmap --script ssl-enum-ciphers -p 443 example.com

# Monitor TLS handshake latency (bpftrace)
bpftrace -e '
kprobe:tls_do_handshake { @start[tid] = nsecs; }
kretprobe:tls_do_handshake { @lat = hist(nsecs - @start[tid]); delete(@start[tid]); }'

# Certificate chain issues
openssl s_client -connect example.com:443 -showcerts 2>/dev/null \
    | awk '/BEGIN CERT/,/END CERT/' | split - cert
for f in cert*; do openssl x509 -in $f -noout -subject -issuer; done

# Test OCSP staple
openssl s_client -status -connect example.com:443 < /dev/null 2>&1 | grep -i ocsp

---

Security Implications

• Forward secrecy: TLS 1.3 mandates ECDHE — compromise of the server's long-term key does not expose past sessions. TLS 1.2 with RSA key exchange (no FS) does expose them; ensure ssl_prefer_server_ciphers on and DHE/ECDHE-only cipher suites in TLS 1.2.
• Certificate pinning: hardcoding expected certificate fingerprints prevents MITM via compromised CAs. Used in high-value mobile apps and microservice mTLS. Breaking on certificate rotation is the main operational risk.
• TLS 1.0/1.1 deprecation: IETF deprecated TLS 1.0 and 1.1 in RFC 8996 (2021). All major browsers removed support. Disable explicitly: ssl_protocols TLSv1.2 TLSv1.3; (minimum TLS 1.2).
• BEAST (TLS 1.0 CBC attack): exploited CBC mode's predictable IV. Fixed in TLS 1.1 by using random IV per record. Fully eliminated in TLS 1.3 which mandates AEAD only.
• Session ticket key exposure: a stolen STEK allows decryption of all sessions using that key. Use short STEK rotation intervals (1–24 hours) and protect STEK storage.

---

Performance Implications

Operation	Latency	Notes
TLS 1.3 handshake	1 RTT + ~1ms CPU	X25519 DH + AES
TLS 1.3 0-RTT	0 RTT + ~0.5ms CPU	PSK + early data
AES-128-GCM encrypt (1 core)	~15 GB/s	AES-NI required
ChaCha20-Poly1305 (1 core)	~4 GB/s	No AES-NI needed
Certificate validation	0.1–1ms	Cached after first
OCSP fetch (no staple)	50–300ms	Network dependent

For a 1 Gbps HTTPS server: - Handshake cost dominates at high connection rates (>10K/s) — use session tickets/0-RTT - Record encryption cost dominates for bulk transfers — use AES-NI, consider kTLS

---

Failure Modes and Real Incidents

Incident: STEK not rotated (2018, major SaaS) Session ticket encryption keys were provisioned once at server startup and never rotated. After a server compromise exposed the STEK, all TLS sessions for the past 90 days could be decrypted from previously captured traffic. Fix: implement STEK rotation with a background key management daemon.

Incident: mTLS clock skew causing certificate rejection (Kubernetes, common) Pod certificates issued by the cluster CA use notBefore = current time. If the pod's system clock is ahead of the CA's clock by >1 minute, the certificate appears invalid (notBefore in the future). Fix: NTP sync on all nodes; certificate issuance with notBefore = now - 5 minutes skew tolerance.

Failure Mode: TLS 1.3 middlebox compatibility Some corporate firewalls (Palo Alto PAN-OS < 9.0, Cisco ASA) cannot parse TLS 1.3 ServerHello correctly, causing handshake failures for internal hosts. Clients fall back to TLS 1.2. Symptom: HTTPS works from home but not from office. Fix: update firewall firmware or disable TLS 1.3 inspection.

---

Modern Usage

• QUIC + TLS 1.3 integration: QUIC uses TLS 1.3 key derivation for all its keys — the QUIC handshake IS the TLS 1.3 handshake, just integrated at the transport layer (see 04-quic-protocol.md)
• Istio/Envoy mTLS: 100% of service mesh deployments use mTLS; certificate rotation is automated via SPIRE/SPIFFE without application involvement
• Let's Encrypt ACME: automated certificate issuance reduced cert management from "weeks per cert" to "minutes"; 90-day certificates with automatic renewal are now standard
• kTLS + sendfile: nginx (>1.21.4) and Apache (>2.4.52) both support kTLS for HTTPS static file serving on Linux 5.2+

---

Future Directions

• TLS 1.3 post-quantum key exchange: IETF is standardizing hybrid X25519+Kyber key exchange (X25519Kyber768Draft00) to protect against future quantum computer attacks. Chrome and Cloudflare have deployed experimental hybrid PQ KEM.
• Delegated credentials (RFC 9345): allows a CDN to use short-lived certificates signed by the origin's long-term cert — limits the blast radius of CDN compromise while enabling CDN TLS termination
• Encrypted ClientHello (ECH): encrypts the SNI and other ClientHello extensions, preventing passive observers from knowing which hostname is being visited. In development for TLS 1.3.

---

Exercises

1. Capture a TLS 1.3 handshake with Wireshark to example.com. Using the NSS key log file (SSLKEYLOGFILE=/tmp/keys.log curl https://example.com), decrypt the capture in Wireshark. Identify each handshake message and the key derivation steps.

2. Implement a minimal TLS 1.3 server in Python using the ssl module. Add OCSP stapling. Benchmark handshake rate (connections/second) with openssl s_time. Then enable ssl.OP_NO_TICKET and measure the impact on resumption rate.

3. Set up mTLS between two services using cfssl to create a private CA. Issue client and server certificates. Configure nginx to require client certificates. Verify that connections without a valid client cert are rejected with ssl_verify_client on.

4. Enable kTLS on a Linux 5.2+ system with OpenSSL 3.0+. Serve a 1GB file via HTTPS with and without kTLS. Use perf stat to compare CPU cycles and strace -c to compare time in the TLS encryption path.

5. Analyze TLS handshake failures from a captured HAProxy access log. Identify the three most common failure reasons (cipher mismatch, expired cert, unknown CA, version mismatch). Write a script that parses openssl s_client output for each failure and categorizes the root cause.

---

References

• RFC 8446 — The Transport Layer Security (TLS) Protocol Version 1.3
• RFC 8996 — Deprecating TLS 1.0 and TLS 1.1
• RFC 6066 — TLS Extensions (OCSP stapling, SNI)
• RFC 9345 — Delegated Credentials for TLS and DTLS
• net/tls/ — Linux kTLS implementation
• ssl/statem/ — OpenSSL TLS state machine
• Rescorla, E. The Transport Layer Security (TLS) Protocol Version 1.3 (RFC 8446). Author commentary. 2018.
• Bhargavan, K. et al. Transcript Collision Attacks: Breaking Authentication in TLS, IKE and SSH. NDSS 2016. (SLOTH attack)
• openssl-s_client(1), openssl-s_server(1) — debugging tools
• nginx docs: ngx_http_ssl_module — ssl_stapling, session tickets
• Cloudflare blog. An overview of TLS 1.3 and Q&A. 2018.