Skip to content

Scalability Fundamentals: Building Systems That Grow

Overview

Scalability is the capacity of a system to handle growing load — more users, more data, more requests — while maintaining acceptable performance and reliability. A scalable system is not just one that is big, but one designed so that making it bigger is achievable without fundamental redesign. Scalability is a property that must be designed in; retrofitting scalability into an existing architecture is extremely expensive.

Understanding scalability requires confronting the specific nature of the load your system will face. "Our system needs to scale" is not actionable. "Our system must handle 50,000 concurrent users, with 95% of requests being reads and 5% writes, where reads are latency-sensitive but writes can tolerate 200ms" is actionable. Every scalability decision — what to cache, how to shard, whether to use synchronous or asynchronous processing — follows from the specific load profile.

Prerequisites

  • Familiarity with HTTP, REST, and basic client-server architecture
  • Understanding of databases and basic SQL
  • Awareness of network fundamentals (latency, bandwidth, throughput)
  • Basic probability and statistics for estimation
  • Understanding of the CAP theorem

Vertical vs Horizontal Scaling

The first decision in scaling is whether to scale up (more powerful hardware) or scale out (more instances of less powerful hardware).

VERTICAL SCALING (Scale Up):

  [Server: 16 CPU, 64 GB RAM] → [Server: 128 CPU, 1 TB RAM]

  Pros:
  - Simple: no application code changes required
  - No distributed systems complexity
  - Strong consistency (single machine)
  - Fast inter-component communication (shared memory)

  Cons:
  - Hardware limits: you eventually cannot buy bigger hardware
  - Cost: bigger machines have superlinear price increases
  - Single point of failure: one machine = one failure domain
  - Downtime for upgrades: stopping for hardware upgrade

  Reality: vertical scaling is a useful short-term strategy but
           not a long-term one. At some point you hit the hardware
           ceiling or the cost becomes prohibitive.

HORIZONTAL SCALING (Scale Out):

  [1 × 16-CPU server] → [10 × 16-CPU servers]

  Pros:
  - Linear cost scaling: 10x servers ≈ 10x cost (roughly)
  - No hardware limit: add as many instances as needed
  - Fault tolerance: losing one of 10 servers loses 10% capacity
  - Rolling upgrades: update instances one at a time

  Cons:
  - Distributed systems complexity: coordination, consensus, consistency
  - Network overhead: communication between nodes
  - Session management: which server handles which user?
  - Shared state: databases, caches must be accessible to all instances

  Prerequisite: services must be stateless (or state must be
                externalized to a shared data store).

Stateless vs Stateful Services

The stateless/stateful distinction is the most important architectural decision for horizontal scalability.

STATELESS SERVICE:

  Each request contains all information needed to process it.
  No session state stored on the server.

  [Load Balancer]
      |
      +---- [Server 1] ←── any request can go here
      |
      +---- [Server 2] ←── any request can go here
      |
      +---- [Server 3] ←── any request can go here

  Adding Server 4 is trivially easy: point LB at it.
  Servers can crash without losing user state.
  Session state lives in: JWT tokens, cookies with state in DB.

STATEFUL SERVICE:

  Server retains state about a client across requests.

  [Load Balancer with session affinity]
      |
      +---- [Server 1] ←── User A must ALWAYS come here (has A's state)
      |
      +---- [Server 2] ←── User B must ALWAYS come here (has B's state)
      |
      +---- [Server 3] ←── User C must ALWAYS come here (has C's state)

  Problems:
  - If Server 1 crashes, User A loses their state
  - Adding Server 4 requires migrating some state to it
  - Load balancer must track session affinity (adds complexity)
  - Cannot easily scale individual users without redistributing state

EXTERNALIZED STATE (best of both):
  Service is stateless, but uses shared stateful data stores.

  [Load Balancer]
      |
      +---- [Server 1] ──→ [Redis: sessions, caches]
      |
      +---- [Server 2] ──→ [PostgreSQL: persistent data]
      |
      +---- [Server 3]

  Servers are stateless (horizontally scalable).
  State lives in dedicated data stores (scaled independently).

Load Balancing Algorithms

Load balancers distribute incoming requests across server instances. The algorithm choice affects performance, session handling, and resource utilization.

ROUND ROBIN:
  Requests distributed cyclically: 1→2→3→1→2→3→...

  Simple and fair assuming all requests are equal.
  Problem: requests are not equal. A heavy query to Server 1
           while Server 2 gets lightweight requests.

WEIGHTED ROUND ROBIN:
  Assign weights based on server capacity.
  Server 1 (32 CPU): weight 2 → 2x more requests
  Server 2 (16 CPU): weight 1 → normal requests

LEAST CONNECTIONS:
  New request goes to server with fewest active connections.

  Better than round-robin when request durations vary significantly.
  Accounts for slow requests holding connections.

LEAST RESPONSE TIME:
  New request goes to server with lowest average response time.
  More sophisticated: accounts for actual server performance.

IP HASH / CLIENT HASH:
  hash(client_IP) % num_servers → determines server

  Same client always goes to same server.
  Good for: session affinity without session stores.
  Problem: loses session when a server is added or removed.

CONSISTENT HASHING:
  Places servers and keys on a ring. Request goes to the
  nearest server clockwise on the ring.

  Adding/removing a server only reassigns ~1/n of keys.
  Essential for distributed caches (see caching strategies).

  [Ring: 0 ─── Server A ─── Server B ─── Server C ─── 360°]
  Key K hash → falls between A and B → assigned to B
  Add Server D between A and B → only keys between A-D move to D

Capacity Planning

Capacity planning translates business growth projections into infrastructure requirements. It prevents two bad outcomes: over-provisioning (wasted cost) and under-provisioning (degraded user experience).

CAPACITY PLANNING PROCESS:

1. Measure current load:
   - Requests/second (peak, average, daily variation)
   - CPU, memory, disk, network utilization at various loads
   - Database query volume, connection count, query latency
   - Storage growth rate per day/month

2. Model growth:
   - Business projections: 3x users in 12 months?
   - Historical growth rate: 10% MoM?
   - Planned events: product launch, Super Bowl, Black Friday?

3. Determine resource-per-request ratios:
   - CPU-seconds per 1000 requests
   - DB reads per request, DB writes per request
   - Storage per user, per transaction

4. Project requirements:
   Current load × growth multiplier × headroom factor

5. Add safety margin:
   - Provision for 2-3x peak load
   - Systems should operate comfortably at 60-70% peak utilization
   - At 80%+ utilization, latency climbs nonlinearly (queuing theory)

EXAMPLE:
  Current: 1,000 req/sec, 10 servers at 50% CPU
  Growth: 5x over 12 months
  Peak multiplier: Black Friday = 3x normal

  Required: 5,000 req/sec sustained, 15,000 req/sec peak
  Current capacity: 10 servers at 100% = 2,000 req/sec
  Required capacity: 15,000 req/sec / 200 req/sec/server = 75 servers

  → Need 75 servers provisioned, running at ~27% average utilization,
    which leaves headroom for peak and unexpected growth.

Back-of-the-Envelope Estimation

Back-of-the-envelope estimation is the skill of quickly computing rough capacity numbers from first principles. It is essential for system design discussions and interviews.

MEMORIZE THESE NUMBERS:

Latency numbers (circa 2024):
  L1 cache access:        0.5 ns
  L2 cache access:        7 ns
  Main memory access:     100 ns
  SSD random read:        100 µs (100,000 ns)
  HDD random seek:        10 ms (10,000,000 ns)
  Round trip same DC:     0.5 ms
  Round trip CA → NL:     150 ms

Throughput:
  SSD sequential read:    500 MB/s
  Network:                1 Gbps = 125 MB/s (gigabit)
  Memory bandwidth:       10-50 GB/s

Storage sizes:
  ASCII char: 1 byte
  UTF-8 char: 1-4 bytes (assume 2-3 average)
  Int/float:  4 bytes
  UUID:       16 bytes
  Timestamp:  4-8 bytes
  URL:        100 bytes average
  Tweet:      300 bytes
  Photo:      300 KB average
  Video:      1 MB/sec (720p)

Common time conversions:
  1 day = 86,400 seconds ≈ 100,000
  1 month = 2.6M seconds ≈ 2.5M
  1 year = 31.5M seconds ≈ 30M

ESTIMATION TEMPLATE:
  System: Twitter (for reference)

  Users: 300M daily active users
  Tweets/day: 300M users × 0.1 tweets/user = 30M tweets/day
  Write QPS: 30M / 86,400 ≈ 350 writes/sec
  Read QPS: assume 100 reads per write = 35,000 reads/sec

  Storage per tweet: 300 bytes text + metadata ≈ 1 KB
  Storage/day: 30M × 1 KB = 30 GB/day
  Storage/year: 30 GB × 365 ≈ 10 TB/year
  Storage/5 years: 50 TB (just text — images/video vastly more)

Eliminating Single Points of Failure

A Single Point of Failure (SPOF) is any component whose failure brings down the entire system. Every SPOF is an availability ceiling.

IDENTIFYING SPOFs:

  Draw the architecture. For each component, ask:
  "If this component fails, can the system continue serving users?"

  YES → not a SPOF (redundant or non-critical path)
  NO  → SPOF — must be addressed

COMMON SPOFS:
  - Single load balancer (add redundant LBs with IP failover)
  - Single database (add read replicas, standby failover)
  - Single Kafka broker (use replication factor ≥ 3)
  - Single region/AZ deployment (deploy to multiple AZs)
  - Single DNS provider (use secondary DNS)
  - Hardcoded dependency URLs (use service discovery)
  - Single external API provider (add fallback provider)

SPOF ELIMINATION PATTERN:

  Before:
  Client → [Single Load Balancer] → [Server Pool] → [Single Database]

  After:
  Client → [DNS with health checks]
               ↓
          [LB Active]──[LB Standby (hot standby)]
               ↓
          [Server Pool] (horizontal, stateless)
               ↓
          [DB Primary] ──replication──> [DB Replica]
               ↓
          [DB Replica used for reads]

  Availability = product of component availabilities:
  Single DB at 99.9%: system availability ≤ 99.9%
  Primary + Replica with auto-failover: 99.99%+

The Fallacies of Distributed Computing

Peter Deutsch (Sun Microsystems, 1991) identified eight assumptions that developers make when building distributed systems, all of which are false:

1. THE NETWORK IS RELIABLE
   Reality: packets are lost, connections drop, routers fail.
   Implication: all network calls need retries, timeouts, circuit breakers.

2. LATENCY IS ZERO
   Reality: network calls add 0.5ms–150ms depending on distance.
   Implication: N+1 queries, synchronous chains of service calls, and
                sequential DB queries are expensive. Batch and parallelize.

3. BANDWIDTH IS INFINITE
   Reality: bandwidth is limited and shared.
   Implication: compress data, avoid large payloads, page responses.

4. THE NETWORK IS SECURE
   Reality: the network is not secure; data must be encrypted in transit.
   Implication: TLS everywhere, mutual TLS for service-to-service.

5. TOPOLOGY DOESN'T CHANGE
   Reality: servers come and go, IPs change, services scale up and down.
   Implication: use service discovery, not hardcoded IPs.

6. THERE IS ONE ADMINISTRATOR
   Reality: multiple teams manage different components with different policies.
   Implication: design for operational autonomy and clear API contracts.

7. TRANSPORT COST IS ZERO
   Reality: serialization, deserialization, and network cost are real.
   Implication: design APIs to minimize round trips, use efficient encoding.

8. THE NETWORK IS HOMOGENEOUS
   Reality: different network segments have different properties.
   Implication: design for heterogeneous latency and reliability profiles.

CAP Theorem Applied

The CAP theorem (Brewer, 2000; formalized by Gilbert and Lynch, 2002) states that a distributed system can guarantee at most two of: - Consistency (C): all nodes see the same data at the same time - Availability (A): every request receives a response - Partition tolerance (P): the system continues operating despite network partitions

Since network partitions are a reality (not a choice), the practical choice is between CP (consistency over availability) and AP (availability over consistency) during a partition.

CAP IN PRACTICE:

  When a network partition occurs, choose:

  CP systems (consistency priority):
    Example: Traditional RDBMS, ZooKeeper, etcd, HBase
    Behavior: During partition, reject writes to minority partition
              (or stall until quorum restored).
    Use when: financial transactions, inventory counts, user authentication.
    Cost: brief unavailability during partition.

  AP systems (availability priority):
    Example: Cassandra, DynamoDB, CouchDB
    Behavior: During partition, continue serving reads and writes
              on both sides. After partition heals, reconcile.
    Use when: product catalogs, user profiles, social graphs, logs.
    Cost: temporary inconsistency (eventual consistency).

  Mixed: use CP for critical data, AP for eventually consistent data.

CONSISTENCY SPECTRUM:
  Strong ←───────────────────────────────────→ Weak

  Linearizable | Sequential | Causal | Monotonic | Eventual

  Linearizable: operations appear instantaneous and ordered globally
  Sequential: all nodes see operations in same order (not necessarily real-time)
  Causal: causally related operations are seen in order
  Eventual: if no new writes, all nodes eventually converge

Scalability Patterns Decision Tree

SYSTEM DESIGN SCALABILITY DECISION TREE:

Is the service stateless?
  NO → Externalize state to Redis/database → YES
  YES ↓

Can one server handle peak load?
  YES → Scale vertically first (simpler)
  NO  → Scale horizontally ↓

Is the load read-heavy or write-heavy?
  READ-HEAVY (>90% reads):
    → Add read replicas (database)
    → Add caching layers (Redis, CDN)
    → Consider eventual consistency (AP model acceptable?)

  WRITE-HEAVY:
    → Consider write sharding (partition data)
    → Consider async processing (queue + workers)
    → Consider CQRS (separate write and read models)

  MIXED:
    → Read replicas + write primary
    → Caching for reads
    → Queue for write spikes

Is data volume the bottleneck?
  YES → Shard data (horizontal partitioning)
  NO  → Scale compute (more servers)

Is the bottleneck a single service?
  YES → Identify which tier (web, app, DB) and scale that tier independently
  NO  → Bottleneck is the whole system → architectural redesign needed

Production Example: Twitter's Scalability Journey

Twitter's early architecture (2006-2009) was a single Rails application with a MySQL database. At scale, it had famously catastrophic "Fail Whale" outages. The key scalability improvements:

  1. Split the monolith into services (tweet storage, timeline service, social graph, etc.)
  2. Replaced the pull-based timeline (query DB for all tweets from followed users) with a fanout/push model (pre-computed timeline cache per user using Redis)
  3. Moved from MySQL to a combination: Manhattan (Twitter's proprietary distributed key-value store) for tweets and user data, Redis for timeline caches
  4. The timeline fanout challenge: a user with 10 million followers tweets → write to 10 million cache entries. Solution: hybrid model — pre-compute timelines for most users, compute on-the-fly for users following celebrities.

This illustrates the core scalability theme: no single solution works for all load profiles. Twitter's timeline problem required a custom solution because the access pattern (write once, read billions of times) had specific characteristics.

Debugging Notes

The 80/20 CPU utilization cliff: At around 70-80% CPU utilization, latency climbs non-linearly due to queuing (Little's Law). If your system is "only at 80% CPU," it may be in a degraded latency state. Provision for 60% average utilization.

Thundering herd on scale-out: When you add new servers to a cache cluster, the new servers start cold. All keys they own miss the cache, causing a spike of DB queries. Use consistent hashing (keys assigned to specific servers) and warm new servers before adding to the pool.

Connection pool exhaustion: As you scale horizontally (more app servers), each server has its own connection pool to the database. Total connections = servers × pool_size. 100 servers × 20 connections = 2,000 DB connections. PostgreSQL's limit is typically 200-500 without a pooler. Use PgBouncer or similar.

Security Implications

  • Horizontal scaling increases attack surface: more instances mean more exposure. Ensure security group rules and network policies scale with instance count.
  • Session management in stateless systems: JWT tokens carry all user claims; a compromised token cannot be revoked without a token blacklist. Design token expiry aggressively.
  • Load balancer as DDoS target: the load balancer is itself a SPOF for attacks. Use anycast, DDoS scrubbing services, and rate limiting at the edge.

Performance Implications

  • Amdahl's Law: the speedup from parallelization is limited by the serial fraction. If 20% of work is serial, maximum speedup is 5x regardless of how many CPUs you add.
  • Universal Scalability Law (Gunther): adds contention and coherence costs to Amdahl's model. Real systems see throughput peak and then decline as parallelism increases due to coordination overhead.
  • Network latency dominates at scale: 10ms × 10 serial service calls = 100ms baseline latency before any processing. Minimize serial dependency chains.

Failure Modes

Premature optimization: Scaling for a load that does not yet exist, at the cost of complexity that slows development. Follow the advice: "do not design for 1M users when you have 1000."

Database as the bottleneck: Application servers scale horizontally, but the database is a single instance. Almost every scaling problem eventually becomes a database scaling problem. Anticipate this.

Single-region architecture: Scaling within a region without multi-region consideration. A region-level failure takes down the entire system regardless of how many servers you have.

Modern Usage and Future Directions

Auto-scaling (AWS Auto Scaling Groups, Kubernetes Horizontal Pod Autoscaler) automates the "add more servers" response to increased load. Serverless architectures (AWS Lambda, Vercel Functions) make horizontal scaling invisible — the platform scales automatically. However, serverless introduces cold start latency and per-request cost models that require different capacity planning.

The future of scalability includes adaptive routing (route traffic to the nearest or least-loaded region dynamically), edge computing (run compute near users globally), and AI-driven capacity planning (predict load spikes from business signals before they occur).

Exercises

  1. Estimate the storage requirements for a photo-sharing service like Instagram: 500M daily active users, 10% post at least one photo per day, photos average 2 MB, stored for 10 years. Calculate both raw storage and bandwidth requirements.

  2. A web service runs on 10 servers at 50% CPU utilization. Traffic is expected to 5x in 6 months. How many servers are needed? What other components need scaling?

  3. Design the data tier for a read-heavy service (95% reads, 5% writes) with 100,000 requests/second. What does the scaling architecture look like?

  4. Identify all single points of failure in a simple architecture: one load balancer, three app servers, one database, one Redis instance. For each SPOF, propose a solution.

  5. Apply the CAP theorem to the following use cases: a bank account balance, a shopping cart, a user's profile bio, a real-time collaborative document. For each, which consistency model is appropriate and why?

References

  • Kleppmann, M. Designing Data-Intensive Applications. O'Reilly, 2017. Chapter 1: Reliable, Scalable, and Maintainable Applications.
  • Fowler, M. Patterns of Enterprise Application Architecture. Addison-Wesley, 2002.
  • Deutsch, P. "The Eight Fallacies of Distributed Computing." Sun Microsystems, 1994-1997.
  • Brewer, E. "Towards Robust Distributed Systems." PODC Keynote, 2000.
  • Gilbert, S., Lynch, N. "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services." ACM SIGACT News, 2002.
  • Gunther, N.J. Analyzing Computer System Performance with Perl::PDQ. Springer, 2005.
  • Krikorian, R. "New Tweets per second record, and how!" Twitter Engineering Blog, 2013.