05 — Networking and Internet History: From ARPANET to the Always-On World
Technical Overview
The internet is arguably the most transformative infrastructure project in human history. What began as a US military research network with four nodes in 1969 became a global communication system connecting 5 billion people by 2024. This file traces the engineering decisions, research papers, and cultural moments that built this infrastructure: packet switching as a theoretical framework, ARPANET as the first implementation, TCP/IP as the universal protocol, DNS as the naming system, Ethernet as the local networking standard, the BSD TCP/IP stack that made the internet accessible, the World Wide Web that made it usable, and the commercial expansion that made it ubiquitous. These are not just historical events — they are the design choices embedded in every network interaction you make.
Prerequisites
- Basic understanding of what a network protocol is
- Familiarity with the concept of IP addresses and ports
- Awareness of what a client/server model means
Theoretical Foundations: Packet Switching (1961–1965)
Paul Baran and Distributed Networks (1964)
Paul Baran at RAND Corporation, funded by the US Air Force, was tasked with designing a communication network that could survive nuclear attack. The existing telephone network was circuit-switched: a dedicated connection was established between two endpoints for the duration of a call. If key switching nodes were destroyed, the network would fail.
Baran's 1964 RAND report, "On Distributed Communications," proposed three topologies:
Centralized (fragile): Decentralized (better): Distributed (resilient):
* *---* *---* *-*-*-*-*
/|\ /| |\ / |\ |/|/|/|\|\
* | * * | | * | * *-*-*-*-*-*-*
\|/ \| |/ \ |/ |/|/|/|\|\|
* *---* *---* *-*-*-*-*-*
Single hub = single Regional hubs = No single node
point of failure multiple SPOFs critical; traffic
routes around damage
Baran's distributed network would use message blocks (what we now call packets): break messages into fixed-size chunks, route each independently, reassemble at the destination. Each node in the network would receive a block, store it briefly, and forward it toward the destination — "store-and-forward."
Independently, Donald Davies at the UK National Physical Laboratory in 1965 proposed the same concept, coining the term "packet switching."
Neither Baran's nor Davies's networks were built directly, but their papers became the theoretical foundation for ARPANET.
ARPANET: The First Packet-Switched Network (1969)
Context and Motivation
ARPA (Advanced Research Projects Agency, now DARPA) was created in 1958 after Sputnik demonstrated that the US could be surprised by technological advances. ARPA funded basic research at universities and laboratories. By the mid-1960s, ARPA-funded research sites had computers, but each computer used different hardware and software. Researchers at one site couldn't easily communicate with researchers at another.
J.C.R. Licklider's 1963 memo "Memorandum for Members and Affiliates of the Intergalactic Computer Network" proposed a network connecting ARPA-funded researchers. Lawrence Roberts, at ARPA, turned Licklider's vision into a funded engineering project.
The First Four Nodes
Bolt Beranek and Newman (BBN) won the contract to build the Interface Message Processors (IMPs) — the packet-switching routers, built on Honeywell 516 minicomputers. The first four nodes:
ARPANET - October 29, 1969 (first message)
UCLA SRI (Stanford Research Institute)
(Sigma 7) (SDS 940)
*----[IMP]--------[IMP]----*
| |
| ARPANET |
| |
*----[IMP]--------[IMP]----*
UCSB University of Utah
(IBM 360/75) (DEC PDP-10)
Distance: UCLA to SRI = 400 miles
First message sent: October 29, 1969, 10:30 PM
Message: "LO" (attempting to type "LOGIN" before system crashed)
By December 1969: all four nodes connected
The first message sent over ARPANET was "LO" — Charley Kline at UCLA was trying to log into the SRI computer by typing "LOGIN" when the remote system crashed after receiving the 'O'. The internet's first transmission was thus a crash. By December 1969, all four nodes were connected and working.
Growth and NCP
The initial ARPANET protocol, NCP (Network Control Program), was implemented in 1970. By 1972, ARPANET had 37 nodes; by 1977, over 100. The first public demonstration was at the International Computer Communication Conference in Washington DC in October 1972, where ARPANET's capabilities were demonstrated to 900 attendees — the first public internet demo.
Email emerged almost immediately. Ray Tomlinson at BBN sent the first email between ARPANET machines in 1971 and chose the @ sign to separate user from host. By 1973, three-quarters of all ARPANET traffic was email.
TCP/IP: The Universal Protocol (1974)
The Problem with NCP
NCP assumed a homogeneous network. As new networks emerged — satellite networks, packet radio networks, Ethernet LANs — the need arose for an "internetwork" protocol: a protocol for connecting networks of different types. NCP could not route between heterogeneous networks.
Cerf and Kahn's Paper (1974)
Vinton Cerf (Stanford) and Robert Kahn (ARPA) published "A Protocol for Packet Network Intercommunication" in IEEE Transactions on Communications in May 1974. This paper described what became TCP/IP.
Key design principles: 1. Networks are independent: Each network can operate according to its own internal design. The internetwork protocol runs on top. 2. No global control: No central authority manages routing. Each router makes forwarding decisions independently. 3. Best-effort delivery: The network makes no guarantees. Packets may be lost, reordered, or duplicated. Higher-level protocols handle reliability. 4. Gateways (routers): Devices that connect networks, translate between protocols, and forward packets. 5. End-to-end principle: Complex functions (reliability, error detection, flow control) should be implemented at endpoints, not in the network. The network core should be simple.
TCP/IP Split (1978)
Originally, Cerf and Kahn's protocol was a single protocol (TCP). In 1978, Jon Postel, Cerf, and Danny Cohen split it into two: - IP (Internet Protocol): Unreliable, connectionless datagram delivery. Routes packets hop-by-hop. - TCP (Transmission Control Protocol): Reliable, ordered, connection-oriented delivery built on top of IP.
This separation was crucial: IP could carry other protocols (UDP, ICMP, later OSPF and BGP directly) not just TCP. The layered architecture enabled independent evolution of the transport and network layers.
TCP/IP Layering:
Application Layer | HTTP, FTP, SMTP, DNS, SSH
| (application-specific protocols)
Transport Layer | TCP (reliable) or UDP (unreliable)
| (port-based multiplexing, TCP: connection & reliability)
Internet Layer | IP (IPv4 or IPv6)
| (addressing, routing, hop-by-hop delivery)
Link Layer | Ethernet, WiFi, PPP, etc.
| (local delivery on a single network segment)
Each layer wraps the layer above in its own header:
+--------+---------+-------+------+----------+
| Frame | IP | TCP | HTTP | Payload |
| Header | Header | Header| data | |
+--------+---------+-------+------+----------+
^ ^ ^ ^
Link Network Transport Application
The Flag Day: NCP to TCP/IP (January 1, 1983)
ARPANET switched from NCP to TCP/IP on January 1, 1983 — a "flag day" cutover. Jon Postel coordinated the transition. All ARPANET nodes had to upgrade simultaneously (or be disconnected). This date is considered the operational birthday of the internet. After this date, any network using TCP/IP could interoperate with any other TCP/IP network — the technical precondition for internet growth.
DNS: Naming the Internet (1983)
As ARPANET grew, the mapping of hostnames to IP addresses was maintained in a single file, HOSTS.TXT, manually updated and distributed from the Stanford Research Institute. By 1982, with hundreds of hosts, this system was breaking down: the file was large, updates were slow, and name conflicts were common.
Paul Mockapetris designed and implemented DNS (Domain Name System) in 1983, described in RFC 882 and 883. DNS was a distributed database:
- The namespace was hierarchical: mail.example.com is in the com top-level domain (TLD), under example, under mail
- Name servers were authoritative for particular zones
- Resolvers queried name servers, caching results for the TTL (time-to-live)
- Root servers (initially 13 IP addresses) knew about all TLDs
DNS Hierarchy:
. (root)
/|\
/ | \
.com .org .net .edu .gov ...
/ |
.example .ietf
| |
mail.example www.ietf.org
DNS Resolution (recursive):
1. Client asks local resolver for www.example.com
2. Resolver asks root server: "who handles .com?"
3. Root replies: "ask ns1.verisign.com"
4. Resolver asks ns1.verisign.com: "who handles example.com?"
5. ns1.verisign replies: "ask ns1.example.com"
6. Resolver asks ns1.example.com: "what is www.example.com?"
7. ns1.example.com replies: "192.0.2.1"
8. Resolver returns 192.0.2.1 to client, caches result for TTL
DNS remains one of the most critical infrastructure components of the internet. When DNS fails, the internet appears broken (even if IP routing works). The 2016 Dyn DDoS attack took down major websites by attacking DNS infrastructure rather than the websites themselves.
Ethernet: The Local Area Network Standard (1973–1980)
Metcalfe's Ethernet (1973)
Robert Metcalfe, working on his PhD at Harvard and simultaneously at Xerox PARC, designed Ethernet for his 1973 doctoral thesis. The design was inspired by ALOHAnet — a 1970 packet radio network at the University of Hawaii. Metcalfe's key insight was CSMA/CD (Carrier Sense Multiple Access with Collision Detection):
CSMA/CD Algorithm:
1. Listen to the cable (carrier sense)
2. If cable is idle, start transmitting
3. While transmitting, continue listening
4. If collision detected (two stations transmitted simultaneously):
a. Stop transmitting immediately
b. Send jam signal (reinforces collision)
c. Wait a random backoff time (exponential backoff)
d. Return to step 1
Exponential backoff: after nth collision, wait random(0, 2^n - 1) slot times
This ensures that repeated collisions resolve: probability approaches 0
The original Ethernet at PARC ran at 2.94 Mbps on coaxial cable. Metcalfe co-designed it with David Boggs.
DIX Ethernet (1980) and IEEE 802.3
DEC, Intel, and Xerox (DIX) collaborated to publish the "Blue Book" standard for 10 Mbps Ethernet in 1980. IEEE 802.3, published in 1983, standardized Ethernet internationally. The progression:
Ethernet Speed Timeline:
1973 Experimental Ethernet: 2.94 Mbps (Xerox PARC)
1980 10BASE5: 10 Mbps (thick coax)
1985 10BASE2: 10 Mbps (thin coax, cheaper)
1990 10BASE-T: 10 Mbps (twisted pair, star topology)
1995 100BASE-TX (Fast): 100 Mbps (twisted pair)
1998 1000BASE-T (Gigabit): 1000 Mbps = 1 Gbps (twisted pair)
2002 10GBASE-SR: 10 Gbps (fiber)
2010 40GbE/100GbE: 40/100 Gbps (data center)
2017 400GbE: 400 Gbps (high-performance)
The shift from coaxial bus topology (10BASE5, 10BASE2) to star topology with a central hub/switch (10BASE-T) in the 1990s was a critical change: a break in coax took down the entire segment; a break in a twisted-pair cable only disconnected one device.
BSD TCP/IP: Making the Internet Ubiquitous (1983)
DARPA funded Berkeley to implement TCP/IP in BSD Unix. Bill Joy and Sam Leffler led the implementation. The result shipped in 4.2BSD in 1983:
- A complete, production-quality TCP/IP stack
- BSD sockets API:
socket(),bind(),listen(),accept(),connect(),send(),recv(),close()— the API that every networked program still uses today (on Linux, Windows, macOS, iOS, Android) - UDP, TCP, IP, ICMP, ARP — the full suite
- Free and open source
The BSD socket API was so well-designed that it became universal. Windows Winsock, Java's java.net, Python's socket module — all model their APIs on BSD sockets. The int socket(int domain, int type, int protocol) function signature has been unchanged for 40 years.
/* BSD Sockets API -- 1983, unchanged in form today */
int sockfd = socket(AF_INET, SOCK_STREAM, 0); /* Create TCP socket */
bind(sockfd, (struct sockaddr*)&addr, sizeof(addr)); /* Bind to port */
listen(sockfd, SOMAXCONN); /* Start listening */
int client = accept(sockfd, NULL, NULL); /* Accept connection */
send(client, "Hello\n", 6, 0); /* Send data */
recv(client, buf, sizeof(buf), 0); /* Receive data */
close(client); /* Close connection */
Before BSD's free TCP/IP stack, running TCP/IP required expensive proprietary implementations. After 4.2BSD, any university or company could run TCP/IP for free. This is what made the internet's expansion from military/research network to commercial internet possible.
The World Wide Web (1989–1993)
Tim Berners-Lee at CERN
Tim Berners-Lee, a British physicist at CERN (the European particle physics laboratory), was frustrated that CERN's thousands of researchers used incompatible computer systems, making information sharing difficult. In March 1989, he circulated a proposal: "Information Management: A Proposal." His manager Mike Sendall wrote on the cover: "Vague but exciting."
Berners-Lee proposed three technologies:
1. HTML (HyperText Markup Language): A document format with hyperlinks — text that, when clicked, navigated to another document
2. HTTP (HyperText Transfer Protocol): A protocol for requesting and serving HTML documents
3. URL (Uniform Resource Locator): A naming scheme for documents: http://host/path
HTTP/1.0 Request/Response:
Client sends:
GET /page.html HTTP/1.0
Host: info.cern.ch
<blank line>
Server responds:
HTTP/1.0 200 OK
Content-Type: text/html
<blank line>
<html><body>Hello World</body></html>
That's it. The first web protocol was this simple.
Berners-Lee implemented the first web server on a NeXT workstation. The first website (http://info.cern.ch) went live on August 6, 1991. Crucially, CERN published the web as open technology with no patent restrictions — a decision that made universal adoption possible.
Mosaic Browser (1993)
Marc Andreessen and Eric Bina at the National Center for Supercomputing Applications (NCSA) at the University of Illinois wrote Mosaic — the first graphical web browser — in six weeks, releasing it in January 1993. Mosaic displayed images inline with text (earlier browsers required separate windows for images), supported multiple protocols (FTP, Gopher, HTTP), and ran on Unix, Windows, and Mac. It was free.
Mosaic created the concept of "browsing the web." Within a year, web traffic grew from negligible to a significant fraction of internet traffic.
Netscape and the Commercial Web (1994)
Andreessen left NCSA and co-founded Mosaic Communications, renamed Netscape Communications. Netscape Navigator launched in October 1994. Netscape's IPO (August 9, 1995) — the company had been public for only 16 months — triggered the dot-com investment boom. A company that had never earned a profit was valued at $2.9 billion on its first day of trading.
Netscape introduced: - SSL/HTTPS: Encrypted web communication, enabling e-commerce - JavaScript: Client-side scripting (originally called LiveScript, renamed for marketing) - Cookies: Session state across stateless HTTP connections
The "browser wars" between Netscape and Microsoft's Internet Explorer (IE, bundled free with Windows 95 OSR2 in 1996) eventually destroyed Netscape's business. Microsoft's antitrust trial (US v. Microsoft, 1998–2001) centered largely on IE bundling.
The Dot-Com Era, Crash, and Rebuilding (1995–2003)
The dot-com boom (1995–2000) was characterized by: - Extreme valuations for companies with no earnings - Investment in internet infrastructure: fiber, data centers, networking equipment - Creation of foundational internet companies: Amazon (1994), eBay (1995), Yahoo (1995), Google (1998), Craigslist (1995)
The NASDAQ peaked at 5,048 on March 10, 2000, then crashed to 1,114 by October 2002 — a 78% decline. Trillions of dollars of paper wealth evaporated. But the physical infrastructure (fiber, data centers, networking equipment) survived the crash at reduced prices, creating the foundation for the next wave.
Google (1998): Larry Page and Sergey Brin's PageRank algorithm — ranking web pages by the number and quality of other pages linking to them — solved the web search problem that Yahoo's human-curated directory approach couldn't scale to. Google's technical contribution (large-scale distributed systems, custom hardware, MapReduce) laid the foundation for cloud computing.
Broadband and the Always-On Internet (2000s)
Dial-up internet (28.8 Kbps to 56 Kbps, 1990s) required dialing a phone number; the connection lasted minutes, and you were billed per minute or per hour. The internet was not "on" — you connected to do a task and disconnected.
Broadband (cable modems via DOCSIS, DSL via ADSL, starting late 1990s–early 2000s) provided: - 1–10 Mbps speeds (vs. 56 Kbps dialup): 20–200x faster - "Always-on" — permanently connected - Flat-rate pricing — no per-minute charges
Always-on internet fundamentally changed the internet's role: from a place you visited to infrastructure you depended on constantly. Email notifications, instant messaging, automatic software updates, streaming media — all required always-on connectivity.
Networking Timeline: ARPANET to Internet
1964 Paul Baran -- "On Distributed Communications" -- packet switching theory
|
1969 ARPANET -- first 4-node packet network; first message: "LO"
|
1971 Ray Tomlinson -- first email; '@' symbol for user@host
|
1972 ARPANET public demo -- 900 attendees, 37 nodes
|
1973 Ethernet -- Metcalfe and Boggs at Xerox PARC
| Cerf & Kahn begin TCP design
|
1974 "A Protocol for Packet Network Intercommunication" -- TCP/IP paper
|
1978 TCP/IP split into TCP and IP -- Jon Postel coordinates
|
1980 DIX Ethernet "Blue Book" -- 10 Mbps standard
|
1983 Jan 1: ARPANET NCP->TCP/IP "flag day" -- internet's birthday
| DNS designed -- Paul Mockapetris (RFC 882, 883)
| 4.2BSD -- free TCP/IP stack; BSD sockets API
|
1985 NSFNET -- NSF funds academic internet backbone, 56 Kbps
|
1988 Morris Worm -- first major internet security incident
|
1989 Tim Berners-Lee proposes WWW at CERN
|
1991 WWW goes live -- first website at info.cern.ch
| NSFNET upgraded to T3 (45 Mbps)
|
1993 Mosaic browser -- graphical web browsing, NCSA
| NSFNET opens to commercial traffic
|
1994 Netscape Navigator -- commercial web begins
| Amazon.com founded
|
1995 NSFNET decommissioned -- fully commercial internet backbone
| Netscape IPO -- dot-com boom ignites
| Windows 95 + IE -- mass consumer internet access
| SSL/HTTPS -- secure web commerce possible
|
1998 Google founded -- search dominance begins
|
2000 NASDAQ peaks then crashes -- dot-com bust
|
2003 Broadband penetration > dialup in US -- always-on internet
|
2004 Facebook (limited) -- social networking era
|
2007 iPhone -- mobile internet; smartphone era
|
2010s Cloud computing dominates; AWS/Azure/GCP
|
2020 5G deployment begins; 5 billion internet users
Production Relevance
These historical decisions are engineering constraints you work with today:
-
The Internet Protocol version 4 (IPv4) — 32-bit addresses, announced exhaustion in 2011 — is what most devices still use. IPv6's 128-bit addresses were standardized in 1998 but IPv6 adoption remains at ~40% as of 2024. The internet is running on an addressing scheme designed in 1981 for a research network with 100 nodes.
-
DNS security is an ongoing problem. DNSSEC (DNS Security Extensions) adds cryptographic authentication but adoption is slow. DNS amplification attacks, cache poisoning, and DNS hijacking are current threat vectors.
-
BGP (Border Gateway Protocol) routes packets between autonomous systems (ISPs, cloud providers, enterprises). BGP was designed in 1989 on the back of three cocktail napkins; it has no built-in authentication, and BGP hijacking (accidentally or maliciously announcing wrong routes) causes outages regularly.
-
The end-to-end principle — complexity at endpoints, simplicity in the core — is why the internet has been able to support applications (HTTP, FTP, then BitTorrent, then video streaming, then WebRTC) that nobody imagined when the protocols were designed.
-
HTTP is the universal application protocol. APIs, microservices, webhooks, REST, GraphQL — all use HTTP as transport. The protocol designed in 1991 for HTML documents now carries real-time video, financial transactions, and IoT telemetry.
Key Figures
| Person | Contribution |
|---|---|
| Paul Baran | Packet switching theory, distributed networks |
| J.C.R. Licklider | "Intergalactic Network" vision; ARPA computing program director |
| Lawrence Roberts | ARPANET technical director; made it happen |
| Vinton Cerf | TCP/IP co-designer; "Father of the Internet" |
| Robert Kahn | TCP/IP co-designer; "Father of the Internet" |
| Jon Postel | TCP/IP split, RFC editor, DNS, NCP-to-TCP/IP flag day coordinator |
| Robert Metcalfe | Ethernet invention; 3Com founder |
| Paul Mockapetris | DNS design |
| Tim Berners-Lee | World Wide Web (HTTP, HTML, URL) |
| Marc Andreessen | Mosaic browser, Netscape |
| Ray Tomlinson | Email and the @ convention |
Lessons Learned
-
Decentralization makes systems resilient but hard to secure. The internet's distributed architecture survives node failures but has no central authority to enforce security. This is why spam, DDoS, BGP hijacking, and DNS attacks are endemic.
-
Open, unencumbered standards spread fastest. CERN's decision to give away the web's technology without patents enabled universal adoption within 2 years. Contrast with Gopher (the competing protocol that Gopher's creators tried to monetize in 1993) — it essentially died within a year.
-
Infrastructure investment outlasts the companies that funded it. The fiber and data centers built during the dot-com boom were auctioned at cents on the dollar in 2001–2002. Amazon, Google, and others bought this infrastructure cheaply and built cloud computing on it.
-
End-to-end design enables unanticipated applications. The internet's designers did not anticipate video streaming, cryptocurrency, or real-time gaming. The end-to-end principle's refusal to hard-code assumptions about applications into the network is what made these possible.
-
Email and the Web are accidents of scale. Neither was planned as a primary internet application. Email emerged from researchers wanting to leave messages; the web emerged from a physicist wanting to share documents. The most important internet applications were side effects of other goals.
Exercises
- Write a minimal HTTP/1.0 server in Python using the BSD socket API. Serve a static HTML file. Do not use
http.server— use rawsocketcalls to understand the protocol. - Trace a DNS resolution for a domain you choose: use
dig +trace example.comand identify each server in the chain. How many hops are there? - Implement a simplified version of CSMA/CD in Python simulation: have N stations with random transmission probabilities, simulate collisions and backoff, measure throughput as N increases.
- Calculate: ARPANET in 1969 ran at 50 Kbps. A 400 GbE link today runs at 400 Gbps. What is the ratio? How many 1969 ARPANETs fit in one modern data center link?
- Research the 2010 Pakistan Telecom BGP hijack of YouTube: what happened, how did it happen, and what would RPKI (Resource Public Key Infrastructure) have done to prevent it?
References
- Cerf, V. and Kahn, R. (1974). "A Protocol for Packet Network Intercommunication." IEEE Transactions on Communications.
- Baran, P. (1964). "On Distributed Communications Networks." IEEE Transactions on Communications.
- Hafner, K. and Lyon, M. (1996). Where Wizards Stay Up Late: The Origins of the Internet. Simon & Schuster.
- Leiner, B.M. et al. (2009). "A Brief History of the Internet." ACM SIGCOMM Computer Communication Review.
- Berners-Lee, T. (1999). Weaving the Web. HarperCollins.
- Metcalfe, R.M. and Boggs, D.R. (1976). "Ethernet: Distributed Packet Switching for Local Computer Networks." Communications of the ACM.
- Mockapetris, P. (1983). RFC 882: "Domain Names — Concepts and Facilities."
- Abbate, J. (1999). Inventing the Internet. MIT Press.
- Comer, D.E. (2014). Internetworking with TCP/IP, Volume I. Prentice Hall.