07 — Supply Chain Security

Technical Overview

Software supply chain security addresses the integrity of every step from source code to deployed artifact: source repositories, build systems, CI/CD pipelines, registries, and distribution mechanisms. The fundamental threat is that an attacker who cannot break your production security perimeter can instead compromise something earlier in the pipeline—a dependency, the build system, or a repository—and have their malicious code delivered as a "trusted" signed artifact.

The 2020 SolarWinds attack demonstrated that supply chain compromise can affect every organization that trusts the vendor, regardless of their own security posture. The 2024 XZ Utils backdoor showed that a sophisticated attacker will spend years building trust before inserting a backdoor. These incidents elevated supply chain security from a theoretical concern to a board-level risk.

Prerequisites

Software build systems and package managers.
Digital signatures and PKI.
CI/CD pipeline architecture.
Container image layers and registries.
Git repository structure.

Core Content

Software Supply Chain Attack Taxonomy

Source Code          Build System         Distribution        Runtime
────────────         ────────────         ────────────        ────────
Malicious            Build system         Registry           Package
maintainer ──────────► compromise ─────────► poisoning ───────► confusion
    │                    │                    │
Dependency           CI environment       Typosquatting
confusion            variable inject      (similar names)
    │                    │
Typosquatting        Compiler
    │                backdoor
Social               (Ken Thompson
engineering          attack)
maintainer

SolarWinds Attack (2020): Build System Compromise

Timeline: October 2019 – December 2020. The attackers (attributed to SVR, Russia's foreign intelligence service) spent months doing reconnaissance before inserting malicious code.

Mechanism:

Initial access to SolarWinds' build environment (believed via compromised build server or developer credentials). October 2019: test injections to verify control.
Malware injection: The attackers modified the source code of SolarWinds.Orion.Core.BusinessLayer.dll to add the SUNBURST backdoor. Critically, the backdoor was compiled as part of the normal build process, resulting in a legitimately signed binary.
Code obfuscation: The malicious code mimicked legitimate Orion code patterns, used a dormant period of 12–14 days before activating, and communicated via subdomain generation algorithms (DGA) to avoid static IOC detection.
Supply chain delivery: The malicious DLL was included in the Orion software update (versions 2019.4 through 2020.2.1), downloaded and installed by ~18,000 organizations.
Signed artifact: SolarWinds signed the update with their code-signing certificate. Standard certificate verification confirmed the update was from SolarWinds—because it was. The signing process was not compromised; the code being signed was.

Normal:  Source → Build System → Sign with cert → Distribute → Install
                                        ↑ customers trust this signature

SolarWinds:
  Source [+ SUNBURST injection] → Build System → Sign → Distribute → Install
  ↑ attacked here                                ↑ signature is valid

Key lesson: code signing ensures the artifact came from the signer; it does not verify the integrity of the build process.

Affected: US Treasury, US CISA, Microsoft, FireEye, ~18,000 other organizations.

One of the most sophisticated supply chain attacks discovered. A 2-year social engineering campaign infiltrated the XZ Utils project (a widely-used compression library).

Timeline: - January 2022: A new contributor, "Jia Tan" (JiaT75), begins contributing high-quality patches to XZ Utils. - 2022–2023: Jia Tan builds reputation, eventually gains commit access. Other fake personas pressure the existing maintainer (Lasse Collin) to hand over more responsibility, citing burnout and the need for help. - February 2024: Malicious backdoor code is merged into XZ Utils 5.6.0. - February 29, 2024: Backdoor discovered by Microsoft engineer Andres Freund while investigating anomalous sshd CPU usage. - March 29, 2024: Public disclosure; CVE-2024-3094 issued.

Backdoor mechanism: The malicious code was in a binary .m4 test file (appearing as test data), not in the C source code. The build system cmake / configure.ac extracted and executed the backdoor during the build process. The backdoor: 1. Hooked into the glibc RSA key verification function via IFUNC mechanism. 2. Intercepted sshd authentication before RSA verification completed. 3. Allowed a specific actor holding a secret key to bypass SSH authentication entirely and gain remote code execution on the server.

The backdoor was in liblzma, which systemd links on many Linux distributions (for journal compression). sshd on systemd-enabled systems links systemd, which links liblzma—giving the backdoor access to the sshd process.

sshd ──links─► libsystemd ──links─► liblzma (infected)
                                         │
                              IFUNC hook intercepts RSA
                              key verification in glibc

The attacker chose this indirect path deliberately to obscure the connection between XZ Utils and sshd exploitation.

Affected: Only reached Fedora Rawhide, Fedora 40 beta, Debian testing/unstable, Kali, OpenSUSE Tumbleweed. Stable distributions (Debian stable, Ubuntu LTS, RHEL) were not affected.

Dependency Confusion Attack

Alex Birsan discovered this class in 2021: package managers (npm, pip, RubyGems) may prefer a public registry package over a private internal one if the public package has a higher version number.

Internal package: company.utils v1.2.0 (from internal npm registry)
Attacker creates: company.utils v9.0.0 (on public npm registry)

npm install company.utils
→ npm checks public registry first
→ v9.0.0 > v1.2.0 → installs v9.0.0 (malicious)

Birsan tested this against Apple, Microsoft, PayPal, Netflix, Uber and others—many internal packages were executed on developer machines and CI systems.

Mitigations: - Use npm scoped packages (@company/utils) — scoped packages cannot be registered on public registry without the scope owner's permission. - Use Artifactory/Nexus with explicit registry pinning (publishConfig.registry). - Pin dependencies by exact version AND hash (not semver ranges). - Use npm install --ignore-scripts to block postinstall hooks.

Typosquatting

Registering packages with names similar to popular packages: - request (legitimate) vs. requets (typosquatted) - lodash vs. l0dash, 1odash - cryptography vs. crpytography

Detected and removed by package registry teams, but the window between upload and detection is hours to days—enough to compromise developer machines.

SLSA Framework (Supply-chain Levels for Software Artifacts)

SLSA (Google, 2021, under OpenSSF) is a graduated framework for build integrity. Four levels:

Level	Requirement
SLSA 1	Build process documented, provenance generated (unsigned)
SLSA 2	Signed provenance from a hosted build system (GitHub Actions, etc.)
SLSA 3	Hardened build system: no external access during build, sources verified
SLSA 4	Hermetic, reproducible builds; two-person review of all changes

Provenance: a signed attestation of what was built, when, how, and from what source:

{
  "buildType": "https://github.com/actions/runner/...",
  "builder": {"id": "https://github.com/actions/runner"},
  "invocation": {
    "configSource": {
      "uri": "git+https://github.com/myorg/myrepo@refs/heads/main",
      "digest": {"sha1": "abc123..."}
    }
  },
  "materials": [
    {"uri": "pkg:npm/express@4.18.2", "digest": {"sha256": "def456..."}}
  ]
}

The provenance is signed by the build system's key (not the developer's). A consumer can verify: 1. The artifact hash matches the provenance. 2. The provenance is signed by a trusted build system. 3. The source commit is the one that triggered the build. 4. No external network access occurred during build (SLSA 3+).

SLSA tooling:

# Generate SLSA provenance with GitHub Actions
# .github/workflows/release.yml
- uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.10.0

# Verify provenance
slsa-verifier verify-artifact artifact.tar.gz \
  --provenance-path provenance.json \
  --source-uri github.com/myorg/myrepo

SBOM (Software Bill of Materials)

An SBOM is a machine-readable inventory of all software components in a product, analogous to an ingredient list. When Log4Shell (CVE-2021-44228) hit, organizations that had SBOMs knew within hours which products were affected; those without SBOMs spent weeks auditing.

Formats: - SPDX (Software Package Data Exchange, Linux Foundation, ISO standard 5962:2021). - CycloneDX (OWASP, widely used in container/cloud context).

Generate SBOM:

# Python project
pip install cyclonedx-bom
cyclonedx-py poetry > sbom.json

# Container image
syft scan myimage:latest -o spdx-json > sbom.spdx.json

# Node.js project
npx @cyclonedx/cyclonedx-npm --output-format JSON > sbom.json

# Go binary (using govulncheck for vuln scanning against SBOM)
govulncheck ./...

Scan SBOM for vulnerabilities:

# Grype: scan SBOM or image for known CVEs
grype sbom:./sbom.spdx.json

# OSV Scanner (Google)
osv-scanner --sbom sbom.spdx.json

Sigstore: Keyless Signing

Sigstore (OpenSSF, 2021) solves the key management problem in supply chain security. Traditional artifact signing requires every signer to manage a long-lived private key—which can be lost, stolen, or forgotten.

Cosign (Sigstore): sign container images and artifacts using an identity from an OIDC provider (GitHub Actions, Google, Microsoft) rather than a stored key.

# Sign a container image (keyless — uses GitHub OIDC token in CI)
cosign sign ghcr.io/myorg/myimage:latest

# What happens:
# 1. cosign requests an OIDC token from GitHub Actions
# 2. Creates a short-lived key pair
# 3. Sends CSR to Fulcio (Sigstore's certificate authority) 
# 4. Fulcio issues a cert binding the GitHub identity to the key
# 5. Signs the image with the key
# 6. Appends signature + cert to Rekor (Sigstore's transparency log)
# 7. Short-lived key is discarded

# Verify the image
cosign verify ghcr.io/myorg/myimage:latest \
  --certificate-identity-regexp="https://github.com/myorg/.*" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com"

Rekor: an immutable, append-only transparency log for all Sigstore signatures. Like Certificate Transparency for TLS, it enables auditing who signed what and when.

Container Image Signing (Notary v2 / Notation)

Notary v2 (CNCF, OCI standard) provides a supply chain framework for signing OCI artifacts:

# Sign with Notation (Notary v2 CLI)
notation sign myregistry.io/myimage:latest \
  --key my-signing-key \
  --plugin com.amazonaws.signer.notation.plugin

# Verify
notation verify myregistry.io/myimage:latest

# Policy enforcement (Kubernetes admission controller — Ratify)
# Only allow images signed by specific identities

Dependency Pinning and Auditing

Version pinning: specify exact versions, not semver ranges.

// Bad: "express": "^4.18.0"  (any 4.x.y)
// Good: "express": "4.18.2"  (exact)

// Better: lockfile + integrity hashes
// package-lock.json has sha512 hashes per package

// Even better: reproducible builds with exact hash verification

npm audit / pip audit / trivy / grype:

# npm
npm audit --production
npm audit fix --dry-run

# Python
pip-audit -r requirements.txt

# Container image
trivy image nginx:latest --severity HIGH,CRITICAL

# Binary / filesystem
grype dir:. --fail-on medium

# Go
govulncheck ./...

Private registry + allowlisting (Artifactory, Nexus):

# .npmrc: point to internal registry for scoped packages
@company:registry=https://artifactory.mycompany.com/npm/
//artifactory.mycompany.com/npm/:_authToken=${NPM_TOKEN}

# Block public registry fallthrough by setting registry globally to internal proxy
registry=https://artifactory.mycompany.com/npm-proxy/

Reproducible Builds

If two independent parties build the same source code and get the same binary byte-for-byte, the build is reproducible. This allows independent verification that a distributed binary corresponds to the published source.

Challenges: embedded timestamps, non-deterministic linker output, environment-captured metadata.

# Debian reproducible builds: check if a package is reproducible
diffoscope file1.deb file2.deb  # diff two builds at binary level

# Nix: fully reproducible package builds
nix build github:nixos/nixpkgs/nixos-unstable#hello

# Rust: reproducible builds with --remap-path-prefix
rustc --remap-path-prefix /home/user=/work ...

# SOURCE_DATE_EPOCH: standardize timestamp in builds
export SOURCE_DATE_EPOCH=1577836800  # 2020-01-01

The Reproducible Builds project (https://reproducible-builds.org) documents the work across Debian, Arch Linux, Fedora, FreeBSD, and other distributions.

Historical Context

Supply chain attacks predate the modern software ecosystem. Ken Thompson's "Reflections on Trusting Trust" (ACM TURING AWARD lecture, 1984) described the ultimate supply chain attack: a compiler backdoor that inserts a Trojan Horse into login and hides itself in the compiler—even rebuilding from source code doesn't remove it because the malicious compiler re-inserts the backdoor when compiling the login program or the compiler itself.

The modern supply chain security movement accelerated after: - 2015 npm left-pad incident: a developer removed a small npm package, breaking thousands of builds. Demonstrated ecosystem fragility. - 2016 Kik/npm dispute: naming conflict highlighted registry governance gaps. - 2017 Petya/NotPetya: malware distributed via a Ukrainian accounting software update. Shared characteristics with SolarWinds. - 2020 SolarWinds: legitimized supply chain as a board-level security concern. - 2021 Log4Shell (CVE-2021-44228): a transitive dependency (Log4j) with a critical RCE in millions of Java applications. Highlighted the SBOM gap. - 2024 XZ Utils: showed that well-resourced adversaries will invest 2+ years in social engineering.

Production Examples

Case: GitHub Actions Cache Poisoning (2023). Security researcher demonstrated that GitHub Actions workflow caches are identified by key strings, not content hashes. An attacker who could trigger a build with a specific branch name could poison a cache key used by a privileged workflow. The cached artifact would include malicious code executed in subsequent builds. Fix: GitHub introduced immutable caches; security practitioners should avoid using untrusted forks' caches in privileged workflows.

Case: Log4Shell SBOM value. After CVE-2021-44228 was disclosed on December 9, 2021, organizations with SBOMs (in Syft, Snyk, or JFrog Xray format) searched their catalog and identified affected services within hours. Organizations without SBOMs spent weeks manually auditing Java applications. The contrast was stark enough that NTIA mandated SBOMs for software sold to the US federal government (May 2021 executive order 14028) specifically because of Log4Shell.

Debugging Notes

# Identify what signed a container image
cosign verify-attestation --type slsaprovenance gcr.io/someimage:latest

# Inspect SBOM
syft scan myimage:latest -o table | grep log4j

# Find all log4j versions in a Java project
find . -name "*.jar" -exec unzip -p {} META-INF/MANIFEST.MF \; 2>/dev/null | grep -i "log4"
# Also search nested JARs (fat jars):
find . -name "*.jar" | xargs -I{} sh -c 'jar tf {} 2>/dev/null | grep -i log4j && echo {}'

# Verify git commit signatures
git log --show-signature
git verify-commit <sha>

# Check npm package integrity
npm install --audit
cat package-lock.json | jq '.packages | to_entries[] | select(.value.integrity != null) | {(.key): .value.integrity}'

Security Implications

The provenance gap: code signing (GnuPG, Authenticode, cosign) verifies who signed the artifact. SLSA provenance verifies how it was built. Both are needed. SolarWinds had code signing but not build provenance.

Transitive dependencies: the average npm package has 80+ transitive dependencies. You are trusting the security practices of dozens of maintainers for each package. Minimize dependency surface, prefer packages with active maintenance and security practices, and scan continuously.

Build system privilege: build systems often have access to production credentials (deployment keys, signing keys). A compromised CI system is equivalent to a compromised production environment. Separate build-time and deploy-time credentials; use OIDC token-based authentication.

Insider threat: the XZ Utils backdoor shows that a "trusted" maintainer can be a long-term adversary. Code review, required signed commits, and independent build verification reduce (but cannot eliminate) this risk.

Performance Implications

SBOM generation adds 10–60 seconds to a build depending on the scanner and project size. SLSA provenance generation is negligible (a few seconds for signature operations). Container image scanning with Trivy/Grype adds 30–120 seconds for large images.

These overheads are acceptable in CI/CD pipelines where total build time is minutes. For extremely fast (< 30 second) CI pipelines, consider asynchronous scanning (scan in parallel, block deployment only on high/critical findings).

Failure Modes and Real Incidents

XZ Utils near-miss: the XZ backdoor was discovered because Andres Freund noticed that SSH login on his Debian testing system took 500ms longer than expected. He investigated the anomalous CPU usage in sshd, traced it to liblzma, and reverse-engineered the backdoor. If he had ignored the performance anomaly—or if the backdoor had been less eager with its scanning—it might have reached stable distributions and affected hundreds of millions of systems.

Left-pad removal (2016): A developer unpublished the left-pad npm package (11 lines of code) after a naming dispute. Thousands of projects using left-pad directly or transitively broke immediately. npm subsequently implemented an unpublish window policy (24 hours) and permanent retention for packages with significant download counts.

event-stream backdoor (2018): The event-stream npm package (2 million weekly downloads) was handed to a new maintainer who injected code targeting the Copay Bitcoin wallet application. The malicious code was hidden in a transitive dependency and only activated when running inside a Copay build context. It was present in the npm registry for 2.5 months before discovery.

Modern Usage

The 2021 US Executive Order on Improving the Nation's Cybersecurity mandated SBOMs for software sold to the federal government. CISA (Cybersecurity and Infrastructure Security Agency) published SBOM guidance making it effectively mandatory for federal contractors.

The OpenSSF (Open Source Security Foundation, Linux Foundation) coordinates: - Sigstore: keyless signing infrastructure. - SLSA: provenance framework. - OpenSSF Scorecard: automated security health scores for open-source projects. - Alpha-Omega: funding for security audits of critical open-source packages.

GitHub Actions now generates OIDC tokens for workloads, enabling keyless signing via Sigstore in CI/CD pipelines—eliminating the need for stored signing keys in most cases.

Future Directions

Reproducible builds at scale: Debian's Reproducible Builds project aims for 100% reproducibility. Nix and Guix are already fully reproducible. As reproducibility becomes standard, independent binary verification becomes practical.
TEE-based build attestation: build processes running inside SGX enclaves or Confidential VMs can produce hardware-attested provenance—impossible to fake even by the build infrastructure operator.
SBOM automation in registries: container registries (GitHub Container Registry, Docker Hub) are beginning to automatically attach SBOMs as OCI artifacts, making SBOM-aware deployment policies practical.
Post-quantum signatures: Sigstore and notary are exploring migration from ECDSA to post-quantum signatures (CRYSTALS-Dilithium) for long-term artifact signing.

Exercises

Generate an SBOM for a container image of your choice using syft. Identify all Java libraries. Scan the SBOM with grype for known CVEs. Find the 3 highest-severity findings and explain what each CVE would allow an attacker to do.
Set up a GitHub Actions workflow that uses slsa-github-generator to produce SLSA Level 3 provenance for a released artifact. Verify the provenance using slsa-verifier. Explain what the provenance proves about the build.
Sign a container image with cosign using keyless signing (OIDC). Verify the signature on a different machine. Examine the Rekor log entry for your signature. What information is public in the transparency log?
Simulate a dependency confusion attack: create a private npm registry (using Verdaccio), publish a package named @company/utils. Then publish a higher-version package with the same name on a public test registry and observe which version npm install selects. Implement the mitigation.
Read the XZ Utils backdoor analysis (Freund's openwall post, March 2024). Write a technical explanation of the IFUNC hooking mechanism used. Explain how the build system extracted the backdoor from a binary test file.

References

Thompson, K. "Reflections on Trusting Trust." ACM, 1984. https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf
SolarWinds attack analysis: https://www.mandiant.com/resources/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor
XZ Utils backdoor: Freund, A. "backdoor in upstream xz/liblzma leading to ssh server compromise." oss-security, March 2024.
XZ Utils CVE-2024-3094: https://nvd.nist.gov/vuln/detail/CVE-2024-3094
SLSA: https://slsa.dev/
Sigstore/Cosign: https://docs.sigstore.dev/
SBOM Minimum Elements: https://www.ntia.gov/report/2021/minimum-elements-software-bill-materials
Birsan, A. "Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies." Medium, 2021.
OpenSSF Scorecard: https://github.com/ossf/scorecard
Reproducible Builds: https://reproducible-builds.org/
Log4Shell (CVE-2021-44228): https://nvd.nist.gov/vuln/detail/CVE-2021-44228