Skip to content
pwnsy
threat-intelintermediate#solarwinds#supply-chain#incident-analysis#sunburst#case-study

SolarWinds and SUNBURST: When the Build System Is the Target

How the SolarWinds Orion compromise worked: SUNSPOT hijacking MSBuild to inject SUNBURST into a signed release, the DGA beacon, forged SAML tokens, and why code signing did not help.

Software supply chain attacks usually mean one of two things: a dependency you pulled in was malicious, or a vendor's source code was tampered with. SolarWinds was neither.

The source repository was clean. Developers reviewed clean code, committed clean code, and the code that went into version control stayed clean. The substitution happened during compilation, on the build server, in the window between the last time anyone looked at the source and the moment the compiler produced a binary. The output was then signed by SolarWinds and distributed through the official update channel, because from every system's point of view it was a legitimate release.

That is what makes this the reference case. It attacks the one step in the pipeline that nobody was watching, and it defeats code signing by getting in front of it.

Facts here come from Mandiant's original analysis, CrowdStrike's technical report on the build-time implant, and CISA's advisories and emergency directive.

Timeline

DateEvent
September 2019Attackers access the SolarWinds environment, per the company's later account
October 2019A test modification is inserted into an Orion build, apparently to prove the technique without a payload
February 2020SUNBURST is injected into Orion builds
March to June 2020Affected versions 2019.4 HF5 through 2020.2.1 HF1 ship to customers
June 2020The attackers remove the implant from the build environment
8 December 2020FireEye discloses a compromise of its own environment
13 December 2020The Orion supply chain compromise is disclosed. CISA publishes Emergency Directive 21-01
January 2021CrowdStrike publishes analysis of SUNSPOT, the build-time tool
15 April 2021The US government attributes the activity to Russia's Foreign Intelligence Service
May 2021Executive Order 14028 lands, with software supply chain requirements including SBOM

Note the shape: nine months from initial access to shipped backdoor, then six months of distribution, then discovery by a victim rather than by the vendor or by any scanner.

SUNSPOT: the build-time substitution

The implant that did the work never shipped to a customer. It lived on a SolarWinds build server.

CrowdStrike's analysis describes it monitoring running processes for MsBuild.exe. When it found one, it inspected the command line to determine whether the Orion product was being compiled. If so, it replaced a specific source file with a version containing the backdoor, let the build proceed, and restored the original file afterwards.

The consequences of doing it this way are worth spelling out.

Version control stays clean. Anyone auditing the repository, reviewing commits, or diffing releases against source sees nothing. There is no malicious commit, no suspicious author, no unusual pull request.

The signature is genuine. The build system signs its output as it always does. The certificate is valid, the signature verifies, and every downstream check passes. Code Signing Explained covers what a signature actually asserts, and this is the precise limit of it: a signature attests that a build came from a vendor unaltered since signing. It cannot attest that the vendor's build was clean.

Update channels distribute it. Customers with good patch hygiene, applying vendor updates promptly, were the ones who received it. Diligence was the delivery mechanism.

The defensive answer to this class is reproducible builds: making compilation deterministic so that anyone can rebuild the source and confirm the binary matches, bit for bit. If the shipped artefact does not match a rebuild from reviewed source, something happened in between. That is the only control that would have caught SUNSPOT directly, and it remains rare.

SUNBURST: what shipped

The backdoor lived inside SolarWinds.Orion.Core.BusinessLayer.dll, a legitimate component of the product.

Its tradecraft was aimed squarely at avoiding detection and analysis:

Dormancy. After installation it waited up to two weeks before doing anything. Sandbox analysis runs for minutes, and correlation between an installation event and network activity two weeks later is a link almost no one draws.

Environment checks. It looked for security tooling, analysis processes and specific domains, and stayed inert where it found them. Some environments were excluded from activity entirely.

Blending in. Orion is network monitoring software. It is expected to talk to many devices and to make outbound connections. A backdoor in that product has cover for behaviour that would be obviously wrong in a calculator.

Domain generation for the first hop. It resolved subdomains of avsvmcloud.com, encoding information about the victim in the subdomain. The DNS response returned a CNAME pointing to the actual command and control domain, so the operators could select targets by choosing which victims got a real answer. DNS Tunneling Explained covers the general technique, and Command and Control Explained covers the pattern.

That selection step is the operational heart of it. Roughly 18,000 organisations got the backdoor. The operators chose a small fraction, widely reported as around a hundred organisations and several federal agencies, for follow-on activity. Everyone else had a dormant implant and no intrusion.

Follow-on: forged authentication

In the victims chosen for hands-on activity, the second-stage tooling included memory-resident loaders and commodity post-exploitation frameworks. The part with the most lasting lesson is what happened to identity.

Attackers with sufficient access stole the signing certificate used by an on-premises federation service and used it to mint SAML tokens for arbitrary users, including for cloud services and email. The technique is commonly called Golden SAML.

A forged token is not a stolen password. There is no authentication event to detect, no failed login, no multi-factor prompt to approve or deny, and no credential to rotate. The identity provider receives a correctly signed assertion and honours it, because that is its job. Rotating passwords does nothing. The only remediation is to treat the signing key as compromised and replace it, then invalidate everything issued under it.

The general point for anyone running federated identity: the token signing key is a tier zero secret, in the sense described in Privileged Access Management Explained. Anyone who obtains it can be anyone. It belongs with the strictest possible controls, ideally in hardware, and its use belongs under monitoring.

Why it took a year to find

Discovery came from a victim. FireEye investigated its own compromise, found the path led back to Orion, and published.

Nothing else was positioned to notice:

  • The vendor's own reviews looked at source, which was clean.
  • Signature verification passed, because the signature was real.
  • Endpoint tooling saw a signed component of a trusted product doing things that product does.
  • Network monitoring saw DNS lookups to a domain with no reputation problem, from a host that legitimately queries a great deal.
  • The two-week dormancy broke the temporal link between installation and activity.

This is what an intelligence operation looks like when it is executed patiently, and it is a reasonable argument for detection strategies that do not depend on knowing what is bad in advance. Threat Hunting Guide covers looking for anomalies rather than signatures.

What changed

Executive Order 14028, in May 2021, put software supply chain security into US federal procurement, including requirements around software bills of materials. An SBOM would not have caught SUNSPOT, because the backdoor was inside a first-party component rather than a declared dependency, and the broader push toward provenance and attestation traces to this incident.

Build system security became a named discipline. The idea that a build server is production infrastructure requiring tier zero treatment is now conventional, and it was not before.

Frameworks for supply chain integrity such as SLSA formalised the levels of assurance a build process can provide, with reproducibility and provenance attestation at the top.

Regulatory attention on disclosure. The US Securities and Exchange Commission subsequently brought a civil enforcement action against SolarWinds concerning its security disclosures, and much of it was dismissed on the defendants' motion. The episode remains an open question about what companies must say about their security posture rather than a settled precedent.

What generalises

  1. Signed does not mean safe. A signature binds an artefact to a vendor and asserts nothing about the vendor's pipeline.
  2. The build system is production. It has credentials to everything, it produces the artefacts your customers trust, and it is usually less monitored than the systems it deploys to.
  3. The gap between reviewed source and shipped binary is unwatched almost everywhere. Reproducible builds close it. Almost nobody has them.
  4. Federation signing keys deserve hardware protection, because their compromise cannot be remediated by any credential rotation.
  5. Mass distribution with selective exploitation is the shape of this class. Presence of the implant is not evidence of intrusion, and the triage question after any such event is who was chosen rather than who received it.

Sources & further reading