Skip to content
pwnsy
malwareintermediate#rootkit#malware#persistence#kernel-security#malware-analysis

What Is a Rootkit? Kernel and User-Mode Stealth Explained

A rootkit hides other malware by tampering with the operating system itself. How kernel and user-mode rootkits work, the hiding tricks, and how to detect them.

Most malware tries to do something visible: encrypt files, drain a wallet, beacon to a server. A rootkit has a narrower and more unsettling job. It changes the operating system so that you cannot see the other malware at all. Files that exist do not appear in a directory listing. Processes that are running do not show in the task manager. Network connections that are open do not show in the connection table. The machine reports that everything is fine, because the thing you are asking has been taught to lie.

The name comes from Unix. "Root" is the all-powerful account, and a "kit" was the bundle of tools an intruder installed to keep root access and stay hidden. The idea carried over to every modern platform. A rootkit today is any code that subverts the operating system's own reporting to conceal a presence on the machine.

What a rootkit is for

A rootkit rarely acts alone. It is the concealment layer for a payload: a backdoor, a keylogger, a cryptominer, a bot. The payload does the work; the rootkit makes sure defenders and users never notice it. That single purpose, staying unseen, is why the behaviour maps to the Defense Evasion tactic and specifically to Rootkit (T1014) on MITRE ATT&CK.

Three things a rootkit typically hides:

  • Files and directories, so the payload on disk does not show up.
  • Processes and services, so the running payload does not appear in process lists.
  • Registry keys, network ports, and connections, so persistence and command channels stay invisible.

The common thread is interception. Every tool you use to inspect a system asks the operating system a question. A rootkit sits between the question and the honest answer, and edits the answer.

User-mode versus kernel-mode

Rootkits are usually sorted by where they run. That location decides how much they can hide and how hard they are to find.

PropertyUser-mode rootkitKernel-mode rootkit
Runs inApplication space (ring 3)OS core (ring 0)
Typical methodHooks or patches shared libraries and API calls per processLoads as a driver, patches kernel structures and system call handling
Privilege to installStandard-to-adminAdministrator, plus a way past driver signing
Scope of hidingThe processes it has infectedThe entire system, including security tools
Stability riskLow; a crash kills one processHigh; a bug can crash the whole machine
Detection difficultyModerateHard

A user-mode rootkit works inside normal applications. It intercepts the library functions a program calls to list files or processes, and filters the results before the program sees them. Because it lives at the same privilege level as the tools hunting it, a scanner running with equal or higher rights can often catch the inconsistency. This is the more common and more survivable kind.

A kernel-mode rootkit runs inside the operating system core, usually loaded as a device driver. From there it can alter the data structures the kernel uses to track processes and files, or intercept system calls at the source. Everything that trusts the kernel, which is every tool on the machine, inherits the lie. A kernel rootkit that removes a process from the kernel's own process list is invisible to any tool that reads that list, no matter how privileged.

Below the OS there are deeper variants. Bootkits infect the boot process so they load before the operating system, and firmware implants live in hardware. Those sit outside the scope here and are covered in What Is a Bootkit.

The hiding techniques

The specific tricks vary by platform, but they cluster into a few families.

API and function hooking. The rootkit redirects a call, for example the function that enumerates directory entries, to its own code. Its code calls the real function, then strips out any entry matching the payload before returning the list. The caller gets a filtered view and cannot tell.

Direct kernel object manipulation. Instead of hooking a call, the rootkit edits the kernel's in-memory bookkeeping directly. Unlinking a process from the doubly linked list the scheduler walks for reporting can hide it from tools while the scheduler still runs it. This is a classic kernel technique because it leaves no hook to find.

System call table and handler patching. The rootkit overwrites entries in the tables the kernel uses to dispatch system calls, or patches the handlers themselves, so requests are quietly rerouted.

Driver and execution-flow abuse. Getting kernel code to run in the first place often means abusing a legitimate signed driver with a known flaw, a technique defenders call "bring your own vulnerable driver." It maps to Hijack Execution Flow and related driver-abuse techniques on ATT&CK.

Interrupt and inline hooking. Some rootkits patch the first few bytes of a target function with a jump to their own code, a technique called inline or trampoline hooking. The original bytes are saved so the rootkit can call through to the real function after filtering. Others go after lower-level dispatch structures such as the interrupt descriptor table or a model-specific register that points at the system-call entry, redirecting the very gate the CPU uses to enter kernel mode. These deeper hooks are powerful because they intercept every request of a given type from a single choke point, but they also disturb structures that integrity checks like Kernel Patch Protection watch closely, which is one reason modern kernel rootkits have drifted toward direct object manipulation instead.

Filter and callback abuse. Modern operating systems expose legitimate extension points: file-system minifilters, registry callbacks, and notification routines that drivers register to observe or alter activity. A rootkit that registers itself through these documented interfaces looks, at a glance, like an ordinary security or storage driver. It can drop or rewrite results for the objects it wants hidden while remaining a well-formed, loadable component. This blurs the line between a rootkit and an aggressive but sanctioned driver, which is part of why detection leans on behaviour and provenance rather than on the mere presence of a hook.

A compromised host cannot audit itself

If a kernel rootkit is active, the process list, file listing, and antivirus running on that machine are all asking a subverted kernel for the truth. Clean results from on-box tools are not evidence of a clean system. Trustworthy answers have to come from outside the running OS.

How rootkits get installed

A rootkit is a post-exploitation tool. Something else has to grant the access first, because installing kernel code requires high privilege. The usual chain:

  1. Initial access through phishing, a malicious download, or an exploited service.
  2. Privilege escalation to administrator or system level.
  3. Rootkit installation, loading the driver or patching the libraries.
  4. Payload concealment, hiding the backdoor or miner the operator actually cares about.

This is why least privilege matters so much. If step two fails, a kernel rootkit never gets the foothold it needs. The delivery file at step one is usually one of a small set of abused formats, the same script, archive, and installer types tracked in our File-Format Abuse Atlas, which is the earliest place to break the chain.

How to detect a rootkit

Detection turns on one principle: compare what the operating system claims against ground truth gathered from somewhere it cannot tamper with.

Cross-view analysis. Ask the same question two ways, a high-level API call and a low-level direct read, and compare. If the kernel's raw process structures list a process that the normal enumeration hides, the gap is the rootkit. Several specialist tools automate this.

Memory forensics. Capture the machine's RAM and analyse it on a separate system. A rootkit can hide from live tools, but it still exists in memory. Frameworks that walk memory structures offline surface the hidden processes, unlinked objects, and suspicious hooks.

Offline and boot-time scanning. Boot the machine from trusted external media, or mount its disk read-only on a clean system, and scan from there. The rootkit is not running, so it cannot filter anything. Modern platforms also offer early-launch anti-malware that starts before third-party drivers, narrowing the window a kernel rootkit has to hide.

Behavioural and network signals. The rootkit can hide the connection locally, but traffic still leaves the machine. Monitoring at the network layer, on a separate device, can reveal beacons the host swears are not there.

How to defend against rootkits

Because installation needs privilege, most of the defense sits before the rootkit ever loads.

  1. Enforce least privilege. Users and services that never run as administrator cannot easily install kernel code. This is the single highest-value control.
  2. Require signed drivers and enable Secure Boot. Driver signature enforcement and a verified boot chain block unsigned kernel code and raise the cost of the vulnerable-driver trick.
  3. Block known vulnerable drivers. Maintain and apply a driver blocklist so the common "bring your own vulnerable driver" candidates cannot be loaded.
  4. Patch and reduce attack surface. The escalation step usually relies on an unpatched flaw. Timely patching removes the ladder to kernel access.
  5. Collect telemetry off the host. Ship endpoint and network logs to a separate system in real time, so a later rootkit cannot rewrite the history of its own arrival.
  6. Keep known-good boot media ready. Have trusted offline scanning media prepared before you need it, so verification does not depend on the suspect machine.
Rebuild, do not clean

For a confirmed kernel-mode rootkit, treat the operating system as untrustworthy. Wipe and reinstall from trusted media rather than attempting to remove the rootkit in place, and rotate every credential that was used on the machine. A partial clean can leave the concealment layer behind.

A worked example: hiding one process

To make the mechanism concrete, follow a single hidden process from the inside. Assume an operator has already gained administrator rights and loaded a kernel driver. Their payload is a small backdoor running as a process, and they want it gone from every view.

The operating system tracks every running process with a structure in kernel memory. Those structures are threaded together in a doubly linked list: each entry holds a pointer forward to the next process and a pointer backward to the previous one. When a tool asks for a process listing, the kernel walks this list from one entry to the next and returns what it finds.

The rootkit's move is to unlink the backdoor's entry from that list. It takes the backdoor's forward and backward pointers, then rewrites its neighbours so they point past it: the previous entry now points forward to the next entry, and the next entry points backward to the previous one. The backdoor's own structure still exists and still holds valid pointers, so nothing crashes. It has simply been spliced out of the chain that enumeration walks.

Here is the important part. The scheduler does not use that same list to decide what to run. It uses a separate set of structures keyed on threads ready to execute. So the unlinked process keeps getting CPU time and keeps running the backdoor, while every tool that walks the process list steps right over the gap and never sees it. The process is running and invisible at the same time, from the same kernel, depending on which structure you ask.

That single example captures the whole rootkit problem. There is no hook to find, no patched function, no altered code page. There is only a set of pointers that no longer agree with reality. Detection has to notice the disagreement, which means finding the process through the scheduler's structures or a raw scan of memory and comparing that against the list the OS is willing to show you.

Detection signals

Beyond the general principle of comparing the OS against outside ground truth, a defender has concrete signals to hunt for. None is proof on its own; together they build a case.

  • Cross-view discrepancies. A process, file, or registry key that appears in a low-level enumeration but is missing from the high-level one is the single strongest indicator. The gap itself is the tell.
  • Unlinked or orphaned kernel objects. Memory analysis that finds process or thread structures not present in the standard linked lists points directly at direct-object-manipulation hiding.
  • Unexpected drivers and modules. A loaded kernel module with no matching file on disk, an unusual signer, a name that mimics a system component, or a load time that clusters with the suspected intrusion window all warrant scrutiny.
  • Hook artifacts. System-call table entries, interrupt vectors, or function prologues that point outside the expected module image suggest a hooking rootkit. Comparing in-memory code against the clean on-disk image of the same module surfaces inline patches.
  • Boot and integrity mismatches. Failed measured-boot attestation, an unexpected change in the boot chain, or Secure Boot being silently off on a machine that should have it on can indicate something loading below the OS.
  • Network without a local owner. A connection or beacon visible from a network sensor or upstream device that no process on the host admits to owning is a classic sign the host is filtering its own connection table.
  • Timing and resource anomalies. Hidden work still consumes CPU, memory, and disk. Accounting that does not add up, where the machine is clearly busier than the visible processes explain, can betray a concealed payload.
Detect from the layer below

The reliable rule for rootkit detection is to observe from a layer the rootkit does not control. A user-mode rootkit is caught by a kernel-level view. A kernel rootkit is caught by memory forensics, an offline disk mount, a hypervisor, or a network sensor. A firmware implant needs attestation rooted in hardware. Each layer can be lied to by what runs beneath it, so trust flows downward, and detection follows the same direction.

Rootkit is a specific behaviour, and it is easy to blur with neighbouring ideas. This table separates them.

ConceptCore behaviourRelationship to a rootkit
RootkitSubverts the OS so a presence cannot be seenThe concealment layer itself
BootkitInfects the boot process to load before the OSA rootkit that persists below the OS; see the bootkit guide
BackdoorProvides ongoing unauthorized accessThe payload a rootkit typically hides
Fileless malwareRuns from memory to avoid leaving filesA hiding strategy; may be used with or instead of a rootkit
TrojanDisguises malicious code as something benignA delivery method that may install a rootkit
Living off the landAbuses built-in trusted toolsEvasion by blending in rather than by lying to the OS

The distinction that matters is purpose. A backdoor grants access, a trojan gets code in the door, and a rootkit makes whatever is already inside invisible. They travel together in real intrusions, but they are different jobs, and conflating them leads to defenses aimed at the wrong stage.

How rootkits evolved

The arc of rootkit history is a running argument between concealment and inspection. Early Unix rootkits were user-mode: they replaced common system binaries, the tools an administrator ran to list files and processes, with tampered copies that omitted the intruder's files. The counter was simple integrity checking, comparing those binaries against known-good hashes, so attackers moved down a layer.

Kernel-mode rootkits followed, hooking system-call tables and later manipulating kernel objects directly to escape the file-based checks. Defenders answered with kernel integrity protection that watches critical structures and with driver signing that raises the bar for getting code into the kernel at all. That pressure pushed the most capable techniques toward abusing legitimately signed but vulnerable drivers, the "bring your own vulnerable driver" pattern, because signing checks a driver's provenance, not its safety.

The deepest branch went below the operating system entirely, into the boot chain and firmware, where an implant loads before any OS defense can watch it. Secure Boot, measured boot, and hardware-rooted attestation are the response there. The through-line across every stage is the same: each new hiding place forces a new observation point one layer lower, and the contest keeps descending toward the hardware.

Common misconceptions

"My antivirus is clean, so there is no rootkit." Antivirus running on the infected machine asks the same subverted OS everyone else does. A kernel rootkit can hide from it as easily as from any other on-box tool. A clean on-host scan is weak evidence.

"A rootkit is a type of virus." A rootkit is defined by what it does, hide a presence, not by how it spreads. It usually does not replicate at all. It is a concealment technique attached to some other payload.

"Rootkits steal data." The rootkit itself typically steals nothing. It hides the component that does. Treating the rootkit as the whole threat leads defenders to miss the backdoor or miner it was protecting.

"Reinstalling the OS always fixes it." For user-mode and most kernel rootkits, a clean rebuild from trusted media does remove them. For a firmware implant or bootkit that lives below the OS, reinstalling the operating system leaves the implant untouched, which is why the deepest cases need firmware reflashing or hardware replacement.

"If it were there, the machine would be slow or unstable." Well-written rootkits are careful to stay quiet. Instability is a sign of a buggy rootkit, not a reliable sign of the presence or absence of one.

Frequently asked questions

What is the difference between a rootkit and a bootkit? A bootkit is a rootkit that infects the boot process so it loads before the operating system starts. Both hide a presence; the bootkit simply does its hiding from a layer below the OS, which makes it harder to detect and to remove. The general term rootkit is often used to cover both.

Can a rootkit infect a machine without administrator rights? A user-mode rootkit can operate with the privileges of the processes it infects, which may be standard-user. A kernel-mode rootkit needs high privilege to load, which is why privilege escalation is a prerequisite and least privilege is such an effective control.

Do rootkits work on Linux and macOS as well as Windows? Yes. The concept is platform-independent because every operating system has structures a rootkit can tamper with and system calls it can intercept. The specific techniques differ, and each platform has its own signing, secure-boot, and integrity defenses, but the strategy of subverting the OS to hide is universal.

How is a rootkit different from fileless malware? Fileless malware avoids leaving files on disk by running from memory. A rootkit actively lies about the system's state. They can be combined, and both aim at evasion, but one hides by leaving little to find while the other hides by tampering with what you can find.

Why can a rootkit be running and invisible at the same time? Because different parts of the kernel use different structures. The scheduler runs a process using one set of data, while enumeration tools read another. A rootkit edits the enumeration structures and leaves the scheduling ones alone, so the process executes and disappears from listings simultaneously, as the worked example above shows.

What is the safest response to a confirmed kernel rootkit? Treat the operating system as untrustworthy. Rebuild the machine from known-good media rather than attempting an in-place removal, and rotate every credential that was used on it. If a firmware or boot-level implant is suspected, escalate to firmware reflashing or hardware replacement, because an OS reinstall will not reach that layer.

Can Secure Boot alone stop rootkits? Secure Boot raises the cost of loading unsigned code early in the boot chain, which blocks a class of bootkits and unsigned kernel rootkits. It is one layer. It does not stop a rootkit that arrives through a signed vulnerable driver or one that operates purely in user mode, so it belongs in a stack with driver blocklists, least privilege, and off-host telemetry.

Rootkits are the quiet backbone of long-term intrusions. They do not make noise; they remove it. Once you accept that a compromised system cannot honestly describe itself, the defensive logic follows: keep privilege scarce so the rootkit cannot install, verify the boot chain so it cannot hide below the OS, and check for it from the outside, where its lies do not reach.

Sources & further reading

Sharetwitterlinkedin

Related guides