SIEM: Log Collection, Correlation and Detection

Every system in an organisation writes down what it did. The domain controller records logons, the firewall records connections, the identity provider records authentications, the EDR agent records process launches, the cloud control plane records API calls. Each of those logs is a partial view, and an attack shows up as a few unremarkable entries scattered across several of them.

A SIEM is the place those views are brought together. It collects the logs, converts them into a common shape, adds context the raw log lacks, runs detection logic across the combined stream, and raises an alert a human can act on. The reason it exists is correlation: one failed login on a domain controller means nothing, and the same failed login preceded by a VPN session from a new country and followed by a service being installed on the host is the shape of an intrusion.

This guide covers the pipeline in order, the decisions that determine whether the deployment works, and where a SIEM stops being the right tool. Incident Response 101 owns what happens after the alert fires, Threat Hunting Guide owns searching the same data without an alert, and Windows Event IDs for Security owns which specific events to collect from Windows. This page owns the machine in the middle.

The pipeline

A SIEM is five stages. Every product implements all five, and the names differ.

1. Collection

Logs arrive by agent, by syslog, by API poll, or by a cloud-native stream.

Agents run on the host, read local logs, and forward them. They handle Windows event logs, files that rotate, and hosts that are sometimes offline. Syslog over UDP 514 or TCP 601 is how network gear reports, and it is lossy by design over UDP: a switch under load drops messages and nothing tells you. API polling collects from SaaS platforms, which is a queue you drain on a schedule and which can fall behind. Cloud-native streaming, meaning CloudTrail to a bucket or an event hub subscription, is usually the most reliable of the four because the provider persists the events until you read them.

The failure mode at this stage is silent. A collector that stops receiving from a source does not raise an error, because absence produces nothing. Every SIEM deployment needs a detection for a source going quiet, and it is the single most valuable rule most teams do not have.

2. Parsing and normalization

A raw log is a string. Detection logic needs fields.

Parsing turns Failed password for invalid user admin from 203.0.113.9 port 55231 ssh2 into a structure with a source address, a username, a service and an outcome. Normalization then maps that structure into a common schema, so that a failed authentication from SSH, from Windows, from Okta and from an application all land on the same field names.

This is where deployments quietly break. If one source delivers the account as user.name and another as account, then a detection written against user.name covers half the estate and reports nothing about the other half. It does not error. It returns zero results and looks like a clean environment.

The remedy is unglamorous. Pick a schema, whether the product's own, the Elastic Common Schema or OCSF, and validate that every onboarded source populates the fields your detections use. Onboarding a source is not finished when data arrives; it is finished when the fields the rules need are populated and spot-checked against a known event.

3. Enrichment

The raw event lacks context that the detection needs.

• Identity. An account name resolves to a person, a department, a manager, and whether the account is privileged or a service account.
• Asset. An IP address resolves to a hostname, an owner, a criticality rating and a location. Without this, an alert says 10.4.19.22 and an analyst spends twenty minutes finding out it is the payroll server.
• Threat intelligence. An address or hash matches a feed. Useful, and worth less than teams expect, because commodity indicators age out in days.
• Geolocation and ASN. Weak on their own and useful in combination, for example a login from a hosting provider's ASN for an account that has only ever authenticated from residential addresses.

Enrichment happens at ingest or at search time. At ingest it is fast to query and frozen at the value it had then; at search time it is current and slower. Asset and identity context is usually worth freezing at ingest, because you want to know what the asset was when the event happened.

4. Correlation and detection

Detection logic comes in a few distinct shapes, and knowing which is which prevents expecting the wrong thing from each.

Signature. A single event matches a pattern. A process named mimikatz.exe started. Cheap, precise, and trivially evaded by renaming the file.

Threshold. A count over a window. More than 20 failed authentications for one account in 5 minutes. Catches unsubtle brute force and nothing that paces itself below the threshold.

Sequence. Several events in order across sources. A successful authentication from a country the account has never used, then an inbox rule created, then a mass download from the file share. This is the shape a SIEM is uniquely able to express, and the shape that catches real intrusions.

Statistical or behavioral. A deviation from a learned baseline, marketed as UEBA. Strong on "this account has never touched this system in nine months of history" and weak on everything where the baseline is thin or already contains the attacker's activity.

A worked example, because the sequence shape is the one worth internalising. Consider these events across four sources in eleven minutes:

Time	Source	Event
09:14:02	Identity provider	Successful MFA authentication, user j.marsh, new device
09:15:40	VPN	Session established, source ASN belongs to a hosting provider
09:18:55	Windows DC	4624 logon type 3 to FS-01 as j.marsh
09:21:11	EDR on FS-01	net group "domain admins" /domain executed
09:24:47	Windows DC	4728, j.marsh added to a privileged group

No single source has enough to act on. The identity provider saw a successful login with MFA satisfied. The VPN saw a session. The domain controller saw a network logon and a group change, both of which happen legitimately dozens of times a week. The EDR saw a built-in Windows command, which is the whole point of Living Off the Land. Together the sequence is an account takeover progressing to privilege escalation, and expressing it is what the correlation engine is for.

5. Alerting, triage and response

The output is a queue a human works. What makes that queue workable:

• Severity that means something. If everything is high, nothing is.
• Deduplication. One incident that generated 40 events should be one item.
• Context attached. The asset, the user, the recent activity for both, and what the detection was looking for.
• A documented next step. A detection shipped without a triage note is a puzzle handed to whoever is on shift at 03:00.

Response automation, whether a separate SOAR product or the automation built into the SIEM, closes the loop for the cases where the action is mechanical: disable the account, isolate the host, open the ticket, enrich and requeue. The rule for automating a response is whether you would be comfortable with it firing on a false positive, because eventually it will.

What it is not

Not an EDR. The EDR agent sees a process tree, the command line, the DLLs loaded and the child processes spawned. The SIEM sees whatever the agent chose to forward. Deep on one host beats shallow on all hosts for endpoint detection, and the reverse is true for anything crossing systems.

Not a log archive, though it is often used as one. Compliance retention and detection have different requirements: retention wants cheap, durable and complete, detection wants fast, parsed and recent. Most mature deployments split them, keeping 30 to 90 days hot in the SIEM and the rest in object storage that can be rehydrated.

Not a substitute for prevention. A SIEM detects. Detection is what you build because prevention fails sometimes, and a control that stops the technique outright is worth more than an alert saying it happened.

Not a data lake, although the boundary is blurring. Security data lakes and pipeline tools sit in front of the SIEM, route high-volume low-value data to cheap storage, and send only what detections need into the expensive tier. For an estate generating terabytes a day this is the difference between an affordable program and an unaffordable one.

The cost problem

Pricing keys on volume, so architecture decisions are budget decisions.

The sources that dominate a bill are predictable: firewall accept logs, which are enormous and rarely detect anything on their own; verbose proxy logs; debug-level application logging that a developer enabled during an incident and left on; and NetFlow at full fidelity. The sources that produce the most detections per gigabyte are identity provider events, EDR telemetry, cloud audit logs and DNS.

That gives a workable discipline for onboarding. For each source, name the detections it enables. If the answer is "we might need it for an investigation", route it to cheap storage and query it there. If the answer is a named detection, ingest it and write that detection.

Two other levers matter. Filtering at the collector, dropping events that no detection or investigation uses, is cheaper than filtering after ingest. And tiering, with a hot window for detection and cold storage for the rest, matches the two different jobs the data does.

Detection engineering

The difference between a SIEM that works and one that does not is whether anyone owns the content.

Detections are code. They live in version control, they get reviewed, they have a documented purpose, a false positive history and a triage procedure. Sigma Rules Explained covers the vendor-neutral format that makes this portable, so a detection written once can be converted to the query language of whichever SIEM you are running.

Coverage is measured against ATT&CK. Mapping every detection to a technique turns "are we covered" into a chart with gaps on it. Be honest in the mapping: one detection for one procedure under a technique does not cover the technique, and a coverage map full of optimistic claims is worse than no map.

Every detection gets tested. Atomic Red Team and similar projects execute a technique deliberately so you can confirm the rule fires and see what the telemetry looks like. A detection that has never fired is untested rather than quiet.

Tuning is continuous. A rule that generates more than a few false positives a week either gets narrowed, gets an exception with an expiry date, or gets retired. Rules nobody reads are worse than no rules, because they create the appearance of monitoring.

Getting one working

For an organisation standing one up, an order that produces results:

1. Decide what you are trying to detect. Pick ten scenarios that would matter: account takeover, ransomware precursors, privileged group changes, data staging and exfiltration, cloud credential abuse. Everything downstream serves those.
2. Onboard the sources those scenarios need, and no more. Typically identity provider, EDR, cloud audit, DNS, perimeter, and Windows security events from domain controllers.
3. Validate parsing per source against a real event you generated yourself.
4. Fix time. Every host on NTP, every log stored in UTC. Correlation across sources with skewed clocks produces sequences that did not happen and misses ones that did.
5. Write the ten detections, with triage notes, in version control.
6. Add a source-silence detection, so a collector failing is an alert rather than a discovery made three weeks later during an investigation.
7. Tune for a month before adding more sources.

The verdict

A SIEM is a correlation engine with a log pipeline attached. Its unique value is seeing an attack across systems that each saw only a fragment, and it earns that value only when the parsing is right, the time is right, and someone owns the detection content.

The common failure is buying it as a product and expecting detection to arrive with the licence. The pipeline is a week of work and the content is permanent. Budget for the second one, size the ingest to the detections you actually wrote, and treat any source you cannot name a detection for as an archive candidate rather than a subscription.

Related guides

• Incident Response 101
• Threat Hunting Guide
• Windows Event IDs for Security
• MITRE ATT&CK Framework Explained
• Sigma Rules Explained