What is Log Management?

NIST's guide to the subject, SP 800-92, defines log management as "the process for generating, transmitting, storing, analyzing, and disposing of computer security log data." Each of those stages can fail quietly, and you usually find out in the middle of an investigation, so it pays to know them before deciding what to collect and how long to keep it.

The NIST life cycle, and the revision still in draft

SP 800-92 dates from September 2006 and is still the final version on NIST's CSRC site. Its revision, SP 800-92 Rev. 1 Cybersecurity Log Management Planning Guide, came out as an initial public draft on 11 October 2023. The comment period closed on 29 November 2023 and no final version has followed. The draft rewrites the definition as "generating, transmitting, storing, accessing, and disposing of log data", with accessing in place of analyzing, and reorganizes the guide as a planning playbook of four plays, starting with an inventory of log source types. Its note for federal agencies still sends readers to OMB M-21-31, which OMB rescinded in May 2026 (see the retention section).

What a SOC collects first

The joint guide Best practices for event logging and threat detection (August 2024), led by the Australian Signals Directorate's ACSC with co-authors including CISA, the FBI and the NSA, ranks log sources for enterprise networks. The top five are critical systems and data holdings likely to be targeted; internet-facing services, including remote access; identity and domain management servers; any other critical servers; and edge devices such as boundary routers and firewalls. User computers come 10th, web proxies 12th and DNS services 13th. For cloud services the guide asks for all control plane operations, "including API calls and end user logins."

That list ranks systems. On those systems, these are the events most worth turning on first, with the setting that decides whether each one is any use:

The Windows events we document in depth are on the event ID hub. Sysmon covers what the Security log leaves out: Microsoft documents that its process creation event carries the full command line and a ProcessGUID for correlation, and that Event ID 22 is written whenever a process makes a DNS query, cached or not.

Retention: hot, cold and the federal numbers that changed in 2026

Hot storage is what detections and analysts search directly. Cold storage costs less and needs a restore or rehydration step before you can query it. The ASD guide recommends tiering logs this way and warns that many systems overwrite old logs once their storage allocation is exhausted, which is one more reason to forward logs off the device early.

The most specific public retention numbers come from US federal policy, and they changed this year. OMB M-21-31 (27 August 2021) defined four event logging tiers, from EL0 (Not Effective) to EL3 (Advanced), and for most log categories required 12 months in active storage plus 18 months in cold data storage, with full packet capture kept for 72 hours. OMB M-26-14 (22 May 2026) states "Effective immediately, OMB Memorandum M-21-31 is rescinded", because keeping "vast quantities of logging data without clear utility" had proved neither operationally feasible nor cost-effective for most agencies. The new minimum:

• logs stay searchable for 6 months after creation, which the memo defines as usable by detections and analytics without any preparation step;
• and retrievable for a year, meaning usable after steps such as moving data out of an archive or "thawing" it from cold storage.

M-26-14 replaces EL0 to EL3 with a maturity model that runs from Ineffective (Level 0) to Optimal (Level 4) and tasks CISA with a logging reference architecture. Both memos apply to US federal agencies (national security systems and some Defense and Intelligence Community systems are excluded), not to private companies. Before you trim retention to six months, weigh the ASD guide's warning that it can take up to 18 months to discover a cyber security incident.

Worked example (illustrative figures, not a benchmark). Suppose your sources produce 40 GB of parsed logs a day before compression, with 1 TB counted as 1,000 GB:

The rescinded rule meant twice the hot storage and two and a half times the total volume. That difference is the cost argument M-26-14 makes, and the reason to set retention per source rather than one number for everything: authentication and process events earn the longest searchable window, verbose debug output can go cold early or not be kept at all.

A simulated case: the log that was collected but said nothing

Simulated example generated by SOCSimulator Research. Hosts, domains, IPs and times are invented; the event ID, field names and Group Policy setting are Microsoft's.

The SIEM has a rule that alerts when powershell.exe runs with -EncodedCommand or -enc on its command line. On 5 October a user on WS-022 opens an HTA file from an email. Six days later the web proxy matches 198.51.100.23 against a new threat intelligence feed, and the analyst pulls process creation for the host from the SecurityEvent table:

TimeGenerated        Computer             EventID  ParentProcessName                                          NewProcessName                                             CommandLine
2026-10-05 09:12:40  WS-022.corp.example  4688     C:\Windows\explorer.exe                                    C:\Windows\System32\mshta.exe
2026-10-05 09:12:44  WS-022.corp.example  4688     C:\Windows\System32\mshta.exe                              C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
2026-10-05 09:12:51  WS-022.corp.example  4688     C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe  C:\Windows\System32\rundll32.exe

CommandLine is empty on every row. The host was auditing process creation, its 4688 events reached the SIEM and it never went silent, but the Group Policy setting "Include command line in process creation events" had never been applied to the workstation OU. Microsoft documents that the setting is not configured by default, and that while it is disabled or not configured, 4688 leaves the command line out. The encoded-command rule had nothing to match. Script Block Logging was off as well, so no 4104 recorded the script, and Sysmon ran only on servers.

What was collected still had a use. mshta.exe starting powershell.exe is a parent and child pair that a rule can catch with no command line at all, and that rule went in the same day. The post-incident report filed it as a log management gap: the events existed, in a shape the detection could not use.

This Microsoft Sentinel query finds hosts that send 4688 without command lines before an incident does:

// Hosts sending process creation (4688) with no command lines in the last 24 hours
SecurityEvent
| where TimeGenerated > ago(24h)
| where EventID == 4688
| summarize Processes = count(), WithCommandLine = countif(isnotempty(CommandLine)) by Computer
| where WithCommandLine == 0
| order by Processes desc

And this one lists sources that went quiet, computers that sent Security events during the last week but none in the last 24 hours:

// Windows hosts that stopped sending Security events
SecurityEvent
| where TimeGenerated > ago(7d)
| summarize LastEvent = max(TimeGenerated) by Computer
| where LastEvent < ago(24h)
| order by LastEvent asc

Both use the documented SecurityEvent schema and assume Windows Security events are already collected into Sentinel. Decommissioned machines also appear in the second result, so check the asset inventory before you open a ticket.

Syslog from network devices and Linux hosts

Network devices, Linux hosts and many appliances send syslog. RFC 5424 starts each message with a priority value: the facility number multiplied by 8, plus the severity. Facility 10 (security/authorization) at severity 6 (Informational) becomes <86>. Severities run from 0 (Emergency) to 7 (Debug), and facilities 4 and 10 both carry security/authorization messages. Plain syslog over UDP port 514 has no delivery guarantee: RFC 5426 warns that messages "can be lost in transit without any notice," and that an attacker can intercept and discard them to hide activity. For security sources, use syslog over TLS on TCP port 6514 (RFC 5425). The ASD guide recommends TLS 1.3 and cryptographic verification to protect logs in transit and at rest.

Where log management ends and the SIEM begins

Log management is the pipeline and the archive: collect, parse, store, protect, keep for the right period, delete. A SIEM is the analysis layer on top, where correlation rules and detections run on whatever reaches it. The ASD guide describes the split directly: a centralised logging facility, such as a secured data lake, aggregates logs and forwards select, processed logs to SIEM and XDR tools. A detection can only fire on data that reaches the SIEM in the shape the rule expects. That is why log management problems show up in a SOC as alerts that never fired.