The Threat Landscape That Makes Log Analysis Non-Negotiable
The Unlimited Technology Systems breach, which impacted 3.8 million people, and the student loan breach exposing 2.5 million records share a common thread that investigators keep uncovering: the indicators were present in logs before the full scope of the compromise became apparent. The data was there. The correlation was not.
ESET's Threat Report for H1 2026 highlights a continued rise in credential-harvesting campaigns and lateral movement techniques that deliberately mimic legitimate administrative behavior. Attackers are not crashing through front doors. They are walking in through service accounts, abusing trusted processes, and living inside environments for weeks before triggering any alert that a human investigates.
Meanwhile, the sale of access to Chinese surveillance cameras circulating in criminal marketplaces means that networked devices many organizations never considered part of their log strategy are now potential footholds. Cameras, building management systems, and similar IoT infrastructure generate logs that most security teams have never ingested, normalized, or analyzed. That gap is exactly where initial access happens in an increasing share of incidents.
Log analysis sits at the center of every serious detection program, but the discipline requires more than collecting data and waiting for a SIEM correlation rule to fire. This article walks through what effective log analysis looks like in practice, which sources matter most, how to structure detection logic, and where most implementations break down before they can deliver value.
Why Log Collection Strategy Determines Detection Outcome
Security teams frequently conflate log volume with log coverage. A high-volume environment that ingests gigabytes of firewall data daily can still have significant blind spots if endpoint telemetry, authentication logs, and cloud control plane activity are absent or inconsistently normalized. Detection quality is bounded by collection completeness.
The sources that produce the most actionable signal in modern environments fall into several categories. Authentication and identity logs from Active Directory, Azure AD, Okta, or equivalent systems reveal credential abuse, privilege escalation, and lateral movement. These logs should capture every authentication attempt, token issuance, MFA bypass attempt, and administrative action. Linux shell history, including tools like Atuin that provide structured, timestamped command history with session context, adds a layer of forensic fidelity that traditional bash history files lack. When a threat actor runs reconnaissance commands or deploys persistence mechanisms on a Linux host, Atuin-style logging captures the sequence with enough context to reconstruct attacker behavior during an investigation.
Network flow data from routers, switches, and firewalls documents communication patterns between hosts and external destinations. Application logs from web servers, APIs, and databases capture request patterns that reflect exploitation attempts, data staging, and exfiltration. Cloud control plane logs, specifically AWS CloudTrail, Azure Activity Logs, and GCP Audit Logs, record every API call made to cloud infrastructure and are irreplaceable when investigating identity-based attacks in cloud environments.
Endpoint detection and response telemetry fills the gap between what the network sees and what actually executes on a host. Process creation events, file system changes, registry modifications, and network connections initiated by specific processes all contribute to a coherent picture of attacker behavior that no single log source provides on its own.
Structuring Detection Logic Around Attack Phases
Effective log-based detection maps to attacker behavior rather than to specific tools or signatures. Tools change. Techniques persist. The MITRE ATT&CK framework provides a practical vocabulary for structuring detection logic, but the operational challenge is translating framework concepts into specific queries and alerting rules that reflect your environment.
During the initial access phase, authentication logs reveal credential stuffing, password spray, and brute force activity. The signature of a password spray campaign in authentication logs differs from a brute force attack. A spray distributes attempts across many accounts with low per-account frequency, while brute force concentrates attempts against a small number of accounts. Detection logic that watches only for account lockout thresholds misses sprays almost entirely, because sprays are specifically designed to stay below lockout thresholds. Effective detection looks for a single source making authentication attempts against an unusual breadth of accounts within a defined time window, regardless of whether any individual account reached its lockout limit.
Lateral movement produces characteristic patterns in authentication logs. Pass-the-hash and pass-the-ticket attacks generate Kerberos and NTLM events with specific characteristics. A machine account authenticating to several other hosts in rapid succession, or a user account that authenticates from multiple endpoints within minutes, both warrant investigation. Windows Security Event IDs 4624, 4625, 4648, and 4768 are foundational here and should be ingested and queryable in near real time.
Command and control traffic appears in DNS logs as unusual query patterns, in proxy logs as beaconing behavior with consistent intervals, and in firewall logs as connections to infrastructure with poor reputation or low-prevalence domain registrations. The Meta AI escape-from-testing incident, while specific in nature, illustrates a broader principle: unexpected outbound communication from systems that have no business reason to initiate external connections is a detection opportunity that most teams configure their way out of by treating those systems as low-priority logging sources.
Data staging and exfiltration show up in file access logs, database query logs, and network flow data. Large volumes of data moving from sensitive file shares to workstations, or SQL queries returning unusual row counts at unusual hours, are signals that correlate across log sources. Neither source alone is conclusive. Together, they form an investigation thread.
Detection Checklist for Log Analysis Programs
- Authentication log coverage: Confirm that every identity provider, including legacy LDAP, cloud SSO, and on-premises AD, forwards authentication events to your SIEM with consistent timestamp normalization.
- Endpoint command execution logging: Verify that PowerShell script block logging, Windows command-line auditing, and Linux shell history logging are enabled and ingesting on all managed hosts. For Linux environments, audit whether structured shell history tools like Atuin are deployed on high-value systems.
- Cloud control plane completeness: Audit CloudTrail, Azure Activity Logs, and equivalent sources to confirm that management API calls, IAM changes, and storage access events are all captured with no regional gaps.
- Network device log ingestion: Validate that firewall deny and allow logs, DNS query logs, and proxy access logs are ingesting consistently. Check for gaps caused by log rotation, storage limits, or misconfigured syslog forwarding.
- IoT and non-standard device logging: Identify all networked devices that do not have a defined log source in your SIEM, including cameras, building systems, and industrial controllers. Establish network flow visibility for these devices as a minimum baseline.
- Log retention alignment with investigation needs: Confirm that retention periods match your organization's incident response requirements. Investigations into long-dwell intrusions frequently require 90 to 180 days of log history. Short retention windows destroy evidence before investigations begin.
- Correlation rule testing cadence: Establish a schedule for testing detection rules with synthetic attacker behavior in staging environments. Rules that worked 18 months ago may no longer fire correctly after schema changes, parser updates, or data source modifications.
- Baseline documentation: Maintain documented baselines for authentication volumes, DNS query rates, outbound connection counts, and data transfer volumes by system type. Anomaly detection without a baseline produces noise, not signal.
- Alert triage workflow documentation: Ensure that every detection rule maps to a documented triage procedure. Alerts without response procedures age in queues rather than driving investigations.
- Cross-source correlation rules: Build and maintain at least a core set of detection rules that correlate across two or more log sources. Single-source detections miss multi-stage attack chains.
Real-World Detection Scenarios
Consider a scenario where an attacker purchases access to a compromised surveillance camera connected to a corporate network segment. The camera is not a managed endpoint. It generates no EDR telemetry. But it does generate network connections, and if network flow logging covers that segment, those connections appear in your data. The camera suddenly initiating outbound connections to an external IP on port 4444, or generating DNS queries for domains registered within the last 30 days, represents a detectable anomaly if you have baselining and alerting configured for that network segment.
In authentication log scenarios, consider the pattern that appears when a threat actor has obtained a valid set of credentials through a phishing campaign and begins accessing cloud services from a new geographic location. The authentication succeeds. No lockout fires. No rule based on failure counts triggers. But the combination of a successful authentication from a previously unseen ASN, followed immediately by API calls to cloud storage resources, followed by a file download volume that exceeds the 90-day baseline for that account, creates a correlation that a well-tuned SIEM can surface. Each event alone looks unremarkable. Together they describe an active intrusion.
Database logs provide another underutilized detection surface. When an attacker who has achieved lateral movement reaches a database server and begins executing queries that return large result sets outside of business hours, the query logs capture that activity. Correlating those queries against the authenticated user session that initiated the connection, and then tracing that session back to the workstation that authenticated using a service account that had not generated any logon events in the previous 30 days, builds a complete detection chain from a breach that a single-source alert would have missed entirely.
Normalizing and Enriching Log Data at Ingestion
Detection quality degrades when log data arrives in inconsistent formats, uses different timestamp conventions, or lacks contextual fields that queries depend on. Normalization at ingestion is not optional for mature programs. Every log source should map to a common schema that includes at minimum: source IP, destination IP, username, timestamp in UTC, event type, and outcome. Fields that exist in one source but not another require careful handling to avoid detection gaps caused by schema mismatches in correlation rules.
Enrichment adds context that raw logs lack. Associating an IP address with its ASN, resolving hostnames to current IP addresses, tagging assets with their business function and criticality, and annotating user accounts with their role and associated department all accelerate triage and improve alert fidelity. An authentication event from an IP associated with a residential ISP in an unexpected country means more to an analyst when the enrichment layer has already flagged the ASN as high-risk and noted that the authenticating account belongs to a privileged administrator.
Timestamp integrity deserves specific attention. Distributed environments frequently have hosts with unsynchronized clocks. A two-minute clock skew between an endpoint and an authentication server can cause correlation rules that rely on time-based sequencing to fail silently. NTP compliance monitoring and log ingestion pipelines that flag or reject events with implausible timestamps both help maintain the time integrity that multi-source correlation depends on.
Tuning Detection Rules to Reduce Noise Without Creating Gaps
Alert fatigue is a real operational problem. Analysts who process hundreds of low-fidelity alerts daily develop habits that allow high-fidelity alerts to age in queues. Tuning is the mechanism that maintains the signal-to-noise ratio necessary for human attention to remain focused on genuine threats.
Tuning starts with understanding why rules fire incorrectly. False positives fall into patterns: a backup job that runs at 2 AM and generates authentication events that look like lateral movement, a developer workstation that queries unusual external domains as part of a legitimate software development workflow, or a network scanner that generates probe-like traffic on an authorized schedule. Each of these can be addressed through specific exclusions that target the legitimate behavior without broadly suppressing the detection category.
The discipline in tuning is to make exclusions as narrow as possible. An exclusion that suppresses all authentication events from a specific subnet because one backup job generated false positives also suppresses real lateral movement from that subnet. An exclusion scoped to the specific service account, the specific destination, and a time window matching the backup job's schedule achieves noise reduction without sacrificing coverage.
Review tuning decisions on a defined schedule, typically quarterly or following any significant infrastructure change. Backup job schedules change. Developers move teams. Network scanners get decommissioned. Exclusions that matched legitimate behavior six months ago may now suppress detections for activity that has no legitimate explanation.
Implementation Pitfalls That Undermine Mature Log Programs
The first and most common pitfall is treating SIEM deployment as a completion event rather than an ongoing operational discipline. Organizations invest significant effort in the initial deployment, connect their primary log sources, configure a set of detection rules, and then shift attention elsewhere. Log sources change over time. New cloud services get provisioned. Network segments get added. Applications get deployed. Unless the log collection inventory is actively maintained, coverage gaps accumulate silently.
A second pitfall is building detection logic exclusively around known-bad indicators rather than behavioral patterns. IP blocklists, domain reputation lists, and file hash databases have value, but sophisticated attackers rotate infrastructure faster than threat intelligence feeds update. A detection program that relies entirely on indicator matching misses actors who operate on fresh infrastructure. Behavioral detection, which identifies what attackers do rather than what infrastructure they use, maintains effectiveness against novel campaigns.
A third pitfall is insufficient log retention. Breaches like the Unlimited Technology Systems incident frequently involve dwell times measured in weeks or months. Investigating a breach that began 60 days before discovery requires 60 days of logs. Organizations with 30-day retention windows lose the evidence needed to understand initial access, establish the full timeline, and identify every affected system. Retention decisions should be made based on investigation requirements, not storage cost minimization alone.
A fourth pitfall is collecting logs without validating that they are complete and parseable. A log source that forwards data but whose parser fails silently on certain event types produces a collection that looks healthy in dashboards but has invisible gaps. Regular validation of parse rates, field population completeness, and event volume against expected baselines catches these issues before they matter during an investigation.
Finally, many organizations build log analysis programs that produce alerts but lack the workflow infrastructure to act on them effectively. An alert that identifies a genuine intrusion but sits in a queue for 72 hours while an analyst waits for access to the affected system, authorization to take containment action, or documentation of the response procedure has not delivered its value. Detection and response are coupled. A log analysis program that invests heavily in detection while leaving response workflows underdocumented and under-resourced will consistently fail to convert its detections into timely containment.
Log analysis is not a technology problem that gets solved once. It is an operational discipline that requires sustained investment in collection coverage, detection logic, analyst workflow, and continuous validation. The organizations that close breaches quickly are almost universally the ones that treated log analysis as infrastructure worth maintaining rather than a deployment milestone worth celebrating.
