Log Analysis Has Been Solving the Wrong Problem
Most security teams treat log analysis as a retrospective exercise. Something happens, an alert fires or a user reports something odd, and then analysts go back into the logs to reconstruct what occurred. That backward orientation made sense when attackers moved slowly and infrastructure was simpler. Neither condition holds today.
The exposure of Cavern Manticore, an Iran-linked modular command-and-control framework documented by threat intelligence researchers, illustrates exactly why reactive log review fails against modern adversaries. The framework's modularity meant that individual components generated log activity that looked benign in isolation. A DNS query here, an outbound connection there, a brief authentication event. Only when analysts correlated log sources across time and system boundaries did the campaign become visible. By that point, the dwell time had already stretched into weeks.
The same pattern appeared in the Scattered Spider prosecutions. Investigators reconstructed the attack chain from logs that had been sitting in SIEM storage without triggering meaningful alerts. The evidence was there. The detection logic to surface it in real time was not.
This article is for security operations teams and IT administrators who want to move log analysis from a post-incident forensics tool into a live threat detection capability. The difference is not about buying more tooling. It is about understanding what logs actually contain, where collection gaps typically live, and how to build correlation logic that reflects how real attackers behave rather than how security vendors describe them in datasheets.
What Logs Actually Contain Versus What Teams Assume They Contain
The first practical problem is completeness. Most organizations believe their logging posture is stronger than it is. Audit reviews consistently find gaps between what teams think they are collecting and what is actually reaching the SIEM or log aggregation platform.
Common gaps include authentication events from legacy applications that write to local flat files instead of forwarding to a central collector, DNS query logs that are enabled on corporate resolvers but not on cloud-hosted workloads, process creation events on endpoints running older agents that do not capture command-line arguments, and egress traffic logs from cloud environments that require explicit configuration rather than being enabled by default.
Each gap is a blind spot that an adversary can move through without detection. The WSzero DDoS family, now in its fourth iteration and spreading via 21 distinct vulnerabilities, exploits this fragmentation. Variants are designed to operate on network-adjacent devices, routers, IoT endpoints, and network appliances, precisely because those devices rarely forward logs to enterprise collection infrastructure. The malware spreads and establishes persistence in the parts of the environment that security teams are not watching.
A logging completeness audit should precede any investment in detection logic. Map every authentication surface, every network segment, every cloud workload, and every endpoint category. For each one, verify that log data is reaching your aggregation layer with full fidelity, meaning timestamps are synchronized, fields are parsed correctly, and volume matches expectations. A sudden drop in log volume from a source is itself a detection signal worth alerting on.
Structuring Log Data for Threat Detection Rather Than Compliance
Compliance frameworks drove most enterprise logging programs into existence, and that heritage creates a structural problem. Compliance logging optimizes for auditability, preserving a record of who accessed what and when. Threat detection optimizes for behavioral visibility, surfacing patterns that indicate adversarial activity before or during an incident rather than after.
The difference shows up in what gets logged and how it gets stored. A compliance-oriented program might retain authentication success events for 12 months and discard authentication failure events after 30 days to manage storage costs. From a compliance perspective, successful authentications are the relevant records. From a threat detection perspective, the ratio of failures to successes, the timing distribution of failures, and the source IP patterns around failed authentications are among the most valuable signals available. Discarding failure events is exactly backward for detection purposes.
Restructuring log retention policy around detection value rather than compliance minimums typically requires a conversation with leadership about storage costs. The practical argument is straightforward: the student loan breach that exposed 2.5 million records became a recoverable investigation only because adequate authentication and API access logs survived long enough to reconstruct the attack chain. Organizations that had truncated those logs to save storage could not answer basic questions about scope and timing. The cost of inadequate retention vastly exceeded the cost of keeping the data.
A workable framework for retention prioritization ranks log sources by their detection value. Authentication logs, DNS query logs, process creation events with command-line arguments, network flow records, and API access logs belong in a high-retention category with at minimum 90-day hot storage and 12-month cold storage. Informational application logs with low detection value can be retained shorter or aggregated before storage.
Building Correlation Rules That Reflect Real Attack Behavior
Default SIEM correlation rules are built around known indicators: specific IP addresses, file hashes, domain names, and behavioral thresholds that have appeared in prior incidents. Against sophisticated adversaries, these rules generate significant noise on low-value alerts while missing novel activity entirely.
Effective correlation logic starts from attacker objectives rather than attacker tools. An adversary attempting lateral movement needs to authenticate to systems other than the one they initially compromised. That activity will appear in authentication logs regardless of the specific tools used. A correlation rule that fires when an account authenticates to more than a defined number of unique hosts within a short time window will catch lateral movement from a commodity script and from a custom framework like Cavern Manticore with equal reliability.
The following categories of behavioral correlation rules provide high detection value across a wide range of adversary techniques:
- Authentication anomaly rules: Alerts on accounts that authenticate from two geographically distant locations within a time window that makes travel impossible, accounts that authenticate outside their established working hours pattern, and accounts showing successful authentication immediately following a cluster of failures from a different source IP.
- Process lineage rules: Alerts on web server processes spawning command interpreters, scripting engines executing from unusual directories, and scheduled task creation from processes without administrative context in normal operations.
- DNS anomaly rules: Alerts on hosts querying domains registered within the last 30 days, high-volume subdomain queries to a single parent domain (a common indicator of DNS tunneling used in C2 frameworks), and queries to domains with unusually high entropy in their hostnames.
- Egress anomaly rules: Alerts on hosts initiating connections to IP addresses in ranges not previously contacted by that host, large data transfers to external destinations from hosts that do not routinely produce such transfers, and connections to ports associated with remote management on external infrastructure.
- Privilege escalation indicators: Alerts on token impersonation events, local group membership changes, and service installations outside of authorized change windows.
Building these rules requires baseline data. Each rule that fires against a behavioral deviation needs a baseline to deviate from. Collecting 30 to 60 days of log data before deploying anomaly-based correlation rules gives the detection logic enough context to separate genuine anomalies from normal operational variation. Deploying anomaly rules against a new log collection deployment without baseline data produces alert storms that train analysts to ignore the alert category entirely.
The Temporal Dimension That Most Detection Stacks Miss
Sophisticated threat actors manipulate timing deliberately to evade detection logic. Scattered Spider's social engineering campaigns operated across irregular time intervals, spreading activity thin enough that hourly and daily aggregation logic failed to accumulate the counts that would trigger threshold-based alerts. The attack appeared as a low, steady background signal rather than a spike.
Temporal analysis of log data adds a detection dimension that simple threshold rules cannot provide. Looking at the distribution of events over time, rather than just their count, reveals activity patterns that attackers cannot easily disguise. Legitimate user authentication concentrates during working hours for a given time zone, shows consistent weekly patterns including reduced activity on weekends, and varies predictably around holidays. Automated credential stuffing tools and human operators in different time zones produce distributions that visually and statistically deviate from these patterns even when total event counts remain below alert thresholds.
Implementing temporal analysis does not require machine learning infrastructure. Time-bucketed counts stored in a simple database, compared against rolling averages for the same time bucket from prior weeks, produce workable anomaly signals with straightforward SQL or SPL queries. A host that shows 14 authentication attempts in a given hour where the 30-day average for that hour is 1.2 is worth examining regardless of whether any individual attempt triggered an indicator-based rule.
The same temporal lens applies to network connections, DNS queries, and file system activity. The key is not just measuring what is happening but measuring when it is happening relative to established patterns for that entity, whether the entity is a user account, a host, or a network segment.
Log Analysis Across the Cloud and On-Premises Boundary
Modern enterprise environments span on-premises infrastructure, multiple cloud providers, SaaS applications, and managed services. Each environment generates logs in different formats, with different fields, different timestamp conventions, and different collection mechanisms. Adversaries who understand this fragmentation use the boundary between environments as a natural detection gap.
The Ostium cryptocurrency theft, which cost the platform $23.7 million via an off-chain attack vector, highlights how adversaries target the boundary between systems. Off-chain components often receive less security monitoring attention than on-chain activity because security teams focus tooling where they have the most visibility. Log analysis programs that cover on-premises infrastructure thoroughly but leave cloud and third-party integration surfaces with minimal collection reproduce this vulnerability across enterprise environments.
Normalizing logs from heterogeneous sources into a common data model is the foundational requirement for cross-environment detection. Frameworks like the Elastic Common Schema or Microsoft Sentinel's ASIM (Advanced Security Information Model) provide field normalization standards that allow correlation rules to operate across log sources without source-specific logic in every rule. Investing time in proper normalization at ingestion reduces the long-term maintenance burden on correlation rule sets significantly.
For cloud environments specifically, ensure that the following log sources are enabled and forwarding to your central collection platform: cloud provider authentication and API logs (AWS CloudTrail, Azure Activity Logs, GCP Audit Logs), virtual network flow logs, cloud storage access logs, identity provider logs for SSO and federation, and container orchestration logs from Kubernetes audit logging if applicable. Many of these sources are off by default and require explicit enablement with associated storage costs that teams sometimes avoid configuring.
Operationalizing Log Analysis in a SOC With Limited Bandwidth
Even well-structured log collection and correlation logic produces more alerts than most security operations teams can investigate manually. Prioritization is not optional; it is the mechanism that makes the detection program sustainable.
Alert prioritization should incorporate context beyond the event itself. An authentication failure alert for a standard user account is lower priority than the same alert for an account with elevated privileges or access to sensitive systems. A DNS anomaly from a developer workstation in a test environment is lower priority than the same anomaly from a production database server. Enriching alerts with asset criticality data, user role data, and network segment classification at the time of alert generation allows triage analysts to apply consistent prioritization without manually researching context for each alert.
Threat intelligence integration adds another layer of prioritization context. The Recorded Future Advanced Persistent Threat tracking data and similar intelligence sources identify infrastructure and behavioral indicators associated with specific threat actor groups. When log correlation surfaces activity that matches indicators associated with tracked APT groups, that alert warrants immediate escalation regardless of its severity score on volume-based metrics alone. Iran-linked activity patterns from Cavern Manticore analysis, for example, include specific C2 communication timing signatures and module staging behaviors that can be built into correlation rules to catch similar activity from the same or related actors.
Runbooks for common alert types reduce the time analysts spend on procedural decisions and free cognitive capacity for genuine investigation work. A well-written runbook for an authentication anomaly alert specifies exactly what additional log queries to run, what contextual data to gather, what initial containment steps are appropriate if certain conditions are met, and what escalation path applies. Analysts following a runbook consistently produce higher-quality triage decisions than analysts working from memory under time pressure.
Using Log Data for Threat Hunting Beyond Alert Response
Reactive alert response catches known bad patterns. Threat hunting uses log data proactively to search for adversary activity that has not yet triggered an alert. The distinction matters because sophisticated attackers specifically design their tradecraft to avoid triggering known detection rules. Hunting starts from a hypothesis about what attacker behavior might look like in your environment and searches log data for evidence of that behavior.
Hypothesis generation for threat hunting draws on several sources. Published threat intelligence about active campaigns targeting your industry vertical, analysis of recent incident reports from peer organizations, and the tactics, techniques, and procedures documented in frameworks like MITRE ATT&CK all provide starting points for hypotheses worth investigating.
A practical hunting workflow for a log-based hunt proceeds as follows:
- Select a specific technique from ATT&CK or a recent threat report. For example, T1071.004, Application Layer Protocol: DNS, covers DNS-based C2 communication like the patterns observed in Cavern Manticore campaigns.
- Identify the log sources that would capture evidence of this technique in your environment. DNS query logs from resolvers and endpoint DNS client logs are the primary sources for DNS-based C2 detection.
- Write queries that surface the behavioral pattern associated with the technique. For DNS tunneling, query for hosts generating more than a defined threshold of unique subdomains against a single parent domain within a time window, or for unusually long hostname strings in query records.
- Review results, filtering out known legitimate sources like update infrastructure and CDN services that legitimately use high-volume subdomain patterns.
- Investigate anomalous results with additional log queries to determine whether observed activity represents a genuine threat or a previously unknown legitimate pattern that should be documented and excluded from future hunts.
Documenting hunt results creates institutional value beyond the individual hunt. Patterns identified as legitimate during one hunt become the exclusion list for future hunts. Patterns identified as malicious become candidates for conversion into persistent correlation rules. Over time, a hunting program systematically improves the fidelity of the alert-based detection program while simultaneously discovering threats that alert-based detection misses.
Practical Implementation Priorities for Teams Starting From a Weak Baseline
Organizations that are working to mature their log analysis capability from a low baseline face a prioritization challenge. Comprehensive logging, normalization, and detection logic is a multi-year program. Knowing which improvements deliver the highest detection value in the shortest time guides resource allocation effectively.
The following sequence reflects detection value per implementation effort, favoring sources and rules that catch the widest range of adversary techniques with the most straightforward implementation:
First 30 days: Verify that authentication logs from all identity providers, including Active Directory domain controllers, cloud identity platforms, and VPN infrastructure, are reaching your SIEM with full fidelity. Build three correlation rules covering impossible travel authentication, after-hours authentication anomalies, and high-failure-to-success authentication ratios. These rules alone catch credential-based attacks, which represent a significant proportion of initial access techniques across virtually every industry.
Days 31 to 60: Enable DNS query logging on all corporate resolvers and major cloud workloads. Build correlation rules for high-volume subdomain queries and for queries to recently registered domains. Enable process creation logging with command-line arguments on endpoints if not already in place. Build a rule for web server processes spawning scripting engines or command interpreters.
Days 61 to 90: Implement network flow logging for east-west traffic within your environment, not just perimeter traffic. Build rules for internal hosts initiating connections to large numbers of unique internal targets, which indicates lateral movement or internal scanning activity. Begin a formal baseline collection period for temporal anomaly detection before deploying time-based rules.
Each phase builds on the previous one and creates detection capability that compounds in value as more sources are integrated and correlation rules can reference richer context.
Where Log Analysis Fits in a Broader Threat Detection Program
Log analysis is the foundational layer of a threat detection program, but it functions best when integrated with adjacent capabilities rather than operating in isolation. Threat intelligence feeds provide context that distinguishes routine anomalies from indicators of tracked campaigns. Endpoint detection and response platforms generate rich process and memory event data that complements log data from network and authentication sources. Vulnerability management data enriches alert prioritization by identifying which systems have exploitable conditions that make observed anomalies more consequential.
The AI security capabilities receiving attention in the current threat landscape, referenced in the AI Security Report 2026, are increasingly being integrated into SIEM and SOAR platforms to augment correlation and triage capabilities. CISOs are under pressure from boards and regulators to demonstrate that AI risk management extends to AI used in security tooling itself. Understanding what your detection platform's AI components are doing, what data they train on, and what their failure modes look like is now a legitimate operational concern rather than a theoretical one.
What remains constant across every tooling evolution is the requirement for complete, high-fidelity log data. Machine learning models trained on incomplete or poorly normalized log data produce detection outputs that reflect the gaps in training data rather than the actual threat landscape. The fundamentals of log collection, normalization, retention, and correlation are not legacy practices being replaced by more sophisticated approaches. They are the substrate that every more sophisticated approach depends on to function correctly.
The teams that caught Cavern Manticore activity early were not the ones with the most advanced tooling. They were the ones with complete log collection, correlation logic built around adversary objectives rather than specific indicators, and analysts who understood what the logs were telling them well enough to ask the right follow-on questions. That combination is available to any organization willing to invest the time to build it methodically.
