The Gap Between Collecting Logs and Actually Using Them
Most organizations today generate logs from dozens of sources: firewalls, endpoint detection platforms, authentication systems, cloud service providers, web proxies, and DNS resolvers. The volume is substantial, the tooling exists, and yet compromise assessments continue to surface the same finding year after year: attackers were resident in environments for weeks or months before detection. The July 2026 Threat Intelligence Report highlighted persistent threats and response gaps as the defining characteristic of incidents that escalated into major breaches. The issue is rarely a failure to collect logs. It is a failure to extract meaningful signal from what is already being captured.
This article addresses the practical mechanics of log analysis for threat detection, separating what actually works from the assumptions that leave organizations exposed. The scenarios and techniques below are drawn from real compromise assessment findings, current threat actor behavior, and the kinds of log patterns that consistently surface during incident response.
Why Persistent Threats Survive Inside Log-Rich Environments
The 0ktapus threat group campaign, which victimized over 130 firms, succeeded largely because attackers blended initial access activity into authentication log noise that teams were not actively interrogating. Similarly, recent compromise assessment insights published in June 2026 found that missed incidents were not caused by absent logging. They were caused by logging configurations that captured events without surfacing patterns, and by detection logic that was tuned to known-bad indicators rather than behavioral sequences.
Persistent threat actors, including the Iran-linked group behind the Cavern Manticore modular C2 framework, specifically design their tooling to avoid generating discrete high-confidence indicators. Cavern Manticore uses modular components that communicate over encrypted channels with variable beaconing intervals, making individual log entries appear benign. The threat becomes visible only when log entries are correlated across time and across multiple sources simultaneously.
The practical implication is that log analysis as a detection discipline requires a different mental model than log analysis for compliance or auditing. Compliance logging asks whether an event occurred. Threat detection asks whether a sequence of events, taken together, represents attacker behavior.
The Log Sources That Actually Matter for Threat Detection
Not all log sources carry equal detection value. Security teams frequently over-invest in tuning certain log sources while under-utilizing others that consistently surface early-stage attacker activity.
Authentication and Identity Logs
Authentication logs remain the highest-value source for detecting credential-based attacks, lateral movement, and privilege escalation. The specific fields that matter most are user principal name, source IP address, authentication method, failure reason codes, and session duration. A single failed login generates minimal signal. A sequence of authentication attempts across multiple accounts from a single IP over a six-hour window, with failure reasons indicating account enumeration, represents a clear behavioral pattern.
Azure Active Directory, Okta, and on-premises Active Directory all expose these fields, but teams frequently configure SIEM ingestion to capture only authentication events and discard the failure reason codes that differentiate a locked-out employee from an active credential spray. The 0ktapus campaign exploited exactly this kind of configuration gap, using targeted phishing to harvest credentials and then authenticating through SSO providers in ways that generated minimal alerting.
DNS Query Logs
DNS logs are systematically underutilized for threat detection. Every C2 framework, including Cavern Manticore, requires DNS resolution at some stage of its operation. DNS logs expose domain generation algorithm (DGA) activity, connections to recently registered domains, and the kind of high-frequency, low-byte DNS queries that characterize beacon traffic. The WSzero DDoS family, now in its fourth version, uses DNS-based infrastructure management that appears in query logs before any network-layer signatures become available.
The specific patterns worth monitoring include: domains queried by a single internal host that no other host in the environment has ever queried, NXDOMAIN response rates above baseline for specific hosts, and queries to domains with entropy scores consistent with DGA behavior. Many endpoint detection platforms now ship with DNS telemetry, but teams frequently disable high-volume log sources to manage SIEM storage costs, removing one of the clearest early-warning signals available.
Web Proxy and HTTP Access Logs
Web proxy logs capture the full HTTP request URI, referrer, user agent string, response code, and bytes transferred. These fields collectively expose a range of attacker behaviors that network flow data misses entirely. User agent rotation, which sophisticated actors use to blend C2 traffic into legitimate browser patterns, leaves detectable artifacts when proxy logs are analyzed for user agent diversity per host over time. A host that generates five distinct user agent strings within a single session, across requests to the same destination, is exhibiting behavior inconsistent with legitimate browser activity.
Response code sequences also carry significant signal. A 403 followed by a 404 followed by a 200 across sequential requests to the same URI stem is consistent with directory traversal and discovery behavior. This pattern appears clearly in proxy logs but requires sequence-aware analysis rather than single-event alerting.
Endpoint Process Execution Logs
Windows Event ID 4688 with command line auditing enabled, or equivalent telemetry from EDR platforms, captures process creation events including parent-child process relationships and full command line arguments. Living-off-the-land attacks, which now represent the majority of hands-on-keyboard intrusion activity, are nearly invisible without this telemetry. The execution of certutil.exe with base64 decode flags, PowerShell with encoded command parameters, or wmic.exe spawned from a browser process are examples of behaviors that appear clearly in process execution logs but generate no network-layer signatures.
Building Detection Logic That Catches Behavioral Sequences
The fundamental unit of threat detection in log analysis is the behavioral sequence, not the individual event. This distinction drives architecture decisions, SIEM query design, and alert triage workflows.
Defining Baselines Before Writing Rules
Detection logic written without established baselines produces alert volumes that overwhelm analysts and train teams to dismiss notifications. Before writing any detection rule, analysts need a documented baseline for the specific behavior the rule targets. For authentication failures, this means knowing the average daily failure count per account, the standard deviation, and the distribution across source IPs. For outbound DNS query volume, this means knowing the typical hourly query rate per host category.
Baselining does not require sophisticated tooling. A 30-day historical query against authentication logs that calculates per-account failure rates provides sufficient context to set a meaningful threshold that distinguishes genuine anomalies from routine noise. The investment in baselining directly reduces false positive rates, which is the primary reason detection rules get disabled by overworked analysts.
Correlation Across Time Windows
Most SIEM platforms support time-windowed correlation, and this capability is underused in practice. A rule that fires when an account generates more than 10 authentication failures within 60 minutes is a point-in-time threshold. A rule that fires when an account generates more than 10 failures within 60 minutes AND successfully authenticates within 30 minutes of the last failure adds a second condition that dramatically increases specificity. This two-stage correlation pattern distinguishes a locked-out employee from a successful credential attack.
The Cavern Manticore C2 framework demonstrates why time-window breadth matters. Modular C2 frameworks specifically vary their activity patterns to stay below short-window thresholds. A detection rule examining beacon-consistent outbound connections over a 24-hour window catches patterns that a 60-minute window misses entirely.
Cross-Source Correlation
The highest-confidence detections combine signals from multiple log sources. An alert that fires based on authentication logs alone carries lower confidence than an alert that fires when authentication failure patterns from identity logs correlate with outbound DNS queries to newly registered domains from the same host, which correlate with proxy log entries showing user agent rotation to the same destination. Each individual signal might be explainable. The combination is not.
Building cross-source correlation rules requires clean asset data. If a host identifier in authentication logs cannot be reliably matched to the same host in DNS logs, the correlation query will fail. Asset inventory quality is a direct prerequisite for effective cross-source log analysis, and this dependency is frequently overlooked when teams plan their detection engineering roadmap.
Threat Detection Log Analysis Checklist
The following checklist provides a practical starting point for cybersecurity professionals and IT administrators assessing or improving their log analysis capability for threat detection.
- Confirm log completeness: Verify that authentication logs capture failure reason codes, not just success or failure events. Confirm that process execution logs include full command line arguments. Validate that DNS query logs are forwarded from all internal resolvers, including any split-horizon or cloud-hosted resolvers.
- Establish retention policies aligned with dwell time: Compromise assessment data consistently shows attacker dwell times of 30 to 90 days. Log retention shorter than 90 days eliminates the historical context needed to reconstruct early-stage activity. Set minimum retention at 90 days for high-value sources, with 12-month retention for authentication and identity logs where storage permits.
- Document baselines for critical behaviors: Run 30-day historical queries to establish per-account authentication failure rates, per-host outbound DNS query volumes, per-host outbound bytes to new destinations, and process execution frequency for high-risk binaries such as certutil, mshta, regsvr32, and PowerShell with encoded commands.
- Audit SIEM ingestion configurations: Walk through each log source and confirm that the fields required for detection logic are actually present in indexed events. Confirm that high-cardinality fields such as user agent strings, DNS query names, and full URI paths are indexed rather than stored as raw text only.
- Test detection rules against historical data: Before deploying any new detection rule, run it against 30 days of historical logs and review all matches. This identifies both false positives and tuning opportunities before the rule generates live alerts.
- Implement time-windowed correlation for credential attacks: Add correlation rules that detect authentication failure sequences followed by successful authentication within a defined time window. Tune the window and threshold based on your baseline data.
- Enable and monitor for beacon-consistent traffic patterns: Configure queries that identify outbound connections to the same destination occurring at regular intervals over multi-hour periods. Flag hosts where connection intervals show low variance over a 24-hour window, which is inconsistent with human-driven browsing behavior.
- Review alert disposition data quarterly: Track the ratio of alerts closed as false positives versus true positives per detection rule. Rules with false positive rates above 90 percent are candidates for retuning or retirement. High false positive rates suppress analyst attention and are a direct contributor to missed detections.
- Validate coverage against current threat actor techniques: Map active detection rules to MITRE ATT&CK techniques. Identify technique categories with no coverage and prioritize new rule development accordingly. The Cavern Manticore and 0ktapus campaigns both relied on techniques in the Initial Access and Command and Control categories that many organizations have limited detection coverage for.
- Assign ownership for log source health monitoring: Log source failures, parsing errors, and ingestion gaps create silent blind spots. Assign explicit ownership for monitoring log pipeline health, with alerting configured for sources that stop producing events or drop below expected volume thresholds.
Common Implementation Pitfalls
Treating SIEM Storage Constraints as a Detection Strategy Decision
Organizations frequently disable high-volume log sources such as DNS query logs, full HTTP request URIs, or process execution events to manage SIEM storage costs. This decision is made by operations teams with budget pressure and is presented as a practical tradeoff. In practice, it removes telemetry that consistently surfaces early-stage attacker activity and shifts detection capability toward network-layer indicators, which sophisticated actors actively suppress.
The resolution is to separate storage cost decisions from detection coverage decisions. Use tiered storage to retain high-volume sources at lower cost with longer query latency. Compress and archive DNS and proxy logs to cold storage rather than disabling ingestion entirely. The storage cost of 90 days of DNS query logs for a 1,000-seat organization is measurable. The cost of a compromise that goes undetected for 60 days because DNS telemetry was disabled is substantially higher.
Writing Detection Rules Without Analyst Feedback Loops
Detection engineering teams that operate without structured feedback from the analysts who triage alerts consistently produce rule sets with growing false positive rates and declining analyst trust. When analysts regularly dismiss alert types without documenting their reasoning, detection engineering loses the signal needed to improve rules. When alerts are routinely closed without documentation, management has no visibility into whether the detection program is improving or degrading.
Implementing a lightweight structured triage workflow, where analysts categorize each alert as true positive, false positive, or inconclusive with a one-line rationale, generates the feedback data that detection engineers need. This does not require a commercial platform. A shared spreadsheet with standardized disposition categories is sufficient to identify the rules generating the most noise and to track improvement over time.
Conflating Log Availability With Detection Coverage
A common assumption in log analysis programs is that having a log source connected to the SIEM means that source is covered for detection purposes. In practice, log availability and detection coverage are separate properties. A log source may be ingesting correctly while contributing zero detection value because no rules reference its fields, because critical fields are not indexed, or because the events it generates are not correlated with any other source.
Performing a coverage audit that maps each log source to the specific detection rules that reference it, and identifying sources that have no associated rules, regularly surfaces significant gaps. Authentication logs from cloud-hosted SaaS applications, network device syslogs, and cloud provider audit logs are consistently underrepresented in detection rule coverage despite being available in the SIEM.
Static Thresholds in Dynamic Environments
Detection rules built with static thresholds degrade over time as the environment changes. A threshold for authentication failures that was calibrated during a 500-person baseline becomes inaccurate after an acquisition that doubles headcount. A DNS query volume threshold calibrated before a cloud migration becomes inaccurate after workloads move to cloud infrastructure that generates fundamentally different query patterns.
Static thresholds should be reviewed on a scheduled cadence, with mandatory review triggered by significant infrastructure changes. For high-priority detection rules, consider implementing dynamic thresholding using rolling baseline calculations rather than fixed values. Most modern SIEM platforms support this natively, though the feature is frequently left unconfigured.
Ignoring Internal Traffic in Log Analysis Scope
Log analysis programs frequently focus on perimeter traffic and external-facing systems while applying minimal detection coverage to internal network logs, east-west traffic between segments, and administrative access logs for internal systems. This mirrors exactly the blind spot that threat actors exploit during the lateral movement phase. The Cavern Manticore framework specifically targets east-west movement paths that traverse internal network segments with limited visibility.
Extending log analysis coverage to include internal DNS, internal authentication (not just domain authentication but application-level authentication within internal systems), and process execution on servers in addition to workstations significantly reduces the dwell time advantage that attackers currently exploit in most enterprise environments.
Connecting Log Analysis to Incident Response Timelines
The value of effective log analysis is ultimately measured in time: how quickly can a team move from first alert to confident scope determination, and how much historical log context is available to reconstruct attacker activity prior to initial detection. Compromise assessment findings published in mid-2026 consistently show that teams with mature log analysis capabilities scope incidents in hours rather than days and identify the initial access vector in the majority of cases. Teams without mature log analysis capabilities frequently cannot determine how the attacker entered, which makes remediation incomplete and recurrence likely.
Building log analysis capability is an incremental process. Starting with authentication logs, implementing cross-source correlation for the highest-priority credential attack patterns, establishing baselines, and reviewing detection rule performance quarterly creates a foundation that improves with each iteration. The threat actors currently active, whether operating modular C2 frameworks, running credential-based campaigns at scale, or exploiting newly discovered vulnerabilities, all leave traces in logs. The question is whether the analysis infrastructure is in place to surface those traces before the damage compounds.
