When the Baseline Becomes the Blind Spot
In Q2 2026, ransomware operators continued refining their pre-encryption dwell time, often spending three to six weeks inside target environments before triggering payload delivery. The State of Ransomware Q2 2026 report documented cases where lateral movement traffic blended into normal administrative patterns precisely because the anomaly detection models in place had been trained on data that included earlier, quieter phases of the same intrusion. The model had learned the attacker's presence as normal.
This is the central tension in machine learning-based network anomaly detection: models learn from historical traffic, and if attackers have already shaped that history, the model inherits the deception. Boston Scientific disclosed that a cyberattack disrupted operations globally, and reporting suggested persistent access had existed before the disruptive phase triggered visibility. The disruption was the first signal most monitoring systems caught. Everything before it looked routine.
Cybersecurity professionals applying machine learning to network anomaly detection need to understand both the genuine strengths of these approaches and the specific failure modes that sophisticated threat actors have learned to exploit. This article covers how to build, tune, and operate ML-based anomaly detection with enough operational depth to catch what signature-based systems miss, without creating a model that teaches itself to ignore the threat.
Why Machine Learning Fits Network Anomaly Detection Better Than Signatures Alone
Signature-based detection requires someone to have seen the attack before. It works well against commodity malware, known exploit kits, and previously documented command-and-control infrastructure. It works poorly against novel tradecraft, custom tooling, and attackers who cycle their indicators faster than threat feeds update.
Dark Caracal's recent expansion of its malware arsenal illustrates this directly. The group has added custom implants that do not match existing detection signatures, and the initial phases of their campaigns involve reconnaissance and light beaconing that generates no signature alerts. Machine learning models trained on behavioral baselines can surface these early-phase signals because they focus on statistical deviation rather than pattern matching against a known bad list.
The watering hole campaigns pushing ScanBox keyloggers demonstrate a complementary problem. The initial compromise happens at a third-party site, and the malicious activity inside the victim network starts with a browser-based keylogger exfiltrating credentials over encrypted channels. There is no malware binary on disk to detect. The only signal available is behavioral: unusual outbound connections, slight increases in DNS query volume to newly registered domains, or timing anomalies in HTTPS sessions. These are exactly the kinds of signals ML anomaly detection is built to surface.
The Architecture Choices That Determine What You Can Actually Detect
Machine learning anomaly detection for networks is not a single tool. It is a collection of models, each trained on different data sources, each targeting different threat behaviors. Getting the architecture right before you start tuning matters more than the choice of algorithm.
Telemetry Sources and What Each One Exposes
NetFlow and IPFIX records give you connection metadata: source and destination IP, port, protocol, byte counts, packet counts, and session duration. They miss payload content but cover the full scope of network connections, including encrypted traffic where DPI provides no signal. ML models trained on flow data are effective at detecting port scanning, beaconing, data staging, and lateral movement patterns.
DNS logs surface domain generation algorithm traffic, lookups to newly registered domains, high-frequency subdomain queries used in DNS tunneling, and changes in resolver behavior that indicate compromised hosts. DNS is underused as an anomaly detection source despite being one of the highest-signal telemetry streams in most environments.
Endpoint telemetry through EDR platforms provides process execution chains, network connection initiation by process, and file system activity. Combining endpoint telemetry with network flow data allows you to attribute network anomalies to specific processes rather than just IP addresses, which dramatically accelerates triage.
Authentication logs from Active Directory, SSO platforms, and VPN concentrators expose credential abuse, unusual access times, and service account behavior that deviates from established patterns. The DOUBLECUP campaign, which delivered PNG-embedded payloads, relied on compromised credentials to move laterally after initial access. Authentication log anomalies were the earliest available signal in post-incident analysis.
Choosing the Right Model Type for Each Signal
Isolation Forest algorithms work well for high-dimensional network flow data where anomalies are rare and the feature space is complex. They isolate anomalous points by randomly partitioning the feature space and measuring how few partitions are needed to isolate each point. Traffic from a workstation suddenly generating 10x its normal outbound byte count to an unfamiliar ASN will isolate quickly.
Autoencoders are effective for sequence-based anomaly detection, particularly for identifying command-and-control beaconing. The model learns to reconstruct normal traffic patterns, and connections that cannot be reconstructed accurately produce high reconstruction error. Periodic beaconing to a C2 server produces a distinctive timing pattern that autoencoders surface reliably even when the destination IP is not on any blocklist.
LSTM networks handle temporal dependencies in traffic data, making them useful for detecting slow-moving threats like the week-over-week data exfiltration patterns common in espionage campaigns. The Android malware hijacking car head unit update systems used gradual exfiltration patterns that would evade threshold-based detection but produce detectable anomalies in LSTM-based models trained on normal update traffic timing and volume.
Graph-based anomaly detection models treat the network as a relationship graph and identify devices whose communication patterns suddenly change: new peers, new protocols, new timing. Lateral movement almost always produces graph anomalies because attackers probe new hosts, establish connections between previously unconnected devices, and generate authentication events between accounts that have no history of interacting.
Building a Baseline That Does Not Encode the Intrusion
The baseline period is where most ML anomaly detection deployments make their most consequential decision badly. A 30-day baseline sounds reasonable. In practice, if an attacker has been present for three weeks of that 30-day window, the model learns their traffic as normal. When they escalate, the deviation from baseline is smaller than it should be.
Several practices reduce this risk. First, cross-validate your baseline against threat intelligence. Before you finalize the training period, run your telemetry from that period against current threat feeds and historical IOC data. Flag any devices or connections that appear in threat intelligence databases during the baseline window and exclude or weight them appropriately. This does not catch novel intrusions, but it catches cases where known infrastructure was present during training.
Second, use rolling baselines with drift detection rather than static models. A static model trained once will degrade as the environment evolves through system additions, software updates, and legitimate business changes. Rolling baselines update continuously, but they require drift detection mechanisms that distinguish legitimate environmental change from attacker-introduced change. When your model updates significantly in a short window, that is worth investigating before the new behavior becomes the new normal.
Third, stratify your baseline by device class, user role, and network segment. A model trained on aggregate network traffic will average out the behavior of servers, workstations, OT devices, and IoT endpoints into a single baseline that accurately represents none of them. Stratified models learn what normal looks like for each device class separately, which means anomalies specific to a single class surface clearly rather than disappearing into aggregate variance.
Feature Engineering for Network Traffic Data
Raw network flow records are not directly useful for most ML models. Feature engineering transforms raw telemetry into representations that models can learn from effectively. The features you build determine what the model can and cannot detect.
Time-based features capture behavioral patterns that raw counts miss. Instead of total bytes per hour, compute the inter-arrival time variance of connections from a given host to a given destination. Beaconing produces low variance. Human-driven browsing produces high variance. This single feature distinguishes automated C2 communication from legitimate browsing even when the destination is unknown.
Entropy-based features applied to DNS query subdomains surface DGA traffic. Domain generation algorithms produce subdomains with high character entropy because they are pseudo-random. Legitimate subdomains tend toward low entropy because they are human-readable. Computing the entropy of subdomain strings and tracking per-host entropy distributions gives your model a feature that directly targets DGA-based C2.
Connection ratio features capture reconnaissance activity. The ratio of unique destination IPs to total connections for a given source host spikes during scanning. A workstation that normally connects to 20 distinct IPs per day suddenly reaching 200 in an hour produces a ratio anomaly that is visible even if none of the destination IPs are on any blocklist.
Protocol-specific behavioral features add depth for specific threat types. For HTTPS, session duration distribution, certificate age at first observation, and the ratio of small to large responses are features that distinguish legitimate browsing from exfiltration. Thousands of hacked WordPress sites running the StopAndProtect operation used HTTPS callbacks that differed from legitimate WordPress traffic in exactly these feature dimensions: shorter sessions, higher small-response ratios, certificates registered within the previous 30 days.
Operational Tuning: Moving From Alert Volume to Alert Value
The fastest way to kill an ML anomaly detection program is to leave it untuned. Models trained on network data generate false positives at rates that overwhelm SOC capacity within weeks if no suppression and tuning workflow exists. Security teams stop investigating alerts, and the model becomes decorative.
Threshold Calibration by Risk Tier
Not all anomalies carry equal risk. A workstation browsing to an unfamiliar CDN edge node is a lower-risk anomaly than a domain controller establishing an outbound connection to a residential IP in an unusual geography. Calibrate alert thresholds separately for high-value asset classes and apply stricter sensitivity to assets whose compromise would have the highest impact.
Define your risk tiers explicitly: crown jewel servers, privileged workstations, standard user endpoints, OT/IoT devices. Set anomaly score thresholds for each tier and route alerts to different response queues based on tier. This ensures your analysts see the highest-priority alerts first and that tuning decisions for one tier do not inadvertently suppress alerts in another.
Contextual Enrichment Before Alert Review
Every alert that reaches an analyst should arrive pre-enriched. At minimum, the analyst should see the historical communication pattern for the source device, current threat intelligence matches for destination IPs and domains, user account associations for the source device, recent authentication events for associated accounts, and the specific feature values that drove the anomaly score. Without this context, analysts spend most of their investigation time gathering information rather than making decisions.
Automation pipelines that perform this enrichment at alert creation time reduce mean time to triage significantly. Integrate your anomaly detection output with your SOAR platform so that enrichment queries run automatically against threat intelligence feeds, your CMDB, your authentication log store, and your DNS history database before the alert appears in the analyst queue.
Feedback Loops That Actually Improve the Model
Analyst verdicts on alerts are your most valuable training signal. When an analyst marks an alert as a true positive or false positive, that decision should feed back into the model's training pipeline. This closes the loop between human expertise and model behavior, allowing the model to learn what your specific environment's analysts consider significant.
Implement structured feedback capture in your alert review workflow. A simple true positive, false positive, and uncertain classification is enough to start. Over time, build in sub-classifications: false positive due to scheduled maintenance, false positive due to new application deployment, true positive confirmed C2 activity. Granular feedback produces more targeted model improvements.
Mapping Detection Capabilities to Current Threat Patterns
The exploits and vulnerabilities observed in Q2 2026 show a consistent pattern: initial access via public-facing application vulnerabilities, followed by living-off-the-land techniques for lateral movement, followed by slow exfiltration before the destructive or disruptive payload deploys. This attack chain has specific network signatures at each phase that ML anomaly detection can target if the models are built for it.
Initial access via application exploits generates connection anomalies at the web tier: unusual HTTP method sequences, unexpected server-side process spawning reflected in EDR telemetry, and new outbound connections from application servers that normally only accept inbound traffic. Train your models to flag web server hosts that initiate outbound connections to IPs with no prior relationship to your environment.
Living-off-the-land lateral movement uses tools like WMI, PsExec, and PowerShell remoting over SMB and WinRM. These protocols are legitimate in most Windows environments, but their usage patterns are constrained. Workstations authenticating to other workstations over SMB is unusual. A single service account authenticating to 40 different hosts in a two-hour window is unusual. Graph anomaly detection captures these patterns by comparing current authentication graph structure to historical norms.
Pre-exfiltration staging produces storage and file access patterns that appear in EDR and file server logs before the data leaves the network. When an account accesses a volume of files far outside its normal range, that is a behavioral anomaly that precedes the network-visible exfiltration event. Catching it at the file access stage gives you more response time than waiting for the exfiltration traffic to appear.
Deployment Considerations for Production Environments
Running ML anomaly detection in production requires infrastructure decisions that affect both detection quality and operational cost. The following considerations apply directly to enterprise deployments.
Compute placement matters for latency-sensitive detection. Models that need to produce alerts within minutes of anomalous traffic require inference infrastructure close to the telemetry collection point. Running inference in a centralized cloud environment when telemetry has to travel through a WAN link before processing can introduce enough latency to miss fast-moving threats like scanning activity that completes in under five minutes.
Model versioning and rollback capability is operationally essential. When a model update increases false positive rates or misses a class of threat that the previous version caught, you need to roll back quickly. Treat model versions the same way you treat software versions: maintain a registry, test new versions in a shadow environment against live traffic before promotion, and keep at least two previous versions available for rollback.
Data retention for model retraining should be scoped to your threat dwell time risk tolerance. If you want to detect threats that have been present for up to 90 days, your training data pipeline needs to store at least 180 days of labeled telemetry. Shorter retention windows reduce your ability to train models that catch slow-moving threats. Budget storage accordingly.
Mexico's Cybersecurity Plan 2025-2030 emphasizes building national detection infrastructure with ML capabilities at its core, specifically citing the need for organizations to operationalize anomaly detection across distributed environments. The infrastructure challenges they are addressing at national scale mirror the challenges enterprises face across geographically distributed networks: consistent telemetry collection, model governance across sites, and alert correlation across independently managed segments.
Integrating Anomaly Detection Into the Broader Security Operation
ML anomaly detection produces its highest value when it feeds into a broader security operation rather than operating as a standalone system. The model's job is to surface candidates for investigation. The analysts' job is to confirm, contextualize, and respond.
Threat hunting teams should use anomaly scores as a prioritization layer for their hunts. Rather than hunting against the full traffic corpus, start with the highest-scoring anomalies from the previous week that were not triaged as alerts because they fell below the alert threshold. These sub-threshold anomalies often represent exactly the kind of slow, low-signal activity that precedes major incidents.
Incident response playbooks should include anomaly score review as a standard step when investigating any confirmed intrusion. Pull the anomaly timeline for affected hosts going back 90 days. In most cases, you will find earlier anomalies that were not triaged when they occurred. This retrospective analysis informs both the scope of the current incident and the tuning priorities for the detection model going forward.
Tabletop exercises should include scenarios where the anomaly detection model is the first alert source. Walk your SOC team through a scenario where a model alert is the only initial signal, with no corresponding endpoint alert or threat intelligence match. This forces analysts to build investigation competency around behavioral signals rather than IOC-driven workflows, which is where ML anomaly detection provides capabilities that nothing else does.
Where to Focus Implementation Effort First
For organizations beginning to build or mature their ML anomaly detection capabilities, the following sequence produces the fastest operational value.
- Start with DNS anomaly detection. DNS telemetry is available in virtually every environment, DNS anomalies are high-signal indicators of C2 activity, and DNS-based ML models are among the most mature and well-documented implementations available. Deploy a model targeting DGA detection and DNS tunneling within your first 30 days.
- Add NetFlow-based lateral movement detection next. Focus on authentication protocol anomalies: SMB, WinRM, and Kerberos traffic between hosts with no prior relationship. This directly targets the most common post-exploitation movement pattern in enterprise environments.
- Build the enrichment and feedback infrastructure before you deploy additional models. The value of each additional model depends on analysts being able to triage its alerts effectively. Get the enrichment pipeline and feedback loop working before you increase alert volume.
- Layer in exfiltration detection as your third major use case, using session-level HTTPS behavioral features and DNS data volume anomalies. By this point, your baseline quality and analyst workflow should be mature enough to handle the more complex tuning that exfiltration detection requires.
Network anomaly detection with machine learning is not a replacement for other security controls. It is a detection layer that catches what signature-based systems miss: novel malware like the tools Dark Caracal continues adding to its arsenal, behavioral patterns that precede payloads, and the quiet reconnaissance that happens long before an attack becomes visible enough to trigger conventional alerts. Building it correctly, tuning it continuously, and integrating it into your SOC workflow turns it from an expensive experiment into the detection capability that catches intrusions while there is still time to contain them.
