The Attack Pattern That Started This Conversation
In mid-2025, the WSzero DDoS family reached its fourth version, spreading through 21 distinct vulnerabilities and evolving its behavior between campaigns fast enough to outpace static signature updates. Around the same time, the 'Popa' botnet was traced back to a publicly-traded Israeli firm, demonstrating that malicious network infrastructure increasingly originates from legitimate-looking corporate environments. These campaigns share a common trait: their traffic patterns diverged from baseline behavior in measurable ways before any signature was ever written for them.
Healthcare networks are experiencing a parallel surge. Cybercriminals are targeting healthcare businesses at increasing rates, and a significant reason those attacks succeed is that the initial reconnaissance and lateral movement blend into high-volume clinical data traffic. Rule-based detection systems often generate so many alerts in that environment that security teams tune aggressively just to survive the noise, and in doing so, they create space for anomalies to persist undetected.
Machine learning-based anomaly detection addresses this specific failure mode. This article walks through how these systems work at the telemetry level, where they outperform traditional approaches, and how to deploy them in a way that produces actionable signals rather than a different variety of noise.
Why Rule-Based Detection Reaches Its Ceiling
Signature-based and rule-based intrusion detection operates on a known-bad model. An analyst observes a behavior, encodes it as a rule, and the system matches future traffic against that rule. This works well when attackers reuse infrastructure and techniques without modification. It breaks down when attackers iterate quickly, operate from unexpected source ranges, or deliberately stay below detection thresholds.
The WSzero family is a clear example. By version four, its propagation mechanisms had changed enough that rules written for earlier versions required substantial revision. During the gap between attack evolution and rule update, networks were exposed. A system trained on behavioral baselines would have flagged the scanning activity and exploit attempts based on deviation from normal traffic patterns, regardless of whether the specific CVEs in use matched any rule.
Botnets like Popa present a different challenge. When malicious activity originates from infrastructure registered to a legitimate corporate entity, IP reputation feeds often clear the traffic. The behavior itself, in terms of connection frequency, session duration, command-and-control beacon intervals, and payload structure, remains anomalous relative to what legitimate traffic from that ASN typically looks like. Machine learning models that learn what normal looks like for a given source range can catch this even when the reputation score says nothing is wrong.
What Behavioral Baselines Actually Capture
Before deploying any ML-based detection system, security teams need a clear picture of what data feeds into the model and what constitutes a meaningful baseline. The most useful behavioral signals for network anomaly detection fall into several categories.
Flow-Level Features
NetFlow, IPFIX, and sFlow records give you source and destination IP and port, protocol, byte and packet counts, flow duration, and flags. These are the foundational inputs for most network anomaly detection models. A machine learning model trained on weeks of this data learns the typical conversation patterns for each subnet, the expected ratio of inbound to outbound data, and the normal distribution of connection durations.
When a compromised host begins staging data for exfiltration, the outbound byte count to an unfamiliar external IP spikes in a way that diverges from the baseline even if the destination IP has no reputation score. When a lateral movement campaign begins, the internal east-west traffic volume changes in ways that rule-based systems rarely capture because those rules typically focus on north-south traffic at the perimeter.
DNS Query Behavior
DNS telemetry is underutilized in many environments. A machine learning model watching DNS query frequency, query type distribution, domain entropy scores, and time-to-live patterns can identify C2 communication, domain generation algorithm (DGA) activity, and DNS tunneling with high accuracy. The Popa botnet's traffic would have exhibited DNS query patterns inconsistent with the corporate environment it was attributed to, providing a detection opportunity independent of reputation data.
Authentication Event Patterns
Authentication log telemetry fed into an anomaly detection model captures credential abuse campaigns that traditional systems miss. Rather than thresholding on failed login counts, a behavioral model learns the typical authentication cadence for each account, the source IP ranges from which authentication normally succeeds, and the time-of-day distribution of access events. A device code phishing attack, such as the Microsoft website vector recently observed, involves legitimate authentication events from unexpected devices and geographic locations. The behavioral deviation is detectable even though no individual event triggers a traditional rule.
Protocol Behavior and Payload Statistics
For environments with full packet capture capability, protocol-level behavioral features add significant detection value. HTTP request header ordering, TLS fingerprint distributions, and inter-packet timing are all dimensions along which malicious traffic often deviates from baseline, particularly for encrypted C2 channels that tunnel through allowed protocols to evade signature-based detection.
Choosing the Right Model Architecture for Your Environment
The choice of machine learning approach depends on what you have, what you want to detect, and how much labeled data you can produce. Most enterprise environments start without reliable labeled datasets of malicious traffic, which means unsupervised or semi-supervised approaches are the practical starting point.
Autoencoders for Reconstruction-Based Detection
An autoencoder is a neural network trained to compress input data into a lower-dimensional representation and then reconstruct it. When trained only on normal traffic, it learns to reconstruct normal patterns accurately. Anomalous traffic, which the model has never seen during training, produces higher reconstruction error. Thresholding on reconstruction error gives you an anomaly score for each flow or session.
Autoencoders work well for detecting novel attack patterns, including new botnet families and previously unseen scanning behavior. The tradeoff is that reconstruction error alone does not tell you what kind of anomaly you are looking at. Pairing autoencoder outputs with clustering techniques helps analysts understand whether a flagged session resembles known attack categories or represents something genuinely new.
Isolation Forests for High-Dimensional Flow Data
Isolation forests work by randomly partitioning the feature space and measuring how quickly individual samples are isolated. Anomalous samples, which are rare and unusual, require fewer partitions to isolate. This approach scales well to high-dimensional NetFlow data and runs efficiently enough for near-real-time scoring of large flow volumes.
In practice, isolation forests perform well when the feature engineering is done carefully. Raw flow data needs to be aggregated at meaningful time windows, and features like flow duration and byte counts need to be log-transformed to handle the heavy-tailed distributions typical of real network traffic. Teams that deploy isolation forests directly on un-normalized data tend to see high false positive rates on legitimate bulk data transfers.
Recurrent Models for Sequential Behavior
Long short-term memory (LSTM) networks and transformer-based sequence models capture temporal dependencies that other approaches miss. A compromised host beaconing to a C2 server on a regular interval exhibits a pattern across time that is anomalous relative to the irregular timing of legitimate browsing or application traffic. Sequence models learn these temporal patterns explicitly and flag deviations from expected cadence.
The deployment complexity for recurrent models is higher than for isolation forests or autoencoders. You need a stream processing infrastructure that maintains state across time windows, which introduces latency and infrastructure cost. For environments where detecting periodic C2 communication is a priority, the detection capability justifies the investment. For others, the simpler approaches provide sufficient coverage with lower operational burden.
Graph-Based Models for East-West Traffic
Network traffic is inherently relational, and graph neural networks (GNNs) represent it that way. Hosts become nodes, connections become edges, and the model learns what the normal graph structure of your network looks like. Lateral movement campaigns, which involve connections between hosts that rarely or never communicate under normal conditions, appear as structural anomalies in the graph representation.
Graph-based detection is particularly valuable for detecting the early stages of ransomware deployment, where an attacker moves between systems to map the environment and identify high-value targets before detonating the payload. The connection patterns involved in that reconnaissance are detectable as graph anomalies well before any encryption activity begins.
Integrating ML Anomaly Detection Into Your SOC Workflow
A machine learning model that produces anomaly scores is not a finished detection system. The operational integration determines whether those scores become actionable alerts or get ignored alongside everything else the team cannot process.
Scoring Pipelines and Alert Routing
The output of an anomaly detection model should flow into your SIEM or SOAR platform as a structured event with the anomaly score, the features that contributed most to the score, and the baseline context that makes the deviation meaningful. Analysts need to see not just that something is anomalous, but what normal looked like and in what specific way the flagged event departed from it.
Routing high-confidence anomalies directly to investigation queues and routing medium-confidence anomalies to correlation rules that look for supporting evidence across other telemetry sources reduces the manual triage burden. A single anomalous flow with no supporting context warrants a low-priority review. The same anomalous flow combined with a DNS query to a high-entropy domain and an authentication event from an unusual source warrants immediate escalation.
Feedback Loops and Model Maintenance
Anomaly detection models degrade as network behavior changes. New applications, infrastructure migrations, and changes to user behavior all shift the baseline that the model learned. Healthcare environments are particularly dynamic: new medical devices, telemedicine platforms, and clinical applications regularly introduce traffic patterns that initially appear anomalous simply because they are new.
Build a feedback loop into your deployment from the start. Analysts who investigate flagged events should have a mechanism to classify the outcome: confirmed malicious, confirmed benign, or uncertain. Confirmed benign events with clear business justification become labeled normal data that can be used to retrain or fine-tune the model. Confirmed malicious events become labeled attack data for supervised model evaluation. This continuous feedback cycle is what separates a detection capability that improves over time from one that stagnates and gradually becomes irrelevant.
Handling Encrypted Traffic
A growing proportion of attack traffic travels over TLS, including C2 channels for many modern malware families. Decrypting all traffic for inspection introduces privacy concerns and certificate management complexity. ML-based approaches offer a practical alternative: behavioral features extracted from TLS sessions without decryption, including certificate characteristics, cipher suite selections, handshake timing, and record length distributions, carry enough information to distinguish malicious from legitimate encrypted sessions with reasonable accuracy.
The JA3 and JA3S fingerprinting approaches provide a starting point, but they are well-known enough that sophisticated adversaries evade them through TLS library manipulation. More robust approaches extract a broader feature set from the TLS metadata and train models on those features rather than relying on single-value fingerprints. This makes evasion significantly harder because the adversary must manipulate many independent features simultaneously to avoid detection.
Deployment Scenarios Worth Prioritizing
Healthcare Network Segmentation Monitoring
Given the current surge in attacks on healthcare organizations, monitoring traffic across clinical network segments deserves specific attention. Clinical devices often have constrained update cycles and run legacy software, making them attractive pivot points once an attacker establishes a foothold elsewhere on the network. An anomaly detection model watching east-west traffic between the clinical device segment and the broader network can detect lateral movement attempts that would otherwise be invisible if security monitoring focuses primarily on the perimeter.
In a practical deployment, you would establish baseline flow patterns for each clinical subnet over a four to six week observation period, then deploy scoring against new flows. Deviations in connection destinations, particularly flows from clinical subnets to administrative or research infrastructure that were rare or absent during the baseline period, would generate alerts for investigation.
C2 Beacon Detection Across Encrypted Channels
The regularity of C2 beacon intervals is one of the most reliable behavioral signals available for post-compromise detection. Even when malware authors introduce jitter to randomize the interval, the resulting inter-connection timing distribution differs from legitimate application traffic in ways that sequence models can detect. Deploying an LSTM trained on DNS and flow timing data to watch for regular communication patterns to external IPs that lack business context catches active C2 communication that signature-based detection misses entirely.
Firmware Attack Surface Monitoring
Recent research into U-Boot vulnerabilities highlights a class of attacks that operate below the operating system level and persist across reimaging. While ML-based network anomaly detection does not directly address firmware compromise, the network behavior of a firmware-implanted device often deviates from baseline in ways that are detectable. Unusual outbound connections during boot sequences, communication with IPs outside expected ranges, and changes in protocol behavior for management interfaces are all signals that anomaly detection can surface for investigation.
Measuring Whether the Detection Is Actually Working
Detection efficacy requires measurement. Teams that deploy anomaly detection without establishing evaluation metrics have no reliable way to distinguish a working system from one that is silently failing to flag real attacks.
The most useful metrics for operational anomaly detection are detection rate against known attack replays, false positive rate during normal business cycles, and time-to-alert for simulated attack scenarios. Running periodic red team exercises against the deployed model and measuring how consistently it flags the activity gives you a ground truth calibration point. Tracking false positive rates across business cycles identifies periods where the model struggles, such as month-end data processing jobs that temporarily shift baseline traffic patterns, and informs retraining schedules.
Tracking mean time to detect for incidents that do occur and comparing it against historical baselines gives you a direct measure of operational impact. A reduction in mean time to detect from days to hours represents a concrete security improvement that justifies the investment in model development and maintenance.
Where This Goes From Here
The current threat landscape rewards defenders who can operate faster than attackers can iterate their techniques. The WSzero botnet's progression through four distinct versions in a short timeframe and the Popa botnet's use of legitimate corporate infrastructure both illustrate that signature-based detection operates on a lag that attackers actively exploit.
Machine learning-based anomaly detection closes that lag by shifting the detection logic from known-bad matching to deviation-from-normal scoring. The practical implementation requires careful feature engineering, appropriate model selection for the detection goal, thoughtful integration into SOC workflows, and continuous feedback mechanisms to keep the model calibrated as your environment evolves.
Teams that invest in this capability now will find themselves better positioned as attack techniques continue to evolve, because behavioral detection does not need to know what the next attack looks like in advance. It only needs to recognize that something has changed.
