The Attack Pattern That Started With Normal Traffic
In early 2024, a modified version of the CIA's leaked Hive attack toolkit began circulating in underground markets. Security researchers tracking the campaign noted something that made traditional detection nearly useless: the actors operating the modified Hive variant deliberately mimicked legitimate network behavior during initial access. Connections came in at low volume, timing intervals mimicked browser-based user sessions, and the command-and-control traffic blended into standard HTTPS flows. Signature-based detection saw nothing worth flagging. The traffic was, by every rule-based metric, normal.
This is where machine learning anomaly detection earns its place in a mature security architecture. When an attacker deliberately engineers traffic to pass signature inspection, the only detection surface left is behavioral deviation from an established baseline. ML-based systems do not ask whether a packet matches a known bad pattern. They ask whether behavior at this moment diverges from what this host, this user, or this network segment has historically done.
The distinction matters operationally. Cybersecurity professionals and IT administrators running detection stacks built primarily around static signatures are exposed to an entire class of threats that anomaly detection was specifically designed to catch.
What Network Anomaly Detection Actually Measures
Before getting into implementation, it helps to be precise about what ML-based anomaly detection operates on. The inputs generally fall into four categories.
- Flow metadata: Source and destination IPs, ports, protocols, bytes transferred, packet counts, connection duration, and timing intervals. This is the foundation. Flow data is high-volume and high-fidelity without requiring full packet capture.
- Behavioral sequences: The order and pattern of actions taken by a host or user over time. A workstation that authenticates to one server per day then suddenly authenticates to forty in two hours has changed its behavioral sequence in a way that flow metadata alone might not surface.
- Protocol-level features: DNS query rates, TLS certificate characteristics, HTTP header ordering, and similar signals that reveal how a protocol is being used, not just that it is being used.
- Graph-based relationships: Which hosts communicate with which other hosts, and how that communication topology evolves over days and weeks. Lateral movement leaves marks in the communication graph even when individual connections look benign.
The Novo Nordisk breach that exposed software development pipeline risks is instructive here. Pipeline infrastructure by definition communicates with a wide range of internal and external systems: source control, artifact repositories, build servers, deployment targets. Signature-based tools struggle to monitor this traffic because the volume and variety are high and known-bad patterns are sparse. An ML model trained on the specific communication patterns of a CI/CD pipeline can detect when a build server begins pulling from an external repository it has never contacted, or when deployment credentials are used from an IP that sits outside the normal deployment automation range.
Baseline Construction: The Work Most Teams Skip
The quality of anomaly detection is directly proportional to the quality of the baseline. A model trained on two weeks of Christmas holiday traffic will fire constantly in January. A model trained during a known-clean period that spans normal business cycles, maintenance windows, patch deployments, and end-of-quarter surges will produce far fewer false positives.
Baseline construction requires deliberate planning. The following steps apply whether you are deploying a commercial ML-based NDR platform or building your own detection pipeline on top of flow data.
- Segment your baselines by network zone. OT networks, data center east-west traffic, VDI environments, and developer workstations each have distinct behavioral signatures. Mixing them into a single model degrades detection quality for all segments. Given the current threat landscape around legacy OT systems, where attackers are increasingly targeting operational technology with modern techniques, having a clean OT-specific baseline is operationally critical.
- Log baseline context. When you capture your training data, tag it with contextual metadata: time of day, day of week, whether a patch cycle was running, whether a major application deployment occurred. This context allows the model to learn cyclical patterns rather than treating Friday afternoon traffic as anomalous because it differs from Monday morning.
- Include failure modes in your baseline review. Before training, review the baseline data for any known incidents, pentest activity, or red team engagements. Training a model on data that includes undetected lateral movement will teach the model that lateral movement is normal.
- Plan for baseline drift. Networks change. New services launch, old ones retire, user populations shift. Build a process for incremental baseline updates that does not silently absorb new attack patterns into the normal profile. Most commercial platforms handle this with supervised feedback loops where analysts label alerts as true or false positives, feeding corrections back into the model.
Algorithm Selection for Real Network Environments
The choice of ML algorithm matters, but operations teams tend to overweight this decision relative to the quality of feature engineering and baseline data. That said, understanding which algorithm classes apply to which detection scenarios helps when evaluating platforms or designing custom detection pipelines.
Unsupervised Methods
Unsupervised methods learn what normal looks like without labeled training data. They are the starting point for most network anomaly detection deployments because labeled datasets of network intrusions are rare and often stale by the time they reach production environments.
Isolation Forest works by randomly partitioning the feature space and measuring how quickly an observation gets isolated. Anomalies, being rare and different, get isolated in fewer partitions. This makes it computationally efficient for high-volume flow data and interpretable enough that analysts can understand why a specific flow scored as anomalous.
Autoencoders are neural networks trained to compress and reconstruct input data. A well-trained autoencoder reconstructs normal traffic accurately and struggles to reconstruct anomalous traffic. The reconstruction error becomes the anomaly score. Autoencoders handle complex, high-dimensional feature spaces better than Isolation Forest but require more computational resources and are harder to interpret.
DBSCAN and clustering methods group similar observations together and flag observations that do not fit into any cluster. Useful for identifying hosts with unusual communication patterns, but sensitive to parameter tuning and can produce unstable results as network topology evolves.
Semi-Supervised and Supervised Methods
Once your team has accumulated labeled alert data from analyst feedback, semi-supervised methods become viable. These train primarily on normal data but incorporate known-bad examples to sharpen detection of specific threat categories.
For teams with mature detection programs, gradient boosting classifiers like XGBoost and LightGBM perform well on tabular network flow features. They handle class imbalance reasonably, train quickly, and produce feature importance scores that help analysts understand which behavioral signals are driving detections. The challenge is that they require labeled data, which means investing in alert review and feedback discipline over time.
Feature Engineering That Makes the Difference
Raw flow data fed directly into an ML model produces poor results. Feature engineering, transforming raw fields into derived signals that capture behavioral meaning, is where detection quality is actually determined.
Consider a scenario involving a crypto clipboard hijacker, similar to the campaign documented recently where fake reputation signals were used to distribute malware that silently replaced cryptocurrency addresses in the clipboard. The initial C2 communication might look like standard HTTPS traffic. But the timing behavior often reveals the implant: beaconing at regular intervals with low jitter. A raw flow record shows a connection. A computed feature capturing the coefficient of variation in inter-connection timing across a sliding window exposes the beaconing pattern.
Useful derived features for network anomaly detection include:
- Bytes per packet ratio: Malware often generates flows with characteristic packet sizes. C2 beacons tend toward small, uniform packets.
- Connection fan-out: The number of unique destination IPs contacted by a source within a time window. Scanners, worms, and lateral movement tools drive this metric up sharply.
- Protocol-to-port mismatch frequency: The rate at which traffic on a given port exhibits protocol characteristics inconsistent with the expected protocol. Malware tunneling over port 443 while using non-TLS protocols generates this signal.
- DNS query entropy: Domain generation algorithms produce high-entropy domain names. Computing the entropy of DNS query strings per client per time window is a reliable DGA detector.
- New destination ratio: The proportion of connections in the current window going to destinations not seen in the historical baseline for this source. A host that suddenly begins talking to many new external IPs warrants investigation.
- Authentication event acceleration: The rate of successful and failed authentications per account per time window. Credential stuffing and lateral movement both distort this metric.
Deploying in OT and Legacy Environments
Legacy operational technology environments present specific challenges for ML-based anomaly detection. These networks often run protocols like Modbus, DNP3, and Profibus over hardware that cannot support endpoint agents. Many devices cannot be patched, and network topology is static by design because stability matters more than flexibility in operational environments.
Paradoxically, these constraints are advantages for anomaly detection. OT networks are among the most predictable network environments that exist. A PLC that has communicated with a single SCADA server over Modbus TCP for three years establishes an extraordinarily tight behavioral baseline. Any deviation, a new destination IP, an unusual command code, a change in polling interval, stands out immediately against that baseline.
Passive monitoring via network TAPs or SPAN ports allows ML-based detection without touching the OT devices themselves. Deploy sensors at the IT/OT boundary and at key aggregation points within the OT network. Train models per device class or communication pair rather than treating the entire OT network as a single segment. The granularity of OT baselines is a detection asset.
Teams securing OT environments against modern threats should also consider that attackers who penetrate IT environments increasingly attempt to pivot into OT segments. ML-based detection at the IT/OT boundary, watching for IT hosts that begin communicating with OT protocols or attempting connections to OT IP ranges, catches this pivot pattern before it reaches production systems.
Operationalizing Anomaly Scores: From Model Output to Analyst Action
The gap between a well-performing ML model and a detection program that actually stops attacks is operationalization. An anomaly score is not an alert. Converting model output into analyst workflow requires deliberate design.
Start with score thresholding. Most ML-based NDR tools produce a continuous anomaly score per flow or per entity over a time window. Setting the alert threshold too low floods analysts with low-confidence detections. Setting it too high means the model only fires on obvious outliers that a signature might have caught anyway. The right threshold is calibrated against your analyst capacity and your acceptable false positive rate, and it should be revisited quarterly as the model matures and baseline quality improves.
Context enrichment is what separates actionable alerts from noise. Before an anomaly alert reaches an analyst, it should be automatically enriched with:
- Asset inventory data: What is this host? Who owns it? What applications run on it?
- User identity mapping: Which user account is associated with this flow?
- Threat intelligence correlation: Does the destination IP appear in any threat feeds? Platforms like Recorded Future maintain proprietary collection engines that surface indicators not present in open-source feeds, and integrating this intelligence into alert enrichment closes gaps that open feeds leave open.
- Historical alert context: Has this host generated anomaly alerts before? What was the analyst verdict?
Alert triage workflows should be structured around the entity, not the individual alert. If a single workstation generates six anomaly alerts across two hours, those six alerts represent one potential incident, not six separate tasks. Entity-centric case management prevents analysts from triaging alerts in isolation and missing the pattern they collectively form.
Tuning Against Adversarial Evasion
Sophisticated threat actors know that behavioral detection exists, and some actively attempt to evade it. The actors behind the modified Hive toolkit demonstrated this by timing their C2 traffic to match human browsing session patterns. Teams deploying ML-based anomaly detection need to account for adversarial tuning as part of their detection design.
The primary defensive response is feature diversity. An adversary who has learned to evade detection on timing-based features may not have accounted for volume-based or graph-based features. Detection models that incorporate multiple independent feature classes are harder to evade simultaneously. This is why combining flow-level, protocol-level, and communication graph features in a single model is worth the additional engineering complexity.
Secondary response is model ensemble design. Rather than a single model, deploy multiple models with different algorithm classes and different feature sets. Require an anomaly score above threshold from at least two independent models before generating a high-priority alert. An adversary who has tuned their behavior to evade an autoencoder trained on timing features may not have tuned against an Isolation Forest trained on communication graph topology.
Red team exercises specifically designed to evade your deployed anomaly detection models are operationally valuable in a way that general penetration tests are not. Provide your red team with the feature set your model uses. Have them attempt to conduct a full kill chain while minimizing anomaly score generation. The evasion techniques they discover become requirements for the next iteration of feature engineering.
Integration With the Broader Detection Stack
ML-based anomaly detection delivers the most value when its output feeds into a broader detection and response pipeline rather than operating as a standalone tool. Practically, this means SIEM integration and SOAR playbook development.
Anomaly alerts enriched with asset and threat intelligence data should flow into your SIEM where they can be correlated with authentication logs, endpoint telemetry, and DNS logs. An anomaly detection alert that correlates with a failed authentication burst and a new outbound DNS query to a high-entropy domain in the same time window constitutes a high-confidence incident that warrants immediate response, even if each individual signal would not have triggered an alert on its own.
SOAR playbooks for anomaly-driven incidents should automate the initial enrichment and triage steps. When an anomaly alert fires, the playbook should automatically query the asset inventory, run threat intelligence lookups on all involved IPs and domains, pull the last 24 hours of authentication events for associated accounts, and deliver the enriched case to the analyst queue with a recommended triage priority. This reduces mean time to triage and ensures consistent context gathering regardless of which analyst picks up the case.
Metrics That Tell You Whether Detection Is Actually Working
Detection programs that cannot measure their own effectiveness cannot improve. For ML-based anomaly detection, the metrics that matter are not the ones ML engineers typically report.
Mean time to detect, measured from the earliest observable evidence of an attack to the generation of a high-priority alert, is the primary operational metric. Compute this for every confirmed incident over the past twelve months and compare it against your target detection window. If attacks are consistently reaching later kill chain stages before detection fires, that gap tells you where to focus tuning effort.
Analyst-validated true positive rate per alert category is more useful than aggregate precision. A model that achieves 40% precision overall might achieve 80% precision on DNS anomaly alerts and 15% precision on volume anomaly alerts. That breakdown tells you which detection categories are mature and which need baseline improvement or feature engineering work.
Coverage gaps are the metric most teams fail to track systematically. Map your ML detection coverage against the MITRE ATT&CK framework. For each tactic and technique, identify whether your anomaly detection has any visibility, and document the specific features or data sources that provide that visibility. Gaps in the map are roadmap items for the next detection engineering cycle.
What to Prioritize If You Are Starting Today
For teams building ML-based anomaly detection from scratch, the sequence of priorities makes a material difference in time-to-value.
Start with flow data collection before touching any ML tooling. NetFlow, IPFIX, or sFlow from your perimeter devices, core switches, and any east-west monitoring infrastructure gives you the raw material everything else depends on. Ensure collection is complete, consistent, and being archived with at least 90 days of retention before moving to the modeling phase.
Deploy a commercial NDR platform before attempting to build custom models. The engineering investment required to build production-quality anomaly detection from scratch is substantial, and the operational maturity required to maintain it is higher still. Purpose-built NDR platforms provide pre-built models, feature engineering pipelines, and analyst interfaces that deliver value within weeks. Custom model development is appropriate once you understand your environment well enough to identify detection gaps that commercial platforms do not address.
Instrument your detection program with the metrics described above from the first day. Detection programs that cannot measure their own performance accumulate technical debt silently. Starting with clear metrics from the outset ensures that tuning decisions are driven by evidence rather than analyst intuition.
The threat landscape that drove the development of ML-based anomaly detection, attackers who engineer traffic to evade signatures, toolkits sold in underground markets with explicit evasion capabilities built in, and adversaries who study defensive tooling to stay below detection thresholds, is not going to simplify. The teams that invest in behavioral detection infrastructure now will be significantly better positioned when the next modified toolkit surfaces in their environment than the teams still relying on signatures to do work that signatures were never designed to do.
