When the Alert Queue Goes Quiet and the Breach Is Already Three Weeks Old
A mid-sized financial services firm running a commercial network detection and response platform discovered a data exfiltration campaign only after a threat intelligence partner flagged one of their egress IPs in a shared feed. The internal anomaly detection system had been running for 18 months. It had generated alerts throughout that period, but the alerts flagged routine bursts from a backup job, a misconfigured monitoring agent, and periodic spikes from a marketing analytics tool. The security team had tuned these out. Meanwhile, a low-and-slow exfiltration session averaging 2.3 MB per hour over encrypted HTTPS had been running for 26 days without a single detection trigger.
This is not an edge case. It reflects a structural problem in how many organizations build and maintain machine learning-based anomaly detection: the models get trained on traffic that already includes malicious behavior, the tuning process optimizes for reducing false positives rather than catching novel threats, and the operational feedback loop that should update baselines stops functioning after the initial deployment sprint ends.
With ransomware attacks continuing to climb in frequency and sophistication, China-linked threat actors now demonstrating AI-augmented reconnaissance capabilities in APAC campaigns, and groups like Armored Likho actively expanding their espionage toolkits, the gap between what anomaly detection systems claim to cover and what they actually catch has real consequences. Understanding where that gap lives is a prerequisite for closing it.
What Machine Learning Anomaly Detection Actually Does Under the Hood
Anomaly detection in network security relies on a core assumption: normal traffic has statistical structure, and deviations from that structure indicate potential threats. The ML component attempts to learn that structure automatically rather than requiring analysts to write explicit rules for every possible deviation.
The main algorithmic families in production deployments fall into a few categories:
- Statistical baseline models track metrics like bytes per session, connection frequency, protocol distribution, and port usage, then flag values that exceed configurable standard deviation thresholds. These are computationally cheap and interpretable, but they require clean training data and perform poorly against slow-moving threats that stay within threshold bands.
- Autoencoder-based deep learning models learn compressed representations of normal traffic and flag inputs that the model cannot reconstruct accurately. These handle high-dimensional feature spaces well and can surface subtle deviations, but they are sensitive to training data quality and require significant compute for inference at scale.
- Isolation forests and tree-based ensemble methods identify anomalies by measuring how quickly a data point can be isolated from the rest of the dataset. They generalize well to tabular network features and are relatively robust to noise, making them popular for host-level behavioral detection.
- Graph-based and relationship models analyze communication patterns between hosts rather than traffic volume alone. These are particularly effective against lateral movement because attackers who stay within normal volume ranges often cannot avoid creating abnormal connection graph structures.
Each approach captures different threat signals. Mature detection architectures layer multiple methods because a threat that evades one class of detection often surfaces in another.
The Baseline Problem and Why It Compounds Over Time
Every ML-based anomaly detector depends on a baseline. The quality of that baseline determines what the model treats as normal. This is where most deployments develop serious structural weaknesses.
Baseline training typically happens during an initial observation window, often 30 to 90 days. If that window includes an active intrusion, the malicious traffic becomes part of the definition of normal. If the window coincides with an atypical business period, such as end of quarter, a product launch, or a migration project, the resulting baseline may not represent typical operational traffic at all. Both scenarios produce models that either miss real threats or generate unsustainable alert volumes.
The baseline drift problem is equally damaging. Networks change constantly. New cloud workloads, SaaS integrations, remote access expansions, and infrastructure changes all shift the traffic distribution. A model trained on traffic patterns from 18 months ago is interpreting a meaningfully different network. Without continuous baseline updating, detection sensitivity degrades silently. Security teams often interpret a quieter alert queue as evidence that threats have decreased, when it actually reflects a model that has lost the ability to discriminate between normal and abnormal.
Operationally, the fix requires building baseline refresh cycles into the detection program, not just the initial deployment. A reasonable cadence for most environments is quarterly full retraining with weekly incremental updates to account for drift. The retraining process needs to include a validation step that compares detection rates on labeled historical threat data before the updated model goes into production.
Feature Engineering Determines What the Model Can See
ML models can only detect anomalies in dimensions that are represented in their feature set. This makes feature engineering one of the highest-leverage decisions in building effective anomaly detection, and one of the most frequently underinvested areas.
Common feature sets built around raw NetFlow or packet captures include volume metrics, protocol flags, port numbers, and connection duration. These capture volumetric and protocol-layer anomalies reasonably well. They perform poorly against threats that operate within normal volume ranges, use legitimate protocols, and connect to expected port numbers.
The malware crypting services ecosystem illustrates the limitation clearly. Crypters are specifically designed to make malicious payloads look like legitimate software traffic. A crypted payload communicating over port 443 with a valid TLS certificate to a recently registered but currently clean domain will not trigger volume-based or protocol-based anomaly alerts. Detecting this class of threat requires behavioral features that capture communication timing patterns, certificate transparency data, domain age and registration metadata, and session entropy.
Effective feature sets for modern threat detection include:
- Temporal regularity metrics that measure whether connection intervals follow a regular pattern, which is a signature of C2 beaconing even when individual sessions look normal
- Entropy measurements on DNS query strings, URI paths, and payload sizes, since encrypted malware traffic often has different entropy distributions than legitimate application traffic
- Peer relationship graphs that track which hosts communicate with which external endpoints and flag new relationships that deviate from established communication patterns
- Certificate and domain metadata integrated as features, including domain age, registrar reputation, and certificate issuer, pulled from threat intelligence and passive DNS sources
- Protocol adherence scores that measure whether traffic claiming to be a particular protocol actually conforms to that protocol's behavioral specification
The identity dimension is also underrepresented in most network anomaly detection feature sets. As highlighted in recent SOC operational analysis, identity has become a primary attack surface. Integrating authentication events, privilege escalation events, and access pattern data into network anomaly models significantly improves detection of credential-based attacks that produce minimal network anomalies on their own.
Adversarial Evasion and Why It Changes the Detection Architecture
Sophisticated threat actors study detection systems. The groups expanding their toolkits and the AI-assisted campaigns emerging from nation-state actors are not firing generic malware at networks and hoping for the best. They profile target defenses and adapt their tradecraft accordingly.
Adversarial evasion against ML-based anomaly detection takes several forms in practice. Traffic shaping involves adjusting communication volumes and intervals to stay within the statistical bounds the model has learned as normal. Mimicry attacks involve generating traffic that closely resembles legitimate application behavior, including mimicking the timing and volume patterns of known-good applications. Slow-and-low techniques deliberately avoid triggering threshold-based alerts by distributing malicious activity across long time windows and many sessions.
The Armored Likho group's expanded toolkit reflects this trend. Their operations use legitimate remote administration tools and cloud storage services as communication channels, specifically because these generate traffic that anomaly detectors trained on organizational baselines have learned to treat as normal.
Building detection infrastructure that maintains sensitivity against adversarial evasion requires a few architectural decisions that are more expensive than standard deployments but operationally necessary:
First, detection models should be diversified across feature spaces so that evasion in one dimension does not provide full coverage against all detection layers. An attacker who learns to evade volume-based detection may still surface in graph-based or timing-based models.
Second, the detection architecture should incorporate models trained on threat behavior directly, not just normal behavior. Supervised classifiers trained on labeled attack traffic complement unsupervised anomaly detection by providing recognition of specific attack patterns even when those patterns stay within normal statistical ranges.
Third, continuous red team exercises specifically targeting the anomaly detection system should be part of the operational program. Penetration testing that does not test whether the detection system actually fires is an incomplete test. The output of these exercises should feed directly into model retraining and feature development priorities.
Alert Prioritization and the Human Bottleneck
The Black Hat USA 2026 discussion around the Hugging Face hack highlighted something that applies directly to anomaly detection operations: human responsibility in the loop remains critical, and the places where human judgment gets bypassed or overwhelmed are where serious failures occur.
Anomaly detection systems that generate high alert volumes create a specific failure mode. Analysts develop alert fatigue, triage speed decreases, and the organizational response to alerts that do represent real threats becomes slower and less thorough. Some organizations address this by aggressively suppressing alert categories, which reduces alert volume at the cost of detection coverage. Others attempt to automate response to certain alert types, which creates risk when the automated response is triggered by a false positive.
A more sustainable approach uses a tiered prioritization model built directly into the detection pipeline:
- Tier 1 alerts represent high-confidence anomalies with strong contextual corroboration from multiple detection layers, threat intelligence matching, and identity context. These should trigger immediate analyst review and potentially automated containment actions.
- Tier 2 alerts represent moderate-confidence anomalies from a single detection layer without corroboration. These enter a queue for analyst triage within a defined SLA, typically four to eight hours depending on organizational risk tolerance.
- Tier 3 observations represent low-confidence signals that fall below alert threshold but are stored for retrospective analysis. When a Tier 1 alert fires, analysts should be able to query historical Tier 3 data to reconstruct the timeline of earlier activity that preceded the high-confidence trigger.
The Tier 3 retrospective capability is where many organizations have genuine detection capability gaps. Low-confidence signals from the early stages of an intrusion frequently exist in the data but get discarded or ignored because they never crossed the alert threshold. Building the data retention and query infrastructure to surface these retrospectively can significantly compress dwell time even when the initial detection is delayed.
Integrating Threat Intelligence Into the Detection Loop
Anomaly detection operates on traffic behavior. Threat intelligence operates on known indicators. These two signal types are complementary rather than substitutable, and combining them in the detection pipeline produces better outcomes than running them in parallel silos.
The practical integration points are specific. Known malicious infrastructure indicators, including IP reputation data, malicious domain lists, and file hash feeds, can serve as dynamic feature inputs to anomaly models. When a connection involves an endpoint that appears in a threat intelligence feed, that context should elevate the anomaly score for that session rather than being evaluated in a separate alerting system that the analyst checks independently.
The recency problem with threat intelligence feeds is real and documented. Blocklists represent historical verdicts. By the time an indicator appears in a commercial feed, the infrastructure may have moved. But for anomaly detection purposes, feed data does not need to be current to be useful. Even outdated indicators provide signal about threat actor infrastructure patterns, ASN preferences, hosting provider selection, and registration behaviors that can inform model features without requiring point-in-time accuracy.
Passive DNS data integrated into anomaly detection is particularly valuable for identifying domain generation algorithm activity and fast-flux infrastructure. Models that incorporate historical resolution data for queried domains can flag connections to domains with abnormal resolution histories even when those domains are not yet on any blocklist.
Deployment Architecture and the Visibility Problem
ML-based anomaly detection is only as good as the traffic it can see. This sounds obvious, but many deployments have significant blind spots that the security team does not fully account for.
Encrypted traffic inspection is the most common visibility gap. The majority of enterprise network traffic runs over TLS. Network anomaly detection that operates only on metadata, without any inspection of payload content, misses payload-level indicators but can still capture behavioral anomalies in connection patterns, certificate characteristics, and session timing. Deployments that rely on deep packet inspection for detection accuracy face challenges as TLS 1.3 and certificate pinning reduce the effectiveness of traditional SSL interception approaches.
East-west traffic between internal hosts is frequently underinstrumented. Perimeter-focused detection that captures north-south traffic misses lateral movement, which is a core component of modern intrusion chains. NetFlow collection from internal switching infrastructure and agent-based host telemetry from endpoints are both necessary to cover lateral movement detection scenarios.
Cloud workload traffic presents a third visibility challenge. Organizations running significant workloads in AWS, Azure, or GCP often have network anomaly detection configured for on-premises infrastructure that does not extend to cloud VPC traffic. VPC flow logs and cloud-native network monitoring integrations can fill this gap but require deliberate configuration and integration into the central detection pipeline.
Container and microservices environments add complexity because the volume of legitimate east-west connections between services can be extremely high, making behavioral baselines difficult to establish and maintain. Service mesh telemetry, including tools like Istio or Linkerd, provides structured traffic data that can support anomaly detection within containerized environments without requiring packet-level capture.
Measuring Detection Effectiveness Before an Incident Forces the Question
Most organizations cannot answer a straightforward question: what percentage of intrusion techniques in their threat model would their current anomaly detection actually catch? The absence of incidents is not evidence of detection effectiveness. It may simply reflect that no attacker has targeted the organization recently with techniques that happen to fall outside detection coverage.
Detection engineering programs address this through systematic coverage measurement. The MITRE ATT&CK framework provides a structured way to map detection coverage because techniques are documented with specific observable behaviors that can be used to generate test traffic. Tools like Atomic Red Team and CALDERA allow security teams to simulate specific ATT&CK techniques in controlled environments and measure whether the detection system fires the expected alerts.
Running these simulations against the live anomaly detection system on a quarterly basis provides a coverage measurement that is grounded in actual system behavior rather than vendor claims or theoretical analysis. The output should map directly to a coverage heat map that shows which techniques are detected, which generate partial signals, and which produce no alerts at all. That heat map should drive both tuning priorities and tool investment decisions.
With Microsoft having recently addressed nearly 400 security vulnerabilities in a single patch cycle, the attack surface available to threat actors continues to expand faster than most detection programs can add coverage. Prioritizing detection coverage for techniques actively used by the threat actors most relevant to your sector is more valuable than attempting broad coverage of every possible attack vector.
Operationalizing the Feedback Loop
The difference between anomaly detection that improves over time and anomaly detection that slowly becomes irrelevant is the feedback loop. When analysts investigate an alert and reach a verdict, whether it is a true positive, a false positive, or a benign anomaly, that verdict needs to flow back into the model training process in a structured way.
In practice, most organizations collect analyst verdicts informally at best. Alert notes get written in a ticketing system, but the data is not structured for model training. Building a feedback collection interface that captures verdicts in a format directly usable for retraining is an infrastructure investment that pays for itself quickly in improved model precision.
True positives confirmed by analysts should enter a labeled dataset that gets incorporated into supervised model training alongside threat intelligence-sourced labels. False positives should inform suppression rules and feature weight adjustments. Patterns of false positives from specific asset types or traffic categories often indicate feature engineering problems that are worth addressing at the model level rather than through suppression alone.
The no-filter AI platforms now being marketed to criminal actors and the increasing accessibility of AI-augmented attack tooling mean that the threat landscape is evolving faster than traditional signature-based approaches can track. ML-based anomaly detection built around strong feedback loops and continuous retraining is positioned to adapt more effectively to this evolution than static detection systems, but only if the operational program treats model maintenance as an ongoing discipline rather than a deployment task that ends at go-live.
Where to Start If Your Current Program Has These Gaps
If the failure case at the beginning of this article sounds familiar, the remediation path does not require replacing the entire detection stack. It requires addressing the structural gaps systematically.
Start with a baseline audit. Pull the training data used for your current models and assess whether it is representative of current traffic and whether it is known to be clean. If neither condition is true, schedule a retraining cycle before addressing anything else, because every other improvement built on a corrupted baseline will underperform.
Next, map your visibility coverage. Document which traffic flows are instrumented, which are not, and which present genuine technical constraints. East-west and cloud workload gaps are worth prioritizing because they represent the environments where attackers move laterally after initial access.
Then run a detection coverage exercise against the top five ATT&CK techniques used by threat actors relevant to your sector. Use the results to identify your highest-priority tuning needs rather than trying to address everything simultaneously.
Finally, build or formalize the feedback loop. Even an informal process where analysts tag verified true positives in a structured field before closing tickets provides training data that is significantly more valuable than no feedback at all.
Anomaly detection that improves continuously is achievable. It requires treating the detection system as a living program rather than a product deployment, and it requires organizational commitment to the operational work that makes ML models useful over time rather than just at initial launch.
