Authors: Kyle McNulty and Gary McIntyre
The U.S. government and private enterprises have spent billions trying to defend against modern adversaries, such as those in the recent SolarWinds Orion supply chain attack. Yet of the estimated 18,000 affected SolarWinds customers, presumably not even one was able to detect and prevent the campaign in their environment over the last nine months. For more information on the attack itself and the application security practices that could have helped prevent the attack, check out this blog post.
The Three Hurdles of Detection
So, what made this specific attack so difficult to detect?
First, significant operational challenges coincided with the timing of the attack.
The attack has been timestamped as far back as March, which means it happened when many organizations were moving to remote work due to COVID-19. This shift likely interfered with traditional monitoring mechanisms and alerting rules. Tracking user logins, remote connections, configuration changes, and general anomalies became far more difficult since they were occurring at an unprecedented rate outside of baseline thresholds. Additionally, U.S. government security efforts were focused on the upcoming election, pulling defense resources away from other departments and initiatives.
Second, the attackers exhibited serious caution, patience, and sophistication.
The malware only roared to life 12 to 14 days after the SolarWinds update was installed on customer systems. Additionally, the communications were extremely well-obfuscated, and the attackers used hostnames matching those in the customer environment and IP addresses from each customer’s country. The data access was also hidden among normal traffic and accessed in moderation to avoid raising alarms. These are just a few examples; the attackers were extremely cautious with every step of the attack. Combined with the extensive timeline, it was challenging for threat hunters to piece the various components together as one malicious incident.
Finally, most SOCs struggle with anomaly detection.
As discussed above, the extreme sophistication of the attack resulted in a number of anomalies, not malicious signatures or blatant red flags. The modern SOC has difficulty balancing true positive detection with false positive prevention in an ever-expanding technology footprint. As a result, SOCs tend to prioritize rules that have an extremely low false positive rate. Anomaly detection rules are the antithesis of that idea. Additionally, anomaly detection requires intensive access to historical data, and this is an extremely expensive endeavor for large enterprises. But this method is necessary for detecting advanced tactics that evade traditional security controls.
Seeing the Big Picture
Unfortunately, as defenses improve, attackers become sophisticated. It is natural to reflect on this attack and focus on the weaknesses in the Windows environment. The attackers exploited the caching of Windows password hashes in order to grab additional credentials and move laterally through the environment. The Windows environment continues to pose significant risks due to simple design flaws. However, this focus misses the larger point.
Given the extreme sophistication of so many elements of the attack, the attackers likely did not need obtain additional credentials in that manner, so it is a mistake to think locking down the Windows environment would have prevented the entire attack. This is not an excuse to leave weak security controls in place, but we want to emphasize the weakness in Windows does not qualify as the single point of failure in this attack.
It is important to recognize the series of anomalies that was triggered across the entire attack campaign: hosts, account usage, authentication, web domains, and more. One anomaly does not indicate an incident. Rather, the association of anomalies over a period of time reveal an underlying incident. And the presence of disparate anomalies rather than blatantly malicious activity shows the future of cyber threats presented by powerful nation-states with near-limitless resources.
Anomaly detection capabilities are extremely difficult to build. There are various challenges that render success fleeting, such as technical needs, budget constraints, and false positive rates. However, there are steps the security industry can take to advance security operations reliant on anomaly detection. In order to successfully build anomaly detection capabilities, an underlying infrastructure must be in place to support advanced technical capabilities and the people leveraging them.
1. Capture the Right Data.
To associate all the various attack indicators in the SolarWinds example, the breadth of data required includes Powershell, user accounts, network activity, cloud resources, and more over a period of weeks. And the activity itself has significant depth. Recognizing an opened Powershell prompt is not enough, but rather additional activity data is required to understand the commands that were executed and how those compare to a typical Powershell session for that user type or system.
Therefore, modern security programs must recognize the importance of collecting and analyzing vast datasets for security use cases. Admittedly, there is a significant budgetary challenge that comes with this task. In order to effectively build towards a robust anomaly detection architecture in a cost-effective manner, it is imperative that organizations prioritize their landscape effectively.
Focus on data associated with commonly recurring tactics, techniques, and procedures used by attackers that target you. For example, when building out account activity anomaly rules to flag attacks against Azure cloud infrastructure, start with Azure Administrator accounts before moving to all accounts. A robust data architecture will serve as the backbone for all investigative technology and analysis.
2. Share Anomaly Intelligence.
Companies have grown used to ingesting and leveraging open source threat intelligence as well as commercial intelligence from various vendors and providers. Many organizations belong to formal or informal threat intelligence sharing networks. Threat Intelligence teams have effectively integrated shared indicators into their own proprietary program for improved detection results. Unfortunately, in the SolarWinds example, threat intelligence indicators did not exist until this last weekend (December 12) when the attack was finally discovered as malicious. However, anomalies certainly did exist.
As we look to the future of reliance on anomaly detection for advanced techniques, the principle of organizational resource sharing should still apply in the anomaly domain. Data sets are an important enabler of anomaly detection as mentioned above. By unifying data sets across disparate organizations through anomaly sharing, you have an improved pool to reference. This extensive pool allows your defenders to determine how widespread an anomaly is and thus improve judgement decisions for follow-up investigation. Additionally, once one organization marks an anomaly as an indicator of malicious activity, others can focus threat hunts and investigations accordingly.
3. Prioritize Investing in Your People and the Tools They Need to Succeed.
Every security team ultimately relies on its people to succeed. Well-equipped, smart defenders can best sophisticated attackers, but it is undoubtedly an uphill battle. (We want to clarify that we are not suggesting SolarWinds customers lacked smart threat hunters, but rather the technical capabilities and surrounding resources left them inadequately prepared for an attack of this type and sophistication.) Skilled hunters are the foundation of a strong threat hunt program, and great hunters continue to adapt and refine their approaches to anomaly hunting as further resources are made available to them.
Anomaly-focused threat hunting is extremely challenging. It takes a keen eye to recognize a standard anomaly from one that is malicious. And it takes an extremely knowledgeable security expert to then relate that point-in-time activity to a chain of events that make up an incident. The security industry must overcome significant challenges to shift towards improved anomaly detection, and they will take time to address. In the meantime, and even after, we must rely on the astute observations of hunters that understand attacker profiles, organizational crown jewels, and signs of compromise to identify this new strain of malicious activity. A capable team is irreplaceable and will also contribute to the buildout of your program in an effective manner.
While nontrivial to implement, these anomaly detection and investigation capabilities will result in a significantly more secure future for enterprises and agencies. Traditional security measures remain important for the vast majority of attacks. But many organizations are spending billions on traditional security in an attempt to prevent advanced threats, which is futile. Hopefully this SolarWinds event serves as an accelerator for organizations to recognize the importance of investing in anomaly detection in defending against modern adversaries.
For immediate risk remediation for affected organizations, please update your SolarWinds Orion software to the latest version and review the threat research post from FireEye which outlines immediate detection mechanisms including published IOCs.
If you are concerned about the maturity of your organization’s Security Operations or DevSecOps programs, or need assistance implementing any of the above measures, Focal Point can help.
Want more security updates in your inbox?
Subscribe to Focal Point's Risk Rundown below - a once-a-month newsletter with templates, webinars, interesting white papers, and news you may have missed. Thousands of your colleagues and competitors have signed up! You can unsubscribe at any time.
