As I speak with clients about their penetration testing needs, it has become increasingly clear to me that most organizations are still struggling to figure out what constitutes a good penetration test and how to buy one.

Cars have been around long enough that, as a society, we’ve generally agreed upon the basic standards of what makes a good car and what to look for when purchasing a car. Reliability is important. Speed may matter to you depending on your goals. It most definitely should have headlights. Over the years, we’ve collectively agreed seatbelts are a requirement. But these universally agreed-upon standards don’t exist yet for penetration and security tests. While we may never settle on a standard set of requirements, it is important that your organization establishes its own standards that define what goals and requirements matter most of the time specifically to your organization.

Part of my job as Director of Penetration Testing at Focal Point is to help companies ask the right questions as they shop for a penetration test. It shouldn’t be a surprise that our practice is largely formed by what we believe are the right answers to these questions.

In order to understand what makes a good penetration test, we need to start with the goals of a penetration test. For many organizations, the goal is to simply complete a task and check a box. But better goals are to identify vulnerabilities to your organization and fully understand the impact of those vulnerabilities.

Fulfilling those goals isn’t easy, but a good penetration test will help you get close. Let’s look at the qualities of a good penetration test and how they’ll help you discover and address critical risks to your business.

The Right People

The most important part of your penetration test is the team who performs it. Organizations who have partnered with different penetration testing firms have likely realized that not all pen testing teams are created equal. In fact, there’s been a major shift in the skills desired by companies seeking pen test services. The skills they once regarded as largely unimportant are now being recognized as critical. Gone are the days of the mysterious, anti-social, hooded hackers conducting penetration tests. Organizations have realized that a team’s soft skills are just as important as their technical expertise.

Historically, the language gap between pen testers and non-technical executives has been an ongoing issue, but now the stakes are higher. The costs associated with data breaches are incredible. And it is typically more than just your business on the line – consumers are often the real victims as their personal data, money, and time are consumed in breach after breach. In order to protect their businesses and their customers, executives need to understand the complex, technical problems uncovered during a pen test.

Organizations are now realizing how important it is to not only conduct a penetration test but also to respond to it. And in order to properly respond to a penetration test, you must understand it. This is why a testing team’s soft skills, especially their communication skills, are so important. You simply cannot respond to what you don’t understand. Moreover, you cannot respond to everything. A good penetration test will prioritize the greatest risks to your organization with the greatest clarity.

Additionally, over the years we’ve seen far too many penetration tests delivered into the hands of blue teams only for them to be filed away and never seen again – often because organizations simply don’t know what to do with the results. Poorly communicated results can be cryptic, unactionable, or simply confusing. When the packets have cleared, the deliverables that arrive at the end of a penetration test is what was ultimately purchased. When you buy a penetration test, you are really buying communication on paper, and that communication needs to be clear and actionable.

When shopping for a penetration test, requesting copies of sample reports and conversations with multiple members of the team can give you a better sense of the team’s communication skills. This also gives you a first-hand look at the quality of their deliverables, which should provide prioritized recommendations and clear language even non-technical members of your organization can digest and act on.

The Right Philosophy

There seem to be two dominant philosophies governing penetration testing approaches. The first type can hardly be called a penetration test at all and ends with a simple list of vulnerabilities, usually discovered via an automated scanner. I call this the “breadth” approach because it’s a mile wide but only an inch deep. It covers a lot of data, but the deliverables are shallow. This type of test is often the least expensive, and it should be because it typically fails to deliver what is most important.

If we consider the goals of a penetration test (to identify vulnerabilities within an organization and to better understand the impact of those vulnerabilities), this type of test will – at best – only meet the first goal: identify vulnerabilities. However, it completely fails to communicate any real sense of the risks presented by those vulnerabilities. While regular scanning tests are a necessary and helpful part of a security program, there are significant disadvantages to this approach in isolation.

First, severity ratings associated with vulnerabilities are often misleading and fail to properly communicate the risk to your specific organization. A “medium” severity vulnerability may be the vulnerability that causes the greatest risk to your organization. Secondly, in my experience, most blue teams fail to properly understand the potential impact of most vulnerabilities. Skilled offensive red teams can take minor vulnerabilities and chain them together until the entire castle crumbles. This “breadth” approach fails to demonstrate this type of risk. Moreover, not all vulnerabilities can be discovered by automated scanners. Unless extensive manual testing is conducted, this approach usually misses these types of vulnerabilities.

Finally, this approach also results in many false positives, frequently wasting valuable blue team resources. As a result, these so-called “penetration tests” that simply deliver a list of vulnerabilities discovered via automated scanning do a significant disservice to their clients. In the end, this approach costs more than it saves.

The second common type of penetration test is the “depth” approach. This approach skips the list of vulnerabilities altogether and instead provides clients with a clear narrative of one or two exploit chains, including demonstrations of the risks discovered via those exploit chains. This can be extremely helpful to organizations as it allows them to better understand the impact of particular vulnerabilities. Moreover, it helps you better understand how a vulnerability can be used to move from a simple foothold to complete organizational compromise.

While this approach succeeds in the second half of our goals for penetration testing, it doesn’t fully meet the first goal. You may be made aware of a few vulnerabilities and how they impact the organization, but you’ll lack a full picture of the breadth of your organization’s vulnerabilities and how to mitigate them. Depending on your organization’s security program and maturity, this may be a significant disadvantage.

For most organizations, a good penetration test, from which the greatest value is gained, should have both “breadth” and “depth,” as both goals are important. Most organizations need a full list of all discovered vulnerabilities and demonstrations of how vulnerabilities can be exploited to negatively impact the organization. A good penetration test will combine these two philosophies to help you understand the impact of discovered vulnerabilities on your technology, people, and processes and provide helpful recommendations on how to mitigate those risks.

The Right Priorities

One of the most important questions you can ask yourself when preparing to purchase a penetration test is, “What is important to my organization?” The answers to this question should govern the entire engagement.

Not all organizations care about the same things, and rightly so. Hospitals and clinics may place a premium on their patients’ PHI and medical devices, but an online retailer will want to focus on their payment processes, their brand, and their cardholder data. A good penetration test will be mindful of these things and prioritize them.

There is very little value in the discovery of several minor vulnerabilities if it comes at the expense of time spent on the discovery of issues affecting your most important assets.

Being aware of several minor information disclosure vulnerabilities may be helpful, but the time spent identifying these minor vulnerabilities steals away time that could be spent testing the most important assets. This is why goal-oriented penetration testing is so important. Few penetration tests have unlimited time and resources. Therefore, the most effective tests prioritize the most important assets. Working through the question posed above will help you identify those assets and communicate them to your testing team. In turn, a good pen testing team will then prioritize those assets to maximize the value of the penetration test.

Keeping these priorities front of mind may help you as you build your testing strategy, as well. To dedicate more time to these assets, you may need to start the testing team in privileged areas of the network (“white carding”) or give them access to portions of an application you may have previously determined to be “off-limits.” Or you may need to remove rate-limits, set up white lists, or remove controls that slow down pen testers who are on strict timelines but would merely be nuisances to real adversaries with unlimited time.

The Right Scope

One of the most effective ways to ruin your penetration test is to not spend time thoughtfully scoping the project. As regulatory requirements for penetration tests have increased, organizations are frequently scoping their tests to just get a passing grade. This often means they are intentionally excluding assets with known issues or failing to properly understand regulatory scoping guidelines in the first place.

A good penetration test is comprehensive in nature and includes the full range of organizational assets.

For example, far too many organizations overestimate their segmentation defenses and assume that because they have segmentation in place, they only need to test a subset of assets within their PCI scope. While there can be value in testing a subset of assets, organizations should also regularly conduct more comprehensive tests to fully understand how malicious actors could potentially move from a less-secure to a more-secure environment.

Moreover, if the goals of a penetration test are to understand risks to your organization and the potential impact of those risks, tests should be comprehensive in both asset inclusion and test type. Organizations rightly emphasize web application and external penetration tests but far too frequently leave the social engineering or wireless front doors wide open. Adversaries know this, as recent Verizon DBIR reports show, and have adjusted their tactics accordingly.

As a result, comprehensive scoping and testing approaches ensure vulnerabilities are identified and mitigated. External and web application penetration tests are important, but if the critical controls that protect the internal environment of the organization can be easily bypassed through a simple phishing attack, then those controls can be rendered completely useless.

Organizations need to take a comprehensive approach to security testing and understand that while penetration testers are bound by time and scope, malicious actors are not.

Finding a Good Penetration Test

As with any major purchase, a good thing is hard to find. While a great penetration test is more than the right people, philosophy, priorities, and scope, focusing on these areas will better equip you to make the right choice for your next engagement.

Looking for more pen testing insights? Check out some of our other pen testing guides and blog posts.

You can also learn more about our pen testing services here.

Want more cybersecurity guides and insights?

Subscribe to Focal Point's Risk Rundown below - a once-a-month newsletter with templates, webinars, interesting white papers, and news you may have missed. Thousands of your colleagues and competitors have signed up! You can unsubscribe at any time.