Manual Penetration Testing vs ‘Automated’ Penetration Testing
What is the difference between Manual and Automated Penetration Testing you might ask; aren’t they the same thing? Not quite. Let’s dive right in.
What is a Manual Penetration Test?
The loose definition of a Manual Penetration Test or Pen Test is an authorized simulated attack delivered by a qualified and experienced Offensive Security or Cyber Security Consultant.
So already you may look at that brief definition and say, ‘Doesn’t that make an Automated Penetration Test an Oxymoron?’ Well, Yes and No.
What is the difference between Automated and Manual Testing?
In layman’s terms, an Automated Pen Test is a method of delivering a Security Assessment using automated security testing toolkits and technologies. This is rather than a focused manual assessment delivered by a qualified Security Consultant.
Automated tools have always formed part of a Penetration Tester’s toolkit. But are these commercial products and offerings now advanced enough to be considered as a direct replacement?
A Vulnerability Scanning Vendor or Automated Testing product will likely be developed and maintained by a team consisting of Development and Offensive Security specialists. However, it is important to understand the limitations and challenges of Automated testing when compared to Manual Testing.
An easy way of understanding the differences between both approaches is that a Manual Penetration test’s purpose is, by default, designed to exploit a vulnerability or weakness within a target system, to achieve a valid and contextualized proof of concept for that specific organization.
Whereas, an “Automated Penetration Test” is designed to highlight and identify known weaknesses in a network device, infrastructure service or application.
But like all automated approaches versus a comparative manual service offering, there are limitations to consider with this approach:
- How do we correctly investigate and verify False Positive findings presented by an automated tool?
- Is the level of access supplied for the targets defined in a scanning window suitable based on external risk to our organization?
- How complex is the asset the Automated Tool is reviewing and are their technical limitations?
- How quickly can an automated tool be updated by its respective development team to test for the latest types of vulnerability exploits?
These are just a few examples that security teams need to factor in when deciding the most suitable testing approach. But it is vital to consider this before sending a PO to a supplier with some good marketing and a good sales pitch.
Okay fair point. So how do I know which approach is best for my business?
This, unfortunately, is not an easy question to answer. It all depends on your requirements, drivers and challenges, and will often vary for most organisations.
Certain regulatory and compliance requirements may demand that Security Testing is performed by an Authorized and appropriately certified Penetration Testing Third-Party. However, there are lots of use cases where an Automated Testing approach can provide direct benefits over a more Manual Testing approach.
A common example of this is organisations who, due to the nature of their business, need to publish application and software updates on either a weekly or fortnightly basis. For example, as part of a Software Development Lifecycle a typical organization releases code or software updates every two weeks in line with Development sprints.
Then it can prove a real challenge for Security Project teams to schedule Manual Penetration Testing with a Third-Party provider that matches the speed of those development sprints.
When used intelligently, Automated Testing tools like Vulnerability Scanners can be used alongside custom scripts and other tactics, techniques, and procedures against less complex applications to replicate and simulate the steps used by a skilled attacker.
Allowing IT Security teams to design and plan a regular testing window to assess applications and infrastructure services on a more frequent basis for vulnerabilities is a much more efficient way than scheduling a 3rd party delivered Penetration Test.
In most situations, Automated Testing is designed to support the efforts of a Manual Penetration Tester to identify flaws across larger estates and attack surfaces. This allows a Penetration Tester to focus their limited time and attention on attempting to exploit more complex, logic-based vulnerabilities that automated toolkits simply are not sophisticated enough at this time to replicate.
A major challenge that Penetration Testing consultancies have faced for years is being expected to find every instance of a vulnerability within a limited testing window. When an attacker with no limitations such as time or budget simply needs to find one to be successful.
Ultimately, you need to consider the following challenges and design your approach around what suitably meets your needs.
It is important to remember that in an ideal strategy, neither approach should replace the other. When designed and used properly as part of a defensive, in-depth approach, both approaches complement each other!
A shift in approach and Penetration Testing Expectations
Over the past decade in Information Security, there has been a major shift in the way that Information Security Managers and project teams expect to receive and access Vulnerability Information and Report Findings.
Most businesses who used to commission a full Penetration Test annually would commission a vendor agnostic Third-Party to manually test all of the externally facing infrastructure services, Web Applications and Internal Network devices defined in scope in depth.
Depending on the level of findings identified, this could take between 15-20 Days of Consultant work and Reporting time to deliver a comprehensive overview of an organisations attack surface. Without factoring in the time required to triage these key vulnerabilities internally and agree a delivery plan for remediation, mitigation, and Re-Testing.
For major organisations with strict regulatory requirements in 2020 this process just simply was no longer agile enough, as changes were occurring in major programmes faster than a testing window could be planned.
This has almost made certain aspects of the Manual Penetration Testing process invalid, as a Manual Penetration Test is considered as a snapshot in time and a whole host of new vulnerabilities could have been shared in the wild by Hackers or Security researchers in that time.
This resulted in a different challenge for consumers who wanted to explore new methods of Dynamic Security Testing (DST), and for Penetration Testing consultancies and Vulnerability Scanning vendors to adapt to meet these changing requirements and concerns.
This resulted in hundreds of suppliers selling this as a one-size-fits all solution to cover Re-Testing, Compliance Driven Testing and replacement for Consultant driven testing as a cost-based value proposition.
This creates another challenge for procurement and IT Security teams to navigate; to build trusted relationships with suppliers who understand their businesses and their range of constantly developing challenges.
So, with this problem identified, is there a sensible way to improve the way that customers can find the right supplier in the Information Security marketplace?
What’s the solution to the challenge?
The common issues with the initial stages of planning is a lack of understanding on how an organization can utilise both Automated and Manual Testing approaches, based on their requirements and to their benefit.
A major challenge in the first instance for organisations new to the world of Security Testing is that they don’t actually know what a good baseline of Security Testing looks like. This can present challenges between suppliers and consumers, especially if services have been mis-sold previously.
A project team should ideally consider the following things before a scope of work is even produced for suppliers to quote against:
- What is the baseline we currently have in place for security testing across our External perimeter and Internal network?
- What systems, assets and information do we hold on our customers, staff, suppliers and target market that presents a risk for our business and would be a specific focus for a malicious attacker?
- What is business critical for our organization and has to always be operation and 100% be included in our defined scope for testing as a priority?
- What are our main limitations when it comes to planning Security Testing? Budget? Constantly developing Applications and Infrastructure services? Limited Internal Resource and Skillsets? Conflicting priorities?
If the project team cannot answer any of those four questions, then it is apparent that there are some internal conversations and activities that need to take place before jumping into commissioning a test.
In some cases, to achieve a good baseline of your what your External/Internal Security Posture looks like to a skilled attacker it may be worth investing in a Manual Penetration Test. This provides a detailed, contextualized analysis of what vulnerabilities currently exist and what needs to be addressed as a priority.
However, whilst this may be useful for some organisations, this model is only effective if an internal team can commit the necessary time and resource into appropriate Vulnerability Triage and remediation activities within a certain timeframe.
For a more dynamic development team or organization, it may be that a point-in-time Manual Penetration Test simply is not sufficient and Automated Testing should be first used to achieve a basic understanding of existing vulnerabilities. Once remediated, there is a baseline for future Security Testing activities to be considered against.
Once this baseline is achieved, it makes planning future Security Testing activities a simpler process to navigate. Future planning can be tailored around an organisations growth plans
How do I find the right partner for us?
So, you have taken the above actions and are now ready to explore the market for an appropriate Security Testing partner to help discuss your options for future Security Testing and Auditing requirements.
The good thing is that the UK has some fantastic Offensive Security Testing specialists, and some incredibly specialist suppliers and consultancies.
However, as of 2020, there are over 500 Suppliers offering varying forms of Penetration Testing services. If procuring a Penetration Test is new to you, this can be an incredibly daunting experience.
You may think ‘how on earth is a small IT team supposed to know the differences between over 200 differently branded and marketed Penetration Testing offerings?’ Our advice? Challenge them!
Look for reviews, ask for references, ask your network, review their Sample Reports and Methodologies, ask to schedule a call with one of their Penetration Testing team, allow them the opportunity to propose how they would design a phased testing approach or statement of work.
A competent supplier will be able to meet your basic requirements. A great supplier will work with you to try and exceed your expectations. Challenge a supplier on their offerings and you will have a much more meaningful conversation with a better end product.
The challenge of what creates a great supplier is commonly discussed within the Information Security community. What should choosing a suitable supplier be based upon? Company size? Accreditations and Certifications? Consultant experience? Trading History? Price?
Ultimately, this will vary based on what you as a business require for procurement guidelines. Whilst price is a major consideration in the current climate, our recommendation would always be to work with someone who values your business. Someone that can offer agility, a combination of individual Penetration Tester skill sets and competitive pricing, but that ensures quality of service is maintained and can be consistently repeated.
It is important to state here that as part of a defensive, in-depth approach, you need to challenge suppliers on making sure their offering meets your needs and that they provide sufficient evidence for you to validate it’s the right fit.
Often an Automated or Manual Testing approach alone is not perfect, and planning and considering your scope for testing is the vital part of a successful project and Vulnerability Management process.
Both approaches have costs; whether that’s Capital Expenditure on Professional services or solutions, or Operating Expenditure for the cost of internal resource.
But both approaches have benefits.
Whether that is increased visibility of your attack surface and more frequent testing, or a hybrid approach that gives them agility and focused comprehensive manual assessment where needed in specific situations.
Penetration Testing specialists will always recommend a balanced blend of Vulnerability Scanning, and Manual Penetration Testing should be utilized to play on each approaches strengths.
As Security Advisors Precursor Security ultimately want to educate clients and provided value from working with them. Whether that is advising on a creative solution, or simply recommending a different approach that more adequately meets a requirement.
Therefore, it is so important to build an honest and transparent relationship with your Security Testing partner or build relationships with a couple of suppliers in case rotation is required for quality.
Once a supplier understands your challenges, limitations, drivers and niche ways your business operates. They can make sure their process is tailored around delivering value.
The cost of being proactive will always be more beneficial for your business than the costs involved in being reactive.