Thoughts on QA Security Testing
In several different contexts, I've been involved in conversations concerning testing for security defects in QA. Here's my take on it.
Remember that I'm a fundamentalist. I love the American League (the Senior Circuit), but hate the DH. I believe engineers should be trained in securing, but not hacking, and I believe that coders should be trained in coding correctly, but not security. I know those sound shocking to people who are experts in security, but there's nothing different about eliminating an injection attack than there is in ensuring that the new presentation layer doesn't barf.
That being said, I don't think QA testing needs to do security testing. They do need to perform "out of bounds" testing, however. QA groups generally take the functional requirements of the application, and test to ensure that all of the functional requirements are met, but they don't test to ensure that things outside the requirements are properly handled. For example, if a field is supposed to be non-blank, alpha-numeric characters, with a maximum length of 20, they will make sure that when non-blank, alpha-numeric characters numbering fewer than or equal to 20 are entered, they do not cause the application to barf. They generally don't test to ensure that more than 20 characters cause a proper warning, or that non-alpha, non-numeric characters don't cause an application failure.
Even with a stored cross-site scripting vulnerability, a proper functional test will actually reveal that the later-rendered pages don't render properly - this is assuming you have proper functional tests. But it will also reveal when the input validation and output filtering techniques being employed aren't working - which are the real core to stopping the various types of injection attacks.
I am not advocating that later security testing be eliminated. Nor am I advocating that your developers not perform static analysis. I'm just advocating that QA groups begin testing for things that are outside the functional requirements to determine that the application fails gracefully, and that they report anything suspicious back to the development team. In order for this to work, it's critical that your functional requirements be very specific about what is allowed, and that the QA testing not only ensures that what is allowed works, but that things outside the allowed range don't work. We still need security experts to engineer the application properly and to do real security testing for logical flaws (and yes, the few semantic flaws that will fall through the previous steps) after a proper code/QA/rinse-repeat cycle.
How hard is it to throw a bunch of cheat sheet data (note: i would use the word "taxonomy" but there has been controversy over the use of this word lately) into some testing and inspection tools?
ReplyDeleteTake MITRE. Take OWASP. Take WASC. Take XSSDB or RSnake's cheat sheet or what have you. Try to get the cheat sheets into an XML format, or something close.
Ok now I know that PMD already supports XPath so that's a no-brainer. For FindBugs you have to write a few lines of Java to add the checks. Oh no.
So now we've already covered (and EclEMMA and EMMA or Clover can tell you how well you've covered) that cheat sheet data in both Eclipse and during build time with Ant, Maven, et al. Automated inspection and code coverage - done!
Development testing (also known as a developer doing the testing, not the QA team) can also be done continuously using CT-Eclipse with JUnit, HTTPUnit, HTML Unit, et al - and maybe they can pull similar cheat sheet data into it. At "daily" (note: could be weekly/monthly depending on how lazy or lame the devteam or project is) build-time, using Ant or Maven - again you can add some DbUnit action using mock objects or stubs of all kinds. Then, developers can re-run this cheat sheet data tests during major builds using protocol drivers (e.g. twill for HTTP, JDBC-spy for queries). Developer testing has now integrated all this cheat sheet data into 1) Continuous unit testing, 2) continuous component testing, 3) continuous component integration testing, 4) continuous system testing, and 5) continuous system integration testing.
The developers can even take that one step forward by pair-testing with a QA tester during functional testing. The developer can take that cheat sheet data plugged into an application driver (e.g. Selenium) and the QA tester can provide experience-based testing (also known to QA people secretly as `exploratory testing') to improve the results of the tests using their own more advanced cheat sheets! How hard is this so far?
But wait! There's more! Before the application is released (upon final build), there can be some final component integration tests and system tests using jCUTE (concolic unit testing with smart fuzz capabilities), Java PathFinder (a model checker), JavaNCSS (a code complexity analzyer), and a final EMMA/Clover code coverage analysis. All of these reports can be fed to the entire development team and the results can be look at in something called "peer review". Particular aspects of security cheat sheet data that somehow made it through all of these tests can be closely analyzed by code reviewers, and finally - by a moderator. Then the moderator can have a Fagan inspection done (or even manual DFA or CFA) on parts/pieces of the code that require the most attention, such as stealing all the credit card data - even after they pass all of the continuous testing, inspection, and manual peer review that has already taken place!
As a final step - QA testers can take the release and do a pair-test along with some security professionals for acceptance, operations, and maintenance testing (adding in that special exploratory testing expertise). Acceptance testing means that the application works in the system provided as the QA/build environment, or perhaps we call this "staging". Then operations testing can be done after putting it into the live, production environment (some security professionals like to call operations testing an initial "vulnerability assessment", or more inaccurately, a "penetration test"). Maintenance testing can be done after a bunch of users have filled the database and inputs up with lots of interesting data. This is where the real fun starts! Now, even third-parties can come in and, surprisingly - can work with both developers and QA on co-operative maintenance testing involving these before-mentioned "vulnerability assessments" and "pen-tests".
OH WAIT says the security professional while co-operatively performing a maintenance test with a QA tester (in experience-based testing fashion) and developer (in "oh shit I did a bad thing because I was lazy fashion") - I have FOUND something! The security professional labels the THING a FIND-ING. Then the security professional says, "you guys should perform remediation!". The QA tester says, "wish I would have found that!" while the developer says "Oh no a defect!". Guess what happens next? The developer writes a test case for that defect, the QA tester adds it to the cheat sheet lists, and at every spinning cycle of the CT/CI-IDE, as well as every build - that defect gets LOOKED FOR AGAIN. Now you have the final step you need in your development-QA-security testing/inspection framework: continuous prevention (or regression testing as the QA team would say). This should also assert any new behavior as as result of the defect's fix! Exciting!
Almost everything I just said would be understood by over half of the talented QA testers in the world. Talented developers and security professionals would be at or around 1%. In other words, I think that security professionals and developers give QA testers a bad name because they don't understand what they do. Not every applications requires a strong QA team, or even one person with the title QA tester. But also not every organization needs a security team or anyone with a title with the word "security" in it. But these roles must be filled - and this is how they should work. Anything else is uncivilized, or more appropriately said, "immature" (see: CMMI, TMM, IMM, ISM3).
Now get back to work!
@dre - you're right - any QA tester worth their salt is familiar with the tools, metrics, and communications means available to them. But I still believe that it's not necessary to test specifically for security defects during QA - it *is* necessary to test for lack of validation and filtering of *any* sort, which is how semantic security defects (any injection flaw) comes about.
ReplyDeleteDoes it matter whether the result is XSS or just a mis-rendered page (in QA, not production)? Not really, the fix for both of those problems is the same.
Spot on!
ReplyDelete