On Source Code Review

First of all, Jeremiah Grossman's periodic Web Application Professional's Survey is online - so go take it.

That being said, I've kept quite silent on the value of static source code analysis for awhile now because I'm pretty sure what the reaction will be, but one of the questions on the survey was regarding which measures to application security go first.

There have been several places where static analysis has gotten a dissed, where it might not be necessary. Most notably of which, I think Fortify Software did their own software a disservice at the Iron Chef Blackhat edition at Black Hat last year. The competition was between some of their source code analyzers and some of their dynamic analyzers. The application testers won hands down.

Before I begin, know that I believe that there is no silver bullet to application security.  Nor do I think static source code analysis is the "best" method of finding vulnerabilities.  Here are some of the valid or most important reasons that static analysis should not stand alone:

• Static analysis is really best at finding semantic flaws - bad API use or failure to use certain API's, etc.
• Static analysis doesn't give compelling pretty pictures and videos of your application giving up information.  The results of a static analysis are only meaningful to developers, and then, only meaningful to developers who understand the real risk of the types of findings.
• Static analysis almost always requires really expensive tools to do a really, really good job.  There are grep types of analyzers, but they don't follow taint through an application.
• Static analysis may analyze components of your code that don't get used.  There are still prioritization decisions to be made.
• Static analysis tools can't find logical flaws such as privilege escalation or XSRF.
• Static analysis has different requirements than black box testing:
• Developers who understand the code and can fix it
• The source code
• For many tools, the code needs to at least build (doesn't have to run)
  

However, there are some really, really good reasons static analysis should be a part of your security toolbelt:

• Static analysis can find vulnerabilities that dynamic analysis can't - corner cases.  "This cross-site scripting flaw only exists on Tuesdays" - if your application was tested in a running state on Monday, you won't know that the flaw exists.  Thread safety issues are very bad for an application, but a black box test of an application might never cause one to come up, and if it does, it's nearly impossible to reproduce, and the results don't say to the oracle that it was that type of vulnerability.  (For example, the application gave you access even though you used the wrong password.)
  
• The results of static analysis are meaningful to developers.  They get lines of code back where untrusted data enters the application, where it flows through the application, and when it exits the application.  These are the exact lines that the developers need to fix, which a black box test alone can't give you.
• Since the results of a static analysis are geared toward the developers, it provides "instant training" for developers.  "What does it take to make this shut up?"  (While I prefer developers understand why you want it to shut up, finding all the places is pretty good, too.)
• Static analysis can happen much earlier in the development process, long before the application is functional.  This gives black box testers more time to test the really cool stuff that static analysis can't find.
• Static analysis can take place as part of a build process, automatically generating problem tickets and/or preventing the promotion of code with high-probability, high-risk findings.  This can be done with automated black-box tools, but it requires a running environment - many more moving parts.
  

So I'm convinced that as long as Iron Chef Blackhat is run the same way as it was last year, static analysis will always lose simply because to a spectator, the results are just boring.  However, that doesn't mean that it shouldn't be a very, very vital part of the development process.