How do we classify security flaws?

The common problem with most of the popular web application security vulnerability classifications (OWASP Top 10, WASC, CWE, Seven Pernicious Kingdoms) is that they really mix and match threats and vulnerabilities. (In defense of WASC, it is mostly threats, and is indeed called a "Threat Classification").

When I discuss this, I need to let you know how I'm defining things - this is according to ISC2:

  • A vulnerability is an absence of weakness of a risk-reducing safeguard.
  • A threat is an event that could have an undesirable impact on the well-being of an asset.
So a vulnerability is something the engineering or development team forgot to do. A threat is what the bad guy does to exploit that vulnerability.

There are a couple of problems with mixing and matching:
  • It makes the report inconsistent. When we classify all of them as "vulnerabilities", then a single issue really falls into multiple categories in all of the systems. A single successful SQL Injection in most taxonomies would be classified as a "lack of input validation", "lack of data representation", and "SQL Injection". Seven Pernicious Kingdoms gets closer, because the kingdoms are (generally) vulnerabilities, and the philum are (generally) threats.
  • Documenting threats at the expense of documenting vulnerabilities leaves the developer with no option than to "fix what's exploitable". Yes, those should be the top priority, but if a developer is only trained to remove Cross-site Scripting vulnerabilities, they don't really learn the other values of proper data handling.
I know that priority one should be to fix what is readily exploitable. However, there's virtue in writing code properly. Output filtering fixes more than just cross-site scripting. Not exposing database keys fixes more than privilege escalation.

Does this mean that I'm saying we should get rid of pen testers? No way! Pen testers make the screen shots that end up funding the fixes. But documenting to developers how to write good code is definitely a good thing.