False Positives vs. Non-Exploitables

Another professional and I sat down to coffee a few weeks ago, and we got around to a conversation we seem to have often. I feel almost ashamed to publish a post on it when there's no "other side of the argument" here to defend itself. So please bear with me as I try to do both arguments justice (and tell you why my side is right).

Consumers of security tools have long asked that the tool reduce the number of false positives. And this is certainly understandable - it's very difficult to know what to fix when the list of things that are broken includes so many things that aren't broken. All areas of security try to sell you on their perfect false positive to false negative ratio. Theoretically, the two are on separate paths that at some point intersect - as the sensitivity of the tool is increased, false positives go up, false negatives go down. As the sensitivity is relaxes, false positives go down, false negatives go up. The theoretical ideal of the two is where the two are equal - this would be the lowest sum total of the two.

Tool vendors have to make their customers happy. And false positive to false negative ratio is one metric that all vendors use to prove the value of their tool. None of the tools out there use the same taxonomy, result format, reporting, etc., so the only apples-to-apples comparison we have is false positive to false negative ratio.

Now, the tool vendors work very hard to get the false positives to decrease, and depending on the type of tool, they do it varying ways. Application firewalls may use anomaly detection rather than flagging everything with known-bad data. Static analysis tools build a call tree to see if the data is sanitized anywhere in the process, and if so, it's not reported. Dynamic analysis tools don't report "evidence of", they report real exploits.

This is both good and bad. The good side is that because false positives are low, when something is reported, you can be pretty sure it needs to be fixed. When a static analyzer finds an injection flaw, you know that the data goes soup to nuts without being properly checked or encoded. When a pen testing tool finds something you know it didn't find those pesky points where you got back an angle bracket, but just couldn't get a real injection attack.

However, I think that false positives have been confused with non-exploitables. When pen-testing, those pesky points where you get back a real angle bracket, but can't get a real injection to work, although there's no injection, you have sufficient proof that encoding is not taking place. And in static analysis, if you see a place where data goes to a "presentation layert" (database call, screen, wire, LDAP server, command shell, etc.) without being properly encoded, you have a real flaw, even though it's not exploitable.

There are two primary reasons that not reporting non-exploitables is dangerous:

1. Although the point is not exploitable today, there's a false safety net keeping it from being exploitable - a string length restriction that could change tomorrow, or an input policy that happens to be checked today, but when the same input is accepted from another source tomorrow, the same checks don't take place.
2. When asked to fix problems, developers must take the results of the test to make their decisions on what to fix. If output is not encoded in two places, and only one is reported, how will the developer learn what to fix by output encoding? They'll learn to fix the ones the tool reports. So how so developers learn to code correctly? They don't - they learn to code the way they always do, then count on the tool to correct them later. Even if it's not exploitable, non output-filtered code should be flagged (possibly rated differently in severity).

So for the security practitioners, do your developers a favor and document those "uncomfortable places" where you couldn't get a really good exploit to work, but you know there's something not behaving properly. When you're doing a manual code review, flag all of those places where data goes to a presentation layer without actual encoding.

And developers, do yourselves a huge favor and when a report comes back with flaws, demand that whoever made the report (your tool, your security team, your peer reviewer), explain to you why it's a problem. Don't simply fix that issue. Discover how you fixed it, and how you can apply the same fix to other locations that weren't reported to you. And more importantly, make that fix a part of your programming habits, so that you just write it that way in the future.