News
App-Scanning Security Tools Fall Short
Vendor’s test finds no clear winner among app-scanning tools.
The good news is that application-scanning tools do indeed catch security bugs as advertised.
The unsettling news for dev shops relying on one to spot vulnerabilities is that in a recent experiment by Fortify Software Inc., app-scanning tools all found different bugs in the same program -- even tools that share a similar approach to testing.
No single tool, including Fortify's, found anywhere near all the bugs in the open source blogging and content-management app they were tested against. "This was the most startling result for us, and not at all what anyone on my team would've anticipated," says Jacob West, who manages Fortify's Security Research Group.
West says his team went into the experiment expecting to find some variation in reported bugs, but instead found almost no overlap at all from one tool to the next. A colleague was scheduled to present the results at the Defcon 15 hacker's conference in Las Vegas early this month.
Still, West says he's wary of drawing sweeping conclusions from an experiment of limited scope conducted on a single Web app. "But certainly you can immediately say you'd gain a higher level of security and find more bugs, at least in this specific app, if you used multiple tools," he adds.
For example, only one of the five tested tools -- two of which can run in more than one mode -- found a path-manipulation vulnerability that would let an attacker inject data into the host server file system. The vulnerability was so serious that a tested fuzzing tool, which pummels an app with a flurry of unexpected inputs, managed to crash the machine used in the experiment by inadvertently exploiting the bug.
"It succeeded in overwriting a critical Windows system file. The machine wouldn't boot when we tried to bring it back up," West says. "It really did bring the point home for us. Seeing that happen and thinking about the impact and the potential risk of not finding that vulnerability opened our eyes a little bit."
App-scanning tools seek to find and report security vulnerabilities through technology broadly classified as either static analysis, runtime analysis or binary instrumentation, the latter of which changes the app in order to monitor it, West says. Some of the tools tested in the experiment are fully automated, while others are designed to be run by a skilled operator.
A Fortify researcher verified all reported bugs; only those found to be legitimate security concerns were considered in comparing test results. The one vulnerability found by all five tools concerned simple cross-head scripting issues, West says.
"The really complete lack of overlap was surprising to us," he says, "and we spend a lot of time thinking about this stuff, so I think it'll be surprising to other people, too."