When a bad software update from the security firm CrowdStrike inadvertently caused digital chaos around the world last month, the first signs were Windows computers showing the Blue Screen of Death. As websites and services went down and people scrambled to understand what was happening, conflicting and inaccurate information was everywhere. Rushing to understand the crisis, longtime Mac security researcher Patrick Wardle knew that there was one place he could look to get the facts: crash reports from computers impacted by the bug.
“Even though I am not a Windows researcher, I was intrigued by what was going on, and there was this dearth of information,” Wardle tells WIRED. “People were saying that it was a Microsoft problem, because Windows systems were blue-screening, and there were a lot of wild theories. But actually it had nothing to do with Microsoft. So I went to the crash reports, which to me hold the ultimate truth. And if you were looking there you were able to pinpoint the underlying cause long before CrowdStrike came out and said it.”
At the Black Hat security conference in Las Vegas on Thursday, Wardle made the case that crash reports are an underutilized tool. Such system snapshots give software developers and maintainers insight into possible problems with their code. And Wardle emphasizes that they can particularly be a fount of information about potentially exploitable vulnerabilities in software—for both defenders and attackers.
In his talk, Wardle presented multiple examples of vulnerabilities he has found in software when the app crashed and he combed through the report looking for the possible cause. Users can readily view their own crash reports on Windows, macOS, and Linux, and they’re also available on Android and iOS, though they can be more challenging to access on mobile operating systems. Wardle notes that to glean insights from crash reports, you need a basic understanding of instructions written in the low-level machine code known as Assembly, but he emphasizes that the payoff is worth it.
In his Black Hat talk, Wardle presented multiple vulnerabilities he discovered simply by examining crash reports on his own devices—including bugs in the analysis tool YARA and in the current version of Apple’s macOS operating system. In fact, when Wardle discovered in 2018 that an iOS bug caused apps to crash anytime they displayed the Taiwanese flag emoji, he got to the bottom of what was happening using, you guessed it, crash reports.
“We revealed conclusively that Apple had acquiesced to demands from China to censor the Taiwanese flag, but their censorship code had a bug in it—ridiculous,” he says. “My friend who originally observed this was like, ‘My phone is being hacked by the Chinese. Whenever you text me it crashes. Or are you hacking me?’ And I said, ‘Rude, I wouldn’t hack you. And also, rude, if I did hack you, I wouldn’t crash your phone.’ So I pulled the crash reports to see what was going on.”
Wardle emphasizes that if he can find so many vulnerabilities just by looking at crash reports from his own devices and those of his friends, software developers need to be looking there, too. Sophisticated criminal actors and well-funded state-backed hackers alike are probably already getting ideas from their own crash reports. Over the years, news reports have indicated that intelligence agencies like the US National Security Agency do mine crash logs. Wardle points out that crash reports are also a valuable source of information for detecting malware, since they can reveal anomalous and potentially suspicious activity. The notorious spyware broker NSO Group, for example, would often build mechanisms into into their malware specifically to delete crash reports immediately upon infecting a device. And the fact that malware is often buggy makes crashes more likely and crash reports valuable to attackers as well for understanding what went wrong with their code.
“With crash reports, the truth is out there,” Wardle says. “Or, I guess, in there.”