How Microsoft Embraced 'Big Data'

It's no accident that staid, proprietary software giant Microsoft has opened itself up and embraced open source (and even competing) technologies, a trend perfectly exemplified by the adoption of "big data" and its flagship Apache Hadoop platform.

It comes down to people like Dave Campbell, with the interesting title of "technical fellow" at Microsoft. It results in products like HDInsight, described last week by Campbell as Hadoop on the cloud (Windows Azure), laptop and server.

Campbell was speaking at the Build 2012 developer conference in a presentation titled "Data Options in Windows Azure, What's a Developer to Do?" Attendees of his presentation received some insight into the process of how a huge, monolithic, bureaucracy-laden organization transforms itself with a view toward long-range competitiveness--or even survival--in a changing landscape.

"Part of my job is to figure out what the heck's happening and then what should we be doing about it," Campbell said. "The last year for me was, OK, we've got this story down. We understand what's going on in sort of the big data space. I wanted to schedule a couple of big talks ... and then I said I want to be able to talk to the techies, the CTOs, the guys who were doing the projects .... Then I wanted to get the story through all the way through [so] I can talk to the CIOs, who aren't going to fully appreciate the technology. The folks who were the hardest nut to crack, were the enterprise data warehouse guys, who had the feeling that 'none of this stuff is worth crap unless it's in my data warehouse. I got the one version of the truth.'"

Six or eight months ago, Campbell said, he stumbled on to a way he can quickly get through to all of the skeptics. "I said, 'That's a very, very fine version of the truth, and it's still super valuable.' I said, 'But in this new world, there's a version of the truth about what people are saying about your products and what they think about your brand and how that's playing out in the online world. And that's a different version of the truth.' And I said, 'Another version of the truth is available from all your operational systems, which are just spewing out data. We've just got data coming out of everything. And there's yet a third version of the truth or perspective that you can get out of those systems.' And I look at them and say, 'And if you're looking at one version of the truth, and even if you're doing better than the other guys, but you're not looking at those other two versions of the truth, and your competitors are, how are you keeping up?'"

The response, he said, is like "'oh, yeah,' and so even those guys ... and you actually see it. There's a lot of recognition of this now. And so, it is real. There is, of course, hype, but again, I've tried to arm you here. The trick in this, there's nothing magic. The magic is about deferring the modeling and to be able to do the information production in a very quick, efficient way, and having things sort of pop out that you can subsequently refine."

And so Campbell continues on his untiring mission to convince others of the correct way to a successful future, traveling the country and even the globe, as he pointed out in a demonstration showing how easily he "mapped" and "reduced" GPS tracking data of where he'd been in the past few months.

And judging from his presentation, he keeps long hours. One piece of his presentation, he pointed out, was done at 3:38 am. that day.

So if you want to get clues about where Microsoft may be heading in the future, you might want to keep track of Campbell and what he's saying. Strangely, though, I couldn't find him on Twitter, but he is on LinkedIn and blogs occasionally on TechNet.

It's also worth checking out the Build 2012 video. It goes into detail about how big data is easy for developers, who only have to worry about two functions: map and reduce. And a lot more.

Maybe too much. At one point, when he's summarizing how developers can choose which Windows Azure data storage option to use, he said that if attendees register what he had just said in the previous couple of minutes, "you're going to walk out of here smarter than most bloggers on this topic. Seriously."

Ouch!

What do you think of Microsoft's embrace of big data in general or HDInsight in particular? Please share your thoughts by commenting here or mailto:[email protected] dropping me a line.

Posted by David Ramel on 11/07/2012