Data Driver

Blog archive

How Microsoft Embraced 'Big Data'

It's no accident that staid, proprietary software giant Microsoft has opened itself up and embraced open source (and even competing) technologies, a trend perfectly exemplified by the adoption of "big data" and its flagship Apache Hadoop platform.

It comes down to people like Dave Campbell, with the interesting title of "technical fellow" at Microsoft. It results in products like HDInsight, described last week by Campbell as Hadoop on the cloud (Windows Azure), laptop and server.

Campbell was speaking at the Build 2012 developer conference in a presentation titled "Data Options in Windows Azure, What's a Developer to Do?" Attendees of his presentation received some insight into the process of how a huge, monolithic, bureaucracy-laden organization transforms itself with a view toward long-range competitiveness--or even survival--in a changing landscape.

"Part of my job is to figure out what the heck's happening and then what should we be doing about it," Campbell said. "The last year for me was, OK, we've got this story down. We understand what's going on in sort of the big data space. I wanted to schedule a couple of big talks ... and then I said I want to be able to talk to the techies, the CTOs, the guys who were doing the projects .... Then I wanted to get the story through all the way through [so] I can talk to the CIOs, who aren't going to fully appreciate the technology. The folks who were the hardest nut to crack, were the enterprise data warehouse guys, who had the feeling that 'none of this stuff is worth crap unless it's in my data warehouse. I got the one version of the truth.'"

Six or eight months ago, Campbell said, he stumbled on to a way he can quickly get through to all of the skeptics. "I said, 'That's a very, very fine version of the truth, and it's still super valuable.' I said, 'But in this new world, there's a version of the truth about what people are saying about your products and what they think about your brand and how that's playing out in the online world. And that's a different version of the truth.' And I said, 'Another version of the truth is available from all your operational systems, which are just spewing out data. We've just got data coming out of everything. And there's yet a third version of the truth or perspective that you can get out of those systems.' And I look at them and say, 'And if you're looking at one version of the truth, and even if you're doing better than the other guys, but you're not looking at those other two versions of the truth, and your competitors are, how are you keeping up?'"

The response, he said, is like "'oh, yeah,' and so even those guys ... and you actually see it. There's a lot of recognition of this now. And so, it is real. There is, of course, hype, but again, I've tried to arm you here. The trick in this, there's nothing magic. The magic is about deferring the modeling and to be able to do the information production in a very quick, efficient way, and having things sort of pop out that you can subsequently refine."

And so Campbell continues on his untiring mission to convince others of the correct way to a successful future, traveling the country and even the globe, as he pointed out in a demonstration showing how easily he "mapped" and "reduced" GPS tracking data of where he'd been in the past few months.

And judging from his presentation, he keeps long hours. One piece of his presentation, he pointed out, was done at 3:38 am. that day.

So if you want to get clues about where Microsoft may be heading in the future, you might want to keep track of Campbell and what he's saying. Strangely, though, I couldn't find him on Twitter, but he is on LinkedIn and blogs occasionally on TechNet.

It's also worth checking out the Build 2012 video. It goes into detail about how big data is easy for developers, who only have to worry about two functions: map and reduce. And a lot more.

Maybe too much. At one point, when he's summarizing how developers can choose which Windows Azure data storage option to use, he said that if attendees register what he had just said in the previous couple of minutes, "you're going to walk out of here smarter than most bloggers on this topic. Seriously."


What do you think of Microsoft's embrace of big data in general or HDInsight in particular? Please share your thoughts by commenting here or dropping me a line.

Posted by David Ramel on 11/07/2012 at 1:15 PM

comments powered by Disqus


  • What's New in Visual Studio 2019 v16.5 Preview 2

    The second preview of Visual Studio 2019 v16.5 has arrived with improvements across the flagship IDE, including the core experience and different development areas such as C++, Python, web, mobile and so on.

  • C# Shows Strong in Tech Skills Reports

    Microsoft's C# programming language continues to show strong in tech industry skills reports, with the most recent examples coming from a skills testing company and a training company.

  • Color Shards

    Sharing Data and Splitting Components in Blazor

    ASP.NET Core Version 3.1 has at least two major changes that you'll want to take advantage of. Well, Peter thinks you will. Depending on your background, your response to one of them may be a resounding “meh.”

  • Architecture Small Graphic

    Microsoft Ships Preview SDK, Guidance for New Dual-Screen Mobile Era

    Microsoft announced a new SDK and developer guidance for dealing with the new dual-screen mobile era, ushered in by the advent of ultra-portable devices such as the Surface Duo.

  • How to Create a Machine Learning Decision Tree Classifier Using C#

    After earlier explaining how to compute disorder and split data in his exploration of machine learning decision tree classifiers, resident data scientist Dr. James McCaffrey of Microsoft Research now shows how to use the splitting and disorder code to create a working decision tree classifier.

.NET Insight

Sign up for our newsletter.

Terms and Privacy Policy consent

I agree to this site's Privacy Policy.

Upcoming Events