HDInsight Gets Hadoop Upgrade

Microsoft today announced its cloud-based Hadoop service, HDInsight, now supports Hadoop 2.4, the latest version of the Big Data software.

Unveiled in October 2012, HDInsight exemplifies Microsoft's embrace of the Big Data movement and -- more generally -- its increasing involvement in open source technologies of all kinds. Microsoft partners with Hadoop heavyweight Hortonworks Inc. to provide the 100 percent Hadoop-compatible service on its Microsoft Azure platform, based on the Hortonworks Hadoop distribution.

Apache Hadoop 2.4, the latest update of the open source framework that's synonymous with Big Data, was released in April with enhancements to the often-criticized Hadoop Distributed File System (HDFS). The latest release also includes improvements to YARN -- sometimes referred to as "yet another resource negotiator" -- which is also described as the successor to the even-more-criticized MapReduce technology, a key component of the original Hadoop ecosystem. Various industry efforts aim to improve upon the constraints of the batch-oriented MapReduce with more modern analytics features such interactive queries on streaming data. YARN offers more interaction patterns with HDFS data and provides a more generalized processing platform beyond the MapReduce technology.

"This update includes interactive querying with Hive using advancements based on SQL Server technology, which we are also contributing back to the Hadoop ecosystem through project Stinger," Microsoft said in an announcement on the SQL Server Blog. "With this update to HDInsight, customers can use the speed and scale of the cloud to gain a 100x response time improvement."

Hive is a Hadoop-based data warehousing project also under the auspices of the Apache Software Foundation that allows data queries with its own SQL-like language. Stinger is a community project shepherded by Hortonworks to improve upon Hive with faster performance, increased scale and broader SQL support.

As noted by Oliver Chiu on the Microsoft Azure Blog, HDInsight is also getting an easy-to-use Web UI, letting developers graphically query Hive data.

The SQL Server team used the HDInsight announcement to highlight Microsoft's growing interaction with the open source community.

HDInsight clusters and Azure Blob Storage
[Click on image for larger view.] HDInsight Clusters and Azure Blob Storage
(source: Microsoft)

"We have fully embraced the Hadoop ecosystem and have prioritized contributing back to the community and Apache Hadoop-related projects, for example, Tez, Stinger and Hive," the post said. "All told, we've contributed 30,000 lines of code and put in 10,000-plus engineering hours to support these projects, including the porting of Hadoop to Windows. We've done this in partnership with Hortonworks, a relationship that ensures our Hadoop solutions are based on compatible implementations of Hadoop. One of the results of that partnership is the engineering work that has led to the Hortonworks Data Platform for Windows and Azure HDInsight."

The news came during the ongoing Hadoop Summit, at which T. K. Rengarajan, Microsoft corporate vice president of Data Platform, delivered the keynote address today.

About the Author

David Ramel is an editor and writer for Converge360.

comments powered by Disqus


  • Creating a Progressive Web App with Blazor WebAssembly

    Not surprisingly, it's dead easy to create an app in Blazor that runs outside of the browser window and (potentially) in an offline mode. Before you get carried away, though, there are some key design decisions to make.

  • GitLab Takes Over VS Code Extension, Plans Improvements

    DevOps specialist GitLab has officially taken over the control of a GitLab extension for Microsoft's open source, cross-platform Visual Studio Code editor.

  • VS Code Python Tool Now Does Native Notebooks

    The Python Extension for VS Code Insiders team is previewing the newest implementation of notebooks, used frequently in data science with offerings such as Jupyter Notebooks.

  • As .NET 5 Nears, Content/Documentation Reorganization Starts

    A GitHub project is seeking to reorganize documentation and developer content in advance of the November debut of .NET 5, a unification of all things .NET that combines. .NET Core and other components.

  • Windows Devs Get Cross-Platform Page, Issues Repo

    Developers doing their coding on the Windows OS have received two new resource gifts from Microsoft: a new landing page for those using cross-platform technologies and a new GitHub repo with which to report issues to Windows engineering teams.Developers doing their coding on the Windows OS have received two new resource gifts from Microsoft: a new landing page for those using cross-platform technologies and a new GitHub repo with which to report issues to Windows engineering teams.

.NET Insight

Sign up for our newsletter.

Terms and Privacy Policy consent

I agree to this site's Privacy Policy.

Upcoming Events