News

Azure Data Lake Store, Services and U-SQL Make Debut

Microsoft fills in the gaps on its Big Data offering with previews of its Azure Data Lake and Language Services, which includes the brand-new U-SQL language for crunching the incoming explosion of data.

• By Michael Domingo
• 09/28/2015

Microsoft announced updates to key technologies in its Big Data portfolio, as well as new offerings -- specifically, Azure Data Lake and Language services -- for developers to wrap their code around. Scott Guthrie, executive vice president of the Microsoft Cloud and Enterprise group, announced them on his blog the day before the company's online AzureCon online conference taking place this week.

"Azure Data Lake Store solves the big data challenges of volume, variety, and velocity by enabling you to store data of any type, at any size, and process it at any scale," writes Guthrie. The gist of his announcement calls out the proliferation of data sources, including the immense real-time data crunching involved with Internet of Things, which the new hyper-scale HDFS repository was built to gobble up and manage.

At the heart of the Azure Data Lake Store, which is currently in preview mode, is the Azure Data Lake Analytics service that is used to manage and crunch data in ADLS repositories right from within the Azure Management portal. And the new jewel in the rough is U-SQL, which appears to be a SQL language superset that adds some flexibility for dealing with data at scale.

"We designed U-SQL from the ground-up as an evolution of the declarative SQL language with native extensibility through user code written in C#," said Michael Rys, Principal Program Manager, Microsoft Big Data, in a separate blog that offers up more detail on the language. "This unifies both paradigms, unifies structured, unstructured, and remote data processing, unifies the declarative and custom imperative coding experience, and unifies the experience around extending your language capabilities."

Rys writes that U-SQL is based on his group's experiences working internally with SCOPE (a declarative and extensible scripting language for analyzing large massive data sets and actually borrows from SQL in the way it lays out the data prior to analysis) as well as the T-SQL, ANSI-SQL and Hive languages, and can be extended directly through C# code. Visual Studio developers will be able to tap right into U-SQL through the Aure Management portal with new tools to come that will allow for debugging and optimizaton of U-SQL jobs natively.

Some of the commenters to Rys's blog question U-SQL's ability to scale, citing the lack of examples. "How well does [U-SQL} scale? Say you wanted to run an algorithm on 100 TB of data, or a PB of data? Is that a use case this is designed for? Or, say, if you need a teraflop of computing power on an algorithm on 10 TB of data, is that a possibility?" asks Xsistor.

"Assuming you are adding enough processing capacity, U-SQL can scale over very large files. There will be a demo by us later this week that shows a bit more data than my simple example above (that was just highlighting the language functionality)," responded Rys.

Guthrie also announced that its HDInsight service, another component of the Azure Data Lake Services toolset that's generally available now, has support for Ubuntu Linux. In the Data + Analytics section of the Azure Management portal is an Ubuntu option, which when selected allows for creation of Linux-based HDInsight clusters. Besides Hadoop, it also supports "clusters pre-configured for workloads like Storm, Spark, HBase, etc.," writes Guthrie. Hadoop clusters can then be managed via Apache Ambari, and those clusters can be accessorized with Hadoop components or other apps beyond the default ones installed under HDInsight.

More details will be revealed during this week's AzureCon.