Redmond Review

Data Platform Innovation, Without Disruption

What's next on the data frontier? Microsoft will take us into the world of in-memory transactional databases and big data. And if you know how to write code and queries against SQL Server, then you already have the necessary skills to play there.

• By Andrew J. Brust
• 01/02/2013

I've always been impressed by Microsoft's dedication to bringing new technologies to its developers, while making those technologies available through existing interfaces and within existing skill sets. For an example, you can go back to the 1990s, when Microsoft brought Visual Basic (and Visual Basic developers) to Web browser and server development. You can also look at more recent offerings, such as building RESTful services with the new ASP.NET Web API, or even the development of Windows Store applications using HTML and JavaScript.

The data side has followed this paradigm, too. There's been churn of data-access APIs over the years, but each one has been impressively versatile. DAO worked for Access and ODBC. ADO worked with Exchange message stores almost as easily as it did with SQL Server. And today LINQ works with everything, including business objects, Internet data feeds, client/server databases and even directory services.

What's next on the data frontier? Microsoft will take us into the world of in-memory transactional databases and big data. And if you know how to write code and queries against SQL Server, then you already have the necessary skills to play there.

New Technologies, Same Database
At the PASS Summit in Seattle this past November, Microsoft announced two new projects: Hekaton and PolyBase. Hekaton is a transactional (rather than analytical) in-memory database technology for the next major version of SQL Server. PolyBase integrates Hadoop into the upcoming release of SQL Server Parallel Data Warehouse edition. What's in it for developers? The same Transact SQL (T-SQL) language and data-access APIs used with SQL Server today will work with both of these new technologies.

Hekaton will allow database administrators (many of whom are also developers these days) to design tables to be loaded into memory, where they can then be queried and updated. Hekaton won't be a standalone product -- in fact, in-memory tables and disk-based tables will exist side-by-side in the same SQL Server databases. Applications that query them will in many cases be able to do so agnostically, with developers not even having to know which type of table they're working with.

PolyBase will also allow for the creation of special tables, but these will have a physical basis in files stored in the Hadoop Distributed File System (HDFS). Because the data in these files will behave like physical tables in SQL Server, developers can query them with T-SQL (not MapReduce code), and even join them to conventional tables in their queries.

In the case of both Hekaton and PolyBase, existing developers with existing skill sets will be almost immediately qualified to build new applications that take on very different workloads, for a different world of data.

Should I Stay or Should I Go Now?
I first started working with SQL Server in 1994. It astounds me how a technology that I became familiar with nearly 20 years ago is still so useful today. Is that a bad thing? SQL Server has acquired layers of features over the years as new technologies have appeared on the scene, much as a rental apartment accumulates layers of paint with each new tenant. In the latter case, there's always the temptation to scrape off the paint, go down to the brick and apply the new paint there -- or just move somewhere new.

Some SQL Server developers have moved on to places like MongoDB, DynamoDB or Hadoop. And just as some people dream of owning a mansion, some developers may hope to land work with SAP HANA, an exclusively in-memory database that certainly doesn't come cheap.

But many are happy to stay in their old house, adding extensions, upgrading the wiring and maybe even redecorating once in a while. So it is with many SQL Server pros who keep reinvesting in that platform. They're spared the expense of "moving," and they get to know the ins and outs of their "home" really well -- though they do lose out on the adventure of going somewhere new.

Businesses value savings and experience. Avoiding new recruit­ment or massive retraining, preventing a change of vendors and the procurement adjustments that go with that, and maintaining the continuum of older applications with newer ones are good things for customers. Microsoft knows this well. And while the industry's attention is almost exclusively focused on the client OS and devices, it loses touch with the importance of back-end technology to platform strength and stability. Microsoft must focus on both. On the server side, it's doing very well.