Redmond Review

Fast Data at Redmond High

• By Andrew J. Brust
• 11/01/2009

After the much-delayed release of SQL Server 2005 finally came to pass at the end of its namesake year, Microsoft made a promise to release a new version of SQL Server every two to three years, from that point on. The company made good on that promise with SQL Server 2008. And it looks set to make good on it again with SQL Server 2008 R2, expected sometime next summer.

Don't let the R2 moniker fool you: This release is no mere service pack. It will contain many new features as well as new components that are practically products in their own right. One of those new components is StreamInsight, which until recently was known by its code name, "Orinoco." StreamInsight is Microsoft's first entry into Complex Event Processing (CEP), a class of product that handles high-volume, low-latency data streams. In plain English, StreamInsight handles data feeds that are big and fast.

A Good Foundation, A Market Need
As I understand it, StreamInsight is written in native code, and thus runs "close to the metal," a quality that's essential in low-latency processing. As much as I love .NET and Windows Communication Foundation (WCF), you still need unmanaged code if you want to handle mission-critical, high-speed data. It would seem the StreamInsight team knows this and planned their architecture accordingly. So far, so good.

Meanwhile, .NET is supported as a client interface. In other words, .NET code can query the streams and process what's in them, and there's actually a LINQ provider for StreamInsight to facilitate this. This gives the best of both worlds: You can use the engine from your favorite programming environment and have that engine be written in fast, native code. So the good news continues.

That's why, when I learned about StreamInsight, I was pretty excited. I help run a technology consultancy in New York, a city chock-full of financial services clients who have applications that need StreamInsight's kind of speed. But we're not a C++ shop, and the middleware technologies we use today-namely WCF and BizTalk-don't have the ability to handle the kind of workloads StreamInsight is designed for.

And I know our firm's not alone. High-speed, high-volume data processing has applications in operational systems, business intelligence, audit and compliance, sales and marketing and numerous other lines-of-business and IT functions. There are plenty of firms that want to play in these spaces, but for many, the barrier to entry is too high. So StreamInsight could provide a real solution.

Questions Remain
What about integration between StreamInsight and other Microsoft technologies and products? For example, could BizTalk use StreamInsight, in some fashion, to process and buffer large, high-speed data streams? The BizTalk message bus provides reliability, but not in a low-latency fashion. Could StreamInsight mitigate this shortcoming and take advantage of the strength? StreamInsight, BizTalk and WCF all have adapter models. Are there adapters in the works that could make BizTalk or WCF into StreamInsight clients?

Certainly, BizTalk is a server that needs a low-latency engine. And it appears that StreamInsight will be a low-latency engine that lacks an industrial-strength server/host. These products would seem made for each other, so their integration must be imminent, right? I can't be sure. Despite earnest inquiry on my part with people on the relevant product teams, it's not clear that StreamInsight will make its way into, or federate with, BizTalk, Dublin, WCF or anything else. It isn't even really integrated with SQL Server; that's just the ship vehicle.

I find this all a tad worrisome. If Microsoft wants to compete against the likes of TIBCO in the financial services world, or any other industry where the data comes fast and furious, it needs something like StreamInsight. But StreamInsight needs a host, manageability and a UI, or it needs to be a value-added embedded component within products that have these features. Without such flesh and bones, StreamInsight will wither and die. Don't believe me? I have four words for you: SQL Server Notification Services.

I'm concerned that Microsoft is building something good and backing it up with marketing and planning that's bad. And I'm further worried that this is emblematic of bigger problems: deficient communication between product groups, lousy market analysis and a fascination with engineering that trumps commercial viability. I hope my worries are unfounded, and that StreamInsight has real insight behind it. But until I see tangible evidence of this, I'm inclined to expect otherwise.