Redmond Review

Fast Data at Redmond High

After the much-delayed release of SQL Server 2005 finally came to pass at the end of its namesake year, Microsoft made a promise to release a new version of SQL Server every two to three years, from that point on. The company made good on that promise with SQL Server 2008. And it looks set to make good on it again with SQL Server 2008 R2, expected sometime next summer.

Don't let the R2 moniker fool you: This release is no mere service pack. It will contain many new features as well as new components that are practically products in their own right. One of those new components is StreamInsight, which until recently was known by its code name, "Orinoco." StreamInsight is Microsoft's first entry into Complex Event Processing (CEP), a class of product that handles high-volume, low-latency data streams. In plain English, StreamInsight handles data feeds that are big and fast.

A Good Foundation, A Market Need
As I understand it, StreamInsight is written in native code, and thus runs "close to the metal," a quality that's essential in low-latency processing. As much as I love .NET and Windows Communication Foundation (WCF), you still need unmanaged code if you want to handle mission-critical, high-speed data. It would seem the StreamInsight team knows this and planned their architecture accordingly. So far, so good.

Meanwhile, .NET is supported as a client interface. In other words, .NET code can query the streams and process what's in them, and there's actually a LINQ provider for StreamInsight to facilitate this. This gives the best of both worlds: You can use the engine from your favorite programming environment and have that engine be written in fast, native code. So the good news continues.

That's why, when I learned about StreamInsight, I was pretty excited. I help run a technology consultancy in New York, a city chock-full of financial services clients who have applications that need StreamInsight's kind of speed. But we're not a C++ shop, and the middleware technologies we use today-namely WCF and BizTalk-don't have the ability to handle the kind of workloads StreamInsight is designed for.

And I know our firm's not alone. High-speed, high-volume data processing has applications in operational systems, business intelligence, audit and compliance, sales and marketing and numerous other lines-of-business and IT functions. There are plenty of firms that want to play in these spaces, but for many, the barrier to entry is too high. So StreamInsight could provide a real solution.

Questions Remain
What about integration between StreamInsight and other Microsoft technologies and products? For example, could BizTalk use StreamInsight, in some fashion, to process and buffer large, high-speed data streams? The BizTalk message bus provides reliability, but not in a low-latency fashion. Could StreamInsight mitigate this shortcoming and take advantage of the strength? StreamInsight, BizTalk and WCF all have adapter models. Are there adapters in the works that could make BizTalk or WCF into StreamInsight clients?

Certainly, BizTalk is a server that needs a low-latency engine. And it appears that StreamInsight will be a low-latency engine that lacks an industrial-strength server/host. These products would seem made for each other, so their integration must be imminent, right? I can't be sure. Despite earnest inquiry on my part with people on the relevant product teams, it's not clear that StreamInsight will make its way into, or federate with, BizTalk, Dublin, WCF or anything else. It isn't even really integrated with SQL Server; that's just the ship vehicle.

I find this all a tad worrisome. If Microsoft wants to compete against the likes of TIBCO in the financial services world, or any other industry where the data comes fast and furious, it needs something like StreamInsight. But StreamInsight needs a host, manageability and a UI, or it needs to be a value-added embedded component within products that have these features. Without such flesh and bones, StreamInsight will wither and die. Don't believe me? I have four words for you: SQL Server Notification Services.

I'm concerned that Microsoft is building something good and backing it up with marketing and planning that's bad. And I'm further worried that this is emblematic of bigger problems: deficient communication between product groups, lousy market analysis and a fascination with engineering that trumps commercial viability. I hope my worries are unfounded, and that StreamInsight has real insight behind it. But until I see tangible evidence of this, I'm inclined to expect otherwise.

About the Author

Andrew Brust is Research Director for Big Data and Analytics at Gigaom Research. Andrew is co-author of "Programming Microsoft SQL Server 2012" (Microsoft Press); an advisor to NYTECH, the New York Technology Council; co-moderator of Big On Data - New York's Data Intelligence Meetup; serves as Microsoft Regional Director and MVP; and is conference co-chair of Visual Studio Live!

comments powered by Disqus

Featured

  • Mastering Blazor Authentication and Authorization

    At the Visual Studio Live! @ Microsoft HQ developer conference set for August, Rockford Lhotka will explain the ins and outs of authentication across Blazor Server, WebAssembly, and .NET MAUI Hybrid apps, and show how to use identity and claims to customize application behavior through fine-grained authorization.

  • Linear Support Vector Regression from Scratch Using C# with Evolutionary Training

    Dr. James McCaffrey from Microsoft Research presents a complete end-to-end demonstration of the linear support vector regression (linear SVR) technique, where the goal is to predict a single numeric value. A linear SVR model uses an unusual error/loss function and cannot be trained using standard simple techniques, and so evolutionary optimization training is used.

  • Low-Code Report Says AI Will Enhance, Not Replace DIY Dev Tools

    Along with replacing software developers and possibly killing humanity, advanced AI is seen by many as a death knell for the do-it-yourself, low-code/no-code tooling industry, but a new report belies that notion.

  • Vibe Coding with Latest Visual Studio Preview

    Microsoft's latest Visual Studio preview facilitates "vibe coding," where developers mainly use GitHub Copilot AI to do all the programming in accordance with spoken or typed instructions.

  • Steve Sanderson Previews AI App Dev: Small Models, Agents and a Blazor Voice Assistant

    Blazor creator Steve Sanderson presented a keynote at the recent NDC London 2025 conference where he previewed the future of .NET application development with smaller AI models and autonomous agents, along with showcasing a new Blazor voice assistant project demonstrating cutting-edge functionality.

Subscribe on YouTube