Q&A

ETL, KQL and RTI: Harnessing Data in Motion with Microsoft Fabric

• By David Ramel
• 09/16/2025

As the volume, velocity, and variety of data continue to accelerate, developers are facing a critical shift: data is no longer just stored and queried--it's constantly on the move. From traditional Extract, Transform, Load (ETL) processes to event-driven architectures, the need to harness and act on real-time information is becoming less of a technical luxury and more of a business imperative.

That's where Microsoft Fabric and Real-Time Intelligence (RTI) come in. Fabric offers a unified platform that brings together batch and streaming data pipelines under one roof, while RTI provides the tools needed to process, analyze, and act on data as it happens. And with the growing accessibility of tools like Kusto Query Language (KQL), working with real-time telemetry and operational signals is becoming simpler and more approachable--even for teams new to streaming.

In their full-day workshop at Live! 360 Orlando, Matt Gordon, Microsoft Data Platform MVP, will join Microsoft's Christopher Schmidt to walk attendees through how to navigate this rapidly evolving space.

Their workshop, ETL, KQL, and RTI - Harnessing Data in Motion with Microsoft Fabric, is designed to give developers, analysts, and data professionals the practical guidance they need to rethink traditional ETL pipelines, understand the real-time capabilities of Microsoft Fabric, and get hands-on with KQL--all with an emphasis on solving real-world business problems.

We caught up with Gordon and Schmidt to get a preview of what attendees can expect from the workshop, why this topic matters now more than ever, and how to start experimenting with Fabric and KQL today.

VisualStudioMagazine: What inspired you to present a session on this topic?
Gordon: Helping people understand how much of their data is truly in motion (whether that is through traditional ETL, change data capture from relational databases, or other sources) helps them understand all the different ways they can analyze and visualize that data quickly. There are so many new capabilities and components in this space, especially with the advent of Fabric, that we feel like a full-day session talking about real customer and field experiences of ours will let our attendees know what their options are as far as handling data in motion in their environment. If you know all your options, you're that much closer to making the best decision for your data -- and your business.

Schmidt: As Matt mentions, getting developers and analysts to understand how much of their data is truly in motion is a key inspiration. Batch processing technologies were great because that's what we had available. With the ease in which streaming technologies can be implemented in Fabric, this fundamentally changes the paradigm on how easy it is to move past slow refresh cycles and act on things as they happen in real time. Not just for making data available in reports, but moving data to be actionable. That's the key to me, make data something that delivers genuine value to the business by integrating with their daily flow.

What's one challenge people run into when working with streaming data for the first time?
Gordon: The challenge has typically been trying to understand the variety of APIs, protocols, and services that needed to be understood when we were programmatically putting together a streaming data architecture in the past. With no-code/low-code ways to handle this data in Fabric, the learning curve is dramatically shorter, which leaves much more time to analyze, visualize, and take actions based on this data.

Schmidt: Thinking this is something I don't need because I'm building a report. Streaming data really changes the entire paradigm fundamentally. Although there are new tools and technologies to learn, it's not as complicated as you may think. It's really just about breaking the problem down into more of a row mentality instead of trying to move files across a network. It's a text message if you will, not the entire conversation. With the simplicity in which these types of technologies can be implemented in Fabric, you don't have to be an expert in streaming technologies to start taking advantage of some of these tools. That unlocks new capabilities far beyond reporting.

How does KQL differ from T-SQL in how you approach data problems?
Gordon: As somebody who has been working with SQL Server and Azure SQL for more years than I'd like to admit, the best explanation of how KQL differs from T-SQL is that KQL is at its best when you write the query in the logical way that the SQL Server query optimizer has traditionally executed your T-SQL. Once I realized the differences between the languages were more intuitive than I thought, I really begin enjoying learning and writing KQL.

Schmidt: It's a mindset shift. KQL is optimized for interactive, ad-hoc exploration of time series data, whereas T-SQL is built for transactional and relational workloads. When I think of how we approach a traditional analytics query in T-SQL, I think in terms of joins, transactions, constraints, indexes, and all the other items that come with it. This means that certain considerations had to be made in traditional time-series/data warehouse type scenarios on things like slowly changing dimensions, facts, etc. With KQL, the focus shifts to querying large volumes of data quickly. You're looking for time series based insights, it's less about modifying the data and enforcing constraints and conditions on the data and more about surfacing patterns and the event (fact). I still use SQL most days, but I use KQL too.

What role does Microsoft Fabric play in unifying the ETL and real-time processing experience?
Gordon: Fabric really allows us to do all of that but I think one of the key takeaways of this session is that we don't necessarily need to break things down between traditional ETL (batch) and real-time, streaming processing. When we picture this through the lens of a modern, truly event-driven architecture, much of our data is just data in motion. It's just sometimes in motion at a variety of speeds - but Fabric gives us the variety of tools we need to unify this event-driven experience.

Schmidt: Plus one. With Fabric, for the first time we don't have one tool that everything must conform too. We have a toolbox available to us, and this allows us to think about the problem much more critically. Whereas previously, I typically had to be ETL or streaming, now I can leverage the right tool in Fabric for the right data store. Since it's all going to Onelake though, I have a single, unified plane where all that data is landing. Data can be streamed into Real Time Intelligence and easily joined with reference data. What's really challenging in the Fabric space honestly is how the industry has different terms for the same thing i.e. is it a transaction? A fact? An event? Same thing, different words for different folks. But this makes Fabric a bit of a challenge because you have to reconcile these terms.

Why is real-time insight becoming essential instead of optional?
Gordon: I think many of us feel like the speed of modern life has sped up substantially and, for better or worse, this "I need it now" attitude is certainly something I'm seeing professionally as well. As the performance of modern data platforms and tools continues to push the boundaries of what's possible (and marketing departments continue to put together demos showing data ingestion to insights in seconds), management will continue to expect the time-to-insight from their data to get faster. It's up to us as data professionals to communicate effectively about when that's a worthy goal and investment -- and when it's not.

Schmidt: In short, AI. Data is the fuel for AI driven decisions. As we move into an agentic future and the ability to have data operate at the speed of business, we want to be able to gather that data from our environment as it happening, not in scheduled refreshes. Anomaly detections, for example, are useful in the moment. Telling you yesterday there was an anomaly and you should have acted is much different than telling you something is happening now and you should act on it. Act before your opportunity is gone. Many insights in the business world are valuable in this moment - fraud detection, anomaly alerts, customer behavior signals are a few examples. Being able to harness these events in the moment enables automated workflows, better performing systems, and dynamic resource allocation. Things that are simply not possible with data today.

What's a simple way to get hands-on with Fabric and KQL before this workshop?
Gordon: If you don't have access to a paid capacity at work or school, you can use a personal email to sign up for a 60-day Fabric trial that will let you get hands-on with Real-time Intelligence in Fabric (and all the other workloads as well).

Schmidt: Absolutely! Sign up for a Fabric trial, If you are looking to get hands on with KQL directly, you can also use the free Kusto cluster made available as a part of the Azure Data Explorer offering. It doesn't have all the capabilities in Fabric, but if you just want to get your hands dirty with KQL it's a great place to start. There is also a gamified version of the learning that our product group puts out called the Kusto Detective Agency. It's a fun way to learn!

How can attendees learn more about this topic and prepare for your session?
Gordon: Honestly, Christopher's passion for this topic of event-driven architecture and data in motion mean he's been turning out some really insightful blogs and LinkedIn posts on this topic. In terms of general familiarity with the capabilities of Real-time Intelligence, I would recommend the bike rentals sample data (more information here). I think that sample does a solid job of providing relatable data in a sample that still exercises the versatility of the different Real-time Intelligence components, especially Real-Time Dashboards and Activator.

Schmidt: "Head over to the Microsoft Fabric Real Time Intelligence home page! While I appreciate the flattery from Matt, I'm just super passionate about event driven architecture and solving business problems. On LinkedIn I author the Real Time Dispatch newsletter, which contains a new article and Q&A topic weekly on streaming data and event driven architectures. The only thing you really need to prepare for the session is a desire to solve business problems and work with your data in motion. It's not nearly as intimidating as it seems."

Note: Those wishing to attend the session can save money by registering early, according to the event's pricing page. "Save $400 when you register by the Super Early Bird Savings deadline of Sept. 26," said the organizer of the event, which is presented by the parent company of Visual Studio Magazine.