Q&A

Empowering AI Applications with Vector Search in SQL Server and Azure Cosmos DB

• By David Ramel
• 05/27/2025

As developers look to harness the power of AI in their applications, one of the most exciting advancements is the ability to enrich existing databases with semantic understanding through vector search. Traditionally the domain of specialized AI or search infrastructure, vector-based querying is now making its way into familiar data platforms like SQL Server and Azure Cosmos DB, unlocking new capabilities for developers without the need to adopt an entirely new tech stack.

At the heart of this movement is Retrieval Augmented Generation (RAG) -- an increasingly popular pattern for improving the relevance and contextual quality of responses in AI applications. By combining vector embeddings with natural language queries, RAG enables developers to build intelligent, search-driven interfaces that feel truly conversational. And with Microsoft recently adding native support for vector data types and indexing to both SQL Server and Azure Cosmos DB, these AI capabilities are now closer than ever to where developers already store and manage their domain data.

In his upcoming session titled "Empowering AI Applications with Vector Search in SQL Server and Azure Cosmos DB" at Visual Studio Live! @ Microsoft HQ in Redmond, longtime MVP and Sleek Technologies CTO Leonard Lobel will show attendees how to tap into these capabilities without leaving the environments they already know. The session aims to help developers build smarter, more responsive applications using integrated vector search and embedding techniques -- backed by real-world demos and implementation tips.

We spoke with Lobel to learn more about how developers can bring vector search and RAG into their projects using tools they already rely on every day, and how developers can learn more about the tech and prepare for his session.

VisualStudioMagazine: What inspired you to present a session on this topic?
Lobel: I've always been passionate about bridging the gap between established database technologies and emerging AI capabilities. With the advent of RAG, I recognized an for developers to bring natural language intelligence into their data-driven applications -- without leaving the familiarity of SQL Server or Azure Cosmos DB, and without requiring deep knowledge in machine learning or mathematical AI concepts. The idea that we can empower AI experiences right from within these platforms -- using native vector search and embedding capabilities -- is definitely a message worth amplifying.

I want attendees to walk away knowing they don't need a separate "AI stack" to get started with modern generative applications.

Can you explain how Retrieval Augmented Generation (RAG) enhances AI applications within SQL Server and Azure Cosmos DB?
Your domain data is already being stored in your relational (SQL Server) or non-relational (Azure Cosmos DB) database. You're already leveraging native querying capabilities for traditional search, along with robust management and security features. So it only makes sense to continue enjoying all these benefits -- now enhanced with integrated vector search and vector indexing in both platforms -- by applying a Retrieval Augmented Generation (RAG) pattern directly over your database. RAG enables natural language user queries to retrieve results that are semantically relevant to natural language queries posed by users. The result is an AI-assisted experience that provides more accurate, context-aware answers. And because this all happens within your existing data environment, there's no need for a separate vector database. That simplifies your architecture while keeping your AI pipeline secure, maintainable, and integrated with the rest of your data strategy.

What are the steps involved in generating and storing text embeddings using Azure OpenAI for integration with SQL Server?
All you need to do is provision an Azure OpenAI resource and deploy one of its available text embedding models. Options include text-embedding-3-large, text-embedding-3-small, and text-embedding-ada-002, where you'll want to experiment with each to find the best fit for your scenario. Once you've selected a model, you can serialize each entity -- along with its related child entities -- into a single JSON document, and then call Azure OpenAI to vectorize that text. The model returns a vector: an array of floating-point numbers that captures the semantic meaning of the input. In SQL Server, this vector is stored in a column using the new native VECTOR data type. What's especially powerful is that SQL Server supports making REST API calls directly from T-SQL, enabling end-to-end embedding workflows entirely within the database engine.

How does vector indexing improve the performance of similarity searches in Azure Cosmos DB?
When a user poses a natural language question to your application, you want to find the most semantically meaningful results by issuing a similarity search. This involves vectorizing the query text using Azure OpenAI -- the same way you vectorized your JSON documents in the database -- and then comparing it against stored vectors in the database. But since each vector is a high-dimensional float array, performing brute-force similarity comparisons across a large dataset can become prohibitively expensive, which is where vector indexing comes in. Azure Cosmos DB supports high-performance vector indexing, so that you can retrieve the most relevant vectors in milliseconds, even across massive datasets, promoting scalable and responsive AI experiences.

What are the considerations for choosing between Azure Cosmos DB for NoSQL and Azure Cosmos DB for MongoDB vCore when implementing vector search?
It really depends on your preferred APIs and workloads. Cosmos DB for NoSQL is best suited if you're already using the native Cosmos SDKs or leveraging multi-region distribution, partitioning, and low-latency reads at global scale. It offers built-in support for the vector data type and multiple index strategies. Cosmos DB for MongoDB vCore supports vector search using MongoDB-compatible syntax ($vectorSearch) and is ideal for teams familiar with MongoDB tools and drivers. It's also backed by a dedicated vCore architecture, for a scale-up versus scale-out database. If you're starting from scratch, Cosmos DB for NoSQL offers deeper integration. But if you already have MongoDB expertise or codebases, the vCore API provides a fast onramp to vector search without significant rework.

What are the potential challenges in implementing vector search in existing databases, and how can they be mitigated?
Common implementation challenges include managing schema changes, controlling cost, and optimizing queries. Particularly in the case of SQL Server, introducing vector columns may require schema evolution, which can be disruptive. Of course Azure Cosmos DB is schema-free, meaning that while there is schema, there is no enforced schema and no schema management, which mitigates this concern. Furthermore, vectorizing large datasets can be resource-intensive if not planned carefully. Plus, similarity queries can slow down without vector indexes, especially at scale. These challenges can be addressed by incrementally adopting vector search -- starting with your most valuable or frequently queried content -- and by batching and caching embeddings to control costs. You'll also want to use native indexing and similarity functions to avoid performance bottlenecks.

What resources can attendees use to learn more about this topic and prepare for your session?
Here are a few great starting points:

• Microsoft Learn -- There's excellent material on Azure OpenAI and integrating AI into your apps
• Azure Cosmos DB documentation -- Especially the sections on vector searching, vector indexing, and performance tuning
• SQL Server documentation -- Covers the new vector data type and upcoming AI-related features
• My Multi-Database RAG Solution on GitHub

And of course, nothing beats being there in person -- my session will walk through real examples with live demos and explain the "why" as well as the "how" behind each design choice.

Note: Those wishing to attend the session can save money by registering early, according to the event's pricing page. "Save $400 when you register by the June 6 deadline," said the organizer of the event, which is presented by the parent company of Visual Studio Magazine.