Foundry IQ for Agentic Retrieval: Intro to enterprise RAG

Introduction

Hey!

In today’s post, I would like to discuss Foundry IQ, which in conjunction with Work IQ and Fabric IQ, allows you to build powerful agents that have full context about your organization (Microsoft IQ intelligence layer).

Foundry IQ is related to the concept of RAG and Context Engineering in general, so if RAG-related concepts are new to you, I encourage you to read these articles first:

• Naive RAG Explained: The Core Pattern
• Hybrid Search in RAG: BM25 + Vectors for Better Retrieval
• Semantic Ranking in Azure AI Search: How Cross-Encoders Improve RAG Retrieval
• Multi-Query Retrieval for RAG: Query Rewrites in Azure AI Search
• HyDE for RAG in Azure: Improve Retrieval with Hypothetical Embeddings
• Chunking Strategies for RAG with C#: Fixed-Size, Semantic, Parent-Child and more
• Contextual Retrieval for RAG: Additional Context to Boost Accuracy
• Building Graph RAG: A Beginner’s Guide using C# and Neo4j

The Problem

So imagine you build a v1.0 type of RAG powered system. You have some files in Azure Blob storage, and you index that data and push it to Azure AI Search. You have a single index. You perform a hybrid search (BM25 + Vector Search), and you also enabled Semantic Reranker (L2 phase) to boost the accuracy. That is all great.

But then the business starts using your chat application. They really enjoy using it, but they inform you about a very serious limitation: it does not have all the neccesary context. The data you pushed to the index based on these files from a given storage account is just a small subset of the entire organization’s data.

Moreover, organization data is scattered among different services (like SharePoint, OneLake, Azure SQL Database, and Azure Cosmos DB) and uses different formats: files/binary, JSON, etc.

Ok… “challenge accepted,” you say.

Let’s create a dedicated index for each of these data sources in Azure AI Search. Suppose you now have five indexes.

Now, the 1st question arises: when there is a question asked by a user, against which index should we perform a query?

The 2nd question pops up immediately once you start reviewing some more complicated questions users also ask: should I send the same original question to each of these indexes and should I decompose somehow a more complicated question?

Then you start asking yourself: how can I deduce whether the given data returned from the indexes is really relevant in reference to the query? That’s the 3rd question.

Then you realize that some questions cannot be answered by any of the available data sources and you would need to find the information on the internet. How to approach that is the 4th question, ehh… and one more thing… you realize that you need to integrate data from your CRM somehow into that whole workflow. The good thing is that there is at least an MCP Server which exposes its capabilities, but how to connect it is the 5th question.

Now, let me present a bold definition…

So… Foundry IQ is a capability within Microsoft Foundry which lets you finish all that stuff within a single working day to build a POC solution.

How is that possible? Foundry IQ covers all the questions and doubts we have enumerated before, automatically.

If you are really focused and motivated, then the next day you may publish such an agent, which provides grounded answers based on these various data sources to M365 Copilot or Microsoft Teams, and just let some users test it. Isn’t that great?

Now let’s focus on how it all really works.

How it Really Works: The Agentic Retrieval Engine

At the heart of Foundry IQ is something called the Agentic Retrieval engine.

If you remember our “5 indexes” problem from earlier, the old way (Classic v1.0 RAG) required you to manually decide which index to search. Foundry IQ changes the game by acting as the “brain” of the retrieval process.

Instead of a simple “search and return” flow, it follows a more intelligent, iterative cycle:

1. Query Planning & Decomposition: When a user asks a complex question, an LLM (like GPT 4o) breaks it down into smaller subqueries. It doesn’t just look for one answer, it plans a strategy based on a query, chat history and the retrieval instructions you can provide.
2. Source Selection: It automatically identifies which knowledge sources, whether it’s a SharePoint site, files from a Blob Storage, or even the live web, are most likely to have the information.
3. Parallel Execution: It queries those sources simultaneously.
4. Semantic Reranking & Synthesis: Once the data comes back, it doesn’t just dump a list of links. It reranks the results based on actual meaning and merges them into a single, grounded response with citations.
5. Reflection: If the engine realizes the initial search didn’t provide enough information, it can actually “reflect” and try a different search path.

The building blocks

To understand how Foundry IQ simplifies such a complex process, I want to introduce two core concepts that act as the foundation: the Knowledge Source and the Knowledge Base.

The Knowledge Source

I like to think of the Knowledge Source as the “what” and “where” of your data. It is the direct connection to your information. In terms of Knowledge Sources, we should mention that there are two main types: Indexed and Remote.

• Indexed Knowledge Sources: These are sources where the data is processed, chunked, and stored in an index in Azure AI Search. This allows for very fast searches across large amounts of static data.
• Remote Knowledge Sources: These work differently because the data is accessed in real-time. Examples include a Web Search, an MCP call, or a direct call to SharePoint 365. This is a game-changer because it allows your agent to fetch the most up-to-date information exactly when it is needed, without the need for a pre-built index.

The Knowledge Base

If the Knowledge Source is the data, then the Knowledge Base is the “brain” that manages it.

A Knowledge Base is a logical container where you can group multiple Knowledge Sources together. This is where the real magic happens: it provides a clean separation of concerns. I think of it like a reusable USB stick.

Instead of writing custom plumbing code for every single agent you build, you configure the retrieval logic once within the Knowledge Base. Then, you can simply “plug” it into multiple agents. They don’t need to know how the data is indexed or where it comes from; they just ask the Knowledge Base, and it handles the rest. This makes your agents much lighter and much easier to maintain.

Demo

Now, let’s configure an agent in Microsoft Foundry and run a simple example. I will explain a few things along the way.

First of all, we need to deploy a LLM in Microsoft Foundry.

The next step is creating an agent which will leverage the GPT-5.1-mini model we just created.

Please note that the “Knowledge” section is currently empty. To fix that, let’s create our first Knowledge Base.

This screenshot features two interesting things. First, you may see the name Foundry IQ for the first time here. You can think of it this way, when a knowledge base is connected to your Foundry project, it becomes Foundry IQ. (You can create a Knowledge Base in Azure AI Search and not link it here, using it just as a standalone resource which is supposed to provide that agentic retrieval capability via MCP or HTTP).

Second, there is a Connection label, and you can see that it connects to Azure AI Search. Foundry IQ is fully based on Azure AI Search, so whenever you configure anything related to Foundry IQ directly in Microsoft Foundry, it will, behind the scenes, create resources in Azure AI Search automatically for you.

Within a knowledge base we can configure a few basic properties which allow us to control a few aspects of that agentic retrieval pipeline.

I have already added 3 knowledge sources to that knowledge base.

You can add various data sources, as mentioned before, including OneLake, SharePoint (indexed or remote), or an existing Azure AI Search index. (There are many built-in connectors you can leverage while configuring a data source in Azure AI Search).

Now, once we have our knowledge base configured, the remaining step is to just link it to the agent.

Our agent is now ready to provide grounded anwers by calling that particular knowledge base behind it scenes.

Below is a proof, that the Knowledge Base that we linked was used by the agent ( via MCP call).

That is all! The agent is ready to be published and exposed in Microsoft 365 Copilot or via Microsoft Teams.

You can now let your colleagues know that they can interact with the entire organization’s knowledge, not just a single data source with PDF files.

Is Foundry IQ right for you?

I think this is an important question to answer. Foundry IQ is a great choice when you already have data stored in various places within the Azure and Microsoft ecosystem (like Azure Storage, OneLake, and SharePoint). It is also a fantastic option if you want easy connections to Web Search, remote SharePoint calls, and MCP.

One of the biggest benefits is that even if you are unfamiliar with Azure AI Search, all the data ingestion pipelines are created automatically for you (Data Source, Indexer, Skillset, Index resources).

However, there are a few scenarios where it might not be the best fit:

• External Data: If you don’t store any data within the Microsoft and Azure ecosystem yet, this service will likely be less appealing.
• Full Control: If you want to have full control over the agentic retrieval pipeline, this is not for you. You can only control a few basic aspects of the workflow, as Foundry IQ is designed to handle the complexity for you.

Summary

I hope that this post helped you to build a general understanding of Foundry IQ and that terms like Knowledge Base and Knowledge Source are understandable to you now. I also believe that the perspective of building a powerful agent based on your organization’s data within a day or two (at least as a PoC) seems doable to you now.

This is just an intro to the Foundry IQ topic, if you are interested in the technical details, there is a separate blog post about it.

Thanks for reading and see you in the next post.