Table of Contents

Introduction

Hey everyone!

You can find all the C# code samples here: MicrosoftAgentFramework GitHub Repo

This post is about Microsoft Agent Framework memory, but also about AI agent memory in general. I’d like to cover the basics and the theory, and then show you a real C# implementation using ChatHistoryMemoryProvider.

Before you read this post, it might be beneficial to first read

When building systems with the new Microsoft Agent Framework, managing state correctly makes the difference between an unstable PoC and a reliable, production-ready AI solution. In this deep dive, I will break down the foundations of both short-term memory and long-term memory, map them to cognitive frameworks, and show you exactly how to build a persistent context layer in C# using the native ChatHistoryMemoryProvider.

Short-Term vs. Long-Term Memory

An architectural diagram showing how AI Agent Memory branches into Long-Term Memory and Short-Term Memory. Long-Term Memory connects to a Memory Lifecycle box containing Memory Extraction, State Management, and Memory Retrieval. Short-Term Memory connects to a Context Management box containing Context Trimming, Compaction, and Summarization.

When we talk about building language agents, we often look at the human brain as our main reference point. It shouldn’t surprise anyone that major companies in the AI space are hiring neuroscientists to translate biological memory mechanisms into software patterns. Evolution has already built the perfect blueprint.

In the world of AI engineering, we must use these mechanisms to prevent our agents from being purely reactive. Without memory, every single interaction with a user exists in isolation. By designing structured memory systems, we allow agents to keep continuous context, recognize user preferences, and run long-running, multi-session business processes.

To build an efficient system, we must establish a clear boundary between the two primary classes of agent state:

1. Short-Term Memory

In the simplest terms, short-term memory is bounded by the scope of a single conversation thread. It represents the active conversation window. Because active LLM context windows can quickly become overwhelmed, we use various short-term context management techniques to keep things under control:

  • Sliding Windows: Passing only the top N recent messages to the next prompt.
  • Summarization: Condensing older turns into a clean, unified paragraph before pushing it to the model.
  • Context Compaction: An umbrella term for dynamically shrinking, cutting, or processing raw logs once token thresholds are breached.

2. Long-Term Memory

Long-term memory is data that persists across completely independent sessions. This is where true personalization and agent autonomy happen. If a user states in session one that they live in Poland, the agent should inherently know that fact in session fifty, even if weeks have passed.

Implementing persistent agent memory introduces unique engineering constraints:

  • Memory Extraction: We don’t want to save thousands of lines of conversational fluff (“hello,” “thank you”). We must extract only the highly relevant, distilled nuggets of data.
  • State Management: Users must be able to view, refresh, or delete saved facts to ensure compliance with strict privacy regulations like GDPR.
  • The Decay and Forgetting Problem: Human brains forget things to protect themselves from cognitive overload. Similarly, if I tell an agent I am 33 years old, that data becomes stale in two years. Our long-term storage layers need to adapt, weight repeated interactions heavier, and implement self-healing mechanisms to purge or update obsolete facts.

CoALA: Working/Semantic/Procedural/Episodic Memory

An architectural flowchart of the CoALA Framework branching into four memory categories: Now / Session Context (Working Memory with Context Window and Chat History), What is True (Semantic Memory with Knowledge Base and .md files), How to do it (Procedural Memory with Skills.md), and What happened before (Episodic Memory with Distillation and Learning).

To design a strong AI agent architecture, the CoALA (Cognitive Architectures for Language Agents) framework is an excellent tool to structure the storage layout. CoALA breaks down agent memory into four clear, functional areas based on human thinking patterns:

1. Now / Session Context (Working Memory)

This block handles the immediate moment. Working Memory focuses entirely on what is happening right now during the active interaction. It is split into two main pieces:

  • Context Window: The total token space the LLM can read and process at one time.
  • Chat History: The temporary record of messages exchanged back and forth during the active session.

2. What is True (Semantic Memory)

This block manages the agent’s world knowledge and factual data. Semantic Memory helps the agent store and find objective truths. In engineering, this data comes from:

  • Knowledge Base: Corporate search databases like Azure AI Search, Cosmos DB, or graph databases like Neo4j.
  • .md files: Static markdown documentation used to provide the agent with default factual knowledge.

3. How to do it (Procedural Memory)

This block defines the agent’s internal skills, rules, and workflows. Procedural Memory contains the step-by-step instructions that teach the model how to complete specific tasks. This is typically managed by:

  • Skills.md: A dedicated system file containing formatting rules, tool definitions, and strict guides that show the LLM exactly how to execute custom tools.

4. What happened before (Episodic Memory)

This is focused entirely on the agent’s past experiences and event history. Episodic Memory allows an agent to look back at previous execution cycles and improve over time. It relies on two main processes:

  • Distillation: Automatically extracting only the highly important facts from heavy raw chat transcripts instead of keeping useless fluff.
  • Learning: A continuous optimization cycle that adapts and rewrites the long-term memory store based on past user interactions and feedback.

Long-Term Memory: Hot Path vs. Background Processing

A memory workflow diagram showing how an AI agent processes prompts. It displays a "Context Hydration Layer" where Chat History and Memory Storage combine before agent reasoning. It highlights two storage methods: Approach 1 (Hot Path inline storage) and Approach 2 (Background processing via a queue and worker). The loop ends by creating a "Fully Hydrated Payload" for the next user prompt.

Once you decide to capture a new piece of information for long-term agent memory in .NET, you have to choose between two structural execution pipelines: the Hot Path or Background Processing.

The Hot Path (Inline)

In a hot path pipeline, the memory engine identifies, summarizes, and writes the new memory payload directly inside the active conversation loop.

  • The Problem: This adds immediate network and processing latency to the current turn. If your embedding generation and vector database writes take too long, the user experiences an annoying delay.

Background Processing (Out-of-Band)

Instead of forcing memory management inline, you publish a system event when a turn or session concludes. An asynchronous background listener picks up the conversation logs, processes them out-of-band, and writes them to the persistent store.

  • The Benefit: You get the luxury of execution time. Your background workers can take two minutes to execute hyper-precise, multi-step entity extractions and validations without impacting the end-user’s response latency.

ChatHistoryMemoryProvider in Microsoft Agent Framework

Let’s bridge this theory with a real-world C# example. The Microsoft Agent Framework (Microsoft.Agents.AI) provides a native component called ChatHistoryMemoryProvider designed specifically for cross-session agent memory.

At its core, this component tracks your conversational turns, vectorizes them using an ecosystem-agnostic IEmbeddingGenerator (which comes from Microsoft.Extensions.AI namespace and uses an embedding model deployed in Microsoft Foundry. Secure connection can be established by using DefaultAzureCredential which behind the scenes uses VisualStudioTokenCredential and Foundry User RBAC role assigned to my security principal), and persists them into a IVectorStore (backed by anything from an In-Memory collection to Azure AI Search, PostgreSQL, or Cosmos DB).

public ChatHistoryMemoryProviderExample()
{
    // trimmed for brevity

    IEmbeddingGenerator<string, Embedding<float>> embeddingGenerator = new AzureOpenAIClient(new Uri(Environment.GetEnvironmentVariable("AZURE_OPEN_AI_URI")!), new DefaultAzureCredential())
        .GetEmbeddingClient(Environment.GetEnvironmentVariable("AZURE_OPEN_AI_EMBEDDING_MODEL")!)
        .AsIEmbeddingGenerator();

    VectorStore vectorStore = new InMemoryVectorStore(new InMemoryVectorStoreOptions()
    {
        EmbeddingGenerator = embeddingGenerator
    });
}

Once we have that vector store with a corresponding embedding generator defined we can finally create a new ChatHistoryMemoryProvider:

public ChatHistoryMemoryProviderExample()
{
    // trimmed for brevity
	
    _mafAgent = new ChatClientAgent(responesClient, new ChatClientAgentOptions()
    {
        Name = agentName,
        ChatOptions = new ChatOptions
        {
            Instructions = "You are a helpful assistant.",
            Reasoning = new ReasoningOptions() { Effort = ReasoningEffort.None, Output = ReasoningOutput.None }
        },
        AIContextProviders = [new ChatHistoryMemoryProvider(
            vectorStore,
            collectionName: "vectorized_chat_history",
            vectorDimensions: 1536,
            session => new ChatHistoryMemoryProvider.State(
                storageScope: new() 
                { 
                    UserId = "123", 
                    SessionId = Guid.NewGuid().ToString(), 
                    AgentId = agentName,
                    ApplicationId = "Memory:Intro"
                },
                searchScope: new() 
                { 
                    UserId = "123",
                    //SessionId = null,
                    //AgentId = null,
                    //ApplicationId = null
                }),
            new ChatHistoryMemoryProviderOptions()
            {
                MaxResults = 3,
                SearchTime = ChatHistoryMemoryProviderOptions.SearchBehavior.BeforeAIInvoke,
                SearchInputMessageFilter = null,
                StorageInputRequestMessageFilter = null,
                StorageInputResponseMessageFilter = msgs => []
            }
        )]
    });
}

As you can see, the location of the data is decided by your vectorStore and the collectionName: "vectorized_chat_history". The scopes on the other hand, do something different because they manage the labels used for filtering:

  • storageScope: This defines the properties and labels (like UserId, SessionId, AgentId, and ApplicationId) that get attached to the chat messages when they are saved.
  • searchScope: This defines the filter rules the agent uses when looking for past memories. Because we only set UserId = "123" and leave the other labels empty, the agent filters only by the user ID. This allows the agent to search and find memories from any past session or application for this specific user. This is exactly how true cross-session memory works.

Configuring the Memory Options

Inside the ChatHistoryMemoryProviderOptions, there are a few important settings to manage:

  • MaxResults = 3: This ensures the provider only retrieves the top 3 most relevant matching memories from the vector store, keeping the prompt clean.
  • SearchTime = BeforeAIInvoke: This tells the framework to automatically run a vector similarity search and inject the relevant past context into the prompt right before the LLM is called.
  • Message Filters: Properties like StorageInputResponseMessageFilter allow you to control exactly which messages get saved. This is highly useful for filtering out regular conversational noise and saving only the important facts.

By combining ChatHistoryMemoryProvider with a flexible vector store, the Microsoft Agent Framework handles the complex background work of vectorization, storage, and retrieval automatically. This allows you to focus purely on building the core logic of your application.

Demo

public async Task RunAsync()
{
    AgentSession firstSession = await _mafAgent.CreateSessionAsync();
    Console.WriteLine(await _mafAgent.RunAsync("I live in Poland", firstSession));

    Console.WriteLine("\n---------------\n");

    AgentSession secondSession = await _mafAgent.CreateSessionAsync();
    Console.WriteLine(await _mafAgent.RunAsync("What is the capital of the country I live in", secondSession));
}

First, I create a first agent session and provide information about where I live.

And then…

In the second session, I ask about the capital of the country I live in. Without using AI agent memory, it couldn’t know the answer. But with the current implementation of the Microsoft Agent Framework memory, it can search through past messages and retrieve the one that enriches the context with “I live in Poland”.

ChatHistoryMemoryProvider in action (Microsoft Agent Framework Memory). A debugger screenshot showing a collection of ChatMessage objects. The first message is a user question about their country. A Text Visualizer window highlights the second message containing the injected memory context: "## Memories Consider the following memories when answering user questions: I live in Poland".

This is how it works, and this is a clear example of long-term memory that is persisted across completely separate sessions.

Please also note that in this specific implementation, the strategy is as follows:

  • Save everything (or almost everything): You can control this using the AIContextProvider filters.
  • Be very selective, picky, and clever about what you retrieve: The system stores the history generously but filters the results carefully during search.

There are more advanced techniques which focus on a different workflow:

  1. Identify
  2. Summarize
  3. Store
  4. Forget (under certain conditions)

These advanced techniques will be discussed in dedicated blog posts, so stay tuned if you found this information interesting!

Summary

Designing persistent agent memory isn’t just about saving strings to a table; it’s about building a predictable context hydration layer. By isolating short-term chat threads from long-term episodic storage, utilizing frameworks like CoALA to classify state, and applying tools like the ChatHistoryMemoryProvider within the Microsoft Agent Framework, you can construct robust .NET applications that gracefully recall context across sessions.

Thanks for reading and see you in the next post!

Tags:

Categorized in:

Agents,