intermediate⏱35 minutesLesson 4 of 10

Summary Memory

Implement conversation summarization for long-running agents, summary memory nodes, and truncation strategies for managing context windows.

Summary Memory

When conversations grow too long for the LLM's context window, simple trimming discards valuable context. Summarization preserves key information in a compressed form while keeping the conversation manageable.

The Problem with Long Conversations

LLMs have token limits. Once a conversation exceeds the limit:

Truncation: Discards early messages — loses context
Error: Context window exceeded — request fails

Summarization solves this by compressing history into a summary while retaining important details.

Summary Memory Architecture

Conversation Flow:
[Message 1] → [Message 2] → ... → [Message 10] → [Summarize]
                                                       ↓
[Summary] ← [Message 11] → [Message 12] → ... → [Message 20] → [Update Summary]
                                                                    ↓
[Updated Summary] ← ...

A summary node is triggered periodically to condense the conversation.

Basic Summary Node

python

from langchain_openai import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langgraph.graph import StateGraph, START, END, add_messages
from langgraph.checkpoint.memory import MemorySaver
from typing_extensions import TypedDict, Annotated
from typing import List, Any

llm = ChatOpenAI(model="gpt-4o-mini")

class State(TypedDict):
    messages: Annotated[List[Any], add_messages]
    summary: str

def summarize_conversation(state: State) -> dict:
    current_summary = state.get("summary", "No summary yet.")

    prompt = ChatPromptTemplate.from_messages([
        ("system", "You are a conversation summarizer. Given the current "
                   "summary and the new messages, create an updated summary. "
                   "Keep it concise but include all key information.\n\n"
                   "Current summary:\n{summary}"),
        ("human", "New messages to incorporate:\n{messages}")
    ])

    chain = prompt | llm | StrOutputParser()
    recent = state["messages"][-4:]  # Last 4 messages
    messages_text = "\n".join(f"{m.type}: {m.content}" for m in recent)

    new_summary = chain.invoke({
        "summary": current_summary,
        "messages": messages_text
    })

    return {"summary": new_summary}

ℹ️Note

The summary node reads the current summary and recent messages, then produces an updated summary. This incremental approach is more efficient than re-summarizing the entire conversation.

When to Summarize: Triggering Strategies

Strategy 1: Message Count Threshold

python

def should_summarize(state: State) -> str:
    if len(state["messages"]) > 10:  # Summarize every 10 messages
        return "summarize"
    return "respond"

# In graph:
builder.add_conditional_edges(
    "check",
    should_summarize,
    {
        "summarize": "summarize_node",
        "respond": "chat_node"
    }
)

Strategy 2: Token Count Threshold

python

import tiktoken

def should_summarize_by_tokens(state: State) -> str:
    enc = tiktoken.encoding_for_model("gpt-4o")
    total = sum(len(enc.encode(m.content)) for m in state["messages"])

    if total > 3000:  # Summarize if over 3000 tokens
        return "summarize"
    return "respond"

Strategy 3: Periodic (Every N Turns)

python

def should_summarize_periodic(state: State) -> str:
    # Summarize every 5 turns
    if len(state["messages"]) >= 5 and len(state["messages"]) % 5 == 0:
        return "summarize"
    return "respond"

✅Success

Choose your triggering strategy based on the expected conversation length and message size. Token-based is most accurate; count-based is simplest.

Full Summarization Agent

python

from langgraph.graph import StateGraph, START, END, add_messages
from langgraph.checkpoint.memory import MemorySaver
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
from langchain.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from typing_extensions import TypedDict, Annotated
from typing import List, Any

llm = ChatOpenAI(model="gpt-4o-mini")

class AgentState(TypedDict):
    messages: Annotated[List[Any], add_messages]
    summary: str

def chat_node(state: AgentState) -> dict:
    system_msg = "You are a helpful assistant."
    if state.get("summary"):
        system_msg += f"\n\nSummary of conversation:\n{state['summary']}"

    messages = [SystemMessage(system_msg)] + state["messages"]
    response = llm.invoke(messages)
    return {"messages": [response]}

def summarize_node(state: AgentState) -> dict:
    prompt = ChatPromptTemplate.from_messages([
        ("system", "Update the conversation summary. Include all important "
                   "information from the new messages.\n\n"
                   "Current summary: {summary}"),
        ("human", "New messages:\n{messages}")
    ])
    chain = prompt | llm | StrOutputParser()
    recent = state["messages"][-6:]
    msgs = "\n".join(f"{m.type}: {m.content}" for m in recent)
    return {"summary": chain.invoke({"summary": state.get("summary", ""), "messages": msgs})}

def router(state: AgentState) -> str:
    if len(state["messages"]) > 10:
        return "summarize"
    return "respond"

# Build graph
builder = StateGraph(AgentState)
builder.add_node("chat", chat_node)
builder.add_node("summarize", summarize_node)

builder.add_edge(START, "chat")
builder.add_conditional_edges("chat", router, {
    "summarize": "summarize",
    "respond": END
})
builder.add_edge("summarize", END)

app = builder.compile(checkpointer=MemorySaver())

# Usage
session = {"configurable": {"thread_id": "summary-test"}}
for i in range(15):
    app.invoke({"messages": [HumanMessage(f"Message number {i}")]}, session)

# Check the summary
state = app.get_state(session)
print(state.values.get("summary"))

💡Tip

In the full agent, the summary is injected into the system prompt on every turn. This means the LLM always has access to compressed context from earlier in the conversation.

Truncating After Summarization

After summarizing, you can truncate the message list to save tokens:

python

def summarize_and_truncate(state: AgentState) -> dict:
    # 1. Generate updated summary
    prompt = ChatPromptTemplate.from_messages([
        ("system", "Summarize:\nCurrent: {summary}\nNew: {messages}"),
        ("human", "Create updated summary.")
    ])
    chain = prompt | llm | StrOutputParser()
    recent = state["messages"][-6:]
    msgs = "\n".join(f"{m.type}: {m.content}" for m in recent)
    new_summary = chain.invoke({"summary": state.get("summary", ""), "messages": msgs})

    # 2. Truncate — keep only last 4 messages
    truncated = state["messages"][-4:]

    return {
        "summary": new_summary,
        "messages": truncated  # Replace messages with short history
    }

⚠️Warning

When you replace messages with a truncated list, the old messages are permanently gone (except what the summary captured). Ensure your summary is comprehensive enough.

Running Summary (Incremental)

Rather than summarizing everything at once, maintain a running summary:

python

def update_running_summary(state: AgentState) -> dict:
    existing = state.get("summary", "")
    last_exchange = state["messages"][-2:]  # Last user + assistant turn

    prompt = ChatPromptTemplate.from_messages([
        ("system", "Incorporate this exchange into the running summary. "
                   "Keep the summary under 200 words.\n\n"
                   "Current summary: {existing}"),
        ("human", "Exchange:\n{exchange}")
    ])

    chain = prompt | llm | StrOutputParser()
    exchange_text = "\n".join(f"{m.type}: {m.content}" for m in last_exchange)
    new_summary = chain.invoke({"existing": existing, "exchange": exchange_text})

    return {"summary": new_summary}

Summary with Entity Extraction

Go beyond simple summarization — extract structured entities:

python

from pydantic import BaseModel, Field

class ConversationMemory(BaseModel):
    summary: str = Field(description="Conversation summary")
    user_name: str = Field(default="")
    preferences: List[str] = Field(default_factory=list)
    facts: List[str] = Field(default_factory=list)

def extract_memory(state: AgentState) -> dict:
    parser = PydanticOutputParser(pydantic_object=ConversationMemory)

    prompt = ChatPromptTemplate.from_messages([
        ("system", "Extract conversation memory. {format_instructions}"),
        ("human", "History:\n{messages}")
    ])

    chain = prompt | llm | parser
    msgs_text = "\n".join(f"{m.type}: {m.content}" for m in state["messages"])
    memory = chain.invoke({"messages": msgs_text, "format_instructions": parser.get_format_instructions()})

    return {"summary": memory.summary}

Comparison: Summarization vs Truncation

Aspect	Truncation	Summarization
What it does	Drops old messages	Compresses history into text
Information retention	Loses everything dropped	Retains key info in summary
Token efficiency	Excellent	Good (summary takes some tokens)
LLM awareness	Sees only recent history	Sees compressed full history
Implementation	Simple (slicing)	Moderate (LLM call needed)
Cost	Free	LLM call per summarization

Practice Questions

Practice Question

What is the main advantage of summarization over simple truncation?

Practice Question

How does incremental summarization work?

Practice Question

What is a good trigger condition for running a summary node?

Practice Question

Where in the system prompt should the summary be injected?

Practice Question

What happens to truncated messages after summarization?

Practice Question

What is the most accurate trigger for summarization?

Practice Question

What additional capability does entity extraction add to summarization?

Practice Question

What does the running summary approach trade off?

Practice Question

After summarizing and truncating, what should you keep in the messages list?

Practice Question

What is a limitation of summarization-based memory?

✅Success

Key Takeaways

Summarization preserves key context when conversations exceed token limits
Incremental summarization updates an existing summary with new messages
Trigger summarization by token count or message count thresholds
Inject the summary into the system prompt for every LLM call
Truncate messages after summarization to stay within context windows
Entity extraction adds structured memory alongside free-text summaries
Running summaries trade perfect recall for token efficiency
Summarization costs LLM calls — balance frequency with the cost

Progress40%

Conversation Memory

Human-in-the-Loop with Interrupts

Course Content

1. Advanced State Management 2. Persistence and Checkpointing 3. Conversation Memory 4. Summary Memory 5. Human-in-the-Loop with Interrupts 6. Approval Workflows 7. Multi-Agent Basics 8. Agent Communication and Supervision 9. Streaming Tokens 10. Intermediate Project: Multi-Agent Research System