Giving LLM Agents Persistent Memory with pgvector

Session-based semantic memory for multi-turn agentic workflows using PostgreSQL + pgvector and SQLAlchemy.

An agentic AI job copilot that analyzes job fit, tailors resumes and drafts outreach only feels useful if it remembers what you told it last week. Building that persistent, semantic memory across multi-session chat threads with PostgreSQL + pgvector and SQLAlchemy taught me most of what I now assume about agent memory design.

Diagram of an LLM agent storing and retrieving semantic memories from a pgvector-backed store — Conversation turns are embedded, stored and selectively retrieved across sessions.

Not every message is worth remembering

The naive approach — embed and store every message — produces a memory store full of noise. Instead, a lightweight extraction step runs after each session, pulling out durable facts ("prefers remote roles," "targeting senior backend positions," "already applied to Company X") rather than storing raw transcript. Each memory is embedded and stored with metadata: source session, timestamp, and a confidence/decay signal.

Retrieval at generation time

When a new session starts, the agent doesn't dump the entire memory store into the prompt — it runs a similarity search against pgvector, scoped by recency and relevance to the current message, and injects only the top handful of memories as context. This keeps prompts small and, just as importantly, keeps the agent's behavior grounded in facts that are actually relevant to what the user is asking right now.

Why Postgres + pgvector over a dedicated vector database

One source of truth — user data, session state and embeddings live in the same transactional database, so memory writes are consistent with everything else without a separate sync process.
SQL filtering before vector search — scoping candidates by user ID and timestamp with a normal WHERE clause before the vector comparison runs kept queries fast without a separate metadata store.
Operationally simpler — one database to back up, migrate and monitor, which mattered more than a marginal ANN performance gain at this scale.

The pattern that made the memory feel reliable rather than gimmicky was restraint: store less, retrieve less, but make sure what's retrieved is actually relevant. An agent with a huge memory store and poor retrieval feels worse than one with no memory at all.