← All posts
technicalgraph

Why a relationship graph changes everything for agent memory

·1 min read·Remem Team

When most people think about agent memory, they think about vector search: embed a query, find the closest stored memories, return them. It works. But it misses something fundamental about how knowledge is structured.

Facts don’t exist in isolation. They connect to each other. The user’s preference about meeting cadence connects to their comment about being overwhelmed last quarter. The architecture decision from two weeks ago connects to the bug report from yesterday. Real understanding comes from following those connections — not from finding the closest text chunk.

Remem builds a relationship graph automatically as memories are stored. This post explains how.

The data structure: CSR graph

Remem represents relationships as a compressed sparse row (CSR) graph. Each memory is a node. Edges are typed: related_to, caused_by, or part_of.

CSR was chosen over an adjacency list for traversal performance. In a CSR graph, all outgoing edges for a node are stored contiguously in memory. Traversing the neighbours of a node is a single sequential read — cache-friendly and fast even at high node counts.

Auto-discovery

When store_memory is called, the API returns immediately after writing to the LSM-tree. In the background, an auto-discovery worker runs:

  1. Query the HNSW index for the top-K nearest memories by embedding similarity
  2. Filter candidates by a configurable similarity threshold (default: 0.7 cosine similarity)
  3. For each candidate above threshold, write a related_to edge in both directions into the CSR graph

The HNSW index uses approximate nearest-neighbour search — O(log n) query time, sublinear memory — so auto-discovery is fast even at millions of stored memories.

The traversal API

find_related takes a memory ID and a depth parameter:

Terminal window
curl "http://localhost:4545/api/v1/memories/{id}/related?depth=2" \
  -H "Authorization: Bearer your-key"

At depth=1, you get the direct neighbours. At depth=2, you get neighbours-of-neighbours. The response includes the traversal path, so the agent can understand how two memories are connected — not just that they are.

Why this matters for agents

Consider an agent helping a user manage a project. Over several weeks it stores:

  • “User is stressed about the Q3 deadline” (week 1)
  • “Budget was cut by 20%” (week 2)
  • “Two engineers left the team” (week 3)
  • “User asked to reschedule all external meetings” (week 4)

Vector search for “project pressure” might retrieve the Q3 deadline memory. But find_related traversal from that memory surfaces the budget cut and the team departures — because auto-discovery connected them when they were stored. The agent now has a multi-hop picture of the situation without requiring the user to re-explain context.

What auto-discovery doesn’t do

Auto-discovery creates related_to edges based on semantic similarity. It doesn’t infer causality or hierarchy — those edge types (caused_by, part_of) are written by explicit calls to the connections API:

Terminal window
curl -X POST http://localhost:4545/api/v1/connections \
  -H "Authorization: Bearer your-key" \
  -H "Content-Type: application/json" \
  -d '{
    "source_id": "memory-id-budget-cut",
    "target_id": "memory-id-deadline-stress",
    "relationship_type": "caused_by"
  }'

Agents can reason about these edges explicitly — or you can write custom discovery logic that creates typed edges based on your domain.

Performance characteristics

  • Auto-discovery adds roughly 5–15ms of background latency per stored memory (not on the write path)
  • Graph traversal at depth=2 on a 100K-node graph runs in under 10ms (CSR cache locality)
  • Graph density grows sub-linearly — most memories connect to 3–8 neighbours at threshold 0.7

→ Relationship graph docs