Migrating from Rust Remem

Remem 0.5 uses a different on-disk data format from the earlier Rust implementation. Do not point a Go server at a Rust data directory, and do not run both servers against the same directory.

Migration uses a portable snapshot:

Rust data directory → Rust export → portable .rsnap file → Go import → Go data directory

The snapshot carries memory records, relationships, lifecycle data, and tenant information. Go rebuilds its own indexes during import. Embeddings are recomputed from the original content because the two implementations produced vectors in different spaces; copying the old vectors would make search results look valid while being incorrect.

For the complete command reference and troubleshooting details, see the repository guide docs/MIGRATION.md.

Before you begin

Make sure you run the latest Rust version of Remem 0.4.0. If you don’t, upgrade to that version.

Prepare:

• a Rust remem-export binary that can open your existing deployment;
• a Go remem-admin binary built with the ONNX embedding runtime;
• the Go server binary built with the same embedding support;
• enough disk space for the snapshot and the new Go data directory;
• a maintenance window during which the Rust server can be stopped.

Keep the Rust data directory until the new deployment has been verified. The export file is also a portable logical backup and should be retained until you are satisfied with the migration.

1. Stop the Rust server

Stop Remem cleanly before exporting. The exporter opens the Rust directory read-write while it replays the write-ahead log. Two processes must never write to the same directory.

2. Export a snapshot

Run the Rust exporter against the stopped deployment:

REMEM_DATA_DIR=/var/lib/remem \
  remem-export --out corpus.rsnap

The exporter reports the number of records, vectors, and relationships written. Keep corpus.rsnap; it lets you repeat the Go import without starting the Rust server again.

Some archived Rust memories may be reported as having no active vector. This is expected. The records are still exported and Go recomputes their embeddings during import.

3. Import into a new Go directory

Import into a path that does not contain an existing Go database:

./remem-admin import \
  --data-dir /var/lib/remem-go \
  --in corpus.rsnap \
  --model-path ./.models/all-MiniLM-L6-v2 \
  --onnx-library-path /path/to/libonnxruntime.so

Use the ONNX library path appropriate for your operating system. The exact model and library setup depends on how you built Remem; use the repository’s build instructions for the current release.

The import prints its progress before writing and reports how many embeddings were recomputed. Seeing 0 taken from the snapshot is expected for a Rust snapshot: Go intentionally recomputes every embedding.

If the import is interrupted, run the same command with --resume. The import cursor is stored beside the snapshot. Re-running from the beginning is also safe, but takes longer.

Import warnings and rejections

The importer names data it cannot carry forward. In particular, it may reject:

• self-relationships;
• relationships whose target was hard-deleted from the Rust corpus;
• snapshots written by a newer, unsupported format version;
• imports into a directory that already contains conflicting data.

Read the rejection output before continuing. Rejected relationships are not silently recreated in the Go store.

4. Verify before starting the server

Compare the new Go store with the snapshot:

./remem-admin verify \
  --data-dir /var/lib/remem-go \
  --in corpus.rsnap

Verification should report matching record, vector, and relationship counts and exit successfully. Run it before starting the server: lifecycle maintenance can legitimately change health, retention, or archive fields after the server starts, which makes the store differ from the point-in-time snapshot.

The only expected relationship differences are the self-relationships or orphan relationships explicitly reported during import.

5. Rebuild indexes if desired

The import queues vector index work for the Go server. You can rebuild the approximate vector index before serving traffic:

./remem-admin rebuild \
  --data-dir /var/lib/remem-go \
  --index vector

This is optional. Starting the server without the rebuild allows the normal repair job to complete, but search may initially report that the vector index is degraded.

If you want relationship discovery for memories that were imported from Rust, queue a discovery backfill while the server is stopped:

./remem-admin discovery backfill \
  --data-dir /var/lib/remem-go

The server processes the queued work when it starts. Backfill is safe to run more than once.

6. Start Go Remem

Start the Go server against the new directory. For a local binary:

REMEM_SERVER_HTTP_ADDR=127.0.0.1:4545 \
REMEM_SERVER_API_KEY='<your key>' \
REMEM_VECTOR_INDEX=hnsw \
REMEM_EMBEDDING_MODEL_PATH=$(pwd)/.models/all-MiniLM-L6-v2 \
REMEM_EMBEDDING_ONNX_LIBRARY_PATH=/path/to/libonnxruntime.so \
./remem start --data-dir /var/lib/remem-go

For a container deployment, mount /var/lib/remem-go as the persistent data directory and set REMEM_SERVER_API_KEY in the Compose environment. See the Docker Compose guide for the supported container setup.

7. Check the migrated corpus

Run checks against the Go server before decommissioning Rust:

KEY='<your key>'
BASE='http://127.0.0.1:4545/api/v1'

# Confirm that records are present.
curl -s \
  -H "Authorization: Bearer $KEY" \
  "$BASE/memories?limit=3"

# Exercise each search mode.
for search_type in semantic keyword hybrid; do
  curl -s -X POST \
    -H "Authorization: Bearer $KEY" \
    -H 'Content-Type: application/json' \
    -d "{\"query\":\"a phrase you know is in the corpus\",\"search_type\":\"$search_type\",\"limit\":3}" \
    "$BASE/memories/search"
done

# Check relationships for a known memory ID.
curl -s \
  -H "Authorization: Bearer $KEY" \
  "$BASE/memories/<memory-id>/related?limit=5"

Also check the server logs for unexpected warnings. Confirm the memories you care about, search quality, lifecycle fields, and relationships before removing the old deployment.

Important limitations

Go to Rust is not supported

There is no importer for the Go format in the frozen Rust implementation. Keep the .rsnap file as your portable logical backup, but plan migrations in the Rust-to-Go direction only.

The migration is not an in-place upgrade

The Go server must import into a new data directory. Do not delete or overwrite the Rust directory as part of the import command.

Embeddings take time

Recomputing embeddings is the slowest part of the migration. Its duration depends on the model runtime, hardware, and corpus size. Plan for it in the maintenance window and monitor the importer’s progress.

Migrating older Rust data directories

If the Rust deployment predates the current Rust on-disk format, opening it with the exporter may run the Rust migration chain first. That step changes the Rust directory, so make a filesystem backup before running it if you need to retain the ability to reopen the directory with an older Rust binary.

The exporter creates a pre-migration backup when supported. Treat the backup as an additional safety measure, not as a replacement for your normal deployment backup policy.

After migration

Keep corpus.rsnap and the original Rust backup until the Go deployment has served successfully through your normal verification period. Once the new deployment is trusted, update clients to the Go endpoint and follow the production checklist for persistence, credentials, health checks, and backups.