Keep document IDs stable across RAG reindexing

A source's identity should survive an ordinary edit. If every reindex gives a handbook a new random document ID, old vectors and citations can remain attached to abandoned versions. Separate logical identity from content revision.

For a synthetic handbook, use an application-assigned ID such as ops-handbook. Store a content hash for the exact extracted representation being indexed. A changed hash requests a new revision; an unchanged hash lets the ingestion worker skip identical content.

import hashlib

def revision(text):
    normalized = text.replace("\r\n", "\n")
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

assert revision("one\r\ntwo") == revision("one\ntwo")
assert revision("one") != revision("two")

The hashlib reference documents SHA-256 and byte input. This example's normalization is intentionally narrow. Changing case or removing whitespace can erase distinctions in code, tables or identifiers, so broader normalization needs a documented reason.

Source ID with version hash updating one index record.

Track a replacement as one logical source

An ingestion record can contain the source ID, content hash, parser version and active revision ID. Chunk records reference that revision. Build all new chunks, verify their expected count and metadata, then switch the source's active revision after the new data is ready.

The pgvector documentation supplies vector storage and update operations inside PostgreSQL. Revision ownership and activation are application responsibilities; a vector column does not establish them automatically.

Test a source with two chunks, then edit it so only one remains. Reindexing must retire the chunk that disappeared. Upserting the new chunk alone leaves a stale passage searchable. Also repeat the same ingestion twice and verify that it does not multiply the active records.

Keep source moves deliberate

A URL can be a source identifier when the publisher guarantees its stability. A renamed file or redirected URL can otherwise appear to be a different document. Prefer a stored mapping from external location to durable internal identity, and review moves explicitly.

A content hash identifies a representation, not the document's meaning or access policy. Two tenants can upload identical text and still require separate authorization and retention. Never merge their permissions merely because their hashes match.

Store the model and chunker configuration with the revision too. The source text can remain unchanged while its derived vectors require replacement. Keep old revisions available only for a defined audit or rollback period, and ensure retrieval selects the active authorized revision consistently.