Version prompts with their evaluation fixtures

A prompt change should travel with the examples used to judge it. If a repository stores only the latest prompt string, a later reviewer cannot tell which test cases supported the change or reproduce an earlier failure.

Keep each prompt version alongside fixture inputs, expected output properties and the response schema version. Record the model configuration for live evaluations separately from deterministic application tests.

For a synthetic task categorizer, fixture records could contain a short ticket summary and an allowed category. A deterministic stub can check the surrounding contract:

def fake_categorizer(text):
    return {"category": "export" if "download" in text else "other"}

def test_fixture_contract():
    cases = [
        ("download link expired", "export"),
        ("change display name", "other"),
    ]
    for text, expected in cases:
        assert fake_categorizer(text) == {"category": expected}

test_fixture_contract()

The stub ignores a real prompt and contains no intelligence beyond its written rule. Passing this test proves that the fixture harness can compare structured outputs; it cannot prove that a revised prompt improves a model.

Both prompt versions are evaluated against the same held-back cases; a stub tests only the harness.

Give prompt versions meaningful records

Store the exact prompt text and a version identifier that changes when behavior-relevant instructions change. Avoid using a mutable label such as "latest" as the only record. Include the output schema and any retrieval configuration when they affect the result.

Use pytest fixtures to share controlled setup when tests need reusable data or a provider stub. Keep fixtures small enough that an expected outcome can be understood without reading a large hidden dataset.

Add a regression example when a real failure is investigated, after removing sensitive data and checking that its expected result is correct. A copied model response is not automatically a valid label.

Separate repeatable checks from model evidence

Application tests can verify required fields, category handling and rejection of malformed responses on every change. Live model evaluations need a separate record of actual requests, outputs and judgments under the chosen configuration.

Compare both prompt versions on the same held-back cases. If the new wording improves one frequent category but harms a rare important category, preserve that difference in the review instead of publishing only an aggregate score.

Provider behavior can change even with the same prompt label. Record model versions when available and re-evaluate when the integration changes. A deterministic stub remains useful for application behavior while those external outputs vary.

This example is a fixture-harness check. No real prompt versions were sent to a model, and no model-quality comparison is claimed. The repository record should make that distinction visible to anyone reviewing the next change.