Pack whole passages under a UTF-8 byte budget

AI assisted the preparation of this article, and its Python example was tested locally with synthetic fixtures.

A retrieval application may need a local byte limit on the exact context it serializes. Counting characters or passage text alone misses UTF-8 encoding, source labels and separators. This limit is a byte policy, not a model token limit.

Flow: Passages, then Serialize UTF-8, then Count + reserve, then Keep or skip.

Count the assembled representation

This small function takes already authorized (source_id, text) pairs in priority order. It keeps a whole passage only if the complete assembled UTF-8 payload and a separate framing reserve fit. An oversized passage is skipped; later passages may still fit.

def pack(passages, limit, reserve):
    if type(limit) is not int or type(reserve) is not int or limit < 0 or reserve < 0:
        raise ValueError("limits must be nonnegative integers")
    header = "Sources:\n"
    if len(header.encode("utf-8")) + reserve > limit:
        raise ValueError("fixed framing exceeds budget")
    text = header
    kept = []
    for source_id, passage in passages:
        if not isinstance(source_id, str) or not isinstance(passage, str):
            raise ValueError("source and passage must be strings")
        candidate = text + f"[{source_id}] {passage}\n"
        if len(candidate.encode("utf-8")) + reserve <= limit:
            text = candidate
            kept.append(source_id)
    return text, kept

The header, brackets, ID, separating space and newline are included in every candidate count. reserve represents additional framing bytes the caller plans to add; it must not include bytes already in the returned text. The function does not append that reserved framing itself.

A two-byte UTF-8 character such as é illustrates why character count is insufficient. Test an exact boundary with that character, an oversized middle passage followed by a smaller one, and an invalid negative reserve. Verify both the returned IDs and the serialized byte length. If the fixed header and reserve already exceed the limit, fail rather than return an over-budget header.

Keep other limits separate

A production serializer may add JSON escaping, role labels or protocol fields. Count the exact representation that will actually be sent, including those bytes, or replace this fixture's serializer with the real one and rerun its boundary tests. Reject or escape ambiguous source labels according to the receiving format; this fixture uses labels only for illustration.

Bytes cannot establish whether a model's context window will fit. Tokenizers and provider envelopes differ, and token counts need their own validated enforcement and output allowance. The Python string encoding reference describes UTF-8 encoding; the Transformers truncation guide describes a separate token-based mechanism.

The local fixture does not decide whether a passage is relevant, supports a claim or remains authorized. Keep current access checks before selection and preserve source revisions for later citation review. It also repeatedly assembles strings, so bound input sizes before using it on large collections.