Decode UTF-8 across byte chunks

AI assisted the preparation of this article, and its Python example was tested locally with synthetic fixtures.

A byte stream can split a UTF-8 character between reads. Decoding each read independently may raise an error even when the complete byte stream is valid. Keep one incremental decoder for the stream and flush it when the stream ends.

Flow: Byte chunks, then Decoder state, then Decoded text, then Flush or error.

Keep decoder state between reads

The following function uses strict decoding and returns one string for a small synthetic example. The chunk boundaries are not character boundaries. A network reader could supply the chunks, but this function does not open a connection.

import codecs

def decode_chunks(chunks):
    decoder = codecs.getincrementaldecoder("utf-8")(errors="strict")
    parts = [decoder.decode(chunk, final=False) for chunk in chunks]
    parts.append(decoder.decode(b"", final=True))
    return "".join(parts)

The decoder holds an incomplete sequence until the next call. The final call tells it that no more bytes are coming. With strict error handling, an incomplete sequence at that point raises UnicodeDecodeError instead of silently discarding bytes. An invalid sequence in an earlier chunk also raises.

For example, split the three bytes of the euro sign across three pieces: b'A\xe2', b'\x82', and b'\xacB'. The combined output is A€B. Trying to decode the first piece independently fails because that piece ends before the character is complete.

Check the end of the stream

A useful test needs both the successful split and a truncated last character. Feed the first two bytes of a multibyte sequence and then finish the stream. The error should appear at finalization. Also send a byte that cannot begin a valid UTF-8 sequence and assert that it fails.

For a real service, decide where a decoding failure is reported and whether the upstream connection should be closed or retried. Do not use an ignore policy merely to keep a pipeline moving: it can remove data without telling the caller.

The example collects the decoded parts in memory for clarity. A long-lived stream should consume decoded pieces incrementally and bound the surrounding buffers. Keeping a decoder does not supply message boundaries, ordering guarantees, backpressure or reconnection. An event parser still needs to carry its own incomplete event state.

Python documents the incremental decoder interface. This synthetic fixture was checked on CPython 3.13.5; it does not measure a network service.