Allows the inference engine to correctly yield a single token, even when the streaming response from the backend contains multiple tokens concatenated together.
lgtm