Bytes, code points, glyphs, normalization, and tokens
A person usually reads glyphs and words. Software may compare UTF-8 bytes, enumerate Unicode code points, normalize compatibility forms, apply script restrictions, split on whitespace, or convert the result into model-specific token IDs. Those are different representations, not interchangeable views of one self-evident string.
Major carrier families
- Format and zero-width characters: code points that affect joining, breaking, direction, or formatting without an ordinary visible glyph.
- Confusables and homoglyphs: visually similar characters from different scripts or compatibility forms.
- Whitespace distinctions: ordinary spaces, non-breaking spaces, tabs, line endings, and unusual separators.
- Reversible encodings: Base64, hexadecimal, percent encoding, Unicode escapes, or structured markup that a decoder may interpret.
- Tokenizer-sensitive constructions: leading spaces, punctuation, spelling, or word choices that produce different token boundaries or IDs.
A rendered browser may make project and project look identical. Their underlying code-point sequences differ because the second contains ZERO WIDTH SPACE. NFC and NFKC do not generally remove U+200B, so normalization alone is not a complete sanitizer.
visible A: project
visible B: pro[U+200B ZERO WIDTH SPACE]ject
machine test: enumerate code points before normalization
Do not infer tokenizer behavior from the word “BPE.” Normalization, cleaning, pre-tokenization, byte fallback, vocabulary, and model version all matter. Inspect the exact serialized tokenizer and preprocessing path.