Invisible Unicode and Modern Language-Model Tokenizers

Executive summary

Invisible and near-invisible Unicode characters are not a single tokenizer category. Their fate is determined by a sequence of independent layers: Unicode decoding, normalization, application-specific cleaning, whitespace handling, pre-tokenization, subword segmentation, and finally vocabulary lookup or byte fallback. Two tokenizers that are both described as “BPE” can therefore behave radically differently on the same invisible character. Hugging Face explicitly models normalization, pre-tokenization, the subword model, and post-processing as separate pipeline components, while SentencePiece packages its normalization rules with the model itself. source

The most important findings are:

Unicode normalization is not sanitization. NFC and NFD perform canonical normalization; NFKC and NFKD additionally apply compatibility decomposition. They do not, as a general rule, remove zero-width joiners, zero-width non-joiners, zero-width spaces, bidi controls, C0 controls, private-use characters, or other arbitrary invisibles. A particularly important exception is compatibility-equivalent spacing: NFKC/NFKD map U+00A0 NO-BREAK SPACE to ordinary U+0020 SPACE and similarly fold many Unicode space characters. Unicode itself explicitly illustrates NBSP → SPACE as a compatibility equivalence. source

Whitespace, format controls, and combining marks must not be conflated. Unicode 17.0 distinguishes Zs space separators from Cf format controls, Cc controls, Mn nonspacing marks, Cs surrogates, Co private-use characters, and Cn unassigned code points. ZERO WIDTH SPACE U+200B is specifically Cf, not Zs; Unicode notes that it was reclassified to distinguish it from actual space characters. Thus a “collapse Unicode whitespace” step usually does not catch ZWSP, ZWJ, or ZWNJ. source

BERT-style preprocessing is unusually destructive. The standard Hugging Face BertNormalizer(clean_text=True) removes control characters, converts recognized whitespace to ordinary spaces, lowercases by default, and—when configured like bert-base-uncased—strips accents. In an algorithm-level reproduction with the official bert-base-uncased vocabulary, a<ZWJ>b, a<ZWNJ>b, a<ZWSP>b, a<SHY>b, a<LRM>b, a<RLO>b, a<PUA>b, and a<U+0378>b all became ab and therefore tokenized identically as token ID 11113; Unicode space characters instead created word boundaries. source

GPT-2-style byte-level BPE is almost the opposite. GPT-2 maps the UTF-8 bytes of surviving input into a reversible 256-symbol Unicode alphabet and applies BPE to those symbols. Consequently, valid Unicode invisibles generally cannot disappear merely because the vocabulary lacks their code point: every UTF-8 byte is representable. In our reproduction of OpenAI GPT-2's reference algorithm and merge table, a<ZWJ>b became tokens ['a','âĢ','į','b'] with IDs [64,447,235,65]; NUL became ['a','Ā','b'] [64,188,65]; U+FE0F VARIATION SELECTOR-16 became ['a','ï¸ı','b'] [64,37929,65]. The odd-looking strings are the GPT-2 byte-to-Unicode display alphabet, not textual corruption. OpenAI's published GPT-2 tokenizer is the basis of this scheme, and Hugging Face's current byte-level normalizer documents the same one-to-one byte mapping. source

SentencePiece is neither inherently preserving nor inherently deleting. Its subword model may be BPE or Unigram, but text normalization happens first. The current project documents nmt_nfkc as the default normalization rule, with duplicate whitespace removal and conversion of spaces to the visible meta-symbol ▁ enabled by default. SentencePiece 0.2.2, released in July 2026, updated its normalization rules to newer ICU/Unicode data; an earlier release explicitly changed ZWJ handling so that ZWJ was no longer treated as whitespace. source

In a directly executed SentencePiece 0.2.1 experiment using nmt_nfkc, ZWJ was preserved, while ZWNJ, ZWSP, LRM, RLM, NBSP, OGHAM SPACE, EM SPACE, NNBSP, TAB, and LF were normalized to spaces; DEL disappeared; soft hyphen, bidi override U+202E, VS16, private-use U+E000, and unassigned U+0378 survived. Because the latest stable 0.2.2 could not be installed in the research runtime, these precise edge-case results should be treated as 0.2.1 measurements, not claims about 0.2.2. The 0.2.2 release notes explicitly say the normalization tables changed. source

The practical rule is therefore: inspect the exact serialized tokenizer, not merely its algorithm label. “BPE,” “WordPiece,” and “Unigram” describe how surviving symbols are segmented; they do not tell you whether ZWJ was deleted, NBSP was folded, or whitespace was collapsed before segmentation. This is particularly important for security-sensitive processing, because Unicode warns that join controls can be semantically required in real languages and that mapping/deletion policies should not be confused with Unicode compatibility normalization itself. source

Unicode character taxonomy and normalization

Unicode 17.0.0 is the current Unicode Character Database version as of the research date, August 16, 2026. Its General_Category distinguishes controls (Cc), format characters (Cf), surrogates (Cs), private-use characters (Co), unassigned code points (Cn), space separators (Zs), and nonspacing marks (Mn). Unicode emphasizes that these categories are first-order classifications rather than a complete description of how a character behaves. source

Character classes that matter for tokenization

Unicode explicitly describes Cc as C0/C1 controls, Cf as format controls, Cs as surrogate code points, Co as private-use, and Cn as reserved/unassigned or noncharacters; Zs means a space separator of non-zero width, while Mn is a nonspacing combining mark. It also specifically records U+200B ZERO WIDTH SPACE as Cf, not Zs, and lists U+1680 and U+2000–U+200A under the White_Space property. source

ZWJ and ZWNJ deserve special treatment rather than blanket deletion. Unicode's security guidance notes that some languages require these join controls for correct spelling and shaping; it therefore recommends contextual restrictions rather than universally declaring them invalid. source

Bidi controls are also materially different from ordinary spaces. Unicode identifies characters such as LRE, RLE, LRO, RLO, PDF and the isolate controls as explicit directional formatting controls, and Unicode's security guidance specifically warns about bidi overrides in security-sensitive names and identifiers. source

What NFC, NFD, NFKC, and NFKD actually do

Unicode defines the four normalization forms as follows: NFD is canonical decomposition; NFC is canonical decomposition followed by canonical composition; NFKD is compatibility decomposition; NFKC is compatibility decomposition followed by canonical composition. Compatibility equivalence is deliberately broader than canonical equivalence and can remove distinctions such as width, typographic form, or line-breaking behavior. source

The practical consequences for invisible characters are more limited than many preprocessing descriptions imply:

The NBSP result is explicitly illustrated by Unicode as a compatibility equivalence: the line-breaking distinction between NBSP and SPACE is erased under compatibility normalization. More generally, NFKC/NFKD are intentionally stronger transformations than NFC/NFD and can erase distinctions that are semantically relevant in specialized contexts. source

A local sanity check using Python's Unicode 15.1 tables produced exactly the results in the table above for the representative code points; Unicode normalization stability makes the long-established mappings relevant, but the Unicode 17.0 data remains the normative reference. The experiment also confirmed that plain NFKC did not delete ZWJ, ZWNJ, ZWSP, soft hyphen, bidi controls, VS16, NUL, DEL, PUA, or U+0378. Unicode's own documentation distinguishes plain compatibility normalization from application profiles that add extra mappings. source

This distinction is security-critical. Unicode's security report gives an historical identifier profile in which ZWJ was additionally mapped to nothing, but it presents that deletion as an extra mapping beyond NFKC, not a property of NFKC itself. source

Preprocessing pipelines and where characters disappear

There is no single “modern LLM preprocessing pipeline.” The most useful abstraction is:

flowchart LR
    A["Bytes / API string"] --> B["Decode & validate Unicode"]
    B --> C{"Unicode normalization?"}
    C -->|"None"| D
    C -->|"NFC / NFD"| D["Canonicalized text"]
    C -->|"NFKC / NFKD"| D
    D --> E{"Additional text cleaning?"}
    E -->|"case/lowercase"| F
    E -->|"control filtering"| F
    E -->|"whitespace mapping/collapse"| F
    E -->|"format-char policy"| F["Normalized model input"]
    F --> G["Pre-tokenizer / boundary detection"]
    G --> H{"Tokenizer representation"}
    H -->|"code points / pieces"| I["BPE / WordPiece / Unigram"]
    H -->|"UTF-8 bytes"| J["byte-to-symbol mapping"]
    J --> I
    I --> K["Token strings + token IDs"]
    K --> L["Special tokens / truncation / padding"]

Hugging Face Tokenizers makes these stages explicit: normalizer, pre-tokenizer, model, post-processor and decoder are separable components. The pre-tokenizer is especially important because a split introduced there prevents the model from subsequently merging symbols across that boundary. source

Typical operations and their effects

Unicode's categories explain why a generic regex such as “remove \p{C}” is substantially stronger than “remove controls”: C is the union of Cc, Cf, Cs, Co, and Cn. Such a rule would remove not merely NUL and DEL but also join controls, bidi controls, private-use content and currently unassigned code points. source

BERT cleaning

Hugging Face's current BertNormalizer documents clean_text=True as removing control characters and replacing all whitespace with the ordinary form; it additionally supports lowercasing and accent stripping and defaults to lowercase. source

That combination explains several otherwise surprising results:

• ZWJ/ZWNJ/ZWSP/SHY/bidi controls are Cf and are removed by BERT-style control detection.
• NBSP, OGHAM SPACE, U+2000–U+200A, U+202F, U+205F and U+3000 are spacing characters and become ordinary word-separating spaces.
• TAB/LF/CR are specially interpreted as whitespace rather than simply discarded in the original BERT-style logic.
• Variation selectors such as VS16 are Mn, not Cf; nevertheless bert-base-uncased-style NFD + accent/mark stripping removes them.
• PUA (Co), surrogate (Cs) and unassigned (Cn) values are also caught by an implementation that defines “control” as Unicode category beginning with C.

The Hugging Face implementation documents the overall cleaning contract, while the BERT model itself uses WordPiece subwords. source

SentencePiece normalization

SentencePiece explicitly embeds normalization in its model. Current options document:

normalization_rule_name="nmt_nfkc" by default; nfkc, case-folding variants and identity are alternatives; add_dummy_prefix=true; remove_extra_whitespaces=true; and escape_whitespaces=true, which turns normal spaces into the visible meta-symbol ▁. source

Its current Python documentation demonstrates normalization of "Hello World." to "▁Hello▁World.", showing both duplicate-space collapse and the visible whitespace marker. The same documentation emphasizes that the exact normalization behavior can be read from the model or exercised independently with SentencePieceNormalizer. source

nmt_nfkc should not be equated with bare Unicode NFKC. It is an NMT-oriented mapping layer with additional cleaning/whitespace behavior. This becomes obvious experimentally: in the tested SentencePiece 0.2.1 normalizer, ZWNJ and ZWSP became spaces even though plain Unicode NFKC preserves both. SentencePiece's release history also records normalization behavior as an evolving implementation detail; version 0.1.96 specifically stopped treating ZWJ as whitespace, and 0.2.2 updated the normalization rules to current ICU/Unicode data. source

Hugging Face has its own Nmt normalizer intended to match the original SentencePiece NMT preprocessing. Its current source documentation says it removes controls, normalizes whitespace and replaces selected Unicode characters; its published example maps "Hello\x00World" to "Hello World". This contrasts with the SentencePiece 0.2.1 experiment below, where NUL survived the normalizer and was handled by byte fallback. That is precisely why version and implementation must be recorded when auditing edge cases. source

Tokenization algorithms and byte-level versus character-level behavior

BPE

BPE begins from an initial alphabet and repeatedly applies learned pair merges. The NLP adaptation introduced by Sennrich, Haddow and Birch learns frequent subword combinations so rare words can be represented through smaller units. source

For invisible Unicode, the key question is therefore what the initial symbols are:

• With character/code-point BPE, an invisible code point must either occur in the initial alphabet/vocabulary, be mapped to an unknown token, or be handled by fallback.
• With byte-level BPE, a valid Unicode character first becomes UTF-8 bytes. Because all 256 byte values have representations, an unseen code point is still expressible.
• A pre-tokenizer can impose boundaries before BPE, preventing merges across an invisible or whitespace symbol even when the BPE vocabulary otherwise contains possible merges. Hugging Face explicitly documents that pre-tokenization boundaries constrain subsequent model merging. source

WordPiece

WordPiece, as used by BERT, performs greedy longest-match segmentation within each pre-tokenized word. Hugging Face describes it as choosing long vocabulary pieces first, conventionally using ## to mark non-initial pieces; the BERT paper specifies a WordPiece vocabulary for the model. source

This means an invisible code point that survives BERT's BasicTokenizer can make an otherwise known word fail to match or break into different subwords. In bert-base-uncased, however, many of the troublesome Cf/C* values disappear before WordPiece ever sees them, so insertion can instead create a normalization collision: visually and bytewise different input strings map to the same token sequence. source

Unigram and SentencePiece

SentencePiece is a framework, not one segmentation algorithm: it supports both BPE and a Unigram language-model tokenizer and can train directly on raw sentences. source

With Unigram, the surviving normalized input is segmented using a probabilistic vocabulary rather than deterministic merge ranks. But Unicode questions are still primarily settled before that scoring step: if nmt_nfkc turns ZWNJ into a space, no Unigram candidate can recover the original ZWNJ afterward. Conversely, a surviving code point may become its own piece, join an adjacent piece, map to <unk>, or fall back to byte pieces depending on the particular model. Current SentencePiece also supports byte fallback, in which an unsupported character is represented by its UTF-8 bytes. source

GPT-2 byte-level BPE

GPT-2's important property is not merely “BPE” but byte-level BPE. The implementation converts UTF-8 bytes to a reversible Unicode display alphabet before applying BPE. Hugging Face's current ByteLevel documentation describes this mapping as assigning every byte 0–255 a unique visible character, permitting arbitrary byte values without requiring an unknown token. source

As a result, insertion of an invisible character generally:

1. adds one to four UTF-8 bytes;
2. changes the byte-to-Unicode symbol stream;
3. may create a separate regex pre-tokenization span;
4. may be compressed by learned BPE merges into one or several tokens;
5. changes all relevant token IDs without the character needing a dedicated Unicode-code-point vocabulary entry.

Thus U+200B does not become a token literally named “ZERO WIDTH SPACE.” Its UTF-8 byte sequence E2 80 8B becomes GPT-2's byte alphabet; the learned merges happen to combine those bytes into the token string âĢĭ, ID 9525, in the tested GPT-2 vocabulary.

GPT-NeoX

GPT-NeoX-20B also uses byte-level BPE, but it is a different trained tokenizer. Hugging Face's current GPT-NeoX documentation states that the tokenizer is backed by the Tokenizers library and uses byte-level BPE; it was trained with additional treatment of whitespace. The documentation gives "Hello world" → [15496, 995] and " Hello world" → [18435, 995], demonstrating that the leading-space state affects IDs. source

The implication—not an experimentally measured NeoX ID claim—is that a surviving invisible character is byte-representable in GPT-NeoX as well, but its exact merge pattern and IDs must be read from the NeoX vocabulary/merges rather than copied from GPT-2. source

T5 SentencePiece

The current google-t5/t5-small repository contains a 792-kB spiece.model plus a serialized tokenizer.json, confirming that T5's tokenizer is SentencePiece-based and that tokenizer behavior is embodied in model artifacts rather than just an algorithm name. source

It would therefore be unsafe to report the synthetic SentencePiece IDs below as T5 IDs. The T5 binary SentencePiece model could not be executed in the research sandbox, so T5-specific invisible-character IDs are deliberately reported as not measured rather than inferred from generic SentencePiece behavior.

Hugging Face Tokenizers and Node/“tokenizers.js”

Hugging Face Tokenizers 0.23.1 is the latest Python release found as of August 16, 2026, uploaded April 27, 2026. It supports configurable BPE, Unigram, WordPiece and other pipeline components rather than imposing one Unicode policy. source

There is an official Node.js binding in the same Hugging Face repository; it is a binding over the Rust implementation rather than a separate JavaScript tokenizer algorithm. The repository instructs users to install the package as tokenizers and can load a serialized tokenizer.json. source

The npm package itself is notably stale: npm lists tokenizers 0.13.3, published in 2023, while a 2026 Hugging Face issue records a failure publishing newer Node binary artifacts. Thus “tokenizers.js” should not be assumed to mean the current 0.23.1 Python/Rust release. source

Experimental results and implementation comparison

Methodology and version scope

Current stable releases identified for this report were Hugging Face Tokenizers 0.23.1, Transformers 5.15.0 (uploaded August 10, 2026), SentencePiece 0.2.2 (July 12, 2026), and TensorFlow Text 2.20.1 (March 10, 2026). The official Node tokenizers npm package remains at 0.13.3. source

Not every latest package could be installed in the sandbox. The experiment therefore separates executed results from source-derived/documented behavior:

• GPT-2: algorithm-level execution using OpenAI's published byte-to-Unicode/BPE procedure and the GPT-2 merge/vocabulary ordering; no external Transformers runtime was used. source
• BERT: algorithm-level execution of BERT-style cleaning + greedy WordPiece against the official bert-base-uncased vocabulary; special [CLS]/[SEP] tokens were intentionally omitted so the table shows only text-derived pieces. source
• SentencePiece: the actual installed sentencepiece 0.2.1 Python/C++ implementation was executed. A controlled 277-piece synthetic Unigram model with nmt_nfkc and byte_fallback=True was trained so every surviving test input could receive an ID. Those IDs are experiment-local, not T5 IDs. The latest stable is 0.2.2, whose normalization tables changed. source
• T5, GPT-NeoX and Node bindings: exact invisible-character IDs were not executable in this sandbox, so no IDs are invented. Primary model/library documentation is used for their architecture and preprocessing behavior. source

In the tables, notation such as a<ZWJ>b means that the actual Unicode character, not the ASCII text <ZWJ>, was inserted between a and b.

Requested tokenizer comparison

The implementation descriptions in this table are grounded in the current Hugging Face pipeline/normalizer documentation, SentencePiece's current options, GPT-NeoX's tokenizer documentation, and the T5 model artifacts. source

Character-by-character measured results

The following is the most direct answer to whether characters disappear, cause boundaries, or become tokens. GPT-2 and BERT IDs are from the published model vocabularies; SentencePiece IDs refer only to the controlled synthetic model described above.

The table exposes four qualitatively different outcomes:

Separate token(s). GPT-2 NUL, TAB, DEL and most format characters create byte-derived token pieces of their own.

Merged byte token. GPT-2's ZWSP bytes happen to merge into a single learned byte-BPE piece, ID 9525; VS16 similarly becomes one merged byte sequence, ID 37929.

Boundary creation. BERT converts Unicode spaces to U+0020 and therefore turns ab into two words. SentencePiece nmt_nfkc does the same for several additional zero-width formatting characters in the tested version.

Disappearance / collision. BERT maps many distinct injected values to exactly ab; thus the tokenizer cannot distinguish the original clean string from one containing ZWJ, ZWSP, SHY, bidi controls, PUA, etc. This is a many-to-one preprocessing collision, not a WordPiece property. The documented BertNormalizer cleaning behavior explains why. source

Canonically equivalent text can tokenize differently without normalization

The GPT-2 experiment also demonstrates why Unicode normalization can matter even when no “invisible attack” is intended:

Unicode considers the two spellings canonically equivalent; NFC would make them identical. GPT-2 performs no such Unicode canonical normalization, so byte-level preservation leads to different token sequences. BERT's uncased preprocessing decomposes and strips the accent, making both even more aggressively equivalent as cafe. Unicode defines the canonical equivalence underlying the first pair. source

Lone surrogates are a separate problem

For diagnostic purposes only, the GPT-2 reproduction was also fed Python's internal string a\ud800b using the non-standard surrogatepass UTF-8 error handler. It yielded byte-symbol pieces ['a','í','ł','Ģ','b'], IDs [64,169,254,222,65].

That result must not be interpreted as normal GPT-2 support for U+D800. U+D800–U+DFFF are surrogate code points (Cs) used by UTF-16; a strict UTF-8 serialization/API boundary will ordinarily reject a lone surrogate before BPE runs. Unicode 17.0 explicitly classifies this range as surrogate code points. source

Reproducing and extending the experiments

The safest way to reproduce tokenizer behavior is to print all three layers: the input in escaped form, the tokenizer's tokens, and the integer IDs. Do not rely on terminal rendering, because the entire point of many test characters is that they render invisibly.

Unicode inspection and normalization

import unicodedata as ud

TESTS = {
    "ZWJ": "\u200D",
    "ZWNJ": "\u200C",
    "ZWSP": "\u200B",
    "SOFT_HYPHEN": "\u00AD",
    "NBSP": "\u00A0",
    "OGHAM_SPACE": "\u1680",
    "EM_SPACE": "\u2003",
    "NARROW_NBSP": "\u202F",
    "IDEOGRAPHIC_SPACE": "\u3000",
    "NUL": "\u0000",
    "TAB": "\u0009",
    "DEL": "\u007F",
    "LRM": "\u200E",
    "RLM": "\u200F",
    "RLO": "\u202E",
    "LRI": "\u2066",
    "VS16": "\uFE0F",
    "PRIVATE_USE": "\uE000",
    "UNASSIGNED_EXAMPLE": "\u0378",
}

def esc(s: str) -> str:
    return s.encode("unicode_escape").decode("ascii")

for name, ch in TESTS.items():
    print(
        name,
        f"U+{ord(ch):04X}",
        ud.category(ch),
        ud.name(ch, "<no Unicode name>")
    )
    for form in ("NFC", "NFD", "NFKC", "NFKD"):
        print(" ", form, esc(ud.normalize(form, ch)))

Python's Unicode tables depend on the Python build, so for audit-quality testing record unicodedata.unidata_version alongside the result. The normative definitions of the four forms come from UAX #15, while the current UCD for this report is Unicode 17.0.0. source

Hugging Face / GPT-2 / BERT / NeoX / T5 harness

With current Transformers, a compact test harness is:

from importlib.metadata import version
from transformers import AutoTokenizer

CASES = {
    "plain": "ab",
    "ZWJ": "a\u200Db",
    "ZWNJ": "a\u200Cb",
    "ZWSP": "a\u200Bb",
    "SHY": "a\u00ADb",
    "NBSP": "a\u00A0b",
    "EM_SPACE": "a\u2003b",
    "NUL": "a\u0000b",
    "DEL": "a\u007Fb",
    "LRM": "a\u200Eb",
    "RLO": "a\u202Eb",
    "VS16": "a\uFE0Fb",
    "PUA": "a\uE000b",
    "UNASSIGNED": "a\u0378b",
}

MODELS = {
    "gpt2": "openai-community/gpt2",
    "bert": "google-bert/bert-base-uncased",
    "t5": "google-t5/t5-small",
    "gpt-neox": "EleutherAI/gpt-neox-20b",
}

print("transformers =", version("transformers"))

for label, model_id in MODELS.items():
    print(f"\n### {label}: {model_id}")
    tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)

    # Fast tokenizers expose their serialized normalizer/pre-tokenizer.
    backend = getattr(tok, "backend_tokenizer", None)
    if backend is not None:
        print("normalizer:", backend.normalizer)
        print("pre_tokenizer:", backend.pre_tokenizer)

    for name, text in CASES.items():
        encoded = tok(text, add_special_tokens=False)
        ids = encoded["input_ids"]
        pieces = tok.convert_ids_to_tokens(ids)

        print(
            name,
            repr(text),
            "pieces=", [repr(x) for x in pieces],
            "ids=", ids,
        )

The current Transformers package on PyPI was 5.15.0 on August 16, 2026; Hugging Face Tokenizers was 0.23.1. Pinning versions is essential because tokenizer serialization and normalization implementations are part of the behavior under test. source

For BERT, run both use_fast=True and use_fast=False when compatibility matters. Any difference is itself valuable evidence that the Python and Rust preprocessing paths are not perfectly aligned.

Inspecting a Hugging Face tokenizer directly

Hugging Face Tokenizers exposes individual normalizers:

from tokenizers.normalizers import (
    NFC, NFD, NFKC, NFKD,
    BertNormalizer, ByteLevel
)

samples = [
    "a\u200Db",       # ZWJ
    "a\u200Bb",       # ZWSP
    "a\u00A0b",       # NBSP
    "a\u0000b",       # NUL
    "a\uFE0Fb",       # VS16
]

normalizers = {
    "NFC": NFC(),
    "NFD": NFD(),
    "NFKC": NFKC(),
    "NFKD": NFKD(),
    "BERT": BertNormalizer(
        clean_text=True,
        lowercase=True
    ),
    "BYTELEVEL": ByteLevel(),
}

for name, norm in normalizers.items():
    print("\n", name)
    for s in samples:
        print(repr(s), "=>", repr(norm.normalize_str(s)))

The current Hugging Face documentation explicitly shows its ByteLevel normalizer turning newline into the visible byte symbol Ċ, and documents the BERT normalizer's control cleaning, whitespace normalization, lowercasing and accent handling. source

SentencePiece normalization and token IDs

For an existing model such as T5, the most informative operation is to ask the model itself how it normalizes:

import sentencepiece as spm

sp = spm.SentencePieceProcessor(
    model_file="spiece.model"
)

for name, text in CASES.items():
    print(
        name,
        "input=", repr(text),
        "normalized=", repr(sp.normalize(text)),
        "pieces=", sp.encode(text, out_type=str),
        "ids=", sp.encode(text, out_type=int),
    )

For separating Unicode normalization from model segmentation:

import sentencepiece as spm

for rule in ("identity", "nfkc", "nmt_nfkc"):
    normalizer = spm.SentencePieceNormalizer(
        rule_name=rule,
        add_dummy_prefix=False,
        escape_whitespaces=False,
        remove_extra_whitespaces=False,
    )

    print("\nRULE:", rule)
    for name, text in CASES.items():
        print(name, repr(normalizer.normalize(text)))

Current SentencePiece documentation exposes SentencePieceNormalizer precisely for this purpose, and current training options identify nmt_nfkc as the default normalization rule unless overridden. source

For a production audit, also serialize the package versions:

from importlib.metadata import version
import unicodedata

for package in ("transformers", "tokenizers", "sentencepiece"):
    try:
        print(package, version(package))
    except Exception:
        print(package, "not installed")

print("Python Unicode DB:", unicodedata.unidata_version)

That metadata is not bureaucratic detail: SentencePiece 0.2.2 explicitly updated Unicode normalization rules relative to previous versions, while the npm Hugging Face Node binding is still on a substantially older release. source

Practical recommendations and conclusions

Preserve raw input separately from model input

For systems that care about provenance, security, moderation, code analysis, document integrity or forensic reproducibility, store or hash the original byte sequence before performing Unicode normalization. Once a normalizer has collapsed NBSP into SPACE, removed a bidi control or transformed ZWNJ into a boundary, the tokenizer output cannot reconstruct what was originally supplied.

This is particularly important for BERT-like cleaners, where many distinct strings can intentionally collapse onto the same token sequence, and for SentencePiece models with aggressive NMT normalization. Hugging Face and SentencePiece both treat normalization as an explicit preprocessing stage, so applications should log that stage rather than treating tokens as a lossless representation of source text. source

Use NFC as the conservative default when semantic preservation matters

For ordinary natural-language model input, NFC is generally the safer baseline when the goal is to canonicalize equivalent spellings without removing compatibility distinctions such as NBSP versus SPACE or full-width versus ordinary forms. NFC resolves the café versus cafe + combining acute canonical-equivalence problem while avoiding much of NFKC's deliberately stronger folding. Unicode itself distinguishes canonical from compatibility equivalence for exactly this reason. source

NFKC can be appropriate for search, matching, indexing, deduplication or deliberately compatibility-insensitive NLP, but applications should regard it as a semantic policy decision, not harmless cleanup. It changes NBSP and many other compatibility spacing characters to normal spaces and similarly folds many typographic/width distinctions. source

Define “whitespace” explicitly

Do not write a policy saying merely “collapse whitespace.” Specify whether it means:

U+0020 only; Unicode White_Space; general category Zs; language/runtime isspace(); regex \s; or an application-defined collection that also contains zero-width formatting characters.

Those sets are not identical. Unicode specifically classifies U+200B ZERO WIDTH SPACE as a Cf format control, while characters such as U+1680 and U+2000–U+200A are Unicode whitespace/space separators. source

For source code, indentation-sensitive formats, tables, or prompt structures, collapsing whitespace may also destroy meaningful structure independently of Unicode security concerns.

Do not delete all Cf or all C* characters by default

A blanket category.startswith("C") deletion policy removes:

Cc controls plus Cf formatting characters plus Cs surrogates plus Co private-use characters plus Cn unassigned/noncharacter values. Unicode defines the categories this way. source

Such a rule can therefore destroy legitimate ZWJ/ZWNJ orthography, emoji sequences, private-use notation and intentional directional metadata. Unicode specifically notes legitimate language use of ZWJ and ZWNJ. source

A better policy is contextual:

reject malformed encoding/surrogates at the boundary; control C0/C1 characters according to protocol needs; treat bidi overrides as high-risk; retain or validate join controls according to language context; preserve variation selectors when grapheme/presentation identity matters; and make PUA handling an explicit domain policy.

Distinguish validation from normalization

For security-sensitive fields, there are three separate operations:

mapping, where an input becomes another string;

prohibition, where an input is rejected;

normalization, where equivalent Unicode representations are canonicalized.

They should not be silently collapsed into one regex. Unicode's security guidance specifically distinguishes mapping from prohibition and recommends applying mappings before prohibition where both are used. source

For example, deleting U+202E RLO and accepting the rest is operationally different from rejecting any field that contains bidi overrides. The latter retains evidence that the suspicious character was present and prevents two distinct inputs from silently collapsing to one accepted identifier.

Audit before and after tokenization

For hostile-input or robustness testing, log at least:

raw byte length
Unicode scalar/code-point sequence
Unicode categories/properties
normalized string
pre-tokenized spans
token strings
token IDs
decoded token output

This distinguishes five failure modes that otherwise look identical in the final token sequence: decoder replacement, Unicode normalization, application cleaning, pre-tokenization boundaries, and subword-vocabulary behavior.

Test the exact model artifact, not “the tokenizer family”

The experimental evidence makes this the strongest operational conclusion:

“BPE” is not a Unicode policy. “SentencePiece” is not a Unicode policy. “WordPiece” is not a Unicode policy. “byte-level” is not a complete preprocessing specification.

Hugging Face Tokenizers deliberately allows normalizer and pre-tokenizer configuration independently of the BPE/WordPiece/Unigram model, while SentencePiece embeds normalization rules in each model. GPT-NeoX uses byte-level BPE like GPT-2 but a different whitespace-rich vocabulary; T5 packages a specific SentencePiece model. source

For regression testing, keep a fixed corpus containing at least:

ZWJ, ZWNJ, ZWSP, SHY
NBSP, OGHAM SPACE
U+2000–U+200A
U+202F, U+205F, U+3000
NUL, TAB, LF, CR, DEL
LRM, RLM
U+202A–U+202E
U+2066–U+2069
VS15/VS16
a representative supplementary variation selector
BMP and supplementary PUA
a currently unassigned scalar value
precomposed/decomposed canonical pairs
ordinary, doubled, leading and trailing ASCII spaces

Record normalized text, token strings and IDs in version control. A tokenizer-library upgrade should be treated as a behavior-changing dependency upgrade whenever any of those outputs change. SentencePiece 0.2.2's explicit normalization-table update demonstrates that this is not merely theoretical. source

Bottom line

The central dividing line is not simply “invisible versus visible” but information-preserving versus information-collapsing preprocessing.

GPT-2-style byte-level BPE is strongly preservation-oriented after it receives a valid Unicode string: invisible code points generally become byte-derived tokens, often increasing token count and altering boundaries but remaining distinguishable.

BERT-style preprocessing is strongly collapsing: many format/control/unassigned/private characters disappear before WordPiece, while Unicode spaces become word boundaries. This can be desirable for robust lexical modeling but creates large equivalence classes of distinct source strings.

SentencePiece is configurable and model-specific: nmt_nfkc can transform substantially more than plain NFKC, while identity can preserve text much more literally. Its whitespace marker ▁, model-contained normalization rules and optional byte fallback make it especially important to inspect the actual .model file rather than infer behavior from the word “SentencePiece.” source

T5 and GPT-NeoX inherit these general principles but have model-specific vocabularies and preprocessing artifacts; exact invisible-character IDs should therefore be measured against their actual released tokenizer files rather than extrapolated from T5's generic SentencePiece family or GPT-NeoX's generic byte-BPE label. source

Finally, normalization should be chosen for semantics, filtering should be chosen for policy, and tokenization should be treated as a downstream representation step. Treating NFKC, whitespace collapse, control deletion and subword segmentation as interchangeable “cleanup” is the root cause of most surprising invisible-Unicode tokenizer behavior. Unicode's standards and the implementation experiments above consistently point to the same engineering discipline: preserve the original, make every transformation explicit, test the exact tokenizer artifact, and attach versioned Unicode/tokenizer semantics to every security- or reproducibility-critical pipeline. source