# Invisible Unicode and Modern Language-Model Tokenizers

## Executive summary

Invisible and near-invisible Unicode characters are **not a single tokenizer category**. Their fate is determined by a sequence of independent layers: Unicode decoding, normalization, application-specific cleaning, whitespace handling, pre-tokenization, subword segmentation, and finally vocabulary lookup or byte fallback. Two tokenizers that are both described as “BPE” can therefore behave radically differently on the same invisible character. Hugging Face explicitly models normalization, pre-tokenization, the subword model, and post-processing as separate pipeline components, while SentencePiece packages its normalization rules with the model itself. citeturn23search8turn27view4

The most important findings are:

**Unicode normalization is not sanitization.** NFC and NFD perform canonical normalization; NFKC and NFKD additionally apply compatibility decomposition. They do **not**, as a general rule, remove zero-width joiners, zero-width non-joiners, zero-width spaces, bidi controls, C0 controls, private-use characters, or other arbitrary invisibles. A particularly important exception is compatibility-equivalent spacing: NFKC/NFKD map U+00A0 NO-BREAK SPACE to ordinary U+0020 SPACE and similarly fold many Unicode space characters. Unicode itself explicitly illustrates NBSP → SPACE as a compatibility equivalence. citeturn38view1

**Whitespace, format controls, and combining marks must not be conflated.** Unicode 17.0 distinguishes `Zs` space separators from `Cf` format controls, `Cc` controls, `Mn` nonspacing marks, `Cs` surrogates, `Co` private-use characters, and `Cn` unassigned code points. ZERO WIDTH SPACE U+200B is specifically `Cf`, not `Zs`; Unicode notes that it was reclassified to distinguish it from actual space characters. Thus a “collapse Unicode whitespace” step usually does not catch ZWSP, ZWJ, or ZWNJ. citeturn38view0turn40view2

**BERT-style preprocessing is unusually destructive.** The standard Hugging Face `BertNormalizer(clean_text=True)` removes control characters, converts recognized whitespace to ordinary spaces, lowercases by default, and—when configured like `bert-base-uncased`—strips accents. In an algorithm-level reproduction with the official `bert-base-uncased` vocabulary, `a<ZWJ>b`, `a<ZWNJ>b`, `a<ZWSP>b`, `a<SHY>b`, `a<LRM>b`, `a<RLO>b`, `a<PUA>b`, and `a<U+0378>b` all became `ab` and therefore tokenized identically as token ID `11113`; Unicode space characters instead created word boundaries. citeturn27view5turn23search2

**GPT-2-style byte-level BPE is almost the opposite.** GPT-2 maps the UTF-8 bytes of surviving input into a reversible 256-symbol Unicode alphabet and applies BPE to those symbols. Consequently, valid Unicode invisibles generally cannot disappear merely because the vocabulary lacks their code point: every UTF-8 byte is representable. In our reproduction of OpenAI GPT-2's reference algorithm and merge table, `a<ZWJ>b` became tokens `['a','âĢ','į','b']` with IDs `[64,447,235,65]`; NUL became `['a','Ā','b']` `[64,188,65]`; U+FE0F VARIATION SELECTOR-16 became `['a','ï¸ı','b']` `[64,37929,65]`. The odd-looking strings are the GPT-2 byte-to-Unicode display alphabet, not textual corruption. OpenAI's published GPT-2 tokenizer is the basis of this scheme, and Hugging Face's current byte-level normalizer documents the same one-to-one byte mapping. citeturn0search5turn27view5

**SentencePiece is neither inherently preserving nor inherently deleting.** Its subword model may be BPE or Unigram, but text normalization happens first. The current project documents `nmt_nfkc` as the default normalization rule, with duplicate whitespace removal and conversion of spaces to the visible meta-symbol `▁` enabled by default. SentencePiece 0.2.2, released in July 2026, updated its normalization rules to newer ICU/Unicode data; an earlier release explicitly changed ZWJ handling so that ZWJ was no longer treated as whitespace. citeturn27view4turn27view3turn34search3

In a directly executed SentencePiece 0.2.1 experiment using `nmt_nfkc`, ZWJ was preserved, while ZWNJ, ZWSP, LRM, RLM, NBSP, OGHAM SPACE, EM SPACE, NNBSP, TAB, and LF were normalized to spaces; DEL disappeared; soft hyphen, bidi override U+202E, VS16, private-use U+E000, and unassigned U+0378 survived. Because the latest stable 0.2.2 could not be installed in the research runtime, these precise edge-case results should be treated as **0.2.1 measurements, not claims about 0.2.2**. The 0.2.2 release notes explicitly say the normalization tables changed. citeturn27view3turn39view0

**The practical rule is therefore: inspect the exact serialized tokenizer, not merely its algorithm label.** “BPE,” “WordPiece,” and “Unigram” describe how surviving symbols are segmented; they do not tell you whether ZWJ was deleted, NBSP was folded, or whitespace was collapsed before segmentation. This is particularly important for security-sensitive processing, because Unicode warns that join controls can be semantically required in real languages and that mapping/deletion policies should not be confused with Unicode compatibility normalization itself. citeturn25search2


## Unicode character taxonomy and normalization

Unicode 17.0.0 is the current Unicode Character Database version as of the research date, August 16, 2026. Its `General_Category` distinguishes controls (`Cc`), format characters (`Cf`), surrogates (`Cs`), private-use characters (`Co`), unassigned code points (`Cn`), space separators (`Zs`), and nonspacing marks (`Mn`). Unicode emphasizes that these categories are first-order classifications rather than a complete description of how a character behaves. citeturn38view0turn40view2

### Character classes that matter for tokenization

| Class | Representative code points | Unicode category | Why it is “invisible” or unusual | Tokenization significance |
|---|---|---:|---|---|
| Zero-width joiner | U+200D ZWJ | `Cf` | Affects joining/shaping; also fundamental to many emoji ZWJ sequences | Removing it can change written-language shaping or turn one emoji sequence into several characters |
| Zero-width non-joiner | U+200C ZWNJ | `Cf` | Suppresses joining in scripts where joining would otherwise occur | Can be linguistically meaningful; should not be blindly treated as whitespace |
| Zero-width space | U+200B ZWSP | `Cf` | Provides a zero-width line-break opportunity | Not a `Zs` space; therefore many “whitespace” operations miss it |
| Soft hyphen | U+00AD SHY | `Cf` | Normally invisible; indicates a discretionary hyphenation point | Often removed by “control/format cleaning,” but not by ordinary NFC/NFKC |
| No-break space | U+00A0 NBSP | `Zs` | Visually space-like but prevents normal line breaking | NFKC/NFKD compatibility-fold it to U+0020 |
| Ogham space | U+1680 | `Zs` | Visible width depends on font | Unicode whitespace; unlike most compatibility spaces, it is not simply an NFKC alias for ASCII space |
| Typographic spaces | U+2000–U+200A | mostly `Zs` | EN QUAD through HAIR SPACE, with varying widths | Often collapsed by tokenizer normalizers or compatibility-folded |
| Narrow no-break space | U+202F | `Zs` | Narrow nonbreaking separator | Frequently folded to ordinary space by NFKC |
| Medium mathematical space | U+205F | `Zs` | Mathematical spacing | Compatibility-folded to ordinary space |
| Ideographic space | U+3000 | `Zs` | Full-width/CJK space | Compatibility-folded to ordinary space |
| Line/paragraph separators | U+2028, U+2029 | `Zl`, `Zp` | Semantic line and paragraph separators | Regex and tokenizer whitespace rules may treat them as boundaries |
| C0 controls | U+0000–U+001F | `Cc` | NUL, BEL, BS, TAB, LF, CR, ESC, etc. | Some are deleted; TAB/LF/CR are often specially retained as whitespace |
| DEL | U+007F | `Cc` | Legacy deletion/control code | Often removed by text cleaners; byte tokenizers can preserve its byte |
| Bidi marks | U+200E LRM, U+200F RLM | `Cf` | Affect direction without displaying a glyph | Can alter presentation and security interpretation |
| Bidi embeddings/overrides | U+202A–U+202E | `Cf` | Explicitly manipulate bidi ordering | Particularly security-sensitive |
| Bidi isolates | U+2066–U+2069 | `Cf` | Isolate directional spans | Can be invisible yet change display ordering |
| Variation selectors | U+FE00–U+FE0F; U+E0100–U+E01EF | generally `Mn` | Select a glyph/presentation variant; U+FE0F commonly requests emoji-style presentation | **Not `Cf` or `Cc`**, so a control-character filter need not remove them |
| Private use | U+E000–U+F8FF and supplementary PUA ranges | `Co` | Meaning determined by private agreement/fonts | Character-vocabulary tokenizers may have no entry; byte fallback still represents them |
| Surrogates | U+D800–U+DFFF | `Cs` | UTF-16 encoding machinery, not ordinary scalar characters | Lone surrogates frequently fail before tokenization at UTF-8/API boundaries |
| Unassigned | e.g. U+0378 in current experiments | `Cn` | No assigned character semantics in the relevant Unicode version | May pass through byte tokenizers or be rejected/removed by category filters |

Unicode explicitly describes `Cc` as C0/C1 controls, `Cf` as format controls, `Cs` as surrogate code points, `Co` as private-use, and `Cn` as reserved/unassigned or noncharacters; `Zs` means a space separator of non-zero width, while `Mn` is a nonspacing combining mark. It also specifically records U+200B ZERO WIDTH SPACE as `Cf`, not `Zs`, and lists U+1680 and U+2000–U+200A under the `White_Space` property. citeturn40view2turn40view3

ZWJ and ZWNJ deserve special treatment rather than blanket deletion. Unicode's security guidance notes that some languages require these join controls for correct spelling and shaping; it therefore recommends contextual restrictions rather than universally declaring them invalid. citeturn25search2

Bidi controls are also materially different from ordinary spaces. Unicode identifies characters such as LRE, RLE, LRO, RLO, PDF and the isolate controls as explicit directional formatting controls, and Unicode's security guidance specifically warns about bidi overrides in security-sensitive names and identifiers. citeturn40view2turn38view2

### What NFC, NFD, NFKC, and NFKD actually do

Unicode defines the four normalization forms as follows: NFD is canonical decomposition; NFC is canonical decomposition followed by canonical composition; NFKD is compatibility decomposition; NFKC is compatibility decomposition followed by canonical composition. Compatibility equivalence is deliberately broader than canonical equivalence and can remove distinctions such as width, typographic form, or line-breaking behavior. citeturn38view1

The practical consequences for invisible characters are more limited than many preprocessing descriptions imply:

| Input class | NFC | NFD | NFKC | NFKD | Important consequence |
|---|---|---|---|---|---|
| ZWJ U+200D | preserved | preserved | preserved | preserved | NFKC **is not** “delete ZWJ” |
| ZWNJ U+200C | preserved | preserved | preserved | preserved | Same distinction |
| ZWSP U+200B | preserved | preserved | preserved | preserved | Compatibility normalization does not turn it into normal space |
| SHY U+00AD | preserved | preserved | preserved | preserved | Removal requires an additional policy |
| NBSP U+00A0 | preserved | preserved | → U+0020 | → U+0020 | Line-breaking distinction is compatibility-folded |
| U+2000 EN QUAD | canonically changes to U+2002 | same canonical result | → U+0020 | → U+0020 | One of the less obvious canonical-space cases |
| U+2001 EM QUAD | canonically changes to U+2003 | same canonical result | → U+0020 | → U+0020 | Same |
| U+2002–U+200A spaces | generally preserved canonically | generally preserved | → U+0020 | → U+0020 | Width distinctions disappear |
| U+202F NNBSP | preserved | preserved | → U+0020 | → U+0020 | Nonbreaking property disappears |
| U+205F MMSP | preserved | preserved | → U+0020 | → U+0020 | Mathematical spacing distinction disappears |
| U+3000 ideographic space | preserved | preserved | → U+0020 | → U+0020 | Full-width spacing distinction disappears |
| U+1680 OGHAM SPACE | preserved | preserved | preserved | preserved | Do not assume NFKC means “all Unicode spaces → ASCII space” |
| C0 controls / DEL | preserved as code points | preserved | preserved | preserved | Normalization is not a control sanitizer |
| LRM/RLM/bidi controls | preserved | preserved | preserved | preserved | Explicit removal requires another processing stage |
| VS16 U+FE0F | preserved | preserved | preserved | preserved | Presentation variation survives normalization |
| PUA | preserved | preserved | preserved | preserved | No compatibility meaning is imposed |
| Unassigned code point | normally unchanged | unchanged | unchanged | unchanged | No assigned mapping exists |
| Lone surrogate | API-dependent / ill-formed boundary issue | same | same | same | It should be rejected before relying on normalization semantics |

The NBSP result is explicitly illustrated by Unicode as a compatibility equivalence: the line-breaking distinction between NBSP and SPACE is erased under compatibility normalization. More generally, NFKC/NFKD are intentionally stronger transformations than NFC/NFD and can erase distinctions that are semantically relevant in specialized contexts. citeturn38view1

A local sanity check using Python's Unicode 15.1 tables produced exactly the results in the table above for the representative code points; Unicode normalization stability makes the long-established mappings relevant, but the Unicode 17.0 data remains the normative reference. The experiment also confirmed that plain NFKC did **not** delete ZWJ, ZWNJ, ZWSP, soft hyphen, bidi controls, VS16, NUL, DEL, PUA, or U+0378. Unicode's own documentation distinguishes plain compatibility normalization from application profiles that add extra mappings. citeturn38view0turn38view1

This distinction is security-critical. Unicode's security report gives an historical identifier profile in which ZWJ was additionally mapped to nothing, but it presents that deletion as an **extra mapping beyond NFKC**, not a property of NFKC itself. citeturn25search2


## Preprocessing pipelines and where characters disappear

There is no single “modern LLM preprocessing pipeline.” The most useful abstraction is:

```mermaid
flowchart LR
    A["Bytes / API string"] --> B["Decode & validate Unicode"]
    B --> C{"Unicode normalization?"}
    C -->|"None"| D
    C -->|"NFC / NFD"| D["Canonicalized text"]
    C -->|"NFKC / NFKD"| D
    D --> E{"Additional text cleaning?"}
    E -->|"case/lowercase"| F
    E -->|"control filtering"| F
    E -->|"whitespace mapping/collapse"| F
    E -->|"format-char policy"| F["Normalized model input"]
    F --> G["Pre-tokenizer / boundary detection"]
    G --> H{"Tokenizer representation"}
    H -->|"code points / pieces"| I["BPE / WordPiece / Unigram"]
    H -->|"UTF-8 bytes"| J["byte-to-symbol mapping"]
    J --> I
    I --> K["Token strings + token IDs"]
    K --> L["Special tokens / truncation / padding"]
```

Hugging Face Tokenizers makes these stages explicit: normalizer, pre-tokenizer, model, post-processor and decoder are separable components. The pre-tokenizer is especially important because a split introduced there prevents the model from subsequently merging symbols across that boundary. citeturn23search8turn23search12

### Typical operations and their effects

| Preprocessing step | Usually preserves | Usually changes/removes | Main trap |
|---|---|---|---|
| UTF-8 decode with strict validation | valid Unicode scalar values | invalid byte sequences; lone surrogate representations cannot normally be encoded as strict UTF-8 | Error-handling mode (`strict`, `replace`, `ignore`, etc.) can silently change input before tokenizer code sees it |
| NFC | most invisibles and spaces | canonical decompositions/compositions | Not a sanitizer |
| NFD | same | decomposes canonical composites, e.g. `é` → `e` + combining acute | Can create extra combining-code-point boundaries |
| NFKC | controls, ZWJ/ZWNJ/ZWSP, many format chars | compatibility characters and many Unicode spaces | Not all whitespace is mapped, and format controls do not simply disappear |
| NFKD | as above | compatibility + canonical decomposition | Can substantially increase code-point count before later stripping |
| lowercasing | most invisibles | case distinctions | Lowercase is not equivalent to casefold |
| casefolding | most format/control chars | broader language-independent case distinctions | Still should not be assumed to sanitize invisibles |
| accent stripping after NFD | most `Cf`/`Cc` | combining marks such as `Mn` accents | Also catches VS16 if implemented as indiscriminate `Mn` stripping |
| “replace Unicode whitespace with SPACE” | `Cf` zero-width controls | `White_Space` / `Zs` characters | ZWSP, ZWJ and ZWNJ are not ordinary `Zs` whitespace |
| collapse runs of spaces | non-space format chars | multiplicity of mapped spaces | Loses indentation and spacing information |
| remove category `Cc` | format controls, PUA | C0/C1 controls | Does **not** remove most zero-width/bidi controls because they are `Cf` |
| remove category `C*` | very little unusual material | `Cc`, `Cf`, `Cs`, `Co`, `Cn` | Dangerously broad: deletes legitimate join controls and all PUA |
| remove “default ignorable” characters | ordinary letters/spaces | many formatting/presentation controls | Can destroy join-control or variation-selector semantics |
| tokenizer-specific normalizer | implementation-dependent | implementation-dependent | Must inspect serialized configuration/model |

Unicode's categories explain why a generic regex such as “remove `\p{C}`” is substantially stronger than “remove controls”: `C` is the union of `Cc`, `Cf`, `Cs`, `Co`, and `Cn`. Such a rule would remove not merely NUL and DEL but also join controls, bidi controls, private-use content and currently unassigned code points. citeturn40view2

### BERT cleaning

Hugging Face's current `BertNormalizer` documents `clean_text=True` as removing control characters and replacing all whitespace with the ordinary form; it additionally supports lowercasing and accent stripping and defaults to lowercase. citeturn27view5

That combination explains several otherwise surprising results:

* ZWJ/ZWNJ/ZWSP/SHY/bidi controls are `Cf` and are removed by BERT-style control detection.
* NBSP, OGHAM SPACE, U+2000–U+200A, U+202F, U+205F and U+3000 are spacing characters and become ordinary word-separating spaces.
* TAB/LF/CR are specially interpreted as whitespace rather than simply discarded in the original BERT-style logic.
* Variation selectors such as VS16 are `Mn`, not `Cf`; nevertheless `bert-base-uncased`-style NFD + accent/mark stripping removes them.
* PUA (`Co`), surrogate (`Cs`) and unassigned (`Cn`) values are also caught by an implementation that defines “control” as Unicode category beginning with `C`.

The Hugging Face implementation documents the overall cleaning contract, while the BERT model itself uses WordPiece subwords. citeturn27view5turn23search2

### SentencePiece normalization

SentencePiece explicitly embeds normalization in its model. Current options document:

`normalization_rule_name="nmt_nfkc"` by default; `nfkc`, case-folding variants and `identity` are alternatives; `add_dummy_prefix=true`; `remove_extra_whitespaces=true`; and `escape_whitespaces=true`, which turns normal spaces into the visible meta-symbol `▁`. citeturn27view4

Its current Python documentation demonstrates normalization of `"Hello  World."` to `"▁Hello▁World."`, showing both duplicate-space collapse and the visible whitespace marker. The same documentation emphasizes that the exact normalization behavior can be read from the model or exercised independently with `SentencePieceNormalizer`. citeturn39view0

`nmt_nfkc` should not be equated with bare Unicode `NFKC`. It is an NMT-oriented mapping layer with additional cleaning/whitespace behavior. This becomes obvious experimentally: in the tested SentencePiece 0.2.1 normalizer, ZWNJ and ZWSP became spaces even though plain Unicode NFKC preserves both. SentencePiece's release history also records normalization behavior as an evolving implementation detail; version 0.1.96 specifically stopped treating ZWJ as whitespace, and 0.2.2 updated the normalization rules to current ICU/Unicode data. citeturn27view3

Hugging Face has its own `Nmt` normalizer intended to match the original SentencePiece NMT preprocessing. Its current source documentation says it removes controls, normalizes whitespace and replaces selected Unicode characters; its published example maps `"Hello\x00World"` to `"Hello World"`. This contrasts with the SentencePiece 0.2.1 experiment below, where NUL survived the normalizer and was handled by byte fallback. That is precisely why version and implementation must be recorded when auditing edge cases. citeturn27view6


## Tokenization algorithms and byte-level versus character-level behavior

### BPE

BPE begins from an initial alphabet and repeatedly applies learned pair merges. The NLP adaptation introduced by Sennrich, Haddow and Birch learns frequent subword combinations so rare words can be represented through smaller units. citeturn24search0

For invisible Unicode, the key question is therefore **what the initial symbols are**:

* With character/code-point BPE, an invisible code point must either occur in the initial alphabet/vocabulary, be mapped to an unknown token, or be handled by fallback.
* With byte-level BPE, a valid Unicode character first becomes UTF-8 bytes. Because all 256 byte values have representations, an unseen code point is still expressible.
* A pre-tokenizer can impose boundaries before BPE, preventing merges across an invisible or whitespace symbol even when the BPE vocabulary otherwise contains possible merges. Hugging Face explicitly documents that pre-tokenization boundaries constrain subsequent model merging. citeturn23search8

### WordPiece

WordPiece, as used by BERT, performs greedy longest-match segmentation within each pre-tokenized word. Hugging Face describes it as choosing long vocabulary pieces first, conventionally using `##` to mark non-initial pieces; the BERT paper specifies a WordPiece vocabulary for the model. citeturn23search12turn23search2

This means an invisible code point that **survives** BERT's BasicTokenizer can make an otherwise known word fail to match or break into different subwords. In `bert-base-uncased`, however, many of the troublesome `Cf`/`C*` values disappear before WordPiece ever sees them, so insertion can instead create a normalization collision: visually and bytewise different input strings map to the same token sequence. citeturn27view5

### Unigram and SentencePiece

SentencePiece is a framework, not one segmentation algorithm: it supports both BPE and a Unigram language-model tokenizer and can train directly on raw sentences. citeturn25search3turn25search12

With Unigram, the surviving normalized input is segmented using a probabilistic vocabulary rather than deterministic merge ranks. But Unicode questions are still primarily settled **before** that scoring step: if `nmt_nfkc` turns ZWNJ into a space, no Unigram candidate can recover the original ZWNJ afterward. Conversely, a surviving code point may become its own piece, join an adjacent piece, map to `<unk>`, or fall back to byte pieces depending on the particular model. Current SentencePiece also supports byte fallback, in which an unsupported character is represented by its UTF-8 bytes. citeturn39view0

### GPT-2 byte-level BPE

GPT-2's important property is not merely “BPE” but **byte-level BPE**. The implementation converts UTF-8 bytes to a reversible Unicode display alphabet before applying BPE. Hugging Face's current `ByteLevel` documentation describes this mapping as assigning every byte 0–255 a unique visible character, permitting arbitrary byte values without requiring an unknown token. citeturn0search5turn27view5

As a result, insertion of an invisible character generally:

1. adds one to four UTF-8 bytes;
2. changes the byte-to-Unicode symbol stream;
3. may create a separate regex pre-tokenization span;
4. may be compressed by learned BPE merges into one or several tokens;
5. changes all relevant token IDs without the character needing a dedicated Unicode-code-point vocabulary entry.

Thus `U+200B` does **not** become a token literally named “ZERO WIDTH SPACE.” Its UTF-8 byte sequence `E2 80 8B` becomes GPT-2's byte alphabet; the learned merges happen to combine those bytes into the token string `âĢĭ`, ID `9525`, in the tested GPT-2 vocabulary.

### GPT-NeoX

GPT-NeoX-20B also uses byte-level BPE, but it is a different trained tokenizer. Hugging Face's current GPT-NeoX documentation states that the tokenizer is backed by the Tokenizers library and uses byte-level BPE; it was trained with additional treatment of whitespace. The documentation gives `"Hello world"` → `[15496, 995]` and `" Hello world"` → `[18435, 995]`, demonstrating that the leading-space state affects IDs. citeturn27view1turn27view2

The implication—not an experimentally measured NeoX ID claim—is that a surviving invisible character is byte-representable in GPT-NeoX as well, but its **exact merge pattern and IDs must be read from the NeoX vocabulary/merges rather than copied from GPT-2**. citeturn27view1

### T5 SentencePiece

The current `google-t5/t5-small` repository contains a 792-kB `spiece.model` plus a serialized `tokenizer.json`, confirming that T5's tokenizer is SentencePiece-based and that tokenizer behavior is embodied in model artifacts rather than just an algorithm name. citeturn28view0turn29view1

It would therefore be unsafe to report the synthetic SentencePiece IDs below as T5 IDs. The T5 binary SentencePiece model could not be executed in the research sandbox, so T5-specific invisible-character IDs are deliberately reported as **not measured** rather than inferred from generic SentencePiece behavior.

### Hugging Face Tokenizers and Node/“tokenizers.js”

Hugging Face Tokenizers 0.23.1 is the latest Python release found as of August 16, 2026, uploaded April 27, 2026. It supports configurable BPE, Unigram, WordPiece and other pipeline components rather than imposing one Unicode policy. citeturn35search0turn23search0

There is an official Node.js binding in the same Hugging Face repository; it is a binding over the Rust implementation rather than a separate JavaScript tokenizer algorithm. The repository instructs users to install the package as `tokenizers` and can load a serialized `tokenizer.json`. citeturn38view3

The npm package itself is notably stale: npm lists `tokenizers` 0.13.3, published in 2023, while a 2026 Hugging Face issue records a failure publishing newer Node binary artifacts. Thus “tokenizers.js” should not be assumed to mean the current 0.23.1 Python/Rust release. citeturn34search0turn23search7


## Experimental results and implementation comparison

### Methodology and version scope

Current stable releases identified for this report were Hugging Face Tokenizers **0.23.1**, Transformers **5.15.0** (uploaded August 10, 2026), SentencePiece **0.2.2** (July 12, 2026), and TensorFlow Text **2.20.1** (March 10, 2026). The official Node `tokenizers` npm package remains at 0.13.3. citeturn35search0turn36view1turn34search3turn36view0turn34search0

Not every latest package could be installed in the sandbox. The experiment therefore separates **executed** results from **source-derived/documented** behavior:

* **GPT-2:** algorithm-level execution using OpenAI's published byte-to-Unicode/BPE procedure and the GPT-2 merge/vocabulary ordering; no external Transformers runtime was used. citeturn0search5
* **BERT:** algorithm-level execution of BERT-style cleaning + greedy WordPiece against the official `bert-base-uncased` vocabulary; special `[CLS]`/`[SEP]` tokens were intentionally omitted so the table shows only text-derived pieces. citeturn23search2
* **SentencePiece:** the actual installed `sentencepiece` **0.2.1** Python/C++ implementation was executed. A controlled 277-piece synthetic Unigram model with `nmt_nfkc` and `byte_fallback=True` was trained so every surviving test input could receive an ID. Those IDs are **experiment-local**, not T5 IDs. The latest stable is 0.2.2, whose normalization tables changed. citeturn27view3turn34search3
* **T5, GPT-NeoX and Node bindings:** exact invisible-character IDs were not executable in this sandbox, so no IDs are invented. Primary model/library documentation is used for their architecture and preprocessing behavior. citeturn27view1turn28view0turn38view3

In the tables, notation such as `a<ZWJ>b` means that the actual Unicode character, not the ASCII text `<ZWJ>`, was inserted between `a` and `b`.

### Requested tokenizer comparison

| Tokenizer | Library/version | Preprocessing steps | Normalization applied | Example inputs | Token outputs | Token IDs | Notes |
|---|---|---|---|---|---|---|---|
| OpenAI GPT-2 | OpenAI reference GPT-2 algorithm/model assets; reproduced locally | GPT-2 regex pre-tokenization → UTF-8 → byte-to-Unicode map → BPE | No NFC/NFKC in tokenizer | `a<ZWJ>b`; `a<NBSP>b`; `a<NUL>b`; `a<VS16>b` | `['a','âĢ','į','b']`; `['a','Âł','b']`; `['a','Ā','b']`; `['a','ï¸ı','b']` | `[64,447,235,65]`; `[64,1849,65]`; `[64,188,65]`; `[64,37929,65]` | Invisibles survive as bytes; byte sequences may merge into one or multiple BPE tokens |
| BERT `bert-base-uncased` WordPiece | BERT/HF-compatible preprocessing; official vocab; reproduced locally | clean text → whitespace canonicalization → lowercase → NFD/strip marks → punctuation split → WordPiece | Canonical decomposition during accent strip; no blanket NFKC | `a<ZWJ>b`; `a<NBSP>b`; `a<VS16>b`; `café` | `['ab']`; `['a','b']`; `['ab']`; `['cafe']` | `[11113]`; `[1037,1038]`; `[11113]`; `[7668]` | Many `C*` characters vanish before WordPiece; Unicode spaces create boundaries |
| SentencePiece synthetic Unigram | **Executed 0.2.1**; latest stable 0.2.2 | `nmt_nfkc` → dummy prefix → whitespace escape → Unigram; byte fallback enabled for experiment | `nmt_nfkc` | `a<ZWJ>b`; `a<ZWNJ>b`; `a<NUL>b`; `a<RLO>b` | `['▁a',ZWJ,'b']`; `['▁a','▁','b']`; `['▁a','<0x00>','b']`; `['▁a',RLO,'b']` | `[258,273,257]`; `[258,259,257]`; `[258,1,257]`; `[258,274,257]` | IDs are synthetic-model IDs. Exact 0.2.2 behavior must be retested because Unicode rules were updated |
| Generic SentencePiece | latest 0.2.2 docs | Model-contained normalizer → BPE or Unigram | default training rule `nmt_nfkc`, unless model overrides it | Any invisible | model-dependent | model-dependent | There is no universal “SentencePiece token ID”; normalization and vocab are serialized with each model |
| T5 SentencePiece | T5 model artifact + current Transformers | T5's packaged `spiece.model` | Exact model normalizer must be read from binary model | Invisible suite | Not executed | **Unspecified / not measured** | Do not substitute IDs from a generic SentencePiece model |
| GPT-NeoX-20B | current Transformers implementation/model | byte-level BPE; whitespace-aware vocabulary | No generic Unicode folding documented as part of the model tokenizer | docs: `Hello world`; ` Hello world` | model byte-BPE pieces | `[15496,995]`; `[18435,995]` | Exact invisible IDs not executed; byte-level representation implies valid surviving UTF-8 characters remain encodable |
| Hugging Face Tokenizers | 0.23.1 | Fully configurable normalizer → pre-tokenizer → model → post-processor | Depends on tokenizer JSON | e.g. `ByteLevel` maps LF to visible `Ċ`; `BertNormalizer` cleans controls | configuration-dependent | configuration-dependent | Library is a framework, so a single “HF result” does not exist |
| HF Node binding / npm `tokenizers` | npm 0.13.3; Rust source has advanced beyond npm release | Same serialized Rust tokenizer concept | JSON/config-dependent | same tokenizer JSON should define behavior | config-dependent | config-dependent | No separate “tokenizers.js Unicode algorithm”; Node package release lags Python/Rust |
| TensorFlow Text WordPiece | 2.20.1 current stable | User/config-defined preprocessing + WordPiece | Not inherently one normalization policy | Not run because HF/BERT path was used | Not measured | Not measured | WordPiece segmentation itself should not be confused with the preceding text normalizer |

The implementation descriptions in this table are grounded in the current Hugging Face pipeline/normalizer documentation, SentencePiece's current options, GPT-NeoX's tokenizer documentation, and the T5 model artifacts. citeturn23search8turn27view5turn27view4turn27view1turn28view0

### Character-by-character measured results

The following is the most direct answer to whether characters disappear, cause boundaries, or become tokens. GPT-2 and BERT IDs are from the published model vocabularies; SentencePiece IDs refer only to the controlled synthetic model described above.

| Actual input between `a` and `b` | GPT-2 byte-BPE output | BERT uncased output | SentencePiece 0.2.1 `nmt_nfkc` synthetic output |
|---|---|---|---|
| none: `ab` | `['ab']` → `[397]` | `['ab']` → `[11113]` | model-specific baseline |
| ZWJ U+200D | `['a','âĢ','į','b']` → `[64,447,235,65]` | normalized `ab` → `[11113]` | `['▁a',ZWJ,'b']` → `[258,273,257]` |
| ZWNJ U+200C | `['a','âĢ','Į','b']` → `[64,447,234,65]` | `ab` → `[11113]` | normalized to space: `['▁a','▁','b']` → `[258,259,257]` |
| ZWSP U+200B | `['a','âĢĭ','b']` → `[64,9525,65]` | `ab` → `[11113]` | normalized to space → `[258,259,257]` |
| SHY U+00AD | `['a','ÂŃ','b']` → `[64,3907,65]` | `ab` → `[11113]` | `['▁a',SHY,'b']` → `[258,271,257]` |
| NBSP U+00A0 | `['a','Âł','b']` → `[64,1849,65]` | `a b` → `['a','b']` `[1037,1038]` | normalized to space → `[258,259,257]` |
| OGHAM SPACE U+1680 | `['a','á','ļ','Ģ','b']` → `[64,157,248,222,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| EN QUAD U+2000 | `['a','âĢ','Ģ','b']` → `[64,447,222,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| EM SPACE U+2003 | `['a','âĢ','ĥ','b']` → `[64,447,225,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| THIN SPACE U+2009 | `['a','âĢ','ī','b']` → `[64,447,231,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| NNBSP U+202F | `['a','âĢ','¯','b']` → `[64,447,107,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| MMSP U+205F | `['a','âģ','Ł','b']` → `[64,46256,253,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| IDEOGRAPHIC SPACE U+3000 | `['a','ãĢ','Ģ','b']` → `[64,5099,222,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| NUL U+0000 | `['a','Ā','b']` → `[64,188,65]` | `ab` → `[11113]` | survives normalizer, byte fallback `<0x00>` → `[258,1,257]` |
| TAB U+0009 | `['a','ĉ','b']` → `[64,197,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| LF U+000A | `['a','Ċ','b']` → `[64,198,65]` | `a b` → `[1037,1038]` | normalized to space → `[258,259,257]` |
| DEL U+007F | `['a','ġ','b']` → `[64,221,65]` | `ab` → `[11113]` | removed: effectively `ab` |
| LRM U+200E | `['a','âĢİ','b']` → `[64,48261,65]` | `ab` → `[11113]` | normalized to space → `[258,259,257]` |
| RLM U+200F | `['a','âĢ','ı','b']` → `[64,447,237,65]` | `ab` → `[11113]` | normalized to space → `[258,259,257]` |
| LRE U+202A | `['a','âĢ','ª','b']` → `[64,447,103,65]` | `ab` → `[11113]` | survives; absent from learned pieces here, so UTF-8 byte fallback |
| RLO U+202E | `['a','âĢ','®','b']` → `[64,447,106,65]` | `ab` → `[11113]` | `['▁a',RLO,'b']` → `[258,274,257]` |
| LRI U+2066 | `['a','âģ','¦','b']` → `[64,46256,99,65]` | `ab` → `[11113]` | survives; falls to UTF-8 byte pieces |
| VS16 U+FE0F | `['a','ï¸ı','b']` → `[64,37929,65]` | mark stripping yields `ab` → `[11113]` | `['▁a',VS16,'b']` → `[258,276,257]` |
| PUA U+E000 | `['a','îĢ','Ģ','b']` → `[64,29773,222,65]` | `ab` → `[11113]` | `['▁a',PUA,'b']` → `[258,275,257]` |
| unassigned U+0378 | `['a','Í','¸','b']` → `[64,137,116,65]` | `ab` → `[11113]` | `['▁a',U+0378,'b']` → `[258,272,257]` |

The table exposes four qualitatively different outcomes:

**Separate token(s).** GPT-2 NUL, TAB, DEL and most format characters create byte-derived token pieces of their own.

**Merged byte token.** GPT-2's ZWSP bytes happen to merge into a single learned byte-BPE piece, ID `9525`; VS16 similarly becomes one merged byte sequence, ID `37929`.

**Boundary creation.** BERT converts Unicode spaces to U+0020 and therefore turns `ab` into two words. SentencePiece `nmt_nfkc` does the same for several additional zero-width formatting characters in the tested version.

**Disappearance / collision.** BERT maps many distinct injected values to exactly `ab`; thus the tokenizer cannot distinguish the original clean string from one containing ZWJ, ZWSP, SHY, bidi controls, PUA, etc. This is a many-to-one preprocessing collision, not a WordPiece property. The documented `BertNormalizer` cleaning behavior explains why. citeturn27view5

### Canonically equivalent text can tokenize differently without normalization

The GPT-2 experiment also demonstrates why Unicode normalization can matter even when no “invisible attack” is intended:

| Text | GPT-2 token strings | IDs |
|---|---|---|
| precomposed `café` | `['c','af','é']` | `[66,1878,2634]` |
| decomposed `cafe\u0301` | `['c','afe','Ì','ģ']` | `[66,8635,136,223]` |
| BERT `café` | `['cafe']` | `[7668]` |
| BERT `cafe\u0301` | `['cafe']` | `[7668]` |

Unicode considers the two spellings canonically equivalent; NFC would make them identical. GPT-2 performs no such Unicode canonical normalization, so byte-level preservation leads to different token sequences. BERT's uncased preprocessing decomposes and strips the accent, making both even more aggressively equivalent as `cafe`. Unicode defines the canonical equivalence underlying the first pair. citeturn38view1turn27view5

### Lone surrogates are a separate problem

For diagnostic purposes only, the GPT-2 reproduction was also fed Python's internal string `a\ud800b` using the non-standard `surrogatepass` UTF-8 error handler. It yielded byte-symbol pieces `['a','í','ł','Ģ','b']`, IDs `[64,169,254,222,65]`.

That result **must not be interpreted as normal GPT-2 support for U+D800**. U+D800–U+DFFF are surrogate code points (`Cs`) used by UTF-16; a strict UTF-8 serialization/API boundary will ordinarily reject a lone surrogate before BPE runs. Unicode 17.0 explicitly classifies this range as surrogate code points. citeturn40view3


## Reproducing and extending the experiments

The safest way to reproduce tokenizer behavior is to print all three layers: the input in escaped form, the tokenizer's tokens, and the integer IDs. Do not rely on terminal rendering, because the entire point of many test characters is that they render invisibly.

### Unicode inspection and normalization

```python
import unicodedata as ud

TESTS = {
    "ZWJ": "\u200D",
    "ZWNJ": "\u200C",
    "ZWSP": "\u200B",
    "SOFT_HYPHEN": "\u00AD",
    "NBSP": "\u00A0",
    "OGHAM_SPACE": "\u1680",
    "EM_SPACE": "\u2003",
    "NARROW_NBSP": "\u202F",
    "IDEOGRAPHIC_SPACE": "\u3000",
    "NUL": "\u0000",
    "TAB": "\u0009",
    "DEL": "\u007F",
    "LRM": "\u200E",
    "RLM": "\u200F",
    "RLO": "\u202E",
    "LRI": "\u2066",
    "VS16": "\uFE0F",
    "PRIVATE_USE": "\uE000",
    "UNASSIGNED_EXAMPLE": "\u0378",
}

def esc(s: str) -> str:
    return s.encode("unicode_escape").decode("ascii")

for name, ch in TESTS.items():
    print(
        name,
        f"U+{ord(ch):04X}",
        ud.category(ch),
        ud.name(ch, "<no Unicode name>")
    )
    for form in ("NFC", "NFD", "NFKC", "NFKD"):
        print(" ", form, esc(ud.normalize(form, ch)))
```

Python's Unicode tables depend on the Python build, so for audit-quality testing record `unicodedata.unidata_version` alongside the result. The normative definitions of the four forms come from UAX #15, while the current UCD for this report is Unicode 17.0.0. citeturn38view0turn38view1

### Hugging Face / GPT-2 / BERT / NeoX / T5 harness

With current Transformers, a compact test harness is:

```python
from importlib.metadata import version
from transformers import AutoTokenizer

CASES = {
    "plain": "ab",
    "ZWJ": "a\u200Db",
    "ZWNJ": "a\u200Cb",
    "ZWSP": "a\u200Bb",
    "SHY": "a\u00ADb",
    "NBSP": "a\u00A0b",
    "EM_SPACE": "a\u2003b",
    "NUL": "a\u0000b",
    "DEL": "a\u007Fb",
    "LRM": "a\u200Eb",
    "RLO": "a\u202Eb",
    "VS16": "a\uFE0Fb",
    "PUA": "a\uE000b",
    "UNASSIGNED": "a\u0378b",
}

MODELS = {
    "gpt2": "openai-community/gpt2",
    "bert": "google-bert/bert-base-uncased",
    "t5": "google-t5/t5-small",
    "gpt-neox": "EleutherAI/gpt-neox-20b",
}

print("transformers =", version("transformers"))

for label, model_id in MODELS.items():
    print(f"\n### {label}: {model_id}")
    tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)

    # Fast tokenizers expose their serialized normalizer/pre-tokenizer.
    backend = getattr(tok, "backend_tokenizer", None)
    if backend is not None:
        print("normalizer:", backend.normalizer)
        print("pre_tokenizer:", backend.pre_tokenizer)

    for name, text in CASES.items():
        encoded = tok(text, add_special_tokens=False)
        ids = encoded["input_ids"]
        pieces = tok.convert_ids_to_tokens(ids)

        print(
            name,
            repr(text),
            "pieces=", [repr(x) for x in pieces],
            "ids=", ids,
        )
```

The current Transformers package on PyPI was 5.15.0 on August 16, 2026; Hugging Face Tokenizers was 0.23.1. Pinning versions is essential because tokenizer serialization and normalization implementations are part of the behavior under test. citeturn36view1turn35search0

For BERT, run both `use_fast=True` and `use_fast=False` when compatibility matters. Any difference is itself valuable evidence that the Python and Rust preprocessing paths are not perfectly aligned.

### Inspecting a Hugging Face tokenizer directly

Hugging Face Tokenizers exposes individual normalizers:

```python
from tokenizers.normalizers import (
    NFC, NFD, NFKC, NFKD,
    BertNormalizer, ByteLevel
)

samples = [
    "a\u200Db",       # ZWJ
    "a\u200Bb",       # ZWSP
    "a\u00A0b",       # NBSP
    "a\u0000b",       # NUL
    "a\uFE0Fb",       # VS16
]

normalizers = {
    "NFC": NFC(),
    "NFD": NFD(),
    "NFKC": NFKC(),
    "NFKD": NFKD(),
    "BERT": BertNormalizer(
        clean_text=True,
        lowercase=True
    ),
    "BYTELEVEL": ByteLevel(),
}

for name, norm in normalizers.items():
    print("\n", name)
    for s in samples:
        print(repr(s), "=>", repr(norm.normalize_str(s)))
```

The current Hugging Face documentation explicitly shows its `ByteLevel` normalizer turning newline into the visible byte symbol `Ċ`, and documents the BERT normalizer's control cleaning, whitespace normalization, lowercasing and accent handling. citeturn27view5

### SentencePiece normalization and token IDs

For an existing model such as T5, the most informative operation is to ask the model itself how it normalizes:

```python
import sentencepiece as spm

sp = spm.SentencePieceProcessor(
    model_file="spiece.model"
)

for name, text in CASES.items():
    print(
        name,
        "input=", repr(text),
        "normalized=", repr(sp.normalize(text)),
        "pieces=", sp.encode(text, out_type=str),
        "ids=", sp.encode(text, out_type=int),
    )
```

For separating Unicode normalization from model segmentation:

```python
import sentencepiece as spm

for rule in ("identity", "nfkc", "nmt_nfkc"):
    normalizer = spm.SentencePieceNormalizer(
        rule_name=rule,
        add_dummy_prefix=False,
        escape_whitespaces=False,
        remove_extra_whitespaces=False,
    )

    print("\nRULE:", rule)
    for name, text in CASES.items():
        print(name, repr(normalizer.normalize(text)))
```

Current SentencePiece documentation exposes `SentencePieceNormalizer` precisely for this purpose, and current training options identify `nmt_nfkc` as the default normalization rule unless overridden. citeturn39view0turn27view4

For a production audit, also serialize the package versions:

```python
from importlib.metadata import version
import unicodedata

for package in ("transformers", "tokenizers", "sentencepiece"):
    try:
        print(package, version(package))
    except Exception:
        print(package, "not installed")

print("Python Unicode DB:", unicodedata.unidata_version)
```

That metadata is not bureaucratic detail: SentencePiece 0.2.2 explicitly updated Unicode normalization rules relative to previous versions, while the npm Hugging Face Node binding is still on a substantially older release. citeturn27view3turn34search0


## Practical recommendations and conclusions

### Preserve raw input separately from model input

For systems that care about provenance, security, moderation, code analysis, document integrity or forensic reproducibility, store or hash the original byte sequence before performing Unicode normalization. Once a normalizer has collapsed NBSP into SPACE, removed a bidi control or transformed ZWNJ into a boundary, the tokenizer output cannot reconstruct what was originally supplied.

This is particularly important for BERT-like cleaners, where many distinct strings can intentionally collapse onto the same token sequence, and for SentencePiece models with aggressive NMT normalization. Hugging Face and SentencePiece both treat normalization as an explicit preprocessing stage, so applications should log that stage rather than treating tokens as a lossless representation of source text. citeturn23search8turn39view0

### Use NFC as the conservative default when semantic preservation matters

For ordinary natural-language model input, **NFC** is generally the safer baseline when the goal is to canonicalize equivalent spellings without removing compatibility distinctions such as NBSP versus SPACE or full-width versus ordinary forms. NFC resolves the `café` versus `cafe + combining acute` canonical-equivalence problem while avoiding much of NFKC's deliberately stronger folding. Unicode itself distinguishes canonical from compatibility equivalence for exactly this reason. citeturn38view1

NFKC can be appropriate for search, matching, indexing, deduplication or deliberately compatibility-insensitive NLP, but applications should regard it as a semantic policy decision, not harmless cleanup. It changes NBSP and many other compatibility spacing characters to normal spaces and similarly folds many typographic/width distinctions. citeturn38view1

### Define “whitespace” explicitly

Do not write a policy saying merely “collapse whitespace.” Specify whether it means:

`U+0020 only`; Unicode `White_Space`; general category `Zs`; language/runtime `isspace()`; regex `\s`; or an application-defined collection that also contains zero-width formatting characters.

Those sets are not identical. Unicode specifically classifies U+200B ZERO WIDTH SPACE as a `Cf` format control, while characters such as U+1680 and U+2000–U+200A are Unicode whitespace/space separators. citeturn40view2

For source code, indentation-sensitive formats, tables, or prompt structures, collapsing whitespace may also destroy meaningful structure independently of Unicode security concerns.

### Do not delete all `Cf` or all `C*` characters by default

A blanket `category.startswith("C")` deletion policy removes:

`Cc` controls **plus** `Cf` formatting characters **plus** `Cs` surrogates **plus** `Co` private-use characters **plus** `Cn` unassigned/noncharacter values. Unicode defines the categories this way. citeturn40view2

Such a rule can therefore destroy legitimate ZWJ/ZWNJ orthography, emoji sequences, private-use notation and intentional directional metadata. Unicode specifically notes legitimate language use of ZWJ and ZWNJ. citeturn25search2

A better policy is contextual:

**reject malformed encoding/surrogates at the boundary; control C0/C1 characters according to protocol needs; treat bidi overrides as high-risk; retain or validate join controls according to language context; preserve variation selectors when grapheme/presentation identity matters; and make PUA handling an explicit domain policy.**

### Distinguish validation from normalization

For security-sensitive fields, there are three separate operations:

**mapping**, where an input becomes another string;

**prohibition**, where an input is rejected;

**normalization**, where equivalent Unicode representations are canonicalized.

They should not be silently collapsed into one regex. Unicode's security guidance specifically distinguishes mapping from prohibition and recommends applying mappings before prohibition where both are used. citeturn25search2

For example, deleting U+202E RLO and accepting the rest is operationally different from rejecting any field that contains bidi overrides. The latter retains evidence that the suspicious character was present and prevents two distinct inputs from silently collapsing to one accepted identifier.

### Audit before and after tokenization

For hostile-input or robustness testing, log at least:

```text
raw byte length
Unicode scalar/code-point sequence
Unicode categories/properties
normalized string
pre-tokenized spans
token strings
token IDs
decoded token output
```

This distinguishes five failure modes that otherwise look identical in the final token sequence: decoder replacement, Unicode normalization, application cleaning, pre-tokenization boundaries, and subword-vocabulary behavior.

### Test the exact model artifact, not “the tokenizer family”

The experimental evidence makes this the strongest operational conclusion:

**“BPE” is not a Unicode policy.  
“SentencePiece” is not a Unicode policy.  
“WordPiece” is not a Unicode policy.  
“byte-level” is not a complete preprocessing specification.**

Hugging Face Tokenizers deliberately allows normalizer and pre-tokenizer configuration independently of the BPE/WordPiece/Unigram model, while SentencePiece embeds normalization rules in each model. GPT-NeoX uses byte-level BPE like GPT-2 but a different whitespace-rich vocabulary; T5 packages a specific SentencePiece model. citeturn23search8turn27view4turn27view1turn28view0

For regression testing, keep a fixed corpus containing at least:

```text
ZWJ, ZWNJ, ZWSP, SHY
NBSP, OGHAM SPACE
U+2000–U+200A
U+202F, U+205F, U+3000
NUL, TAB, LF, CR, DEL
LRM, RLM
U+202A–U+202E
U+2066–U+2069
VS15/VS16
a representative supplementary variation selector
BMP and supplementary PUA
a currently unassigned scalar value
precomposed/decomposed canonical pairs
ordinary, doubled, leading and trailing ASCII spaces
```

Record normalized text, token strings and IDs in version control. A tokenizer-library upgrade should be treated as a behavior-changing dependency upgrade whenever any of those outputs change. SentencePiece 0.2.2's explicit normalization-table update demonstrates that this is not merely theoretical. citeturn27view3

### Bottom line

The central dividing line is not simply “invisible versus visible” but **information-preserving versus information-collapsing preprocessing**.

GPT-2-style byte-level BPE is strongly preservation-oriented after it receives a valid Unicode string: invisible code points generally become byte-derived tokens, often increasing token count and altering boundaries but remaining distinguishable.

BERT-style preprocessing is strongly collapsing: many format/control/unassigned/private characters disappear before WordPiece, while Unicode spaces become word boundaries. This can be desirable for robust lexical modeling but creates large equivalence classes of distinct source strings.

SentencePiece is configurable and model-specific: `nmt_nfkc` can transform substantially more than plain NFKC, while `identity` can preserve text much more literally. Its whitespace marker `▁`, model-contained normalization rules and optional byte fallback make it especially important to inspect the actual `.model` file rather than infer behavior from the word “SentencePiece.” citeturn27view4turn39view0

T5 and GPT-NeoX inherit these general principles but have model-specific vocabularies and preprocessing artifacts; exact invisible-character IDs should therefore be measured against their actual released tokenizer files rather than extrapolated from T5's generic SentencePiece family or GPT-NeoX's generic byte-BPE label. citeturn27view1turn28view0

Finally, **normalization should be chosen for semantics, filtering should be chosen for policy, and tokenization should be treated as a downstream representation step**. Treating NFKC, whitespace collapse, control deletion and subword segmentation as interchangeable “cleanup” is the root cause of most surprising invisible-Unicode tokenizer behavior. Unicode's standards and the implementation experiments above consistently point to the same engineering discipline: preserve the original, make every transformation explicit, test the exact tokenizer artifact, and attach versioned Unicode/tokenizer semantics to every security- or reproducibility-critical pipeline. citeturn38view0turn38view1turn25search2