MachineTradecraft
Global site search

Search guides, labs, glossary, and research

Type two or more characters to search.

Start with a channel, artifact, or defense term

Examples include zero-width, metadata, tokenizer, or prompt injection.
    Model Signals · Boundary mismatch

    Educational Tokenization Differential

    The lab illustrates why tokenization differs while explicitly refusing to claim compatibility with a proprietary model tokenizer.

    Browse all labs Open universal Machine View
    Quick answer

    What does the Tokenization Differential show?

    The lab illustrates why tokenization differs while explicitly refusing to claim compatibility with a proprietary model tokenizer.

    Human visibility
    Subword and byte boundaries
    Machine receiver
    Educational local tokenizer profiles
    Robustness
    Vocabulary-specific

    Research boundary: this page uses bounded, inert data and fixed safe examples. It never executes decoded content, requests secrets, calls third-party services, or performs actions against external systems.

    FIRST RUN / THREE STEPS

    Start with the prepared, bounded workflow.

    Nothing runs automatically
    1. Review the prepared starter input

      A bounded benign input is already present. Change it only when you are ready to test a different authorized artifact.

    2. Run Compare tokenizer profiles

      Run the normal first-pass analysis for the prepared values above.

    3. Scan before expanding

      Read the summary first, then scan findings and expand only the machine views you need.

    INPUT / CONTROL PLANE

    Prepare the input and choose one action.

    Ready

    The recommended first run is separated from alternate analyses. Inputs and selected files stay on this host.

    Current input state Unicode and identifier mix loaded

    These bounded starter values are ready to inspect. Review them before running the recommended action.

    61 / 4,096 bytes

    Enter or paste one bounded, authorized artifact. The original value remains in the form so you can revise and rerun it.

    Maximum: 4,096 UTF-8 bytes.

    Enter the bounded numeric value used for this run.

    Allowed range: 4 to 128 in steps of 1.

    Enter the bounded numeric value used for this run.

    Allowed range: 0 to 32 in steps of 1.

    Switch prepared example6 options

    Loading a sample changes only the form values. Review the result and run an action yourself.

    Prepared benign examples
    ACTION HIERARCHY

    Run the recommended first pass.

    Alternate actions remain available below, but the first pass is the clearest place to start.

    Inputs remain on this host. Text operations are size-limited; uploaded files are processed from PHP’s temporary upload and are not retained by the application.

    OUTPUT / MACHINE VIEWS

    Scan the result from summary to evidence.

    Run Compare tokenizer profiles to create the first result.

    The prepared starter input is ready. The output will lead with a summary and visible qualifications before the expandable machine views.

    SummaryFindingsMachine views
    Interpretation framework

    The same artifact can produce several valid observations.

    01

    Human view

    What a person naturally reads, sees, or hears.

    02

    Structural view

    What a parser, DOM, container reader, or metadata extractor exposes.

    03

    Decoder view

    What becomes meaningful only with a rule, key, tokenizer, model, or tool.

    04

    Defensive view

    What normalization, rendering, OCR, canonicalization, or policy changes.

    Evidence and decision boundary

    Use the result as bounded evidence, not as a universal verdict.

    The lab illustrates why tokenization differs while explicitly refusing to claim compatibility with a proprietary model tokenizer.

    LOCAL MODEDeterministic educational token-boundary simulation
    REVIEW DATE2026-08-26
    SOURCE BODYPreserved separately from implementation claims
    Computed locally

    Deterministic output produced by this bounded runtime.

    • Exact input bytes, code-point units, simple whitespace/punctuation pretokens, chunk limits, and overlap
    • Deterministic synthetic pieces and IDs generated from a bundled educational rule set
    • Exact local chunking output for the synthetic sequence
    Bounded approximation

    Useful subset or model that does not establish full conformance.

    • The BPE-, WordPiece-, and Unigram-like labels explain concepts but do not reproduce full training or inference algorithms
    • Synthetic token IDs have no relationship to a vendor vocabulary
    • Normalization and byte-fallback examples are illustrative unless the exact serialized tokenizer is supplied
    Escalate for

    Claims that require an exact parser, trust system, model, or human review.

    • Exact boundaries require the model's tokenizer files, normalizer, pre-tokenizer regex, vocabulary, merge or score tables, special-token map, chat template, and immutable version
    • Security or moderation conclusions require evaluation against the actual model and surrounding application pipeline
    • Tokenizer drift must be tested across deployment artifacts, not inferred from names
    Interpretation rule

    Record the receiver and transformation.

    Machine-decodable is receiver-relative. Record the parser, preprocessing, codebook, tokenizer, key, model, and transformation path before generalizing from one result.

    Limitations

    What this page does not prove

    The bundled vocabularies and scores are intentionally tiny educational fixtures. Exact production behavior requires the serialized tokenizer, model, template, and version.

    Deterministic review material

    Download the exact benign fixtures used for the evidence boundary.

    These local files are supplied for repeatable inspection. The application does not fetch them automatically, execute their content, or treat a fixture result as external verification.

    Tokenization boundary text

    Exercises Unicode normalization differences, a ZWJ emoji, a mixed-script identifier, casing, underscores, punctuation, and indentation in the educational tokenizer simulations.

    Type
    Text fixture
    Bytes
    112
    SHA-256
    ee7bc947159c7f9ce493…
    Download fixture

    Download the complete evidence-review fixture pack · SHA-256 sidecar included beside the archive

    Continue the investigation

    Related labs and source reports

    Tokenizer Explorer Exact tokenizer and normalizer Probability Signals Distribution-aware statistic Watermark Simulator Key-aware statistical detector Tokenization, Normalization, and Boundary Differentials in AI Pipelines Research report Invisible Unicode and Modern Language-Model Tokenizers Research report The Interpretation of Encoded Data by Large Language Models Research report
    Continue exploring

    Move from the current explanation to a related guide, evidence source, or controlled workbench.

    Laboratory catalogChoose a focused local experimentBrowse 38 bounded laboratories spanning representation, files, messages, models, provenance, agents, supply chains, and defense.Open this path Channel atlasCompare machine-readable channel familiesMatch each carrier to its receiver, failure modes, survival properties, and defensive comparison.Open this path Defensive architectureBuild a safer model-ingestion boundaryPreserve evidence, compare representations, canonicalize allowlisted fields, retain provenance, and minimize privilege.Open this path