MachineTradecraft
Global site search

Search guides, labs, glossary, and research

Type two or more characters to search.

Start with a channel, artifact, or defense term

Examples include zero-width, metadata, tokenizer, or prompt injection.
    Model Signals · Tokenizer-dependent signal

    Tokenizer-Sensitive Representation Explorer

    The same surface string can produce different boundaries after normalization and pre-tokenization. The page deliberately avoids claiming compatibility with proprietary token IDs.

    Browse all labs Open universal Machine View
    Quick answer

    What does the Tokenizer Explorer show?

    The same surface string can produce different boundaries after normalization and pre-tokenization. The page deliberately avoids claiming compatibility with proprietary token IDs.

    Human visibility
    Rendered text looks unchanged
    Machine receiver
    Exact tokenizer and normalizer
    Robustness
    Highly version-specific

    Research boundary: this page uses bounded, inert data and fixed safe examples. It never executes decoded content, requests secrets, calls third-party services, or performs actions against external systems.

    FIRST RUN / THREE STEPS

    Start with the prepared, bounded workflow.

    Nothing runs automatically
    1. Review the prepared starter input

      A bounded benign input is already present. Change it only when you are ready to test a different authorized artifact.

    2. Run Compare token views

      Run the normal first-pass analysis for the prepared values above.

    3. Scan before expanding

      Read the summary first, then scan findings and expand only the machine views you need.

    INPUT / CONTROL PLANE

    Prepare the input and choose one action.

    Ready

    The recommended first run is separated from alternate analyses. Inputs and selected files stay on this host.

    Current input state Prepared starter input loaded

    The form is prefilled with a bounded safe starting point. Nothing runs until you choose an action.

    41 / 4,096 bytes

    Enter or paste one bounded, authorized artifact. The original value remains in the form so you can revise and rerun it.

    Maximum: 4,096 UTF-8 bytes.

    Choose the local educational profile used for this run. The selection changes only this local analysis.

    Switch prepared example3 options

    Loading a sample changes only the form values. Review the result and run an action yourself.

    Prepared benign examples
    ACTION HIERARCHY

    Run the recommended first pass.

    Alternate actions remain available below, but the first pass is the clearest place to start.

    Inputs remain on this host. Text operations are size-limited; uploaded files are processed from PHP’s temporary upload and are not retained by the application.

    OUTPUT / MACHINE VIEWS

    Scan the result from summary to evidence.

    Run Compare token views to create the first result.

    The prepared starter input is ready. The output will lead with a summary and visible qualifications before the expandable machine views.

    SummaryFindingsMachine views
    Interpretation framework

    The same artifact can produce several valid observations.

    01

    Human view

    What a person naturally reads, sees, or hears.

    02

    Structural view

    What a parser, DOM, container reader, or metadata extractor exposes.

    03

    Decoder view

    What becomes meaningful only with a rule, key, tokenizer, model, or tool.

    04

    Defensive view

    What normalization, rendering, OCR, canonicalization, or policy changes.

    Evidence and decision boundary

    Use the result as bounded evidence, not as a universal verdict.

    The same surface string can produce different boundaries after normalization and pre-tokenization. The page deliberately avoids claiming compatibility with proprietary token IDs.

    Interpretation rule

    Record the receiver and transformation.

    Machine-decodable is receiver-relative. Record the parser, preprocessing, codebook, tokenizer, key, model, and transformation path before generalizing from one result.

    Limitations

    What this page does not prove

    These are transparent educational approximations, not GPT, Claude, Gemini, or Llama tokenizers. Exact security analysis must use the serialized tokenizer and preprocessing version actually deployed.

    Continue the investigation

    Related labs and source reports

    Probability Signals Distribution-aware statistic Watermark Simulator Key-aware statistical detector Pattern Detector Statistical analyzer Invisible Unicode and Modern Language-Model Tokenizers Research report Hidden Structure in Text: Human-Subtle, Model- and Decoder-Detectable Patterns Research report
    Continue exploring

    Move from the current explanation to a related guide, evidence source, or controlled workbench.

    Laboratory catalogChoose a focused local experimentBrowse 40 bounded laboratories spanning representation, files, messages, models, provenance, agents, supply chains, and defense.Open this path Channel atlasCompare machine-readable channel familiesMatch each carrier to its receiver, failure modes, survival properties, and defensive comparison.Open this path Defensive architectureBuild a safer model-ingestion boundaryPreserve evidence, compare representations, canonicalize allowlisted fields, retain provenance, and minimize privilege.Open this path