Global site search

Search guides, labs, glossary, and research

Type two or more characters to search.

Start with a channel, artifact, or defense term

Examples include zero-width, metadata, tokenizer, or prompt injection.

    REPRESENTATION INTEGRITY

    Unicode Canonicalization and Representation Security

    Trace text from UTF-8 bytes through code points, normalization, grapheme boundaries, bidirectional layout, confusable analysis, and tokenizer-facing forms.

    Quick answer

    What does this Machine Tradecraft expansion explain?

    Unicode security is the discipline of preserving and comparing the byte, code-point, grapheme, identifier, logical-order, visual-order, and tokenizer representations of text before making identity, authorization, moderation, or execution decisions.

    Core differential
    Visually identical text can have different bytes and code points, while different code points can render as nearly identical glyphs.
    Safe default
    Preserve raw UTF-8, validate strictly, generate normalized copies separately, and apply identifier policy only in identifier contexts.
    Important limit
    NFKC, confusable mapping, or control stripping can destroy legitimate language, mathematical notation, emoji, and accessibility semantics.
    Reviewed implementation boundary

    Know what is measured, approximated, and still external.

    The submitted report remains byte-identical. This separate review, checked 2026-08-26, narrows implementation claims and gives visitors a decision path before they generalize from a local result.

    IMPLEMENTATION MODE Bounded educational Unicode differential
    VISIBLE SOURCE PROFILE 5 standards/specifications · 1 research · 0 government · 0 implementation
    SOURCE BODY Preserved; corrections live in this review layer
    Directly computed

    Output produced deterministically by the local runtime.

    • Exact submitted UTF-8 bytes, byte count, SHA-256, and local code-point rows for the bounded input
    • Presence of the explicitly allowlisted bidi controls and selected default-ignorable candidates
    • Separate diagnostic outputs that never replace the original input
    Bounded approximation

    Useful model or subset that must not be mistaken for full conformance.

    • Grapheme counting is an educational approximation rather than complete UAX #29 conformance
    • NFC, NFD, NFKC, and NFKD use a transparent bounded map because the public runtime has no intl dependency
    • Script and confusable-skeleton results cover a tested subset rather than the complete Unicode 17.0 data files
    Requires external verification

    Conclusion that needs an exact implementation, trust system, model, parser, or human review.

    • Full conformance requires a version-pinned Unicode implementation and the complete UCD, UAX #15, UAX #29, and UTS #39 data
    • Identifier acceptance still requires language, locale, product, and account-risk policy
    • Renderer and tokenizer behavior must be tested with the exact production versions
    Decision support

    Choose the next evidence step instead of treating one result as a verdict.

    QuestionWhat the local page can answerWhat it does not establishNext evidence step
    Are two inputs byte-identical?Yes — use the exact byte and SHA-256 views.Whether they are canonically, visually, or semantically equivalent.Compare normalized copies and apply context-specific policy.
    Does the input contain selected bidi or invisible controls?Yes — for the explicitly allowlisted code points.A complete Unicode security verdict.Use a full Unicode 17.0 property implementation and language-aware review.
    Is an identifier safe to register?No — this page provides evidence only.Confusable, mixed-script, locale, or account-policy acceptance.Run a version-pinned UTS #39 profile and high-value-name collision policy.
    Focused deterministic fixture

    Canonical, mixed-script, bidi, and emoji boundary set

    Demonstrates why byte, normalization, identifier, and grapheme views must remain separate.

    Expected boundary: The original input remains unchanged while bounded diagnostics expose differences and limitations.

    Open prepared laboratory
    Version and authority checkpoints
    • Unicode 17.0 finalCurrent final version used for review; Unicode 18.0 remains pre-release
    • UAX #15Normative normalization algorithm; implementation must pin a Unicode version
    • UTS #39Normative security mechanisms and data; full tables are not bundled in this runtime
    Compare independent views

    One artifact, several machine-readable representations

    No single view is automatically authoritative. Preserve the source, identify each parser or receiver, and compare their outputs before authorizing a consequential decision.

    01Bytes

    Exact UTF-8 octets retained as evidence.

    02Code points

    Logical Unicode scalar sequence and selected properties.

    03Approximate grapheme view

    A bounded educational cluster count; full UAX #29 conformance requires current Unicode data.

    04Normalization copies

    NFC, NFD, NFKC, and NFKD are compared without replacing the original.

    05Identifier view

    Scripts, default ignorables, bidi controls, and confusable skeleton candidates.

    06Visual order

    Renderer-dependent layout remains distinct from logical order.

    Bounded method

    Analysis workflow

    The workflow preserves evidence before transformation and keeps structural inspection separate from execution, remote verification, or model behavior.

    1. Validate UTF-8 and preserve the raw bytes.
    2. Enumerate code points, scripts, controls, and grapheme boundaries.
    3. Generate normalization copies and compare exact differences.
    4. Apply identifier-specific confusable and mixed-script policy only where appropriate.
    5. Log the chosen representation and keep the original for review.
    Defense in depth

    Controls carried into implementation

    These controls are contextual. They reduce a defined risk; they do not guarantee safety, truth, attribution, or resistance to every adaptive attack.

    01

    Use NFC consistently for selected storage/comparison contexts, not as an indiscriminate rewrite.

    02

    Wrap untrusted directional text in isolates and visibly flag override controls.

    03

    Use current UTS #39 data for high-value identifiers and review mixed scripts contextually.

    04

    Truncate at grapheme boundaries and validate after concatenation.

    05

    Test the exact tokenizer and parser versions used by the application.

    Shared vocabulary

    Key terms

    Definitions are linked into the site-wide glossary and back to the full report.

    Canonical equivalence

    Distinct Unicode sequences that represent the same abstract character under canonical normalization.

    Grapheme cluster

    A user-perceived character that may contain multiple Unicode code points.

    Confusable skeleton

    A standardized prototype mapping used to compare visually confusable identifiers.

    Default ignorable

    A code point normally omitted from rendering when unsupported, while remaining present in the logical text.

    Logical order

    The stored code-point order consumed by parsers, compilers, and tokenizers.

    Visual order

    The glyph order produced after bidirectional resolution and shaping.

    Mixed-script identifier

    An identifier combining characters from multiple scripts, requiring contextual policy review.

    Continue with primary material

    External standards and research

    These links are provided for visitors who want the governing specification, paper, framework, or implementation documentation. Links open in a new tab; the site does not fetch them during runtime analysis.

    Primary research Conference paper

    Trojan Source

    Peer-reviewed study of bidi-based source-code display differentials.

    Original research or formal conference publication.
    www.usenix.org
    Continue the investigation

    Read the evidence, then test the bounded model

    The full submitted report is preserved byte-for-byte in the governed research library and in durable repository documentation. The laboratory turns selected concepts into deterministic local output without external calls or hidden persistence.

    Detailed report

    Unicode Canonicalization, Confusables, Bidirectional Controls, and Machine-View Security

    A deep treatment of Unicode bytes, scalar values, normalization, grapheme segmentation, default ignorables, bidirectional controls, confusable skeletons, identifiers, tokenization, and internationalization-safe defenses.

    Read governed report
    Focused laboratory

    Unicode Canonicalization Differential

    Compare raw UTF-8, code points, an explicitly approximate grapheme view, four bounded normalization copies, scripts, bidi controls, selected default ignorables, and an educational confusable skeleton without overwriting the original.

    Open bounded laboratory