Semantic-Category and Language-Model Text Steganography

Executive summary#

The techniques in scope all exploit the same basic resource: natural language has multiple acceptable realizations of roughly the same communicative intent. A sender can therefore use the choice among synonyms, paraphrases, semantic categories, entities, syntactic realizations, or language-model next tokens as a secondary channel. The central technical problem is not merely to produce grammatical text; it is to make the distribution of encoded choices look like the distribution an innocent writer or language model would have produced. Christian Cachin's information-theoretic formulation makes this precise: steganalysis can be treated as hypothesis testing between cover and stego distributions, with relative entropy/KL divergence quantifying distinguishability. source

Four families are especially relevant:

The empirical figures are not directly comparable: papers use different models, tokenizers, corpora, sequence lengths, truncation rules, and definitions of “word.” Ueoka et al. explicitly contrast the low payload of classical edit-based systems with generation systems in the 1–5 bits/word range; SAAC's GPT-2 experiments and ADG's RNN experiments illustrate how strongly capacity depends on the predictive distribution and coding rule. source

The main progression in the field has been from fixed symbolic codebooks toward probability-matched encoding. A binary synonym codebook gets one nominal bit whenever a usable synonym slot occurs, but forcing uniformly distributed secret bits through nonuniform linguistic choices changes word frequencies. Huffman, arithmetic, adaptive grouping, minimum-entropy coupling, and related probabilistic methods instead try to allocate more bit patterns to more probable linguistic outcomes. Ziegler, Deng, and Rush showed that ideal arithmetic coding can make the induced distribution extremely close to the underlying LM; their untruncated experiment measured KL divergence around \(4\times10^{-8}\) nats. source

That result should not be read as practical undetectability. In 2026, Cao, Wang, and Hu showed that finite-precision implementations can produce “low-probability vanishing” artifacts and proposed RRNs-HT, a hypothesis-test detector aimed specifically at the sampling mechanism. Their homologous experiments—comparing ordinary and steganographic outputs from the same model rather than merely distinguishing machine text from human text—detected arithmetic-coding and METEOR steganography even where semantic detectors approached chance. source

There is also a fundamental robustness split. Token-level codes are fragile: paraphrasing, insertion/deletion, re-tokenization, normalization, or translation can destroy exact decoding. Nozaki and Murawaki demonstrated that subword segmentation ambiguity alone can cause receiver-side failures after detokenization. By contrast, semantic/entity-based systems can survive lexical rewriting because their information resides at a higher semantic level; Bai et al. report substantial robustness to token edits and paraphrasing, although those figures come from a 2024 preprint and a particular semantic extractor/generator setup. source

A rigorous evaluation should therefore treat steganography as a multi-objective optimization problem:

\[
\text{maximize useful payload}
\quad\text{subject to}\quad
\text{decode reliability},\;
\text{semantic fidelity},\;
\text{distributional similarity},\;
\text{robustness},
\]

rather than ranking systems by bits per word alone. A high-rate system that is easy to classify—or that breaks after ordinary text normalization—may have less usable covert-channel capacity than a lower-rate scheme. Zhang et al. captured this intuition with an “effective embedding rate” that discounts raw capacity by detector success. source

Unstated assumptions used below. “Semantic category” is interpreted broadly enough to include synonym sets, semantic bins, ontology/entity types, paraphrase classes, and semantically filtered LM candidate sets. Unless otherwise stated, the sender and receiver are assumed to share deterministic preprocessing, model/tokenizer versions, and any codebook; secret payloads are assumed encrypted or otherwise whitened so their bits are approximately uniform; a passive warden observes text unless an active paraphrasing/normalization channel is explicitly discussed; and “capacity” means payload before error-correction overhead. These assumptions matter because biased plaintext bits, model-version differences, and tokenizer discrepancies can all alter the stego distribution or make decoding fail. Bai et al., for example, explicitly apply a keyed PRF before semantic sampling, while neural token systems generally require sender and receiver to reproduce the same conditional distribution. source

Conceptual and information-theoretic framework#

At the most abstract level, let \(M\) be a secret message, \(K\) a shared key, \(X\) the innocuous linguistic context or cover, and \(Y\) the stegotext. An encoder implements

\[
Y = E_K(M,X),
\]

and the receiver computes

\[
\hat M=D_K(Y,X).
\]

Reliability requires \(\Pr[\hat M=M]\) near one. Security requires the distribution \(P_S(Y)\) induced by embedding to be difficult to distinguish from an innocent distribution \(P_C(Y)\). Cachin's passive-warden model frames Eve's task precisely as deciding between those two hypotheses and uses relative entropy as the information-theoretic security measure. source

A useful conceptual encoder–decoder architecture is:

flowchart LR
    A[Secret message] --> B[Encrypt / whiten payload]
    B --> C{Carrier mapping}
    C -->|Lexical| D[Synonym / vocabulary classes]
    C -->|Semantic| E[Entity / ontology / paraphrase class]
    C -->|Constrained| F[Allowed lexical or structural realizations]
    C -->|Probabilistic LM| G[Token probability intervals / groups]
    D --> H[Fluency + semantic filter]
    E --> H
    F --> H
    G --> H
    H --> I[Stegotext]
    I --> J[Channel: normalization, edits, paraphrase]
    J --> K[Reconstruct classes / token probabilities]
    K --> L[Inverse mapping]
    L --> M[Decrypt / dewhiten]
    M --> N[Recovered payload]

The important distinction is where the equivalence classes live. In lexical encoding the classes are sets of words; in paraphrase systems they are alternative sentences; in semantic steganography they are sets of sentences sharing an entity/type representation; in LM steganography the “classes” are portions of the model's next-token probability mass. Chang and Clark's contextual synonym work is an archetypal lexical approach, whereas Bai et al.'s ontology/entity method explicitly defines a semantic class as the set of possible sentences sharing a type. source

Capacity from choice entropy#

For a carrier position with \(m\) equally acceptable alternatives, a fixed-length mapping carries at most

\[
b=\lfloor\log_2 m\rfloor
\]

whole bits per use. Four alternatives permit two bits; two alternatives permit one. If only a fraction \(r\) of words can safely be modified and an eligible position \(i\) has \(m_i\) choices, a crude lexical upper bound is

\[
R_{\text{lex}}
\approx
\frac{1}{N}
\sum_{i\in\text{eligible}}
\log_2 m_i
\quad \text{bits/word}.
\]

This is an optimistic upper bound because contextual filtering, ambiguous decoding, error correction, and distribution matching all consume usable capacity. The masked-LM work of Ueoka et al. illustrates precisely this control: masking fewer positions lowers payload but improves resistance to automatic detection. source

If choices have natural probabilities \(p_1,\ldots,p_m\), the relevant quantity becomes entropy,

\[
H(P)=-\sum_i p_i\log_2 p_i .
\]

A distribution-matched entropy coder can asymptotically approach this average number of bits while selecting outcomes with the proper nonuniform probabilities. This is why arithmetic coding is so attractive: ordinary arithmetic coding compresses samples toward their entropy, and reversed arithmetic coding maps near-uniform secret bits into outcomes according to the desired language-model distribution. Ziegler et al. explicitly exploit this reversal. source

The same argument applies one level higher to semantic steganography. Let \(S\) be a semantic class or entity-type combination with innocent probability \(q(s)\). A distribution-preserving semantic coder has an average capacity bounded by approximately

\[
H(S)=-\sum_s q(s)\log_2 q(s)
\quad \text{bits per semantic realization}.
\]

Bai et al. make this connection explicit: the capacity of their ontology/entity construction depends on the entropy of the estimated distribution of entities and entity combinations, not on the cosmetic shape of the hand-built upper ontology. source

Security as distribution matching#

The canonical quantity is

\[
D_{\mathrm{KL}}(P_S\Vert P_C)=
\sum_y P_S(y)\log
\frac{P_S(y)}{P_C(y)}.
\]

A large value means there are outcomes or frequency patterns favored disproportionately by embedding. At zero, the two modeled distributions coincide. Cachin uses relative entropy to connect steganographic security with hypothesis-testing capability. source

There is, however, an important modeling caveat. In LM steganography, many papers compare \(P_S\) with the language model's distribution \(P_{\mathrm{LM}}\), not with the true distribution of human language \(P_{\mathrm{human}}\). Ziegler et al. explicitly state that their KL analysis evaluates the gap to the LM and therefore measures the embedding algorithm separately from the language model's own modeling error. Perfect matching to a poor LM still produces detectable machine-like text. source

A second caveat is implementation. Mathematical equivalence \(P_S=P_{\mathrm{LM}}\) assumes ideal arithmetic/probability computations and exact shared state. Finite numerical precision, probability truncation, altered tokenization, and deterministic implementation details can create a measurable sampling fingerprint. The 2026 LPV result is a direct demonstration of this gap between asymptotic/probabilistic proofs and actual implementations. source

Technique families and encoder–decoder mappings#

Vocabulary-class encoding#

Vocabulary-class encoding is the most literal realization of the user's description: partition a set of words into semantic or lexical classes, then communicate bits through the choice of a member while keeping the visible sentence plausible.

Early large-scale linguistic systems such as Chapman, Davida, and Rennhard's 2001 work used large dictionaries categorized by linguistic properties and sentence-generation structures. Later synonym systems became more semantically conservative: Chang and Clark developed contextual synonym substitution together with vertex coding to handle ambiguity in synonym relationships, treating contextual appropriateness as central rather than assuming that arbitrary dictionary synonyms can be exchanged. source

A simple keyed lexical encoder can be expressed as:

LEXICAL-ENCODE(text, payload, key):
    for each candidate position i:
        sense_i  <- disambiguate_sense(text, i)
        C        <- codebook[sense_i]
        C        <- filter_for_POS_morphology(C, context)
        C        <- filter_for_semantic_similarity(C, context)
        C        <- filter_for_fluency(C, context)

        b <- floor(log2 |C|)
        if b == 0:
            continue

        C <- keyed_permutation(C, key, position=i)
        j <- integer(next b payload bits)
        replace word_i with C[j]

    return text

The decoder repeats the contextual analysis, reconstructs the candidate class and keyed permutation, observes which member appears, and converts its index back to bits. The difficulty is that encoder and decoder must infer the same candidate set. A word whose sense is ambiguous, or a candidate accepted under one tokenizer/model version but rejected under another, can desynchronize every later payload bit. Chang and Clark's vertex coding was specifically motivated by coding ambiguities in synonym graphs. source

A minimal binary example is the familiar synonym choice:

Visible text: “She will wed him.” Shared contextual codebook: marry → 0, wed → 1 Decoded payload: 1

Ueoka et al. use this type of example to explain edit-based linguistic steganography before replacing manually built synonym groups with masked-LM alternatives. source

A richer exact example from their paper starts from “We completed the charitable task.” After masking selected words, their candidate mapping includes finished → 0, started → 1 and task → 00, project → 01, job → 10, labor → 11. Thus:

Stegotext: “We started the charitable project.” Decoded payload: 1 || 01 = 101

The attraction is obvious: three bits are carried by two ordinary-looking lexical choices. The danger is equally obvious: “completed” → “started” does not preserve the original proposition. Ueoka et al.'s system optimizes contextual fit rather than guaranteeing strict synonymy, illustrating the capacity-versus-semantic-fidelity tension. source

Mapping variants. A codebook can use fixed binary classes, \(2^b\)-way bins, keyed permutations, frequency-weighted variable-length codes, or graph/vertex coloring. In a fixed \(2^b\)-bin scheme, each usable carrier contributes exactly \(b\) nominal bits, but uniformly choosing among bins will distort language whenever the natural probability masses of those bins differ. Context-conditioned coding alleviates the worst substitutions but does not itself correct frequency bias. Chang and Clark's work, and later masked-LM editing, can be viewed as progressively stronger methods for constructing \(C_i\), the context-valid equivalence class. source

Capacity. The theoretical capacity is determined by the number and frequency of safely substitutable positions. Classical contextual synonym systems are therefore sentence-limited: Ueoka et al. describe prior edit-based work such as Chang–Clark as around two bits per sentence, versus much higher generation-based rates. Their masked-BERT configuration with masking frequency \(f=3\) and candidate threshold \(p=.02\) obtained about 0.204 bits/word while a BERT-based detector achieved 58.6% accuracy—only moderately above random guessing on a balanced test. source

Detectability. The decisive weakness of an unbalanced synonym codebook is that encrypted payload bits are essentially random, while ordinary authors do not choose synonyms uniformly. Xiang et al. explicitly exploit this: synonym substitution reduces the prevalence of high-frequency members of ordered synonym sets and increases lower-frequency ones. Chen et al. similarly use context-cluster fitness to detect substitutions that are statistically atypical for their surroundings. source

Semantic steganography#

Semantic steganography moves the carrier above individual surface words. Instead of encoding 0 by “marry” and 1 by “wed,” the sender maps bits to semantic classes, then asks a generator to produce any fluent sentence belonging to the selected class. The resulting words may all change under paraphrasing while the hidden semantic class survives.

A particularly explicit version is Bai et al.'s 2024 Semantic Steganography preprint. They construct an ontology/entity tree whose upper nodes contain concepts such as person and location, lower nodes refine them, and leaves are entities such as tourist locations. A sentence's “type” is defined by the entities and their multiplicities; all sentences sharing a type belong to the same semantic class. The empirical probabilities of those classes are estimated from entities extracted from LCCC, a cleaned corpus of 12 million conversations. source

Their encoder is probabilistic rather than assigning arbitrary fixed bit strings to entities. Cipher bits are randomized with a keyed PRF and interpreted as a position in \([0,1]\). The probability masses of ontology branches partition this interval. Following the branch containing the secret fraction eventually selects entities whose probability interval shares a binary prefix with the payload. A generation agent then realizes those required entities in fluent text; a checking agent rejects or requests regeneration when the semantic constraints are not met. At decoding time, an extraction agent reconstructs the sentence type and thereby the corresponding probability interval and bit prefix. source

A simplified version is:

SEMANTIC-ENCODE(ciphertext, key, semantic_distribution, ontology):
    bits <- PRF(key, ciphertext)
    u    <- binary_fraction(bits)          # approximately Uniform[0,1)

    target_type <- empty
    while not STOP:
        masses <- conditional_entity_masses(target_type)
        e, interval <- interval_containing(u, masses)
        target_type.add(e)
        renormalize u within interval

    repeat:
        sentence <- LLM.generate(constraint=target_type)
    until semantic_extractor(sentence) == target_type

    return sentence

This is structurally an entropy coder applied to a semantic distribution, with an LLM serving as a realization mechanism. Bai et al. emphasize that the ontology's tree shape does not create free capacity: maintaining the estimated entity distribution means capacity is tied to semantic entropy. source

An illustrative toy codebook, not a claim about Bai et al.'s specific implementation, makes the distinction clearer. Suppose a discourse naturally permits one of four semantically appropriate examples:

A payload 00 would constrain the generator to mention a cultural venue while remaining responsive to the same broad conversational goal. Unlike lexical substitution, the exact nouns, verbs, syntax, and sentence length need not be fixed. In a secure system the four codes should not simply be equiprobable unless the four classes are naturally equiprobable; the probability interval assigned to each should match its cover distribution, as in the ontology sampler. source

The reported capacity is striking but should be read as system-specific. With ChatGLM2-6B, Bai et al. report 28.5088 bits/sentence and 0.3958 bit/token, versus about 2.15–2.57 bits/sentence for METEOR, DISCOP, and arithmetic-coding baselines under their setup. Their 4-bit-quantized ChatGLM2 variant produced 27.8945 bits/sentence and 0.4130 bit/token. source

The apparent paradox—many bits per sentence but only about 0.4 bit/token—comes from using long, semantically constrained realizations. A class may encode a relatively large interval prefix, but generating a sentence reliably exhibiting its entities consumes many surface tokens. Thus semantic steganography can be high-capacity per communicative act without being high-capacity per surface token. source

Its principal advantage is robustness. Token insertion, deletion, synonym substitution, punctuation changes, or re-tokenization may leave the target entities and relations intact. Bai et al. report far higher successful decoding than arithmetic coding under random token edits and meaningful survival under paraphrasing; their paper explicitly motivates semantic encoding by the fragility of symbolic/token-level steganography to transcoding, word blocking, and tokenizer ambiguity. source

Its principal new detection surface is semantic distributional abnormality. If secret bits make an author mention people, locations, dates, or topics at frequencies inconsistent with the conversational setting, a detector need not look for unusual words at all. It can estimate \(P(\text{semantic type}\mid\text{context})\) and test whether the observed class sequence is plausible. This is the semantic analogue of synonym-frequency steganalysis and follows directly from Bai et al.'s own requirement that entity/type probabilities match their empirical distribution. source

Constrained text generation and transformation#

“Constrained generation” is best regarded as a bridge between fixed lexical coding and unrestricted LM generation. The sender first defines an admissible set of realizations and then embeds bits by choosing among them. Constraints can operate at several levels:

• lexical: only context-compatible synonyms;
• morphological or POS: preserve inflection and syntactic category;
• syntactic: choose among equivalent constructions;
• paraphrastic: select among sentence-level paraphrases;
• translation-based: exploit alternative translations or pivot-language realizations;
• masked-token: hold most of an existing text fixed and let an MLM alter only selected positions;
• semantic: require particular entities, concepts, or meanings.

Chang and Clark explored automatically generated paraphrases as linguistic carriers and later developed contextual synonym substitution in substantially greater depth. Yang et al.'s semantic-preserving pivot-translation method modifies a given text by translating through another language and then uses semantic-aware bins to embed information in alternative realizations; the stated goal is to improve payload while retaining semantic consistency. source

Masked-LM steganography is a particularly clean example because the constraint is “change only these positions.” Ueoka et al. select mask positions, obtain BERT's context-conditioned replacement distribution, discard candidates below a probability threshold, and assign bit chunks to the remaining alternatives. Increasing the distance between masked positions lowers capacity while improving detectability; the method therefore exposes a direct tunable security-rate curve. source

A general constrained encoder looks like this:

CONSTRAINED-ENCODE(cover, bits, key):
    slots <- choose_edit_slots(cover, key)

    for slot in slots:
        candidates <- generate_alternatives(cover, slot)

        candidates <- [
            c for c in candidates
            if grammar_ok(c)
            and semantic_similarity(c, cover) >= tau_sem
            and contextual_probability(c) >= tau_prob
        ]

        if too_few(candidates):
            continue

        # Prefer bins of similar total probability, not equal cardinality.
        bins <- probability_balanced_partition(candidates)

        symbol <- next_payload_symbol(bits, number_of_bins(bins))
        choice <- sample_within(bins[symbol],
                                proportional_to=natural_probability)

        cover <- apply(choice, cover)

    return cover

The important phrase is similar total probability. Four equal-size word lists do not imply four equally likely linguistic events. Probability-balanced bins reduce the frequency shift produced when the next payload bits are uniform. This principle reappears in generation-based adaptive grouping and distribution-preserving samplers. source

Constrained systems face a three-way trade-off. Strong semantic constraints reduce drift but shrink the admissible set and capacity. Weak constraints increase alternatives but admit replacements that subtly change entailments or register. Aggressive fluency filtering usually keeps only high-probability alternatives, again reducing capacity. Ueoka et al.'s comparison is informative: a conservative masked-LM setting transmitted only a fraction of a bit per word but was markedly harder for their classifier to detect than their GPT-2 generation baseline. source

Pivot translation adds another degree of freedom. Translation systems commonly admit multiple target-language expressions for similar source meaning, so a sender can deliberately steer a round-trip translation toward a code-compatible paraphrase. Yang et al.'s IEEE work explicitly uses pivot translation followed by semantic-aware bin coding to address semantic inconsistency and limited payload in direct modification. source

The cost is decoder synchronization. Unless the decoder knows exactly which choices were available to the encoder, free-form paraphrases are difficult to invert. Practical systems therefore require a shared deterministic paraphrase generator, a recoverable semantic class, a canonical ordering, or rejection rules that both sides can reproduce. This problem is analogous to subword-tokenization ambiguity in neural token coding, where Nozaki and Murawaki found that ordinary detokenization/re-tokenization can invalidate the receiver's reconstructed sequence. source

LM-generated steganographic text#

Generation-based systems dispense with an original sentence and encode directly while generating. Fang, Jaggi, and Argyraki's 2017 LSTM paper was an early neural generation milestone; later systems combined pretrained language models with fixed bins, Huffman coding, arithmetic coding, adaptive grouping, and increasingly formal distribution-preserving samplers. source

Fixed bins. Suppose the vocabulary is partitioned into \(2^b\) bins and the next \(b\) secret bits select one. The generator then samples a plausible token from that bin. The rate is exactly \(b\) nominal bits/token, but the induced probability distribution can be badly distorted because a payload bit pattern makes each bin equally likely even when the LM's total probability mass is not equal across bins. Zhang et al.'s experiments show the consequence clearly: fixed-bin encoders at 1–5 bits/word produced detector accuracies often around 0.72–0.89 rather than chance. source

Huffman coding. Construct a Huffman tree over likely tokens using LM probabilities; secret bits traverse the tree to a token. High-probability tokens receive shorter bit strings and low-probability tokens longer ones, so the mapping better follows the LM distribution than fixed bins. The discretization of probabilities into dyadic codeword lengths still creates mismatch. In SAAC's summary of prior methods, RNN-Stega builds a Huffman tree over the top \(2^H\) likely tokens and follows ciphertext bits through it. source

Arithmetic coding. Interpret an approximately uniform secret bit stream as a number in an interval. Partition that interval according to the LM's conditional probabilities; emit the token whose subinterval contains the secret number; renormalize and repeat. The decoder replays the same LM distributions and progressively narrows the message interval. Ziegler et al. describe exactly this reversal of compression: ordinary arithmetic coding maps language to uniform bits, while steganographic arithmetic coding maps uniform bits to language. source

LM-ARITHMETIC-ENCODE(cipher_bits, shared_LM, context):
    u <- interpret_cipher_bits_as_fraction_in_[0,1)

    while payload_not_determined:
        p <- shared_LM.next_token_distribution(context)
        p <- agreed_truncation_and_normalization(p)

        intervals <- cumulative_probability_intervals(p)
        token, [lo, hi) <- interval_containing(u, intervals)

        emit(token)
        context.append(token)

        # map chosen subinterval back to [0,1)
        u <- (u - lo) / (hi - lo)

The decoder computes the same intervals from the observed tokens and reconstructs the common binary prefix. In practice the parties must agree on model weights, tokenizer, floating-point/quantization behavior, prompt/context, truncation, termination, and coding precision. Ziegler et al.'s implementation used fixed-precision arithmetic; later work identified finite precision itself as a steganalysis surface. source

The asymptotic attraction is that the expected rate approaches the entropy of the model's conditional distribution. Ziegler et al. evaluated rates between roughly one and five bits/word; in their human experiment, arithmetic and Huffman generations were judged plausible continuations more than half the time up to around three bits/word, although the best human indistinguishability occurred at the lower-rate settings. Their unmodulated arithmetic coder closely matched the LM distribution in KL terms. source

Self-adjusting arithmetic coding. Shen, Ji, and Han's SAAC explicitly imposes an imperceptibility parameter \(\delta\) and adjusts the usable next-token distribution rather than taking an arbitrary fixed top-\(K\) cut. Across Drug, News, COVID-19, and Random settings, their strongest reported \(\delta=.01\) configurations were roughly 4.78–4.86 bits/word with KL values around 0.093–0.117, outperforming the paper's arithmetic, Huffman, and bin baselines on its joint rate/divergence criterion. source

Adaptive dynamic grouping. Zhang et al.'s ADG dynamically groups candidates so each bit-defined group has approximately balanced total model probability. In their Movie, News, and Tweet experiments it obtained 5.147, 5.650, and 5.411 bits/word, while two steganalysis detectors were near chance—approximately 0.50–0.55 accuracy. This result shows how probability-matched grouping can defeat detectors that easily catch fixed bins; it does not imply universal indistinguishability outside that model/domain/threat model. source

More recent coding. The line has continued toward coding mechanisms intended to combine controllable payload and closer distribution matching. A 2026 Computational Linguistics paper proposes self-adjusting asymmetric-number-system coding, while OD-Stega formulates next-token replacement as an optimization under KL or total-variation constraints. These developments preserve the same basic objective: maximize transmission subject to a bound on the distortion of the LM's sampling distribution. source

The historical trajectory is therefore:

Capacity, fluency, semantic fidelity, and implementation trade-offs#

There is no single “capacity of text steganography.” Capacity is conditional on what linguistic properties are required to remain invariant.

If the sender must preserve the exact proposition of an existing sentence, only truly interchangeable lexical/syntactic variants are usable and capacity is usually low. If the requirement is merely to stay on topic, far more semantic alternatives become available. If the sender may generate an entirely new innocuous passage from an LM, the usable entropy can approach several bits per token. If semantic classes rather than tokens carry the code, many bits can be attached to one long generated sentence even though the rate per token remains modest. These regimes are represented respectively by Chang–Clark, Ueoka et al., neural arithmetic coding/SAAC/ADG, and Bai et al. source

Comparative capacity and trade-offs#

Empirical entries above come from experiments that are not apples-to-apples. Ueoka et al. report the edit/generation rate gap; Shen et al. use GPT-2 Medium and different prompt domains; Zhang et al. train/evaluate on Movie, News and Tweet corpora; Bai et al. use ChatGLM2 and LCCC-derived semantic statistics. source

Why increasing rate usually increases detectability#

Consider a natural candidate distribution

\[
P=(0.70,0.20,0.08,0.02).
\]

Its entropy is only about 1.16 bits. A naïve two-bit code that maps 00, 01, 10, and 11 uniformly to the four words instead induces

\[
Q=(0.25,0.25,0.25,0.25),
\]

dramatically increasing the rarest word from 2% to 25%. The text may remain grammatical, yet a corpus-level detector sees a huge distributional change. This is the fundamental reason that “four synonyms = two free bits” is false under a steganographic threat model. Probability-aware coding instead gives the common alternative a much larger fraction of the bit interval, accepting a rate closer to entropy. This is the principle underlying arithmetic coding and the balanced-distribution objectives of later methods. source

The same phenomenon occurs at semantic level. Suppose travel conversations mention restaurants 60% of the time, museums 20%, parks 15%, and universities 5%. A fixed two-bit semantic code would force each to 25%; a semantic classifier could detect the resulting overproduction of universities even though every sentence is individually natural. Bai et al.'s use of empirical semantic distributions and probability-interval sampling is designed specifically to avoid that class-frequency flattening. source

Semantic drift and pragmatic distortion#

Lexical similarity does not guarantee proposition preservation. “Started” and “completed” can both have high probability in “We ___ the charitable project” while encoding opposite event phases. A masked LM optimizes contextual likelihood, not logical equivalence. The Ueoka example demonstrates why an embedding system that only checks LM probability should additionally use entailment, word-sense, temporal, negation, and entity-consistency checks when semantic fidelity matters. source

Sentence-level semantic coding has a different failure mode: the generated sentence may contain every required code entity but still be pragmatically strange. The stronger the hidden constraint, the more likely the generator is to mention an otherwise irrelevant place/person/object. Bai et al.'s checking-and-regeneration loop exists because simply prompting an LLM with the required semantic class does not reliably produce a compliant, high-quality sentence. source

Accordingly, “semantic similarity” should be decomposed during evaluation rather than reduced to one embedding cosine:

\[
\text{fidelity} =
\{\text{bidirectional entailment},
\text{entity preservation},
\text{negation},
\text{temporal relations},
\text{coreference},
\text{topic relevance},
\text{pragmatic naturalness}\}.
\]

This decomposition is an analytical recommendation based on the documented contextual-substitution and semantic-constraint failure modes, rather than a metric prescribed by a single paper. source

The synchronization problem#

Shared vocabulary classes alone are insufficient for robust decoding. Sender and receiver must agree on which locations were carriers, how a word was tokenized, which sense it had, which alternatives were admissible, and in what canonical order they were mapped. A change in any of these can cause an insertion/deletion in the recovered bitstream rather than one isolated bit error.

Neural generation raises the synchronization requirement still further. Sender and receiver commonly need identical model weights, tokenizer vocabulary, numerical precision, conditional context, truncation/top-\(k\) policy, temperature, token ordering, and coding state. SAAC notes that its compared methods require the employed LM to be shared between sender and receiver. Nozaki and Murawaki subsequently showed that subword detokenization can lead to a different re-tokenization and hence decoding failure. source

For practical experimentation, therefore, the protocol should version every component:

protocol_id = hash(
    model_weights_id,
    tokenizer_version,
    prompt_template,
    candidate_filter,
    semantic_model,
    probability_precision,
    coding_algorithm,
    termination_rule
)

The key is separate from this protocol identifier. A key can randomize codebook orderings or drive a PRF, but it cannot repair disagreement over the underlying candidate probabilities.

Steganalysis, detection signals, and countermeasures#

Detection should be organized by what invariant the stegosystem attempts to imitate. Lexical systems may preserve meaning but alter lexical statistics. Probability-coded LM systems may reproduce token statistics but still expose numerical-sampling artifacts. Semantic systems may preserve token-level naturalness yet skew which concepts appear.

Statistical signals#

N-gram and collocational shifts. Synonym substitution often replaces the locally expected word with a less frequent synonym. Context-cluster and n-gram/collocation statistics therefore detect choices that are individually grammatical but unlikely in that neighborhood. Chen et al.'s context-cluster method targets exactly this substitution-induced loss of contextual fitness. source

For a token \(w_t\), a simple score is

\[
s_t=-\log P_{\text{ref}}(w_t\mid w_{t-n:t-1}),
\]

and a document detector can compare the mean, tail quantiles, or distribution of \(s_t\) against same-domain controls. A substitution system often creates a heavier tail of surprisingly chosen words.

Synonym-frequency shifts. Xiang et al. order synonym alternatives by frequency and observe that substitution tends to reduce high-frequency attribute pairs while increasing low-frequency ones. A detector can build histograms over (rank within synonym set, set size) and apply a classifier or a goodness-of-fit test. source

Perplexity. A reference LM can compute

\[
\operatorname{PPL}(x)=
\exp\left[
-\frac1T \sum_{t=1}^T \log P_\theta(x_t\mid x_{<t})
\right].
\]

High PPL can reveal forced low-probability choices. Yet it is a weak standalone detector because topic/domain shift, model mismatch, creativity, nonnative writing, and text length also move perplexity. Moreover, probability-matched arithmetic or ADG schemes can deliberately stay close to their generating model even while carrying substantial payload. Ziegler's and Zhang's results illustrate both points. source

Distributional divergence. With sufficient samples, compare empirical cover and suspect distributions using KL divergence, Jensen–Shannon divergence, total variation, chi-square tests, or kernel/two-sample statistics. Token unigrams are only the first level; conditional next-token distributions, sentence embeddings, POS sequences, dependency motifs, and semantic classes can all be compared. Cachin provides the core hypothesis-testing rationale for divergence-based security evaluation. source

POS and structural anomalies. Dictionary/template encoders and constrained substitution can inadvertently alter the relative frequency of adjectives, adverbs, uncommon verb forms, dependency structures, or sentence lengths. Deep steganalyzers have therefore moved beyond unigram frequency: Wu et al.'s graph-neural steganalysis work models text graphically to capture syntactic and global associations rather than treating it solely as a flat token sequence. source

Entropy profiles. A useful detector for LM steganography compares the entropy of the model's next-token distribution with the realized token ranks. Fixed-bin encoders may choose unusually low-ranked tokens at low-entropy positions because a payload forces a particular bin. “Patient” strategies explicitly skip such dangerous positions when divergence exceeds a threshold, which indirectly confirms why conditional-entropy stratification is a strong evaluation tool. source

Semantic and pragmatic signals#

A semantically aware detector asks whether the text says something plausible for the discourse, not merely whether every word is probable.

Useful features include topical continuity between previous context and generated sentence, named-entity type/count distributions, semantic-role consistency, bidirectional entailment between original and modified text, unexpected changes of event state or polarity, unusual specificity, unexplained introduction of named places/people, and discourse connectives that do not fit the pragmatic relation.

These signals follow naturally from known stegosystem constraints. Contextual lexical systems risk accepting contextually probable but non-equivalent words, while ontology steganography deliberately controls entities. Bai et al.'s system even defines the code-bearing semantic type by entity identities and counts; therefore an adversary can invert the perspective and test whether those type distributions are anomalous under the surrounding discourse. source

A practical semantic anomaly score could be:

\[
S(x,c)=
\alpha\,[-\log P(T(x)\mid c)]
+\beta\,\text{NLI-contradiction}(x,c)
+\gamma\,\text{topic-drift}(x,c)
+\delta\,\text{entity-surprisal}(x,c),
\]

where \(T(x)\) is a semantic type and \(c\) is preceding discourse. This is a proposed detector architecture, not a result from one cited system; its component choices are motivated by the semantic carriers described above.

Classifiers and hypothesis tests#

A binary neural steganalyzer trains on

\[
H_0:x\sim P_C,\qquad H_1:x\sim P_S
\]

and predicts the source. Architectures in the literature range from engineered context/frequency features through neural semantic-space models to graph neural networks. Zhang et al., for example, evaluated stegotext against FastText- and TextCNN-based steganalysis and reported accuracy approaching 0.5 for ADG; the important lesson is not that ADG is universally secure, but that detector choice and experimental provenance matter enormously. source

Accuracy alone is a poor metric when operational base rates are low. Report ROC-AUC, PR-AUC, equal-error rate, and especially TPR at low FPR such as 0.1%, 1%, and 5%. A censor scanning millions of innocent messages may require extremely low false-positive rates, so a detector with good balanced accuracy can still be operationally useless.

The statistically optimal conceptual test, when \(P_C\) and \(P_S\) are known, is a likelihood-ratio test:

\[
\Lambda(x)
=
\log\frac{P_S(x)}{P_C(x)}
\mathop{\gtrless}_{H_0}^{H_1}
\tau.
\]

Cachin's model directly motivates such hypothesis-testing formulations. In practice neither distribution is exactly known, so classifiers and density-ratio estimators approximate this decision. source

A particularly important methodological advance is homologous evaluation: cover and stego samples should come from the same model, prompts, domains, and decoding setup except for the steganographic sampler. Otherwise a detector may simply learn “human versus GPT,” “temperature 0.7 versus 1.0,” or topic differences. Cao et al.'s 2026 RRNs-HT work explicitly adopts this homologous setup to isolate finite-precision sampling artifacts. source

Their Low-Probability Vanishing result also changes how claims of “provable security” should be interpreted. Arithmetic or cryptographic sampling proofs generally concern ideal probability distributions; actual finite-precision arithmetic can eliminate outcomes whose mathematical probability is nonzero. RRNs-HT constructs representative random numbers and applies hypothesis testing to the sampling mechanism, detecting AC and METEOR even when semantic steganalyzers fall toward random guessing. source

Detection matrix#

Sender-side anti-detection techniques#

There are several academically important ways a stegosystem reduces its statistical footprint.

Payload whitening. Encrypting or PRF-randomizing the secret prevents biased plaintext from overproducing particular classes. Bai et al. explicitly randomize cipher bits with a keyed PRF before semantic interval sampling. This does not by itself make a bad codebook distribution-preserving; it merely gives the probabilistic encoder the uniform input it assumes. source

Probability-balanced mapping. Assign classes according to total natural probability rather than equal cardinality. Arithmetic coding is the canonical continuous version; ADG dynamically approximates balanced groups; SAAC modifies the usable distribution under an imperceptibility constraint. source

Rate adaptation. Embed more bits only where linguistic entropy is high. At a deterministic phrase such as “United States of ___,” forcing a rare alternative is conspicuous; in a genuinely high-entropy continuation, several alternatives may be safe. Patient-Huffman explicitly waits when its divergence criterion is too high. source

Candidate smoothing/filtering. Removing implausible long-tail tokens improves local fluency, but truncation itself changes the distribution. Ziegler et al. note that their small positive KL under top-\(k\) arithmetic coding arises from truncating the long tail. Thus “smoothing” is not intrinsically secure; it trades rare grotesque outputs for a systematic sampling bias. source

Cover/context selection. High-entropy, stylistically diverse contexts support more choices than formulaic ones. For edit-based schemes, choosing covers with many safely replaceable locations raises capacity without forcing marginal synonyms. For semantic systems, contexts with naturally broad semantic possibilities offer more class entropy. Bai et al. explicitly note that low-semantic-entropy prompts restrict redundancy, while the entropy analysis of token systems gives the analogous word-level principle. source

Semantic verification. Bidirectional entailment and entity/relation preservation can reject contextually likely but meaning-changing substitutions. This is especially important for masked LMs, where high token probability is not a synonym guarantee, as the “started/completed” example demonstrates. source

Warden-side disruption and normalization#

An active defender need not perfectly classify stegotext if it can harmlessly destroy the covert channel.

Paraphrasing is an effective active defense against exact lexical and token codes because it changes the code-bearing word identities while preserving visible meaning. It is much less effective against semantic/entity coding: Bai et al. explicitly design their carrier to survive semantics-preserving rewriting and report substantial paraphrase robustness. source

Lexical canonicalization replaces interchangeable variants with preferred canonical forms. For example, every member of a known synonym class could be normalized to its most common context-valid realization. This collapses a lexical alphabet from \(m\) symbols to one, destroying its payload while preserving much of the text's semantics. The trade-off is stylistic damage and false transformations; synonym steganalysis research demonstrates why synonym rank/frequency is a natural basis for such normalization. source

Tokenization and Unicode normalization can remove formatting- or subword-dependent channels, although legitimate text must be preserved. Nozaki and Murawaki's finding that ordinary detokenization/re-tokenization can break neural decoding demonstrates how powerful even benign normalization can be against exact token-sequence carriers. source

Round-trip translation or controlled regeneration is a stronger semantic normalizer: translate or ask a trusted model to restate the proposition in a canonical style. This can destroy lexical, syntactic, and many token-probability codes simultaneously. However, a semantic carrier based on entities/topics may survive precisely because those features are what a good paraphraser preserves. Yang et al.'s use of pivot translation as an encoding resource and Bai et al.'s paraphrase-resistant semantic carrier expose both sides of this trade-off. source

This yields a useful defensive hierarchy:

\[
\text{surface normalization}
\rightarrow
\text{lexical canonicalization}
\rightarrow
\text{paraphrase/regeneration}
\rightarrow
\text{semantic canonicalization}.
\]

Each step attacks a higher-level carrier but carries a greater risk of damaging legitimate user meaning. Semantic steganography is difficult to neutralize without changing some semantic degrees of freedom, because those degrees of freedom are the channel.

Evaluation and reproducibility blueprint#

A strong experiment should separate capacity, reliability, linguistic quality, statistical security, semantic security, robustness, and computational cost. Collapsing them into one score hides important failure modes.

Recommended metrics#

SAAC explicitly uses bits/word and KL divergence as its main rate/imperceptibility measures, while Zhang et al. supplement rate and KL-style metrics with automatic steganalysis. Bai et al. add mission-success rate, Dist-3, semantic evaluation and channel-attack robustness. These choices motivate the broader matrix above. source

Do not compare raw perplexities across different tokenizers and scoring models as though they are on one universal scale. A sentence's PPL is defined relative to a particular probabilistic model and tokenization. The safest protocol is to score every system's outputs with the same held-out reference model and accompany PPL with human and semantic evaluation. Ziegler et al.'s separation between the steganographic distribution and the underlying LM's modeling quality is the conceptual reason for this distinction. source

Experimental protocol#

A reproducible benchmark should use a factorial design:

Carrier family: lexical codebook, masked-LM edit, paraphrase/pivot, fixed-bin LM, Huffman, arithmetic, SAAC/ADG-style probability matching, semantic-class encoding.

Payload rate: sweep from near-zero toward the algorithm's practical limit rather than reporting one cherry-picked operating point. Ueoka, Ziegler, SAAC, and Zhang all demonstrate that security changes materially with rate or its control parameter. source

Cover provenance: create paired samples from the same domain and, for generative systems, the same LM and prompt distribution. Each prompt should generate an ordinary-control sample and steganographic samples at several payload rates. Cao et al.'s homologous methodology is particularly important here because it prevents a detector from exploiting unrelated machine-versus-human artifacts. source

Train/test separation: split by source document, author/topic, and prompt template rather than by individual sentence wherever possible. Otherwise near-duplicate contexts can make detector numbers misleading.

Detector generalization: evaluate not only in-domain detection but cross-rate, cross-topic, cross-model, and leave-one-stegosystem-out detection. A detector that recognizes a specific fixed-bin implementation is qualitatively different from one that learns a general signature of covert sampling.

Statistical power: bootstrap confidence intervals across prompts/documents and run multiple random seeds. For hypothesis tests, report effect sizes and power curves versus number of observed sentences, because a divergence too small to detect from one sentence may be easy to detect from hundreds.

Suggested datasets and language-model baselines#

For historical reproduction, CNN/DailyMail context with GPT-2 345M/Medium is especially valuable because Ziegler et al. use CNN/DailyMail news context and GPT-2, and SAAC explicitly implements its comparisons with pretrained GPT-2 Medium. This provides a direct bridge to arithmetic, Huffman, bin, and SAAC results. source

For steganalysis benchmarking, Zhang et al.'s Movie, News, and Tweet corpora provide three substantially different styles. Their processed datasets contain roughly 1.00 million/111k train/test movie examples, 1.46 million/162k news examples, and 1.57 million/175k tweet examples, giving a useful historical baseline for reproducing ADG and its detectors. source

For semantic carriers, LCCC is appropriate because Bai et al. construct their semantic entity distribution from its 12 million cleaned daily conversations. A reproduction should publish the extracted ontology, entity frequency tables, extractor version/prompts, and train/evaluation splits; without those, nominally identical semantic coding may induce a different class distribution. source

A useful model ladder would be:

Crucially, the same experiment should not silently mix tokenizers. Report both bits per model token and bits per whitespace-delimited word, because subword length varies by language and model; Nozaki and Murawaki's segmentation results make this more than a cosmetic reporting issue. source

Recommended ablations#

For vocabulary-class systems, vary synonym-set size, minimum candidate frequency, semantic-similarity threshold, WSD confidence, and fraction of modified words. Plot payload versus detector AUC and payload versus semantic error.

For constrained/MLM systems, vary mask spacing, candidate probability cutoff, entailment threshold, and whether bins are equal-cardinality or probability-balanced. Ueoka et al.'s results predict a strong rate/detectability curve as masking becomes more aggressive. source

For LM arithmetic systems, sweep top-\(k\), temperature, arithmetic precision, sequence length, and payload rate. Include an ideal high-precision implementation and deliberately reduced precision to measure the finite-precision leakage highlighted by Cao et al. source

For semantic systems, vary semantic-type entropy, number of required entities, entity frequency, generator strength, extractor strength, and number of regeneration attempts. Measure how many payload bits survive paraphrase while separately measuring how unusual the chosen semantic classes are in context. Bai et al.'s framework makes both the entity-count capacity effect and rejection-generation cost directly relevant. source

The most informative visualization is a Pareto frontier, not a single leaderboard:

\[
x=\text{net bits/token},\qquad
y=\text{detector AUC or divergence},
\]

with point shape representing semantic-error rate and perhaps separate curves for channel robustness. Systems below and to the right—more capacity with less detectability—dominate those above and to the left. Ziegler's KL-versus-bits/word experiment is an early example of exactly this style of trade-off analysis. source

Open research questions and ethical considerations#

What is the correct cover distribution? Matching \(P_{\mathrm{LM}}\) is mathematically convenient but not necessarily equivalent to matching actual human communication. A perfect sampler from a recognizably machine-like model can be detected as generated text even if the covert embedding leaves no additional token-distribution footprint. Ziegler et al. explicitly distinguish LM mismatch from embedding mismatch, and the distinction becomes more important as steganography moves into conversational LLMs. source

Can security proofs survive real computation? The 2026 LPV result establishes that finite-precision arithmetic can invalidate the practical implication of ideal distribution-matching arguments. An important research direction is therefore implementation-aware steganographic security: proofs should model fixed-point arithmetic, quantized probabilities, truncation, tokenizer normalization, and serialization rather than treating the sampler as an exact real-valued oracle. source

What is the semantic analogue of perfect distribution matching? Token security asks for \(P_S(x)=P_C(x)\). A semantic system needs something stronger and more contextual:

\[
P_S(T\mid C)=P_C(T\mid C),
\]

where \(T\) is the semantic class and \(C\) the discourse context. Matching global entity frequencies is insufficient if a selected entity is surprising in this conversation. Bai et al.'s empirical semantic distribution is an important step, but conditional semantic modeling remains a much harder problem. source

How much semantic entropy is actually usable? There may be large theoretical entropy in “all possible things an LLM could plausibly say,” yet only a small subset preserves the user's communicative intent. A useful theory would condition capacity on a semantic equivalence relation:

\[
C_{\text{meaning}}(x)
=
H(Y\mid Y\equiv_{\text{meaning}} x),
\]

and distinguish proposition-preserving capacity from merely topic-preserving capacity. The gap between contextual fit and exact semantic preservation in edit systems, and between entity compliance and discourse naturalness in semantic systems, shows why this distinction matters. source

Can semantic detectors generalize across encoders? A classifier trained against one vocabulary partition often learns its particular lexical artifacts. A more durable approach may model entity surprisal, discourse coherence, semantic-role distributions, or conditional topic transitions independent of any specific codebook. Graph-based linguistic steganalysis is one move away from narrowly engineered surface statistics, but robust cross-system semantic detection remains open. source

How should active defenses preserve utility? Paraphrasing easily disrupts token-level codes but risks changing legitimate nuance, quotations, legal terms, technical vocabulary, names, or style. Semantic steganography deliberately survives meaning-preserving paraphrase, implying that eliminating a high-level covert channel may require narrowing the very semantic freedom that makes natural-language systems useful. Bai et al.'s robustness motivation and Nozaki and Murawaki's tokenization fragility illustrate the two ends of this spectrum. source

How should multi-message detection work? A single sentence can be statistically innocuous while a long conversation reveals a persistent category bias. Information-theoretically, small divergences can accumulate with repeated observations, so evaluations should report detection as a function of the number of messages available to the warden rather than only single-sentence accuracy. Cachin's hypothesis-testing framing is naturally suited to this aggregated setting. source

How should robustness be priced into capacity? A semantic code carrying 20 bits but requiring large error-correcting redundancy under paraphrase may deliver fewer net bits than a nominally lower-rate system. Conversely, fragile arithmetic token coding may approach entropy on a pristine channel yet deliver essentially zero information after routine rewriting. Future work should therefore report net reliably recovered payload after a declared channel model, not only encoder-side bit counts. Bai et al.'s robustness experiments and the documented tokenizer ambiguity of neural schemes make this distinction concrete. source

The ethical picture is intrinsically dual-use. Linguistic steganography can protect privacy, enable communication under censorship, support watermarking/provenance research, and provide a scientifically useful adversarial test of language-model sampling and detection. The same methods can hide data exfiltration, evade monitoring, facilitate collusion between automated systems, or conceal malicious instructions. The semantic-steganography authors explicitly motivate covert communication in terms of privacy and censorship, while the modern detection literature treats covert channels as a security problem requiring active steganalysis. source

For research publication, a defensible norm is therefore to release evaluation harnesses, detectors, controlled synthetic payloads, and reproducible probability/capacity measurements, while treating operational deployment in sensitive channels separately. Especially important is paired disclosure: a new high-capacity encoder should be evaluated against strong statistical, semantic, homologous, and implementation-level attacks rather than demonstrating only fluency and raw payload. The recent progression from nominally distribution-preserving arithmetic systems to finite-precision steganalysis shows why offensive and defensive evaluation need to advance together. source

The deepest technical conclusion is that selecting words from semantic categories is not fundamentally about synonyms. It is about constructing an equivalence class of acceptable linguistic realizations and using its residual entropy as a communications channel. Vocabulary-class encoding defines that class lexically; constrained generation defines it through explicit linguistic constraints; LM steganography defines it through a next-token probability distribution; semantic steganography defines it through meanings, entities, or ontology types. The closer the encoder's selection probabilities match the genuine conditional distribution of those alternatives, the harder statistical detection becomes—but the farther the carrier moves from exact surface symbols toward meaning, the more detection and defense must move toward discourse, pragmatics, and semantic distribution modeling as well. source