Beyond Frequency Analysis

Panels 1–5 above use simple frequency counting — ranking code groups by how often they appear and matching the distribution to expected English. This works against simple substitution ciphers, but codebook ciphers (especially with homophones) are specifically designed to defeat it.

Panel 6 takes the first steps beyond counting. Without homophones it uses the shape of the text to identify the space code, the [CAP] marker, the commonest punctuation mark and the word “the” (the standalone code followed by the greatest variety of other codes), and it flags the codes that most often come right before “the” as function words without being able to name them. With homophones, a distributional clustering attack groups codes by the company they keep to find which ones are homophones of the same token. Panel 7 lets you toggle between frequency-only and context-enhanced reconstruction to see the difference, and panel 6½ is where real progress comes from.

But even context analysis is just the beginning. Historically, every major codebook system was eventually broken. Here is the full toolkit a modern cryptanalyst would bring to bear:

1. Known-Plaintext and Crib Attacks

The single most powerful technique. If the analyst knows (or can guess) even a small fragment of the original message — a greeting, a date, a signature, a formulaic opening — they can match those words against the code groups at each position. Every confirmed match recovers codebook entries, and those entries can then be recognized wherever they appear in the rest of the message (or in other messages using the same codebook). This bootstrapping effect means a single good crib can unravel a large portion of the codebook.

This is how the British Room 40 broke German naval codes in WWI, and how Bletchley Park attacked Enigma in WWII. The principle is the same for codebooks: find something you know, and use it as a lever.

2. N-gram Context Analysis

Instead of looking at each code group in isolation, analyze which groups appear next to which other groups. In English, certain word sequences are extremely common: “of the”, “in the”, “to the”, “it is”. If you have identified the code for “the” (the second most frequent group), then whatever groups appear immediately before it most often are likely “of”, “in”, and “to”. Each confirmed mapping opens up more context-based deductions.

A computer can build a co-occurrence matrix — for every pair of code groups, how often do they appear adjacent? — and compare this matrix against expected English word-pair frequencies. This is far more powerful than single-group frequency counting.

3. Homophone Clustering

The hardest part of attacking a homophonic codebook is figuring out which code groups are homophones of each other (i.e., different codes for the same plaintext token). Panel 6 above demonstrates a basic version of this when homophones are detected. Modern approaches use distributional similarity: two groups that consistently appear in the same contexts (same neighbors, same positions in sentences) are likely homophones. Techniques include:

• Mutual information: measure how much knowing one group’s neighbors tells you about another group’s neighbors. High mutual information → likely homophones.
• Context vectors: represent each group by a vector of its neighbor frequencies, then cluster groups whose vectors are similar (cosine similarity, k-means, etc.).
• Expectation-maximization (EM): iteratively guess a homophone clustering, decode the message under that clustering, score how “English-like” the result is, and refine.

Once homophones are clustered, their combined frequency matches the original token frequency, and standard frequency analysis works again.

4. Constraint Propagation and Grammar

English has rigid grammatical structure. Once a few words are known, the possibilities for neighboring words are heavily constrained. “the ___ of” can only be a noun. “___ is” is likely a pronoun or short noun. A computer can use language models (even simple n-gram models) to score candidate decodings and prune impossible ones. Modern large language models make this even more powerful — they can evaluate whether a partial decoding “sounds like English” with high accuracy.

5. Multiple Messages

Real codebooks are used for more than one message. Each additional message using the same codebook provides more data for frequency analysis, more context for n-gram analysis, and more opportunities for cribs. Historically, codebook security depended on changing the codebook frequently — but producing and distributing new codebooks was expensive and slow, so in practice books were used for months or years, giving analysts ample material.

6. Null Detection and Filtering

If null (dummy) code groups are present, they appear in the frequency distribution as a cluster of low-frequency groups with nearly identical counts — because each null code is inserted with equal probability. An analyst can identify this cluster statistically: look for a plateau in the frequency tail where 10+ groups all appear the same number of times. Once identified, filtering them out restores the original distribution and all standard techniques apply. Nulls add noise but do not change the fundamental structure of the real token frequencies.

7. Computational Brute Force

If the analyst knows the codebook structure (vocabulary size, group length, whether homophones are used), they can use computers to search the space of possible codebooks. For CODEX specifically, the codebook is generated from a seed string and a set of configuration options. If the configuration is known (or can be guessed), the search space reduces to just the seed — and modern computers can test millions of candidate seeds per second.

The Limit of Pure Frequency Analysis (and Why)

Here is what the Decryptor actually does on its own. Fed 60,000 words of a novel — 180,000 code groups, no homophones, no cipher — it identified three codes with confidence: the space, “the”, and the commonest punctuation mark. It flagged twelve more codes as function words that precede “the” (they really were of, in, and, to, with, by, at, all, from, for, on, that) but could not say which was which. Three times as much text would not change that. This is the structural limit of frequency analysis against a codebook, and it is worth understanding.

A codebook cipher is not a letter substitution. The output mixes about 1,300 different symbols — whole words, three- and two-letter chunks, single letters, markers — and their frequency ranks line up with no fixed table. In one text “of” outranks “to”, in the next it does not; a letter code such as s or d (word endings the book has no chunk for) can outrank every word but “the”. The textbook rule “Nth most frequent code = Nth most frequent English word” is therefore wrong from rank two onward, and an earlier version of this page that applied it produced confident, wrong reconstructions. The Decryptor now commits only to what the structure of the text proves.

What does work is what worked in 1917: a lever. Guess a word the message must contain, find where its shape fits, and let each identified code constrain the next guess (panel 6½, and the worked example). Read more than one message in the same book, so that rare words recur. And, today, let a computer search for the assignment of codes to tokens that makes the whole text most English-like — a language-model attack this page does not attempt.

How Long Would It Take?

Against CODEX specifically, a modern analyst with standard tools would likely proceed as follows:

• No cipher layer, no homophones: The structural codes (space, [CAP], punctuation, “the”) fall out of a few thousand groups. Content words do not: frequency alone tops out at a sentence skeleton. Cribs recover the names and places and their letters within minutes (try the worked example); a full solution needs a computer search with an English language model, which is beyond this page.
• Cipher layer, no homophones: A Caesar shift or a substitution on the code characters changes nothing that matters (it only relabels the groups). A Vigenère layer is found by its period and aligned away without the key, as panel 3 does; then proceed as above.
• No cipher layer, with homophones: Harder, but clustering techniques on a few thousand groups would identify the most common homophones within hours. With 10,000+ groups, most of the codebook is recoverable.
• Cipher layer + homophones: The hardest combination. Strip the cipher, then cluster homophones, then do frequency/context analysis. Still breakable with enough ciphertext (tens of thousands of groups) and standard computational tools. Days to weeks of analyst effort, not months.
• With null insertion: Nulls add noise but are straightforward to detect. The null codes form a visible plateau in the frequency tail (many groups with identical low counts). Once identified and filtered out, all other techniques proceed normally. Nulls slow analysis slightly but do not fundamentally strengthen the cipher.

The fundamental weakness of any codebook system is that the mapping is fixed: the same plaintext token always produces codes from the same set. No matter how many homophones or cipher layers are added, this structural regularity leaks information with every message. This is why codebook ciphers were progressively abandoned in the 20th century in favor of stream and block ciphers, which produce entirely different output for every encryption even with the same key.