Chris Spackman – Cryptography

Hitt, Friedman, and the Birth of Scientific Cryptanalysis

For centuries, codebreaking was an art. Skilled cryptanalysts relied on intuition, pattern recognition, and deep experience with languages. In the early 20th century, two Americans changed that — first Parker Hitt, who systematized the practical craft of military codebreaking, and then William Friedman, who introduced statistical and mathematical methods that could determine cipher type, find key lengths, and break encryption systematically. Together, they transformed cryptanalysis from a craft into a branch of applied mathematics.

Parker Hitt: The First American Textbook

Before Friedman, there was Parker Hitt (1878–1971), a U.S. Army officer stationed at Fort Leavenworth, Kansas, and later on the Mexican border. Hitt was fascinated by codes and ciphers, and he realized that the Army had no systematic training for codebreaking. Officers were expected to figure it out on their own.

In 1916, Hitt published Manual for the Solution of Military Ciphers — the first American military textbook on cryptanalysis. It was a slim, practical book, but it covered techniques that would remain relevant for decades:

• Frequency analysis — systematic methods for breaking substitution ciphers by counting letter frequencies and comparing them to known English patterns.
• Digraph and trigraph analysis — looking at pairs and triples of letters (like TH, THE, ING) rather than just individual letters, which gives much stronger clues.
• Breaking Vigenère ciphers — techniques for finding the key length and then breaking each alphabet separately.
• Solving transposition ciphers — methods for reconstructing the original letter order when the message has been scrambled.

The Manual was not a theoretical work — it was written for field officers who needed to break enemy messages quickly and correctly. Hitt filled it with worked examples and practical tips. It became the training manual for a generation of American military cryptanalysts, including many who later joined Yardley's MI-8 and Friedman's SIS.

The Strip Cipher

Hitt was also an inventor. Around 1914 he built a strip cipher device: a set of paper strips, each printed with the 26 letters in a different scrambled order, sliding side by side in a wooden frame. To encrypt a message, you slid the strips so that one row spelled out your plaintext, then read off a different row as the ciphertext. He also made a version with the alphabets on wooden discs stacked on a rod, the form Joseph Mauborgne turned into the Army's M-94 cylinder (25 aluminium discs, 1922). Thomas Jefferson had designed a similar wheel cipher in the 1790s, but it had been forgotten.

The strip cipher was a significant innovation. It was portable, fast to use in the field, and much harder to break than pencil-and-paper ciphers. Most importantly, it was a cipher device — a physical machine that performed encryption mechanically rather than relying on manual letter-by-letter work. Hitt's strips were a direct ancestor of the U.S. Army's M-138 strip cipher, which remained in service through World War II. It also foreshadowed the principle behind rotor machines like Enigma: using interchangeable, rearrangeable components to create complex substitutions.

Hitt's Legacy

Hitt's direct contributions were practical rather than theoretical. He did not develop new mathematics — he organized and systematized existing techniques and made them teachable. But that was exactly what the field needed. Before Hitt, American military cryptanalysis barely existed. After Hitt, there was a trained community of practitioners, a standard reference text, and a cipher device good enough for field use. When war came in 1917, the United States had something to build on.

Friedman himself acknowledged Hitt's foundational role. The Manual was among the texts Friedman studied at Riverbank, and several of the techniques Friedman later formalized mathematically were first described in practical terms by Hitt.

Friedman: From Genetics to Codebreaking

Friedman's path to cryptanalysis was unusual. Born in Kishinev in the Russian Empire (today Chișinău, Moldova) in 1891, he came to the United States as an infant. He studied genetics at Cornell and took a job at Riverbank Laboratories in Illinois, working for the eccentric millionaire George Fabyan. Fabyan was paying for a search for secret messages supposedly hidden in Shakespeare's plays. Elizebeth Smith had been hired for that project (she and Friedman married in 1917), and Friedman was drawn into the cipher work through it. The Shakespeare theory was wrong — the Friedmans themselves demolished it in a 1957 book — but it was how both of them learned cryptanalysis.

His scientific training proved invaluable. Where other cryptanalysts saw patterns intuitively, Friedman asked: can we measure these patterns? Can we express them as numbers? Can we build a systematic method that works even when intuition fails?

The Index of Coincidence

Friedman's most important contribution was the Index of Coincidence (IC), written in 1920 and printed by Riverbank in 1922. The idea is elegantly simple: if you pick two letters at random from a text, what is the probability that they are the same letter?

Why Does This Matter?

In English text, some letters are very common (E, T, A) and others are rare (Z, Q, X). If you pick two random letters from English text, there is a relatively high chance they will match, because common letters show up often. The IC of English is about 0.0667 (roughly 1 in 15).

In completely random text (all 26 letters equally likely), the probability of a match is much lower: 0.0385 (exactly 1/26, roughly 1 in 26).

This difference is the key insight. Different types of encryption affect the IC in predictable ways:

• Monoalphabetic ciphers (like Caesar or substitution) just rearrange which letter is which. They do not change the underlying frequency distribution, so the IC stays near 0.0667.
• Polyalphabetic ciphers (like Vigenère) flatten the frequency distribution by using multiple alphabets. The IC drops toward 0.0385, depending on the key length.
• Perfectly random text has an IC of exactly 0.0385.

The Formula

For a text of N letters where letter i appears n_i times:

IC = Σ n_i(n_i − 1) / N(N − 1)

The sum is over all 26 letters. Each term n_i(n_i − 1) counts the number of ways to pick two copies of letter i, and N(N − 1) is the total number of ways to pick any two letters.

Try It: Index of Coincidence Calculator

Frequency Distribution

The Kasiski Examination

The Index of Coincidence tells you what kind of cipher was used. But for a polyalphabetic cipher like Vigenère, you also need to find the key length. The Kasiski examination, developed by Friedrich Kasiski in 1863, does this by looking for repeated sequences in the ciphertext.

How It Works

In a Vigenère cipher, if the same plaintext sequence happens to align with the same part of the key, it produces the same ciphertext sequence. The distance between these repeated sequences is always a multiple of the key length.

By finding all repeated sequences of three or more letters (trigrams), measuring the distances between them, and finding the greatest common divisor (GCD) of those distances, you can determine the likely key length. Once you know the key length, you can split the ciphertext into groups (one per key letter) and attack each group as a simple Caesar cipher.

Try It: Kasiski Examination

Chi-Squared Test

The same statistical turn of mind brought standard tests into cryptanalysis, among them the chi-squared (χ²) test that Karl Pearson had introduced in 1900. (Friedman's own "chi test" in his Army manuals is a related but different calculation.) The chi-squared test measures how well an observed distribution matches an expected distribution. In cryptanalysis, you compare the letter frequencies in a ciphertext to the expected frequencies of English.

The formula is:

χ² = Σ (O_i − E_i)² / E_i

Where O_i is the observed count for each letter and E_i is the expected count (based on English frequencies and the total text length). A low χ² value means the text's frequency distribution closely matches English. A high value means it does not.

Using Chi-Squared to Break Caesar Ciphers

For a Caesar cipher, you can try all 26 shifts and calculate χ² for each one. The shift that produces the lowest χ² value is most likely to be correct, because that shift makes the frequencies best match English.

Try It: Chi-Squared Shift Finder

Friedman's Legacy

In 1929, when the Black Chamber was shut down, the Army established the Signal Intelligence Service (SIS) and put Friedman in charge. He hired a team of mathematicians — including Abraham Sinkov, Solomon Kullback, Frank Rowlett, and John Hurt — and trained them using the statistical techniques he had developed.

This team went on to break the Japanese PURPLE cipher machine in 1940, one of the greatest cryptanalytic achievements in history. They did it without ever seeing the machine — they deduced its structure entirely from analysis of intercepted messages, then built a replica.

Friedman's key innovations:

• Index of Coincidence — a universal measure for classifying ciphers and determining key lengths.
• Statistical testing — applying statistical tests to systematically evaluate cryptanalytic hypotheses instead of relying on a feel for the text.
• Formalization — defining precise terminology and mathematical notation for cryptanalysis, replacing informal, ad-hoc methods with reproducible techniques.
• Training — building a team and an institution (SIS, later NSA) that carried his methods forward.

Things to Try

• Calculate the IC of the English sample and compare it to the random sample. How different are they?
• Load the Caesar ciphertext into the IC calculator. Is the IC close to English or close to random? Why?
• Load the Vigenère ciphertext into the IC calculator. How does its IC compare? What does this tell you about the cipher type?
• Use the Kasiski examination on the Vigenère sample. What key length does it suggest?
• Use the chi-squared test on the Caesar sample. Which shift produces the lowest χ²? Does the decoded text make sense?
• Go to the Vigenère cipher page, encrypt a message with a keyword, then come back here and use the IC and Kasiski tools to figure out the key length.