Text frequency analysis counts how many times each word or character appears in a text. In English, this exercise has a famous literary connection — Edgar Allan Poe built an entire mystery story around exactly this technique.
"E" and the shape of English
"E" is the most common letter in English text, showing up in roughly one out of every eight or nine letters typed — a byproduct of how many common words ("the," "he," "she," "be") and endings rely on it. Letters like "Q," "X," and "Z" sit at the opposite end, appearing so rarely that their presence barely registers in a frequency table for ordinary prose.
The Gold-Bug: cryptanalysis as detective fiction
Edgar Allan Poe's 1843 short story "The Gold-Bug" centers on a character who deciphers a treasure map's cryptogram purely through letter frequency analysis — counting which symbol appears most often, guessing it stands for "E," and working outward from there. The story predates modern cryptography education but demonstrates the same principle taught in computer science courses today: a simple substitution cipher can't hide the statistical fingerprint of the language underneath it.
Why frequency analysis eventually stopped working
Once cryptographers realized letter frequency could break substitution ciphers, they designed polyalphabetic ciphers (like the Vigenère cipher) specifically to flatten that fingerprint by using multiple substitution alphabets in rotation. Modern encryption goes much further, producing ciphertext that's statistically indistinguishable from random noise — the entire arms race started with the simple observation that "E" shows up too often to hide.
Why you need this
- Quickly grasping the main topics of a long document from its most frequent words.
- Checking text for a natural letter distribution (educational cryptography exercises).
- Analyzing word repetition for SEO optimization of a page's text.
Zipf's law
In natural languages, a word's frequency is inversely proportional to its rank in a list sorted by descending frequency: the most common word occurs roughly twice as often as the second most common, three times as often as the third, and so on. This pattern (Zipf's law) is so stable for ordinary text that a significant deviation from it is a signal — the text may be artificially generated, heavily keyword-stuffed, or written in a different language than expected.