Word Frequency
Problem statement
Count how often each word appears in a text file and print the words from most to least frequent, with ties in alphabetical order. The trick is to turn the text into one word per line first; after that it is the same count-and-rank pipeline you use for logs. The same steps find the most common error messages, user agents or failing test names.
notes.txt
The deploy failed. The rollback worked,and the second deploy worked.Deploy again? The team said: deploy!Treat upper and lower case as the same word, and ignore punctuation.
- Show the first 6 words after the text is turned into one word per line.
- Print every word with its count, most frequent first. When two words have the same count, put them in alphabetical order.
- Print only the top 3.
Expected output:
== one word per line (first 6) ==thedeployfailedtherollbackworked== every word, most frequent first ==deploy 4the 4worked 2again 1and 1failed 1rollback 1said 1second 1team 1== top 3 ==deploy 4the 4worked 2Hints
tr 'A-Z' 'a-z' lowercases text. tr -cs 'a-z' '\n' turns every run of characters that are not letters into a single newline, which leaves one word per line.Approach
Optimal: tr into words, then count
Covers: tr, tr -c, tr -s, tr -d, grep -v '^$', sort -k1,1nr -k2,2, sort keys, uniq -c, head, shell functions.
Turn text into a list, then count the list. Counting words is hard while they are inside sentences. Counting lines is easy, because sort | uniq -c already does it. So the real work is reshaping the text until every word sits on its own line.
lowercase"]):::purple L --> W(["tr -cs a-z newline
one word per line"]):::purple W --> E(["grep -v '^$'
drop empty lines"]):::purple E --> C(["sort | uniq -c
count each word"]):::purple C --> R(["sort -k1,1nr -k2,2
rank, ties A to Z"]):::purple R --> OUT["deploy 4
the 4
worked 2"]:::green classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px
tr swaps or removes characters. tr (translate) reads text and changes it one character at a time. It does not understand words, only characters, which makes it fast and simple.
| Command | Does | Example |
|---|---|---|
tr 'A-Z' 'a-z' |
each capital becomes its lowercase letter | The → the |
tr -d '0-9' |
delete every digit | v2 → v |
tr -s ' ' |
squeeze runs of spaces into one | a b → a b |
tr -c 'a-z' '\n' |
every character that is not a-z becomes a newline | a, b → a\n\nb |
tr -cs 'a-z' '\n' |
the same, and squeeze the newlines | a, b → a\nb |
-c means complement: "every character not in this set". -s means squeeze: "turn a run of the same output character into one". Together, tr -cs 'a-z' '\n' replaces each gap between words, whether it is a space, a comma and a space, or a full stop and a newline, with exactly one newline.
Why grep -v '^$'? If the text starts with punctuation or a space, the first gap becomes a newline at the very start, which is an empty line. ^$ matches a line with nothing between its start and end, and -v removes it. Without this, the empty line would be counted as a "word".
Sorting by two keys. Plain sort -rn sorts by the count, but words with the same count come out in no fixed order. sort -k1,1nr -k2,2 gives two rules, used in order:
| Part | Means |
|---|---|
-k1,1nr |
first look only at field 1 (the count), as a number, biggest first |
-k2,2 |
when counts are equal, look at field 2 (the word), A to Z |
The ,1 and ,2 matter: -k1 alone means "from field 1 to the end of the line", which mixes the word into the first rule.
Walking through the code. The # Setup: lines only create the sample text, so skip past them.
words() { ... }is a shell function: a name for a group of commands, so the three pipelines below do not repeat the cleaning steps. Callingwordsruns the lowercase, split and drop-empty steps onnotes.txt.words | head -n 6shows the list: one lowercase word per line, punctuation gone.sort | uniq -ccounts each word,sort -k1,1nr -k2,2ranks them with alphabetical ties, andawk '{print $2, $1}'printsword count.head -n 3before the finalawkkeeps the top three.
deploy and the both appear 4 times, so the second key puts deploy first.
Edge cases. An empty file gives empty output. tr 'a-z' only knows plain English letters, so a word like café is split at the é. Apostrophes split it's into it and s; add ' to the set (tr -cs "a-z'" '\n') if you want to keep them.
# Setup: create the sample text in a fresh temporary folder
cd "$(mktemp -d)"
cat > notes.txt << 'TEXT'
The deploy failed. The rollback worked,
and the second deploy worked.
Deploy again? The team said: deploy!
TEXT
words() {
tr 'A-Z' 'a-z' < notes.txt | # 1. make everything lowercase
tr -cs 'a-z' '\n' | # 2. every run of non-letters becomes one newline
grep -v '^$' # 3. drop any empty line
}
echo "== one word per line (first 6) =="
words | head -n 6
echo "== every word, most frequent first =="
words | sort | uniq -c | sort -k1,1nr -k2,2 | awk '{print $2, $1}'
echo "== top 3 =="
words | sort | uniq -c | sort -k1,1nr -k2,2 | head -n 3 | awk '{print $2, $1}'Interview follow-ups
Show only the top 5, and ignore common words like "the" and "and".
Put the common words in a file, one per line, for example
stop.txtwiththe,and,a,to. Addgrep -vxFf stop.txtright after the word list, beforesort.-f stop.txtreads the patterns from the file,-Ftreats them as plain text,-xmatches whole lines only, and-vremoves the matches. Then end the pipeline withhead -n 5.
Frequently asked questions
[:upper:] and [:lower:] are named classes that follow the locale, so on systems that support it they can also handle letters outside plain English. They also avoid a rare problem where A-Z covers more characters than you expect in some locales. For plain English text, both forms give the same result. In scripts that others will run on many systems, the named classes are the safer habit, along with [:alpha:] for letters and [:digit:] for digits.
The LeetCode problem gives words already separated by spaces, all lowercase, and promises no ties. So tr -s ' ' '\n' is enough to split, and a plain sort -rn is enough to rank. Real text has capitals, punctuation and ties, which is why this page adds lowercasing, tr -cs, the empty-line filter and a second sort key. Practise both: the LeetCode version for speed in an interview, this one for real files.
Yes: awk '{ $0 = tolower($0); gsub(/[^a-z]+/, " "); for (i = 1; i <= NF; i++) c[$i]++ } END { for (w in c) print c[w], w }' notes.txt | sort -k1,1nr -k2,2. It lowercases each line, turns non-letters into spaces, and counts every field in an array. It reads the file once and keeps one counter per distinct word, so it is faster on big files than sorting every word. The tr | sort | uniq version is easier to read and to change, which is why it is the one to learn first.