Compare Two Files
Problem statement
Find out exactly what changed between two versions of a file with diff, and which items are in one list but not the other with comm. Engineers do this before and after every config change, when a server behaves differently from its twin, and when checking that an inventory matches what is really running.
app.conf.old
port=8080workers=4log_level=infotimeout=30app.conf.new
port=8080workers=8log_level=infotimeout=30cache=onhosts.old
api-1db-1web-1web-2hosts.new
api-1cache-1web-1web-2web-3Do these steps in order:
- Check whether the two configs are the same, using
diff's exit code, and print the code. - Show the changes as a unified diff, without the two header lines.
- Print the hosts only in the new list, then only in the old list, then in both.
Expected output:
configs are different (diff exit code 1)== diff -u ==@@ -1,4 +1,5 @@ port=8080-workers=4+workers=8 log_level=info timeout=30+cache=on== only in hosts.new ==cache-1web-3== only in hosts.old ==db-1== in both ==api-1web-1web-2Hints
diff -u old new marks removed lines with - and added lines with +. A changed line shows up as one of each.Approach
Optimal: diff and comm
Covers: diff, diff -q, diff -u, diff -r, exit codes, comm, comm -12, comm -13, comm -23.
Two tools, two jobs. diff compares two files line by line, in order, and tells you how to turn one into the other. It is the tool for configs, scripts and anything where line order matters. comm compares two sorted lists and tells you which items are only in one, only in the other, or in both. It is the tool for sets: hosts, users, packages.
Reading diff -u output. The -u flag gives the unified format, the same format git diff uses. A full diff starts like this:
--- app.conf.old (time of the old file)+++ app.conf.new (time of the new file)@@ -1,4 +1,5 @@| Line starts with | Means |
|---|---|
--- |
the old file |
+++ |
the new file |
@@ -1,4 +1,5 @@ |
this block covers lines 1 to 4 of the old file and 1 to 5 of the new |
| a space | the line is the same in both (context) |
- |
the line is only in the old file (removed) |
+ |
the line is only in the new file (added) |
So -workers=4 followed by +workers=8 means "this line changed", and +cache=on means "this line is new". The --- and +++ lines include file times, which differ on every machine, so the code skips them with tail -n +3 (start from line 3).
Using diff in an if. Every command returns an exit code when it ends. diff returns:
| Code | Means |
|---|---|
| 0 | the files are the same |
| 1 | the files are different |
| 2 | something went wrong, for example a missing file |
if runs its first branch when the command returns 0. So if diff -q a b > /dev/null reads as "if the files are the same". -q (quiet) only reports whether they differ, and > /dev/null hides even that message. Scripts use this to decide whether a config really changed before restarting a service.
How comm sees two lists.
comm hosts.old hosts.new prints three columns, one for each group above. You pick a group by hiding the other two:
| Command | Keeps |
|---|---|
comm -13 old new |
only in new (hide columns 1 and 3) |
comm -23 old new |
only in old (hide columns 2 and 3) |
comm -12 old new |
in both (hide columns 1 and 2) |
Both files must be sorted. comm walks the two files side by side, like merging two sorted piles of cards. If either file is not sorted, it gives wrong answers. The sample lists are already sorted. In real use, sort them first with sort old > old.sorted, or in bash with comm -13 <(sort old) <(sort new).
Walking through the code. The # Setup: lines only create the four sample files, so skip past them. Step 1 uses diff -q inside if, and prints $?, which still holds diff's exit code inside the else branch. Step 2 prints the unified diff from line 3 onward. Step 3 runs comm three times with different columns hidden.
Edge cases. If both files are the same, diff prints nothing and returns 0. If a file is missing, it prints an error and returns 2, so a script should treat 2 differently from 1. Trailing spaces count as a change, so two lines that look the same can still show up in a diff.
# Setup: an old and a new version of a config, and two host lists
cd "$(mktemp -d)"
cat > app.conf.old << 'CONF'
port=8080
workers=4
log_level=info
timeout=30
CONF
cat > app.conf.new << 'CONF'
port=8080
workers=8
log_level=info
timeout=30
cache=on
CONF
printf 'api-1\ndb-1\nweb-1\nweb-2\n' > hosts.old
printf 'api-1\ncache-1\nweb-1\nweb-2\nweb-3\n' > hosts.new
# 1. Are they the same? Look at the exit code: 0 = same, 1 = different
if diff -q app.conf.old app.conf.new > /dev/null; then
echo "configs are the same"
else
echo "configs are different (diff exit code $?)"
fi
# 2. Show what changed, in unified format
# tail -n +3 skips the two header lines, which hold file times
echo "== diff -u =="
diff -u app.conf.old app.conf.new | tail -n +3
# 3. comm compares two SORTED lists line by line
echo "== only in hosts.new =="
comm -13 hosts.old hosts.new
echo "== only in hosts.old =="
comm -23 hosts.old hosts.new
echo "== in both =="
comm -12 hosts.old hosts.newInterview follow-ups
How do you compare two whole folders, like the config folder on two servers?
diff -r dirA dirBwalks both folders and shows a diff for every file that differs. It also reports files that exist in only one of them.diff -rq dirA dirBgives just the list of differing and missing files, which is easier to read for big folders. To list only the missing files, filter that output withgrep '^Only in'. For folders on two different servers, copy one over first, or compare checksum lists: runfind . -type f -exec sha256sum {} + | sort -k2on each server anddiffthe two lists.
Frequently asked questions
Almost always because one of the files is not sorted, or was sorted in a different way. comm assumes both are sorted the same way and walks them in step, so an out-of-order line breaks its count. Sort both files right before comparing, in the same shell, so they use the same sort order: comm -13 <(sort old.txt) <(sort new.txt). GNU comm also prints a warning like "file 1 is not in sorted order", so read the warnings. Duplicate lines can also surprise you, so add -u to sort if each item should count once.
diff -w ignores all whitespace differences, and -b ignores changes in the amount of whitespace. -B ignores lines that are blank. These are useful when someone re-indented a file and you only care about real changes. For YAML or Python, be careful: there, indentation changes meaning, so a whitespace-only change can matter. diff -y (side by side) is another easy way to read small files.
diff compares text, line by line, and shows the changes. cmp compares bytes, stops at the first difference, and tells you where it is. cmp is the right tool for binary files like images or archives, where a line-by-line view means nothing. cmp -s a b prints nothing and only sets the exit code, which makes it a quick "are these files identical?" check in scripts. For checking a downloaded file, comparing checksums with sha256sum is the usual way.