Clean Up a Config
Problem statement
Clean a messy config so only the real settings are left: remove Windows line endings, comment lines (even indented ones) and blank lines, then print just one [section] of it. You do this to see what a config really sets, to compare two servers' configs, or to fix a file copied from Windows that breaks scripts.
config.ini (every line ends with a Windows \r)
# main config[server]port = 8080 # indented commenthost = 0.0.0.0 [database]user = app; old style commentpass = secret[cache]size = 64- Count the lines that end with a Windows
\r. - Print only the real settings and section headers: no
\r, no comments (#or;, indented or not), no blank lines. - Print only the
[database]section. - Save the cleaned file and count the
\rlines again.
Expected output:
== lines ending in a Windows \r ==12== real settings only ==[server]port = 8080host = 0.0.0.0[database]user = apppass = secret[cache]size = 64== only the [database] section ==[database]user = apppass = secret== lines ending in \r after cleaning ==0Hints
sed '/pattern/d' deletes every line that matches. ^[[:space:]]*# matches a line that starts with optional spaces and then #.Approach
Optimal: sed addresses: d, p and ranges
Covers: sed addresses (/pattern/, line numbers, $, ranges /a/,/b/), commands d and p, -n, -e, [[:space:]], s/\r$//, grep -c $'\r'.
Every sed command can have an address. An address picks which lines the command works on. Without one, the command runs on every line. With one, it only runs on lines that match:
d, p or s"]:::green A --> SKIP["leave the line
as it is"]:::gray classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px
| Address | Picks |
|---|---|
3 |
line 3 |
$ |
the last line |
/^#/ |
lines that start with # |
/^\[database\]/,/^\[/ |
from the [database] line to the next line starting with [ |
1! |
every line except line 1 (! flips the address) |
| Command | Does |
|---|---|
d |
delete the line (do not print it) |
p |
print the line |
s/a/b/ |
substitute, as on the previous page |
Deleting comments and blanks. [[:space:]] matches a space or a tab, and * means "any number of them, including none". So:
/^[[:space:]]*[#;]/ddeletes lines that start with optional spaces, then#or;. That catches# indented commenttoo./^[[:space:]]*$/ddeletes lines that are empty or only spaces.
The Windows \r problem. Windows ends each line with two characters, \r\n; Linux uses only \n. A file copied from Windows keeps an invisible \r at the end of every line. It breaks scripts (bash: $'\r': command not found), makes grep 'secret$' fail, and makes values like 8080\r invalid. s/\r$// removes it. grep -c $'\r' counts lines that contain one; the $'\r' form is how bash writes a real \r character.
Printing a range with -n and p. -n turns off sed's normal printing, so only lines you p (print) appear. /^\[database\]/,/^\[/p starts printing at [database] and stops after the next line that starts with [, which is [cache]. The brackets are escaped as \[ because [ is special in a pattern. The second sed '1!{/^\[/d}' deletes header lines on every line except the first, which removes the extra [cache].
range starts"]:::green ~~~ S2["user = app"]:::green ~~~ S3["pass = secret"]:::green ~~~ S4["[cache]
range ends, removed after"]:::red end classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px style R fill:transparent,stroke:#7c3aed,stroke-width:2px
Walking through the code. The # Setup: line uses printf with \r\n so every line really ends the Windows way, so skip past it.
grep -c $'\r'counts 12 lines with a\r.- The
cleanfunction runs three-ecommands: strip\r, delete comment lines, delete blank lines. - The range command prints the
[database]block. clean config.ini > clean.inisaves the result, and the count is now 0.
Edge cases. A comment after a value, like port = 8080 # main, is not removed by these commands; add s/[[:space:]]+#.*// with -E if your format allows that. If [database] is the last section, the range runs to the end of the file, which is still correct.
# Setup: write the sample config with Windows line endings (\r\n) in a fresh temporary folder
cd "$(mktemp -d)"
printf '%s\r\n' '# main config' '[server]' 'port = 8080' ' # indented comment' 'host = 0.0.0.0' '' \
'[database]' 'user = app' '; old style comment' 'pass = secret' '[cache]' 'size = 64' > config.ini
clean() {
sed -e 's/\r$//' \
-e '/^[[:space:]]*[#;]/d' \
-e '/^[[:space:]]*$/d' "$1"
}
echo "== lines ending in a Windows \\r =="
grep -c $'\r' config.ini
echo "== real settings only =="
clean config.ini
echo "== only the [database] section =="
clean config.ini | sed -n '/^\[database\]/,/^\[/p' | sed '1!{/^\[/d}'
clean config.ini > clean.ini
echo "== lines ending in \\r after cleaning =="
grep -c $'\r' clean.ini || trueInterview follow-ups
Also strip trailing spaces and print the settings sorted, to compare two servers.
Add one more command,
-e 's/[[:space:]]*$//', which removes spaces and tabs at the end of every line. Then sort:clean server1.ini | sort > a.txtand the same for server2.diff a.txt b.txtnow shows only real differences, not changes in comments, blank lines or order. This clean-sort-diff trick is a quick way to find config drift between machines.
Frequently asked questions
cat -A file shows them: a \r appears as ^M, a tab as ^I, and the end of each line as $. file config.ini also tells you "with CRLF line terminators" when a file has Windows endings. Many editors can show them too. Once you know they are there, sed -i 's/\r$//' file or dos2unix file removes them.
Plain sed prints every line unless you delete it with d, so you say what to throw away. sed -n prints nothing unless you print it with p, so you say what to keep. Use d when most lines stay, like removing comments. Use -n ... p when you want a small part, like one section or a line range. Mixing p without -n prints matching lines twice, which is a common surprise.
For comments and blank lines, grep -v -E '^[[:space:]]*([#;]|$)' does the same job in one step, and that is fine. sed earns its place when you also need to change lines, like removing the \r, and when you want ranges such as "from [database] to the next header", which grep cannot do. Knowing both lets you pick the shortest clear command.