Pick Columns with cut
Problem statement
Pull columns out of a file with cut, join lines with paste, and see when cut breaks and how to fix it. Inventories, CSV exports and command output are all columns of text, and "give me just the names and regions" is an everyday request.
hosts.csv
name,region,ip,roleweb-1,us-east,10.0.1.10,webweb-2,eu-west,10.0.2.10,webdb-1,us-east,10.0.1.20,dbcache-1,ap-south,10.0.3.30,cachehosts.txt (lined up with several spaces, like many command outputs)
web-1 10.0.1.10db-1 10.0.1.20Print, in this order:
- The name and region of every host, without the header line.
- Everything from the third column to the end.
- All host names on one line, joined by commas.
- The IP column of
hosts.txtwith a plaincut(it fails), then after squeezing the spaces.
Expected output:
== name and region, no header ==web-1,us-eastweb-2,eu-westdb-1,us-eastcache-1,ap-south== from column 3 to the end ==10.0.1.10,web10.0.2.10,web10.0.1.20,db10.0.3.30,cache== all names on one line ==web-1,web-2,db-1,cache-1== IP column with plain cut (empty, because of the extra spaces) == == IP column after squeezing spaces ==10.0.1.1010.0.1.20Hints
cut -d, -f1,2 splits each line at commas and keeps fields 1 and 2. tail -n +2 starts at line 2, which skips the header.Approach
Optimal: cut, paste and tr -s
Covers: cut -d -f, field lists and ranges (1,2, 3-), cut -c, tail -n +2, paste, paste -s -d, tr -s, when to use awk instead.
A line is a row, a separator makes columns. In a CSV each line is a record, and commas cut it into fields. cut numbers the fields from 1:
web-1"]:::blue L --> F2["-f2
us-east"]:::yellow L --> F3["-f3
10.0.1.10"]:::green L --> F4["-f4
web"]:::purple classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px
| You write | You get |
|---|---|
cut -d, -f1 |
field 1 only |
cut -d, -f1,3 |
fields 1 and 3 |
cut -d, -f2- |
field 2 to the end |
cut -d, -f-2 |
fields 1 and 2 |
cut -c1-5 |
characters 1 to 5 of each line (no separator needed) |
-d sets the separator (delimiter). Without it, cut uses a tab. The fields come out in file order, so -f3,1 still prints field 1 first.
Skipping the header. tail -n +2 hosts.csv prints from line 2 to the end. Feeding that into cut removes the name,region,... line, which you rarely want in results.
Joining lines with paste. paste -s (serial) joins all lines of its input into one line, and -d, puts a comma between them. The - means "read from the pipe". Without -s, paste a b puts two files side by side, line by line, which is handy for joining two lists.
Where cut breaks. cut treats every single separator as a column break. In web-1 10.0.1.10 there are four spaces, so cut -d' ' sees empty fields between them, and field 2 is empty. Two fixes:
- Squeeze the spaces first:
tr -s ' 'turns any run of spaces into one, thencut -d' ' -f2works. - Use
awk '{print $2}', which splits on runs of spaces and tabs by itself. For command output likepsordf, awk is the better tool.
Walking through the code. The # Setup: lines only create the two sample files, so skip past them.
tail -n +2 hosts.csv | cut -d, -f1,2printsname,regionfor each host.cut -d, -f3-prints the IP and role columns.tail -n +2 | cut -d, -f1 | paste -sd, -collects the names into one comma-separated line, a common way to build a list for another command.- The plain
cutprints empty lines, and thetr -s ' 'version prints the IPs.
Edge cases. A line with fewer fields than you ask for prints what it has; a line with no separator at all is printed whole unless you add -s. CSV files with quoted commas, like "Pune, India", cannot be split safely by cut; use a real CSV tool for those.
# Setup: create the two sample files in a fresh temporary folder
cd "$(mktemp -d)"
cat > hosts.csv << 'DATA'
name,region,ip,role
web-1,us-east,10.0.1.10,web
web-2,eu-west,10.0.2.10,web
db-1,us-east,10.0.1.20,db
cache-1,ap-south,10.0.3.30,cache
DATA
printf 'web-1 10.0.1.10\ndb-1 10.0.1.20\n' > hosts.txt
echo "== name and region, no header =="
tail -n +2 hosts.csv | cut -d, -f1,2
echo "== from column 3 to the end =="
tail -n +2 hosts.csv | cut -d, -f3-
echo "== all names on one line =="
tail -n +2 hosts.csv | cut -d, -f1 | paste -sd, -
echo "== IP column with plain cut (empty, because of the extra spaces) =="
cut -d' ' -f2 hosts.txt
echo "== IP column after squeezing spaces =="
tr -s ' ' < hosts.txt | cut -d' ' -f2Interview follow-ups
The columns are in a different order in every file. Pick the region column by its name.
Read the header to find the column number, then use it. With awk:
awk -F, 'NR == 1 { for (i = 1; i <= NF; i++) if ($i == "region") c = i; next } { print $c }' hosts.csv. The first line setscto the position ofregion, andnextskips printing the header. Every other line prints that column. This keeps working when someone adds or moves a column.
Frequently asked questions
Use cut for files with one clean separator, like CSV without quotes or /etc/passwd. Use awk when columns are lined up with several spaces or tabs, as in ps, df or ls -l output, because awk splits on runs of whitespace. awk can also filter rows ($3 > 80), reorder columns (print $3, $1) and do maths, which cut cannot. cut is slightly faster on huge files, but in practice both are fast.
A tab is cut's default separator, so cut -f2 already splits on tabs. To type one explicitly in bash, write cut -d$'\t' -f2. cut -d only accepts a single character, so for a separator like :: or , use awk: awk -F'::' '{print $2}'. awk's -F also accepts a pattern, like -F'[,;]' for commas or semicolons.
On many systems cut -c counts bytes, not letters. Letters like é or Hindi characters take more than one byte in UTF-8, so a byte range can cut a letter in half. GNU cut accepts -c but treats it like -b (bytes). For text with such characters, use awk's substr in a UTF-8 aware awk, or a scripting language.