Pipe

Linux hub

A pipe (|) connects the standard output of one command to the standard input of the next, creating a data processing chain. Pipes let you compose simple commands into powerful one-liners without temporary files. The kernel implements pipes as in-memory buffers between processes.

Understanding Linux Pipes

What Is a Pipe in Simple Terms

A pipe is a conveyor belt between commands. Instead of saving a command's output to a file and feeding that file to the next command, a pipe connects them directly — output from the left flows immediately into the input on the right.

ps aux | grep nginx | awk '{print $2}' — three commands chained together. ps produces output, grep filters it, awk extracts just the PID column. No temporary files, no intermediate steps.

How It Works

◈ DIAGRAM

+------------------+     +------------------+     +------------------+
| ps aux           |     | grep nginx       |     | awk '{print $2}' |
|                  |     |                  |     |                  |
| stdout ------->  | --> | stdin  stdout -> | --> | stdin  stdout -> | --> terminal
|                  |     |                  |     |                  |
+------------------+     +------------------+     +------------------+
 
All three run simultaneously in parallel.
The kernel buffer (65536 bytes) between each pair
backs up the producer if the consumer is slow.

Practical Commands

Bash

## Basic pipe
ps aux | grep nginx
 
## Chain multiple pipes
cat /var/log/nginx/access.log | grep '200' | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
## Finds top 10 IPs making successful requests
 
## Count lines matching a pattern
grep 'ERROR' /var/log/app.log | wc -l
 
## Find most common errors
grep 'ERROR' /var/log/app.log | sort | uniq -c | sort -rn | head -5
 
## Process JSON output
curl -s https://api.internal/health | jq '.services[] | select(.status == "down")'
 
## Named pipes (FIFOs) -- persist on filesystem
mkfifo /tmp/mypipe
echo "data" > /tmp/mypipe &   ## write in background
cat /tmp/mypipe               ## read from it
rm /tmp/mypipe
 
## Check exit code of specific command in pipeline
## $PIPESTATUS array holds each command's exit code
cat file.txt | grep pattern | wc -l
echo ${PIPESTATUS[@]}
## 0 0 0 -- all succeeded
## 0 1 0 -- grep found no matches (exit 1)
 
## With set -o pipefail, pipeline fails if any command fails
set -o pipefail
cat missing-file.txt | grep pattern
## Error: cat returns 1, whole pipeline returns 1

Troubleshooting

Symptom	Command	What to Check
Pipeline always succeeds	`echo ${PIPESTATUS[@]}`	Last command exit code hides earlier failures
Data lost in pipeline	Check buffer size	Producer much faster than consumer
Cannot pipe to sudo	`echo cmd	sudo bash`

Tip
Use tee to send output to both a file and the next pipe simultaneously: command | tee output.log | grep ERROR. The file gets everything, grep gets everything, and you lose nothing.

Remember
In bash, set -o pipefail makes the entire pipeline return the exit code of the rightmost failed command. Without it, false | true returns 0 (success) because only the last command's exit code counts. Always use set -o pipefail in production scripts.

Frequently Asked Questions

Why are pipes faster than writing to a temp file between commands?

A pipe is a kernel-managed ring buffer (typically 64KB on modern Linux) that holds data in memory between two processes' file descriptors — there's no disk I/O, no filesystem metadata, and the reading process can start consuming output before the writing process finishes. This lets `producer | consumer` run concurrently as two processes scheduled in parallel, rather than serially writing then re-reading a file, which is both slower and leaves cleanup work behind.

Why does a pipeline sometimes report success even when an early command failed?

By default, a shell pipeline's exit status is the exit status of the last command only, so `false | true` reports success even though `false` failed. This silently masks failures in scripts, especially with `grep` (which exits non-zero when it finds nothing) buried mid-pipeline. Use `set -o pipefail` in bash scripts so the pipeline's exit code reflects any failing stage, not just the last one.