DevOps NetworkDevOpsNetwork
Join Free
Bash and Linux

Handle Command Failures

mediumBash scripts and automation

Problem statement

Run a three-step deploy where the second step fails, report which step failed and its exit code, and stop with that code. Along the way, see how exit codes work in pipelines, what PIPESTATUS shows, and two places where set -e does not stop a script. A script that hides a failure is worse than no script, because everyone trusts its "done".

The script uses three fake steps:

steps

TEXT
fetch succeeds
build fails with exit code 3
deploy would succeed, but must not run after a failed build
  1. Show the exit code of a pipeline with and without pipefail, and every code with PIPESTATUS.
  2. Show the codes for "command not found" and "not executable".
  3. Show that local out=$(false) hides the failure, and how to catch it.
  4. Run the steps, report the failure, skip deploy, and print the exit code the script would end with.

Expected output:

TEXT
== pipelines ==
without pipefail: exit 0
with pipefail: exit 1
PIPESTATUS: 1 0 2
== common codes ==
command not found: 127
not executable: 126
== local hides the failure ==
local out=$(false): exit 0
local out; out=$(false): exit 1
== deploy ==
step fetch
fetching source
ok
step build
compiling
FAILED: build exited with 3
stopped early; the script would end with: exit 3

Hints

Hint 1: $? is the exit code of the last command. In a pipeline, it is the last command's code unless set -o pipefail is on. PIPESTATUS is an array with every command's code.

Approach

Optimal: Exit codes, PIPESTATUS and run_step

Covers: exit codes, $?, if cmd, cmd || handle, pipelines and pipefail, PIPESTATUS, codes 1, 2, 126, 127, 130, 137, where set -e does not stop, local x=$(cmd).

Strict mode, used on every page in this section. The second line, set -euo pipefail, makes bash stop on mistakes instead of carrying on:

Option Means
-e exit as soon as a command fails (with some exceptions, see Handle Command Failures)
-u treat an unset variable as an error, instead of silently using empty text
-o pipefail a pipeline fails if any command in it fails, not just the last one

Put it right after the shebang line #!/usr/bin/env bash in every script you write.

Every command returns a number. 0 means success; anything from 1 to 255 means some kind of failure. $? holds the code of the last command. Common ones:

Code Usually means
0 success
1 general failure
2 wrong usage, bad arguments
126 found, but not executable
127 command not found
130 stopped with Ctrl+C (128 + 2)
137 killed with KILL (128 + 9)

Pipelines hide failures by default. A pipeline's code is the code of its last command. So false | true "succeeds", and a curl | tar where curl failed looks fine. set -o pipefail makes the pipeline fail if any part fails. To see every part, read PIPESTATUS right after the pipeline, before any other command overwrites it:

%%{init: {"flowchart": {"padding": 18, "nodeSpacing": 30, "rankSpacing": 40, "htmlLabels": true}, "themeVariables": {"fontSize": "18px"}}}%% flowchart LR subgraph P["false | true | (exit 2)"] direction TB P1["false
1"]:::red ~~~ P2["true
0"]:::green ~~~ P3["exit 2
2"]:::red end P --> S["PIPESTATUS = 1 0 2"]:::yellow classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px style P fill:transparent,stroke:#7c3aed,stroke-width:2px

Where set -e does not stop the script. set -e is helpful, but it is switched off in a few places, and people rely on it there by mistake:

  • in a condition: if cmd, while cmd, and on the left of && or ||. Inside a function called that way, too.
  • in local out=$(cmd): the code you see is the code of local, which is always 0. Declare first, then assign: local out; out=$(cmd).

So for important steps, check the code yourself instead of trusting set -e.

A step runner. run_step name command... runs the command inside if. On success it prints ok; on failure it captures $?, prints the step name and code, and returns that code. run_all chains the steps with || return, so the first failure stops the rest and passes its code upward.

%%{init: {"flowchart": {"padding": 18, "nodeSpacing": 30, "rankSpacing": 40, "htmlLabels": true}, "themeVariables": {"fontSize": "18px"}}}%% flowchart TB F["fetch
ok"]:::green --> B["build
exit 3"]:::red B --> D["deploy
skipped"]:::gray B --> E(["script exits with 3"]):::red classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px

Walking through the code. The # Setup: lines only define the three fake steps, so skip past them.

  1. set +o pipefail in a subshell shows the default; then the same pipeline with pipefail fails. set +e around the PIPESTATUS line lets the failing pipeline finish so its codes can be printed.
  2. no-such-command gives 127; a file without the run permission gives 126.
  3. local out=$(false) reports 0; the split form reports the real 1.
  4. run_all || code=$? runs the chain. build fails with 3, deploy never runs, and the script would end with exit "$code".

Edge cases. A function's exit code is that of its last command unless it uses return N. Exit codes above 255 wrap around (exit 256 is 0), so keep them small. ! in front of a pipeline flips its result, and set -e ignores it too.

#!/usr/bin/env bash
set -euo pipefail

# Setup: three fake deploy steps in a fresh temporary folder
cd "$(mktemp -d)"
step_fetch()  { echo "    fetching source"; }
step_build()  { echo "    compiling"; return 3; }
step_deploy() { echo "    deploying"; }

echo "== pipelines =="
( set +o pipefail; false | true; echo "without pipefail: exit $?" )
false | true || echo "with pipefail:    exit $?"
set +e
false | true | (exit 2)
codes=("${PIPESTATUS[@]}")
set -e
echo "PIPESTATUS: ${codes[*]}"

echo "== common codes =="
no-such-command 2>/dev/null || echo "command not found: $?"
echo 'echo hi' > notexec.sh
./notexec.sh 2>/dev/null || echo "not executable:    $?"

echo "== local hides the failure =="
f() { local out=$(false); echo "local out=\$(false):       exit $?"; }
f
g() { local out; out=$(false) || echo "local out; out=\$(false): exit $?"; }
g

run_step() {
  local name=$1; shift
  echo "step $name"
  if "$@"; then
    echo "  ok"
  else
    local code=$?
    echo "  FAILED: $name exited with $code"
    return "$code"
  fi
}
run_all() {
  run_step fetch  step_fetch  || return
  run_step build  step_build  || return
  run_step deploy step_deploy || return
}

echo "== deploy =="
code=0
run_all || code=$?
echo "stopped early; the script would end with: exit $code"

Interview follow-ups

  • Always print a summary at the end, even when a step fails and the script stops.

    Record each step's result in a variable or array as it runs, and print them from an EXIT trap: trap 'echo "summary: ${results[*]:-none}"' EXIT. The trap runs whether the script ends normally, with exit 3, or because set -e stopped it. The next pages show trap in detail, including cleaning up temp files the same way.

Frequently asked questions

Yes, as a safety net: it catches the many small commands you would never check by hand. But do not rely on it for the important steps, because of the exceptions on this page: conditions, && and || chains, functions called from them, and local x=$(cmd). For those, check the code explicitly with if or || { handle; }. Many teams use both, plus shellcheck, which warns about several of these traps.

Add || true to the command: rm -f old.tmp || true or grep -c error app.log || true. That tells readers you expected it to fail sometimes and it is fine. For grep, remember that exit code 1 means "no match" and 2 means a real error, so grep ... || [[ $? -eq 1 ]] ignores only the harmless case. Never wrap a whole script in set +e to get past a failure.

(( n++ )) returns the old value, and (( 0 )) counts as false, which is exit code 1, so set -e stops the script. It is a well-known surprise. Write n=$((n + 1)) or (( n += 1 )) instead, which never evaluate to 0 when counting up from 0. The same applies to let.