DevOps NetworkDevOpsNetwork
Join Free
Bash and Linux

Search a Whole Repo

easygrep and regex

Problem statement

Search every file in a folder tree at once with grep -r, limit the search to one kind of file, skip folders you do not care about, list which files match (or do not), and show lines around a match. You do this when you need to know "which manifests set replicas?", "where is this timeout configured?" or "which services forgot resource limits?".

The script builds this small repo:

repo/

Bash
k8s/web.yaml Deployment with replicas, resources and timeout
k8s/worker.yaml Deployment with replicas and timeout, no resources
docs/scaling.md mentions replicas in plain text
app/settings.py REQUEST_TIMEOUT = 15
.git/ORIG_HEAD an old copy that mentions replicas

k8s/web.yaml

TEXT
kind: Deployment
name: web
replicas: 3
resources:
limits:
memory: 512Mi
timeout: 30

k8s/worker.yaml

TEXT
kind: Deployment
name: worker
replicas: 1
timeout: 120

Print, in this order:

  1. The YAML files that set replicas, without looking inside .git.
  2. The YAML files that have no resources: block.
  3. Every line that mentions a timeout, in any case, with its file and line number.
  4. The memory: line in web.yaml with the 2 lines above it.

Expected output:

Bash
== YAML files that set replicas (skip .git) ==
./k8s/web.yaml
./k8s/worker.yaml
== YAML files with NO resources block ==
./k8s/worker.yaml
== every timeout, any case, with file and line ==
./app/settings.py:1:REQUEST_TIMEOUT = 15
./k8s/web.yaml:7:timeout: 30
./k8s/worker.yaml:4:timeout: 120
== the memory limit with 2 lines of context above it ==
4-resources:
5- limits:
6: memory: 512Mi

Hints

Hint 1: -r makes grep walk into every folder. --include='*.yaml' keeps only some files, and --exclude-dir=.git skips a folder.

Approach

Optimal: grep -r with filters

Covers: grep -r, --include, --exclude-dir, -l, -L, -n, -i, -A, -B, -C, reading file:line:text, sorting results.

-r searches a whole tree. grep -r 'replicas' . starts at . (this folder), goes into every folder below it, and searches every file it finds. Each match is printed as file:text, and with -n as file:line:text, so you always know where it came from.

Choosing what to search. A real repo has things you do not want: .git holds old copies of every file, node_modules can hold thousands of files, and docs mention words in plain English. Two flags fix that:

%%{init: {"flowchart": {"padding": 18, "nodeSpacing": 30, "rankSpacing": 40, "htmlLabels": true}, "themeVariables": {"fontSize": "18px"}}}%% flowchart TB R(["grep -r starts at ."]):::purple R --> K["k8s/"]:::blue R --> D["docs/"]:::gray R --> G[".git/"]:::red K --> W["web.yaml
searched"]:::green K --> WK["worker.yaml
searched"]:::green D --> S["scaling.md
not .yaml"]:::gray G --> O["skipped by
--exclude-dir"]:::red classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px
Flag Does Example
--include='*.yaml' only search files whose name matches YAML manifests only
--exclude='*.min.js' skip files whose name matches skip built files
--exclude-dir=.git do not go into folders with this name skip git history

Quote the patterns, like '*.yaml', so the shell passes the * to grep instead of expanding it itself. You can repeat a flag: --include='*.yaml' --include='*.yml'.

Choosing what to print.

Flag Prints
(none) file:matching line
-n file:line number:matching line
-l only the names of files with at least one match
-L only the names of files with no match
-c file:count for every file

-L is the hidden gem of audits. "Which manifests have no resource limits?" is really "which files do not contain resources:?", and grep -rL answers it in one line.

Seeing a match in its place. One matching line often makes no sense alone. In YAML, memory: 512Mi only means something once you see it sits under resources: and limits:. Context flags print neighbours:

Flag Adds
-B 2 2 lines before each match
-A 2 2 lines after each match
-C 2 2 lines on both sides

In the output, the matching line uses : after the line number, and context lines use -. When matches are far apart, grep prints -- between the groups.

%%{init: {"flowchart": {"padding": 18, "nodeSpacing": 30, "rankSpacing": 40, "htmlLabels": true}, "themeVariables": {"fontSize": "18px"}}}%% flowchart TB subgraph OUT["grep -n -B2 'memory:'"] direction TB L4["4-resources:
context"]:::yellow ~~~ L5["5- limits:
context"]:::yellow ~~~ L6["6: memory: 512Mi
the match"]:::green end classDef blue fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px classDef yellow fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px classDef green fill:#d1fae5,stroke:#059669,color:#064e3b,stroke-width:2px classDef red fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px classDef purple fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:2px classDef gray fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:2px linkStyle default stroke:#94a3b8,stroke-width:2px style OUT fill:transparent,stroke:#7c3aed,stroke-width:2px

Why | sort? grep visits folders in the order the disk returns them, which can change between machines. Sorting the result gives the same order every time, which matters for comparing outputs and for tests.

Walking through the code. The # Setup: lines only build the sample repo, so skip past them.

  1. grep -rl 'replicas' --include='*.yaml' --exclude-dir=.git . lists the two manifests. docs/scaling.md is not a YAML file, and .git is skipped.
  2. grep -rL 'resources:' ... lists the one manifest with no resources block.
  3. grep -rni 'timeout' --exclude-dir=.git . searches every file in any case, so it also finds REQUEST_TIMEOUT in the Python file.
  4. grep -n -B2 'memory:' k8s/web.yaml prints the match and the two lines above it.

Edge cases. With no matches, -l prints nothing and grep exits with 1. A folder you cannot read prints "Permission denied" on stderr; add 2>/dev/null to hide it. Binary files print "binary file matches" instead of the line; add -I to skip them.

# Setup: a small repo in a fresh temporary folder
cd "$(mktemp -d)"
mkdir -p k8s docs app .git
cat > k8s/web.yaml << 'F'
kind: Deployment
name: web
replicas: 3
resources:
  limits:
    memory: 512Mi
timeout: 30
F
cat > k8s/worker.yaml << 'F'
kind: Deployment
name: worker
replicas: 1
timeout: 120
F
cat > docs/scaling.md << 'F'
Raise replicas before a big sale.
F
cat > app/settings.py << 'F'
REQUEST_TIMEOUT = 15
F
echo 'replicas: old copy in git history' > .git/ORIG_HEAD

echo "== YAML files that set replicas (skip .git) =="
grep -rl 'replicas' --include='*.yaml' --exclude-dir=.git . | sort

echo "== YAML files with NO resources block =="
grep -rL 'resources:' --include='*.yaml' --exclude-dir=.git . | sort

echo "== every timeout, any case, with file and line =="
grep -rni 'timeout' --exclude-dir=.git . | sort

echo "== the memory limit with 2 lines of context above it =="
grep -n -B2 'memory:' k8s/web.yaml

Interview follow-ups

  • Only search the files changed in the last commit.

    Ask git for the list, then hand it to grep: git diff --name-only HEAD~1 | grep '\.yaml$' | xargs grep -n 'replicas'. The middle grep keeps only YAML names. Plain xargs splits names at spaces, so for safety use git diff -z --name-only HEAD~1 | xargs -0 grep -n 'replicas', where -z and -0 separate names with a null character instead. This is how CI checks often look only at what a pull request changed.

Frequently asked questions

Without quotes, the shell expands *.yaml into the names of matching files in the current folder before grep even starts. If there are none, it may pass the text through unchanged and it seems to work, which hides the bug until the day a .yaml file appears there. Always quote: --include='*.yaml'. Also note that --include takes a file name pattern, while the search pattern is a regular expression, so . and * mean different things in each.

Inside a git repo, git grep -n 'replicas' only searches files tracked by git, so it skips .git, build output and ignored files without extra flags, and it is very fast. rg (ripgrep) does the same using .gitignore, and it is faster still on big trees, but it is not installed on most servers. Learn grep -r first, because it works on every Linux box. Reach for the others on your own laptop.

For lines with either word, use grep -rE 'timeout|retries', where | means "or" in an extended pattern (-E). You can also give several -e options: grep -r -e timeout -e retries .. For lines with both words, chain two greps: grep -r 'timeout' . | grep 'web'. For files that contain both words anywhere, use grep -rl timeout . | xargs grep -l retries, keeping the spaces-in-names problem in mind.