Overview and What You Will Learn
Production servers always have scheduled work. Database backups run nightly. Certificate renewal runs weekly. Log rotation runs daily. Cache warming runs before peak hours. Health checks run every minute. Cron and systemd timers are the two mechanisms that drive all of this.
By the end of this lab you will:
- Write correct crontab entries using all five time fields
- Debug the most common cron failure (PATH mismatch)
- Capture cron job output reliably to log files
- Create systemd timer units as the modern alternative to cron
- Build production-grade patterns for backup, log parsing, and health checks
Why This Matters in Production
A cron job that works when run manually but silently fails in cron is one of the most frustrating debugging scenarios in Linux. The cause is almost always the same: the cron execution environment has a minimal PATH that does not include the directories where your tools are installed. Understanding this saves hours of debugging.
At PhonePe, database backups run as cron jobs every night at 1 AM. If those backups silently fail because of a PATH issue, nobody knows until the day they need to restore — and the backup is not there.
Core Principles
Crontab time field syntax:
+------------------------------------------+| minute hour day month weekday || 0-59 0-23 1-31 1-12 0-7 |+------------------------------------------+| || * = every value (any) || , = list of values (1,15 = 1st,15th) || - = range (1-5 = Mon through Fri) || / = step (*/15 = every 15 minutes) |+------------------------------------------+ Examples: 0 2 * * * Every day at 02:00 */15 * * * * Every 15 minutes 0 9 * * 1-5 Weekdays at 09:00 0 0 1 * * First day of each month 30 23 * * 0 Sundays at 23:30Cron vs systemd timers:
+------------------------+ +------------------------+| cron | | systemd timer || | | || Simple syntax | | More powerful syntax || No dependency support | | After= dependencies || Output to email | | Output to journald || Missed jobs lost | | Persistent= catches up || No status command | | systemctl status timer |+------------------------+ +------------------------+ Use for simple Use for complex scheduled tasks production servicesDetailed Step-by-Step Practical Lab
Milestone 1 — Create and manage crontab entries
## Open your personal crontab in editorcrontab -e ## List current crontabcrontab -l ## Remove all crontab entries (DANGEROUS -- no confirmation)## crontab -r ## View root's crontabsudo crontab -l ## Edit root's crontabsudo crontab -e## Example crontab entries:## m h dom mon dow command ## Database backup at 1 AM every night0 1 * * * /opt/scripts/backup-postgres.sh >> /var/log/backup.log 2>&1 ## Health check every 5 minutes*/5 * * * * /opt/scripts/health-check.sh ## Log cleanup every Sunday at 3 AM0 3 * * 0 find /var/log/app -name "*.log" -mtime +30 -delete ## Cache warming every morning at 8:45 AM on weekdays45 8 * * 1-5 /opt/scripts/warm-cache.sh >> /var/log/cache-warm.log 2>&1 ## First day of each month -- rotate archived logs0 0 1 * * /opt/scripts/archive-old-logs.sh ## Every 15 minutes -- metrics collection*/15 * * * * /opt/scripts/collect-metrics.sh > /dev/null 2>&1Milestone 2 — Fix the most common cron failure: PATH
## Test this: does your script work in cron's environment? ## Cron's default PATH is minimal:## /usr/bin:/bin ## Your interactive shell's PATH is much richer:echo $PATH## /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ## Tools often installed in /usr/local/bin:which node ## /usr/local/bin/nodewhich python3 ## /usr/bin/python3which aws ## /usr/local/bin/aws ## The fix: always use absolute paths in cron scripts## BAD:## node /opt/app/index.js ## GOOD:## /usr/local/bin/node /opt/app/index.js ## Or set PATH at the top of your crontab:## PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ## Or set PATH inside the script itself:cat /opt/scripts/backup-postgres.sh## Always set explicit PATH at top of cron scriptsPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ## Always use strict modeset -euo pipefail ## Use absolute paths for everythingPGDUMP=/usr/bin/pg_dumpBACKUP_DIR=/opt/backupsDATE=$(date +%Y%m%d-%H%M%S) $PGDUMP -U postgres mydb > "$BACKUP_DIR/mydb-$DATE.sql"echo "Backup completed: $BACKUP_DIR/mydb-$DATE.sql"Milestone 3 — Capture cron output reliably
## By default cron emails output to the local user## On a server with no mail configured, output is lost silently ## Option 1: Redirect both stdout and stderr to log file0 1 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1 ## Option 2: Redirect to separate files0 1 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>> /var/log/backup-errors.log ## Option 3: Discard all output (only for idempotent scripts you trust)*/5 * * * * /opt/scripts/metrics.sh > /dev/null 2>&1 ## Option 4: Use logger to send to syslog/journald0 1 * * * /opt/scripts/backup.sh 2>&1 | logger -t backup-cron ## Then view with:journalctl -t backup-cron ## Disable cron email for the whole crontabMAILTO="" ## Or send to a specific addressMAILTO=ops@devops-network.inMilestone 4 — System-wide cron locations
## System cron directories (no crontab -e needed, just drop files)ls /etc/cron.d/ ## arbitrary schedule filesls /etc/cron.daily/ ## runs once dailyls /etc/cron.hourly/ ## runs once hourlyls /etc/cron.weekly/ ## runs once weeklyls /etc/cron.monthly/ ## runs once monthly ## Add a system-wide cron jobsudo tee /etc/cron.d/payment-health << 'EOF'SHELL=/bin/bashPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binMAILTO="" ## Run health check every 5 minutes as payment-svc user*/5 * * * * payment-svc /opt/payment/scripts/health-check.sh >> /var/log/payment-health.log 2>&1EOF ## Verify cron service is runningsystemctl status cron ## Debian/Ubuntusystemctl status crond ## RHEL/Amazon LinuxMilestone 5 — Create a systemd timer as modern alternative
## Step 1: Create the service unit (what runs)sudo tee /etc/systemd/system/db-backup.service << 'EOF'[Unit]Description=PostgreSQL Database BackupAfter=postgresql.service [Service]Type=oneshotUser=backup-svcExecStart=/opt/scripts/backup-postgres.shStandardOutput=journalStandardError=journalEOF ## Step 2: Create the timer unit (when it runs)sudo tee /etc/systemd/system/db-backup.timer << 'EOF'[Unit]Description=Daily PostgreSQL backup timerRequires=db-backup.service [Timer]## Run daily at 01:00OnCalendar=*-*-* 01:00:00 ## If server was off at 01:00, run at next bootPersistent=true [Install]WantedBy=timers.targetEOF ## Step 3: Enable and start the timersudo systemctl daemon-reloadsudo systemctl enable --now db-backup.timer ## Step 4: Verify timer is scheduledsystemctl list-timers db-backup.timer## NEXT LEFT UNIT ACTIVATES## Mon 2024-01-16 01:00:00 UTC 12h left db-backup.timer db-backup.service ## Step 5: Run immediately to testsudo systemctl start db-backup.service ## Step 6: Check resultsjournalctl -u db-backup.service -n 20Milestone 6 — Production cron patterns
## Pattern 1: Backup with retentioncat > /opt/scripts/backup-postgres.sh << 'SCRIPT'#!/usr/bin/env bashset -euo pipefailPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin BACKUP_DIR=/opt/backups/postgresRETENTION_DAYS=7DATE=$(date +%Y%m%d-%H%M%S)BACKUP_FILE="$BACKUP_DIR/mydb-$DATE.sql.gz" mkdir -p "$BACKUP_DIR" echo "Starting backup at $(date)"/usr/bin/pg_dump -U postgres mydb | gzip > "$BACKUP_FILE"echo "Backup created: $BACKUP_FILE ($(du -sh $BACKUP_FILE | cut -f1))" ## Delete backups older than RETENTION_DAYSfind "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -deleteecho "Cleanup done: removed files older than $RETENTION_DAYS days"echo "Backup completed at $(date)"SCRIPT chmod +x /opt/scripts/backup-postgres.sh ## Add to crontab## 0 1 * * * /opt/scripts/backup-postgres.sh >> /var/log/backup.log 2>&1Production Best Practices and Common Pitfalls
| Mistake | Problem | Fix |
|---|---|---|
| No absolute paths in script | Works manually, fails in cron | Use full paths or set PATH at top of script |
| No output redirection | Silent failures, no debugging | Always >> logfile 2>&1 |
crontab -r typo |
Deletes all cron jobs | Use crontab -e to edit, never -r |
| Script not executable | Cron exits silently | chmod +x /opt/scripts/myscript.sh |
| Cron job overlaps itself | Two instances run simultaneously | Use flock to prevent concurrent execution |
Quick Reference and Troubleshooting Commands
| Task | Command |
|---|---|
| Edit crontab | crontab -e |
| List crontab | crontab -l |
| List timers | systemctl list-timers |
| Run timer now | sudo systemctl start name.service |
| Check timer logs | journalctl -u name.service |
| Debug PATH issues | Add env > /tmp/cron-env.txt to crontab |
| Check cron daemon | systemctl status cron |
| Validate cron syntax | https://crontab.guru |
TipAdd
* * * * * env > /tmp/cron-environment.txtto your crontab temporarily to see exactly what environment cron uses. This immediately reveals PATH differences and explains why commands work manually but fail in cron.
RememberThe cron execution environment has a minimal PATH (
/usr/bin:/bin). Commands that work in your interactive shell may silently fail in cron because the binary is not in cron's PATH. Always use absolute paths or setPATH=at the top of every cron script.
SecurityCron jobs run with the permissions of the user whose crontab contains them. A root crontab entry that executes a world-writable script is a privilege escalation vulnerability. Always ensure cron scripts are owned by root and have
chmod 750or stricter permissions.
Common MistakeWriting
crontab -rwhen you meantcrontab -e. The-rflag removes all cron entries with no confirmation and no undo. Always use-eto edit and-lto list. If you accidentally run-r, immediately check if you have a backup in~/crontab.bakor restore from your configuration management system.
