Overview and What You Will Learn
A default Linux server installation is not production-ready from a security perspective. Root SSH is enabled. Password authentication is allowed. Unnecessary packages are installed. No rate limiting exists on login attempts. Kernel parameters use defaults that do not reflect production security requirements.
By the end of this lab you will:
- Apply SSH hardening that eliminates the most common attack vectors
- Configure user accounts following least-privilege principles
- Remove unnecessary packages and services to reduce the attack surface
- Apply kernel security parameters via sysctl
- Deploy fail2ban to automatically block brute-force attempts
- Run lynis to get an automated security score and remediation list
Why This Matters in Production
Zerodha operates under SEBI regulations. Every server handling trading data must meet security baselines. An unpatched server with default configurations is not just a security risk — it is a compliance failure that can result in regulatory action. The checklist in this topic is the starting point for any server that touches production workloads or regulated data.
Core Principles
Defence in depth — multiple layers:
+------------------------------------------+| Layer 1: Network (firewall, VPC, SG) || Block unwanted traffic before it arrives |+------------------------------------------+| Layer 2: SSH (key auth, no root, MFA) || Secure the entry point |+------------------------------------------+| Layer 3: Users (least privilege, audit) || Limit what authenticated users can do |+------------------------------------------+| Layer 4: OS (packages, kernel, sysctl) || Minimise the attack surface |+------------------------------------------+| Layer 5: Detection (fail2ban, auditd) || Detect and respond to attacks in flight |+------------------------------------------+| Layer 6: Compliance (lynis, CIS) || Measure and track security posture |+------------------------------------------+Detailed Step-by-Step Practical Lab
Milestone 1 — SSH hardening
SSH hardening is the highest-impact single action for server security. Disabling password authentication eliminates brute-force attacks entirely.
## Backup the original config firstsudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.original ## Apply production SSH hardeningsudo tee /etc/ssh/sshd_config.d/hardening.conf << 'EOF'## Disable root login completelyPermitRootLogin no ## Disable password authentication -- keys onlyPasswordAuthentication noChallengeResponseAuthentication noKbdInteractiveAuthentication no ## Disable PAM if not needed (reduces attack surface)## UsePAM yes <- keep yes if using PAM-based 2FA ## Allow only specific usersAllowUsers rahul deploy jenkins monitoring-agent ## Limit authentication attemptsMaxAuthTries 3MaxSessions 10LoginGraceTime 30 ## Disconnect idle sessionsClientAliveInterval 300ClientAliveCountMax 2 ## Disable X11 forwarding (servers do not need it)X11Forwarding no ## Disable agent forwarding by default (enable per-user in Match blocks if needed)AllowAgentForwarding no ## Use only modern key exchange algorithmsKexAlgorithms curve25519-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512 ## Restrict ciphers to strong ones onlyCiphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com ## Verbose logging for audit trailLogLevel VERBOSE ## Disable TCP forwarding for regular users (enable in Match blocks if needed)AllowTcpForwarding noEOF ## Validate config syntax before applyingsudo sshd -t## No output = valid ## Apply changessudo systemctl reload sshd ## IMPORTANT: Test from a SECOND terminal before closing this sessionssh rahul@$(hostname -I | awk '{print $1}')## If this works, your hardening is correctMilestone 2 — User and sudo hygiene
## Audit all accounts that can log ingrep -v '/nologin\|/false' /etc/passwd | cut -d: -f1,7## Only accounts with real shells should be here## Remove any unexpected accounts ## Audit accounts with sudo accesssudo grep -r 'ALL' /etc/sudoers /etc/sudoers.d/ 2>/dev/null ## Remove NOPASSWD: ALL from any user that does not absolutely require it## Jenkins CI runner might need it -- but scope it to specific commandssudo visudo -f /etc/sudoers.d/jenkins## jenkins ALL=(ALL) NOPASSWD: /bin/systemctl restart payment-api, /bin/systemctl reload nginx ## Lock accounts that should never log in## (service accounts, old employee accounts)sudo passwd --lock old-employee-accountsudo usermod --shell /sbin/nologin old-employee-account ## Find accounts with no password set (critical security issue)sudo awk -F: '($2 == "" || $2 == "!") {print $1}' /etc/shadow ## Check password expiry policysudo chage -l rahul## Minimum: 0 days, Maximum: 99999 days, Warning: 7 days ## Set password expiry for accounts using passwordssudo chage -M 90 -W 14 rahul ## expire every 90 days, warn 14 days before ## Review last loginslast | head -20lastlog | grep -v 'Never' ## users who have actually logged inMilestone 3 — Package and service minimisation
## List all installed packagesdpkg -l | wc -l ## Debian/Ubunturpm -qa | wc -l ## RHEL/Amazon Linux ## Find and remove services not needed on this server## Common unnecessary services on cloud servers:sudo systemctl list-units --type=service --state=running ## Disable and remove common unnecessary packagessudo apt remove --purge telnet ## use SSH insteadsudo apt remove --purge rsh-client ## old, insecure remote shellsudo apt remove --purge ftp ## use sftp insteadsudo apt remove --purge sendmail postfix ## if not a mail serversudo apt autoremove ## Disable unnecessary servicessudo systemctl disable --now avahi-daemon ## mDNS -- not needed on serverssudo systemctl disable --now cups ## printing -- not needed on servers ## List all listening services and evaluate each oness -tulpn## For every port listed: do we need this exposed?## If not: stop the service or bind it to localhost ## Bind services to localhost where external access is not needed## Example: PostgreSQL should only accept local connectionsgrep listen_addresses /etc/postgresql/*/main/postgresql.conf## Change to: listen_addresses = 'localhost'Milestone 4 — Kernel hardening with sysctl
sysctl parameters control kernel behaviour. These settings harden the network stack and restrict information leakage.
## Create a production hardening sysctl configsudo tee /etc/sysctl.d/99-hardening.conf << 'EOF'## -- Network Security --------------------------------------## Disable IP forwarding (enable only on routers/VPN servers)net.ipv4.ip_forward = 0net.ipv6.conf.all.forwarding = 0 ## Protect against SYN flood attacksnet.ipv4.tcp_syncookies = 1 ## Ignore ICMP redirects (prevent route manipulation)net.ipv4.conf.all.accept_redirects = 0net.ipv4.conf.all.send_redirects = 0net.ipv6.conf.all.accept_redirects = 0 ## Ignore broadcast pings (smurf attack prevention)net.ipv4.icmp_echo_ignore_broadcasts = 1 ## Log suspicious packets (spoofed, source-routed, redirects)net.ipv4.conf.all.log_martians = 1 ## Disable source routingnet.ipv4.conf.all.accept_source_route = 0 ## -- Kernel Security ---------------------------------------## Restrict access to kernel logskernel.dmesg_restrict = 1 ## Restrict kernel pointer exposurekernel.kptr_restrict = 2 ## Disable magic SysRq key (can be used to crash server)kernel.sysrq = 0 ## Restrict ptrace to parent processes onlykernel.yama.ptrace_scope = 1 ## -- Memory Security ---------------------------------------## Randomise memory layout (ASLR -- makes exploits harder)kernel.randomize_va_space = 2EOF ## Apply immediatelysudo sysctl -p /etc/sysctl.d/99-hardening.conf ## Verify settings took effectsysctl net.ipv4.tcp_syncookies## net.ipv4.tcp_syncookies = 1Milestone 5 — Deploy fail2ban for brute-force protection
fail2ban monitors log files and automatically bans IPs that show signs of brute-force attacks.
## Install fail2bansudo apt install fail2ban ## Create local configuration (never edit main jail.conf)sudo tee /etc/fail2ban/jail.local << 'EOF'[DEFAULT]## Ban for 1 hour after 5 failures in 10 minutesbantime = 3600findtime = 600maxretry = 5 ## Use iptables for banningbanaction = iptables-multiport ## Email alerts (if mail server configured)destemail = ops@devops-network.insendername = fail2banaction = %(action_mwl)s [sshd]enabled = trueport = sshlogpath = %(sshd_log)smaxretry = 3bantime = 86400 ## 24 hours for SSH failures (stricter) [nginx-http-auth]enabled = trueport = http,httpslogpath = /var/log/nginx/error.log [nginx-limit-req]enabled = trueport = http,httpslogpath = /var/log/nginx/error.logEOF ## Enable and start fail2bansudo systemctl enable --now fail2ban ## Check statussudo fail2ban-client status## Status## |- Number of jail: 2## `- Jail list: sshd, nginx-http-auth ## Check SSH jail specificallysudo fail2ban-client status sshd## Status for the jail: sshd## |- Filter## | `- Number of recently found: 3## `- Actions## |- Currently banned: 1## `- Banned IP list: 192.0.2.45 ## Manually unban an IPsudo fail2ban-client set sshd unbanip 192.0.2.45Milestone 6 — Automated compliance scan with lynis
lynis audits the server against security best practices and produces a prioritised list of improvements.
## Install lynissudo apt install lynis## Or install from source for latest version:## git clone https://github.com/CISOfy/lynis## cd lynis && sudo ./lynis audit system ## Run full system auditsudo lynis audit system ## Key output sections:## [+] System tools <- checks installed tools## [+] Boot and services <- checks running services## [+] SSH <- SSH configuration score## [+] Users, groups <- account security## [+] Networking <- network configuration ## Hardening index at the end:## Hardening index : 67 [############# ]## Tests performed: 257## Plugins enabled: 0#### Components:## - Firewall [V]## - Malware scanner [X] <- not installed## - File integrity tool [X] <- not installed ## View all warnings foundsudo lynis audit system 2>&1 | grep Warning ## View all suggestions (prioritised list of improvements)sudo lynis audit system 2>&1 | grep Suggestion | head -20 ## Save report for compliance documentationsudo lynis audit system --report-file /var/log/lynis-report.datsudo lynis show details /var/log/lynis-report.datProduction Best Practices and Common Pitfalls
| Area | Wrong | Correct |
|---|---|---|
| SSH | Root login enabled, password auth on | PermitRootLogin no, PasswordAuthentication no |
| Users | Shared accounts, NOPASSWD: ALL | Named accounts, scoped sudo commands |
| Packages | Never remove anything | Audit and remove unused services quarterly |
| Kernel | Default sysctl settings | Apply hardening.conf with network protections |
| Monitoring | No failed login visibility | fail2ban + auth.log monitoring |
Quick Reference and Troubleshooting Commands
| Task | Command |
|---|---|
| Validate sshd config | sudo sshd -t |
| Check who has sudo | sudo grep -r ALL /etc/sudoers.d/ |
| List listening services | ss -tulpn |
| Apply sysctl | sudo sysctl -p /etc/sysctl.d/99-hardening.conf |
| fail2ban status | sudo fail2ban-client status sshd |
| Unban an IP | sudo fail2ban-client set sshd unbanip IP |
| Run lynis audit | sudo lynis audit system |
| Check auth logs | `grep 'Failed|Invalid' /var/log/auth.log |
TipRun
sudo lynis audit systemon every new server before it goes into production. The hardening index score and suggestion list give you a concrete, prioritised remediation plan. Aim for a hardening index above 80 for production servers handling sensitive data.
RememberSecurity hardening is not a one-time task. New CVEs are discovered regularly. Enable unattended security upgrades on every server, schedule quarterly lynis scans, and review sudo access lists whenever team members join or leave.
SecurityNever disable
UsePAM yesin sshd_config unless you are absolutely certain you do not need PAM-based authentication. Disabling PAM can break two-factor authentication, account locking, and session management. If hardening SSH algorithms or disabling password auth, leaveUsePAM yesin place.
Common MistakeApplying
PasswordAuthentication nobefore verifying that key-based authentication works. If you lock out password auth and your key is not inauthorized_keys, you need console access to recover. Always test key login from a second terminal before disabling passwords.
