It is 3 AM. A Prometheus alert fires. The payment service on one of Razorpay's backend servers has crashed. The on-call engineer logs in, checks the logs, types `systemctl restart payment-service`, confirms it is running, and goes back to sleep. Forty minutes gone. Ticket closed. Same thing happens next Tuesday. And the Tuesday after that. This is the runbook problem. Your team knows exactly how to fix it. The steps are written somewhere in a Confluence page. But every fix still requires a human to wake up, read the page, and type the same three commands. **Ansible** is automation software that lets you encode those three commands into a playbook and run them automatically — across one server or a thousand — without logging in to any of them. ### What a runbook actually is A **runbook** is a documented set of steps to resolve a known operational problem. The steps exist because someone figured out the fix the first time and wrote it down. Every time that problem recurs, someone follows those steps. The gap between a runbook and automation is execution. A runbook lives in a wiki. Ansible turns it into code that executes itself. ### How Ansible changes the equation Before Ansible, fixing infrastructure meant: - SSH into each affected server manually - Run commands one at a time - Hope you did not forget a step on server 7 of 12 - Repeat next week With Ansible: - One playbook runs the same steps on all servers simultaneously - Output tells you exactly what changed and what did not - The same playbook runs identically every time - A webhook from Prometheus can trigger it automatically with no human involved ### Where this fits in the AIOps stack Ansible sits in the automation layer. Prometheus detects the problem. Alertmanager routes the alert. Ansible fixes it. The agents module covered how AI agents reason and take actions — Ansible is the execution engine those agents call when remediation needs to happen on real infrastructure. Prometheus detects anomaly | v Alertmanager fires alert | v Webhook triggers Ansible | v Playbook runs on affected servers | v Problem resolved, verified, logged
Ansible does not need software installed on the servers it manages. This is the first thing that surprises people coming from tools like Chef or Puppet, which require an agent running on every machine. ### Control node vs managed nodes Ansible works with two types of machines: A **control node** is the machine where Ansible is installed — your laptop, a CI server, or a dedicated automation server. Commands and playbooks run from here. **Managed nodes** are the servers Ansible connects to and configures. They need nothing special installed. Just Python 3 and SSH access. Your laptop / CI server (Control Node — Ansible installed here) | | SSH connection v prod-web-01 (Managed Node — no Ansible needed) prod-web-02 (Managed Node — no Ansible needed) prod-db-01 (Managed Node — no Ansible needed) ### How Ansible connects and runs tasks When you run a playbook, here is what happens under the hood for each task: 1. Ansible reads the module code from the control node 2. Copies it as a temporary Python script to the managed node via SSH/SFTP 3. Executes the script on the managed node 4. Reads the JSON output back to the control node 5. Deletes the temporary script 6. Reports success, failure, or no change No persistent connection. No daemon running on the server. Each task is a brief SSH session. ### Idempotency — the most important property in automation **Idempotency** means running the same operation multiple times produces the same result as running it once. The system ends up in the desired state whether it was already there or not. This matters enormously in ops automation. Your self-healing playbook might run 10 times in an hour if alerts keep firing. Without idempotency, you could end up creating the same user 10 times, appending the same config line 10 times, or breaking something by running a fix twice. Good Ansible modules are idempotent by design: ```yaml ## The service module checks if nginx is already running ## If it is running: changed=false, nothing happens ## If it is stopped: changed=true, Ansible starts it - name: Ensure nginx is running service: name: nginx state: started ``` > **Note:** `state: started` means "make sure it is running" — not "start it right now regardless." Ansible only acts if the state does not match. This is the core of idempotency. The shell module is NOT idempotent: ```yaml ## This runs every single time, no matter what ## If run 10 times, appends the line 10 times - name: Add config line (BAD) shell: echo "max_connections=100" >> /etc/myapp.conf ``` Always prefer modules over shell commands. Modules know the current state. Shell commands do not.
The **inventory** is the list of servers Ansible manages. Before running any playbook, Ansible needs to know which machines to target and how to connect to them. ### Static inventory file format The default inventory location is `/etc/ansible/hosts` but you can keep it anywhere and pass it with `-i`. ```text ## Simple inventory — two web servers and one database server ## Each line is a hostname or IP address [webservers] prod-web-01 ansible_host=10.0.1.10 prod-web-02 ansible_host=10.0.1.11 [databases] prod-db-01 ansible_host=10.0.1.20 [monitoring] prod-mon-01 ansible_host=10.0.1.30 ``` > **Note:** `[webservers]` is a group name. Groups let you run playbooks against a specific set of servers without listing them individually each time. ### Host variables and group variables You can set variables per host or per group: ```text ## Per-host variables — set on the same line prod-web-01 ansible_host=10.0.1.10 ansible_user=ubuntu ansible_port=22 ## Or in a separate group_vars directory ## group_vars/webservers.yml ``` ```yaml ## group_vars/webservers.yml ## These variables apply to every host in [webservers] ansible_user: ubuntu ansible_ssh_private_key_file: ~/.ssh/prod_key app_port: 8080 ``` ### Verify your inventory Before running any playbook, confirm Ansible can see your inventory: ```bash ## List all hosts and their variables ansible-inventory -i inventory.ini --list ## Ping all hosts to verify SSH connectivity ansible all -i inventory.ini -m ping ## Ping only the webservers group ansible webservers -i inventory.ini -m ping ``` Expected output when connectivity works: ```text prod-web-01 | SUCCESS => { "changed": false, "ping": "pong" } prod-web-02 | SUCCESS => { "changed": false, "ping": "pong" } ``` > 📌 **Remember:** If ping fails, the problem is almost always SSH — wrong key, wrong user, or the host is not reachable. Fix SSH manually first before debugging Ansible. ### Dynamic inventory for cloud environments For AWS, GCP, or Kubernetes environments where servers come and go, a static file becomes outdated immediately. You add a server on Monday, terminate it Friday — a static file never reflects this. **Dynamic inventory plugins** query the cloud API at runtime and return the current list of hosts automatically. ```bash ## Install the AWS collection — includes the ec2 dynamic inventory plugin ansible-galaxy collection install amazon.aws ## Confirm it installed ansible-galaxy collection list | grep amazon ``` Create an `aws_ec2.yml` file instead of a `hosts` file: ```yaml ## aws_ec2.yml — dynamic inventory plugin for AWS EC2 ## Ansible reads this file and queries EC2 for live instances plugin: amazon.aws.aws_ec2 ## which plugin to use regions: - ap-south-1 ## Mumbai region — change to your region ## Group instances automatically by their tags ## An EC2 instance tagged Environment=production goes into group "tag_Environment_production" keyed_groups: - key: tags.Environment prefix: env - key: tags.Role prefix: role ## Only return running instances — skip terminated, stopped filters: instance-state-name: running ## Use the private IP for SSH — change to public_ip_address for internet-facing hostnames: - private-ip-address ## Variables to add to every host compose: ansible_host: private_ip_address ``` > **Note:** `keyed_groups` is the powerful part. If your EC2 instances have a tag `Role=webserver`, Ansible automatically creates a group called `role_webserver` containing all those instances. You can then run `--limit role_webserver` without maintaining a list of hostnames manually. See what the plugin returns: ```bash ## Query EC2 and list all discovered hosts and groups ansible-inventory -i aws_ec2.yml --list ## Ping all instances in the production environment group ansible env_production -i aws_ec2.yml -m ping ## Run a playbook against only web servers ansible-playbook deploy.yml -i aws_ec2.yml --limit role_webserver ``` Example output showing auto-discovered groups: ```text { "env_production": { "hosts": ["10.0.1.10", "10.0.1.11", "10.0.1.12"] }, "role_webserver": { "hosts": ["10.0.1.10", "10.0.1.11"] }, "role_database": { "hosts": ["10.0.1.12"] } } ``` As instances are added or removed in AWS, the inventory updates automatically on the next playbook run — no file editing needed.
**Modules** are the tools that do the actual work. Every task in a playbook calls one module. Ansible ships with thousands of built-in modules covering Linux, Windows, cloud providers, databases, network devices, and more. ### How modules work When a task runs, Ansible: 1. Takes the module code from the control node 2. Passes your arguments to it 3. Runs it on the managed node 4. Gets a JSON response back — `{"changed": true}` or `{"changed": false}` or an error You never write module code. You just call the module by name and give it arguments: ```yaml ## Module: ansible.builtin.service ## Arguments: name and state - name: Start the payment service ansible.builtin.service: name: payment-service state: started ``` ### Most used ops modules These are the modules you will use in 80% of ops runbooks: | Module | What it does | Common use | |:-------|:-------------|:-----------| | `service` / `systemd` | Start, stop, restart, enable services | Restart crashed services | | `apt` / `yum` / `dnf` | Install, remove, update packages | Patch management | | `copy` | Copy files from control node to managed node | Deploy config files | | `file` | Create, delete, set permissions on files/dirs | Clean up disk space | | `command` | Run a command (not through shell) | One-off system commands | | `shell` | Run shell commands with pipes and redirects | Complex one-liners (use sparingly) | | `lineinfile` | Ensure a line exists or does not exist in a file | Config updates | | `template` | Copy a Jinja2 template, substituting variables | Dynamic config files | | `uri` | Make HTTP requests | Health checks, API calls | | `debug` | Print a variable or message during a run | Troubleshooting | ### Reading module output Every task produces output with four possible states: ```text ok: [prod-web-01] -- task ran, nothing changed (already in desired state) changed: [prod-web-01] -- task ran and made a change skipped: [prod-web-01] -- task was skipped (when condition was false) failed: [prod-web-01] -- task failed, playbook stops here by default ``` The PLAY RECAP at the end summarises every host: ```text PLAY RECAP prod-web-01 : ok=4 changed=2 unreachable=0 failed=0 skipped=1 prod-web-02 : ok=4 changed=0 unreachable=0 failed=0 skipped=1 ``` `failed=0` means everything succeeded. `changed=0` on the second run means the system was already in the desired state — idempotency working correctly.
A **playbook** is a YAML file that describes what you want done and on which servers. It contains one or more **plays**, and each play contains one or more **tasks**. ### YAML structure — the anatomy of a playbook ```yaml ## Every playbook starts with three dashes --- ## A play begins with a name and a hosts selector - name: Ensure payment service is healthy hosts: webservers ## which inventory group to target become: yes ## run tasks as root (sudo) ## vars block — define variables for this play vars: service_name: payment-service max_retries: 3 ## tasks — the list of things to do, in order tasks: - name: Check if service is running systemd: name: "{{ service_name }}" ## {{ }} inserts variable value state: started - name: Verify service responds on port 8080 uri: url: http://localhost:8080/health status_code: 200 ``` > **Note:** YAML indentation is everything. Two spaces per level, no tabs. The most common Ansible error for beginners is wrong indentation — the task is at the wrong level and the playbook either fails or silently ignores it. ### Handlers — run only when something changes A **handler** is a special task that only runs if another task notifies it. The classic use case is restarting a service only when its config file changed. ```yaml --- - name: Configure nginx hosts: webservers become: yes tasks: - name: Copy nginx config copy: src: files/nginx.conf dest: /etc/nginx/nginx.conf notify: Restart nginx ## sends a notification to the handler - name: Copy SSL certificate copy: src: files/cert.pem dest: /etc/nginx/cert.pem notify: Restart nginx ## same handler, still only runs once handlers: - name: Restart nginx ## name must match exactly what notify uses service: name: nginx state: restarted ``` Even if both tasks notify the handler, it only runs once at the end of the play. This prevents restarting nginx twice unnecessarily. ### Variables and extra-vars Variables make playbooks reusable across environments: ```yaml ## vars/main.yml — default values service_name: payment-service restart_limit: 3 alert_webhook: http://10.0.1.30:9093/alert ``` ```bash ## Override variables at runtime with --extra-vars ## This is how Prometheus alert data gets passed into a playbook ansible-playbook remediate.yml \ --extra-vars "target_host=prod-web-03 service_name=order-service" ``` ### Running and reading output ```bash ## Syntax check before running — catches YAML errors without executing ansible-playbook -i inventory.ini playbook.yml --syntax-check ## Dry run — shows what WOULD change without changing anything ansible-playbook -i inventory.ini playbook.yml --check ## Run for real ansible-playbook -i inventory.ini playbook.yml ## Run only on one host from the group ansible-playbook -i inventory.ini playbook.yml --limit prod-web-01 ## Run only tasks tagged with a specific label ansible-playbook -i inventory.ini playbook.yml --tags restart ``` ### Tags and limits — run exactly what you intend Two flags you will use every day in production ops: `--limit` restricts which hosts the playbook runs on, even if the `hosts:` selector matches more: ```bash ## Run on the whole webservers group ansible-playbook -i inventory.ini deploy.yml ## Run only on one specific host — useful for testing before rolling out ansible-playbook -i inventory.ini deploy.yml --limit prod-web-01 ## Run on two specific hosts ansible-playbook -i inventory.ini deploy.yml --limit "prod-web-01,prod-web-02" ``` `--tags` runs only tasks that have a matching tag — useful when you want to run just the restart step of a long playbook: ```yaml ## Tag tasks in your playbook tasks: - name: Deploy new application version copy: src: app.tar.gz dest: /opt/app/ tags: deploy ## only runs when --tags deploy is used - name: Restart application service systemd: name: myapp state: restarted tags: restart ## only runs when --tags restart is used - name: Run health check uri: url: http://localhost:8080/health status_code: 200 tags: verify ## only runs when --tags verify is used ``` ```bash ## Run only the restart and verify steps — skip the deploy step ansible-playbook -i inventory.ini playbook.yml --tags "restart,verify" ## Run everything EXCEPT the deploy step ansible-playbook -i inventory.ini playbook.yml --skip-tags deploy ``` > 🔴 **Common Mistake:** Running a playbook directly in production without `--check` first. Always dry-run on production before applying changes. One wrong `state: absent` on a file task can delete something important.
As you write more playbooks, you will notice patterns repeating. The same nginx restart logic appears in your deployment playbook, your health check playbook, and your monitoring setup playbook. **Roles** solve this by bundling related tasks, variables, and templates into a reusable package. ### What a role is and why it exists A role is a directory with a standard structure. Instead of pasting the same 20 tasks into every playbook, you write them once in a role and call the role by name. Think of it like a Python module — you import it, you do not copy-paste it. ### Directory structure ```bash ## Create a role skeleton with ansible-galaxy ansible-galaxy init roles/service_restarter ``` This creates: ```text roles/ service_restarter/ tasks/ main.yml <- the actual task list handlers/ main.yml <- handlers (restart triggers) defaults/ main.yml <- default variable values (easily overridden) vars/ main.yml <- fixed variables (not meant to be overridden) templates/ *.j2 <- Jinja2 config file templates files/ * <- static files to copy meta/ main.yml <- role metadata and dependencies ``` ### A real ops role — service restarter ```yaml ## roles/service_restarter/tasks/main.yml --- - name: Check current service status systemd: name: "{{ target_service }}" ## variable — caller provides this register: service_status ## save the output for the next task - name: Restart service if not running systemd: name: "{{ target_service }}" state: restarted when: service_status.status.ActiveState != "active" ## only if not running - name: Wait for service to become healthy uri: url: "{{ health_check_url }}" status_code: 200 retries: 5 ## try up to 5 times delay: 10 ## wait 10 seconds between retries register: health until: health.status == 200 ``` ```yaml ## roles/service_restarter/defaults/main.yml --- ## Default values — callers can override any of these target_service: nginx health_check_url: http://localhost:80/health ``` ### Using the role in a playbook ```yaml ## playbook.yml --- - name: Remediate payment service hosts: "{{ target_host }}" ## passed from webhook alert become: yes roles: - role: service_restarter vars: target_service: payment-service health_check_url: http://localhost:8080/health ``` One line calls the entire role. Same role works for nginx, payment-service, order-service — just pass different variables. > 💡 **Tip:** Use `defaults/main.yml` for variables you want callers to override. Use `vars/main.yml` for constants internal to the role. The difference matters because Ansible variable precedence treats them differently.
It is 3 AM. A Prometheus alert fires. The payment service on one of Razorpay's backend servers has crashed. The on-call ...
Ansible does not need software installed on the servers it manages. This is the first thing that surprises people coming...
The inventory is the list of servers Ansible manages. Before running any playbook, Ansible needs to know which machines ...
Modules are the tools that do the actual work. Every task in a playbook calls one module. Ansible ships with thousands o...
A playbook is a YAML file that describes what you want done and on which servers. It contains one or more plays, and eac...
As you write more playbooks, you will notice patterns repeating. The same nginx restart logic appears in your deployment...
Every real ops automation eventually needs credentials — database passwords, API keys, webhook tokens, SSH passphrases. ...
Modern Ansible organises reusable content into collections. A collection bundles related modules, roles, plugins, and do...
Now that you understand playbooks and roles, here are the runbooks ops teams use most often. Each one maps to a real inc...
This is where individual runbooks become a self-healing system. The goal: Prometheus fires an alert, a webhook receiver ...
Auto-remediation is powerful enough to fix things at 3 AM without waking anyone up. It is also powerful enough to make t...
You will build a complete self-healing system: a Python app that occasionally "crashes," a Prometheus rule that detects ...
Ansible command cheat sheet Command What it does ansible all -m ping Verify connectivity to all hosts ansible-inventory ...
Running ansible-playbook from your laptop works well for a single engineer. When your team grows to 10 engineers, all ru...
The self-healing system you have built connects every layer of the AIOps stack: Infrastructure (Linux servers, K8s, clou...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.