Runtime Security and Detection Engineering | Engineering Modules | DevOps Network

Engineering ModuleADVANCED

Runtime Security and Detection Engineering

Build a production-grade runtime security and detection pipeline -covering eBPF fundamentals, Falco syscall-based threat detection with custom rules, Tetragon eBPF enforcement, MITRE ATT&CK mapping for Cloud and Containers, Sigma rules and SIEM conversion, Elasticsearch SIEM with alert correlation and noise reduction, auditd Linux syscall monitoring, OpenTelemetry for security observability, and automated incident response runbooks.

~195 Minutes

13 Topics

Hands-on Scenarios

What You'll Learn

Introduction

Shift-left security catches vulnerabilities before deployment. Runtime security catches what slips through — and everything that happens after code reaches production. Containers get compromised, credentials get stolen, attackers move laterally, and misconfigured workloads get exploited at runtime. None of those events are visible to a static scanner. Runtime security and detection engineering is the discipline of observing everything that happens inside running systems — syscalls, file access, network connections, process executions — and turning that observation into alerts, investigations, and automated responses. Topics covered: - eBPF — the technology powering modern runtime observability - Falco — syscall-based threat detection for containers and Kubernetes - Tetragon — eBPF-powered runtime enforcement with Cilium - MITRE ATT&CK for Cloud and Containers — mapping detections to the threat landscape - Sigma rules — vendor-agnostic detection logic - SIEM with Elasticsearch — ingesting, correlating, and querying security events - Alert correlation and noise reduction - auditd — Linux syscall monitoring - OpenTelemetry for security observability - Incident response automation and runbooks ---

eBPF — The Technology Behind Modern Runtime Security

eBPF (extended Berkeley Packet Filter) allows programs to run safely inside the Linux kernel without modifying kernel source code or loading kernel modules. For security, this means observing every syscall, network packet, file access, and process event — with minimal overhead and no blind spots. ``` Traditional monitoring: Application → OS kernel → [no visibility here] → hardware eBPF monitoring: Application → OS kernel → eBPF program hooks at kernel events → security tool ↑ Sees everything: syscalls, network, files, processes Cannot be bypassed by user-space code Kernel verifier ensures the eBPF program is safe ``` Why eBPF matters for security: ``` 1. Kernel-level visibility An attacker who compromises a container cannot hide from eBPF The container runtime sits above the kernel — eBPF is in the kernel 2. Low overhead eBPF filtering happens before data reaches user space Only relevant events are forwarded to your security tool Typically <1% CPU overhead even at high event rates 3. No TOCTOU attacks Time-Of-Check-Time-Of-Use: traditional tools check a file, then use it eBPF enforces at the kernel syscall level — the check and use are atomic 4. Kubernetes-aware context eBPF tools like Tetragon correlate kernel events with pod/namespace/label metadata A syscall is attributed to "payments/processor" not just "pid 1234" ``` ---

Falco — Syscall-Based Threat Detection

Falco is the CNCF standard for runtime security. It monitors Linux syscalls and Kubernetes audit logs, evaluates them against a rules engine, and generates alerts when something suspicious happens. ### How Falco Works ``` Process in container makes a syscall (e.g., execve, open, connect) ↓ Falco's kernel driver (eBPF or kernel module) captures the event ↓ Event enriched with container/pod/namespace metadata ↓ Rules engine evaluates: does this match a rule condition? ↓ Alert generated with output fields: user, command, container, image, etc. ``` ### Installing Falco on Kubernetes ```bash # Install Falco with Helm — eBPF driver (preferred, no kernel module needed) helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update helm install falco falcosecurity/falco \ --namespace falco \ --create-namespace \ --set driver.kind=ebpf \ --set falcosidekick.enabled=true \ --set falcosidekick.webui.enabled=true \ --set tty=true # Verify Falco is running kubectl get pods -n falco kubectl logs -n falco daemonset/falco | grep "Falco initialized" | tail -1 ``` ### Understanding Falco Rules A Falco rule has four components: condition, output, priority, and tags. ```yaml # rules/custom-rules.yaml # Rule 1: Detect shell spawned inside a container # This catches most post-exploitation scenarios — attacker gets RCE, spawns bash - rule: Shell Spawned in Container desc: > A shell was spawned inside a container. This is unusual for production workloads and may indicate a compromise or debugging session. condition: > container and proc.name in (shell_binaries) and not container.image.repository in (allowed_shell_images) and not proc.pname in (shell_binaries) output: > Shell spawned in container (user=%user.name user_loginuid=%user.loginuid image=%container.image.repository container=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline) priority: WARNING tags: [container, shell, MITRE_TA0002_execution] # Rule 2: Detect write to sensitive file paths - rule: Write to Sensitive File desc: Detect writes to files that should never change at runtime condition: > open_write and fd.name in (/etc/passwd, /etc/shadow, /etc/sudoers, /etc/hosts) and not proc.name in (passwd, shadow, useradd, usermod) output: > Sensitive file opened for writing (user=%user.name command=%proc.cmdline file=%fd.name container=%container.name image=%container.image.repository) priority: ERROR tags: [filesystem, MITRE_TA0003_persistence, MITRE_T1098_account_manipulation] # Rule 3: Detect outbound connection to unexpected port from container - rule: Unexpected Outbound Connection desc: Container made an outbound connection to an unusual port condition: > outbound and container and not fd.sport in (allowed_outbound_ports) and not container.image.repository in (network_tools_images) output: > Unexpected outbound connection from container (user=%user.name command=%proc.cmdline connection=%fd.name container=%container.name image=%container.image.repository) priority: NOTICE tags: [network, MITRE_TA0010_exfiltration] # Rule 4: Detect privilege escalation — setuid binary executed - rule: Setuid Binary Executed desc: > A setuid binary was executed inside a container. Attackers use setuid binaries to escalate from container user to root. condition: > container and proc.is_suid_exe = true and not proc.name in (allowed_setuid_binaries) output: > Setuid binary executed in container (user=%user.name binary=%proc.name parent=%proc.pname container=%container.name image=%container.image.repository) priority: WARNING tags: [container, privilege_escalation, MITRE_TA0004_privilege_escalation] ``` ### Falco Macros and Lists Macros and lists keep rules readable and maintainable: ```yaml # Reusable lists - list: shell_binaries items: [bash, csh, ksh, sh, tcsh, zsh, dash] - list: allowed_shell_images items: [debug-tools, busybox] - list: allowed_outbound_ports items: [80, 443, 8080, 8443, 5432, 6379, 9200] - list: allowed_setuid_binaries items: [ping, su, sudo] # Reusable macros - macro: container condition: container.id != host - macro: outbound condition: > (evt.type = connect and evt.dir = <) and fd.typechar = 4 and fd.connected = true - macro: open_write condition: > (evt.type in (open, openat, openat2) and evt.is_open_write = true and fd.typechar = 'f' and fd.num >= 0) ``` ### Production Rule Tuning New Falco deployments generate many false positives. Tuning is not about turning off rules — it is about scoping them to your actual threat model: ```yaml # Before tuning: alerts on ALL containers writing to /proc - rule: Write below root condition: open_write and fd.name startswith /proc # After tuning: exclude known-good paths, scope to containers - rule: Write below root condition: > open_write and fd.name startswith /proc and container and not fd.name in (/proc/self/fd, /proc/self/attr/current) and not container.image.repository in (monitoring_images) except: - name: known_write_paths fields: [proc.name, fd.name] values: - [fluentd, /proc/*/net/dev] # Log collector reads proc - [node-exporter, /proc/stat] # Metrics collector reads proc # Severity tuning — not every alert is the same urgency # Development environments: WARNING on most rules # Staging: ERROR on rules that are CRITICAL in production # Production: CRITICAL rules page the on-call engineer immediately ``` ### Falco Output — Forwarding to SIEM ```bash # falco.yaml — configure outputs outputs: stdout_output: enabled: true file_output: enabled: true keep_alive: false filename: /var/log/falco/events.log http_output: enabled: true url: http://falcosidekick:2801 # Falcosidekick forwards to Elasticsearch, Slack, etc. user_agent: falcosecurity/falco # Falcosidekick routes alerts to multiple destinations simultaneously # helm values for falcosidekick config: elasticsearch: hostport: http://elasticsearch:9200 index: falco-events type: _doc slack: webhookurl: https://hooks.slack.com/services/... minimumpriority: warning pagerduty: apikey: <pagerduty-api-key> minimumpriority: critical ``` ---

Tetragon — eBPF Runtime Enforcement

Falco detects and alerts. Tetragon can detect **and enforce** — blocking malicious actions at the kernel level before they complete, eliminating the window between detection and response (the TOCTOU gap). ### Installing Tetragon ```bash # Install Tetragon alongside Falco (complementary tools) helm repo add cilium https://helm.cilium.io helm repo update helm install tetragon cilium/tetragon \ --namespace kube-system \ --set tetragon.exportFilename=/var/run/cilium/tetragon/tetragon.log kubectl rollout status -n kube-system daemonset/tetragon -w ``` ### Tetragon Tracing Policies Tetragon uses `TracingPolicy` CRDs to define what to observe and what to enforce: ```yaml # Policy 1: Monitor all process executions in a specific namespace apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: monitor-exec-payments spec: kprobes: - call: "sys_execve" syscall: true args: - index: 0 type: "string" # filename - index: 1 type: "string_array" # argv selectors: - matchNamespaces: - namespace: Mnt values: - "/payments" # Only in payments namespace mounts --- # Policy 2: BLOCK binary execution from /tmp (common malware drop location) apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: block-tmp-execution spec: kprobes: - call: "sys_execve" syscall: true args: - index: 0 type: "string" selectors: - matchArgs: - index: 0 operator: "Prefix" values: - "/tmp/" - "/dev/shm/" matchActions: - action: Sigkill # Kill the process immediately at kernel level --- # Policy 3: Monitor sensitive file access apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: monitor-sensitive-files spec: kprobes: - call: "sys_openat" syscall: true args: - index: 1 type: "string" # filename selectors: - matchArgs: - index: 1 operator: "Postfix" values: - "/etc/passwd" - "/etc/shadow" - "/.aws/credentials" - "/.ssh/id_rsa" ``` ### Reading Tetragon Events ```bash # Watch process events in real time kubectl exec -n kube-system daemonset/tetragon -- \ tetra getevents -o compact --pods my-app # Sample output: # 🚀 process payments/processor /bin/python app.py # 📬 open payments/processor /etc/ssl/certs/ca-certificates.crt # 🌐 connect payments/processor tcp 10.0.1.15:5432 [postgres] # 💥 process payments/processor /tmp/malware ← BLOCKED by policy # JSON output for SIEM ingestion kubectl exec -n kube-system daemonset/tetragon -- \ tetra getevents --output json | jq '{ time: .time, type: .process_exec.process.binary // .process_kprobe.function_name, pod: .process_exec.process.pod.name // .process_kprobe.process.pod.name, namespace: .process_exec.process.pod.namespace, binary: .process_exec.process.binary, arguments: .process_exec.process.arguments }' ``` ---

MITRE ATT&CK for Cloud and Containers

MITRE ATT&CK is a knowledge base of adversary tactics and techniques observed in real attacks. The Cloud and Container matrices map these techniques to cloud-native environments — giving detection engineers a structured framework for coverage. ### Container ATT&CK Matrix — Key Techniques ``` Initial Access ├── T1190 Exploit Public-Facing Application (vulnerable API exposed) ├── T1133 External Remote Services (exposed Kubernetes API) └── T1078 Valid Accounts (stolen service account token) Execution ├── T1059 Command and Scripting Interpreter (bash in container) ├── T1204.003 Malicious Image (backdoored container image) └── T1609 Container Administration Command (kubectl exec) Persistence ├── T1525 Implant Internal Image (backdoor pushed to registry) ├── T1078 Valid Accounts (create new service account) └── T1053 Scheduled Task/Job (CronJob created in cluster) Privilege Escalation ├── T1611 Escape to Host (privileged container, hostPath mount) ├── T1548 Abuse Elevation Control Mechanism (sudo, setuid) └── T1134 Access Token Manipulation (steal service account token) Defense Evasion ├── T1070 Indicator Removal (delete audit logs, Falco logs) ├── T1562.001 Disable or Modify Tools (kill Falco process) └── T1036 Masquerading (rename binary to look like system process) Credential Access ├── T1552.001 Credentials in Files (read .env, mounted secrets) ├── T1528 Steal Application Access Token (read K8s SA token) └── T1212 Exploitation for Credential Access Lateral Movement ├── T1021 Remote Services (SSH to other pods using stolen keys) └── T1210 Exploitation of Remote Services (pivot from compromised pod) Exfiltration ├── T1041 Exfiltration Over C2 Channel └── T1537 Transfer Data to Cloud Account ``` ### Mapping Falco Rules to ATT&CK ```yaml # Detection coverage mapping — which rules cover which techniques # T1611: Escape to Host - rule: Privileged Container Launched condition: > container_started and container.privileged = true and not container.image.repository in (allowed_privileged_images) tags: [MITRE_T1611_container_escape] # T1552.001: Credentials in Files - rule: Sensitive File Read in Container condition: > open_read and container and fd.name in ( /root/.aws/credentials, /home/*/.aws/credentials, /etc/kubernetes/admin.conf, /var/run/secrets/kubernetes.io/serviceaccount/token ) and not proc.name in (aws, kubectl, node) tags: [MITRE_T1552_unsecured_credentials] # T1562.001: Disable Security Tools - rule: Falco Process Killed condition: > (evt.type = kill or evt.type = tkill) and proc.name = falco output: > Attempt to kill Falco security process detected (user=%user.name command=%proc.cmdline) priority: CRITICAL tags: [MITRE_T1562_impair_defenses] ``` ### ATT&CK Navigator for Coverage Gaps ```bash # Export your Falco rule tags to visualize ATT&CK coverage # Parse all enabled rules and extract MITRE tags grep -r "MITRE_T" /etc/falco/rules/ \ | grep -oP 'MITRE_T\d+[\w.]*' \ | sort -u \ | sed 's/MITRE_//' \ | tr '\n' ',' > covered_techniques.txt # Upload to MITRE ATT&CK Navigator (https://mitre-attack.github.io/attack-navigator/) # to visualize your detection coverage heatmap and identify gaps ``` ---

Sigma Rules — Vendor-Agnostic Detection Logic

Sigma is to log files what Snort is to network traffic — a generic, open signature format for sharing detection logic. Write a rule once, convert it to any SIEM's query language. ### Sigma Rule Structure ```yaml # rules/container-escape.yml title: Container Escape via Privileged Mount id: 4f8b2c1a-9e3d-4f2b-8a1c-5e6f7a8b9c0d status: experimental description: > Detects a container accessing host filesystem paths via a privileged mount, which is a common container escape technique. references: - https://attack.mitre.org/techniques/T1611/ author: DevSecOps Team date: 2024/01/15 tags: - attack.privilege_escalation - attack.t1611 logsource: product: falco category: process_creation detection: selection: proc.name: - bash - sh - python - perl selection_path: fd.name|startswith: - '/host/' - '/proc/1/' # Accessing PID 1 from container = host process access condition: selection and selection_path falsepositives: - Debug containers with intentional host access level: high ``` ```yaml # rules/k8s-rbac-abuse.yml title: Kubernetes ClusterRole with Wildcard Permissions Created id: 7c3a9b2e-1f4d-4c8a-9b2e-3f5a7c9b1e2f status: stable description: > A ClusterRole granting wildcard permissions was created. This grants the subject full control over all Kubernetes resources. references: - https://attack.mitre.org/techniques/T1078/ author: DevSecOps Team date: 2024/01/15 tags: - attack.privilege_escalation - attack.t1078 logsource: product: kubernetes service: audit detection: selection: objectRef.resource: clusterroles verb: create requestObject.rules|contains: - '"*"' # Wildcard verb or resource condition: selection falsepositives: - Intentional admin role creation during cluster setup level: critical ``` ### Converting Sigma Rules to SIEM Queries ```bash # Install sigma-cli pip install sigma-cli # Install backend for your SIEM sigma plugin install elasticsearch # For Elasticsearch/OpenSearch sigma plugin install splunk # For Splunk sigma plugin install qradar # For IBM QRadar # Convert a rule to Elasticsearch query sigma convert \ --target elasticsearch \ --pipeline ecs_windows \ rules/container-escape.yml # Output: # {"query": {"bool": {"must": [{"terms": {"process.name": ["bash","sh","python","perl"]}}, {"prefix": {"file.path": "/host/"}}]}}} # Convert an entire rules directory sigma convert \ --target elasticsearch \ --pipeline ecs_kubernetes \ --output-format ndjson \ rules/ > elasticsearch-rules.ndjson # Load converted rules into Elasticsearch detection engine curl -X POST "http://elasticsearch:9200/_security/detection_engine/rules/_import" \ -H "Content-Type: application/ndjson" \ --data-binary @elasticsearch-rules.ndjson ``` ### Building a Custom Sigma Rule from a Threat Report When a threat intelligence report describes a new attack technique, convert it to a Sigma rule: ```yaml # Scenario: Threat report describes attackers using curl to exfiltrate data # from compromised Kubernetes pods to external C2 servers title: Data Exfiltration via curl from Container id: 2b4d6f8a-0c2e-4g6i-8k0m-2o4q6s8u0w2y status: experimental description: > Detects curl being used inside a container to make outbound connections to non-internal IP ranges. May indicate data exfiltration. logsource: product: falco category: network_connection detection: selection_process: proc.name: curl filter_internal: fd.rip|cidr: - '10.0.0.0/8' - '172.16.0.0/12' - '192.168.0.0/16' - '127.0.0.0/8' condition: selection_process and not filter_internal falsepositives: - Legitimate external API calls from curl-based health checks level: medium tags: - attack.exfiltration - attack.t1041 ``` ---

Runtime Security and Detection Engineering

What You'll Learn

Introduction

eBPF — The Technology Behind Modern Runtime Security

Falco — Syscall-Based Threat Detection

Tetragon — eBPF Runtime Enforcement

MITRE ATT&CK for Cloud and Containers

Sigma Rules — Vendor-Agnostic Detection Logic

Skills You'll Master

Curriculum Index

Introduction

eBPF — The Technology Behind Modern Runtime Security

Falco — Syscall-Based Threat Detection

Tetragon — eBPF Runtime Enforcement

MITRE ATT&CK for Cloud and Containers

Sigma Rules — Vendor-Agnostic Detection Logic

SIEM with Elasticsearch — Ingest, Correlate, Query

Alert Correlation and Noise Reduction

auditd — Linux Syscall Monitoring

OpenTelemetry for Security Observability

Incident Response Automation and Runbooks

💻 Lab: Build a Detection Pipeline

Summary

Career Impact

Troubleshooting & Lab Guides

How to Troubleshoot a Linux Production Server: A Systematic Approach

Next Modules

S3 to FSx for OpenZFS

DevSecOps Basics

Event Correlation & Alert Noise Reduction - prometheus and Alert manager

Related Guides

What Is AI SRE? How AI Agents Are Changing Incident Response

OpenTelemetry Explained: Unifying Metrics, Logs, and Traces

Shift-Left Security: Adding SBOM and Supply Chain Scanning to Your CI/CD Pipeline