Auto-Remediate GuardDuty Findings with Lambda
Build an AWS auto-remediation pipeline: GuardDuty detects threats, EventBridge routes them, and Lambda quarantines the instance and alerts Slack.
Blueprint Walkthrough
Architecture Overview & Problem Statement
This project builds an automated incident-response pipeline on AWS. GuardDuty detects a threat, EventBridge routes high-severity findings to a Lambda function, and Lambda quarantines the affected EC2 instance, deactivates a leaked IAM access key, and posts the evidence to Slack, with no human needing to be online. You build it with the AWS CLI and Python, and you add the guard rails (dry-run mode, an opt-out tag and a rollback runbook) that make unattended remediation safe.
What You Will Learn
- How GuardDuty findings are structured (type, severity, resource) and how they reach EventBridge
- How to write an EventBridge event pattern that matches a numeric severity
- How to isolate an instance without destroying forensic evidence
- How to write a remediation function that is safe to run unattended: dry-run mode, idempotency, an opt-out tag and least-privilege IAM
- How to keep a secret (the Slack webhook) out of environment variables
What You Can Show
A GitHub repo with the Lambda code, a README of the CLI steps, a screenshot of a real Slack alert (webhook removed), your measured response time, and the rollback runbook.
Prerequisites, Time and Cost
- An AWS account you can safely experiment in (not a shared production account)
- AWS CLI v2 configured, basic Python, and a Slack workspace where you can create an incoming webhook
- About 2-3 hours
- GuardDuty bills on the data it analyses, and new accounts typically get a free-trial period. Lambda, EventBridge and an SSM standard parameter cost effectively nothing at this volume. Check the current GuardDuty pricing page and run the cleanup at the end.
Architecture Overview
A security alert with no automation is just noise. When GuardDuty flags an EC2 instance talking to a known command-and-control server at 3am, nobody is awake to respond. By the time someone at a company like Hotstar reads the alert in the morning, the instance may have been compromised for hours during peak streaming traffic.
This project automates the first response step. GuardDuty detects the threat. EventBridge routes the finding to a Lambda function. Lambda quarantines the instance by swapping its security groups for a deny-all group, tags it with what happened, and sends a Slack alert, with no human needing to be online.
activity] --> B[GuardDuty
finding] B --> C[EventBridge
rule] C --> D[Lambda] classDef blue fill:#326CE5,stroke:#1E4FB8,color:#fff classDef green fill:#10B981,stroke:#0B7A56,color:#fff classDef orange fill:#F97316,stroke:#B85410,color:#fff classDef red fill:#EF4444,stroke:#B02D2D,color:#fff classDef purple fill:#8B5CF6,stroke:#6238C2,color:#fff classDef slate fill:#475569,stroke:#2B3544,color:#fff class A slate class B orange class C purple class D blue
When the rule matches, one Lambda function does the response. It first checks DRY_RUN and the opt-out tag, then takes up to three actions:
and tags] L --> K[Deactivate
IAM key] L --> S[Slack alert] classDef blue fill:#326CE5,stroke:#1E4FB8,color:#fff classDef green fill:#10B981,stroke:#0B7A56,color:#fff classDef orange fill:#F97316,stroke:#B85410,color:#fff classDef red fill:#EF4444,stroke:#B02D2D,color:#fff classDef purple fill:#8B5CF6,stroke:#6238C2,color:#fff classDef slate fill:#475569,stroke:#2B3544,color:#fff class L blue class Q red class K red class S green
TipDo not promise a response time you have not measured. In the Validation section you measure the part you control (EventBridge to Lambda to isolated instance) and write the number in your README. GuardDuty's own detection delay is separate and is not something your code can speed up.
Problem Solved
Manual incident response has a timing problem: the gap between "a threat occurs" and "a human notices and reacts" can be hours, especially overnight or at weekends. In that gap a compromised instance can exfiltrate data, mine cryptocurrency on your bill, or pivot to other resources, a real risk for any company running workloads around the clock, like Zerodha during market hours.
This project shrinks that gap by automating the first step. GuardDuty analyses CloudTrail activity, VPC Flow Logs and DNS logs using threat intelligence and anomaly detection, so you do not write any detection logic yourself. Your job is the response.
Isolation is not termination. The standard pattern is to cut a compromised instance off from the network while keeping its disk and memory for forensics, not to destroy the evidence by terminating it.
SecurityAutomated remediation limits blast radius, but it can also cause an outage when a finding is a false positive. That is why this project adds three guard rails that most tutorials skip: a
DRY_RUNswitch, an opt-out tag for instances that need a human decision, and a rollback runbook. Treat the automation as the first responder, not the investigator.
Milestone 1: Enable GuardDuty
Concept
Enabling GuardDuty is one API call. There is no agent to install. GuardDuty is regional: a detector in ap-south-1 only watches ap-south-1. This project works in a single region. In a real company you would enable it in every region you use, usually through AWS Organizations with a delegated administrator account.
Steps
Check whether a detector already exists, because an account can only have one per region:
export AWS_REGION=ap-south-1aws guardduty list-detectors --region $AWS_REGION{ "DetectorIds": []}If the list is empty, create one:
DETECTOR_ID=$(aws guardduty create-detector \ --enable \ --finding-publishing-frequency FIFTEEN_MINUTES \ --region $AWS_REGION \ --query 'DetectorId' \ --output text)echo $DETECTOR_IDIf a detector already exists, reuse its ID instead:
DETECTOR_ID=$(aws guardduty list-detectors --region $AWS_REGION \ --query 'DetectorIds[0]' --output text)Tip
FIFTEEN_MINUTEScontrols how often updates to an existing finding are re-published to EventBridge. New findings are published to EventBridge as they are generated, and GuardDuty detects continuously.
Common MistakeRunning
create-detectorwhen a detector already exists. The call fails, so always list detectors first and reuse the existing ID.
Milestone 2: Create the Deny-All Quarantine Security Group
Concept
Isolation relies on one security group with zero inbound and zero outbound rules. Swapping it onto an instance blocks new network traffic in both directions without touching the disk or memory.
A security group belongs to one VPC, and an instance can only use groups from its own VPC. The Lambda in this project therefore looks up the quarantine group by name inside the instance's VPC, instead of hard-coding one group ID. You create a group with the same name in each VPC you want to protect.
Steps
VPC_ID=$(aws ec2 describe-vpcs \ --filters Name=isDefault,Values=true \ --query 'Vpcs[0].VpcId' \ --output text \ --region $AWS_REGION) QUARANTINE_SG=$(aws ec2 create-security-group \ --group-name "QUARANTINE-deny-all" \ --description "Quarantine - no inbound or outbound traffic" \ --vpc-id $VPC_ID \ --region $AWS_REGION \ --query 'GroupId' \ --output text)echo $QUARANTINE_SGA new group has no inbound rules, but it does have a default allow-all outbound rule. Remove it:
aws ec2 revoke-security-group-egress \ --group-id $QUARANTINE_SG \ --ip-permissions '[{"IpProtocol": "-1", "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}]' \ --region $AWS_REGIONConfirm both rule lists are empty:
aws ec2 describe-security-groups \ --group-ids $QUARANTINE_SG \ --region $AWS_REGION \ --query 'SecurityGroups[0].{In:IpPermissions,Out:IpPermissionsEgress}'{ "In": [], "Out": []}Common MistakeForgetting to revoke the default outbound rule. The group would block inbound traffic but still allow outbound, so a compromised instance could keep exfiltrating data.
SecuritySecurity groups track connections. Per AWS's documentation on security group connection tracking, connections that were already established before the change can keep working for a while, even though the new group has no rules. If you must cut every existing connection immediately, a network ACL change on the subnet or stopping the instance is stronger, but stopping loses memory contents. Decide this trade-off before you rely on this pattern in production. The Extensions section covers it.
Milestone 3: Store the Slack Webhook Securely
Concept
A Slack incoming-webhook URL is a secret: anyone who has it can post to your channel. Putting it in a Lambda environment variable leaves it visible in the console and in get-function-configuration. Instead you store it in SSM Parameter Store as a SecureString and let the function read it at runtime.
Steps
Create an incoming webhook in Slack (Slack app, then Incoming Webhooks, then add to a test channel). Then read it into a shell variable without it landing in your shell history:
read -rs SLACK_WEBHOOKaws ssm put-parameter \ --name /security/slack-webhook-url \ --type SecureString \ --value "$SLACK_WEBHOOK" \ --region $AWS_REGION unset SLACK_WEBHOOKCommon MistakePasting the webhook URL into your README, a screenshot, or a Git commit. If it leaks, delete the webhook in Slack and create a new one.
Milestone 4: Write the Lambda Remediation Function
Concept
The function does three things per finding: contain the resource, tag what happened and why, and notify. It is written to be safe to run unattended:
DRY_RUNlogs what it would do without changing anything. You start with it on.- Idempotent: a second finding for an already-quarantined instance is skipped, not re-processed.
- Opt-out tag: an instance tagged
SecurityAutoResponse=exemptis never auto-isolated. A human decides. - Narrow blast radius: for access-key findings it deactivates only the key named in the finding, not every key the user has.
- Deactivate, never delete: deactivation is reversible and keeps the audit trail.
Findings about assumed-role or instance-profile credentials are not handled automatically here. Those need a different response (for example a deny policy on the role session), so the function reports "manual review" to Slack instead of guessing.
Steps
mkdir -p lambdacat > lambda/remediation.py << 'EOF'import jsonimport loggingimport osimport urllib.requestfrom datetime import datetime, timezone import boto3from botocore.exceptions import ClientError logger = logging.getLogger()logger.setLevel(logging.INFO) QUARANTINE_SG_NAME = os.environ.get('QUARANTINE_SG_NAME', 'QUARANTINE-deny-all')SLACK_WEBHOOK_PARAM = os.environ['SLACK_WEBHOOK_PARAM']DRY_RUN = os.environ.get('DRY_RUN', 'true').lower() == 'true'SEVERITY_THRESHOLD = float(os.environ.get('SEVERITY_THRESHOLD', '7.0'))EXEMPT_TAG_KEY = 'SecurityAutoResponse' ssm = boto3.client('ssm')_slack_url = None def now_iso(): return datetime.now(timezone.utc).isoformat() def get_slack_url(): """Read the webhook once per container, not once per invocation.""" global _slack_url if _slack_url is None: resp = ssm.get_parameter(Name=SLACK_WEBHOOK_PARAM, WithDecryption=True) _slack_url = resp['Parameter']['Value'] return _slack_url def lambda_handler(event, context): """GuardDuty findings arrive from EventBridge as event['detail'].""" finding = event.get('detail', {}) finding_id = finding.get('id', 'unknown') finding_type = finding.get('type', 'Unknown') severity = float(finding.get('severity', 0)) account_id = finding.get('accountId', 'unknown') region = finding.get('region', os.environ.get('AWS_REGION', 'unknown')) # Log identifiers only - the full finding can contain sensitive details logger.info("Finding %s type=%s severity=%s dry_run=%s", finding_id, finding_type, severity, DRY_RUN) if severity < SEVERITY_THRESHOLD: logger.info("Severity below threshold - skipping") return {'status': 'skipped', 'reason': 'below_severity_threshold'} resource = finding.get('resource', {}) resource_type = resource.get('resourceType', 'Unknown') actions = [] if resource_type == 'Instance': instance_id = resource.get('instanceDetails', {}).get('instanceId') if instance_id: actions.extend(isolate_ec2_instance(instance_id, finding_id, finding_type, region)) else: actions.append("No instance ID in finding - manual review needed") elif resource_type == 'AccessKey': details = resource.get('accessKeyDetails', {}) if details.get('userType') == 'IAMUser' and details.get('userName') and details.get('accessKeyId'): actions.extend(deactivate_access_key( details['userName'], details['accessKeyId'], finding_id, finding_type)) else: actions.append( f"No automatic action for principal type {details.get('userType')} - manual review needed") else: actions.append(f"No automatic action for resource type {resource_type} - manual review needed") send_slack_alert(finding_type, severity, finding_id, account_id, region, resource_type, actions) failed = any(a.startswith('ERROR') for a in actions) return {'status': 'error' if failed else 'handled', 'finding_id': finding_id, 'dry_run': DRY_RUN, 'actions': actions} def isolate_ec2_instance(instance_id, finding_id, finding_type, region): ec2 = boto3.client('ec2', region_name=region) try: reservations = ec2.describe_instances(InstanceIds=[instance_id])['Reservations'] instance = reservations[0]['Instances'][0] tags = {t['Key']: t['Value'] for t in instance.get('Tags', [])} if tags.get(EXEMPT_TAG_KEY) == 'exempt': return [f"SKIPPED {instance_id}: tagged {EXEMPT_TAG_KEY}=exempt - human decision required"] if tags.get('SECURITY_STATUS') == 'QUARANTINED': return [f"SKIPPED {instance_id}: already quarantined"] # The quarantine group must live in the same VPC as the instance vpc_id = instance['VpcId'] groups = ec2.describe_security_groups(Filters=[ {'Name': 'group-name', 'Values': [QUARANTINE_SG_NAME]}, {'Name': 'vpc-id', 'Values': [vpc_id]}, ])['SecurityGroups'] if not groups: return [f"ERROR {instance_id}: no security group named {QUARANTINE_SG_NAME} in {vpc_id}"] quarantine_sg = groups[0]['GroupId'] original_sgs = [sg['GroupId'] for sg in instance['SecurityGroups']] if DRY_RUN: return [f"DRY RUN: would replace security groups {original_sgs} on {instance_id} with {quarantine_sg}"] # Record the original groups BEFORE changing anything, so rollback is always possible ec2.create_tags(Resources=[instance_id], Tags=[ {'Key': 'ORIGINAL_SGS', 'Value': ','.join(original_sgs)}, {'Key': 'FINDING_ID', 'Value': finding_id}, {'Key': 'FINDING_TYPE', 'Value': finding_type}, ]) ec2.modify_instance_attribute(InstanceId=instance_id, Groups=[quarantine_sg]) ec2.create_tags(Resources=[instance_id], Tags=[ {'Key': 'SECURITY_STATUS', 'Value': 'QUARANTINED'}, {'Key': 'QUARANTINE_TIME', 'Value': now_iso()}, ]) return [f"Quarantined {instance_id}: security groups {original_sgs} replaced with {quarantine_sg}"] except (ClientError, IndexError, KeyError) as e: logger.error("Failed to isolate %s: %s", instance_id, e) return [f"ERROR isolating {instance_id}: {e}"] def deactivate_access_key(user_name, access_key_id, finding_id, finding_type): iam = boto3.client('iam') try: if DRY_RUN: return [f"DRY RUN: would deactivate key {access_key_id} for user {user_name}"] iam.update_access_key(UserName=user_name, AccessKeyId=access_key_id, Status='Inactive') iam.tag_user(UserName=user_name, Tags=[ {'Key': 'SECURITY_STATUS', 'Value': 'KEY_DEACTIVATED'}, {'Key': 'DEACTIVATED_KEY_ID', 'Value': access_key_id}, {'Key': 'FINDING_ID', 'Value': finding_id}, {'Key': 'FINDING_TYPE', 'Value': finding_type}, ]) return [f"Deactivated access key {access_key_id} for user {user_name}"] except ClientError as e: logger.error("Failed to deactivate key for %s: %s", user_name, e) return [f"ERROR deactivating key for {user_name}: {e}"] def send_slack_alert(finding_type, severity, finding_id, account_id, region, resource_type, actions): if severity >= 9.0: emoji, color, label = ":rotating_light:", "#FF0000", "CRITICAL" elif severity >= 7.0: emoji, color, label = ":warning:", "#FF8C00", "HIGH" else: emoji, color, label = ":information_source:", "#FFA500", "MEDIUM" mode = " (DRY RUN)" if DRY_RUN else "" actions_text = "\n".join(f"- {a}" for a in actions) message = { "text": f"{emoji} GuardDuty {label} finding{mode}", "attachments": [{ "color": color, "blocks": [ {"type": "header", "text": {"type": "plain_text", "text": f"Security Alert - {label}{mode}"}}, {"type": "section", "fields": [ {"type": "mrkdwn", "text": f"*Finding Type:*\n{finding_type}"}, {"type": "mrkdwn", "text": f"*Severity:*\n{severity}/10"}, {"type": "mrkdwn", "text": f"*Account:*\n{account_id}"}, {"type": "mrkdwn", "text": f"*Region:*\n{region}"}, {"type": "mrkdwn", "text": f"*Resource Type:*\n{resource_type}"}, {"type": "mrkdwn", "text": f"*Finding ID:*\n{finding_id}"}, ]}, {"type": "section", "text": {"type": "mrkdwn", "text": f"*Actions:*\n{actions_text}"}}, ], }], } req = urllib.request.Request( get_slack_url(), data=json.dumps(message).encode('utf-8'), headers={'Content-Type': 'application/json'}, ) try: with urllib.request.urlopen(req, timeout=10) as resp: logger.info("Slack notification sent: %s", resp.status) except Exception as e: # a Slack outage must never undo a successful remediation logger.error("Failed to send Slack notification: %s", e)EOF cd lambdazip function.zip remediation.pycd ..Common MistakeDeleting a compromised access key instead of deactivating it. Deletion is irreversible and destroys evidence. If the finding is a false positive, or you need to trace what the credential accessed, a deactivated key is far more useful.
Milestone 5: Create the Lambda IAM Role
Concept
The responder is itself a high-value target: it can change security groups and disable credentials. So its permissions are scoped as tightly as the APIs allow. Read-only EC2 describe calls need *. The write actions are limited to instances and security groups in your account and region, to IAM users in your account, and to a single SSM parameter.
Steps
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) cat > trust-policy.json << 'EOF'{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole" }]}EOF ROLE_ARN=$(aws iam create-role \ --role-name guardduty-remediation-role \ --assume-role-policy-document file://trust-policy.json \ --query 'Role.Arn' \ --output text) aws iam attach-role-policy \ --role-name guardduty-remediation-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRoleThe policy file below is not quoted (EOF without quotes), so $ACCOUNT_ID and $AWS_REGION are filled in by the shell:
cat > remediation-policy.json << EOF{ "Version": "2012-10-17", "Statement": [ { "Sid": "ReadEc2", "Effect": "Allow", "Action": ["ec2:DescribeInstances", "ec2:DescribeSecurityGroups"], "Resource": "*" }, { "Sid": "QuarantineInstances", "Effect": "Allow", "Action": ["ec2:ModifyInstanceAttribute", "ec2:CreateTags"], "Resource": [ "arn:aws:ec2:$AWS_REGION:$ACCOUNT_ID:instance/*", "arn:aws:ec2:$AWS_REGION:$ACCOUNT_ID:security-group/*" ] }, { "Sid": "DeactivateKeys", "Effect": "Allow", "Action": ["iam:UpdateAccessKey", "iam:TagUser"], "Resource": "arn:aws:iam::$ACCOUNT_ID:user/*" }, { "Sid": "ReadSlackWebhook", "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:$AWS_REGION:$ACCOUNT_ID:parameter/security/slack-webhook-url" } ]}EOF aws iam put-role-policy \ --role-name guardduty-remediation-role \ --policy-name remediation-permissions \ --policy-document file://remediation-policy.jsonTipIf a call later fails with
AccessDenied, the error message names the action and resource that was denied. Read it before widening the policy. Widen only the one statement that needs it.
Security
iam:UpdateAccessKeyonuser/*is still broad. For a production version, add a condition or limit it to a path prefix (for exampleuser/service-accounts/*) so the responder cannot disable your break-glass admin user.
Common MistakePutting
Resource: "*"on every statement just to makeAccessDeniederrors disappear. Read which action and resource were denied, then widen only that statement.
Milestone 6: Deploy the Lambda Function
Concept
IAM is eventually consistent: the role you just created may not be visible to the Lambda service for a few seconds. The function starts with DRY_RUN=true, so nothing is changed until you have seen the logs and Slack message and are happy with them.
Steps
## Wait for IAM role propagationsleep 10 aws lambda create-function \ --function-name guardduty-remediation \ --runtime python3.12 \ --role $ROLE_ARN \ --handler remediation.lambda_handler \ --zip-file fileb://lambda/function.zip \ --timeout 30 \ --environment "Variables={QUARANTINE_SG_NAME=QUARANTINE-deny-all,SLACK_WEBHOOK_PARAM=/security/slack-webhook-url,DRY_RUN=true,SEVERITY_THRESHOLD=7.0}" \ --region $AWS_REGION aws lambda wait function-active-v2 \ --function-name guardduty-remediation \ --region $AWS_REGIONWhen you change the code later, redeploy with:
aws lambda update-function-code \ --function-name guardduty-remediation \ --zip-file fileb://lambda/function.zip \ --region $AWS_REGIONCommon MistakeSkipping the
sleep. Creating the function immediately after the role often fails with an error that the role cannot be assumed, because the role has not propagated yet. Wait and retry.
TipThe runtime name
python3.12was current when this project was written. Lambda retires old runtimes over time, so check the Lambda runtimes page and use a currently supported Python version if this one is rejected.
Milestone 7: Create the EventBridge Rule
Concept
The event pattern decides which findings ever reach your Lambda. GuardDuty severity is a number (0.1 to 10.0), not a label, so the pattern needs a numeric comparison. Common bands are Low (below 4.0), Medium (4.0-6.9) and High (7.0-8.9). Newer GuardDuty documentation also describes a Critical band at 9.0 and above. Confirm the current bands in the GuardDuty docs. The threshold of 7.0 used here catches High and anything above it.
The severity threshold is checked twice, once in the rule and once in the Lambda. That is intentional: if someone loosens the rule later, the function still refuses to act on low-severity findings.
Steps
aws events put-rule \ --name guardduty-high-severity-findings \ --event-pattern '{ "source": ["aws.guardduty"], "detail-type": ["GuardDuty Finding"], "detail": { "severity": [{"numeric": [">=", 7.0]}] } }' \ --state ENABLED \ --description "Route High-severity GuardDuty findings to the remediation Lambda" \ --region $AWS_REGION LAMBDA_ARN=$(aws lambda get-function \ --function-name guardduty-remediation \ --region $AWS_REGION \ --query 'Configuration.FunctionArn' --output text) aws events put-targets \ --rule guardduty-high-severity-findings \ --targets "Id=RemediationLambda,Arn=$LAMBDA_ARN" \ --region $AWS_REGION aws lambda add-permission \ --function-name guardduty-remediation \ --statement-id eventbridge-invoke \ --action lambda:InvokeFunction \ --principal events.amazonaws.com \ --source-arn "arn:aws:events:$AWS_REGION:$ACCOUNT_ID:rule/guardduty-high-severity-findings" \ --region $AWS_REGIONCommon MistakeMatching severity as a string.
{"severity": ["HIGH"]}never matches, because GuardDuty sends a number. Use{"numeric": [">=", 7.0]}.
Validation & Testing
Verification Steps
Test in three stages, from safest to most realistic. Do not skip stage 1.
Stage 1: Dry Run Against a Disposable Instance
Launch a throwaway instance (it runs in your default VPC, where you created the quarantine group):
INSTANCE_ID=$(aws ec2 run-instances \ --image-id resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \ --instance-type t3.micro \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=guardduty-test-victim}]' \ --query 'Instances[0].InstanceId' --output text \ --region $AWS_REGION)aws ec2 wait instance-running --instance-ids $INSTANCE_ID --region $AWS_REGIONecho $INSTANCE_IDBuild a fake finding that points at it:
cat > test-event.json << EOF{ "detail": { "id": "test-finding-0001", "type": "Backdoor:EC2/C&CActivity.B!DNS", "severity": 8.0, "accountId": "$ACCOUNT_ID", "region": "$AWS_REGION", "resource": { "resourceType": "Instance", "instanceDetails": {"instanceId": "$INSTANCE_ID"} } }}EOF aws lambda invoke \ --function-name guardduty-remediation \ --cli-binary-format raw-in-base64-out \ --payload file://test-event.json \ --region $AWS_REGION \ response.json && cat response.json{"status": "handled", "finding_id": "test-finding-0001", "dry_run": true, "actions": ["DRY RUN: would replace security groups ['sg-...'] on i-... with sg-..."]}Check Slack for the alert marked (DRY RUN), and confirm the instance's security groups did not change.
Stage 2: Real Isolation
Turn dry-run off. Note that update-function-configuration replaces the whole variable set, so resend every variable:
aws lambda update-function-configuration \ --function-name guardduty-remediation \ --environment "Variables={QUARANTINE_SG_NAME=QUARANTINE-deny-all,SLACK_WEBHOOK_PARAM=/security/slack-webhook-url,DRY_RUN=false,SEVERITY_THRESHOLD=7.0}" \ --region $AWS_REGION aws lambda wait function-updated-v2 --function-name guardduty-remediation --region $AWS_REGION aws lambda invoke \ --function-name guardduty-remediation \ --cli-binary-format raw-in-base64-out \ --payload file://test-event.json \ --region $AWS_REGION \ response.json && cat response.jsonVerify the instance is quarantined and tagged:
aws ec2 describe-instances --instance-ids $INSTANCE_ID --region $AWS_REGION \ --query 'Reservations[0].Instances[0].{SGs:SecurityGroups[*].GroupId,Tags:Tags}'You should see only the quarantine group, plus the SECURITY_STATUS, QUARANTINE_TIME, FINDING_ID, FINDING_TYPE and ORIGINAL_SGS tags. Invoke the function a second time and confirm the response says SKIPPED ... already quarantined (idempotency).
Stage 3: End to End Through GuardDuty and EventBridge
Generate a sample finding:
aws guardduty create-sample-findings \ --detector-id $DETECTOR_ID \ --finding-types 'Backdoor:EC2/C&CActivity.B!DNS' \ --region $AWS_REGIONTipUse single quotes around finding types. The
!character triggers history expansion inside double quotes in interactive bash.
Then read the Lambda logs:
aws logs tail /aws/lambda/guardduty-remediation --since 15m --region $AWS_REGIONSample findings use synthetic instance IDs that do not exist in your account, so the EC2 step will log an ERROR ... InvalidInstanceID.NotFound action. That is expected. What you are proving is that the whole chain fires: GuardDuty to EventBridge to Lambda to Slack. The sample finding's severity depends on its type, so confirm in the log line that it was at or above 7.0. If nothing arrives, check the rule state and the Lambda permission first.
Measure Your Response Time
Compare the QUARANTINE_TIME tag from Stage 2 against the time you invoked the function, and look at the Duration line in the CloudWatch REPORT entry. Write the numbers you measured in your README. Do not copy a number from anyone else.
Rollback Runbook
A false positive that isolates a production instance is an outage. Practise the rollback now, not during an incident:
## 1. Read the original security groups from the tagORIG=$(aws ec2 describe-tags \ --filters Name=resource-id,Values=$INSTANCE_ID Name=key,Values=ORIGINAL_SGS \ --query 'Tags[0].Value' --output text --region $AWS_REGION) ## 2. Restore them (the tag stores a comma-separated list)aws ec2 modify-instance-attribute \ --instance-id $INSTANCE_ID \ --groups ${ORIG//,/ } \ --region $AWS_REGION ## 3. Clear the quarantine marker so the function can act again if a new finding arrivesaws ec2 delete-tags --resources $INSTANCE_ID \ --tags Key=SECURITY_STATUS Key=QUARANTINE_TIME \ --region $AWS_REGIONFor a deactivated key, re-enable it with aws iam update-access-key --user-name <user> --access-key-id <key> --status Active, but only after a human has confirmed the finding was a false positive.
Cleanup
Remove everything so you are not billed for it:
aws events remove-targets --rule guardduty-high-severity-findings --ids RemediationLambda --region $AWS_REGIONaws events delete-rule --name guardduty-high-severity-findings --region $AWS_REGIONaws lambda delete-function --function-name guardduty-remediation --region $AWS_REGIONaws iam delete-role-policy --role-name guardduty-remediation-role --policy-name remediation-permissionsaws iam detach-role-policy --role-name guardduty-remediation-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRoleaws iam delete-role --role-name guardduty-remediation-roleaws ssm delete-parameter --name /security/slack-webhook-url --region $AWS_REGIONaws ec2 terminate-instances --instance-ids $INSTANCE_ID --region $AWS_REGIONaws ec2 wait instance-terminated --instance-ids $INSTANCE_ID --region $AWS_REGIONaws ec2 delete-security-group --group-id $QUARANTINE_SG --region $AWS_REGIONaws guardduty delete-detector --detector-id $DETECTOR_ID --region $AWS_REGIONCommon MistakeDeleting the security group before terminating the instance that uses it. The delete fails with a dependency error. Terminate the instance first.
Common Mistakes Recap
| Mistake | Why It Breaks | Fix |
|---|---|---|
| Default outbound rule left on the quarantine group | The instance can still send data out | Revoke the allow-all egress rule |
| Quarantine group created in only one VPC | Instances in other VPCs cannot use it | Create a group with the same name in each VPC |
| String match on severity | Rule never fires | Use a numeric comparison |
No sleep after role creation |
Role cannot be assumed yet | Wait a few seconds and retry |
Missing lambda add-permission |
Rule matches but Lambda never runs | Grant EventBridge invoke permission |
update-function-configuration with only one variable |
All other variables are wiped | Always resend the full variable set |
| Slack webhook in an env var or Git | Secret is exposed | Keep it in SSM SecureString |
| Deleting keys or terminating instances | Destroys forensic evidence | Deactivate and isolate first |
| Never testing rollback | A false positive becomes a long outage | Practise the runbook before you need it |
Portfolio Deliverable
Publish a repo containing: lambda/remediation.py, the CLI steps as a README, an architecture diagram, a screenshot of a real Slack alert (webhook URL removed), the response time you measured, and the rollback runbook. In the README add a short "limitations" section that covers connection tracking, single-region scope, and the credential types the function does not handle. Hiring managers value honest limits.
Extensions
- Snapshot the instance's EBS volumes before isolation, so evidence is preserved even if someone later stops the instance
- Add a network ACL step for faster cut-off of established connections
- Add an SQS dead-letter queue so failed invocations are not lost
- Rebuild the whole stack in Terraform
- Enable GuardDuty across an AWS Organization with a delegated administrator