Automate file transfer from Amazon S3 to FSx for OpenZFS so that every file uploaded to a specific S3 path automatically appears on FSx - no manual commands needed after setup. --- ### What You Will Learn * What dummy values are used throughout this guide and where to replace them * Why EventBridge + DataSync is the best approach for FSx OpenZFS * How to compare all available solutions so you can make an informed choice * How to set up the full pipeline using Terraform (infrastructure as code) * How to verify the setup using AWS Console and CLI * How to test that a file uploaded to S3 automatically lands on FSx --- ### Why This Matters In real DevOps and data engineering workflows, teams store raw files in S3 (cheap, durable, scalable) but applications and compute clusters need those files on a fast, POSIX-compliant file system like FSx for OpenZFS. Manually copying files every time is error-prone and does not scale. This guide eliminates that manual step entirely using a fully automated, event-driven AWS-native pipeline. ---
> 📌 **Remember:** Every value in the table below is a placeholder. Before running any command or applying any Terraform code, replace each dummy value with your real one. | What | Dummy Value Used | Where to Find / Replace | |---|---|---| | AWS Account ID | `123456789012` | AWS Console top-right corner | | AWS Region | `us-east-1` | Your preferred AWS region | | S3 Bucket Name | `mycompany-data-bucket` | You choose this name | | S3 Folder/Prefix | `uploads/incoming/` | The folder you upload files to in S3 | | FSx File System ID | `fs-0abc1234def56789` | Auto-generated when FSx is created | | FSx Volume Path | `/fsx/incoming` | Mount path on the FSx volume | | FSx Subnet ID | `subnet-0a1b2c3d4e5f` | Your VPC subnet (same AZ as FSx) | | VPC ID | `vpc-0123456789abcdef0` | Your existing VPC | | Security Group ID | `sg-0abc123456def7890` | Created during setup | | DataSync S3 Location ARN | `arn:aws:datasync:us-east-1:123456789012:location/loc-src` | Auto-generated | | DataSync FSx Location ARN | `arn:aws:datasync:us-east-1:123456789012:location/loc-dst` | Auto-generated | | DataSync Task ARN | `arn:aws:datasync:us-east-1:123456789012:task/task-abc123` | Auto-generated | | IAM Role — DataSync S3 | `datasync-s3-access-role` | Created during setup | | IAM Role — EventBridge | `eventbridge-datasync-role` | Created during setup | | Terraform State Bucket | `mycompany-tf-state` | You choose this name | | Project Tag | `data-pipeline` | Your project name | | Environment Tag | `production` | Your environment label | ---
> 📌 **Remember:** FSx Lustre has a native S3 link feature — but that only works for Lustre. Since we are using OpenZFS, we need a different approach. | Solution | Works with OpenZFS | Auto on S3 Upload | No Lambda | Production Ready | Verdict | |---|---|---|---|---|---| | EventBridge + DataSync | Yes | Yes | Yes | Yes | **Best — Use This** | | FSx Lustre S3 Native Link | No — Lustre only | Yes | Yes | Yes | Not for OpenZFS | | DataSync Scheduled Only | Yes | No — runs on a timer | Yes | Yes | OK but adds delay | | Lambda + S3 Trigger | Yes | Yes | No | Limited by 15 min timeout | Not recommended | | EC2 Cron Job | Yes | No — scheduled only | Yes | Fragile | Manual effort | | rclone on EC2 | Yes | Needs extra setup | No — EC2 always on | Self-managed | High maintenance | | AWS Transfer Family | Indirect only | No | Yes | Yes | Overkill for this | **Winner: EventBridge + DataSync** — natively supported, no Lambda, no EC2, real-time trigger on every S3 upload, handles retries and failures automatically. ---
You upload a file to S3 | v S3 fires an Object Created event | v Amazon EventBridge receives the event and matches it to your rule | v EventBridge triggers DataSync task execution automatically | v DataSync copies the file from S3 path --> FSx for OpenZFS path | v File is available on FSx No manual step needed ---
> 📌 **Remember:** Terraform manages all infrastructure as code. Run this once and everything is created automatically. You never need to click through the console for setup. ### Prerequisites Before running Terraform, make sure you have: * AWS CLI installed and configured (`aws configure`) * Terraform installed (v1.5 or above) * An existing VPC and subnet in `us-east-1` * FSx for OpenZFS file system already created (or use the Terraform block below to create it) --- ### Project Folder Structure ```bash s3-to-fsx-pipeline/ ├── main.tf ├── variables.tf ├── outputs.tf ├── iam.tf ├── datasync.tf ├── eventbridge.tf └── terraform.tfvars ``` --- ### variables.tf ```hcl variable "aws_region" { default = "us-east-1" } variable "account_id" { default = "123456789012" } variable "s3_bucket_name" { default = "mycompany-data-bucket" } variable "s3_prefix" { default = "/uploads/incoming/" } variable "fsx_subnet_id" { default = "subnet-0a1b2c3d4e5f" } variable "vpc_id" { default = "vpc-0123456789abcdef0" } variable "fsx_storage_capacity" { default = 64 } variable "project" { default = "data-pipeline" } variable "environment" { default = "production" } ``` --- ### main.tf ```hcl terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } backend "s3" { bucket = "mycompany-tf-state" key = "s3-to-fsx/terraform.tfstate" region = "us-east-1" } } provider "aws" { region = var.aws_region } resource "aws_s3_bucket" "data_bucket" { bucket = var.s3_bucket_name tags = { Project = var.project Environment = var.environment } } resource "aws_s3_bucket_notification" "bucket_eventbridge" { bucket = aws_s3_bucket.data_bucket.id eventbridge = true } resource "aws_security_group" "fsx_sg" { name = "fsx-openzfs-sg" description = "Allow NFS traffic to FSx OpenZFS" vpc_id = var.vpc_id ingress { description = "NFS TCP" from_port = 2049 to_port = 2049 protocol = "tcp" cidr_blocks = ["10.0.0.0/8"] } ingress { description = "NFS UDP" from_port = 2049 to_port = 2049 protocol = "udp" cidr_blocks = ["10.0.0.0/8"] } egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } tags = { Project = var.project Environment = var.environment } } resource "aws_fsx_openzfs_file_system" "main" { storage_capacity = var.fsx_storage_capacity subnet_ids = [var.fsx_subnet_id] deployment_type = "SINGLE_AZ_1" throughput_capacity = 64 security_group_ids = [aws_security_group.fsx_sg.id] root_volume_configuration { data_compression_type = "LZ4" } tags = { Name = "fsx-openzfs-main" Project = var.project Environment = var.environment } } ``` --- ### iam.tf ```hcl resource "aws_iam_role" "datasync_s3_role" { name = "datasync-s3-access-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Service = "datasync.amazonaws.com" } Action = "sts:AssumeRole" }] }) } resource "aws_iam_role_policy" "datasync_s3_policy" { name = "datasync-s3-policy" role = aws_iam_role.datasync_s3_role.id policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Action = [ "s3:GetBucketLocation", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:GetObject", "s3:GetObjectTagging", "s3:GetObjectVersion" ] Resource = [ "arn:aws:s3:::${var.s3_bucket_name}", "arn:aws:s3:::${var.s3_bucket_name}/*" ] } ] }) } resource "aws_iam_role" "eventbridge_datasync_role" { name = "eventbridge-datasync-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Service = "events.amazonaws.com" } Action = "sts:AssumeRole" }] }) } resource "aws_iam_role_policy" "eventbridge_datasync_policy" { name = "eventbridge-datasync-policy" role = aws_iam_role.eventbridge_datasync_role.id policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Action = "datasync:StartTaskExecution" Resource = aws_datasync_task.s3_to_fsx.arn }] }) } ``` --- ### datasync.tf ```hcl resource "aws_datasync_location_s3" "source" { s3_bucket_arn = aws_s3_bucket.data_bucket.arn subdirectory = var.s3_prefix s3_config { bucket_access_role_arn = aws_iam_role.datasync_s3_role.arn } tags = { Project = var.project Environment = var.environment } } resource "aws_datasync_location_fsx_openzfs_file_system" "destination" { fsx_filesystem_arn = aws_fsx_openzfs_file_system.main.arn security_group_arns = [aws_security_group.fsx_sg.arn] protocol { nfs { mount_options { version = "AUTOMATIC" } } } tags = { Project = var.project Environment = var.environment } } resource "aws_datasync_task" "s3_to_fsx" { name = "s3-to-fsx-openzfs-task" source_location_arn = aws_datasync_location_s3.source.arn destination_location_arn = aws_datasync_location_fsx_openzfs_file_system.destination.arn options { overwrite_mode = "ALWAYS" transfer_mode = "CHANGED" preserve_deleted_files = "REMOVE" verify_mode = "ONLY_FILES_TRANSFERRED" log_level = "TRANSFER" } tags = { Project = var.project Environment = var.environment } } ``` --- ### eventbridge.tf ```hcl resource "aws_cloudwatch_event_rule" "s3_upload_trigger" { name = "s3-upload-trigger-datasync" description = "Fires when a file is uploaded to the S3 incoming prefix" event_pattern = jsonencode({ source = ["aws.s3"] detail-type = ["Object Created"] detail = { bucket = { name = [var.s3_bucket_name] } object = { key = [{ prefix = "uploads/incoming/" }] } } }) tags = { Project = var.project Environment = var.environment } } resource "aws_cloudwatch_event_target" "datasync_target" { rule = aws_cloudwatch_event_rule.s3_upload_trigger.name arn = aws_datasync_task.s3_to_fsx.arn role_arn = aws_iam_role.eventbridge_datasync_role.arn } ``` --- ### outputs.tf ```hcl output "s3_bucket_name" { value = aws_s3_bucket.data_bucket.id } output "fsx_filesystem_id" { value = aws_fsx_openzfs_file_system.main.id } output "fsx_filesystem_arn" { value = aws_fsx_openzfs_file_system.main.arn } output "datasync_task_arn" { value = aws_datasync_task.s3_to_fsx.arn } output "eventbridge_rule_name" { value = aws_cloudwatch_event_rule.s3_upload_trigger.name } ``` --- ### terraform.tfvars ```hcl aws_region = "us-east-1" account_id = "123456789012" s3_bucket_name = "mycompany-data-bucket" s3_prefix = "/uploads/incoming/" fsx_subnet_id = "subnet-0a1b2c3d4e5f" vpc_id = "vpc-0123456789abcdef0" fsx_storage_capacity = 64 project = "data-pipeline" environment = "production" ``` --- ### Run Terraform ```bash cd s3-to-fsx-pipeline terraform init terraform plan terraform apply ``` > 💡 **Tip:** Always run `terraform plan` before `terraform apply` to review exactly what AWS resources will be created, changed, or destroyed. ---
Use this if you prefer the visual interface over Terraform. ### Step 1 — Create S3 Bucket * Go to **S3 Console** → Click **Create bucket** * Bucket name: `mycompany-data-bucket` * Region: `us-east-1` * Leave all other settings default → Click **Create bucket** * Inside the bucket, create a folder called `uploads/incoming/` ### Step 2 — Enable EventBridge on S3 * Open your bucket → Go to **Properties** tab * Scroll to **Amazon EventBridge** section * Click **Edit** → Toggle **Send notifications to Amazon EventBridge** to **On** * Click **Save changes** > 📌 **Remember:** Without this step, S3 upload events will never reach EventBridge and the automation will not work. ### Step 3 — Create FSx for OpenZFS * Go to **FSx Console** → Click **Create file system** * Choose **Amazon FSx for OpenZFS** → Click **Next** * Deployment type: **Single-AZ** * Storage capacity: **64 GiB** * VPC: select `vpc-0123456789abcdef0` * Subnet: select `subnet-0a1b2c3d4e5f` * Security group: create or select one that allows NFS port 2049 * Click **Next** → Review → **Create file system** * Wait 5-10 minutes for status to become **Available** ### Step 4 — Create DataSync Locations **Source — S3:** * Go to **DataSync Console** → **Locations** → **Create location** * Location type: **Amazon S3** * S3 bucket: `mycompany-data-bucket` * S3 folder: `uploads/incoming/` * IAM role: select `datasync-s3-access-role` * Click **Create location** **Destination — FSx OpenZFS:** * Click **Create location** again * Location type: **Amazon FSx for OpenZFS** * File system: select your FSx file system * Mount path: `/fsx/incoming` * Security groups: select your FSx security group * Protocol: **NFS** → Version: **Automatic** * Click **Create location** ### Step 5 — Create DataSync Task * Go to **DataSync Console** → **Tasks** → **Create task** * Source: select the S3 location you just created * Destination: select the FSx OpenZFS location * Task name: `s3-to-fsx-openzfs-task` * Transfer mode: **Transfer only changed files** * Overwrite mode: **Always** * Deleted files: **Remove from destination** * Click **Create task** and copy the **Task ARN** ### Step 6 — Create EventBridge Rule * Go to **EventBridge Console** → **Rules** → **Create rule** * Name: `s3-upload-trigger-datasync` * Event bus: **default** * Rule type: **Rule with an event pattern** * AWS service: **S3** → Event type: **Object Created** * Specific bucket: `mycompany-data-bucket` * Prefix filter: `uploads/incoming/` * Click **Next** * Target: **AWS service** → **DataSync** * DataSync task ARN: paste the Task ARN from Step 5 * IAM role: select `eventbridge-datasync-role` * Click **Create rule** > 💡 **Tip:** The prefix filter on the EventBridge rule ensures only files uploaded to `uploads/incoming/` trigger DataSync — not every upload to the entire bucket. ---
Automate file transfer from Amazon S3 to FSx for OpenZFS so that every file uploaded to a specific S3 path automatically...
> 📌 Remember: Every value in the table below is a placeholder. Before running any command or applying any Terraform cod...
> 📌 Remember: FSx Lustre has a native S3 link feature — but that only works for Lustre. Since we are using OpenZFS, we ...
You upload a file to S3 v S3 fires an Object Created event v Amazon EventBridge receives the event and matches it to you...
> 📌 Remember: Terraform manages all infrastructure as code. Run this once and everything is created automatically. You ...
Use this if you prefer the visual interface over Terraform. Step 1 — Create S3 Bucket Go to S3 Console → Click Create bu...
Use this if you want to set things up without Terraform and without the console. Step 1 — Create S3 Bucket Step 2 — Enab...
Command What It Does terraform init Downloads providers and sets up Terraform backend terraform plan Shows what will be ...
EventBridge not enabled on S3 — You create the EventBridge rule but forget to turn on EventBridge notifications on the S...
Follow these steps after setup to confirm everything is working. Upload a test file to the S3 incoming path: Go to Event...
Service What You Pay For Estimated Cost AWS DataSync $0.0125 per GB transferred 1 TB = $12.50 Amazon S3 $0.023 per GB pe...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.