### Why storage choice is a system design decision, not a config setting A 2 AM page reads: "uploads are inconsistent across servers." Three web servers behind a load balancer. Server 1 handled the upload, saved it locally. The next request lands on Server 2, which has never seen that file. This is not a bug in your code - it is a storage architecture mistake made weeks earlier, when someone picked local EBS volumes for a multi-server application instead of shared storage. AWS provides several storage and data-transfer options, each designed around a different access pattern - block, object, file, and migration/hybrid tooling. No single storage type fits every workload. Picking the wrong one does not just cost money. It creates outages that look like application bugs. * **Block storage** - a raw disk one server owns. Fast, private, tied to one instance. * **Object storage** - files accessed over HTTP as flat, keyed items. Massively scalable, without traditional filesystem directories. * **File storage** - a shared filesystem multiple servers mount at once. Looks like a local directory to every client. * **Migration and hybrid tools** - moving huge datasets when the internet is too slow, or bridging on-premises storage to AWS. > 📌 **Remember:** The question is never "which storage service is best." It is "what access pattern does this specific workload need." A database needs block storage. A shared upload folder needs file storage. A static website needs object storage. Get the pattern right first. ### The AWS storage decision tree Every storage decision on AWS starts with three questions: how many servers need to read this data, does it need to persist past instance termination, and how fast does access need to be. Does ONE EC2 instance own this data exclusively? Yes -> Block storage Must survive instance stop/terminate? -> EBS Pure scratch space, fastest possible? -> Instance Store Do MULTIPLE servers need the SAME files at once? Yes -> File storage Linux servers? -> EFS Windows servers? -> FSx for Windows File Server HPC/ML workload? -> FSx for Lustre Is this accessed via URL/API, not mounted as a disk? Yes -> Object storage -> S3 Is this cold data you rarely touch, kept for years? Yes -> S3 Glacier tiers (still S3, different storage class) Is this on-premises data that needs to reach AWS? Too slow over the network (100 TB+, week+ transfer time)? -> Snowball Ongoing scheduled sync? -> DataSync Legacy protocol (NFS/SMB/iSCSI) bridging to S3? -> Storage Gateway Partners sending files via FTP/SFTP? -> Transfer Family This module builds the mental model across all of these options. Full API-level depth for S3 lives in the AWS Hub - this module teaches when and why to reach for each service, and the production patterns that connect them.
### What makes S3 fundamentally different from a filesystem S3 is not a hard drive in the cloud. It has no real directory tree, no mount point, no file locking. Every object lives in a flat bucket, addressed by a key - a string that looks like a path but is really just a name with slashes in it. Object Storage (S3) File System (EFS/EBS) ------------------------ ------------------------ Stored as flat objects Stored in real directories Accessed via HTTP/HTTPS API Accessed via mount point "Folders" are a UI illusion Folders are real structures Massively scalable, no Tied to provisioned capacity capacity provisioning needed This matters practically: you cannot `cd` into an S3 bucket from your application code the way you can `cd` into an EFS mount. Every read or write is an API call. That is the trade-off for massive scalability and durability without provisioning capacity ahead of time. > 💡 **Tip:** S3 buckets are created in a specific region and their names must be globally unique across every AWS account on Earth - not just your account. `mycompany-logs` might already be taken by someone else entirely. ### S3 storage classes and choosing by access pattern S3 is not one storage tier. It is a family of tiers with identical durability (eleven nines, 99.999999999%) but very different availability and cost - and availability is the real lever you are pulling. | Storage Class | Access Pattern | Retrieval | Min. Storage Duration | Best For | |:---|:---|:---|:---|:---| | S3 Standard | Frequent | Instant, free | None | Active application data | | S3 Standard-IA | Monthly or less | Instant, per-GB fee | 30 days | Backups needing fast recovery | | S3 One Zone-IA | Monthly or less, recreatable | Instant, per-GB fee | 30 days | Secondary copies, thumbnails | | S3 Glacier Instant | Quarterly | Milliseconds | 90 days | Rarely accessed but urgent when needed | | S3 Glacier Flexible | Yearly | Minutes to hours | 90 days | Long-term backups, compliance | | S3 Glacier Deep Archive | Once every 7-10 years | Standard 12h / Bulk 48h | 180 days | Regulatory retention, tape replacement | | S3 Intelligent-Tiering | Unknown/changing | Automatic | None | Unpredictable access patterns | Four things shape a real storage class decision, and they are not the same thing: * **Durability** - the odds your data survives at all. Identical across every class, at eleven nines (99.999999999%). Durability is never the differentiator. * **Availability** - how reliably the object can be served on demand, expressed as a percentage (for example 99.99% for Standard vs 99.5% for One Zone-IA). This does differ by class and is a real factor for latency-sensitive workloads. * **Resilience characteristics** - how many Availability Zones the class spans. Standard, Standard-IA, Glacier, and Intelligent-Tiering replicate across 3+ AZs. One Zone-IA deliberately stores in a single AZ, trading resilience for a lower price. * **Retrieval behavior and cost** - how fast you get the object back, and whether that speed costs extra. This is the main lever that separates the Glacier tiers from each other. Minimum storage duration is a separate cost trap worth calling out on its own: store a file in Glacier Deep Archive for one day and delete it, and you are still billed for the full 180-day minimum. These classes punish short-term use by design. > 🔴 **Common Mistake:** Choosing S3 One Zone-IA for data that cannot be recreated. One Zone-IA lives in a single Availability Zone - if that AZ is destroyed, the data is permanently gone. Only use it for data you can regenerate, like image thumbnails derived from an original still stored in Standard. ### Lifecycle policies that move data automatically Manually moving objects between storage classes does not scale. Lifecycle rules run on a schedule you define, based on object age. Upload log file to S3 Standard | After 30 days -> transition to Standard-IA | After 90 days -> transition to Glacier Flexible Retrieval | After 365 days -> expire (delete permanently) This is the standard pattern for application logs, compliance archives, and backups - active for a short window, then progressively cheaper as the odds of needing them drop. ```json { "Rules": [ { "ID": "logs-lifecycle", "Status": "Enabled", "Filter": { "Prefix": "logs/" }, "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" }, { "Days": 90, "StorageClass": "GLACIER" } ], "Expiration": { "Days": 365 } } ] } ``` > **Note:** The `Filter` field targets only objects under the `logs/` prefix - other objects in the same bucket are untouched by this rule. You can run multiple lifecycle rules on one bucket, each targeting a different prefix or tag. > 💡 **Tip:** Before writing lifecycle rules from a guess, run S3 Storage Class Analysis for 24-48 hours. It reports your actual access patterns as a CSV, so your transition days are based on real usage instead of assumption. ### Controlling who can access an S3 bucket S3 access control operates at two layers, and understanding which one to reach for prevents most security misconfigurations. IAM Policy (attached to a user or role) "Can THIS PRINCIPAL call THIS ACTION?" Use for: your own IAM users, EC2 instance roles, Lambda execution roles Bucket Policy (attached to the bucket itself) "Who is allowed to touch THIS BUCKET?" Use for: public access, cross-account access, enforcing encryption The starting point is separating "who is asking" from "what are they asking to touch": * Your own IAM user needs S3 access -> an identity-based (IAM) policy is typically sufficient. No bucket policy needed. * An EC2 instance needs S3 access -> attach an IAM role to the instance. No bucket policy needed. * A different AWS account needs access -> resource policies become necessary - a bucket policy naming that account's ARN as principal. * The public internet needs to read objects (static website) -> bucket policy with `Principal: "*"`. > **Note:** This module covers the common cases. A real access decision can also involve permissions boundaries, session policies, Service Control Policies at the AWS Organizations level, VPC endpoint policies, and KMS key policies if the object is encrypted with a customer-managed key. The IAM module covers the full evaluation logic in depth - this module focuses on the S3-specific patterns. > ⚠️ **Security:** An explicit `Deny` in any applicable policy - identity-based or resource-based - always wins, even when another policy explicitly allows the same action. Block Public Access settings sit above both and override bucket policies entirely; they are enabled by default on every new bucket for exactly this reason. ### Serving private content through CloudFront with OAC Public S3 buckets are rarely the right answer for production content. Origin Access Control (OAC) lets CloudFront read from a private S3 bucket while the bucket itself stays fully locked down from direct internet access. Browser | v CloudFront Distribution <-- HTTPS, caching, global edge locations | | (OAC - signed requests only) v S3 Bucket (Block Public Access: ON, no public bucket policy) This is the standard production pattern for static websites, images, and downloads: CloudFront is the only thing allowed to read the bucket, and the bucket policy explicitly trusts only that one CloudFront distribution's ARN. > 📌 **Remember:** OAC replaces the older OAI (Origin Access Identity) approach - OAC supports all S3 encryption types and is the current AWS recommendation for any new CloudFront + S3 setup. ### Versioning as your recovery mechanism Versioning does not prevent mistakes. It gives you a way back from them. Versioning disabled: delete file.txt -> gone immediately, permanently Versioning enabled: delete file.txt -> S3 adds a delete marker -> file appears gone in the console -> the actual object still exists -> remove the delete marker to restore it Once enabled, every upload to the same key creates a new version instead of overwriting the old one. All versions coexist until you explicitly delete a specific version ID. > 🔴 **Common Mistake:** Assuming "suspend versioning" deletes existing versions. It does not. Suspending only stops new versions from being created going forward - every version created while versioning was active remains in the bucket, still consuming storage and still billable.
### Cross-region replication for compliance and latency Replication copies objects from a source bucket to a destination bucket automatically, either within the same region (SRR) or across regions (CRR). Source bucket (ap-south-1) | | async replication v Destination bucket (us-east-1) CRR is the answer when regulations require data to physically exist in a specific region, or when you need to serve users in multiple regions from a nearby copy. SRR is common for centralizing logs from multiple buckets into one, or replicating between production and staging accounts. Two rules trip people up every time: * **Only new objects replicate after the rule is enabled.** Anything already in the bucket before you turned on replication stays put - you need S3 Batch Replication to backfill existing objects. * **No chaining.** If Bucket A replicates to Bucket B, and Bucket B replicates to Bucket C, an upload to A reaches B but never reaches C. Each replication rule is a single hop. ### Event-driven processing with S3 notifications S3 can trigger other AWS services the moment something happens in a bucket - no polling required. User uploads photo to S3 | | S3:ObjectCreated event fires v Lambda function invoked automatically | v Thumbnail generated, written back to S3 S3 can send events directly to three destinations - SNS for fan-out notifications, SQS for queued async processing, and Lambda for immediate compute. For more complex routing (multiple consumers, filtering by metadata, event replay for debugging), route through EventBridge instead. > **Note:** The destination must be configured to accept events from S3, using the appropriate resource-based permission or service configuration for that destination - for SNS and SQS this is a resource policy trusting `s3.amazonaws.com`, for Lambda it is a resource-based policy granting `lambda:InvokeFunction`. Configuring the event rule on the S3 side alone is not sufficient - the destination must grant permission too, or delivery silently fails. ### Temporary access without exposing credentials Pre-signed URLs grant time-limited access to a single private object, without making the bucket public or issuing IAM credentials to the requester. You (owner) generate a signed URL -> embeds: object key + operation + expiry + your signature -> share URL with a user -> user accesses the object directly until expiry -> URL stops working automatically after that ```bash ## Generate a pre-signed URL that expires in 1 hour aws s3 presign s3://prod-mumbai-uploads/invoice-2847.pdf --expires-in 3600 ``` This is the standard pattern for letting a logged-in user download a private file - a generated invoice, a premium video - without ever making that object publicly reachable.
### EBS as a network-attached disk EBS is a virtual hard drive that connects to your EC2 instance over the network rather than sitting on the same physical hardware. That network hop is the source of both its biggest strength (you can detach and reattach it) and its biggest constraint (it is locked to a single Availability Zone). EC2 Instance ---- AWS internal network ---- EBS Volume (small latency on every I/O) ap-south-1a ap-south-1b +------------------+ +------------------+ | EC2 | | EC2 | | [EBS Volume] | | [EBS Volume] | +------------------+ +------------------+ Cannot attach directly across AZs. Must snapshot, then restore in the target AZ. Every EC2 instance gets a root EBS volume automatically. Additional data volumes attach independently and, critically, survive termination by default - only the root volume is deleted on termination unless you change that setting. ### Choosing an EBS volume type by workload | Type | Category | Max IOPS | Boot Volume | Best For | |:---|:---|:---|:---|:---| | gp3 | SSD | 16,000 | Yes | Default choice for most workloads | | gp2 | SSD | 16,000 | Yes | Legacy - IOPS tied to volume size | | io2 | SSD | 256,000 | Yes | Mission-critical databases, sub-ms latency | | st1 | HDD | 500 | No | Big data, sequential throughput | | sc1 | HDD | 250 | No | Cold archives, lowest cost | The single most important fact for choosing between gp2 and gp3: gp2 ties IOPS directly to volume size (3 IOPS per GiB), so getting more performance means provisioning a bigger disk whether you need the space or not. gp3 decouples IOPS from size entirely - a 10 GiB gp3 volume can still get 3,000 baseline IOPS, and you can raise it to 16,000 without touching the size at all. This is why gp3 is the default recommendation for nearly every workload today. > 📌 **Remember:** st1 and sc1 cannot be boot volumes. If a workload needs to boot an OS, it must be gp2, gp3, io1, or io2. ### Encrypting an existing volume - the four-step process There is no in-place toggle to enable encryption on an existing EBS volume. For an existing unencrypted volume, the standard conversion workflow is snapshot, encrypted snapshot copy, new encrypted volume, then replacement attachment: 1. Snapshot the unencrypted volume 2. Copy the snapshot, enabling encryption during the copy operation 3. Create a new volume from the encrypted snapshot 4. Detach the old volume, attach the new encrypted one Unencrypted Volume | Snapshot (still unencrypted) | Copy snapshot WITH encryption enabled | New encrypted snapshot | Create volume from encrypted snapshot | Detach old, attach new > 💡 **Tip:** Enable "EBS encryption by default" at the account level once, and every volume and snapshot created afterward is automatically encrypted - no manual step needed on future resources. ### When Instance Store beats EBS Instance Store is a physical NVMe disk soldered to the same hardware your EC2 instance runs on. No network hop means it delivers IOPS numbers EBS cannot touch - some i3 instance sizes exceed 3 million random read IOPS. The trade-off is absolute: Instance Store is ephemeral. Stop the instance, terminate it, or have the underlying hardware fail, and the data is gone with no recovery path. Only a reboot leaves data intact. Question mentions "highest possible IOPS" + "temporary/cache/scratch/recreatable" -> Instance Store Question mentions "high IOPS" + "must survive stop/start" or "persistent" or "database" -> EBS io2, not Instance Store > ⚠️ **Security:** Never place data on Instance Store that cannot be regenerated or that is not backed up elsewhere. There is no recovery mechanism if the host fails - this is not a corner case, it is the expected behavior. ### EBS Multi-Attach - the exception to "one instance at a time" If EBS is described as a single-instance disk, a natural question follows: how can multiple instances ever share one volume? The answer is Multi-Attach, a specific capability of io1 and io2 volumes only. * Supported only on io1 and io2 volume types - not gp2, gp3, st1, or sc1 * Up to 16 EC2 instances can attach to the same volume simultaneously * All attached instances must be in the same Availability Zone as the volume * Your application or cluster software must handle concurrent write coordination itself - EBS does not manage file locking for you * Used for specific clustered applications built to coordinate shared block access, not as a general-purpose alternative to shared file storage > 📌 **Remember:** Multi-Attach is not a substitute for EFS. It solves a narrow problem - specific clustered applications that need shared block-level access within one AZ - while EFS solves the general shared-file-access problem across multiple AZs. ### Connecting snapshots to AMIs and backup strategy A snapshot is not just a standalone backup - it is also the foundation of an AMI (Amazon Machine Image). When you create a custom AMI from an EC2 instance, AWS snapshots the underlying EBS volumes and packages that snapshot as part of the image. Launching a new instance from that AMI creates new EBS volumes from those snapshots. EBS Volume -> Snapshot -> AMI (bundles snapshot + launch config) -> New EC2 instance launched from AMI gets new EBS volumes restored from the snapshot For ongoing backup strategy beyond manual snapshots, AWS Backup can centrally manage EBS snapshot schedules, retention, and cross-region copies across your entire fleet from one policy, rather than scripting `create-snapshot` calls yourself. This module keeps that connection conceptual - full disaster recovery and backup strategy is covered later in the roadmap's architecture module.
### EFS solves the multi-server shared storage problem EBS gives one instance a private drive. The moment you run multiple EC2 instances that all need to see the same files - the classic three-web-servers-behind-a-load-balancer setup - EBS breaks down completely. Without shared storage (each server has its own EBS): Upload lands on Server 1 -> saved to Server 1's local EBS Next request routed to Server 2 -> file not found -> broken With EFS: Upload lands on Server 1 -> saved to EFS Next request routed to Server 2 -> mounts the same EFS -> file found -> works EFS is a managed NFS filesystem that looks and behaves like a local directory to any Linux instance that mounts it - no code changes required. It scales automatically to petabyte size, spans multiple Availability Zones for high availability, and supports thousands of concurrent clients. > 📌 **Remember:** EFS is designed for Linux and POSIX-based workloads, because it relies on the POSIX filesystem interface. For Windows and SMB workloads, use FSx for Windows File Server instead. ### EFS storage tiers and automatic cost reduction Not every file in a shared filesystem is accessed at the same rate. A lifecycle policy moves files automatically between tiers based on how long they have gone untouched. Standard tier -> files accessed regularly, full price, fastest Infrequent Access -> untouched 60+ days, ~91% cheaper, small retrieval fee Archive -> rarely touched, cheapest, for compliance data Example - 1 TB total, only 100 GB actively used: Without lifecycle policy: 1,000 GB x full price = high monthly cost With lifecycle policy: 100 GB Standard + 900 GB IA = significant savings This is the same principle as S3 lifecycle rules, applied to a shared filesystem instead of an object store - active data stays fast and expensive, cold data becomes cheap automatically, with zero manual intervention. ### FSx - managed third-party file systems FSx runs specific enterprise filesystems as a fully managed AWS service, for teams that need a filesystem EFS cannot provide. | FSx Type | Solves | |:---|:---| | FSx for Windows File Server | Windows workloads needing SMB, NTFS, Active Directory integration | | FSx for Lustre | Extreme-throughput ML/HPC workloads, direct S3 integration | | FSx for NetApp ONTAP | Teams already running NetApp ONTAP on-premises | | FSx for OpenZFS | Teams already running ZFS on-premises | FSx for Lustre deserves particular attention for ML pipelines because it can read training data directly from S3 and write results straight back, at hundreds of GB/s - no manual data staging required. S3 Bucket (training data) | | FSx reads S3 as if it were local v FSx for Lustre (compute cluster reads/writes at extreme speed) | | results written back automatically v S3 Bucket (model output) > 💡 **Tip:** FSx for Lustre offers two deployment types - Scratch (cheaper, no replication, for short jobs where data already lives safely in S3) and Persistent (replicated within the AZ, for long-running jobs with data that cannot be lost).
### When the network is too slow - AWS Snowball Uploading 100 TB over a typical office internet connection can take well over a hundred days. AWS's own rule of thumb: if a network transfer would take more than a week, ship a physical device instead. Time to transfer 100 TB over the network: 100 Mbps -> ~124 days 1 Gbps -> ~12 days 10 Gbps -> ~30 hours Snowball ships you an encrypted physical appliance. You copy data onto it locally at full local network speed, ship it back, and AWS imports it directly into S3. Your servers --local network--> Snowball --shipped--> AWS --> S3 > 🔴 **Common Mistake:** Assuming Snowball can import data directly into Glacier. It cannot - Snowball always imports into S3 first. Use an S3 Lifecycle Policy afterward to transition the imported data into Glacier if long-term archival is the goal. ### Bridging on-premises protocols with Storage Gateway On-premises servers speak NFS, SMB, or iSCSI. S3 speaks its own HTTP-based API. Storage Gateway is the translation layer - a virtual appliance you run on-premises that makes S3 look like a normal file or block device to your existing servers. On-premises server | | NFS / SMB / iSCSI (protocol it already speaks) v Storage Gateway (VM on-premises) | | translated to S3 API calls over HTTPS v Amazon S3 / Glacier The core idea to hold onto: **Storage Gateway bridges on-premises storage protocols to AWS storage services.** Three variants apply that bridge to three different protocols - **S3 File Gateway** for NFS/SMB file access backed by S3, **Volume Gateway** for iSCSI block storage backed by S3 (Cached mode keeps only recent data local, Stored mode keeps the full dataset local with cloud backup), and **Tape Gateway** for replacing physical backup tape libraries with a virtual one backed by S3 and Glacier, compatible with existing backup software like Veeam and Commvault. ### Scheduled data movement with DataSync DataSync moves large datasets between on-premises storage and AWS, or between AWS storage services, on a defined schedule - not continuously, and with automatic retry, bandwidth throttling, and metadata preservation built in. On-premises to AWS (requires an agent): On-premises NFS/SMB -> DataSync Agent -> AWS DataSync -> S3 / EFS / FSx AWS to AWS (no agent needed): S3 -> DataSync -> EFS EFS -> DataSync -> FSx > 📌 **Remember:** On-premises-to-AWS transfers require a DataSync Agent installed locally. AWS-to-AWS transfers between storage services run entirely within AWS, with no agent involved.
Why storage choice is a system design decision, not a config setting A 2 AM page reads: "uploads are inconsistent across...
What makes S3 fundamentally different from a filesystem S3 is not a hard drive in the cloud. It has no real directory tr...
Cross-region replication for compliance and latency Replication copies objects from a source bucket to a destination buc...
EBS as a network-attached disk EBS is a virtual hard drive that connects to your EC2 instance over the network rather th...
EFS solves the multi-server shared storage problem EBS gives one instance a private drive. The moment you run multiple E...
When the network is too slow - AWS Snowball Uploading 100 TB over a typical office internet connection can take well ove...
The complete decision framework Situation Service Web app needs to store user-uploaded images, accessed via URL S3 Datab...
Create an S3 bucket in ap-south-1, enable versioning, and upload a file twice to confirm two versions exist: > Note: Eac...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.