Amazon S3 - Object Storage, Bucket Policies, Versioning, and Lifecycle Rules

What you will learn

• What S3 is and how it is different from a file system or a database
• Bucket naming rules and how objects are actually stored
• Bucket policies vs IAM policies — how S3 decides who gets access
• Block Public Access — the safety switch that overrides everything
• Versioning — how delete markers work and how to recover deleted files
• Replication — CRR and SRR, what gets replicated and what does not
• All storage classes and how lifecycle rules move objects between them automatically
• S3 performance — multipart upload, Transfer Acceleration, and byte-range fetches
• Encryption — four methods and when to use each
• Pre-signed URLs for time-limited access to private files

Why this matters

Hotstar serves video thumbnails and metadata to millions of users through S3 and CloudFront. Razorpay stores payment compliance documents with Object Lock so they cannot be modified or deleted. Swiggy aggregates application logs from hundreds of services into a single S3 bucket and queries them with Athena. Zerodha stores trading records for regulatory retention requirements using Glacier Deep Archive at a fraction of the cost of a database.

S3 is not just file storage. It is the integration point for almost every AWS service. And it is involved in one of the most common AWS security incidents — accidentally public buckets containing sensitive data. Understanding S3 properly means understanding both how powerful it is and where the traps are.

What is Amazon S3

S3 (Simple Storage Service) is object storage. You store files — called objects — inside containers called buckets. Unlike a file system, there are no real folders. Unlike a database, there is no schema or query language.

S3 is accessed via HTTP/HTTPS using an API — not mounted like a disk. Your application calls the S3 API to put, get, or delete objects.

What makes S3 special:

Buckets and Bucket Naming Rules

A bucket is the top-level container. Every object lives inside a bucket.

Naming rules — every one matters:

Buckets are created in a specific region. Your data stays in that region unless you explicitly replicate it. Despite looking global in the console, every bucket has a home region.

Objects and Keys

An object is any file stored in S3. The key is the full path to that object inside the bucket.

The key has two parts:

Remember

There are no real folders in S3. What looks like a folder hierarchy is just slashes in the key name. The entire path is the filename. This matters for performance — spreading objects across many different prefixes dramatically increases read and write throughput.

Object size limits:

S3 Security — Who Gets Access

S3 access is controlled at two levels and can come from two directions.

The access decision rule:

Bucket Policy — the resource-side rule:

A bucket policy is a JSON document attached directly to the bucket. It controls access to that bucket for anyone — IAM users, other accounts, anonymous public users, AWS services.

Remember

arn:aws:s3:::my-bucket targets the bucket itself (for listing). arn:aws:s3:::my-bucket/* targets objects inside the bucket (for reading files). Many access problems come from using the wrong ARN. Listing needs the bucket ARN. Getting objects needs the /* ARN. Often you need both statements.

Block Public Access

Block Public Access is a safety override that ignores bucket policies and ACLs that try to make your bucket public. It was created after too many data breaches from accidentally public S3 buckets.

Four settings — each can be on or off individually:

Default: All four are ON for every new bucket. AWS also applies them at the account level.

Even if a bucket policy explicitly says "allow public access", Block Public Access overrides it and blocks anyway. This is the safety net.

Only turn off Block Public Access when you specifically need a public bucket — static website hosting, public file downloads. For every other bucket keep all four settings on.

Versioning

Versioning keeps every version of every object. Once enabled, uploading the same key creates a new version — it does not overwrite the old one.

Key facts:

How deletes work with versioning:

Tip

Enable versioning on any bucket that holds data you cannot afford to lose. It protects against two things: accidental overwrites (restore a previous version) and accidental deletes (remove the delete marker). The extra storage cost is usually worth it.

Replication — CRR and SRR

Replication automatically copies new objects from one bucket to another in the background.

Cross-Region Replication (CRR):

Same-Region Replication (SRR):

Requirements for either type:

What replication does and does not do:

Storage Classes — The Cost vs Speed Tradeoff

All S3 storage classes have the same 11 nines of durability. What changes is availability, retrieval speed, and cost.

The minimum billing duration trap:

Intelligent-Tiering — fully automated:

Monitors access for each object. Moves it between tiers automatically based on actual usage patterns. No retrieval fees. Small monitoring fee per object. Use when you genuinely do not know how often data will be accessed.

Lifecycle Rules — Automatic Cost Reduction

A lifecycle rule automatically transitions objects between storage classes after a set number of days, and optionally deletes them after another set of days.

Rules can target everything in the bucket or just specific prefixes:

Remember

Always add a lifecycle rule to delete incomplete multipart uploads after 7 days. A failed large upload leaves behind parts that bill you but never become a complete object. Without this rule those orphaned parts accumulate silently.

S3 Performance

S3 scales automatically but some patterns can squeeze much more throughput out of it.

Baseline performance:

Spread objects across many prefixes and the numbers multiply:

Multipart Upload — for large files:

Breaks a large file into parts and uploads all parts in parallel. Faster. If one part fails, retry only that part, not the entire file.

S3 Transfer Acceleration:

Routes uploads through the nearest AWS Edge Location. Short fast hop to the edge, then AWS's fast private network the rest of the way to your bucket.

Useful when uploading from distant locations or across continents. Small per-GB cost.

Byte-Range Fetches — download only what you need:

Request a specific byte range of an object instead of downloading the entire file.

Encryption

Four methods. The right one depends on who should control the keys and your compliance requirements.

SSE-S3 — default, zero effort:

AWS encrypts every object with AES-256. You do nothing. Best for most workloads that just need encryption without special compliance requirements.

SSE-KMS — full audit trail:

Keys managed in AWS KMS. Every encryption and decryption operation appears in CloudTrail. Use for compliance requirements where you need proof of who accessed encrypted data and when.

One trap: every S3 upload with SSE-KMS calls KMS GenerateDataKey. Every download calls KMS Decrypt. KMS has API quotas. At high request rates (millions of S3 operations per hour) you can hit the KMS quota and get throttled. If this happens, request a quota increase or switch to SSE-S3.

SSE-C — keys you manage, AWS never stores:

You send your encryption key in the request header. AWS encrypts, then immediately forgets your key. HTTPS is mandatory because your key travels in the header. AWS stores the encrypted data, not your key.

Client-Side — AWS never sees plaintext:

You encrypt on your machine before uploading. S3 stores an already-encrypted blob. S3 has no idea what is inside. Only clients with the correct key can decrypt. Maximum control.

Pre-Signed URLs

A pre-signed URL gives temporary access to a private S3 object without making the bucket public.

Use cases:

S3 Event Notifications — React to What Happens in Your Bucket

Every time an object is uploaded, deleted, or restored, S3 can automatically notify other services. No polling needed. No cron jobs. The bucket tells the world something happened.

Four destinations for event notifications:

EventBridge is the most powerful option — it lets you filter events by prefix, suffix, or object metadata and route them to dozens of different targets including Step Functions, Kinesis, and other AWS accounts.

One important rule: S3 sends each event notification at least once. Your consumer must handle duplicates.

S3 Object Lock and Glacier Vault Lock — WORM Storage

WORM stands for Write Once Read Many. Once written, the object cannot be modified or deleted — by anyone, including the root user — until the retention period expires.

Razorpay uses Object Lock on compliance documents that regulators require to be kept unmodified for 7 years. Nobody can delete or overwrite them accidentally or maliciously.

Two Object Lock modes:

Retention Period vs Legal Hold:

Glacier Vault Lock:

Same concept but for Glacier Vaults. Once a Vault Lock policy is locked (confirmed), it cannot be changed or deleted. Complete regulatory compliance for archived data.

Remember

Object Lock must be enabled when the bucket is created. You cannot enable it on an existing bucket. Versioning is automatically enabled when Object Lock is enabled. Plan for this before creating compliance buckets.

S3 Access Points — Simplify Multi-Team Access

A large company has one S3 bucket with data for many teams — Finance, Engineering, Analytics, Compliance. Managing bucket policies that correctly grant each team access to only their data gets complicated fast.

S3 Access Points solve this. Each team gets their own Access Point — their own hostname and their own policy — pointing to the same underlying bucket.

Each Access Point has its own policy. Adding a new team means creating a new Access Point — the bucket policy stays simple and unchanged.

Object Lambda — Transform Data on the Way Out:

Object Lambda sits between S3 and the application. When an object is retrieved through an Object Lambda Access Point, a Lambda function runs and can modify the response before it reaches the caller.

Same underlying data. Different views for different teams. No duplicate storage needed.

Static Website Hosting and CORS

S3 can serve a static website — HTML, CSS, JavaScript files — directly. No web server. No EC2.

For a public website the bucket must allow public read access. Combine with CloudFront to add HTTPS, caching, and a custom domain.

CORS — Cross-Origin Resource Sharing:

If your website at https://myapp.in makes API calls to https://api.myapp.in, the browser blocks the request unless the API explicitly says the other origin is allowed. This is CORS.

When your web app hosted on one domain needs to fetch assets from an S3 bucket on another domain, S3 needs a CORS configuration.

This tells browsers: requests from https://myapp.in are allowed to fetch objects from this bucket.

Hands-on Lab — Versioning, Lifecycle, and Event Notifications

Step 1 — Create a versioned bucket

Step 2 — Upload, overwrite, see versions

Step 3 — Delete and restore

Step 4 — Create a lifecycle rule

Step 5 — Generate a pre-signed URL

Step 6 — Cleanup

Common Mistakes to Avoid

Common Mistake

Setting the S3 access log destination to the same bucket being logged. Every access creates a log file, which is another access, which creates another log file. The bucket grows uncontrollably. Bills spike. Always use a separate dedicated logging bucket.

Common Mistake

Turning off Block Public Access without realising what it does. Block Public Access protects the entire bucket even if a policy tries to make it public. Turning it off does not make the bucket public — but it means a bucket policy CAN make it public. Many engineers turn it off "to debug" and forget to turn it back on.

Common Mistake

Using SSE-KMS for high-throughput S3 workloads without checking KMS quotas. At millions of S3 operations per hour, KMS Decrypt is called for every download. The default KMS quota is 5,500-30,000 requests per second depending on region. Hitting the quota causes S3 throttling even though S3 itself is fine. Use SSE-S3 for high-throughput workloads unless audit trail is specifically required.

Security

Never disable Block Public Access on a bucket unless you have a specific reason that justifies it. The number of data breaches from accidentally public S3 buckets has been enormous — enough that AWS made Block Public Access the default and added account-level enforcement. Treat any bucket with Block Public Access disabled as a security exception that needs review.

Tip

Run S3 Storage Lens on your account at least once. It shows you across all your buckets: which ones have versioning disabled, which have unencrypted objects, which have incomplete multipart uploads silently costing you money, and which storage classes your data is in. Most teams are surprised by what it finds — especially orphaned multipart uploads.