05 — S3: Object Storage, Buckets, and the Access Math

"A folder in the sky" was my mental model of S3, and it quietly explained nothing — not the bucket names, not the access errors, not the pricing. The two ideas that straightened it out: S3 is a flat object store where the "folders" are an illusion — every object is just a key in a bucket — and every access question resolves to policy math between what the caller's identity allows and what the bucket itself allows [1]. Once those two were clear, the rest of S3 (lifecycle rules, presigned URLs, CORS) read like straightforward configuration rather than a maze.

What S3 actually is

Amazon S3 — Simple Storage Service — is object storage [1]. I store arbitrary blobs ("objects") in containers ("buckets"), and each object is identified by a unique key. The unit of retrieval is a whole object: there is no editing in place, no byte-range writes, no filesystem lock. S3 is not a disk; it is a key→blob map with eleven 9s of durability.

The mental shift I had to make: the slash in reports/2024/q1.csv is not a directory separator. It is a literal character in the key. S3 has no folders; the console renders prefixes as folders for my convenience, but underneath there is one flat keyspace per bucket. Tools that "create a folder" are actually just creating a zero-byte object whose key ends in /.

Buckets and objects

A bucket is the top-level container [5]. Three rules worth memorizing:

• Bucket names are globally unique across all of AWS. my-bucket is taken; I'll be choosing my-company-reports-2024-xyz.
• A bucket lives in one region. I pick the region at creation; the data stays there unless I replicate it. (Region choice matters for latency, cost, and data residency.)
• There is no limit on the number of objects in a bucket, and objects can be from 0 bytes to 5 TB. The scale is genuinely vast.

An object is the blob plus its metadata (content-type, custom tags, encryption settings). Object keys must be unique within a bucket; pushing the same key overwrites the previous object (unless versioning is on, in which case both are retained).

The access math: identity-based vs resource-based

This is where S3 security used to lose me. There are two policy surfaces, and S3 consults both [2][3]:

• Identity-based policy — attached to the IAM user or role making the call. "This caller may s3:GetObject on arn:aws:s3:::my-bucket/*."
• Resource-based policy — the bucket policy, attached to the bucket itself. "Anyone (or these specific principals) may read objects in me."

For a same-account caller, S3 grants access if either policy allows it. For a cross-account caller, both must allow it (the caller's identity policy needs to permit the action, and the bucket policy needs to trust the caller's account). That "same-account = either, cross-account = both" rule is the part I had to write on a sticky note.

A third surface — ACLs (access control lists) — exists, and AWS now recommends disabling them entirely in favor of bucket policies. Treat ACLs as legacy.

Instance profiles and assuming roles, in S3 terms

Two IAM concepts keep coming up with S3, and they're worth pinning down here [3][4]:

• Instance profiles are how an EC2 instance gets a role attached to it. The role's identity-based policy can grant s3:GetObject on a bucket, and code on the instance reads objects without ever holding a long-lived key.
• Assuming roles is how one identity temporarily takes on another role's permissions via STS — used for cross-account access ("the CI account assumes a role in the prod account to push a build to an S3 bucket").

In both cases the S3 access question collapses to the same identity-vs-resource math above. The delivery mechanism is different; the policy evaluation is identical.

Bucket and object lifecycle

S3 is cheap per GB, but at scale the bill and the clutter add up. Lifecycle rules automate transitions and deletions [6]:

• Transition actions — move objects to a cheaper storage class after N days. "Logs older than 30 days go to S3 Standard-IA; older than 90 days go to Glacier."
• Expiration actions — delete objects after N days. "Drop everything in tmp/ after 7 days."

Lifecycle rules are how a bucket stays maintainable without a cleanup cron. The pattern I reach for: every "log" or "upload" bucket gets a lifecycle rule on creation; every long-term archive bucket gets a transition-to-Glacier rule. The notes on storage classes cover what each class costs and how fast it returns data.

A real shell snippet

Uploading and reading objects is directly observable with the AWS CLI:

aws s3 cp ./report.pdf s3://my-bucket/reports/2024/report.pdf
aws s3 presign s3://my-bucket/reports/2024/report.pdf --expires-in 3600

The first copies a local file into a bucket. The second generates a presigned URL — a time-limited link anyone can use to fetch the object, without holding AWS credentials. Presigning is the clean answer for "give this client a download link valid for an hour."

How I use this

S3 is the storage default for anything that isn't a database. My standing rules: bucket names include a project prefix and a random suffix to survive global uniqueness; every bucket gets versioning and default encryption turned on at creation; public access is blocked by default and only opened deliberately through a bucket policy I can defend; and lifecycle rules are part of the bucket's definition, not an afterthought. For uploads from a browser or mobile client I use presigned URLs (or a pre-signed POST) so the bytes never transit my own servers. The flat-keyspace picture is the part I keep closest — once "folders are an illusion" is internalized, every weird S3 path bug becomes obvious.

References

[1] Amazon Web Services, "Welcome to Amazon Simple Storage Service," S3 User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

[2] Amazon Web Services, "Identity-based policies for S3," S3 User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_identity-vs-resource.html

[3] Amazon Web Services, "Using instance profiles," IAM User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html

[4] Amazon Web Services, "Assuming a role," IAM User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_manage-assume.html

[5] Amazon Web Services, "Working with buckets," S3 User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html

[6] Amazon Web Services, "Managing your storage lifecycle," S3 User Guide, 2024. [Online]. Available: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html