Technology · Object storage
S3 and object storage
Cheap, durable storage for files of any size. Buckets and keys, durability and consistency, presigned and multipart uploads, storage classes, events and the limits that matter.
10 min read
Amazon S3, and its equivalents (Google Cloud Storage, Azure Blob Storage, MinIO, Cloudflare R2), store objects: blobs of bytes addressed by a key, from a few bytes to terabytes. It's where images, videos, uploads, backups, logs and data lake files belong. Databases hold the metadata; object storage holds the bytes.
When to reach for it
- Any file or binary payload: photos, videos, documents, attachments.
- Large outputs: exports, ML datasets, analytics tables in Parquet.
- Backups and archives with lifecycle rules.
- Static website assets served through a CDN.
The model
- A bucket holds objects; each object has a key (like
users/42/avatar/v3.jpg), bytes, and metadata. - Keys look like paths, but the namespace is flat. "Folders" are just shared prefixes, useful for listing and permissions.
- Objects are written whole and replaced whole; there's no editing in place, only overwriting or appending a new version.
- Access goes through an HTTP API: PUT, GET (including byte ranges), DELETE, LIST.
- Durabilitydesigned annual object durability
- 11 nines
- Availability (standard)
- 99.99% designed
- Max object sizesingle PUT up to 5 GB; use multipart above ~100 MB
- 5 TB
- Request ratespread keys across prefixes to go higher
- ~3,500 writes and ~5,500 reads per second per prefix
- First-byte latencyput a CDN in front for users
- tens of ms
S3 now offers strong read-after-write consistency for all operations, so a successful PUT is immediately visible to GETs and LISTs.
Uploads and downloads
- Presigned URLs let clients upload or download directly, without bytes passing through your servers. Your API checks permissions, then signs a URL for one key, one operation, a few minutes.
- Multipart upload splits large files into parts uploaded in parallel and retried individually.
- Range GETs fetch part of an object, for resumable downloads and video seeking.
- Event notifications on object creation trigger processing: thumbnails, transcoding, virus scanning.
Storage classes and lifecycle
Standard
- Instant access
- No retrieval fee
- Highest storage price
Pick it when: Hot data: active user content.
Infrequent access
- Cheaper storage
- Retrieval fee
- Minimum storage duration
Pick it when: Older content read occasionally.
Glacier / archive
- Cheapest storage
- Minutes to hours to restore
Pick it when: Backups and compliance archives.
Lifecycle rules move objects between classes by age and delete expired ones automatically. Intelligent tiering does it based on observed access.
Protecting data
- Versioning keeps old versions on overwrite and delete, so mistakes can be undone.
- Object Lock (write once, read many) stops deletion for a retention period, protecting backups from attackers.
- Cross-region replication copies objects to another region for disaster recovery or latency.
- Block public access by default; serve public content through a CDN with an origin access policy.
Where it bites
- Request costs add up with millions of tiny objects; batch small items into larger files.
- Egress to the internet is expensive; serve through a CDN.
Try it
File Storage HLD
Design a file storage service like Dropbox or Google Drive.