LESSON 13.1
Object storage
Separate file metadata from bytes, understand durability versus availability, and choose storage classes and lifecycle rules around access patterns.
6 min read
A database is good at finding a user's files, checking permissions and updating a filename. It's rarely the best place to put a 2 GB video. Object storage holds large byte sequences behind keys; the database holds the small records that explain what those bytes mean.
Metadata here, bytes there
An object has a bucket, a key such as uploads/u7/asset92/original, bytes, and a little metadata. Retrieve it by key rather than by joining tables or searching its contents. Treat application objects as immutable: replacing a photo creates a new key or version.
A metadata row might contain asset_id, owner_id, object_key, size_bytes, content_type, checksum, status and created_at. Index it by owner and creation time to list files. Check permissions there before issuing access to the object.
Databases can store binary data, and tiny transactional attachments may fit. At media scale, large payloads inflate database replication, backups and recovery, and compete with queries for memory and I/O. Object storage scales the byte workload separately from transactional metadata.
How much storage?
Use rough arithmetic before discussing classes or providers:
- Original photos
- 100 million × 5 MB = 500 TB
- Thumbnails and resized copies
- 1 MB extra per photo = 100 TB
- Total stored payloadBefore old versions and backups
- 600 TB
- Metadata at 1 KB per photoBefore indexes and database overhead
- About 100 GB
That difference is why the split matters. Also estimate growth, download traffic, API request volume and retained versions. A cheap storage tier doesn't make repeated downloads free; transfer and retrieval charges can dominate the bill.
Durability isn't availability
S3 Standard is designed for 99.999999999% annual durability, often called eleven nines, and 99.99% availability. It stores data redundantly across multiple availability zones. Those are different properties: an object can be safe while temporarily unreachable. AWS's data protection guide states these design targets.
High durability doesn't protect against every application mistake. An authorized delete, a compromised account, or a lifecycle rule that expires the wrong prefix can still remove data. Use least-privilege permissions, versioning and retention where needed; consider a separate regional or account copy for disaster recovery.
Storage classes
Frequent access
- Immediate reads
- Good for active content
- Higher storage cost
Pick it when: New uploads, popular media and user-facing files.
Infrequent access
- Lower storage cost
- Immediate reads in suitable classes
- Retrieval fees
- Minimum storage durations may apply
Pick it when: Older files that are rarely read but must open immediately.
Archive
- Low cost for long retention
- Some classes require hours to restore
- Retrieval and early deletion costs
Pick it when: Backups and records whose restore delay is acceptable.
Archive isn't one latency tier: some classes support instant access, others require a restore job. Single-zone classes also have different failure protection from multi-zone classes. Choose from the required access and recovery behavior rather than the class name alone.
Lifecycle rules
A lifecycle policy automates transitions and expiration: keep originals hot for 30 days, move rarely accessed originals to a cheaper class, and expire temporary exports after 7 days. These are example product rules, not universal defaults. Check minimum durations, retrieval costs and legal retention first.
Try it
File Storage HLD
Separate file metadata from bytes and choose retention and recovery rules.