Pattern · Storage
Handling large blobs
Files from megabytes to gigabytes. Keep bytes out of your servers and database, upload directly to object storage, process asynchronously, and deliver through a CDN.
10 min read
Photos, videos, documents, backups and datasets are too big to push through application servers or store in database rows. Doing so ties up threads for minutes, bloats backups and replicas, and makes every failure restart a huge transfer. The pattern separates metadata, which lives in your database, from bytes, which move directly between clients and object storage.
The shape
Uploading
- The client asks the API to start an upload, with the file name, size and type.
- The API checks permissions and quotas, creates a metadata row in an UPLOADING state, and returns a presigned URL (or multipart upload id and part URLs) for a key it chooses.
- The client uploads directly to object storage. Large files go in parts of 5–100 MB, in parallel, each retried on failure.
- The client tells the API it's done (or a storage event does). The API verifies the object exists with the expected size and checksum, and marks the row UPLOADED.
Resumable uploads keep the upload id and the list of completed parts on the client, so a dropped connection resumes instead of restarting. Clean up abandoned multipart uploads with a lifecycle rule.
Processing
Uploaded files often need work: thumbnails, transcoding into several bitrates, virus scanning, text extraction. Trigger it from the upload-complete event into a queue, process in workers, and write outputs under derived keys (photos/42/thumb_256.jpg). Treat this as a long-running task: idempotent, retryable, with status on the metadata row so the UI can show "processing".
Downloading
Through the app server
- Easy access checks
- Servers stream every byte
- Slow and expensive
Pick it when: Avoid for anything large.
Presigned GET URL
- Direct from storage
- Per-request permission
- No edge caching per user
Pick it when: Private files downloaded occasionally.
CDN with signed URLs or cookies
- Cached near users
- Cheap egress
- Range requests for video
- Invalidation is coarse
Pick it when: Media and popular files.
Use immutable, versioned keys for derived files so CDN caching can be aggressive: a new version gets a new key instead of an invalidation.
Sync and deduplication
File-sync products (Dropbox, Drive) go further:
- Split files into chunks of a few MB, addressed by their content hash.
- Upload only chunks the server doesn't already have, so editing a large file sends just the changed chunks, and identical files are stored once.
- The file's metadata becomes an ordered list of chunk hashes, versioned per edit.
- 1 GB upload over a 20 Mbps link
- ~7 minutes
- Through an app server
- one connection and its memory held for 7 minutes
- Direct to storage with 10 MB parts
- 100 retryable parts; the server is used for a few ms
Try it
File Storage HLD
Design a file storage and sharing service.