Deduplication, deletion and cost

Object storage makes keeping bytes easy. At scale, three questions decide the bill and the risk: are you storing the same bytes many times, can you delete safely, and what does moving data cost?

Content-addressed blobs

If the blob's key is a hash of its content (say SHA-256), identical files map to the same key. On upload, the client sends the hash first; if the server already has it, the upload is skipped entirely and only a metadata row is added. This is how Dropbox, Git and container registries avoid storing the same data repeatedly.

Chunking

Whole-file hashes help with exact copies. Editing a big file still re-uploads all of it. Splitting files into chunks of a few megabytes, each stored and hashed separately, means only changed chunks move.

• Fixed-size chunks are simple, but inserting a byte near the start shifts every boundary and changes every chunk.
• Content-defined chunking picks boundaries where a rolling hash of the last few bytes matches a pattern. Boundaries move with the content, so an insert changes only the nearby chunks.

A file is then a list of chunk hashes, and two versions of the file share most of their chunks.

Deleting shared data

Sharing makes deletion a garbage-collection problem. A user's delete removes their metadata; the bytes can go only when nothing references them.

Always add a grace period (days) before deleting bytes, so an upload whose metadata isn't written yet, or a restore from trash, still finds its blob. Legal holds and retention rules may also forbid deletion for a time.

What it costs

Storage is cheap; moving it is not. Rough public-cloud list prices:

For a media app, serving each stored byte a few times a month costs more than storing it. That's why CDNs, caching, efficient formats and lifecycle rules (move old data to cheaper classes) matter more than squeezing storage itself.