Uploads and processing

An API server shouldn't spend minutes relaying a user's video to storage. Let the client move bytes directly, while the API controls who may upload, where the bytes land, and when they're safe to serve. Processing then happens asynchronously, as in the queues module.

Authorize the upload

The API authenticates the user, checks quota and allowed file types, and creates a pending asset with a unique key. It returns a short-lived presigned URL for uploading that object.

A signed URL is a bearer capability: anyone holding it can use its permitted operation until it expires. Keep lifetimes short, don't log it, and never let callers choose another user's key. Configure browser CORS for the intended origin. Use signed request constraints or an upload policy where supported; declared size and MIME type alone aren't proof of the actual content.

Multipart and resume

For large files, split the upload into parts. The server starts a multipart upload and authorizes individual parts; the client keeps the upload id and completed part identifiers. It can upload several parts in parallel and retry only failed parts. A custom upload service can offer the same idea through resumable byte offsets.

Persist the upload id, verify ownership on resume, refresh expired part URLs, and confirm which parts storage accepted before completing the upload. Abort abandoned multipart uploads with a lifecycle rule so unfinished parts don't accumulate charges.

Processing pipeline

Images need thumbnails and resized variants; videos need transcodes at several resolutions. Jobs carry an asset id, immutable source version and transformation version, rather than the file bytes. Workers fetch the source, write deterministic output keys, and update metadata with the result.

Treat storage events as duplicateable and potentially out of order. Record progress per asset version, ignore obsolete work, retry transient failures with backoff, and send repeated failures to a dead-letter queue. Reconcile pending assets periodically so a missed event doesn't strand an upload forever.

Verify before serving

Don't trust the filename extension or the client's completion message. Verify the object's actual size and checksum, inspect its format, and scan it for malware. A checksum detects corruption; it doesn't prove the content is harmless. A multipart ETag isn't a universal whole-file checksum.

Keep uploaded bytes in a private quarantine area while checks run. Limit dimensions, decompressed size and processing time, and run decoders with restricted resources. Otherwise a tiny compressed file or malformed video can exhaust a worker even without a known virus.

Downloads

For private downloads, check permission and issue a short-lived signed GET URL, or delegate to a CDN with signed access. Keep the origin private. Byte-range requests let downloads resume; stable object versions prevent mixing bytes from different revisions.