LESSON 8.4
Deploying without downtime
Rolling, blue-green and canary releases, draining connections and shutting down gracefully, and keeping old and new versions compatible while both are running.
6 min read
Scaling out gives you many identical servers. That's also what makes deploys safe: you can replace them a few at a time while the rest keep serving. But only if the release process and the code are built for it.
Three release strategies
Rolling
- No extra fleet
- Simple
- Old and new versions run together for a while
- Rollback is another rolling deploy
Pick it when: The default for stateless services.
Blue-green
- Instant switch and instant rollback
- Double capacity during the switch
- Everyone moves at once
Pick it when: You need a clean cutover, or rollback speed matters most.
Canary
- Small blast radius
- Real traffic decides
- Needs good metrics and automation
Pick it when: Large user bases where a bad release is expensive.
A canary typically goes 1% → 10% → 50% → 100%, pausing at each step while error rate and latency are compared with the old version. If they get worse, the rollout stops and reverts automatically.
Draining and graceful shutdown
Every replaced server must leave without dropping requests:
- Deregister. Remove the server from the load balancer, or fail its readiness check so traffic stops arriving.
- Drain. Let in-flight requests finish, with a deadline (say 30 seconds). Long-lived connections like WebSockets get a close message so clients reconnect elsewhere.
- Stop. Exit on SIGTERM only after draining. Stop pulling queue messages first so none are half-processed.
New servers need the opposite: don't report ready until caches, connection pools and config are warm. Otherwise the first requests after a deploy are the slowest of the day.
Two versions at once
Rolling and canary deploys mean old and new code serve traffic at the same time, often for an hour. Everything they share must work with both:
- Database schema. Use expand and contract: add new columns or tables first, deploy code that writes both, backfill, switch reads, and only drop the old shape in a later release.
- APIs and messages. Add fields, don't rename or remove them; readers ignore fields they don't know. A message written by v2 may be read by v1.
- Caches. If the cached format changes, version the key so old servers don't choke on new entries.
Rollback is a feature
A deploy is only safe if undoing it is fast and practiced. That means keeping the previous build ready, avoiding one-way migrations in the same release as the code that needs them, and alerting on the metrics a canary watches. Most outages start with a change; the best defense is making every change small, gradual and reversible.
Try it
Fleet Configuration Rollout Service HLD
Design a staged rollout with automatic halts and rollback.