Technology · Edge layer
API Gateways
The single front door for a service fleet. Routing, authentication, rate limiting, TLS termination, request shaping, and aggregation. How gateways differ from load balancers and service meshes, and how to keep them from becoming a single point of failure.
10 min read
An API gateway sits between the outside world and your services. Every request from a client or a partner enters through it: the gateway checks the token, enforces rate limits, rewrites the URL, terminates TLS and forwards the call to whichever service owns that route. Done well, it keeps cross-cutting concerns out of every service; done badly, it becomes the most dangerous single point of failure in the stack.
What gateways do
Routing
- One hostname for all services
- Versioned routes without client changes
- Routing rules grow into a maintenance burden
Pick it when: Always — this is the core job.
Authentication and authorization
- Validates tokens once at the edge
- Services trust the gateway's headers
- Gateway must stay up-to-date with the auth server
- Fine-grained per-resource checks still live in services
Pick it when: JWT or API key validation before requests reach services.
Rate limiting
- Protects services from overload
- Enforces per-customer quotas
- Distributed counters add latency
- Exact limits require coordination
Pick it when: Any public or partner-facing API.
TLS termination
- Services communicate on plain HTTP inside the cluster
- Certificates managed in one place
- Traffic is unencrypted inside the network
Pick it when: Standard practice — re-encrypt to services if the network is untrusted.
Request shaping
- Add, remove or rename headers
- Transform JSON payloads
- Handle CORS
- Complex transforms belong in services, not the gateway
Pick it when: Minor adaptations and header injection.
Aggregation and BFF
- One round-trip for the client instead of many
- Mobile and web can get different shaped responses
- The gateway grows service-specific logic
- Harder to keep lightweight
Pick it when: Backend-for-frontend patterns for mobile apps or SPAs.
Gateway vs load balancer vs service mesh
These three layers overlap in marketing but have different jobs.
- A load balancer (L4 or L7) distributes TCP connections or HTTP requests across a pool of instances. It knows IP addresses and ports, not API semantics. It's the right tool for scaling a single service's fleet.
- An API gateway (L7) understands API contracts: routes, authentication, rate limits and policies. It's the entry point for external traffic.
- A service mesh (such as Istio or Linkerd) handles service-to-service traffic inside the cluster: mutual TLS, retries, circuit breaking and observability at the sidecar level. It's not the entry point for external clients.
In a typical architecture all three exist: external traffic hits the gateway, which routes through a load balancer to a pod fleet, while inter-service calls flow through the mesh.
The latency cost
Every gateway plugin that runs on the hot path adds latency. A round-trip to a remote token introspection endpoint costs 1–5 ms; a Redis lookup for a rate-limit counter costs 0.2–1 ms; TLS handshake overhead on the first connection costs tens of milliseconds.
- Gateway base overheadfor a pass-through with no plugins
- < 1 ms
- JWT verification (local)signature check in-process
- ~0.1 ms
- Token introspection (remote)a network call to the auth server
- 1–5 ms
- Redis rate-limit counterone round-trip per request
- ~0.3–1 ms
- TLS terminationper-connection handshake amortized with keep-alive
- < 1 ms steady state
Keep the hot path thin. Validate JWTs locally using a cached JWKS; only call the auth server to introspect opaque tokens or revocation-sensitive routes.
Single point of failure and how to scale it
Scaling strategies:
- Run multiple gateway instances behind a cloud load balancer. Gateways are stateless (they read config from a control plane and counters from Redis); they scale horizontally.
- Keep distributed rate-limit state in Redis with a short TTL. Losing the Redis cluster degrades accuracy of limits — not the ability to serve requests — so fail open and alert.
- Store routing and policy config in a version-controlled store (etcd, a database or a git-backed config file) with a control-plane API. Never configure gateways by hand in production.
- Run the gateway in each region with the load balancer doing geographic routing. Cross-region gateway calls add 50–100 ms; avoid them on the hot path.
Common examples
Kong
- Open source, plugin ecosystem
- Lua and Go plugins
- Declarative or DB-backed config
- Self-hosted cluster to operate
- Plugin quality varies
Pick it when: On-prem or cloud-agnostic deployments with custom plugin needs.
Envoy (via Istio or standalone)
- High performance
- xDS API for dynamic config
- Native gRPC support
- Config is verbose
- Steep learning curve
Pick it when: Service meshes and advanced traffic management at Kubernetes scale.
AWS API Gateway
- No infrastructure to manage
- Integrates with Lambda, IAM, Cognito
- Expensive at high volume
- Limited customization
- AWS vendor lock-in
Pick it when: Serverless or AWS-native architectures, lower to medium traffic.
Nginx (as gateway)
- Extremely fast
- Familiar config model
- No native rate limiting across instances
- Plugins via Lua or Nginx Plus
Pick it when: Simple routing with TLS termination where custom logic is minimal.
Where it bites
- Config drift. Without a single source of truth for gateway config, different instances diverge. Routes added by hand disappear on the next deploy or reappear after a rollback. Use declarative config with CI/CD.
- Plugin creep. Over time, teams add auth logic, data transformations and business rules to the gateway because it's convenient. The gateway becomes a deployment bottleneck and a testing nightmare. Keep it to cross-cutting concerns.
- Thundering herd on restart. After a gateway restart, all services that rely on the gateway health check for service discovery may simultaneously re-register, causing a spike of traffic. Use connection draining and rolling restarts.
- Rate limits that don't match promises. A gateway that rounds down a 100 req/s limit to 99 due to clock skew between instances frustrates API customers. Document whether limits are exact or approximate and pick accordingly.
Try it
API Gateway HLD
Design an API gateway that handles routing, authentication and rate limiting for a multi-tenant platform.