LESSON 29.3
TLS and secrets management
Securing data in transit and at rest with TLS termination, mutual TLS for service meshes, secret rotation, and envelope encryption.
6 min read
Protecting sensitive user data requires two fundamental guarantees: encrypting data while moving across networks (data in transit) and encrypting stored bytes on persistent media (data at rest).
Furthermore, application services require database passwords, API credentials, and private keys. Storing these credentials in Git repositories or container images is a critical security vulnerability.
TLS termination vs end-to-end encryption
Edge TLS Termination
- Offloads heavy cryptographic handshake CPU from application backends to edge load balancers
- Enables perimeter inspection, WAF filtering, and HTTP header routing
- Internal traffic between load balancer and microservices travels in plaintext unless on a secure VPC
Pick it when: High-traffic public web applications within hardened private subnets.
Mutual TLS (mTLS) Everywhere
- Zero Trust architecture; network traffic is encrypted and authenticated across every internal microservice RPC
- Guarantees caller identity without relying on IP addresses
- Higher CPU handshake overhead (mitigated by TLS session resumption)
- Requires automated certificate renewal infrastructure
Pick it when: Service meshes (Envoy, Istio), financial and compliance workloads.
Mutual TLS (mTLS) in service meshes
In a microservice fleet, service A must know with cryptographic certainty that incoming RPC calls originate from service B and not an attacker:
- A central internal Certificate Authority (CA) issues short-lived X.509 certificates (e.g. 24-hour lifetime) to each service pod.
- When Service A opens a TCP connection to Service B, both sidecar proxies exchange certificates.
- Service A verifies that Service B's certificate is signed by the trusted internal CA.
- Service B verifies that Service A's certificate contains the expected SAN identity (e.g.
spiffe://cluster.local/ns/prod/sa/checkout). - Symmetric session keys are derived via Diffie-Hellman, establishing a secure, mutually authenticated channel.
Envelope encryption with Key Management Services
Encrypting a 50 GB database backup directly with a cloud Key Management Service (AWS KMS, Google Cloud KMS) is impossible because KMS APIs enforce strict payload limits (typically 4 KB) and high network latency.
Production systems use envelope encryption:
- Request keys: The application calls
kms.GenerateDataKey(master_key_id). - KMS returns two keys: A plaintext Data Encryption Key (DEK) and the same DEK encrypted by the KMS Master Key (Ciphertext DEK).
- Local fast encryption: The application encrypts the 50 GB dataset in memory using the plaintext DEK with AES-256-GCM.
- Wipe plaintext DEK: The application immediately scrubs the plaintext DEK from memory buffers.
- Store envelope: The application stores the encrypted data alongside the Ciphertext DEK. The master key never leaves the hardware security module (HSM).
Secrets distribution and zero-downtime rotation
Hardcoded API tokens in configuration files cause severe security breaches. Dedicated secret managers (HashiCorp Vault, AWS Secrets Manager) provide secure storage and automated rotation:
- Dynamic secrets: Rather than static database passwords, Vault generates ephemeral database users valid for 1 hour.
- Short-lived tokens: Services authenticate to Vault using cloud IAM or Kubernetes Service Account tokens.
- Dual-credential rotation: When rotating database credentials, the database creates a secondary user account while keeping the primary active. Applications transition to the new credential before the old account is terminated, eliminating downtime.
- Recommended mTLS certificate lifetime
- 12 to 24 hours with automated renewal
- AES key length for envelope encryption
- AES-256-GCM (authenticated encryption)
- Signing keys
- RSA at least 2048-bit; prefer Ed25519 or ECDSA P-256 where supported
- Secrets rotation frequency
- Automated rotation every 30 to 90 days
Try it
Secret Distribution Service HLD
Design a zero-trust secret distribution service that dynamically rotates credentials across thousands of nodes.