- 01 Terminology Glossary
- 02 Real-Time Chat (WhatsApp)
- 03 Video Streaming (YouTube)
- 04 Low-Latency Systems
- 05 Ordered Message Queues
- 06 URL Shortener
- 07 Rate Limiter
- 08 Notification System
- 09 News Feed
- 10 Offline Messaging
- 11 Circuit Breaker
- 12 Self-Healing Infrastructure
- 13 Scaling to 5M Users
- 14 Handling 5M Requests
- 15 End-to-End Encryption
- 16 Real-Time React Dashboard
- 17 Fast-to-Slow File Transfer
- 18 Ride-Hailing (Uber, Ola)
- 19 Ticket Booking (BookMyShow)
- 20 Unique ID Generator
Blueprint 12 · 6 diagrams
Design Self-Healing (Auto-Healing) Infrastructure
Self-healing (auto-healing) infrastructure
1. Summary
Self-healing infrastructure finds failures and repairs them automatically, with no human action. It uses one main idea: a control loop. The system compares the actual state with the desired state. When the two states are different, the system takes action to make them the same.
Self-healing does not prevent failures. It decreases the time to repair (MTTR). It also decreases the number of alerts that wake up engineers.
2. The control loop
flowchart LR
D["Desired state: 5 healthy pods, v2.3, 60% CPU max"] --> CMP{"Compare"}
OBS["Observe: health checks, metrics, logs"] --> CMP
CMP -- "same" --> OBS
CMP -- "different" --> DEC["Decide: which action?"]
DEC --> ACT["Act: restart, replace, scale, fail over, roll back"]
ACT --> VER["Verify: is the system healthy now?"]
VER -- "yes" --> OBS
VER -- "no, after N tries" --> ESC["Escalate to a human"]
The four steps
- Detect. Find the failure with health checks, metrics, and logs.
- Decide. Choose the correct repair action for this failure.
- Act. Do the repair action automatically.
- Verify. Check the result. If the repair fails, try a different action or call a human.
3. Healing layers
Repair each failure at the lowest possible layer. A low-layer repair is fast and has a small blast radius.
flowchart TB
L1["1. Process: restart the process"] --> L2["2. Container: restart the container"]
L2 --> L3["3. Service: replace the instance, keep replica count"]
L3 --> L4["4. Node: drain and replace the machine"]
L4 --> L5["5. Capacity: scale out or scale in"]
L5 --> L6["6. Data: fail over to the database replica"]
L6 --> L7["7. Release: roll back the bad deployment"]
L7 --> L8["8. Region: move traffic to a different region"]
| Layer | Failure | Repair action | Tools |
|---|---|---|---|
| Process | Process crashes | Restart the process | systemd, PM2, supervisord |
| Container | Container stops or does not respond | Restart the container | Kubernetes liveness probe, Docker restart policy |
| Service | Pod or instance is lost | Create a new instance | Kubernetes ReplicaSet, Deployment |
| Node | Machine becomes unhealthy | Drain the node. Create a new node. | AWS Auto Scaling Group, GKE node auto-repair, Karpenter |
| Capacity | Load is too high | Add instances | Kubernetes HPA, Cluster Autoscaler, AWS ASG scaling policies |
| Data | Primary database fails | Promote a replica | AWS RDS Multi-AZ, Patroni, MongoDB replica set, Kafka leader election |
| Release | New version causes errors | Roll back automatically | Argo Rollouts, Flagger, Spinnaker |
| Configuration | Manual change causes drift | Apply the correct configuration again | Argo CD self-heal, Flux, Terraform |
| Region | Full region fails | Move traffic to a healthy region | Route 53 health checks, Global Load Balancer |
4. Reference architecture on Kubernetes
flowchart TB
subgraph Control plane
API["Kubernetes API"]
RS["ReplicaSet controller"]
HPA["Horizontal Pod Autoscaler"]
CA["Cluster Autoscaler"]
ARGO["Argo CD: GitOps sync"]
ROLL["Argo Rollouts: canary"]
end
subgraph Worker nodes
K1["kubelet: runs probes"] --> P1["Pod A"]
K1 --> P2["Pod B"]
K2["kubelet"] --> P3["Pod C"]
NPD["Node Problem Detector"]
end
GIT[("Git repo: desired state")] --> ARGO
ARGO --> API
PROM["Prometheus metrics"] --> HPA
PROM --> ROLL
PROM --> AM["Alertmanager"]
AM --> AUTO["Automated runbooks"]
AM --> PD["PagerDuty: human"]
RS --> API
HPA --> API
CA --> CLOUD["Cloud provider: add or remove nodes"]
NPD --> API
5. Detection: health checks
Detection is the most important part. A bad health check causes wrong repairs.
Three types of probes
| Probe | Question | Action on failure |
|---|---|---|
| Liveness | Is the process alive and not stuck? | Restart the container. |
| Readiness | Can the instance accept traffic now? | Remove the instance from the load balancer. Do not restart. |
| Startup | Has the application finished startup? | Do not run other probes until startup is complete. |
Kubernetes example
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-service
spec:
replicas: 3 # Desired state: 3 pods
selector:
matchLabels: { app: order-service }
template:
metadata:
labels: { app: order-service }
spec:
containers:
- name: app
image: registry/order-service:2.3.0
resources:
requests: { cpu: 250m, memory: 256Mi }
limits: { memory: 512Mi }
startupProbe:
httpGet: { path: /health/startup, port: 8080 }
failureThreshold: 30 # 30 x 5 s = 150 s maximum startup time
periodSeconds: 5
livenessProbe:
httpGet: { path: /health/live, port: 8080 }
periodSeconds: 10
failureThreshold: 3 # Restart after 3 failures
readinessProbe:
httpGet: { path: /health/ready, port: 8080 }
periodSeconds: 5
failureThreshold: 2 # Stop traffic after 2 failures
Rules for health check endpoints
- Liveness must check only the process itself. Do not check the database in the liveness probe. If the database fails, Kubernetes restarts all pods. The restarts do not repair the database, and they cause more load.
- Readiness can check critical dependencies. If the database is not available, the pod stops getting traffic. It does not restart.
- Keep checks fast and cheap. A health check must not cause load.
- Detect "stuck" states. For example, the liveness endpoint can check that the event loop or worker thread made progress in the last 30 seconds.
6. Capacity healing: autoscaling
sequenceDiagram
participant M as Metrics (Prometheus)
participant HPA as Pod autoscaler
participant K as Kubernetes
participant CA as Cluster autoscaler
participant C as Cloud provider
M->>HPA: CPU = 85% (target 60%)
HPA->>K: Increase replicas 5 to 8
K->>K: 3 pods cannot be scheduled (no free node)
K->>CA: Pending pods
CA->>C: Add 2 nodes
C-->>K: Nodes ready
K->>K: Schedule pending pods
M->>HPA: CPU = 55%
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: order-service
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: order-service }
minReplicas: 3
maxReplicas: 30
metrics:
- type: Resource
resource:
name: cpu
target: { type: Utilization, averageUtilization: 60 }
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # Wait 5 min before scale-in. This prevents flapping.
7. Release healing: automatic rollback
Many production incidents start with a deployment. An automatic rollback repairs these incidents quickly.
flowchart LR
NEW["New version v2.4"] --> C1["Canary: 5% traffic"]
C1 --> AN{"Analysis: error rate < 1%, p99 < 300 ms?"}
AN -- "pass" --> C2["25% traffic"]
C2 --> AN2{"Analysis"}
AN2 -- "pass" --> FULL["100% traffic"]
AN -- "fail" --> RB["Automatic rollback to v2.3"]
AN2 -- "fail" --> RB
Procedure
- Deploy the new version to a small part of the traffic.
- Compare the error rate and latency with the old version.
- If the metrics are good, increase the traffic in steps.
- If the metrics are bad, send all traffic back to the old version.
- Send an alert to the team with the analysis result.
8. Configuration healing: GitOps
- Git holds the desired state of all infrastructure and applications.
- Argo CD or Flux compares the cluster with Git every few minutes.
- If an engineer changes the cluster manually, the tool applies the Git state again ("self-heal").
- To change the system, change Git. Thus, each change has a review and a history.
9. Automated runbooks
Some failures need a specific action. Write the runbook as code. Start the code from an alert.
| Alert | Automated action |
|---|---|
| Disk usage more than 90% | Delete old logs and temporary files. Increase the volume size if necessary. |
| Certificate expires in 7 days | Renew the certificate (cert-manager). |
| Memory leak: memory grows without limit | Restart the pods one at a time. Create a ticket for the team. |
| Queue backlog grows | Add consumer instances (KEDA). |
| Node has a kernel or disk error | Cordon and drain the node. Replace it. |
| Database connection pool is full | Restart the application with the leak. Increase the pool limit for a short time. |
Tools: AWS Systems Manager Automation, Azure Automation, StackStorm, Rundeck, PagerDuty Runbook Automation, Kubernetes operators.
10. Data healing: database failover
sequenceDiagram
participant APP as Application
participant DNS as DB endpoint (DNS or proxy)
participant P as Primary DB
participant R as Replica DB
participant O as Failover manager (Patroni / RDS)
APP->>DNS: Write
DNS->>P: Write
Note over P: Primary fails
O->>P: Health check fails 3 times
O->>R: Promote replica to primary
O->>DNS: Point endpoint to new primary
APP->>DNS: Write (retry)
DNS->>R: Write
O->>O: Create a new replica
- Use synchronous replication if you cannot lose any write. This adds write latency.
- Use a consensus system (etcd, ZooKeeper) to choose the new primary. This prevents "split brain" (two primaries at the same time).
- The application must retry failed writes and reconnect after failover.
11. Safety controls
Automation can make an incident worse. Use these controls.
| Risk | Control |
|---|---|
| Restart loop | Use exponential backoff between restarts (Kubernetes CrashLoopBackOff does this). |
| Too many repairs at the same time | Limit the rate of repair actions. Repair one node or one pod at a time. |
| Repair removes too much capacity | Use a PodDisruptionBudget. Keep a minimum number of healthy pods. |
| Repair hides a real problem | Record each repair action. Create a ticket if the same repair occurs many times. |
| Wrong diagnosis | Use more than one signal before a large action (for example, failover). |
| Automation fails | After N failed tries, stop and call a human. |
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: order-service-pdb
spec:
minAvailable: 2 # Always keep 2 pods running during repairs
selector:
matchLabels: { app: order-service }
12. Design rules for applications
The infrastructure can heal only applications that support it.
- Stateless services. Keep state in a database or cache. Then any instance can replace any other instance.
- Fast startup. A new instance must be ready in seconds.
- Graceful shutdown. Handle
SIGTERM. Finish current requests. Then stop. - Idempotent operations. A retry after a failure must not cause a duplicate result.
- Resilience patterns. Use timeouts, retries, and circuit breakers (see
11-circuit-breaker.md). - Immutable infrastructure. Replace a broken server. Do not repair it manually.
13. Verify with chaos engineering
You must test the self-healing system. If you do not test it, you do not know if it works.
- Write a hypothesis. Example: "If one node stops, the service continues with no errors."
- Inject a failure in a controlled way. Start in a test environment.
- Measure the result.
- Fix the gaps that you find.
- Repeat the test regularly. Then move to production with a small blast radius.
Tools: Chaos Monkey, LitmusChaos, Chaos Mesh, AWS Fault Injection Service, Gremlin.
14. Metrics to measure success
| Metric | Meaning |
|---|---|
| MTTD (mean time to detect) | Time from failure to detection. |
| MTTR (mean time to repair) | Time from failure to full recovery. |
| Auto-remediation rate | Percentage of incidents repaired without a human. |
| Repeated repairs | Number of times the same repair occurs. A high number shows a root cause to fix. |
| Pages per week | Number of alerts that call an engineer. |
15. Trade-offs
- Automation vs control: More automation gives faster repair. But a bug in the automation can cause a large incident.
- Sensitive checks vs stable system: Sensitive health checks find failures fast. But they can cause unnecessary restarts.
- Extra capacity vs cost: Spare instances make repairs fast. But they cost money when not in use.
- Self-heal vs root cause: Automatic restarts hide problems. Always record repairs and fix the root cause.
16. Interview follow-up questions
- What is the difference between a liveness probe and a readiness probe?
- Why must a liveness probe not check the database?
- How do you prevent the automation from restarting all instances at the same time?
- How do you prevent split brain during a database failover?
- How do you decide if a failure needs an automatic repair or a human?
- How do you test that the self-healing system works?