Engineering FAQ

DevOps is a cultural philosophy that bridges development and operations through shared ownership, fast feedback, and CI/CD. SRE (Site Reliability Engineering), pioneered by Google, is a concrete implementation of DevOps principles using software engineering to solve operational problems. The key SRE additions are quantified reliability targets (SLOs), error budgets that govern release velocity, and toil reduction as a first-class engineering goal.

Defined in the Google SRE book: Latency (time to serve a request — distinguish successful vs error latency), Traffic (demand on the system: requests/sec, messages/sec), Errors (rate of failed requests — explicit 500s and implicit wrong-data 200s), and Saturation (how 'full' the service is — CPU, memory, queue depth, disk I/O). If you can only instrument four things, instrument these.

An error budget is the allowed downtime or failure derived from your SLO. If your availability SLO is 99.9%, you have 0.1% budget — that's 43.8 minutes/month. If your error budget is exhausted, you freeze releases until it refills. If budget remains with weeks left, teams can ship riskier changes. The formula is: Error Budget = (1 − SLO target) × measurement window. Error budgets align development and operations by giving both sides a shared, objective measure of reliability.

Toil is manual, repetitive, automatable operational work that scales linearly with service growth and produces no lasting value. Examples: manually restarting pods, resolving the same alert by hand each time, manually rotating credentials. The SRE book recommends keeping toil below 50% of an engineer's time, with the rest on project work that reduces toil or improves reliability. High toil is a scaling problem — it grows as your service grows.

The four key DevOps Research and Assessment (DORA) metrics that predict software delivery performance: Deployment Frequency (how often you deploy to production), Lead Time for Changes (time from commit to production), Change Failure Rate (percentage of deployments causing a production failure), and Failed Deployment Recovery Time (time to restore service after a failure). Elite performers deploy multiple times per day with lead times under one hour and recovery times under one hour.

CrashLoopBackOff means the container is crashing and Kubernetes is applying exponential backoff before restarting it. Debug order: (1) kubectl logs <pod> --previous to see the crash output. (2) kubectl describe pod <pod> — check Events for OOMKilled (exit code 137 = OOM), ImagePullBackOff, or failed volume mounts. (3) kubectl get events --sort-by='.lastTimestamp'. Common causes: application crash on startup, missing env vars or secrets, resource limits too low (OOMKilled), or a broken liveness probe killing the pod before it's ready.

Deployments manage stateless pods — all replicas are interchangeable, scheduled on any node, and replaced with a new randomly-named pod. StatefulSets manage stateful pods that need stable network identity (pod-0, pod-1), stable persistent storage (PVCs that survive pod deletion), and ordered startup/shutdown. Use StatefulSets for databases, Kafka brokers, Elasticsearch nodes, or any service where pod identity matters. StatefulSets are more complex to operate — prefer managed databases when possible.

HPA (Horizontal Pod Autoscaler) adds or removes pod replicas based on metrics (CPU, memory, custom). It scales out. VPA (Vertical Pod Autoscaler) adjusts the CPU and memory requests/limits of existing pods based on observed usage — it scales up. VPA in Recommend mode is useful for rightsizing without disruption; in Auto mode it evicts and restarts pods to apply new resource values, which is disruptive. HPA and VPA conflict if both react to CPU — use VPA for Recommendation only alongside HPA, or use KEDA for event-driven scaling.

A PodDisruptionBudget (PDB) limits how many pods of a deployment can be simultaneously unavailable during voluntary disruptions (node drain, cluster upgrades, kubectl rollout). Example: minAvailable: 2 means at least 2 pods must stay Running during a drain. Without a PDB, a node drain can terminate all pods of a single-node deployment simultaneously, causing downtime. You need a PDB for any production workload with more than one replica. Set minAvailable or maxUnavailable based on how many replicas you have.

Readiness probe: determines whether the pod should receive traffic. If it fails, the pod is removed from the Service endpoint list — no traffic is sent, but the pod is not restarted. Use this for startup warmup or dependency unavailability. Liveness probe: determines whether the pod is alive. If it fails repeatedly, kubelet restarts the container. Use this to recover from deadlocks or infinite loops that leave the process running but non-functional. A misconfigured liveness probe that fires too early causes CrashLoopBackOff — always set initialDelaySeconds conservatively.

Kubernetes Secrets are base64-encoded, not encrypted by default — anyone with etcd access can read them. Three better approaches: (1) External Secrets Operator (ESO): syncs secrets from AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault into Kubernetes Secrets — the source of truth is always external. (2) Vault Agent Sidecar / CSI Driver: injects secrets directly into pod filesystems at runtime without creating Kubernetes Secret objects. (3) Sealed Secrets: encrypt secrets with a cluster-specific public key; the sealed YAML is safe to commit to Git. Enable Kubernetes etcd encryption at rest as a baseline regardless of approach.

Helm is a package manager and templating engine — it distributes versioned, reusable chart packages and manages release lifecycle (install, upgrade, rollback). Use Helm when consuming third-party software (cert-manager, ArgoCD, Prometheus). Kustomize is a configuration overlay system — no templates, just strategic merges and JSON patches on top of base manifests. Use Kustomize for managing your own app manifests across environments (dev/staging/prod). They compose well: use Helm to install third-party charts and Kustomize to patch the outputs.

ALB (Application Load Balancer) operates at Layer 7 (HTTP/HTTPS/WebSocket/gRPC). Use it for host/path-based routing, header-based routing, authentication (Cognito/OIDC), and WAF integration. NLB (Network Load Balancer) operates at Layer 4 (TCP/UDP/TLS). Use it when you need static IP addresses or Elastic IPs (required for IP-based whitelisting), ultra-low latency, PrivateLink endpoints, or non-HTTP protocols. NLB preserves the client source IP natively; ALB requires the X-Forwarded-For header.

mTLS (mutual TLS) requires both the client and server to present valid TLS certificates — each authenticates the other's identity. Standard TLS only authenticates the server. Use mTLS for service-to-service communication in microservices where you need to verify the caller's identity (not just encrypt the channel). Service meshes (Istio, Linkerd, Cilium) automate mTLS issuance and rotation via short-lived certificates without code changes. Use it when network-level identity enforcement matters — e.g., HIPAA ePHI flows or PCI-DSS CDE internal traffic.

An API Gateway sits at the edge of your system — it's the entry point for external clients. It handles authentication/authorisation, rate limiting, request routing, and protocol translation (REST → gRPC). A Service Mesh is inside the cluster — it manages east-west (service-to-service) traffic using sidecar proxies (Envoy in Istio, Linkerd-proxy). It provides mTLS, observability (per-service latency/error traces), and traffic policies between services without changing application code. You typically need both: API Gateway for north-south (external) traffic, Service Mesh for east-west (internal) traffic.

Drift occurs when the actual state of your cloud resources diverges from what your IaC code describes — usually from manual console changes, automated processes, or resource replacement. terraform plan compares state file against real cloud state and shows drift as changes it would make. Prevent drift by: making production consoles read-only for humans, routing all changes through the deploy pipeline, and running terraform plan on a schedule with alerting on non-empty plans.

Terraform (HCL): mature ecosystem, 3,000+ providers, best community support, declarative state model. Best for teams wanting an ops-centric tool with a broad provider ecosystem. Pulumi: uses real programming languages (TypeScript, Python, Go) — full loops, functions, and type safety. Best for teams with strong software engineering backgrounds or complex conditional logic. AWS CDK: TypeScript/Python constructs that synthesise to CloudFormation. Best for AWS-only shops comfortable with CloudFormation and wanting higher-level constructs. For multi-cloud or greenfield projects, Terraform or OpenTofu is the lowest-risk default.

A canary release deploys a new version to a small percentage of traffic (1–5%) before rolling it out fully, letting you observe error rates, latency, and business metrics on real users before committing. If metrics degrade, roll back — only a fraction of users were affected. Implement at the load balancer (ALB weighted target groups, Nginx split_clients), Kubernetes (Argo Rollouts, Flagger with Istio/Linkerd), or feature flag layer. Progressive delivery tools like Argo Rollouts automate promotion and rollback based on Prometheus metrics.

RBAC (Role-Based Access Control) in Kubernetes controls who can perform which verbs (get, list, create, delete, patch) on which resources (pods, secrets, configmaps) in which namespaces. Without it, any compromised service account or misconfigured pod can read all Secrets cluster-wide or escalate privileges. Best practices: give service accounts namespace-scoped Roles (not ClusterRoles) with only the permissions the workload actually needs, audit with kubectl auth can-i --list --as=<serviceaccount>, and use tools like rbac-lookup or Rakkess to visualise effective permissions.

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that lets you run sandboxed programs in the kernel without modifying kernel source or loading kernel modules. For platform engineering it enables: (1) Observability — trace syscalls, network packets, and function calls with near-zero overhead (Pixie, Tetragon). (2) Networking — high-performance, programmable packet processing in Kubernetes networking (Cilium replaces kube-proxy with eBPF). (3) Security — detect and block malicious syscalls at the kernel level (Falco, Tetragon). eBPF-based tools eliminate the sidecar proxy overhead of traditional service meshes.