Kubernetes Operators: Building Controllers with controller-runtime
Kubernetes Operators automate the operational knowledge of stateful applications — creating databases, scaling clusters, managing backups, handling failover. The controller-runtime library (used by Kubebuilder and Operator SDK) provides the reconciliation loop, informer caching, and webhook scaffolding. This covers the core concepts: Custom Resource Definitions, reconciliation loops, status conditions, leader election, and the patterns that make operators production-ready.
An Operator is a Kubernetes controller that understands the operational requirements of a specific application. Where Kubernetes knows how to run a Deployment (keep N replicas running), an Operator knows how to run a PostgreSQL cluster (create a primary, provision replicas, configure streaming replication, take backups, handle failover). The operator encodes the operational runbook in code.
All Kubernetes controllers — including the ones built into kube-controller-manager — follow the same pattern: watch resources, compare desired state to actual state, act to reconcile the difference. The controller-runtime library provides this infrastructure for custom controllers.
Core Concepts
Custom Resource Definition (CRD) — extends the Kubernetes API with a new resource type. A CRD for PostgresCluster lets users create PostgresCluster objects the same way they create Deployment objects.
Custom Resource (CR) — an instance of a CRD. kubectl apply -f postgres-cluster.yaml creates a CR that the operator acts on.
Reconciler — the Go struct that implements the Reconcile(ctx, Request) (Result, error) method. The reconciler receives the name/namespace of a resource that may have changed, reads the current state, and makes API calls to drive toward the desired state.
Informer/Cache — watches the Kubernetes API for changes and caches resources locally. The controller-runtime manager sets this up automatically; reconcilers read from the cache (not the API server directly) to avoid rate limiting.
Project Setup with Kubebuilder
1# Install Kubebuilder — dynamically fetch the latest release
2KUBEBUILDER_VERSION=$(curl -s https://api.github.com/repos/kubernetes-sigs/kubebuilder/releases/latest | jq -r .tag_name)
3curl -L -o kubebuilder "https://github.com/kubernetes-sigs/kubebuilder/releases/download/${KUBEBUILDER_VERSION}/kubebuilder_linux_amd64"
4chmod +x kubebuilder && mv kubebuilder /usr/local/bin/
5
6# Initialize a new operator project
7mkdir postgres-operator && cd postgres-operator
8kubebuilder init --domain codingprotocols.com --repo github.com/codingprotocols/postgres-operator
9
10# Scaffold a new API (CRD + controller)
11kubebuilder create api --group db --version v1alpha1 --kind PostgresClusterThis generates:
api/v1alpha1/postgrescluster_types.go— CRD type definitionsinternal/controller/postgrescluster_controller.go— reconciler skeletonconfig/crd/bases/— CRD manifests generated from Go types
Defining the CRD
1// api/v1alpha1/postgrescluster_types.go
2
3// PostgresClusterSpec defines the desired state
4type PostgresClusterSpec struct {
5 // +kubebuilder:validation:Minimum=1
6 // +kubebuilder:validation:Maximum=7
7 Replicas int32 `json:"replicas"`
8
9 // +kubebuilder:validation:Pattern=`^\d+Gi$`
10 StorageSize string `json:"storageSize"`
11
12 // +kubebuilder:default="16"
13 // +kubebuilder:validation:Enum="14";"15";"16";"17"
14 PostgresVersion string `json:"postgresVersion,omitempty"`
15}
16
17// PostgresClusterStatus defines the observed state
18type PostgresClusterStatus struct {
19 // ReadyReplicas is the number of replicas that are ready
20 ReadyReplicas int32 `json:"readyReplicas,omitempty"`
21
22 // Phase is the current lifecycle phase
23 // +kubebuilder:validation:Enum=Creating;Running;Updating;Degraded;Failed
24 Phase string `json:"phase,omitempty"`
25
26 // Conditions records the detailed status
27 Conditions []metav1.Condition `json:"conditions,omitempty"`
28}
29
30// +kubebuilder:object:root=true
31// +kubebuilder:subresource:status
32// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
33// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
34// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
35// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
36type PostgresCluster struct {
37 metav1.TypeMeta `json:",inline"`
38 metav1.ObjectMeta `json:"metadata,omitempty"`
39 Spec PostgresClusterSpec `json:"spec,omitempty"`
40 Status PostgresClusterStatus `json:"status,omitempty"`
41}The +kubebuilder: markers generate validation rules in the CRD manifest via make generate. The +kubebuilder:subresource:status marker creates a separate status subresource so that status updates don't conflict with spec updates.
The Reconciler
1// internal/controller/postgrescluster_controller.go
2
3type PostgresClusterReconciler struct {
4 client.Client
5 Scheme *runtime.Scheme
6}
7
8func (r *PostgresClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
9 log := log.FromContext(ctx)
10
11 // 1. Fetch the PostgresCluster resource
12 cluster := &dbv1alpha1.PostgresCluster{}
13 if err := r.Get(ctx, req.NamespacedName, cluster); err != nil {
14 // Resource was deleted — nothing to do
15 return ctrl.Result{}, client.IgnoreNotFound(err)
16 }
17
18 // 2. Check if the resource is being deleted (finalizer pattern)
19 if !cluster.DeletionTimestamp.IsZero() {
20 return r.handleDeletion(ctx, cluster)
21 }
22
23 // 3. Add finalizer if not present
24 if !controllerutil.ContainsFinalizer(cluster, "db.codingprotocols.com/finalizer") {
25 controllerutil.AddFinalizer(cluster, "db.codingprotocols.com/finalizer")
26 if err := r.Update(ctx, cluster); err != nil {
27 return ctrl.Result{}, err
28 }
29 }
30
31 // 4. Reconcile the StatefulSet
32 if err := r.reconcileStatefulSet(ctx, cluster); err != nil {
33 // Update status to reflect the error
34 meta.SetStatusCondition(&cluster.Status.Conditions, metav1.Condition{
35 Type: "Ready",
36 Status: metav1.ConditionFalse,
37 Reason: "ReconcileError",
38 Message: err.Error(),
39 ObservedGeneration: cluster.Generation,
40 })
41 _ = r.Status().Update(ctx, cluster)
42 return ctrl.Result{}, err
43 }
44
45 // 5. Update status
46 meta.SetStatusCondition(&cluster.Status.Conditions, metav1.Condition{
47 Type: "Ready",
48 Status: metav1.ConditionTrue,
49 Reason: "Reconciled",
50 Message: "PostgresCluster is healthy",
51 ObservedGeneration: cluster.Generation,
52 })
53 if err := r.Status().Update(ctx, cluster); err != nil {
54 return ctrl.Result{}, err
55 }
56
57 log.Info("Reconciled PostgresCluster", "name", cluster.Name)
58
59 // Requeue after 5 minutes to detect drift even without watch events
60 return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
61}Key patterns:
- IgnoreNotFound on Get: the resource may have been deleted between the watch event and the reconcile call. This is normal — don't return an error.
- Finalizer: add a finalizer before creating external resources (cloud storage, databases). This prevents the CR from being deleted before the operator cleans up external state.
- Status conditions: use
metav1.Condition(the standard condition type) withObservedGenerationso clients can detect whether conditions are stale. - RequeueAfter: periodic requeue catches drift that watch events miss (external changes to the StatefulSet, etc.).
Controller Setup
1// cmd/main.go
2
3func main() {
4 mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
5 Scheme: scheme,
6 // Leader election: only one controller instance is active at a time
7 LeaderElection: true,
8 LeaderElectionID: "postgres-operator.codingprotocols.com",
9 LeaderElectionNamespace: "postgres-operator-system",
10
11 // Metrics server (BindAddress moved to nested struct in recent controller-runtime)
12 Metrics: metricsserver.Options{
13 BindAddress: ":8080",
14 },
15 HealthProbeBindAddress: ":8081",
16
17 // Cache only the namespaces this operator manages
18 Cache: cache.Options{
19 DefaultNamespaces: map[string]cache.Config{
20 "payments": {},
21 "analytics": {},
22 },
23 },
24 })
25
26 if err := (&controller.PostgresClusterReconciler{
27 Client: mgr.GetClient(),
28 Scheme: mgr.GetScheme(),
29 }).SetupWithManager(mgr); err != nil {
30 setupLog.Error(err, "unable to create controller")
31 os.Exit(1)
32 }
33
34 mgr.AddHealthzCheck("healthz", healthz.Ping)
35 mgr.AddReadyzCheck("readyz", healthz.Ping)
36
37 if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
38 setupLog.Error(err, "problem running manager")
39 os.Exit(1)
40 }
41}1// SetupWithManager registers watches
2func (r *PostgresClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
3 return ctrl.NewControllerManagedBy(mgr).
4 For(&dbv1alpha1.PostgresCluster{}). // Watch the primary resource
5 Owns(&appsv1.StatefulSet{}). // Also reconcile when owned StatefulSets change
6 Owns(&corev1.Service{}).
7 WithOptions(controller.Options{
8 MaxConcurrentReconciles: 3, // Process up to 3 clusters concurrently
9 }).
10 Complete(r)
11}Owns sets up watches on StatefulSets and Services that have the PostgresCluster as their owner reference. When the StatefulSet changes (pod restarted, replica count changed), the controller requeues the parent PostgresCluster for reconciliation.
Idempotent Resource Creation
Reconcilers run repeatedly — the code that creates the StatefulSet must handle "already exists" without erroring:
1func (r *PostgresClusterReconciler) reconcileStatefulSet(ctx context.Context, cluster *dbv1alpha1.PostgresCluster) error {
2 desired := r.buildStatefulSet(cluster)
3
4 // Set owner reference so the StatefulSet is garbage-collected with the CR
5 if err := controllerutil.SetControllerReference(cluster, desired, r.Scheme); err != nil {
6 return err
7 }
8
9 existing := &appsv1.StatefulSet{}
10 err := r.Get(ctx, client.ObjectKeyFromObject(desired), existing)
11 if apierrors.IsNotFound(err) {
12 return r.Create(ctx, desired)
13 }
14 if err != nil {
15 return err
16 }
17
18 // Merge: only update fields we manage, preserve fields set by other controllers
19 patch := client.MergeFrom(existing.DeepCopy())
20 existing.Spec.Replicas = desired.Spec.Replicas
21 existing.Spec.Template = desired.Spec.Template
22 return r.Patch(ctx, existing, patch)
23}Using client.MergeFrom + Patch (instead of Update) preserves fields that the Kubernetes controllers set (e.g., resourceVersion, status). Only the fields you explicitly set are sent in the patch.
Modern Reconciliation: Server-Side Apply (SSA)
In 2026, Patch and Update are increasingly replaced by Server-Side Apply (SSA). SSA allows the controller to declare ownership of specific fields, leaving other fields (like HPA-managed replicas) to be managed by other controllers without conflicts:
Important: The object passed to client.Apply must have TypeMeta (.Kind and .APIVersion) explicitly set. Omitting TypeMeta causes the API server to return 'invalid apply configuration'. This is the most common SSA gotcha.
1func (r *PostgresClusterReconciler) reconcileWithSSA(ctx context.Context, cluster *dbv1alpha1.PostgresCluster) error {
2 desired := r.buildStatefulSet(cluster)
3 // TypeMeta must be set explicitly — SSA requires Kind and APIVersion
4 desired.TypeMeta = metav1.TypeMeta{
5 Kind: "StatefulSet",
6 APIVersion: "apps/v1",
7 }
8
9 // SSA uses a "Patch" with an "Apply" type
10 return r.Patch(ctx, desired, client.Apply, &client.PatchOptions{
11 FieldManager: "postgres-operator",
12 Force: ptr.To(true),
13 })
14}SSA simplifies the "Merge" logic by shifting the responsibility for field merging to the API server. This eliminates the need to Get the resource before patching, reducing API calls and improving reconciliation performance.
Frequently Asked Questions
When should I build an Operator vs use Helm or plain manifests?
Operators are worth the complexity when the application has operational requirements that can't be expressed as static YAML: failure detection and remediation, multi-step provisioning workflows, backup scheduling, version upgrades with data migrations. If the application is stateless or has a well-maintained upstream Helm chart, that's usually the right starting point. Operators are for encoding operational knowledge, not just packaging.
What's the difference between Kubebuilder and Operator SDK?
Both use the same controller-runtime library underneath. Kubebuilder is the upstream project from the Kubernetes SIG; Operator SDK is Red Hat's extension of Kubebuilder with Ansible and Helm operator support (for teams that don't want to write Go). For Go-based operators, either is fine — the code is essentially identical. Most production Go operators use the Kubebuilder scaffolding directly.
How do I handle long-running operations (database backups) in a reconciler?
Reconcilers should return quickly — a reconcile loop that blocks for 10 minutes holds up all other reconciliations for that controller instance. Pattern: start the backup as a Kubernetes Job, record the Job name in the CR status, and requeue. On the next reconcile, check the Job status. If complete, update the CR status. If still running, requeue again with RequeueAfter.
For operator design principles and CRD best practices (when to build vs adopt, CRD schema design, RBAC for operators), see Kubernetes Operators: Building Custom Controllers with CRDs. For StatefulSets that operators commonly manage, see Kubernetes StatefulSets: Running Stateful Workloads in Production. For RBAC that controls what operator service accounts can do in the cluster, see Kubernetes RBAC Advanced Patterns.
Building a custom Operator for your platform or evaluating whether an existing operator meets your production requirements? Talk to us at Coding Protocols — we help platform teams implement operators that automate complex stateful workloads without accumulating operational debt.
