Ubuntu Zero-Downtime Patching: Unattended Upgrades and Livepatch

Ubuntu Zero-Downtime Patching: Unattended Upgrades and Livepatch

Security patching on production servers involves two distinct problems: keeping packages up to date (which usually requires service restarts) and keeping the kernel up to date (which requires a reboot). Ubuntu has dedicated tools for both:

• unattended-upgrades — automatically applies security patches to packages on a schedule
• Livepatch — patches the running kernel in memory without a reboot

Used together, most CVEs get patched without any planned downtime.

unattended-upgrades

Install and enable

apt update
apt install unattended-upgrades apt-listchanges -y

# Enable automatic upgrades
dpkg-reconfigure -plow unattended-upgrades
# Answer: Yes

Or enable manually:

cat > /etc/apt/apt.conf.d/20auto-upgrades << 'EOF'
APT::Periodic::Update-Package-Lists "1";     // Update package lists daily
APT::Periodic::Unattended-Upgrade "1";       // Run upgrades daily
APT::Periodic::AutocleanInterval "7";        // Remove old packages weekly
EOF

Configure what gets upgraded

/etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Allowed-Origins {
    // Security patches only — safe for automatic application
    "${distro_id}:${distro_codename}-security";

    // Ubuntu Pro ESM security patches (if you have Ubuntu Pro)
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";

    // Uncomment to also apply general updates (higher risk — test first)
    // "${distro_id}:${distro_codename}-updates";
};

// Packages to never auto-update (you manage these manually)
Unattended-Upgrade::Package-Blacklist {
    // "nginx";
    // "postgresql-*";
};

// Reboot handling
Unattended-Upgrade::Automatic-Reboot "false";       // Don't auto-reboot — handle manually
Unattended-Upgrade::Automatic-Reboot-Time "02:00";  // If you do auto-reboot, use this window

// Email on failure
Unattended-Upgrade::Mail "[email protected]";
Unattended-Upgrade::MailReport "on-change";

// Remove unused kernel packages after upgrade
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";

Verify it's working

1# Run manually to confirm config is correct
2unattended-upgrade --debug --dry-run
3
4# Check the log
5tail -f /var/log/unattended-upgrades/unattended-upgrades.log
6
7# See what would be upgraded
8apt list --upgradable 2>/dev/null | grep -i security

Livepatch — Kernel Patching Without Reboots

Livepatch patches CVEs directly into the running kernel — in memory — without requiring a reboot. The patched kernel binary on disk is also updated, so the next boot starts a clean patched kernel.

Livepatch is free for up to 5 machines via Ubuntu Pro (previously Ubuntu Advantage). For more machines, Ubuntu Pro subscription is required.

Enable Livepatch

# 1. Install the Livepatch client
snap install canonical-livepatch

# 2. Get a token from ubuntu.com/security/livepatch (free tier: up to 5 machines)
# Then enable:
sudo canonical-livepatch enable <your-token>

Check status

sudo canonical-livepatch status

client-version: 10.x.x
machine-id: <uuid>
status:
- kernel: 6.8.0-49-generic
  livepatch:
    checkState: checked
    patchState: applied
    version: "99.1"
    fixes: "CVE-2024-xxxx, CVE-2024-yyyy"

# Verbose — see each individual patch
sudo canonical-livepatch status --verbose

patchState: applied means the kernel is patched and running the latest Livepatch fixes.

What Livepatch covers

Livepatch covers high and critical kernel CVEs. It does not cover every kernel update — only security-relevant fixes for the running kernel version. You still need to reboot eventually to pick up new kernel versions (feature updates, major version upgrades, security fixes outside Livepatch scope).

Detecting Required Reboots and Service Restarts

Even with Livepatch, some patches require action:

Is a reboot required?

1# The reboot-required file is created by the update-notifier-common package
2# after a kernel update, libc update, or other reboot-requiring change
3test -f /var/run/reboot-required && echo "REBOOT REQUIRED" || echo "No reboot needed"
4
5# Which packages triggered the requirement
6cat /var/run/reboot-required.pkgs
7# linux-image-6.8.0-51-generic
8# libc6

Which services need restarting (without a full reboot)?

When shared libraries (glibc, OpenSSL) are updated, running services continue using the old in-memory versions until restarted. needrestart identifies them:

1apt install needrestart -y
2
3# Show services that need restarting (non-interactive)
4needrestart -b -r l
5
6# Output looks like:
7# NEEDRESTART-SVC: nginx.service
8# NEEDRESTART-SVC: postgresql.service

# Restart all affected services automatically
needrestart -b -r a

# Or restart specific services manually
systemctl restart nginx postgresql

Running needrestart after applying security patches to OpenSSL or glibc is important — services keep using the vulnerable library until restarted.

Rolling Node Patching for Kubernetes Clusters

For Kubernetes nodes, patch one node at a time to maintain cluster availability:

1#!/usr/bin/env bash
2# patch-node.sh — drain, patch, reboot if needed, uncordon
3set -euo pipefail
4
5NODE=${1:?Usage: $0 <node-name>}
6
7echo "==> Draining $NODE"
8kubectl drain "$NODE" \
9    --ignore-daemonsets \
10    --delete-emptydir-data \
11    --grace-period=60 \
12    --timeout=300s
13
14echo "==> Patching $NODE"
15ssh "$NODE" "apt update && apt upgrade -y && apt autoremove -y"
16
17echo "==> Restarting affected services"
18ssh "$NODE" "needrestart -b -r a 2>/dev/null || true"
19
20if ssh "$NODE" "test -f /var/run/reboot-required"; then
21    echo "==> Reboot required — rebooting $NODE"
22    ssh "$NODE" "reboot" || true
23
24    echo "==> Waiting for $NODE to come back..."
25    sleep 30
26    until ssh -o ConnectTimeout=5 "$NODE" "echo alive" 2>/dev/null; do
27        echo "   Still waiting..."
28        sleep 10
29    done
30    sleep 15    # Extra wait for kubelet to re-register
31fi
32
33echo "==> Uncordoning $NODE"
34kubectl uncordon "$NODE"
35
36echo "==> Done. $NODE is back in rotation."
37kubectl get node "$NODE"

Usage:

# Patch nodes one at a time
for node in $(kubectl get nodes -o name | sed 's/node\///'); do
    ./patch-node.sh "$node"
    sleep 60    # Allow workloads to reschedule before draining next node
done

Tracking Ubuntu Security Notices (USNs)

Ubuntu publishes all security advisories as Ubuntu Security Notices at ubuntu.com/security/notices. Subscribe to the ubuntu-security-announce mailing list or use the CLI:

1# Show security status of installed packages
2ubuntu-security-status
3# Output shows which packages have available security updates
4# and whether Ubuntu Pro (ESM) is needed for extended support
5
6# Check for CVEs affecting your installed packages
7apt install debian-goodies -y
8checkrestart     # Lists services that need restarting after upgrades

Recommended Configuration Summary

See Also

• Linux Commands for Advanced Engineers — systemd, sysctl, and service management
• Fix: Kubernetes Pending Pods — what happens to pods when you cordon a node

Frequently Asked Questions

Does Livepatch remove the need to reboot?

No, it defers it. Livepatch applies critical fixes to the running kernel so you can postpone the reboot. Separately, apt keeps installing new kernel packages, and the machine must eventually restart to run one. Treat Livepatch as scheduling flexibility, not a permanent substitute.

How do I know a reboot is actually required?

Check for the reboot-required marker file, which the package system creates when an update needs one. For service restarts rather than reboots, needrestart identifies processes still running against replaced libraries — that is the case people miss, since the patch is installed but the vulnerable code is still in memory.

Is unattended-upgrades safe on production?

For security updates on a well-tested distribution, generally yes, and the risk of not patching is usually higher. Configure it to security updates only, exclude packages you pin deliberately, and stagger across nodes so an unexpected regression does not land everywhere at once.

How should I patch Kubernetes nodes?

Cordon, drain, patch, reboot, uncordon — one node at a time, respecting PodDisruptionBudgets so draining does not take capacity below what your workloads need. On managed node groups, replacing the node with a new image is usually cleaner than patching in place, since it also refreshes everything else.