CAPLV Demo

Three operators work together on a Single Node OpenShift (SNO) cluster to schedule ephemeral libvirt VMs as CAPI worker nodes.

Component 1: CAPLV (Cluster API Provider Libvirt)

Provisions libvirt VMs over SSH. Manages the full VM lifecycle: domain creation, UEFI boot, networking, disk cloning, and teardown.

LibvirtCluster CAPLV

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: LibvirtCluster
metadata:
  name: spot
  namespace: default
spec:
  controlPlaneEndpoint:
    host: api.spot.labdroid.net
    port: 6443

LibvirtHost — laptop CAPLV

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: LibvirtHost
metadata:
  name: laptop
  namespace: default
spec:
  uri: qemu+ssh://caplv@192.168.122.1/system
  secretRef:
    name: laptop-ssh-key
  hostKeyFingerprint: SHA256:hAoYrhR9rv7VrGx4JAyeZcpDNdy30MOv23Bbo72iwo8
  firmwarePath: /usr/share/OVMF/OVMF_CODE.fd
  nvramTemplatePath: /usr/share/OVMF/OVMF_VARS.fd
  healthCheckIntervalSeconds: 300
# status: 16 vCPUs, 31560 MB total, reachable

Cluster CAPI

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: spot
  namespace: default
spec:
  controlPlaneEndpoint:
    host: api.spot.labdroid.net
    port: 6443
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: LibvirtCluster
    name: spot

Component 2: capi-bootstrap-ignition

Generates Ignition/Butane bootstrap data for Fedora CoreOS VMs. Points to a Secret containing the raw Ignition JSON.

IgnitionConfig Ignition

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: IgnitionConfig
metadata:
  name: spot-worker-01   # auto-generated by 5-Spot
  namespace: default
spec:
  secretRef:
    name: worker-ignition  # Secret with key "value" containing Ignition JSON

worker-ignition Secret (pre-created)

apiVersion: v1
kind: Secret
metadata:
  name: worker-ignition
  namespace: default
type: Opaque
data:
  value: <base64-encoded Ignition JSON>
# Contains: SSH keys, NM cleanup scripts, kubelet join config, etc.

Component 3: 5-Spot (Scheduled Machine Controller)

Time-based machine lifecycle manager. Creates and destroys CAPI Machine resources (plus their bootstrap and infrastructure refs) based on a day/hour schedule.

ScheduledMachine 5-Spot

This is what you create to add a spot worker:

apiVersion: 5spot.finos.org/v1alpha1
kind: ScheduledMachine
metadata:
  name: spot-worker-01
  namespace: default
spec:
  clusterName: spot

  schedule:
    enabled: true
    daysOfWeek:
      - mon-fri
    hoursOfDay:
      - "9-17"
    timezone: America/New_York

  infrastructureSpec:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: LibvirtMachine
    spec:
      hostRef:
        name: laptop
      bootstrapFormat: ignition
      domain:
        firmware: uefi
        machine: q35
        vcpus: 2
        memoryMB: 8192
      rootDisk:
        baseImage: rhcos-qemu.x86_64.qcow2
        storagePool: default
        size: 20Gi
      network:
        name: default
        type: network
        addresses:
          - 192.168.122.100/24
        gateway: 192.168.122.1

  bootstrapSpec:
    apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
    kind: IgnitionConfig
    spec:
      secretRef:
        name: worker-ignition

  machineTemplate:
    labels:
      node-role.kubernetes.io/worker: ""

  priority: 50
  gracefulShutdownTimeout: 5m
  nodeDrainTimeout: 5m
  killSwitch: false

Static IP Networking with OVN-Kubernetes

On OpenShift clusters using OVN-Kubernetes, the physical NIC is enslaved to an OVS bridge (br-ex) during boot. The configure-ovs.sh script handles this migration: if the physical NIC already has a static IP, it preserves it on br-ex. This is the same mechanism the Assisted Installer uses for bare metal nodes.

The worker-ignition secret is shared by all workers — it contains the generic Ignition config fetched from the cluster's Machine Config Server. At provisioning time, CAPLV merges per-VM configuration into a copy of this shared Ignition before booting the VM. For static networking, CAPLV injects an NMConnection file that configures the physical NIC with method=manual before any services start, so DHCP is never used.

Per-VM Ignition injection (done by CAPLV controller)

# From the shared worker-ignition secret, CAPLV injects:
/etc/hostname                           ← VM hostname
/etc/systemd/system/kubelet.service.d/
  90-provider-id.conf                   ← CAPI provider ID
/etc/NetworkManager/system-connections/
  enp1s0.nmconnection                  ← static IP for physical NIC

Boot sequence for static IP assignment

1. Ignition writes enp1s0.nmconnection (method=manual, no DHCP)
2. NetworkManager starts, enp1s0 gets the static IP immediately
3. nodeip-configuration detects the static IP (no DHCP race)
4. MCO pulls images, configures the node, reboots
5. configure-ovs.sh migrates enp1s0's static IP to br-ex
6. CRI-O and kubelet start with the correct IP on br-ex
7. Node joins the cluster as a worker

Lifecycle Flow

Quick Commands

# Create the scheduled worker
./demo-create-worker.sh

# Watch it come up
kubectl get scheduledmachines -w

# Check all resources
kubectl get cluster,libvirtcluster,libvirthost,libvirtmachine,machine,ignitionconfig,scheduledmachine

# Stack status
./deploy.sh status

# Tear down the worker
kubectl delete scheduledmachine spot-worker-01

Architecture