# CAPLV — *EXPERIMENTAL* Cluster API Provider for LibVirt

CAPLV is an *EXPERIMENTAL* [Cluster API](https://cluster-api.sigs.k8s.io/)
infrastructure provider that provisions KVM virtual machines on libvirt
hosts. It works standalone as a plain CAPI infrastructure provider —
create `LibvirtMachine` and `Machine` resources directly to bring up
workers — and pairs naturally with
[5-Spot](https://github.com/finos/5-spot), which schedules OpenShift
worker nodes on and off physical RHEL/KVM infrastructure based on
time-of-day rules.

The target hosts are machines with incumbent workloads — DR standby
systems or servers with predictable load schedules that have idle
capacity during off-peak periods. CAPLV is designed to be minimally
disruptive to these hosts: worker-node VMs are fully ephemeral and,
when backed by a tmpfs storage pool, won't touch persistent storage
on the device at all.

## How It Works

```
5-Spot: schedule becomes active
  → Creates IgnitionConfig + LibvirtMachine + CAPI Machine
    → capi-bootstrap-ignition copies ignition data, reports ready
      → CAPLV injects hostname + providerID into ignition config
        → Connects to libvirt host over SSH
          → Clones RHCOS base image, writes ignition config
            → Defines and starts KVM domain
              → VM boots, kubelet starts, CSR auto-approved
                → Node joins this OpenShift cluster as worker

5-Spot: schedule becomes inactive
  → Deletes CAPI Machine
    → CAPI drains pods
      → CAPLV destroys domain, cleans up all artifacts, deletes Node object
```

CAPLV runs on the same OpenShift cluster that the worker VMs join.

Each libvirt host runs exactly one CAPLV-managed VM — enforced by an
admission webhook that rejects a second machine targeting the same host.
Hundreds or thousands of hosts can come online simultaneously — the
controller runs up to 50 concurrent reconcilers by default
(`--max-concurrent-reconciles`), each operating against a different host
over its own SSH connection.

## Cluster Setup

Before creating any ScheduledMachine resources, the OpenShift cluster
needs these one-time prerequisites:

**1. Install CAPI, 5-Spot, CAPLV, and
[capi-bootstrap-ignition](https://github.com/atgreen/capi-bootstrap-ignition)**
on the OpenShift cluster. The bootstrap provider bridges OpenShift's
ignition-based worker bootstrap with the CAPI bootstrap contract.

**2. Create a CAPI Cluster resource** pointing to itself:

```yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: my-ocp-cluster
  namespace: default
spec:
  controlPlaneEndpoint:
    host: "api.my-ocp-cluster.example.com"
    port: 6443
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: LibvirtCluster
    name: my-ocp-cluster
    namespace: default
```

**3. Create a LibvirtCluster resource** (CAPLV verifies the endpoint is
reachable before reporting ready):

```yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: LibvirtCluster
metadata:
  name: my-ocp-cluster
  namespace: default
spec:
  controlPlaneEndpoint:
    host: "api.my-ocp-cluster.example.com"
    port: 6443
```

**4. Register libvirt hosts** (one resource per physical host):

```yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: LibvirtHost
metadata:
  name: rhel-host-01
spec:
  uri: "qemu+ssh://caplv@rhel-host-01.example.com/system"
  secretRef:
    name: rhel-host-01-ssh-key
  hostKeyFingerprint: "SHA256:abc123..."
  reservedResources:             # leave this much for the incumbent workload
    vcpus: 2                     # default: 2
    memoryMB: 4096               # default: 4096
  healthCheckIntervalSeconds: 300  # default: 300 (5 min), only while machines exist
```

The controller discovers total host capacity via `virsh nodeinfo` and
reports available resources (total minus reserved) in
`status.capacity.availableVCPUs` / `status.capacity.availableMemoryMB`.

**5. Create a service account on each host** with minimal privileges:

```bash
# Create the caplv user with libvirt group membership.
useradd -r -s /bin/bash -G libvirt caplv
mkdir -p /home/caplv/.ssh
# Deploy the SSH public key matching the Kubernetes Secret.
cat > /home/caplv/.ssh/authorized_keys <<< "<public-key>"
chmod 700 /home/caplv/.ssh
chmod 600 /home/caplv/.ssh/authorized_keys
chown -R caplv:caplv /home/caplv/.ssh
```

Grant restricted sudo for tmpfs and file operations only:

```
# /etc/sudoers.d/caplv
caplv ALL=(root) NOPASSWD: /usr/bin/mkdir -p /run/caplv/*
caplv ALL=(root) NOPASSWD: /usr/bin/mount -t tmpfs tmpfs /run/caplv/*
caplv ALL=(root) NOPASSWD: /usr/bin/mount -t tmpfs -o size=* tmpfs /run/caplv/*
caplv ALL=(root) NOPASSWD: /usr/bin/umount /run/caplv/*
caplv ALL=(root) NOPASSWD: /usr/bin/rmdir /run/caplv/*
caplv ALL=(root) NOPASSWD: /usr/bin/tee /run/caplv/*
caplv ALL=(root) NOPASSWD: /usr/bin/rm -f /run/caplv/*
```

The `libvirt` group grants access to `virsh` commands against
`qemu:///system` without sudo. Only tmpfs mount/unmount and file writes
under `/run/caplv/` require elevated privileges.

The account name is arbitrary — the ansible playbook parameterizes it as
`caplv_user` (`-e caplv_user=<name>`), and the controller uses whatever
user appears in the `LibvirtHost` URI.

### Session mode (unprivileged libvirt)

By default VMs are managed by the root-owned system libvirt daemon
(`/system` in the URI). Hosts that must not run a privileged VM manager
can instead use the service account's per-user daemon by ending the URI
with `/session`:

```yaml
spec:
  uri: "qemu+ssh://caplv@rhel-host-01.example.com/session"
```

In session mode QEMU runs as the service account itself. Because an
unprivileged process cannot create tap devices, bridge attachment goes
through QEMU's setuid `qemu-bridge-helper`, which only attaches to
bridges whitelisted in `/etc/qemu/bridge.conf`. The ansible playbook
configures all of this:

```bash
ansible-playbook -i <host>, -u root deploy/ansible/setup-host.yaml \
  -e libvirt_mode=session \
  -e '{"session_allowed_bridges": ["br0"]}'
```

which additionally: adds the service account to the `kvm` group (direct
`/dev/kvm` access), enables `loginctl` lingering (so VMs survive SSH
disconnect and the user daemon can autostart), makes `qemu-bridge-helper`
setuid, whitelists the bridges, and creates the storage pool under
`~caplv/.local/share/libvirt/images` in the user session.

The `LibvirtHost` health probe verifies session hosts end to end: on top
of the usual connectivity and KVM checks (which run against the session
daemon), it confirms `qemu-bridge-helper` is setuid (or has
`cap_net_admin`) and that lingering is enabled, marking the host
`Ready=false` with reason `SessionModeMisconfigured` otherwise.

Session mode limitations:

- Machines must use `network.type: bridge`. Libvirt-managed NAT networks
  (`network.type: network`) require the system daemon. The machine
  controller rejects this combination up front (terminal
  `NetworkTypeUnsupported` condition).
- `qemu-bridge-helper` does not support multi-queue virtio-net.
- On SELinux-enforcing hosts, sVirt isolation between guests is reduced
  (session guests run unconfined as the service account). If a guest
  image fails to open, check file contexts under the session pool path.

**6. Pre-stage RHCOS base images** on each libvirt host in the persistent
storage pool (e.g., `/var/lib/libvirt/images/rhcos.qcow2`). The RHCOS
version should match the OpenShift cluster version (e.g., use the 4.21
image for an OCP 4.21 cluster). Download from
`https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/<version>/latest/`.

**7. Deploy a MachineHealthCheck** (recommended — see below).

## Custom Resources

| CRD | Purpose |
|-----|---------|
| **LibvirtHost** | Reusable connection details for a libvirt hypervisor (URI, SSH key, OVMF paths, reserved resources). Discovers host capacity via `virsh nodeinfo`. Health-checked only while machines are active. |
| **LibvirtCluster** | CAPI contract resource — verifies the control plane endpoint is reachable (TCP dial) before reporting ready. No infrastructure provisioned. |
| **LibvirtMachine** | Core resource — a single KVM VM with static IP, UEFI firmware, and ignition or cloud-init bootstrap. VM size can be auto-derived from host capacity. One per host (webhook-enforced). |

## Example

### Schedule a worker via 5-Spot

```yaml
apiVersion: 5spot.finos.org/v1alpha1
kind: ScheduledMachine
metadata:
  name: weekday-worker-01
spec:
  clusterName: my-ocp-cluster
  schedule:
    daysOfWeek: ["mon-fri"]
    hoursOfDay: ["9-17"]
    timezone: "America/Toronto"
    enabled: true
  infrastructureSpec:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: LibvirtMachine
    spec:
      hostRef:
        name: rhel-host-01
      domain: {}                   # auto-sized from host capacity minus reserved
      rootDisk:
        size: "100Gi"
        storagePool: "vm-disks"
        baseImagePool: "default"
        baseImage: "rhcos.qcow2"
        ephemeralPool: true
      network:
        type: "bridge"
        name: "br0"
        addresses:
          - "192.168.1.50/24"
        gateway: "192.168.1.1"
        dns:
          nameservers:
            - "192.168.1.1"
      bootstrapFormat: "ignition"
```

### Create a worker directly

First, fetch the worker ignition from the Machine Config Server on a
cluster node and create a bootstrap secret:

```bash
ssh core@<node-ip> "sudo curl -sk \
  -H 'Accept: application/vnd.coreos.ignition+json;version=3.2.0' \
  https://localhost:22623/config/worker" > worker-ignition.json

kubectl create secret generic worker-bootstrap -n default \
  --from-file=value=worker-ignition.json \
  --from-literal=format=ignition
```

Then create a Machine + LibvirtMachine:

```yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: LibvirtMachine
metadata:
  name: worker01
  namespace: default
spec:
  hostRef:
    name: rhel-host-01
  domain:
    vcpus: 4
    memoryMB: 8192
  network:
    type: "bridge"
    name: "br0"
    addresses:
      - "192.168.1.50/24"
    gateway: "192.168.1.1"
    dns:
      nameservers:
        - "192.168.1.1"
  rootDisk:
    baseImage: "rhcos.qcow2"
    baseImagePool: "default"
    storagePool: "default"
    size: "100Gi"
  bootstrapFormat: "ignition"
---
apiVersion: cluster.x-k8s.io/v1beta2
kind: Machine
metadata:
  name: worker01
  namespace: default
  labels:
    cluster.x-k8s.io/cluster-name: my-ocp-cluster
spec:
  clusterName: my-ocp-cluster
  bootstrap:
    dataSecretName: worker-bootstrap
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: LibvirtMachine
    name: worker01
```

CAPLV will automatically:
1. Inject hostname (`default-my-ocp-cluster-worker01`) and providerID
2. Provision the VM on the libvirt host
3. Auto-approve the kubelet CSR
4. The node joins the cluster as a worker

To delete, just `kubectl delete machine worker01` — CAPLV destroys the
VM, cleans up storage, and removes the Node object.

### Using with 5-Spot ScheduledMachines

When using [5-Spot](https://github.com/finos/5-spot) to schedule workers,
install the [capi-bootstrap-ignition](https://github.com/atgreen/capi-bootstrap-ignition)
provider. This satisfies the CAPI bootstrap contract so 5-Spot's existing
`bootstrapSpec` flow works without modification:

```yaml
apiVersion: 5spot.finos.org/v1alpha1
kind: ScheduledMachine
metadata:
  name: spot-worker-01
spec:
  clusterName: my-ocp-cluster
  schedule:
    daysOfWeek: ["mon-fri"]
    hoursOfDay: ["9-17"]
    timezone: "America/New_York"
    enabled: true
  bootstrapSpec:
    apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
    kind: IgnitionConfig
    spec:
      secretRef:
        name: worker-ignition
  infrastructureSpec:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: LibvirtMachine
    spec:
      hostRef:
        name: rhel-host-01
      # ... VM configuration
```

The same `worker-ignition` Secret serves all ScheduledMachines in the cluster.

### Ephemeral storage with tmpfs

VMs are ephemeral — created and destroyed on demand by 5-Spot schedules.
To avoid touching persistent storage on hosts with incumbent workloads,
set `ephemeralPool: true`:

```yaml
rootDisk:
  storagePool: "vm-disks"        # CAPLV creates this as tmpfs on demand
  baseImagePool: "default"       # persistent pool with pre-staged base image
  baseImage: "rhcos.qcow2"
  ephemeralPool: true
  ephemeralPoolSize: "80%"       # optional cap on tmpfs RAM (default: kernel's 50%)
```

**No host setup required.** CAPLV creates a per-machine tmpfs mount and
libvirt storage pool when the VM is provisioned, and tears both down when
the VM is deleted. RAM is only consumed while the VM exists. The host's
persistent storage is never touched.

`ephemeralPoolSize` accepts tmpfs `size=` syntax: a percentage of physical
RAM (`"80%"`) or an absolute size (`"16G"`). When unset, the kernel's
default applies (50% of physical RAM per mount).

### Node labels and annotations

`LibvirtMachine.spec.nodeLabels` and `LibvirtMachine.spec.nodeAnnotations`
apply arbitrary labels and annotations to the Kubernetes `Node` object
that backs the VM, once kubelet has registered it with the cluster:

```yaml
spec:
  # ...existing fields...
  nodeLabels:
    node-role.kubernetes.io/app: ""
    k8s.ovn.org/egress-assignable: ""
    dynatrace: "true"
    aqua: "true"
  nodeAnnotations:
    example.com/owner: "platform-team"
```

Unlike the existing CAPI `Machine.spec.nodeLabels` field (and `kubelet
--node-labels` / `K0sWorkerConfig.spec.args` / ignition kubelet drop-ins),
this is **not subject to the NodeRestriction admission allow-list**.
NodeRestriction polices labels self-set by kubelet (`system:nodes:<name>`)
and CAPI mirrors the same allow-list on its own `nodeLabels` field. CAPLV
instead patches the `Node` from the controller's own identity after
kubelet registration, so arbitrary keys like `dynatrace` or
`k8s.ovn.org/egress-assignable` are accepted.

**Ownership.** CAPLV owns only the keys it has applied. It tracks them
via two annotations it writes onto the `Node` itself:

- `infrastructure.cluster.x-k8s.io/libvirt-managed-labels`
- `infrastructure.cluster.x-k8s.io/libvirt-managed-annotations`

On each reconcile, keys that have disappeared from spec are removed from
the `Node`; admin-applied keys CAPLV never set are left untouched. The
result of the patch is visible on
`status.conditions[?(@.type=="NodeLabelled")]`:

| Status | Reason          | Meaning                                                |
| ------ | --------------- | ------------------------------------------------------ |
| True   | `NodeLabelled`  | All declared keys are present on the Node              |
| False  | `NodeNotJoined` | Waiting for kubelet to register the Node (retried 15s) |

The condition is set only when at least one of `nodeLabels` /
`nodeAnnotations` is non-empty; clearing both removes the condition. Node
labelling does **not** block `status.ready` — infrastructure is reported
ready as soon as the domain is running, and the patch is applied as a
follow-on step.

When using 5-Spot, put the fields inside `infrastructureSpec.spec` and
they pass through to the underlying `LibvirtMachine`:

```yaml
apiVersion: 5spot.finos.org/v1alpha1
kind: ScheduledMachine
spec:
  # ...
  infrastructureSpec:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: LibvirtMachine
    spec:
      # ...VM config...
      nodeLabels:
        k8s.ovn.org/egress-assignable: ""
        dynatrace: "true"
```

### Fast first boot with `bootArtifacts` (direct kernel boot)

The default first-boot path delivers ignition via QEMU `fw_cfg`. That works
on every OS image but has a known kernel issue: `qemu_fw_cfg` does O(n²)
offset reads, which adds tens of seconds of wall-clock time for multi-MB
ignition payloads (see [kernel bug #218394](https://bugzilla.kernel.org/show_bug.cgi?id=218394)).
Setting `LibvirtCluster.spec.bootArtifacts` switches first-boot ignition
delivery to libvirt direct-kernel-boot plus a virtio-blk ignition disk:

- The controller fetches kernel + initramfs from a configured source.
- Blobs are content-addressed on the libvirt host under `<hostPath>/<sha256>/`
  and reused across machines and reconciles.
- The domain XML is rendered with `<kernel>` / `<initrd>` / `<cmdline>` plus
  a virtio-blk disk with `serial=ignition` carrying the rendered config.

Three transports are supported: **HTTPS**, **OCI** (single artifact with two
layers), and **S3** (any S3-compatible store).

```yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: LibvirtCluster
metadata:
  name: prod
spec:
  controlPlaneEndpoint:
    host: 10.0.0.10
    port: 6443
  bootArtifacts:
    hostPath: /var/lib/caplv/boot      # cache dir on each libvirt host
    kernelArgs: |-
      ignition.platform.id=metal
      ignition.config.url=oem:/dev/disk/by-id/virtio-ignition
      console=ttyS0
    source:
      type: HTTPS                       # one of: HTTPS, OCI, S3
      https:
        kernelURL:       https://artifacts.example.com/rhcos/vmlinuz
        initramfsURL:    https://artifacts.example.com/rhcos/initramfs.img
        kernelSHA256:    <hex>          # optional integrity check
        initramfsSHA256: <hex>
```

**OCI source.** Push a single `oras`-style artifact with the kernel and
initramfs as two layers, identified by the
`org.opencontainers.image.title` annotation:

```bash
oras push ghcr.io/example/boot:v1 \
    vmlinuz:application/octet-stream \
    initramfs.img:application/octet-stream
```

```yaml
source:
  type: OCI
  oci:
    reference: ghcr.io/example/boot:v1
    # Optional layer-title overrides (defaults shown):
    # kernelLayerTitle:    vmlinuz
    # initramfsLayerTitle: initramfs.img
    # plainHTTP:            false       # for in-cluster mirrors
    # insecureSkipTLSVerify: false      # dev/self-signed only
    credentialsSecretRef:                # optional for private registries
      name: ghcr-pull-secret
    kernelSHA256:    <hex>               # optional
    initramfsSHA256: <hex>
```

The credentials secret can be either a `kubernetes.io/dockerconfigjson`
Secret (preferred) or a Secret with plain `username` / `password` keys. It
is read from the LibvirtCluster's namespace unless
`credentialsSecretRef.namespace` is set.

**S3 source.** Works with AWS S3, MinIO, and Ceph RGW:

```yaml
source:
  type: S3
  s3:
    endpoint:     s3.amazonaws.com       # host[:port]
    region:       us-east-1              # required by AWS, optional elsewhere
    bucket:       my-boot-artifacts
    kernelKey:    rhcos/vmlinuz
    initramfsKey: rhcos/initramfs.img
    # usePathStyle:           true       # required by MinIO/Ceph
    # insecure:               false      # plain HTTP endpoint
    # insecureSkipTLSVerify:  false
    credentialsSecretRef:                # optional for private buckets
      name: s3-read-creds
    kernelSHA256:    <hex>               # optional
    initramfsSHA256: <hex>
```

The S3 credentials Secret recognizes either:
- `accessKeyID` / `secretAccessKey` / `sessionToken` (preferred), or
- `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN`.

Leaving `credentialsSecretRef` unset means anonymous reads.

**Transport gzip.** Artifacts that arrive gzip-wrapped (detected by the
`1f 8b` magic bytes — no naming convention required) are transparently
decompressed before the sha256 is computed. So a `.gz`-wrapped kernel in
Artifactory, an OCI layer pushed with `application/gzip`, and a raw
`vmlinuz` all produce the same digest and the same on-host cache path.
The `*SHA256` fields you configure describe the *decompressed* payload —
i.e. what the kernel actually boots.

**Caching and integrity.** Optional `*SHA256` fields are verified after
download (and decompression, if applicable). Bytes are cached in-memory
per controller process (keyed by the source spec digest), and on the
libvirt host they live under `<hostPath>/<sha256>/{vmlinuz,initramfs.img}`
so multiple machines reuse the same staged files.

### Cluster-wide base image via URL (`baseImage`)

Setting `LibvirtCluster.spec.baseImage` makes the controller responsible for
distributing the cluster's root-disk qcow2 to every libvirt host. Hosts no
longer need the qcow2 pre-staged via Ansible — they just need libvirt and a
storage pool. The controller fetches the qcow2 once into its local cache
and SCPs it onto each host the first time a machine targeting that host is
scheduled; subsequent machines on the same host reuse the staged volume.

```yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: LibvirtCluster
metadata:
  name: prod
spec:
  controlPlaneEndpoint:
    host: 10.0.0.10
    port: 6443
  baseImage:
    pool: default                       # libvirt storage pool on each host
    volumeName: rhcos-4.18.qcow2        # the libvirt volume name the controller registers
    source:
      type: HTTPS                        # one of HTTPS, OCI, S3
      https:
        url: https://artifactory.example.com/rhcos/rhcos-4.18.qcow2.gz
        sha256: <hex>                    # optional but strongly recommended for mutable URLs
        credentialsSecretRef:
          name: artifactory-creds        # optional
        # insecureSkipTLSVerify: false   # dev/self-signed only; prefer SSL_CERT_FILE (see below)
```

`LibvirtMachine.spec.rootDisk.baseImage` continues to refer to the staged
volume by name (here `rhcos-4.18.qcow2`), so existing machine manifests work
unchanged once you point them at the cluster-managed volume name.

**Transports.** `HTTPS` is the typical mirror case; `OCI` reads a
single-blob artifact (`oras push <ref> rhcos.qcow2:application/octet-stream`)
with optional `blobTitle` disambiguation; `S3` reads a single object from
any S3-compatible store. All three accept a `credentialsSecretRef` with the
same secret shapes used by `bootArtifacts`: `kubernetes.io/dockerconfigjson`
or `username`/`password` for OCI/HTTPS basic auth, and
`accessKeyID`/`secretAccessKey`/`sessionToken` (or the `AWS_*` env-var
spellings) for S3.

**Transparent gzip.** Same magic-byte sniff used by `bootArtifacts`:
`.qcow2.gz` mirrors are decompressed in-stream before the digest is
computed. The `sha256` field describes the *decompressed* qcow2.

**HTTPS trust (private CAs).** The `HTTPS` transport verifies the server
certificate against the controller container's trust store. When the mirror
is served by an internal Artifactory (or any endpoint) fronted by a private
or corporate CA, the fetch fails with `x509: certificate signed by unknown
authority`. Rather than rebuild the controller image, mount the CA bundle
and point Go's `SSL_CERT_FILE` at it — the controller's HTTPS client honors
it automatically:

```bash
# CA certs are not secret, so a ConfigMap is fine.
oc -n <controller-namespace> create configmap caplv-ca \
  --from-file=ca-bundle.crt=/path/to/ca-bundle.pem
```

Add the mount and env var to the manager Deployment (works with the shipped
`readOnlyRootFilesystem: true` security context, since the bundle is a
read-only volume rather than a write into `/etc/pki`):

```yaml
        env:
        - name: SSL_CERT_FILE
          value: /etc/pki/caplv/ca-bundle.crt
        volumeMounts:
        - name: caplv-ca
          mountPath: /etc/pki/caplv
          readOnly: true
      volumes:
      - name: caplv-ca
        configMap:
          name: caplv-ca
```

`SSL_CERT_FILE` *replaces* the system bundle rather than adding to it. If the
controller also fetches from public-CA endpoints, concatenate your CA with
the system bundle into the ConfigMap (`cat ca.pem /etc/pki/tls/certs/ca-bundle.crt`),
or use `SSL_CERT_DIR` to point at a directory of trusted certs instead. On
OpenShift you can also have the cluster populate the bundle for you by
creating an empty ConfigMap labeled
`config.openshift.io/inject-trusted-cabundle: "true"` and mounting it the
same way — useful when the CA is already in the cluster's proxy/additional
trust bundle.

As a last resort for development or self-signed endpoints, the `HTTPS` source
also accepts `insecureSkipTLSVerify: true` (matching the `OCI` and `S3`
transports), which disables certificate verification entirely. Prefer the
`SSL_CERT_FILE` approach above for anything production — it keeps verification
on.

**Cache.** The controller stores fetched payloads under
`--base-image-cache-dir` (default `/var/cache/caplv/baseimages`), mounted as
an `emptyDir` in the shipped manager Deployment. Files are content-addressed
by sha256; concurrent fetches for the same artifact are coalesced via
`singleflight`, and concurrent uploads of the same artifact to the same
host are coalesced per-`(host, sha256)`. A controller restart re-downloads
on first use after the restart.

**Status.** The first machine on a fresh host blocks while the qcow2 is
SCP'd over and registered with libvirt — for a ~1 GB image this is minutes,
not seconds. Watch
`LibvirtMachine.status.conditions[?(@.type=="BaseImageStaged")]`:

| Status | Reason             | Meaning                                                 |
| ------ | ------------------ | ------------------------------------------------------- |
| False  | `BaseImageStaging` | qcow2 is being uploaded to this host                    |
| True   | `BaseImageStaged`  | Volume present in `spec.baseImage.pool`, ready to clone |

Subsequent machines on the same host hit the volume-already-present fast
path and don't surface the condition at all.

### MachineHealthCheck (recommended)

When the designed deletion path is followed (5-Spot → CAPI → CAPLV),
CAPI drains pods and deletes the `Node` object from OpenShift before the
VM is destroyed. Everything cleans up automatically.

However, if a VM dies unexpectedly (host crash, OOM kill, admin
`virsh destroy`), the `Node` object lingers in `NotReady` state
indefinitely. A CAPI `MachineHealthCheck` detects this and triggers
remediation — deleting the orphaned Machine and its stale Node.

```yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineHealthCheck
metadata:
  name: caplv-worker-health
  namespace: default
spec:
  clusterName: my-ocp-cluster
  selector:
    matchLabels:
      node-role.kubernetes.io/worker: ""
  unhealthyConditions:
    - type: Ready
      status: "False"
      timeout: 5m           # node was Ready, now isn't — remediate after 5 min
    - type: Ready
      status: "Unknown"
      timeout: 5m           # node stopped reporting — remediate after 5 min
  nodeStartupTimeout: 10m   # time for a new VM to boot and join the cluster
```

**Tuning `nodeStartupTimeout`:** This is how long CAPI waits for a newly
provisioned VM to boot, run ignition, start kubelet, get its CSR approved,
and report `Ready`. Typical CAPLV startup is 2-4 minutes (VM provision
~30s, RHCOS boot ~60s, ignition + kubelet ~60s, CSR approval ~30s). The
10-minute default gives comfortable headroom. Increase to 15m+ if your
hosts are slow, base images are large, or the network path to the API
server has high latency.

This is the safety net for the known limitation that CAPLV does not
monitor VM health after provisioning. Without it, dead VMs leave orphaned
Node objects in the cluster.

## Key Design Decisions

- **virsh-over-SSH** — pure Go binary (`CGO_ENABLED=0`), distroless container.
  No `libvirt-dev`, no CGo. The `Client` interface allows swapping to native
  libvirt bindings later.
- **One VM per host** — enforced by admission webhook. Each LibvirtHost runs
  at most one CAPLV-managed VM. No contention, no resource accounting complexity.
- **Auto-sizing** — omit `vcpus` and `memoryMB` and CAPLV sizes the VM from
  the host's available capacity (total minus `reservedResources`). Hosts with
  varying specs just work.
- **Static IPs required** — every VM must declare its network identity upfront.
  No DHCP. This matches 5-Spot's model of predetermined machine configurations.
- **Control plane health gate** — LibvirtCluster verifies the worker-facing
  API endpoint is reachable (TCP dial, 60s recheck) before allowing machine
  provisioning. Since CAPLV runs on the same cluster, this catches the case
  where the internal API works but the external endpoint (that workers use
  to join) is down.
- **Parallel at scale** — 50 concurrent reconcilers by default. Each host runs
  one VM, so there are no contention issues across parallel provisions.
- **Ephemeral storage** — `ephemeralPool: true` creates a per-machine tmpfs
  mount and libvirt pool on demand, destroys both on cleanup. No host setup
  required, no persistent storage touched, RAM reclaimed when the VM goes away.
- **Metadata injection** — CAPLV injects the machine hostname and providerID
  into ignition configs before writing them to the host. The hostname is set
  via `/etc/hostname` (using the domain name `<namespace>-<cluster>-<machine>`)
  and the providerID is set via a kubelet systemd drop-in. For ignition
  (OpenShift/RHCOS), the config is delivered via QEMU `fw_cfg` — the standard
  libvirt method, no ISO needed. For cloud-init, a NoCloud ISO is created.
- **Built-in CSR approver** — CAPLV includes a CSR approver controller that
  auto-approves kubelet CSRs (both bootstrap and serving) for nodes whose
  hostname matches a CAPI Machine backed by a LibvirtMachine. No external
  machine-approver integration required.
- **Deterministic artifact naming** — domain names, disk volumes, and ISOs are
  named `<namespace>-<cluster>-<machine>`. Artifacts can be rediscovered after a
  controller crash without relying on status writes.
- **Immutable spec** — all spec fields are frozen after creation. To change VM
  configuration, delete and recreate. Enforced by admission webhook.
- **Finalizer safety** — the finalizer is never removed until cleanup is confirmed
  on the libvirt host. If the host is unreachable, the resource stays and a
  `CleanupStalled` condition is surfaced for operator intervention.

## Host Security Model

CAPLV connects to each libvirt host over SSH using a dedicated `caplv`
service account. The account is designed with minimal privileges:

| Operation | Privilege | Mechanism |
|-----------|-----------|-----------|
| `virsh define/start/destroy/undefine` | libvirt group | Group membership grants `qemu:///system` access — no sudo |
| `virsh vol-create-as/vol-upload/vol-delete` | libvirt group | Same — storage pool operations via libvirt API |
| `virsh nodeinfo/dominfo/version` | libvirt group | Read-only host and domain queries |
| `mkdir -p /run/caplv/*` | root | `sudo` — create tmpfs mount point |
| `mount -t tmpfs tmpfs /run/caplv/*` | root | `sudo` — mount ephemeral storage |
| `mount -t tmpfs -o size=* tmpfs /run/caplv/*` | root | `sudo` — mount with size cap (`ephemeralPoolSize`) |
| `umount /run/caplv/*` | root | `sudo` — unmount on cleanup |
| `rmdir /run/caplv/*` | root | `sudo` — remove mount point |
| `tee /run/caplv/*` | root | `sudo` — write ignition config |
| `rm -f /run/caplv/*` | root | `sudo` — delete ignition config |
| `cat > /tmp/caplv-*` | caplv | No sudo — `/tmp/` is world-writable (temp files for virsh define/vol-upload) |

All sudo rules are restricted to paths under `/run/caplv/`. The service
account cannot escalate beyond these specific commands.

Note that even in the default `/system` mode the VMs themselves do not run
as root: the system libvirt daemon launches QEMU as the unprivileged
`qemu` user. For hosts that must not run a root-owned VM management
daemon at all, see [session mode](#session-mode-unprivileged-libvirt),
which runs both the daemon and QEMU as the service account and confines
the remaining privilege to the setuid `qemu-bridge-helper` plus the same
`/run/caplv/` sudo rules.

## Host Storage Layout (with ephemeralPool: true)

```
/var/lib/libvirt/images/                    (persistent pool "default")
  └── rhcos.qcow2                     ← pre-staged by operator (read-only)

/run/caplv/ns-cluster-worker01/             (tmpfs, CAPLV creates/destroys)
  ├── ns-cluster-worker01-root.qcow2       ← CoW overlay (in RAM)
  └── ignition.json                        ← ignition config (delivered via fw_cfg)

/var/lib/libvirt/qemu/nvram/
  └── ns-cluster-worker01_VARS.fd           ← UEFI NVRAM (libvirt manages)
```

The tmpfs mount, libvirt pool, and ignition file are created when the VM
is provisioned and destroyed when the VM is deleted. No permanent host
storage impact. For cloud-init guests, a NoCloud ISO replaces the
ignition file.

## Failure Modes

### Host failures

| Scenario | What happens | Recovery |
|----------|-------------|----------|
| **Host dies or reboots** | VM vanishes. LibvirtHost controller marks host not-ready on the next health check (per-host `spec.healthCheckIntervalSeconds`, default 5 min). LibvirtMachine finalizer stalls cleanup (`CleanupStalled` condition). | Operator fixes host. Once reachable again, CAPLV retries cleanup automatically. If the VM no longer exists, cleanup completes (not-found errors are ignored). |
| **Host network becomes unreachable** | Same as host death from CAPLV's perspective. SSH connections fail, host goes not-ready on next health check. | Network restored, host re-verified on next check cycle. |
| **libvirtd crashes or restarts** | SSH works but virsh commands fail. Host Ping check (`virsh version`) catches this on the next cycle, marks host not-ready. Running VMs may or may not survive depending on libvirt config. | libvirtd restarts, next health check restores host to ready. |
| **Incumbent workload reclaims resources** | OOM killer terminates the VM process, or host admin runs `virsh destroy`. Domain goes to `shutoff` state. | CAPLV does not currently detect post-ready VM state changes (see Known Limitations). CAPI MachineHealthCheck can detect the node going `NotReady`. |

### VM failures

| Scenario | What happens | Recovery |
|----------|-------------|----------|
| **VM killed externally** (`virsh destroy`) | Domain state changes to `shutoff`. CAPLV does not detect this after initial `ready=true`. | CAPI MachineHealthCheck detects the node disappearing and can trigger remediation. |
| **VM crashes** | Same as killed — domain state changes but CAPLV's status still shows `ready=true`. | Same as above. |
| **VM boots but never joins cluster** | Not CAPLV's responsibility. The node will sit in `NotReady` state. | CAPI MachineHealthCheck handles this (`nodeStartupTimeout`). The bootstrap provider and machine approver must be functional. |

### Controller failures

| Scenario | What happens | Recovery |
|----------|-------------|----------|
| **Controller crashes mid-provisioning** | Partially created artifacts may exist on the host. | Automatic. Deterministic artifact naming allows the controller to discover existing artifacts and resume from where it left off. |
| **Controller crashes mid-deletion** | Finalizer remains on the resource, blocking garbage collection. | Automatic. On restart, the controller retries cleanup. Idempotent delete operations (not-found errors ignored) ensure safe retry. |
| **Leader election lost** | New leader picks up all pending reconciles. | Automatic. All operations are idempotent. |

### Control plane failures

Since CAPLV runs on the same cluster, control plane issues directly
affect CAPLV's ability to operate.

| Scenario | What happens | Recovery |
|----------|-------------|----------|
| **OpenShift API fully down** | CAPLV cannot reconcile at all — it runs on this cluster. No new machines are created. Already-running VMs continue to serve workloads but cannot be managed (no drain, no deletion). | API recovers, CAPLV resumes reconciliation automatically. |
| **API reachable internally but not from worker endpoint** | CAPLV can reconcile, but the LibvirtCluster TCP dial against the external endpoint fails. Status goes `ready=false` with condition `ControlPlaneUnreachable`. CAPI blocks new machine creation. Already-running VMs are unaffected. | External endpoint restored, LibvirtCluster rechecks every 60s. |
| **Cluster under resource pressure** | CAPLV pods may be evicted or throttled. Reconciliation slows. If 5-Spot activates a large schedule during a stressed period, hundreds of concurrent reconciles add API server and etcd load. | CAPLV pods restart via Deployment. Consider PodDisruptionBudget and resource requests to keep CAPLV running during pressure. |
| **etcd degradation** | Status patches and Machine updates slow down. All concurrent reconciles compete for etcd bandwidth. | etcd recovers, backlogged reconciles drain. |
| **CAPLV CSR approver not running** | VMs boot and request CSRs, but certificates are never approved. Nodes stay `NotReady`. CAPLV includes a built-in CSR approver that auto-approves CSRs from nodes matching a CAPI Machine backed by a LibvirtMachine. | Restart the CAPLV controller. Pending CSRs are approved and nodes join. |

### Storage failures (with ephemeralPool)

| Scenario | What happens | Recovery |
|----------|-------------|----------|
| **Host runs out of RAM** | tmpfs can't allocate pages for CoW writes. VM disk I/O fails, VM likely crashes. | Operator intervention. Reduce `reservedResources` or increase host RAM. The crashed VM can be deleted and recreated by 5-Spot. |
| **Base image pool goes offline** | New VMs can't be provisioned (backing image unreachable). Existing CoW VMs may also fail if the backing chain is broken. | Operator restores the persistent storage pool. |

### Health check behavior

Host health checks only run while machines actively reference the host.
When 5-Spot deletes all machines on a host, health checks stop — no SSH
traffic to idle hosts. When a new machine is created, the host controller
wakes up and resumes checking. The interval is per-host via
`spec.healthCheckIntervalSeconds` (default 300, minimum 30).

### Known Limitations

**No post-ready VM health monitoring.** Once a LibvirtMachine reaches
`ready=true`, CAPLV does not currently recheck the domain state. If the
VM dies after provisioning, `status.ready` remains `true` until the
resource is deleted. Deploy a `MachineHealthCheck` (see example above) to
detect node-level failures and trigger remediation. Without it, dead VMs
leave orphaned `Node` objects and `Terminating` pods in OpenShift.

## Prerequisites

- An OpenShift cluster (CAPLV runs on the same cluster that workers join)
- CAPI installed on the cluster
- libvirt hosts reachable over SSH from the cluster network
- RHCOS qcow2 base images pre-staged in libvirt storage pools on each host
- OVMF firmware packages installed on libvirt hosts (for UEFI boot)

## Development

```bash
# Build
make build

# Run tests
make test

# Generate CRDs and deepcopy
make manifests generate

# Build container image (pure Go, no CGo)
make podman-build IMG=ghcr.io/atgreen/caplv:latest

# Deploy to cluster
make deploy IMG=ghcr.io/atgreen/caplv:latest
```

## Project Structure

```
api/v1alpha1/          CRD type definitions (LibvirtHost, LibvirtCluster, LibvirtMachine)
internal/
  controller/          Reconcilers for all three CRDs + CSR approver (50 concurrent machine workers)
  ignition/            Ignition config manipulation (hostname and providerID injection)
  libvirt/             Client interface, virsh-over-SSH, domain XML generation
  iso/                 Cloud-init NoCloud ISO creation (pure Go, go-diskfs)
  ssh/                 SSH client with host key verification
  scope/               MachineScope — gathers reconciliation context, artifact naming
  webhook/             Admission webhook (immutable spec, required static IP, one VM per host)
config/                Kubernetes manifests (CRDs, RBAC, deployment, webhook)
```

## License

Copyright 2026 Anthony Green.

Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.