Simulating Node failures

Code that manages workloads, or that sets PodDisruptionBudgets, has to cope with Nodes going away. On a dedicated plane with Nodes, a test can cause each failure itself, in the order it chooses.

What you can cause

Failure	How
The kubelet stops reporting	node.SetReady(t, false): the Node turns NotReady, gets the not-ready taint, and its Pods freeze.
An administrator drains the Node	node.Drain(t): cordon, then evict through the Eviction API, honouring budgets.
A Pod is evicted	cp.Evict(t, pod), refused while a budget forbids it.
The Node disappears	node.Delete(t): its Pods are force-deleted, as the Pod garbage collector does.
A Pod's process exits	cp.FailPod(t, pod, exitCode) or cp.CompletePod(t, pod).
A Pod stops being ready	cp.SetPodReady(t, pod, false).

A NotReady Node

notready_test.go

package nodes_test

import (
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"github.com/yyewolf/petri"
)

func TestNotReadyNodeHoldsItsPods(t *testing.T) {
	t.Parallel()

	cp := dedicatedPlane(t)
	ns := cp.Namespace(t)
	pods := cp.Client().CoreV1().Pods(ns)
	node := cp.AddNode(t, petri.NodeOptions{})

	// The kubelet stops reporting: the Node is not Ready and gets the
	// not-ready taint. Nothing is evicted; petri runs no node lifecycle
	// controller.
	node.SetReady(t, false)

	pod, err := pods.Create(t.Context(), &corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{Name: "app"},
		Spec:       corev1.PodSpec{Containers: []corev1.Container{{Name: "app", Image: "app"}}},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatal(err)
	}
	cp.Bind(t, pod, node)

	// To check that something does not happen, wait for it briefly and expect
	// the wait to time out.
	_, err = petri.WaitUntil(t.Context(), pods, "app", running, petri.WithTimeout(2*time.Second))
	if err == nil {
		t.Fatal("pod started on a NotReady node")
	}

	// Back to Ready: the kubelet catches up and starts the Pod.
	node.SetReady(t, true)
	petri.WaitFor(t, pods, "app", running)
}

func running(p *corev1.Pod) (bool, error) {
	return p != nil && p.Status.Phase == corev1.PodRunning, nil
}

To check that something does not happen, wait for it with a short timeout and expect the wait to fail. The test does this for the Pod that must not start.

Draining under a budget

drain_test.go

package nodes_test

import (
	"testing"

	corev1 "k8s.io/api/core/v1"
	policyv1 "k8s.io/api/policy/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"

	"github.com/yyewolf/petri"
)

func TestBudgetBlocksEvictionUntilLifted(t *testing.T) {
	t.Parallel()

	cp := dedicatedPlane(t)
	ns := cp.Namespace(t)
	cs := cp.Client()
	node := cp.AddNode(t, petri.NodeOptions{})
	cp.EnableControllers(t, petri.DisruptionController)

	labels := map[string]string{"app": "db"}
	pod, err := cs.CoreV1().Pods(ns).Create(t.Context(), &corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{Name: "db", Labels: labels},
		Spec:       corev1.PodSpec{Containers: []corev1.Container{{Name: "db", Image: "postgres"}}},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatal(err)
	}
	cp.Bind(t, pod, node)

	pdbs := cs.PolicyV1().PodDisruptionBudgets(ns)
	_, err = pdbs.Create(t.Context(), &policyv1.PodDisruptionBudget{
		ObjectMeta: metav1.ObjectMeta{Name: "db"},
		Spec: policyv1.PodDisruptionBudgetSpec{
			MinAvailable: new(intstr.FromInt32(1)),
			Selector:     &metav1.LabelSelector{MatchLabels: labels},
		},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatal(err)
	}
	// The disruption controller has counted the running Pod.
	petri.WaitFor(t, pdbs, "db", func(p *policyv1.PodDisruptionBudget) (bool, error) {
		return p != nil && p.Status.ObservedGeneration == p.Generation && p.Status.CurrentHealthy == 1, nil
	})

	// One Pod, one required: the Eviction API refuses, as on a cluster.
	err = cp.EvictCtx(t.Context(), pod)
	if !apierrors.IsTooManyRequests(err) {
		t.Fatalf("eviction under budget: %v, want TooManyRequests", err)
	}

	// Lift the budget and drain: cordon, then evict everything but DaemonSet
	// Pods. Drain retries evictions a budget refuses until its timeout.
	err = pdbs.Delete(t.Context(), "db", metav1.DeleteOptions{})
	if err != nil {
		t.Fatal(err)
	}
	node.Drain(t)

	petri.WaitFor(t, cs.CoreV1().Pods(ns), "db", petri.Deleted[*corev1.Pod]())
}

What petri leaves to you

petri runs no node lifecycle controller. A NotReady Node is never tainted NoExecute, and its Pods are never evicted by themselves. If a test needs that, it evicts or deletes the Pods, or deletes the Node, when it chooses. Because nothing happens on a timer, failure tests are deterministic.