Nodes and scheduling

petri simulates Nodes the way kwok does. No container runs. A Pod placed on a Node becomes Running and Ready at once, with an IP from the Node's PodCIDR, and a deleted Pod goes away at once. The workload controllers then behave as they do on a cluster.

A Node can run any test's Pods, so only dedicated planes have Nodes. The shared plane refuses every Node operation. The samples on this page share a helper that starts a dedicated plane:

plane_test.go

package nodes_test

import (
	"context"
	"testing"
	"time"

	"github.com/yyewolf/petri"
)

// dedicatedPlane starts a control plane for one test and stops it when the
// test ends. Nodes only exist on dedicated planes: the shared one refuses them.
func dedicatedPlane(t *testing.T) *petri.ControlPlane {
	t.Helper()

	ctx, cancel := context.WithTimeout(t.Context(), time.Minute)
	defer cancel()

	cp, err := petri.New(ctx)
	if err != nil {
		t.Fatal(err)
	}
	t.Cleanup(func() {
		if stopErr := cp.Stop(); stopErr != nil {
			t.Error(stopErr)
		}
	})

	return cp
}

Adding Nodes

cp.AddNode(t, petri.NodeOptions{...}) creates a Ready Node and deletes it when the test ends. All options are optional:

Option	Default
Name	petri-node- and a random suffix; set it when the test needs to know the name
Labels, Annotations	yours, plus the well-known kubernetes.io/hostname, kubernetes.io/os (linux) and kubernetes.io/arch (amd64) labels
Taints	none
Capacity	32 CPUs, 256Gi of memory, 1Ti of ephemeral storage, 110 Pods
Unschedulable	false; true creates the Node cordoned

Each Node gets its own PodCIDR and InternalIP. It reports a kubelet version and conditions, as a real kubelet would.

kube-scheduler

Enable petri.SchedulerController to run upstream kube-scheduler with its default profile. It places Pods by requests, taints and tolerations, node selectors, affinities and topology spread:

scheduler_test.go

package nodes_test

import (
	"testing"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"github.com/yyewolf/petri"
)

func TestSchedulerKeepsWebOffTheGPUNode(t *testing.T) {
	t.Parallel()

	cp := dedicatedPlane(t)
	ns := cp.Namespace(t)
	cs := cp.Client()

	general := cp.AddNode(t, petri.NodeOptions{Name: "general"})
	cp.AddNode(t, petri.NodeOptions{
		Name:   "gpu",
		Labels: map[string]string{"pool": "gpu"},
		Taints: []corev1.Taint{{Key: "gpu", Effect: corev1.TaintEffectNoSchedule}},
	})
	cp.EnableControllers(t,
		petri.SchedulerController, petri.DeploymentController, petri.ReplicaSetController)

	labels := map[string]string{"app": "web"}
	deployments := cs.AppsV1().Deployments(ns)
	_, err := deployments.Create(t.Context(), &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{Name: "web"},
		Spec: appsv1.DeploymentSpec{
			Replicas: new(int32(2)),
			Selector: &metav1.LabelSelector{MatchLabels: labels},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec:       corev1.PodSpec{Containers: []corev1.Container{{Name: "web", Image: "nginx"}}},
			},
		},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatal(err)
	}

	// Scheduled, started by the simulated kubelet, counted by the controllers.
	petri.WaitFor(t, deployments, "web",
		petri.StatusCondition[*appsv1.Deployment]("Available", metav1.ConditionTrue))

	pods, err := cs.CoreV1().Pods(ns).List(t.Context(), metav1.ListOptions{LabelSelector: "app=web"})
	if err != nil {
		t.Fatal(err)
	}
	for _, pod := range pods.Items {
		if pod.Spec.NodeName != general.Name() {
			t.Errorf("pod %s runs on %s; it does not tolerate the gpu taint", pod.Name, pod.Spec.NodeName)
		}
	}
}

Without the scheduler, Pods stay Pending until the test places them. A plane uses either the scheduler or manual placement; see manual placement and Pod expectations.

Changing Nodes

AddNode returns a *petri.Node. Its methods change the Node the way an administrator or a failing kubelet would:

Method	Effect
Taint, Untaint	Add or remove taints.
Cordon, Uncordon	Mark the Node unschedulable, or not.
Drain	Cordon, then evict every Pod but DaemonSet Pods through the Eviction API, retrying while a PodDisruptionBudget refuses.
SetReady	Report the Node Ready or not, with the matching not-ready taint.
Delete	Delete the Node and force-delete the Pods bound to it, as the Pod garbage collector does on a cluster.

drain_test.go

package nodes_test

import (
	"testing"

	corev1 "k8s.io/api/core/v1"
	policyv1 "k8s.io/api/policy/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"

	"github.com/yyewolf/petri"
)

func TestBudgetBlocksEvictionUntilLifted(t *testing.T) {
	t.Parallel()

	cp := dedicatedPlane(t)
	ns := cp.Namespace(t)
	cs := cp.Client()
	node := cp.AddNode(t, petri.NodeOptions{})
	cp.EnableControllers(t, petri.DisruptionController)

	labels := map[string]string{"app": "db"}
	pod, err := cs.CoreV1().Pods(ns).Create(t.Context(), &corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{Name: "db", Labels: labels},
		Spec:       corev1.PodSpec{Containers: []corev1.Container{{Name: "db", Image: "postgres"}}},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatal(err)
	}
	cp.Bind(t, pod, node)

	pdbs := cs.PolicyV1().PodDisruptionBudgets(ns)
	_, err = pdbs.Create(t.Context(), &policyv1.PodDisruptionBudget{
		ObjectMeta: metav1.ObjectMeta{Name: "db"},
		Spec: policyv1.PodDisruptionBudgetSpec{
			MinAvailable: new(intstr.FromInt32(1)),
			Selector:     &metav1.LabelSelector{MatchLabels: labels},
		},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatal(err)
	}
	// The disruption controller has counted the running Pod.
	petri.WaitFor(t, pdbs, "db", func(p *policyv1.PodDisruptionBudget) (bool, error) {
		return p != nil && p.Status.ObservedGeneration == p.Generation && p.Status.CurrentHealthy == 1, nil
	})

	// One Pod, one required: the Eviction API refuses, as on a cluster.
	err = cp.EvictCtx(t.Context(), pod)
	if !apierrors.IsTooManyRequests(err) {
		t.Fatalf("eviction under budget: %v, want TooManyRequests", err)
	}

	// Lift the budget and drain: cordon, then evict everything but DaemonSet
	// Pods. Drain retries evictions a budget refuses until its timeout.
	err = pdbs.Delete(t.Context(), "db", metav1.DeleteOptions{})
	if err != nil {
		t.Fatal(err)
	}
	node.Drain(t)

	petri.WaitFor(t, cs.CoreV1().Pods(ns), "db", petri.Deleted[*corev1.Pod]())
}

Node failures

A Node that is not Ready behaves as if its kubelet were frozen. Pods bound to it stay Pending and Pods deleted from it stay Terminating until it is Ready again.

notready_test.go

package nodes_test

import (
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"github.com/yyewolf/petri"
)

func TestNotReadyNodeHoldsItsPods(t *testing.T) {
	t.Parallel()

	cp := dedicatedPlane(t)
	ns := cp.Namespace(t)
	pods := cp.Client().CoreV1().Pods(ns)
	node := cp.AddNode(t, petri.NodeOptions{})

	// The kubelet stops reporting: the Node is not Ready and gets the
	// not-ready taint. Nothing is evicted; petri runs no node lifecycle
	// controller.
	node.SetReady(t, false)

	pod, err := pods.Create(t.Context(), &corev1.Pod{
		ObjectMeta: metav1.ObjectMeta{Name: "app"},
		Spec:       corev1.PodSpec{Containers: []corev1.Container{{Name: "app", Image: "app"}}},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatal(err)
	}
	cp.Bind(t, pod, node)

	// To check that something does not happen, wait for it briefly and expect
	// the wait to time out.
	_, err = petri.WaitUntil(t.Context(), pods, "app", running, petri.WithTimeout(2*time.Second))
	if err == nil {
		t.Fatal("pod started on a NotReady node")
	}

	// Back to Ready: the kubelet catches up and starts the Pod.
	node.SetReady(t, true)
	petri.WaitFor(t, pods, "app", running)
}

func running(p *corev1.Pod) (bool, error) {
	return p != nil && p.Status.Phase == corev1.PodRunning, nil
}

petri runs no node lifecycle controller, so nothing reacts to a NotReady Node by itself. There is no taint-based eviction and no tolerationSeconds. The test decides what happens next, for example with Drain or Delete.

What is not simulated

No container runs, so there are no probes, images, volumes, resource usage or restarts, and readiness gates are ignored. A Pod stays Running until the test finishes it with CompletePod or FailPod, or deletes it. Nodes send no heartbeats and have no Leases.