Skip to main content

Control planes

A petri control plane is upstream kube-apiserver, serving on a loopback port from inside your test process. It stores data in kine's in-memory stand-in for etcd and generates its certificates and keys at startup.

There are two kinds: the shared plane and dedicated planes.

The shared control plane​

petri.Start(t) returns one control plane for the whole test binary. The first call starts it, which takes about two seconds, and later calls return the same plane. It is never stopped; it goes away when the process exits.

shared_test.go
package controlplanes_test

import (
	"testing"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"github.com/yyewolf/petri"
)

// Both tests run in parallel against the one control plane petri.Start shares
// with the whole test binary. Each works in its own namespace, so neither sees
// the other's objects.

func TestTeamA(t *testing.T) {
	t.Parallel()
	seesOnlyItsOwnConfigMap(t, "team-a")
}

func TestTeamB(t *testing.T) {
	t.Parallel()
	seesOnlyItsOwnConfigMap(t, "team-b")
}

func seesOnlyItsOwnConfigMap(t *testing.T, name string) {
	t.Helper()

	cp := petri.Start(t)
	configMaps := cp.Client().CoreV1().ConfigMaps(cp.Namespace(t))

	_, err := configMaps.Create(t.Context(), &corev1.ConfigMap{
		ObjectMeta: metav1.ObjectMeta{Name: name, Labels: map[string]string{"example": "shared"}},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatal(err)
	}

	list, err := configMaps.List(t.Context(), metav1.ListOptions{LabelSelector: "example=shared"})
	if err != nil {
		t.Fatal(err)
	}
	if len(list.Items) != 1 || list.Items[0].Name != name {
		t.Fatalf("listed %d ConfigMaps, want only %s", len(list.Items), name)
	}
}

Each test works in its own namespace (see namespaces and clients). Cluster-wide changes only ever add to the plane: a CRD a test installs or a controller it enables stays for every later test. Installing or enabling the same thing twice is safe. Write tests that still pass when other tests share the plane.

If startup fails, Start fails the test and prints the control plane's logs. Every later Start call in the binary reports the same failure.

Dedicated control planes​

petri.New(ctx) starts a plane that belongs to the caller. Use one when a test needs a cluster to itself, for example for:

  • Nodes and the scheduler, which only dedicated planes have;
  • envtest's behaviour, with petri.WithoutDefaultControllers();
  • cluster-wide state the test must own, such as CRD conversion webhooks.
dedicated_test.go
package controlplanes_test

import (
	"context"
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"github.com/yyewolf/petri"
)

func TestEnvtestSemantics(t *testing.T) {
	t.Parallel()

	// A dedicated plane belongs to this test alone. The context bounds startup
	// only; the plane runs until Stop.
	ctx, cancel := context.WithTimeout(t.Context(), time.Minute)
	defer cancel()

	cp, err := petri.New(ctx, petri.WithoutDefaultControllers())
	if err != nil {
		t.Fatal(err)
	}
	t.Cleanup(func() {
		if stopErr := cp.Stop(); stopErr != nil {
			t.Error(stopErr)
		}
	})

	ns := cp.Namespace(t)
	namespaces := cp.Client().CoreV1().Namespaces()
	err = namespaces.Delete(t.Context(), ns, metav1.DeleteOptions{})
	if err != nil {
		t.Fatal(err)
	}

	// Without the namespace controller, nothing empties and removes a deleted
	// namespace: it stays Terminating, as it would with envtest.
	petri.WaitFor(t, namespaces, ns, func(n *corev1.Namespace) (bool, error) {
		return n != nil && n.Status.Phase == corev1.NamespaceTerminating, nil
	})
}

The context only bounds startup. Stop shuts the plane down and returns once every goroutine it started has exited. Calling it again returns the same result. Parallel tests can each run their own dedicated plane, and the planes do not see each other.

What runs by default​

Every plane runs the namespace controller and the garbage collector. Start and New return once both are ready. Other controllers stay off until a test enables them; see built-in controllers.

petri.WithoutDefaultControllers() starts a dedicated plane without those two, for tests written against envtest. Deleted namespaces then stay Terminating, and dependents outlive their owner. The shared plane always runs them.

The API server runs upstream admission, except for the ServiceAccount admission plugin, which is off as in envtest. A Pod created in a new namespace therefore does not wait for the default ServiceAccount. If your code needs that account, enable the ServiceAccount controller.

Logs​

petri captures the control plane's logs and prints them when startup fails. PETRI_VERBOSE=1 also prints them while tests run; see debugging.