Fast parallel suites

A petri suite is fast when its tests run in parallel on the shared control plane. The plane starts once per test binary. After that, a test costs only its API calls.

The rules

1. Use the shared plane: call petri.Start(t) in every test. Start a dedicated plane only for what the shared one cannot do; see shared or dedicated.
2. Call t.Parallel() first, in every test and subtest.
3. Give each test and subtest its own namespace with cp.Namespace(t), and keep its objects inside it.
4. Give each subtest its own deadline (see below).
5. Wait with WaitFor and Retry, never with sleeps.
6. Install CRDs, controllers and webhooks in each test that needs them. Several tests can install the same ones safely. Do not rely on TestMain, or on another test having run first.

table_test.go

package parallel_test

import (
	"context"
	"testing"
	"time"

	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"github.com/yyewolf/petri"
)

func TestConfigMapNames(t *testing.T) {
	t.Parallel()

	cp := petri.Start(t)

	for _, tt := range []struct {
		name  string
		valid bool
	}{
		{name: "settings", valid: true},
		{name: "Settings", valid: false},
		{name: "settings.v2", valid: true},
	} {
		t.Run(tt.name, func(t *testing.T) {
			t.Parallel()

			// Each subtest takes its own deadline. A parallel subtest only starts
			// once its parent returns and a -parallel slot frees up, so a deadline
			// set in the parent may already be spent by then.
			ctx, cancel := context.WithTimeout(t.Context(), time.Minute)
			defer cancel()

			// Each subtest has its own namespace too.
			configMaps := cp.Client().CoreV1().ConfigMaps(cp.Namespace(t))
			_, err := configMaps.Create(ctx, &corev1.ConfigMap{
				ObjectMeta: metav1.ObjectMeta{Name: tt.name},
			}, metav1.CreateOptions{})

			if tt.valid && err != nil {
				t.Fatalf("valid name refused: %v", err)
			}
			if !tt.valid && !apierrors.IsInvalid(err) {
				t.Fatalf("invalid name: got %v, want Invalid", err)
			}
		})
	}
}

Deadlines and parallel subtests

A parallel subtest does not start when t.Run is called. It first waits for its parent's function to return, then for a free -parallel slot (by default, one per CPU). In a busy package that can take a while. A deadline the parent set before the loop keeps counting during the wait. The subtest can then start with its context already expired, and fail at once with context deadline exceeded even though nothing was slow.

Create deadlines inside the subtest, from its own t.Context(), as the example does. petri's helpers that take a t already do this.

Cluster-wide state

On the shared plane, everything that is not namespaced is shared by every test in the binary: CRDs, ClusterRoles, enabled controllers, and webhooks not limited to a namespace. Write tests that work whatever other tests have created:

• name cluster-scoped objects after the test, or use GenerateName;
• limit webhooks to the test's namespace with AdmissionWebhookOptions.Namespace;
• assert on your own objects, never on counts across the cluster.

Packages

Each package is its own test binary with its own shared plane, and go test ./... runs packages in parallel. Many small packages that each start a plane use more memory than a few larger ones; see running in CI.