Webhooks

The API server calls webhooks over HTTPS. petri generates a webhook CA for each control plane, and gives you what a local webhook server needs to work with it: a serving certificate the API server trusts, and installers that point your webhook configurations at your server.

Admission webhooks

admission_test.go

package webhooks_test

import (
	"context"
	"embed"
	"encoding/json"
	"net"
	"net/http"
	"path/filepath"
	"strings"
	"testing"
	"time"

	admissionv1 "k8s.io/api/admission/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

	"github.com/yyewolf/petri"
)

//go:embed testdata/manifests.yaml
var manifests embed.FS

const denied = "ConfigMaps need an owner label"

func TestConfigMapsNeedAnOwner(t *testing.T) {
	t.Parallel()

	cp := petri.Start(t)
	ns := cp.Namespace(t)
	port := serve(t, cp, http.HandlerFunc(requireOwner))

	// Points the manifest's webhook at 127.0.0.1:port with the plane's CA, and
	// scopes it to this test's namespace. Removed when the test ends.
	cp.InstallAdmissionWebhooks(t, petri.AdmissionWebhookOptions{
		Host: "127.0.0.1", Port: port, Namespace: ns,
	}, manifests)

	configMaps := cp.Client().CoreV1().ConfigMaps(ns)

	// The API server picks up a new webhook configuration within moments, so
	// retry until a ConfigMap without an owner is refused by the webhook.
	petri.Retry(t, func(ctx context.Context) (bool, error) {
		_, err := configMaps.Create(ctx, &corev1.ConfigMap{
			ObjectMeta: metav1.ObjectMeta{GenerateName: "orphan-"},
		}, metav1.CreateOptions{})

		return err != nil && strings.Contains(err.Error(), denied), nil
	})

	_, err := configMaps.Create(t.Context(), &corev1.ConfigMap{
		ObjectMeta: metav1.ObjectMeta{Name: "owned", Labels: map[string]string{"owner": "team-a"}},
	}, metav1.CreateOptions{})
	if err != nil {
		t.Fatalf("a ConfigMap with an owner was refused: %v", err)
	}
}

// requireOwner is the webhook: it refuses ConfigMaps without an owner label.
func requireOwner(w http.ResponseWriter, r *http.Request) {
	review := &admissionv1.AdmissionReview{}
	if err := json.NewDecoder(r.Body).Decode(review); err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)

		return
	}

	cm := &corev1.ConfigMap{}
	if err := json.Unmarshal(review.Request.Object.Raw, cm); err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)

		return
	}

	response := &admissionv1.AdmissionResponse{UID: review.Request.UID, Allowed: cm.Labels["owner"] != ""}
	if !response.Allowed {
		response.Result = &metav1.Status{Message: denied}
	}

	review.Response = response
	review.Request = nil
	w.Header().Set("Content-Type", "application/json")
	_ = json.NewEncoder(w).Encode(review)
}

// serve runs handler over TLS on a free port with the plane's serving
// certificate, the pair a controller-runtime webhook server would load from
// its CertDir.
func serve(t *testing.T, cp *petri.ControlPlane, handler http.Handler) int {
	t.Helper()

	ln, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		t.Fatal(err)
	}

	certDir := cp.ServingCertDir(t)
	server := &http.Server{Handler: handler, ReadHeaderTimeout: 10 * time.Second}
	go func() {
		_ = server.ServeTLS(ln, filepath.Join(certDir, "tls.crt"), filepath.Join(certDir, "tls.key"))
	}()
	t.Cleanup(func() { _ = server.Close() })

	return ln.Addr().(*net.TCPAddr).Port
}

testdata/manifests.yaml

# As a kubebuilder project generates it: the webhook points at an in-cluster
# Service, which InstallAdmissionWebhooks rewrites to the test's server.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: require-owner
webhooks:
  - name: require-owner.example.com
    admissionReviewVersions: ["v1"]
    sideEffects: None
    clientConfig:
      service:
        name: webhook-service
        namespace: system
        path: /validate-configmap
    rules:
      - operations: ["CREATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["configmaps"]

• cp.ServingCertDir(t) writes tls.crt and tls.key for 127.0.0.1 and localhost, signed by the plane's CA. A controller-runtime webhook server can load them as they are through its CertDir option. The directory is removed with the test. It is the only thing petri writes to disk.
• cp.InstallAdmissionWebhooks(t, options, sources...) installs the mutating and validating webhook configurations it finds in its sources, which work as in InstallCRDs. It points each service-based webhook at options.Host and options.Port, keeps its path, and sets the plane's CA as its caBundle.
• Each configuration gets a unique name and is deleted when the test ends, so parallel tests can install the same manifests.
• options.Namespace limits every webhook to one namespace, on top of any selector the manifest already has. Leave it empty only for webhooks that must see the whole cluster: on the shared plane, a webhook without a namespace fires for every test's objects.
• cp.WebhookCABundle() returns the CA, for configurations that set their own URL.

The API server finds new webhook configurations through a watch, so the first requests after installing may miss the webhook. Retry the first request that depends on it, as the example does.

Conversion webhooks

First install the CRDs, with the conversion.webhook section your project generates. Then call cp.InstallConversionWebhooks with the CRD names and a petri.ConversionWebhookOptions{Host, Port}. It points each named CRD's conversion webhook at your server, keeps the path, and sets the plane's CA as the caBundle. It leaves alone CRDs whose webhook sets its own URL.

A CRD applies to the whole plane, so the shared plane refuses conversion webhooks. Use a dedicated plane.

Without a testing.T

InstallAdmissionWebhooksCtx returns an error and a cleanup function that removes the configurations. InstallConversionWebhooksCtx returns an error. WriteServingCert(dir) writes the certificate and key to a directory you manage.