Skip to main content

Debugging

A wait timed out​

A wait that times out reports the last state it saw:

petri: wait for "web": timed out; last observed state: ...

Start by reading that state. Common causes:

  • A controller is not enabled. Enabling one never enables another: a Deployment only gets Pods with both DeploymentController and ReplicaSetController.
  • There are no Nodes, so Pods stay Pending. This is the case on the shared plane, and on a dedicated plane until the test calls AddNode; see built-in controllers.
  • Nothing places the Pods. With Nodes but without SchedulerController, Pods wait for the test to bind them; see manual placement.
  • The condition checks something that never changes, such as a status field set only by a controller the test did not enable.

Startup failed​

When a control plane fails to start, the error includes the last lines the API server and kine logged:

petri: start control plane: ...
--- recent control plane logs ---
...

On the shared plane, every test in the binary reports the same failure. The first one is enough to read.

Seeing the control plane's logs​

kube-apiserver and kine log through klog and logrus. petri redirects both, for the whole process, into a buffer, so if your own code logs through klog or logrus, its output is captured too. Set PETRI_VERBOSE=1 to also print the logs to stderr while the tests run:

PETRI_VERBOSE=1 go test -run TestReconcile -v ./controllers/

The logs are long, so run one test at a time.

Looking at the cluster​

A test can print anything it can read. Before failing, list the objects in its namespace and log them with t.Log. Events often explain the most quickly, because controllers and the scheduler record why they did not act. cp.Client().CoreV1().Events(ns).List(...) returns them.

A test passes alone but fails in the suite​

The cause is usually state shared on the shared plane: a cluster-scoped object another test also creates, a webhook not limited to a namespace, or an assertion that counts objects across namespaces. See fast parallel suites. If the failure is context deadline exceeded at the start of a subtest, see deadlines and parallel subtests.