Kueue Pending Jobs: A Headlamp Checklist for Admission Failures

A Kubernetes Job asks for three Pods at one CPU each, the ClusterQueue behind it has a nominal quota of two CPUs, and the Job sits suspended with no Pod to inspect. That is the worked example in the Kubernetes blog’s guide to tracing Kueue admission with Headlamp, and its point is that the first question isn’t why a Pod is pending but whether Kueue has admitted the Workload. For platform engineers running clusters managed by Kueue, a Job stuck like this can be completely normal: an admission-capacity problem, not evidence that kube-scheduler failed.

The Framework

Kueue and the scheduler answer different questions. Kueue decides whether a Workload should start now, given queue policy, quota, ResourceFlavors, admission checks and other waiting Workloads. The scheduler decides which nodes can run the Pods only after the Workload is admitted and the Job resumes. The new Kueue plugin for Headlamp, a 0.1.0-alpha release, puts the admission side in one UI, so the investigation follows links instead of commands and large YAML objects.

Find the Workload

Kueue creates a Workload to represent each Job’s resource request and admission state. In Headlamp, open Kueue, then Workloads, select the namespace and find the Workload, whose name normally contains the Job name. The list already shows the LocalQueue, priority, and admitted and completion state. If there is no Workload, check the namespace, whether the Job was submitted to a LocalQueue, and whether the account can list Workloads.

Read the Conditions Before the YAML

The two conditions that matter are QuotaReserved and Admitted. Admitted=True means Kueue’s part is done. QuotaReserved=False with a reason such as Pending means quota wasn’t reserved, and the condition message says why, for example that a ResourceFlavor couldn’t be assigned because of insufficient CPU quota. An Inadmissible status means the Workload can’t fit the current queue configuration.

Compare the Total Request With Its Assignment

Kueue evaluates PodSets together. A Job with parallelism: 3 and a one-CPU request per Pod asks for three CPUs, so looking at one Pod template underestimates the Workload. In the example, the request is three Pods, three CPUs and 600 MiB of memory.

Follow the Queue Chain

From the Workload, follow the LocalQueue link to confirm the Job entered the expected queue and whether it is stopped. Then follow its ClusterQueue to compare the request with nominal quota, reservations, borrowing and lending limits, and preemption and fair-sharing settings. When the message mentions flavor assignment, check the ResourceFlavor; when the queue shares quota through a Cohort, check whether it can borrow enough.

Switch Views Once Admitted

If the Workload is admitted but Pods are still Pending, leave the Kueue section and inspect Job and Pod events for scheduler causes: node capacity, affinity, taints, topology constraints, volume binding or image pulls.

Where It Doesn’t Apply

The alpha plugin is read-only for Kueue v1beta2 resources. It doesn’t create, edit, stop or resume anything, and owner references aren’t yet clickable. The guide is clear that it doesn’t replace kubectl or kueuectl for scripts, exact output or changes. Teams still on the archived Kubernetes Dashboard need Headlamp first, as Stack Archive covered in Kubernetes Dashboard Is Archived — Headlamp Is the Official Successor.

Source: Kubernetes Blog

Content created with AI assistance and reviewed for accuracy.