Blog: Inside Kueue : How Kubernetes decide what runs next

[ekam-walia]

Fixes Issue

NONE

Changes proposed

Added a Blog

Screenshots

NONE

Note to reviewers

This PR covers the blog Inside Kueue: How Kubernetes decide what runs next

[ekam-walia]

cc @saiyam1814

[saiyam1814]

Hey @ekam-walia, thanks for sending this in. I read the whole thing properly and also got a preview build up so I could see it rendered rather than just reading the diff.

Quick note on that: Cloudflare doesn't build previews for PRs that come from forks, so I pushed your commits to a branch on our repo to get one. You can see the post live here:

https://blog-inside-kueue.website-dab.pages.dev/inside-kueue-how-kubernetes-decides-what-runs-next

Worth opening that yourself before you start editing, because a few things read fine in the markdown but fall apart on the page.

The good part first, and I mean it. Your instinct for explaining things is the strongest thing about this post. The "four team members, one still waiting for a computer" framing of gang scheduling is the clearest version of that idea I have read anywhere, and Demo 1 has a real story to it: job A takes the room, job B waits, A leaves, B gets in. That is exactly the shape a good hands-on post has. The microservices vs batch comparison table is useful too.

So this is worth finishing. But I can't merge it as it stands, and I would rather be straight with you about why than send it back with vague notes.

What's blocking merge

minimumReservedResources is not a real Kueue field. Strategy 2 is built entirely on it, and it doesn't exist in the API. A ClusterQueue resource entry takes nominalQuota, borrowingLimit, and lendingLimit, and that's it (see the ClusterQueue concepts page). The API server would reject that YAML. If part of this came out of an AI draft, no judgment at all, that is a fine way to get a first draft, but everything has to get applied to a real cluster before it ships. The genuine answers to "job B is starving" are workload priorities with preemption, or cohorts with borrowingLimit so one queue can lend spare capacity without giving up its floor. Either one makes a better Strategy 2 than what's there now.

Strategy 1 would break admission rather than prevent deadlock. Both demo jobs request CPU and memory. A ClusterQueue whose coveredResources only lists cpu can't admit a workload that also asks for memory, it gets marked inadmissible. So splitting into a cpu-only queue and a memory-only queue doesn't route jobs by their dominant resource, it just stops both jobs from ever being admitted. If the point is isolation between job types, that's separate ClusterQueues each covering both resources, optionally in a cohort.

The version is about two years stale. You're pinning v0.6.1 and current is v0.19.2. Please re-run on a current release and update the URL in both places it appears (install step and the Resources list at the end).

The command output looks reconstructed rather than pasted. A few tells: kubectl get workloads shows the generated workload name, so it would be something like job-job-a-big-2f4c1, not job-a-big. The QUEUE column shows the LocalQueue the job went through, which in your setup is default, not demo-queue. And kubectl describe workload job-b-small wouldn't resolve, since that's the Job's name and not the workload's. Same thing in Demo 2. "Real commands, real output" is honestly the main thing we ask for (it's the first bullet under "What we look for" in WRITING-A-BLOG.md), and it's also the thing readers can smell instantly. Please paste what your terminal actually printed, hashes and odd column widths and all.

Demo 2 isn't really a deadlock. Job B waiting for job A to finish its sleep 600 is queuing, or starvation if you want to be dramatic about it. It resolves on its own, which the post half admits when it says "when Job A finishes, resources are released". The actual deadlock is the one you describe correctly up in the gang scheduling section: a distributed job gets 3 of its 4 pods placed, those 3 hold resources, and nothing progresses. That's the thing worth demoing. Either rename Demo 2 to what it is, or build the real one with two gang jobs competing.

Render issues

All of these are visible on the preview link:

• Eight <h1>s on the page. Body headings need to start at ##, since the frontmatter title is already the H1. Right now seven of your section headings compete with the title and clutter the table of contents.
• Prose stuck inside a code block. Under Strategy 2 the "How it works" explanation renders as YAML, because the fence never gets closed before it. Close the fence after the last YAML line.
• A stray ** renders literally in "Strategy 2: Use minimumReservedResources**".
• The arrow flowcharts collapse into run-on paragraphs. Markdown eats single newlines, so "Pod lands → Scheduler checks if it fits on any node." and the two lines under it become one blob. Same problem hits the "What Just Happened" recap, the Pod 1 through Pod 4 dependency chain, the car analogy, and the three links at the end. Make those bullet lists, or wrap them in a ```text block if you want the arrow-diagram look.
• Unfenced output in Demo 1 Step 8: the Conditions: block runs together as a paragraph while every other output block is fenced.
• Headings used as emphasis. "##### Kueue checks: Can I fit all 4?" and "##### Queue is stuck" are sentence fragments sitting in the heading hierarchy and the TOC. Bold text does that job.
• Missing space: "Which job gets them?Jobs from all namespaces compete here" (and the quote before it never closes).

Style, small stuff

• No em dashes on the blog. Two here: the seoDescription ("workloads—with a hands on demo") and "someone deploys a Batch Job—maybe it's...". A comma or a period does the job. While you're in that line, "hands on" wants a hyphen: hands-on.
• Tags should be lowercase, hyphenated, 2 to 5 of them, no slashes. "AI/ML" breaks the tag page URL because of the slash. I'd go with ["kubernetes", "kueue", "scheduling", "batch-workloads"].
• Images are heavy: cover is 1.2MB, and the two diagrams are 1.6MB and 1.8MB. Please get them under ~300KB each, WebP if you can.
• Typos worth a pass: "schedular" should be scheduler, "Core principal" should be principle, "it's fix" should be its fix, and "until desired resources don't get vacant" reads oddly (try "until resources free up"). The heading "Now Lets jump on to how Fix this DEADLOCK" needs a rewrite, and "Now let's first understand what Batch Jobs are and the concept of Gang Scheduling, also how they create a DEADLOCK Issue through an example" is a whole sentence doing a heading's job. Curly quotes have crept in from a word processor in a few spots, straight quotes please.
• The post opens directly on a heading and ends on a bare link dump. Two or three sentences up top saying who this is for, and a short wrap-up at the end, would round it off.

One process thing

None of the four commits have a DCO sign-off, which we do require. Easiest fix is to squash them into one:

git reset --soft origin/main
git commit -s -m "Add blog post: Inside Kueue: How Kubernetes Decides What Runs Next"
git push --force

That also cleans up the commit messages, which are currently "..", "...", and "Is this reqd".

---

I know that's a long list. It isn't a rejection, and none of it is about your voice, which is fine. It comes down to: fix the two strategies so they use mechanisms Kueue actually has, re-run the demos on a current version and paste what you really saw, and clean up the heading levels. Do that and this is a solid post that I'll be glad to publish with your byline on it.

Ping me here when you've pushed and I'll refresh the preview and take another look. If you get stuck on the priorities/preemption or cohort setup, ask, that part of Kueue is genuinely fiddly.

[ekam-walia]

Hi @saiyam1814, I have made all the suggested changes, I have added a new block The Real Deadlock Demo (Gang Scheduling) in which i have explained why and how deadlock occurs, thru examples and have also added yaml files that can cause Deadlock Scenario and how Kueue resolves it

[saiyam1814]

Thanks @ekam-walia, this is a big step up and I can see the work that went into it. Most of what I flagged is genuinely fixed, so let me confirm that first and then give you the short list that's still open.

Fixed and verified:

• minimumReservedResources is gone, and so is the Strategy 1 section that would have broken admission. Every coveredResources block now lists both cpu and memory.
• You're on v0.19.2 instead of v0.6.1.
• The workload names in the output are real now (job-job-a-big-cb1a1, job-job-b-small-c8c54), the QUEUE column shows the LocalQueue with RESERVED IN alongside it, and describe targets the workload rather than the job. That reads like an actual terminal, which is exactly what I was after. The couldn't assign flavors to pod set main: insufficient unused quota for cpu in flavor default, 3 more needed message is a nice touch.
• All eight H1s are gone, the fences are balanced, the stray ** is gone.
• Tags are down to four, lowercase and hyphenated.
• Images are WebP and all under 105KB, down from 1.6MB and 1.8MB.
• schedular, principal, "it's fix" and "don't get vacant" are all cleaned up.
• Commits squashed to one with a proper DCO sign-off and a real message.

That's most of the list. Here's what's left, and unfortunately the first two are the kind that have to be right before it goes out.

The install command is broken

Line 219 currently reads:

kubectl apply -f kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.2/manifests.yaml

The kubectl apply -f prefix got duplicated, so the shell would treat the literal string kubectl as a filename. This is the very first command anyone runs, so the tutorial stops there. The version in your Resources section at the bottom is correct, just copy that one up:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.2/manifests.yaml

The closing takeaway isn't right

The wrap-up ends on this:

The best part? One line fixes it: suspend: true. That's it.

That's not what does the work, and I'd hate for the last thing a reader takes away to be the one wrong thing. What enrolls a Job with Kueue is the kueue.x-k8s.io/queue-name label. Kueue's webhook then sets spec.suspend itself when it admits or holds the job. You can confirm it from Kueue's own sample job, which has the queue-name label and no suspend field anywhere:

https://github.com/kubernetes-sigs/kueue/blob/main/site/static/examples/jobs/sample-job.yaml

This also affects your "Before Kueue" and "After Kueue" pair, because both of those YAMLs already carry the queue-name label. As written, Kueue would manage both of them identically, so the only difference you're showing isn't the thing doing the work. Rewriting that section around the label (and mentioning that Kueue handles suspension for you) would land the point properly, and it's honestly a better ending: you add one label and Kueue takes over admission.

Worth a look while you're in there: job-b-needs-6-gets-3 has completions: 1 with parallelism: 2, which is an odd combination. I think you want both set to 2, the way your "Before Kueue" example has it.

The deadlock section is a good explanation but still isn't a demo

The new "The Real Deadlock Demo (Gang Scheduling)" block explains partial admission much better than the old Demo 2 did, and the reasoning is correct. But everything in it is narrative and YAML, with no captured output showing the deadlock actually happening. Right now it's labelled a demo and reads as a description.

Two ways to fix it, either is fine. Rename it to something like "Understanding the Deadlock" so it doesn't promise a run. Or, better, actually run those two jobs and paste what kubectl get workloads and kubectl get pods show while it's stuck. Given you clearly have a working cluster (your Demo output proves it), the second option would make this the strongest section in the post.

Smaller things

• Both em dashes are still there. Line 11 ("works great for microservices—until someone") and line 43 ("someone deploys a Batch Job—maybe it's"). Line 43 is the exact one I flagged last time. We don't use em dashes on the blog, a comma or a full stop is fine.
• Arrow lines still collapse into paragraphs. You fixed several by wrapping them in ```text blocks, which is exactly right, but these are still bare and will render as run-on paragraphs: the Pod 1 through Pod 4 dependency chain, the "Result / Status / Wasted resources" block, the "What happens?" recap, the "Kubernetes scheduler doesn't understand gang scheduling" list, the "What Just Happened" recap after Step 9, and the three links under Resources. Same fix as the ones you already did.
• them?Jobs in the ClusterQueue explainer still has no space, and the quote before it doesn't close.
• Mixed API versions. ClusterQueue appears as v1beta1 in the demo and v1beta2 in the deadlock section. Both are served in v0.19.2 so nothing breaks, but pick one, and I'd use v1beta2 since that's the storage version now. Your LocalQueue is still v1beta1 too.
• Heading levels jump around. You go from ## straight to #### and #####, and in one spot ######. Since the levels feed the table of contents, keeping to ## and ### will make the sidebar readable.
• The heading at line 49 is still a full sentence ("Now let's first understand what Batch Jobs are and the concept of Gang Scheduling, also how they create a DEADLOCK Issue through an example."). Something like "Batch Jobs and Gang Scheduling" would do the same job.
• Your bio in authors.json is still the placeholder "Writing some cool stuff ... ". Worth a real sentence, it shows on your author page.

Fix the install command and the suspend: true ending, decide what to do about the deadlock section, and do a pass on the arrow blocks and em dashes, and I'm happy to publish this. The demo section is in good shape now, which was the hard part.

[ekam-walia]

Hi @saiyam1814, I have made all the changes that were suggested by you, I have made the whole deadlock block again
I have added the yaml files as well as the output one should receive in the deadlock scenario
Thank you !

[saiyam1814]

Thanks @ekam-walia. The deadlock section is the real thing now: actual FailedCreate quota events, controller UIDs, your own shell prompt in the pastes. That was the hardest ask on the list and you delivered it. Fresh preview is building here:

https://blog-inside-kueue.website-dab.pages.dev/inside-kueue-how-kubernetes-decides-what-runs-next

Confirmed fixed from last round: the install command (line 228), the suspend: true ending replaced with the label, all API versions on v1beta2, the them?Jobs spacing and the unclosed quote, the odd completions: 1 / parallelism: 2 job is gone, the Pod 1-4 chain and the other arrow blocks you wrapped in text fences render correctly, and your bio is a real bio. Good pass.

One genuinely interesting thing came out of verifying this round, and it's the main thing left to fix.

The label isn't where you think it is, and your own demo proves it

I went to check why your outputs worked, because the YAMLs say they shouldn't have:

• In the first demo, job-a-big (line 363) and job-b-small (line 429) carry kueue.x-k8s.io/queue-name only on the pod template labels. Kueue's Job integration reads that label from the Job's own metadata.labels, not the template (official sample).
• In the deadlock demo, Job A (Step 5) has no queue-name label anywhere, yet your output shows workload job-job-a-takes-800m-c8bc8 admitted into smart-queue.

So why did it all work on your cluster? Because both of your LocalQueues are named default, and Kueue has LocalQueue defaulting (stable since v0.17): any unlabeled job in a namespace that has a LocalQueue named default gets adopted automatically. Every job in this post was enrolled by the queue's name, not by your labels. The template-level labels are doing nothing.

That matters because the post ends on "The magic? One label", and a reader who copies your YAMLs into a namespace whose LocalQueue is named anything else will watch Kueue ignore their jobs completely, with no idea why.

The fix is small: put kueue.x-k8s.io/queue-name: default in the Job's metadata.labels in every Kueue-managed YAML (Step 6 already has it there, just drop the duplicate from its template), add it to deadlock-demo Job A, and remove the template-only copies in the first demo. If you want, add one sentence noting that a LocalQueue named default adopts unlabeled jobs automatically. That's a nice pro tip and it explains your own terminal output honestly.

Output pastes that don't match the commands next to them

Same class of issue as round one, four spots:

• Step 6's YAML says echo 'All pods running together!'; sleep 60 (line 943), but the describe output shows echo 'Pod started. Waiting for partner pod...'; sleep 600 (line 1015). One of them was edited after the capture. Re-run with the YAML as published, or change the YAML back to what you ran.
• Line 713 says "check pods" but the command is kubectl get jobs, and the output below it is from kubectl get pods.
• Line 660 gives kubectl get pods -n deadlock-demo -o wide, but the output has no NODE/IP columns and your prompt line shows you ran it without -o wide. Drop the flag.
• Line 980: kubectl describe job-b-gang -n kueue-demo is missing the resource type, it would error as written. Your own paste right below has the correct kubectl describe job job-b-gang. Also that output fence starts with leftover scrollback (the previous get workloads run repeated), trim it.

The two demos collide if someone runs the post top to bottom

Demo 1 creates LocalQueue default in kueue-demo pointing at demo-queue. The deadlock solution then reuses the same namespace and applies LocalQueue default pointing at smart-queue. A reader following along hits a conflict there, and demo 1's ClusterQueue and jobs may still be holding quota. Add a short cleanup step between the two (kubectl delete ns kueue-demo plus kubectl delete clusterqueue demo-queue), or give part two its own namespace. Related: Step 1 creates kueue-demo with kubectl create ns and Step 4's YAML creates the same Namespace again, keep one.

The abstract deadlock story contradicts its own numbers

The "Real Problem (Without Kueue)" block runs on 12 CPUs, Job A takes 8, Job B needs 6 as 2 pods. But then "Pod 1 starts with 4 CPUs" (a pod requests a fixed amount, it can't soak up whatever is free; 2 pods needing 6 total is 3 each) and "Pod 2 is waiting for 3 CPUs" while the setup says only 4 of 6 were available. It also says "Kubernetes admits it anyway" with Pod 2 Pending, but your actual demo blocks pod creation via ResourceQuota, so Pod 2 never exists at all. Simplest fix: rewrite that block using the demo's real numbers (1500m quota, A takes 800m, B is 2 x 400m, pod 2 forbidden because 1200m + 400m > 1500m) so the story and the terminal output match.

Render and style pass

• Bare lines that will collapse into run-on paragraphs, same fix as the ones you already did (bullets or a text fence): the "With gang scheduling (Kueue)" block at lines 136-144 (the two-space indent on the Yes?/No? lines isn't enough to make a code block), the "Kubernetes scheduler doesn't understand gang scheduling" list at 148-152, the "THIS is the difference! Intercepted / Calculated / Checked / Suspended" list at 1034-1039, and the wrap-up's "Without Kueue: / With Kueue: / The magic?" at 1050-1055.
• Three new em dashes arrived with the new section: lines 726, 1039, 1059. And line 45, where the old one was deleted, now reads "a Batch Job maybe it's a machine learning..." and needs the comma that replaced it.
• Curly quotes from a word processor at lines 28, 45, 47, 51, 158, 170. Straight quotes please.
• Headings: line 51 is still a full sentence ("Now let's first understand what are Batch Jobs..."), something like "Batch Jobs, Gang Scheduling, and Deadlock" does the job. "## What happens?" at line 95 is a fragment sitting in the TOC as an H2, make it bold text. "Now Lets use Kueue to Solve this problem" at line 802 wants "Let's" and normal capitalization.
• Line 592: kubectl create namespace deadlock-demo is fenced as yaml, should be bash.
• "As we can see in line 13 of the terminal's output that is ... shows that how Kueue automatically suspended" reads broken. Try: "Note the Suspend: true line: Kueue set that itself when it held the job back."
• Optional: the Resources links render as unclickable monospace inside the text fence, a plain bullet list would let people click them.

That's the whole list, and none of it is structural this time. Fix the label placement (the one that changes what readers learn), align the four pastes, add the cleanup step, and do the render pass, and this merges. I'll bump datePublished to the actual publish date when it goes in.

The instinct to verify why something works, not just that it works, is what separates this post from most Kueue content out there. You're one pass away.