Add job pod ephemeral-storage request/limit flags#308
Draft
wesleyjellis wants to merge 1 commit into
Draft
Conversation
Add --job-pod-requests-ephemeral-storage and --job-pod-limits-ephemeral-storage (both in MB, default 0 = unset) so job pods can declare ephemeral storage. Without a request, the scheduler packs disk-hungry jobs onto one node until the kubelet hits DiskPressure and evicts them mid-run. Both flags default to unset, so existing deployments are unaffected until they opt in. A pod config file can still override or extend Resources as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wesleyjellis
force-pushed
the
add-job-pod-ephemeral-storage-flags
branch
from
0603bf7 to
ccd0a52
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
During the 2026-07-06 us-prod incident,
runner-jobspods each wrote 14–26 GB of ephemeral storage with no request/limit declared (request is 0in the kubelet eviction events). The scheduler packed ~10 such jobs per node, driving node root filesystems from ~24% to 95–99%, putting six nodes intoDiskPressure, and causing the kubelet to evict job pods mid-run — surfacing as runner job timeouts. Investigation notebook: https://app.datadoghq.com/notebook/14931018What
--job-pod-requests-ephemeral-storageand--job-pod-limits-ephemeral-storage(both in MB, matching the memory flags' units)0= unset, so existing deployments see no behavior change until they opt inReadPodConfiginsrc/pkg/k8s_config.go; ephemeral-storage entries are added to the ResourceLists only when > 0 (a literal0limit would kill any pod that writes anything). A pod config file can still override/extendresourcesas before.Rollout
After release, set
--job-pod-requests-ephemeral-storage=5120(or similar) in opslevel-kubernetes runner deployments. A limit can follow once we settle on how to bound customer repo clones.🤖 Generated with Claude Code