Debounce transient Kubernetes cluster-health failures

[mterwill]

#6742 changed tilt ci so that any cluster error immediately fails the CI session. The cluster error it relies on, however, is produced by a single health-check poll and cleared by the next successful poll.

Kubernetes evaluates health over a series of probes rather than treating one failed check as terminal. Its probe model uses a failureThreshold and only considers the overall check failed after successive failures. Tilt should apply similar tolerance before turning a point-in-time health observation into an irreversible CI result.

We see this most often on busy clusters with many concurrent CI jobs, where one brief control-plane interruption can cascade into failures across every active Tilt session.

This change requires health checks to fail continuously for one minute—five successive failed polls at the current 15-second interval—before reporting a cluster error. A successful check resets the timer immediately. Sustained failures still terminate CI, but one failed poll no longer amplifies a brief interruption into many permanent job failures.

[nicks]

this pr is marked as draft -- is this something you're still working on?

[mterwill]

Thanks @nicks! The original branch was meant to add observability into a failure we were seeing internally. I’ve investigated further and reworked it around what I believe is the underlying issue.