Skip to content

Avoid train model progress hanging - #952

Open
microbit-grace wants to merge 5 commits into
mainfrom
handle-train-fails
Open

Avoid train model progress hanging#952
microbit-grace wants to merge 5 commits into
mainfrom
handle-train-fails

Conversation

@microbit-grace

@microbit-grace microbit-grace commented Jul 24, 2026

Copy link
Copy Markdown

Addresses #951

  • Ensured train model failures raise error dialog
  • Added a ml worker watchdog that raises error if no progress is made in 60s
  • Added a storage write 15s timeout (app-wide) in case it hangs
  • Added Sentry logging with reference to reason for failure (timeout/worker error/request error/other thrown error)

The progress dialog can only be dismissed by the stage leaving TrainingInProgress, so any unhandled throw in trainModel (e.g. from updateProject) would otherwise strand it at 100%. On failure, we fall through to TrainingError.
Otherwise, timeout after 60s and raise error dialog.
@github-actions

Copy link
Copy Markdown

Preview build will be at
https://review-createai.microbit.org/handle-train-fails/

@microbit-grace

microbit-grace commented Jul 24, 2026

Copy link
Copy Markdown
Author

I'm not 100% sure on the arbitrary timeout cutoff times, but tried to make them generous so that it is more convincing that it has hung. Alternative suggestions welcomed.

with reason as to failure cause (timeout, worker error, or request error)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant