Score Studio

Recover a failed training attempt

Find the cause and keep the exact data selection instead of starting another run blindly.

Source reviewed 2026-09-23. Availability depends on your installation, permissions, and compatible runtimes. A supported path is not a guarantee of model quality or production readiness.

Outcome

A reviewed replacement setup, not an automatic restart or a fabricated ready model.

Start here

What you need to know first

Check the inputs below before starting. If you are new, begin with the first-project guide. A dataset holds media and labels; a model produces results; a deployment makes a selected model version callable. Creating one does not create the others.

Bring these inputs

  • Failed attempt and its recorded dataset version
  • Named image or provider failure detail

Expected output and limits

  • A reviewed retry setup
  • A new attempt only when explicitly started

Core workflow

  1. 01

    Open the failed attempt

    Open its model page or Runs. Read the failure and inspect retained work; dismissing a notification does not delete the attempt.

  2. 02

    Repair the right input

    For invalid class or box coordinates, use Review training annotations and inspect the named image. Changing compute does not fix annotations. Provider failures need a provider/runtime check instead.

  3. 03

    Preserve the version

    Review dataset and retry retains the recorded source version. A missing saved version needs an explicit selection; it must not silently switch to latest.

  4. 04

    Review, then restart

    Retry your first model initializes from foundation weights when no ready incumbent exists. Earlier ready models remain unchanged. Check independent provider jobs before launching a replacement.

What this surface supports

  • Specific recovery actions
  • Pinned source dataset
  • First-model retry
  • Retained failure history

Expert section

Contracts, signals, and failure modes

Use this section when you are defining acceptance criteria, automating the surface, or reviewing whether its output is safe to promote downstream.

Quality and operating signals

  • Failure category
  • Retained artifact state
  • Exact source version

Common failure modes

  • Retrying invalid labels unchanged
  • Selecting a failed model as ready
  • Starting duplicate provider work

Expert release checklist

  • Inputs and dependencies are pinned to immutable versions.
  • Acceptance metrics include critical classes and operating slices.
  • Failure, retry, cost, and rollback behavior are understood.
  • The resulting artifact has an owner and a downstream review path.

Engineering safeguards

  • Retry creates another attempt
  • A failed version is not runnable
  • Worker retries are bounded; the error inventory is not continuous alerting
Recover a failed training attempt · Score Studio