Skip to content

Fail closed when local output violates its output schema #33

Description

@adaamko

Problem

Pydantic validation errors can currently be printed while the invalid local result is still returned, and dict schemas are treated as valid without equivalent runtime validation.

Direction

  • Compile both Pydantic and dict output schemas into enforceable validators.
  • Validate every locally produced result before considering it handled.
  • On failure, return a structured local error and use the configured fallback when available.
  • Never return a partially validated local result.

Acceptance

  • Invalid Pydantic output is not returned as a successful local result.
  • Invalid dict-schema output is detected.
  • A configured fallback receives schema-invalid local requests.
  • Without a fallback, the caller receives an explicit error/abstention outcome.
  • Tests cover classification, extraction/NER, and transformation outputs.

Non-goals

  • Redesigning the complete public result API in this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    correctnessCorrectness or methodology fix

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions