Skip to content

feat(v2.3.0): add provider-neutral source adapters and dataset catalog - #471

Draft
dmidlo wants to merge 3 commits into
devfrom
agent/issue-78-dataset-catalog
Draft

feat(v2.3.0): add provider-neutral source adapters and dataset catalog#471
dmidlo wants to merge 3 commits into
devfrom
agent/issue-78-dataset-catalog

Conversation

@dmidlo

@dmidlo dmidlo commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Closes #78

Summary

  • add strict provider, adapter, partition, dataset, version, alias, query, cursor, verification, replay, and lineage contracts
  • add first-party HistData and fixture-backed adapters with provider-neutral inventory/reconstruction preflight
  • add immutable catalog resolution plus installed list, describe, resolve, verify, and replay CLI/API surfaces
  • add API-ready observed/synthetic lineage projections while preserving all V1 schemas and HistData identities

Seam audit

  • dataset and inventory identities exclude local artifact paths while retaining paths in serialized manifests
  • requested symbol/period matrices and complete requested intervals fail closed when coverage is sparse
  • cursors bind exact dataset version, origin, scope, series/period or ensemble identity
  • synthetic lineage requires a synthetic manifest and an exact declared parent source version
  • provider/licensing identity remains distinct even for byte-identical artifacts; delivery/broker profiles remain orthogonal
  • Windows clean installs receive platform-scoped IANA timezone data and Unix-only RSS measurement is isolated behind an import-safe helper
  • recovery audit mapped every prior matrix failure to a concrete portability or diagnostic seam: Python 3.10 argparse and datetime compatibility, numerical-golden isolation, failed-fit state metadata, Windows-safe artifact names, writable Parquet fsync, and best-effort directory durability

Verification

  • python -m pytest: 2,065 passed, 4 skipped
  • affected compatibility and dataset seams: 211 passed
  • dedicated Python 3.10 reproduction checks for the prior failures: passed
  • pre-commit run --all-files: passed
  • Commitizen commit hooks and real pre-push hooks: passed
  • Sphinx warnings-as-errors: passed
  • local TestPyPI simple-registry preflight: passed, including build/check, clean install, runtime, CLI, quality, and download smoke
  • GitHub Actions run 29654591057: 17 successful, 2 intentionally skipped, 0 failed, 0 pending; full Python 3.10–3.13 Ubuntu matrix plus Python 3.13 macOS/Windows, wheel smoke, package build, docs, container, dependency audit, and CodeQL all green

Sequencing

This targets dev as v2.3.0 minor work. It does not expand #449 v2.1 certification scope and remains draft pending review and the intended release sequence.

dmidlo added 3 commits July 16, 2026 20:44
Implement issue #78 provider adapters, immutable catalog resolution, and API-ready lineage.
Add Windows timezone data and centralize import-safe peak RSS measurement for #78 CI.
…tforms

Keep failed state-space payloads aligned, make numerical goldens deterministic, and use Windows-compatible timestamp and fsync behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant