feat(v2): Phase 1 — schema registry & versioning (/api/v2)#219
Merged
Conversation
…mpy, PUT/defaults)
Pandera 0.32/pandas 3.0 drops Column.metadata through to_json, so the review widget is carried via a review_widgets side map (spec §13). validate_record_fields separates nulls so a None in a nullable numpy-int column no longer fails coercion.
Adds review_widgets JSONB overlay on SchemaVersion per spec §13 (Pandera to_json drops Column.metadata, so review widgets are carried out-of-band).
Includes review_widgets column; the circular schemas.current_version_id FK is added only on non-SQLite dialects (SQLite can't ALTER-add constraints; tests run on SQLite).
Adds a one-directional Schema.workspace relationship so the SubFactory pattern (and FK population) works as in DatasetFactory.
SchemaVersionCreate/Read carry review_widgets (spec §13).
publish_version flushes the new version before pointing schema.current_version_id at it (avoids the schemas<->schema_versions FK cycle and an unset version id). Threads review_widgets into the stored version and derived columns_cache (spec §13).
…r-workspace authz SchemaPolicy mirrors DatasetPolicy (owner always; admin+member for writes). Mounts a separate api_v2 sub-app at /api/v2. Adds an isolated tests/integration conftest with a v2-aware async_client (overrides get_async_db + get_s3_client on api_v2) and no OpenSearch fixture.
…VersionId) - validate_record_fields: convert numpy->native via .item()/ISO instead of lossy to_json (preserves high-precision floats + datetimes); reject omitted required cols. - put_object: propagate S3 VersionId so schema_versions.object_version_id is populated. - Add model default/uniqueness + float/datetime/missing-required tests. - Document accepted/deferred findings (publish race, 404-vs-403, commit granularity).
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
frame.iloc[0] collapses the validated row into a single Series whose dtype is the columns' common type; an int64 cell alongside a float64 column upcasts to float, silently degrading integer fidelity for record.fields. Extract each cell per-column (frame[col].iloc[0]) instead. Adds an Int+Float (no string column) regression test. Addresses roborev job 62.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of 6 — Schema-centric data model
Stands up the
Schemaentity (the v2 "Dataset") and its object-store-backed, versioned Pandera body as an isolated/api/v2module, alongside untouched v1. Implements the Phase 1 plan from the design spec.What's included
models/v2/—Schema+SchemaVersionORM models (registry pointer + lineage + denormalizedcolumns_cache;review_widgetsoverlay).schemas/schema_versionstables (additive, beside v1).contexts/v2/schema_bodies.py— pure Pandera helpers:derive_columns_cache+ null/missing-tolerantvalidate_record_fields.contexts/v2/schemas.py— schema CRUD + object-store-backedpublish_version(uploads body, derives cache, advancescurrent_version_id, links lineage).api/v2/—/api/v2schema router (CRUD, version publishing,GET .../columns) gated by a newSchemaPolicymirroringDatasetPolicy(owner always; admin+member for writes; member for reads). Mounted as a separateapi_v2sub-app.tests/integration/**/v2/with a v2-aware conftest.Notable decisions / deviations (documented in spec §13)
Column.metadatathroughto_json, so the per-column review widget is carried out-of-band inreview_widgets(the plan's documented fallback).pa.Intis numpy int64 (can't holdNone); validation separates nulls, enforces missing required columns, and converts numpy→native via.item()/ISO (no lossyto_json), preserving high-precision floats + datetimes for the Phase-2record.fieldsJSONB.schemas.current_version_id → schema_versions.idis added only on non-SQLite (SQLite can't ALTER-add constraints; the test suite runs on SQLite).publish_versionflushes the version before pointing at it to avoid the FK cycle.tests/integration/(nottests/unit/) so the v2 suite runs without the legacy OpenSearch session fixture — keeping v2 genuinely isolated.put_objectnow propagates the S3VersionId(was always NULL) soobject_version_idis populated.Review
roborev reviewed each commit; 4 findings addressed (lossy coercion, missing-required enforcement, S3 VersionId, model test coverage). Accepted/deferred (Low): concurrent-publish race (Phase-1 inline sync; hardened at job-queue offload) and v1-consistent 404-vs-403 on fetch-then-authorize.
Out of scope (later phases)
Records + bulk-upsert (2), LanceDB index (3), annotation v2 (4), queues (5), v1→v2 migrator + retirements (6).