docs(#1458): SparkAdapter Codec Protocol spec + explainer#188
Open
dimitri-yatsenko wants to merge 2 commits into
Open
docs(#1458): SparkAdapter Codec Protocol spec + explainer#188dimitri-yatsenko wants to merge 2 commits into
dimitri-yatsenko wants to merge 2 commits into
Conversation
Spec-first pair for the Renderable Protocol landing in DataJoint 2.3 (per user direction 2026-06-23, bringing T3.2 back into 2.3 scope). New files: - src/reference/specs/renderable.md — normative spec for the Renderable Protocol. Covers signature, return-value shape constraints (primitives / lists / dicts mapping to Spark ArrayType / StructType / MapType), why the contract is a Protocol rather than an abstract method on Codec, eligibility detection via isinstance, out-of-scope items, and two worked example codec implementations (FloatArrayCodec, Image2DCodec, PointWithLabelCodec). - src/explanation/renderable-codecs.md — explainer. Covers the Bronze/Silver layer model (CDC mirror vs typed silver layer), why <blob@> is bronze-only, what typed renderable codecs are, the design rationale for the Protocol pattern (smaller OSS surface, cleaner opt-in, no churn for existing plugins, structural typing), what's out of scope, and a decision guide for choosing codecs in a new pipeline. Nav entries added: - Reference > Specifications > Type System > Renderable Codec Protocol - Concepts > Storage > Renderable Codecs Implementation (against this spec) follows in datajoint-python; the addition is small (~10 lines: a runtime_checkable Protocol declaration in src/datajoint/rendering.py, re-exported as dj.Renderable). Examples use core DataJoint types (float64, int32) per project convention. Cross-links to codec-api.md (the base Codec interface that renderable codecs extend by composition, not inheritance).
3 tasks
Renderable conflicts with the broader notion of graphically renderable field types and is too generic for an interface targeted specifically at Spark / Lakehouse Sync. Rename for clarity: - Class: Renderable → SparkAdapter (parallels StorageAdapter) - Method: render_spark → to_spark (matches pandas/Arrow conventions like to_pandas, to_arrow, __dataframe__) - Spec file: renderable.md → spark-adapter.md - Explainer: renderable-codecs.md → spark-adapters.md - Nav entries updated in mkdocs.yaml
dimitri-yatsenko
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Spec-first pair for the Renderable Codec Protocol landing in DataJoint 2.3 (#1458). Brought into 2.3 scope per user direction 2026-06-23 (was previously deferred to 2.4).
Why spec-first
The Protocol itself is tiny (~10 lines: a `@runtime_checkable Protocol` declaration). The design conversation in the issue body settled the shape after #1457 (the earlier abstract-method-on-Codec framing) was rejected. Locking the spec now gives:
Marked draft
Stays draft until the matching implementation PR opens in `datajoint-python` — same pattern as the provenance trinity spec (#183) before #1471 landed against it.
Sequencing
Independent of the provenance trinity (no code overlap). Can land in parallel with T2.2 implementation work.
Test plan