feat!: id-based outer-reference resolution (Substrait 0.89.0)

[nielspardon]

Summary

Updates the substrait submodule to v0.89.0 and adds support for its new id-based outer-reference resolution (RelCommon.rel_anchor + OuterReference.rel_reference) alongside the existing offset-based steps_out mechanism, then makes the isthmus integration resolve correlations natively by id.

Implements #869.

Core (feat(core)!)

• Model: Rel.getRelAnchor() / Rel.withRelAnchor(...) and FieldReference.outerReferenceRelReference().
• Mutual exclusivity: steps_out and rel_reference are a single protobuf oneof, so they are mutually exclusive — a FieldReference carries at most one (enforced by FieldReference#check()), and both proto converters emit/read whichever form is set. (Unlike the URI→URN case in the breaking-change policy, these are not parallel fields, so there is no "dual write / prefer" behavior.)
• OuterReferenceConverter rewrites a plan's outer references from one form to the other (toIdBased / toStepsOut). Assigning plan-wide-unique rel_anchors needs whole-plan context, so it lives in core.

Isthmus (feat(isthmus) → feat(isthmus)!)

Calcite's correlation model is itself id-based (CorrelationId is an identity bound to one relation), so isthmus now resolves outer references natively by anchor rather than by subquery depth:

• Produce: OuterReferenceResolver maps each CorrelationId to a plan-wide rel_anchor and its binding relation; SubstraitRelVisitor stamps the anchor on that relation and RexExpressionConverter emits rel_reference directly. The offset-based depth computation is gone.
• Consume: SubstraitRelNodeConverter.Context resolves rel_reference by anchor (no subquery-depth arithmetic). Offset-based (steps_out) plans are still accepted — they are normalized to the id-based form once at the conversion boundary, so the internal machinery is purely id-based.

This replaces the earlier depth/steps_out bridge with a closer, simpler fit to Calcite's own model.

Breaking changes

• Rel gains an abstract getRelAnchor(); non-Immutables Rel implementations must implement it (withRelAnchor(...) is a throwing default the Immutables override).
• Isthmus's outer-reference API changed: OuterReferenceResolver.apply → resolve + anchorForCorrelationId/anchorForTarget; SubstraitRelVisitor.popFieldAccessDepthMap/getFieldAccessDepth → resolveOuterReferences/getOuterReferenceAnchor; the SubstraitRelNodeConverter.Context correlation API is now anchor-based.

Known limitation

OuterReferenceConverter supports outer references whose binding relation is the input of a single-input host (Filter/Project), covering correlated scalar/IN/EXISTS subqueries. A correlated reference into a multi-input (join/set) scope, or a shared subtree (ReferenceRel), throws UnsupportedOperationException. isthmus does not produce those shapes.

Testing

• New core round-trip tests for both outer-reference encodings, rel_anchor, OuterReferenceConverter (round-trip identity, nesting, shared-scope dedup, unsupported cases), plus a mutual-exclusivity test.
• Isthmus SubqueryConversionTest covers native id-based consume and offset-based (boundary-normalized) consume; SubqueryPlanTest covers id-based produce; OuterReferenceResolverTest covers anchor assignment and the dangling/partially-decorrelated case.
• :core:test, :isthmus:test (incl. TPC-H / TPC-DS correlated-subquery round-trips), :core:javadoc, :isthmus:javadoc, spotlessCheck, and Spark compilation all pass.

🤖 Generated with AI