FAO delivery path is point/DataFrame-based — won't carry rusty_bucket's pooled draws

[Polichinel]

Problem

The unfao delivery manager (views_postprocessing/unfao/managers/unfao.py) is built around point forecasts:

• _read_forecast_data (~L62-100) downloads a single parquet DataFrame from Appwrite and wraps it in PGMDataset (pandas).
• _append_metadata (~L112-122) joins GAUL metadata onto that DataFrame.
• _save (~L206-223) writes pred_*_best / pred_*_prob point columns.

rusty_bucket (views-models#143) is the opposite by design: a PredictionFrameEnsembleManager + aggregation: concat ensemble that ships the full pooled posterior draws (~1024/cell, Track A y_pred.npy of shape (N, S)) uncollapsed, so the principled MAP/HDI collapse happens once downstream in views_frames_summarize.

So the current FAO delivery path would not carry rusty_bucket's draws — it expects a point DataFrame with _best/_prob columns, not (N, S) draw arrays. This is the downstream half of the views-models#143 contract: "the FAO delivery path must not collapse them."

Scope

• Adapt the unfao delivery to read/carry the pooled draws (frames / (N, S) arrays + identifiers) for rusty_bucket, delivering them uncollapsed.
• Keep the metadata enrichment (GaulLookupEnricher) and validation working on the frames/draws path.
• Decide the delivery artifact to FAO (raw draws vs the views_frames_summarize collapse output).

Not blocking yet

rusty_bucket's constituent models are still being tuned (views-models#143), so this is track-and-scope. It must land before rusty_bucket goes live.

Refs

• views-models#143 (rusty_bucket + the downstream draw-preservation contract)
• views_postprocessing/unfao/managers/unfao.py (_read_forecast_data, _append_metadata, _save)
• Related: C-40 in the register (the manager's pipeline-core coupling), which any rework here should keep in mind.

[Polichinel]

Cross-repo note from the views-faoapi side — this issue is the keystone for faoapi going frame-native.

Confirmed end-to-end: unfao/managers/unfao.py _save → to_parquet() on the object-dtype DataFrame is exactly where the nested-array-sample parquet that faoapi consumes is born. Downstream, faoapi does the inverse dance — pd.read_parquet → _convert_to_arrays (decode each object-dtype cell back into an np.ndarray) → (N_time, N_entity, S) tensor (views-faoapi data/handlers.py ~234, ~467–516). So the object-dtype round-trip is pure overhead on both ends.

Suggested solution shape (so it unblocks the consumer cleanly):

• Emit a native views-frames frame carrying the full (N, S) pooled draws uncollapsed (as you note for rusty_bucket), and let the principled MAP/HDI collapse happen once in faoapi via views_frames_summarize (tower_point/hdi_tower — see views-faoapi#89).
• Use the arrow single-file form (views_frames.io.arrow), not the npz multi-file directory — Appwrite buckets are file-oriented (one blob per artifact).
• Carry the 9 GAUL metadata columns in the frame header/sidecar, not as DataFrame cells.

This is the producer half of the wire-format change; the faoapi consumer half + the ADR-046 amendment are tracked in views-faoapi (new issue, linked). Resolving this is what actually lets faoapi retire its parquet decode.

[Polichinel]

Sharpened by the samples/uncertainty requirement (2026-06-27). The reason this issue matters is now concrete: the delivery is moving from point estimates to predictions-with-uncertainty (S samples/cell). The two representations diverge exactly here:

• views-frames PredictionFrame = native contiguous (N, S) float32, sample axis always explicit (point is just S=1).
• pandas PGMDataset = object-dtype list-in-cell (each of N cells holds a length-S numpy array; _convert_to_arrays in pipeline-core data/handlers.py).

The object-dtype cost scales ~linearly with S — memory blow-up (pipeline-core's own OOM #181/#189, ~18 GB), encode/decode tax at every parquet/API boundary (faoapi's inverse decode, already noted on this issue), and a silent np.resize pad on mismatched sample counts. At S=1 invisible; at S≈1000 it dominates.

So this issue is the producer half of carrying uncertainty: emit (N, S) draws uncollapsed as a native arrow frame rather than object-dtype parquet. Gated by C-40 (the inherited pandas container) + rusty_bucket (#143). Register C-40 updated with this priority + the concrete pipeline-core gate (#186/#207/#161/#159/#211).

[Polichinel]

Migration backlog created (2026-06-28): epic #85. The pandas→views-frames map for views-postprocessing is now sequenced as epic #85 ("Push pandas to the seams"), stories #86–#92, tracking #93.

This issue (#45) is the gate + consumer half for S6 (#91) — the forecast outbound to_parquet → views_frames.io.arrow sample-frame wire, the step that actually ships (N,S) to FAO. The unilateral arc S1–S3 (#86/#87/#88) makes vpp's interior carry the draws as a PredictionFrame first, so by the time the wire cutover lands the producer side is already built + parity-tested.

[Polichinel]

Wire contract posted — this issue's scope is now bounded. Contract of record: views-platform/views-models#149 (comment: Sampled-Forecast Wire Contract v1, 2026-07-03); producer leg: views-platform/views-pipeline-core#269.

vpp's obligations under the contract:

• Hop A consumer (source adapter, extends S3 — forecast convert-at-the-door (internal PredictionFrame) #88): read the Track A archive (zip: y_pred.npy (N,S) + identifiers.npz + metadata.json) per (target, month) shard, manifest-gated, → interior per-target PredictionFrame via from_arrays. Validate payload-vs-header consistency at the door.
• Hop B producer (sink adapter, S6 — forecast outbound → arrow sample-frame wire (ships (N,S) to FAO) #91): emit one views_frames.io.arrow file per (target, month) to unfao_bucket, §2 header embedded.
• GAUL sidecar (§5): the 9 columns move to a gid-keyed sidecar (one per run) — faoapi already consumes them per-gid (verified); gid-set identity enforced by the existing coverage/identity invariants.
• The #149 no-collapse boundary is named: vpp delivery/draws.py (§6) — 2-D, S==header, S≥S_min, global non-degeneracy (per-row zero variance explicitly legal). New invariant, sibling of coverage/identity/observed_range/provenance.
• Historical artifact: unchanged (pandas parquet) in v1.

Target vocabulary decided: lr_ged_sb/ns/os (models#146).