Fix COUNT bug when pulling up scalar subqueries (fallback planner) by Alena0704 · Pull Request #397 · open-gpdb/gpdb

Alena0704 · 2026-07-02T16:37:36Z

With the Postgres planner (optimizer=off or an ORCA fallback), a correlated
scalar subquery with an aggregate is pulled up into an INNER join with a
grouped subquery (convert_EXPR_to_join), which drops outer rows that have
no match. The original subquery keeps them: it computes the aggregate over
empty input, so e.g. COUNT yields 0 there:

select ... from t1
where t1.a > (select count(*) from t2 where t2.a = t1.d);

A row with no match in t2 must be compared as "t1.a > 0" and can pass, but
the INNER join dropped it.

To fix this, pull the subquery up into a LEFT join, so no-match rows survive
as null-extended rows, and rewrite the comparison to return the same value
the subquery would:

outer OP CASE WHEN match_flag THEN expr ELSE empty_input_default END

match_flag is a constant TRUE column added to the subquery. For a matched
row the CASE returns the real expression; for a null-extended row the flag
is NULL and the CASE returns the empty-input default (0 for COUNT, NULL for
other aggregates).

The comparison runs above the LEFT join as a filter, not as the join
condition: as a join qual it would null-extend matched rows that fail it,
and the default would let them back in.

The LEFT join is not always needed. If a no-match row cannot pass the
comparison anyway -- e.g. "1 = (select count(*) ...)" turns into "1 = 0"
for it -- dropping it is fine and the INNER join is kept as before. This is
detected by substituting the empty-input default into the comparison and
constant-folding it. Ordinary sum/avg/min/max comparisons fall into this
group: their empty-input value is NULL, and a comparison with NULL does not
pass, so those plans do not change.

If the comparison cannot be placed above the join (the sublink is in an
outer join's ON clause) or the subquery's targetlist is correlated, the
pull-up bails out and the sublink runs as a SubPlan, as before.

Co-Authored-By: excaliiibur excaliiibur@foxmail.com

The issue is #378

…elated scalar subquery with an aggregate is pulled up into an INNER join with a grouped subquery (convert_EXPR_to_join), which drops outer rows that have no match. The original subquery keeps them: it computes the aggregate over empty input, so e.g. COUNT yields 0 there: select ... from t1 where t1.a > (select count(*) from t2 where t2.a = t1.d); A row with no match in t2 must be compared as "t1.a > 0" and can pass, but the INNER join dropped it. To fix this, pull the subquery up into a LEFT join, so no-match rows survive as null-extended rows, and rewrite the comparison to return the same value the subquery would: outer OP CASE WHEN match_flag THEN expr ELSE empty_input_default END match_flag is a constant TRUE column added to the subquery. For a matched row the CASE returns the real expression; for a null-extended row the flag is NULL and the CASE returns the empty-input default (0 for COUNT, NULL for other aggregates). The comparison runs above the LEFT join as a filter, not as the join condition: as a join qual it would null-extend matched rows that fail it, and the default would let them back in. The LEFT join is not always needed. If a no-match row cannot pass the comparison anyway -- e.g. "1 = (select count(*) ...)" turns into "1 = 0" for it -- dropping it is fine and the INNER join is kept as before. This is detected by substituting the empty-input default into the comparison and constant-folding it. Ordinary sum/avg/min/max comparisons fall into this group: their empty-input value is NULL, and a comparison with NULL does not pass, so those plans do not change. If the comparison cannot be placed above the join (the sublink is in an outer join's ON clause) or the subquery's targetlist is correlated, the pull-up bails out and the sublink runs as a SubPlan, as before. Co-Authored-By: excaliiibur <excaliiibur@foxmail.com>

Adapted from open-gpdb/gpdb#397. INCOMPLETE ON CLOUDBERRY -- do not merge as is; see "PG16 gap" below. With the Postgres planner (optimizer=off or an ORCA fallback), a correlated scalar subquery with an aggregate is pulled up into an INNER join with a grouped subquery (convert_EXPR_to_join), which drops outer rows that have no match. The original subquery keeps them: it computes the aggregate over empty input, so e.g. COUNT yields 0 there: select ... from t1 where t1.a > (select count(*) from t2 where t2.a = t1.d); A row with no match in t2 must be compared as "t1.a > 0" and can pass, but the INNER join dropped it. The bug is confirmed present in Cloudberry, and it is reachable with the default optimizer=on whenever ORCA falls back: set optimizer=on; set optimizer_enable_orderedagg=off; select string_agg(a::text, ',' order by a) from t_out where a > (select count(*) from t_in where t_in.a = t_out.a); -> NULL (correct answer: "1") It is not limited to plain count(*); any expression whose empty-input value is non-NULL loses rows, e.g. "0 = (select coalesce(sum(a),0) ...)" and "1 = (select count(*) + 1 ...)". Plain min/sum/avg/max under a strict comparison are unaffected, as intended. The fix pulls the subquery up into a LEFT join so no-match rows survive as null-extended rows, and rewrites the comparison to return the same value the subquery would: outer OP CASE WHEN match_flag THEN expr ELSE empty_input_default END PG16 gap (why this is WIP): PG16 tracks outer-join nullability through Var.varnullingrels, and initsplan.c deconstruct_recurse() only registers an outer join -- outer_join_rels, mark_rels_nulled_by_join() -- when the JoinExpr has a real rtindex. make_join_expr() builds the pulled-up join with rtindex 0, so the planner never learns the subquery rel is nullable, the comparison placed above the join is pushed down, and the LEFT join is then dropped as useless, taking the qual with it: explain (costs off) select * from t_out where 0 = (select count(*) from t_in where t_in.a = t_out.a); Gather Motion 3:1 -> Seq Scan on t_out <- no join, no filter which is a worse wrong answer than the original bug, and the variant with a Var on the left fails outright with "ERROR: no relation entry for relid 2 (relnode.c:555)". Remaining work: give the pulled-up LEFT JoinExpr a real RT entry (addRangeTableEntryForJoin) and set j->rtindex, then mark the subquery rel's Vars in the lifted comparison with that rtindex in varnullingrels. Expected outputs then have to be regenerated -- the ones committed here are gpdb's and are NOT valid for Cloudberry. The upstream PR also touches expected/bfv_dd.out, whose surrounding output already differs in Cloudberry; that file is left untouched. Co-Authored-By: excaliiibur <excaliiibur@foxmail.com>

With the Postgres planner (optimizer=off or an ORCA fallback), a correlated scalar subquery with an aggregate is pulled up into an INNER join with a grouped subquery (convert_EXPR_to_join), which drops outer rows that have no match. The original subquery keeps them: it computes the aggregate over empty input, so e.g. COUNT yields 0 there: select ... from t1 where t1.a > (select count(*) from t2 where t2.a = t1.d); A row with no match in t2 must be compared as "t1.a > 0" and can pass, but the INNER join dropped it. To fix this, pull the subquery up into a LEFT join, so no-match rows survive as null-extended rows, and rewrite the comparison to return the same value the subquery would: outer OP CASE WHEN match_flag THEN expr ELSE empty_input_default END match_flag is a constant TRUE column added to the subquery. For a matched row the CASE returns the real expression; for a null-extended row the flag is NULL and the CASE returns the empty-input default (0 for COUNT, NULL for other aggregates). The comparison runs above the LEFT join as a filter, not as the join condition: as a join qual it would null-extend matched rows that fail it, and the default would let them back in. The LEFT join is not always needed. If a no-match row cannot pass the comparison anyway -- e.g. "1 = (select count(*) ...)" turns into "1 = 0" for it -- dropping it is fine and the INNER join is kept as before. This is detected by substituting the empty-input default into the comparison and constant-folding it. Ordinary sum/avg/min/max comparisons fall into this group: their empty-input value is NULL, and a comparison with NULL does not pass, so those plans do not change. If the comparison cannot be placed above the join (the sublink is in an outer join's ON clause) or the subquery's targetlist is correlated, the pull-up bails out and the sublink runs as a SubPlan, as before. Adapted from open-gpdb/gpdb#397 and from the 6.x version of this commit. Co-Authored-By: excaliiibur <excaliiibur@foxmail.com>

With the Postgres planner (optimizer=off or an ORCA fallback), a correlated scalar subquery with an aggregate is pulled up into an INNER join with a grouped subquery (convert_EXPR_to_join), which drops outer rows that have no match. The original subquery keeps them: it computes the aggregate over empty input, so e.g. COUNT yields 0 there: select ... from t1 where t1.a > (select count(*) from t2 where t2.a = t1.d); A row with no match in t2 must be compared as "t1.a > 0" and can pass, but the INNER join dropped it. To fix this, pull the subquery up into a LEFT join, so no-match rows survive as null-extended rows, and rewrite the comparison to return the same value the subquery would: outer OP CASE WHEN match_flag THEN expr ELSE empty_input_default END match_flag is a constant TRUE column added to the subquery. For a matched row the CASE returns the real expression; for a null-extended row the flag is NULL and the CASE returns the empty-input default (0 for COUNT, NULL for other aggregates). The comparison runs above the LEFT join as a filter, not as the join condition: as a join qual it would null-extend matched rows that fail it, and the default would let them back in. The LEFT join is not always needed. If a no-match row cannot pass the comparison anyway -- e.g. "1 = (select count(*) ...)" turns into "1 = 0" for it -- dropping it is fine and the INNER join is kept as before. This is detected by substituting the empty-input default into the comparison and constant-folding it. Ordinary sum/avg/min/max comparisons fall into this group: their empty-input value is NULL, and a comparison with NULL does not pass, so those plans do not change. If the comparison cannot be placed above the join (the sublink is in an outer join's ON clause) or the subquery's targetlist is correlated, the pull-up bails out and the sublink runs as a SubPlan, as before. Adapted from open-gpdb/gpdb#397 Co-Authored-By: excaliiibur <excaliiibur@foxmail.com>

With the Postgres planner (optimizer=off or an ORCA fallback), a correlated scalar subquery with an aggregate is pulled up into an INNER join with a grouped subquery (convert_EXPR_to_join), which drops outer rows that have no match. The original subquery keeps them: it computes the aggregate over empty input, so e.g. COUNT yields 0 there: select ... from t1 where t1.a > (select count(*) from t2 where t2.a = t1.d); A row with no match in t2 must be compared as "t1.a > 0" and can pass, but the INNER join dropped it. To fix this, pull the subquery up into a LEFT join, so no-match rows survive as null-extended rows, and rewrite the comparison to return the same value the subquery would: outer OP CASE WHEN match_flag THEN expr ELSE empty_input_default END match_flag is a constant TRUE column added to the subquery. For a matched row the CASE returns the real expression; for a null-extended row the flag is NULL and the CASE returns the empty-input default (0 for COUNT, NULL for other aggregates). The comparison runs above the LEFT join as a filter, not as the join condition: as a join qual it would null-extend matched rows that fail it, and the default would let them back in. The LEFT join is not always needed. If a no-match row cannot pass the comparison anyway -- e.g. "1 = (select count(*) ...)" turns into "1 = 0" for it -- dropping it is fine and the INNER join is kept as before. This is detected by substituting the empty-input default into the comparison and constant-folding it. Ordinary sum/avg/min/max comparisons fall into this group: their empty-input value is NULL, and a comparison with NULL does not pass, so those plans do not change. If the comparison cannot be placed above the join (the sublink is in an outer join's ON clause) or the subquery's targetlist is correlated, the pull-up bails out and the sublink runs as a SubPlan, as before. Adapted from open-gpdb/gpdb#397 and from the 6.x version of this commit. Co-Authored-By: excaliiibur <excaliiibur@foxmail.com>

With the Postgres planner (optimizer=off or an ORCA fallback), a correlated scalar subquery with an aggregate is pulled up into an INNER join with a grouped subquery (convert_EXPR_to_join), which drops outer rows that have no match. The original subquery keeps them: it computes the aggregate over empty input, so e.g. COUNT yields 0 there: select ... from t1 where t1.a > (select count(*) from t2 where t2.a = t1.d); A row with no match in t2 must be compared as "t1.a > 0" and can pass, but the INNER join dropped it. To fix this, pull the subquery up into a LEFT join, so no-match rows survive as null-extended rows, and rewrite the comparison to return the same value the subquery would: outer OP CASE WHEN match_flag THEN expr ELSE empty_input_default END match_flag is a constant TRUE column added to the subquery. For a matched row the CASE returns the real expression; for a null-extended row the flag is NULL and the CASE returns the empty-input default (0 for COUNT, NULL for other aggregates). The comparison runs above the LEFT join as a filter, not as the join condition: as a join qual it would null-extend matched rows that fail it, and the default would let them back in. The LEFT join is not always needed. If a no-match row cannot pass the comparison anyway -- e.g. "1 = (select count(*) ...)" turns into "1 = 0" for it -- dropping it is fine and the INNER join is kept as before. This is detected by substituting the empty-input default into the comparison and constant-folding it. Ordinary sum/avg/min/max comparisons fall into this group: their empty-input value is NULL, and a comparison with NULL does not pass, so those plans do not change. If the comparison cannot be placed above the join (the sublink is in an outer join's ON clause) or the subquery's targetlist is correlated, the pull-up bails out and the sublink runs as a SubPlan, as before. Adapted from open-gpdb/gpdb#397. Co-Authored-By: excaliiibur <excaliiibur@foxmail.com>

Alena0704 force-pushed the count-aggr-first-fallback branch from e640835 to 68c09fb Compare July 2, 2026 20:05

Alena0704 marked this pull request as draft July 2, 2026 20:12

Alena0704 force-pushed the count-aggr-first-fallback branch 3 times, most recently from 9ffda49 to 0f6cfed Compare July 2, 2026 21:06

Alena0704 closed this Jul 2, 2026

Alena0704 reopened this Jul 2, 2026

Alena0704 mentioned this pull request Jul 3, 2026

[subquery] Preserve COUNT semantics in scalar subquery pull-up #376

Draft

Alena0704 force-pushed the count-aggr-first-fallback branch 4 times, most recently from 148ead5 to c9b09dc Compare July 14, 2026 13:23

Alena0704 marked this pull request as ready for review July 14, 2026 15:59

Alena0704 force-pushed the count-aggr-first-fallback branch 7 times, most recently from caca6e8 to 49938bd Compare July 16, 2026 15:55

Alena0704 force-pushed the count-aggr-first-fallback branch from 49938bd to acff7ac Compare July 17, 2026 09:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix COUNT bug when pulling up scalar subqueries (fallback planner) - #397

Fix COUNT bug when pulling up scalar subqueries (fallback planner)#397
Alena0704 wants to merge 1 commit into
OPENGPDB_STABLEfrom
count-aggr-first-fallback

Alena0704 commented Jul 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Alena0704 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Alena0704 commented Jul 2, 2026 •

edited

Loading