Speed up the card list for large accounts by jorgemanrubia · Pull Request #2971 · basecamp/fizzy

jorgemanrubia · 2026-07-09T16:13:30Z

What this fixes

CardsController#index (/<acct>/cards?board_ids[]=…) runs hundreds of ms to seconds of DB time on large accounts — the account's card list with the Card.active anti-joins, a board_id IN (…) filter, and the default ORDER BY last_active_at DESC, id DESC.

Why it was slow

Filtering by board goes through creator.accessible_cards (an accesses INNER JOIN) and anchors the plan on index_cards_on_board_id — a range scan over every published card on the boards — then a filesort by last_active_at to return one page.

The fix

Keep creator.accessible_cards. The accesses INNER JOIN stays, so authorization is enforced by the query and never depends on the board list being pre-filtered.
Add a redundant account_id predicate + conditional index pin. accessible_cards scopes by accesses.user_id, not cards.account_id, so (account_id, last_active_at, status) isn't eligible. Every selected board belongs to the creator's account, so where(cards: { account_id: … }) makes that index usable, and pinning it holds the plan on its reverse scan — serving the ordering directly and stopping at the page limit, no filesort. Applied only when boards are filtered and no fan-out join (assignee/tag/term) is present.

Why pin, not just the predicate

With the predicate alone the optimizer's choice depends on how many boards are in the IN list: for a handful of boards it still picks index_cards_on_board_id and filesorts, only switching to the fast index for larger lists — and even there the two plans cost within a few percent, so statistics drift can flip it back. The conditional pin holds the fast plan across every board count.

Before / after

MySQL 8, 40k-active-card account filtered to 15 boards, identical result sets (20,005 == 20,005 ids; accesses join present in both):

Before: ~42.7 ms — index_cards_on_board_id, filesort.
After: ~0.9 ms — (account_id, last_active_at, status) reverse scan, no filesort.

Notes

No new index — reuses the existing (account_id, last_active_at, status) index.
The use_index hint is an optimizer-hint comment; SQLite/MariaDB ignore it (safe no-op).

Copilot

Pull request overview

This PR optimizes Filter#cards (used by CardsController#index) to reduce DB time for large accounts by avoiding an accesses join and by applying DISTINCT only when filters can introduce row fan-out, while preserving access control semantics via board scoping.

Changes:

Use a new base_cards relation to avoid the creator.accessible_cards join when the filter is board-scoped.
Apply DISTINCT only when assignee/tag/term filters are active (needs_deduplication?).
Add regression tests for board scoping access safety and tag-based deduplication.

Tip

If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
app/models/filter.rb	Changes the base relation used for card queries when boards are selected and avoids unconditional DISTINCT unless a fan-out join is involved.
test/models/filter_test.rb	Adds regression tests for board-scoped access safety and multi-tag deduplication behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    # When the filter is scoped to specific boards, those boards are already a
+    # subset of the creator's accessible boards (see Filter::Resources#boards),
+    # so `account.cards.where(board_id: boards.ids)` enforces the same access as
+    # `accessible_cards` while letting the query anchor on account_id. That lets
+    # the (account_id, last_active_at, status) index serve the default
+    # `last_active_at DESC` ordering directly — no accesses join, no filesort over
+    # the whole board — instead of scanning and sorting every card on the boards.


jorgemanrubia · 2026-07-09T16:32:03Z

+    # `last_active_at DESC` ordering directly — no accesses join, no filesort over
+    # the whole board — instead of scanning and sorting every card on the boards.
+    def base_cards
+      if boards.present?


I don't love this, this design feels brittle. Even when we validate that boards are accessible boards, I don't want to lose the .accessible_cards as the starting point. A future bug where we expose cards you don't have access to is scary. Please try to look for an alternative to keep that (e.g alternative index hints)

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

+    def board_scoped(relation)
+      relation = relation.where(cards: { account_id: creator.account_id }).where(board: boards.ids)
+      # Redundant account_id predicate + pin keep the last_active_at page on the
+      # index reverse scan; a fan-out join needs a different plan, so skip it then.
+      needs_deduplication? ? relation : relation.use_index(:index_cards_on_account_id_and_last_active_at_and_status)
+    end


CardsController#index (Filter#cards) got slow on large accounts — hundreds of ms to seconds of DB time for the /<acct>/cards?board_ids[]=... endpoint: the account's card list with the Card.active anti-joins, a board_id IN (...) filter, and the default ORDER BY last_active_at DESC, id DESC. Filtering by board goes through creator.accessible_cards (an accesses INNER JOIN) and anchors the plan on index_cards_on_board_id — a range scan over every published card on the boards — then a filesort by last_active_at to return one page. Keeping accessible_cards preserves the authorization guarantee: the accesses join is enforced by the query and never depends on the board list being pre-filtered. Adding a redundant account_id predicate (every selected board belongs to the creator's account) makes index_cards_on_account_id_and_last_active_at_and_status usable, and pinning it holds the plan on that index's reverse scan — serving the ordering directly and stopping at the page limit, no filesort. With the predicate alone the optimizer estimates the board-index plan as cheaper, so stale statistics can flip it back; the pin keeps the fast plan. It is skipped when an assignee, tag, or term filter adds a has-many join, where a different plan wins. On a 40k-active-card account filtered to 15 boards: 48ms -> 0.9ms of DB time, accesses join retained, no filesort. Result sets unchanged.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

+    def filter_boards(relation)
+      relation = relation.where(cards: { account_id: creator.account_id }).where(board: boards.ids)
+      return relation if fans_out?
+      # Pin the (account_id, last_active_at, status) index so the ordered page is served by a reverse scan, not a filesort.
+      relation.use_index(:index_cards_on_account_id_and_last_active_at_and_status)
+    end


Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

+    def filter_boards(relation)
+      relation = relation.where(cards: { account_id: creator.account_id }).where(board: boards.ids)
+      if joins_has_many?
+        relation
+      else
+        # Pin the (account_id, last_active_at, status) index so the ordered page is served by a reverse scan, not a filesort.
+        relation.use_index(:index_cards_on_account_id_and_last_active_at_and_status)
+      end


+    assert_not_includes filter.cards, inaccessible_card
+    filter.cards.each { |card| assert_includes users(:david).boards, card.board }


Copilot AI review requested due to automatic review settings July 9, 2026 16:13

Copilot started reviewing on behalf of jorgemanrubia July 9, 2026 16:14 View session

jorgemanrubia marked this pull request as draft July 9, 2026 16:18

jorgemanrubia marked this pull request as ready for review July 9, 2026 16:18

Copilot AI reviewed Jul 9, 2026

View reviewed changes

jorgemanrubia commented Jul 9, 2026

View reviewed changes

jorgemanrubia force-pushed the entropy-sweep-materialization branch from 949ecdb to 7cae224 Compare July 9, 2026 16:36

Copilot AI review requested due to automatic review settings July 9, 2026 16:36

Copilot started reviewing on behalf of jorgemanrubia July 9, 2026 16:36 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 16:48

jorgemanrubia force-pushed the entropy-sweep-materialization branch from 7cae224 to 2713913 Compare July 9, 2026 16:48

Copilot started reviewing on behalf of jorgemanrubia July 9, 2026 16:49 View session

jorgemanrubia added 3 commits July 9, 2026 18:50

Use if/else instead of early return in filter_boards

90e654e

Rename fans_out? to filters_by_association?

e629d41

Rename to joins_has_many?

33d98a2

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 16:54

Copilot started reviewing on behalf of jorgemanrubia July 9, 2026 16:54 View session

jorgemanrubia merged commit 9812211 into main Jul 9, 2026
14 checks passed

jorgemanrubia deleted the entropy-sweep-materialization branch July 9, 2026 16:56

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Speed up the card list for large accounts#2971

Speed up the card list for large accounts#2971
jorgemanrubia merged 4 commits into
mainfrom
entropy-sweep-materialization

jorgemanrubia commented Jul 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

jorgemanrubia Jul 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		assert_not_includes filter.cards, inaccessible_card
		filter.cards.each { \|card\| assert_includes users(:david).boards, card.board }

Uh oh!

Conversation

jorgemanrubia commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this fixes

Why it was slow

The fix

Why pin, not just the predicate

Before / after

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jorgemanrubia Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jorgemanrubia commented Jul 9, 2026 •

edited

Loading