Skip to content

Improve Rust reader throughput and resilience#7

Merged
xando merged 1 commit into
xando:masterfrom
sebastian-flyr:improve-rust-reader-performance
Jun 8, 2026
Merged

Improve Rust reader throughput and resilience#7
xando merged 1 commit into
xando:masterfrom
sebastian-flyr:improve-rust-reader-performance

Conversation

@sebastian-flyr

Copy link
Copy Markdown
Contributor

Summary

  • Increase the Rust BigQuery Storage reader decode limit to handle larger Arrow ReadRows messages.
  • Use one BigQuery Storage client/channel per read stream while sharing a fetched auth token.
  • Batch currently available Rust-decoded record batches per Python iterator step and pass batch_size into the Rust reader.
  • Reduce Rust stream fanout from worker_count * 3 to worker_count and retry transient ReadRows stream failures from the last decoded offset.

Test plan

  • cargo check --manifest-path /tmp/pyarrow-bigquery/rust/Cargo.toml
  • uv run --group dev ruff check .
  • Full BigQuery Rust load from consuming project: 695678 rows x 35 columns in 44.37s
  • Five repeated full BigQuery Rust loads from consuming project: 24.43s, 28.38s, 29.37s, 26.19s, 28.49s

Note: uv run --group dev pytest currently fails during collection because the package test venv cannot import pyarrow.bigquery under the installed pyarrow package layout. This appears to be an existing packaging/test-environment issue independent of these reader changes.

@xando
xando merged commit 24818cd into xando:master Jun 8, 2026
0 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants