Skip to content

Drop global Rust reader runtime#8

Merged
xando merged 1 commit into
xando:masterfrom
sebastian-flyr:drop-global-rust-runtime
Jun 9, 2026
Merged

Drop global Rust reader runtime#8
xando merged 1 commit into
xando:masterfrom
sebastian-flyr:drop-global-rust-runtime

Conversation

@sebastian-flyr

Copy link
Copy Markdown
Contributor

Summary

  • Replace the process-global Tokio runtime with a runtime owned by each Rust PyReader.
  • Let Rust reader runtime/thread state be dropped with the reader instead of persisting for the Python process lifetime.
  • Avoid leaving native runtime threads around after an engine=\"rust\" read, which makes mixed engine usage safer for callers that later use the Python multiprocessing path.

Test plan

  • cargo check --manifest-path /tmp/pyarrow-bigquery/rust/Cargo.toml
  • Installed local package into the consuming pre-dqn project and ran same-process mixed engine benchmarks.
  • Mixed benchmark no longer shows the previous Python-after-Rust degradation from ~400-500s; Python-after-Rust now matches Python-only repeated-read behavior around ~102-105s.
  • Rust-only repeated reads stayed stable around ~24-29s.

@xando
xando merged commit 5d0c648 into xando:master Jun 9, 2026
0 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants