Skip to content

refactor(download): split the download.py god module into a package#121

Merged
dylanuys merged 1 commit into
devfrom
refactor/download-package
Jul 13, 2026
Merged

refactor(download): split the download.py god module into a package#121
dylanuys merged 1 commit into
devfrom
refactor/download-package

Conversation

@dylanuys

Copy link
Copy Markdown
Contributor

Splits the 1889-line dataset/download.py into a cohesive download/ package. Pure code-move — no logic changes.

Structure

constants (errors, extensions, archive-type predicates) ← fetch (HTTP/HF downloading) / listing (remote enumeration + URL resolution) / extract (parquet/tar/zip/raw decode) / cache_io (persistent-cache load) ← core (download_and_extract orchestration, incl. gasstation).

download/__init__.py re-exports the public API, so every existing from ...dataset.download import X keeps working unchanged (verified against all call sites: iterator.py, gasstation_utils.py, dataset/__init__.py, and the parquet test).

How it was done (low-risk)

  • Function bodies carved by exact line range (sed), so they are byte-identical to the original — no hand-editing of logic.
  • Cross-module imports were driven by ruff: F821 (undefined-name) told me exactly which sibling each module needed; F401 --fix trimmed the shared header down per module.
  • Dependency graph is a clean DAG (no import cycles): constants ← {fetch, listing, extract, cache_io} ← core.

Verification

  • Def/class inventory identical: 31 defs in, 31 out, diff empty — nothing dropped or duplicated.
  • Full import graph loads (no circular imports); public API + external late-imports resolve.
  • ruff check src tests clean.
  • test_parquet_processing.py: 12/12 pass. Full unit suite: 46 pass, same 5 pre-existing test_config_loading data-drift failures (unrelated).

Note

The _process_*/list_* helper names (incl. the leading-underscore ones) are preserved verbatim to avoid touching call sites; a follow-up could tidy naming/visibility now that they're grouped, but that would no longer be a pure move.

Split the 1889-line dataset/download.py into a download/ package
(constants, fetch, listing, extract, cache_io, core) with __init__
re-exporting the public API so all existing imports keep working.
Pure code-move: all 31 defs preserved, verified via identical def
inventory, full import graph, ruff, and the parquet test.
@dylanuys
dylanuys merged commit 71c006a into dev Jul 13, 2026
2 checks passed
@dylanuys dylanuys mentioned this pull request Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant