Skip to content

chore: add ruff + mypy + CI tooling (and fix a pyarrow NameError)#120

Merged
dylanuys merged 2 commits into
devfrom
chore/lint-typecheck-ci
Jul 13, 2026
Merged

chore: add ruff + mypy + CI tooling (and fix a pyarrow NameError)#120
dylanuys merged 2 commits into
devfrom
chore/lint-typecheck-ci

Conversation

@dylanuys

Copy link
Copy Markdown
Contributor

Sets up linting, type-checking, and CI — and fixes the real bug the linter immediately surfaced.

Real bug found by linting

download.py caught except (OSError, pyarrow.ArrowInvalid) in 5 places, but only import pyarrow.parquet as pq was present — pyarrow was never bound. So whenever a corrupt parquet raised ArrowInvalid, the handler itself raised NameError instead of catching it. Fixed by importing pyarrow.

Tooling

  • ruff ([tool.ruff]): baseline ruleset E4,E7,E9,F — pyflakes (real bugs) plus import/statement errors. Style rules left off for now to keep the diff focused. Blocking in CI.
  • mypy ([tool.mypy]): lenient config (ignore_missing_imports). Advisory in CI (continue-on-error) — the codebase is ~30% typed, so it reports without blocking until coverage improves.
  • dev extra: pip install -e .[dev] now pulls pytest/ruff/mypy (previously no test dep was declared anywhere).
  • CI (.github/workflows/ci.yml): lint (blocking) + typecheck (advisory) on push/PR to dev/main.

Lint fixes applied to reach a clean baseline

  • Real bug: pyarrow import (above).
  • Bare except:except Exception: (4 sites in archive.py/media.py — they were swallowing KeyboardInterrupt/SystemExit).
  • Removed unused imports (16), empty f-strings (27), redundant local imports, and dead local variables (incl. 3 ignored df = await run_*_benchmark(...) captures).
  • Two x = lambdadef in download.py.

All mechanical; no behavior change except the pyarrow fix and the bare-except narrowing.

Verification

  • ruff check src tests → clean.
  • Package imports; pyarrow.ArrowInvalid now resolves.
  • Unit suite unchanged: 39 pass, same 5 pre-existing test_config_loading failures.

Deliberately out of scope (next step)

No test job in CI yet, for two reasons that need your call:

  1. The 5 failing test_config_loading tests are stale/brittle — m4a and mov are valid formats (they're in download.py's extension sets), and the count/membership tests hardcode values that drift on every data change. Fixing them means deciding what they should assert.
  2. A test job pulls the full heavy dep set (torch, transformers, onnxruntime, …) — a CI time/cost decision.

Happy to follow up: fix the stale tests (derive format validity from the real extension constants; make the count tests robust) and add a blocking test job.

dylanuys added 2 commits July 11, 2026 09:28
Configure ruff (pyflakes/import/statement rules, blocking in CI) and mypy
(advisory), add a dev extra and GitHub Actions workflow. Fix issues surfaced:
a real bug where 'except (OSError, pyarrow.ArrowInvalid)' raised NameError
(pyarrow was never imported, only pyarrow.parquet as pq), plus unused
imports/vars, empty f-strings, and bare excepts.
@dylanuys
dylanuys merged commit abdadc0 into dev Jul 13, 2026
2 checks passed
@dylanuys dylanuys mentioned this pull request Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant