Conversation
…hape Softmax of a length-1 logit was always 1.0, scoring every sigmoid detector as always-AI. Also stop guessing the first tuple/dict element and add a last-dim shape guard that fails loudly on ambiguous outputs.
…-abort on inference error Image inference errors previously propagated and aborted the whole dataset, letting a model drop hard samples by crashing. All three drivers now share run_batch_and_record(), which records per-sample errors instead.
refactor(benchmarks): unify batch inference/record + fix image dataset-abort
_download_filtered_sequential threaded seed into row sampling but shuffled shard order with the unseeded global RNG, so sample selection was non-reproducible even with a seed set.
Augmentation seeded the process-global np.random/random inside parallel prefetch workers, so concurrent threads clobbered each other's seed and broke the per-sample determinism the seed promises. Thread a per-call RandomState/random.Random through the transforms instead. Uses the legacy RNG algorithm so seeded outputs (and the aug cache) are byte-identical; verified 0/24 vs 24/24 non-deterministic under 8 threads before/after.
fix(transforms): thread-local RNG for augmentation determinism
Configure ruff (pyflakes/import/statement rules, blocking in CI) and mypy (advisory), add a dev extra and GitHub Actions workflow. Fix issues surfaced: a real bug where 'except (OSError, pyarrow.ArrowInvalid)' raised NameError (pyarrow was never imported, only pyarrow.parquet as pq), plus unused imports/vars, empty f-strings, and bare excepts.
chore: add ruff + mypy + CI tooling (and fix a pyarrow NameError)
Split the 1889-line dataset/download.py into a download/ package (constants, fetch, listing, extract, cache_io, core) with __init__ re-exporting the public API so all existing imports keep working. Pure code-move: all 31 defs preserved, verified via identical def inventory, full import graph, ruff, and the parquet test.
refactor(download): split the download.py god module into a package
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.8.0 (
0.7.5→0.8.0).A correctness, reproducibility, and maintainability pass over the benchmark. Public API is unchanged — no breaking changes for model submitters or callers.
Correctness (scoring)
process_model_outputsoftmaxed a length-1 logit, which is always1.0, so every sigmoid (1-output) detector was scored as "always AI-generated." Now applies sigmoid for single-output heads. (#—,775b934)PyTorchInferenceSession.runno longer grabs the first tuple/dict element blindly; it unwraps only unambiguous cases and raises a clear error otherwise, plus a last-dim shape guard against the declarednum_classes. (775b934)run_batch_and_recordthat records per-sample errors on inference failure, so a model can't exclude hard data by crashing. Also de-duplicates the three near-identical drivers. (refactor(benchmarks): unify batch inference/record + fix image dataset-abort #118)pyarrow.ArrowInvalidhandlers fixed.except (OSError, pyarrow.ArrowInvalid)raisedNameError(onlypyarrow.parquet as pqwas imported), so corrupt-parquet handling crashed instead of catching. (chore: add ruff + mypy + CI tooling (and fix a pyarrow NameError) #120)Reproducibility
RandomState/random.Random; outputs are byte-identical to before (aug cache stays valid). Verified 0/24 vs 24/24 non-deterministic under 8 threads. (fix(transforms): thread-local RNG for augmentation determinism #119)_download_filtered_sequentialshuffled shard order with the unseeded global RNG, making sample selection non-reproducible even with a seed set. (59ec197)Tooling / maintainability
.github/workflows/ci.yml): ruff (blocking) + mypy (advisory).devextra forpip install -e .[dev]. (chore: add ruff + mypy + CI tooling (and fix a pyarrow NameError) #120)download.py(1889 lines) split into a package —constants / fetch / listing / extract / cache_io / core— with the public API re-exported. Pure code-move, all 31 defs preserved. (refactor(download): split the download.py god module into a package #121)transforms.py(previously zero coverage); bareexcept:narrowed toexcept Exception:; dead code / unused imports cleaned.Notes for reviewers
test_config_loadingfailures are stale data-drift assertions (m4a/movare valid formats; hardcoded dataset counts) — not introduced here; a follow-up will make them robust and add a test job to CI.Commits:
775b934, #118,59ec197, #119, #120, #121, version bump.