v0.5.0 — robust remote caching, pandas 3.0 support, comprehensive tutorials & demo GIF

.github/AGENTS.md

@@ -0,0 +1,19 @@
+<!-- Parent: ../AGENTS.md -->
+<!-- Generated: 2026-07-03 | Updated: 2026-07-03 -->
+
+# .github
+
+## Purpose
+GitHub-specific configuration for the repository. Currently holds the continuous-integration workflow.
+
+## Subdirectories
+| Directory | Purpose |
+|-|-|
+| `workflows/` | GitHub Actions workflow definitions (see `workflows/AGENTS.md`) |
+
+## For AI Agents
+
+### Working In This Directory
+- Changes here affect CI behavior on push/PR. Validate YAML before committing.
+
+<!-- MANUAL: Any manually added notes below this line are preserved on regeneration -->

.github/workflows/AGENTS.md

@@ -0,0 +1,29 @@
+<!-- Parent: ../AGENTS.md -->
+<!-- Generated: 2026-07-03 | Updated: 2026-07-03 -->
+
+# workflows
+
+## Purpose
+GitHub Actions CI workflow definitions that run the test suite on every push and pull request that touches package/test/build files.
+
+## Key Files
+| File | Description |
+|-|-|
+| `ci.yaml` | Workflow `wrangler-dev`. Triggers on push/PR affecting `datawrangler/**`, `tests/**`, `requirements*.txt`, `setup.*`, `tox.ini`, `Makefile`, `MANIFEST.in`, or the workflows themselves. Matrix-tests Python 3.9-3.12 on `ubuntu-latest`: installs `requirements.txt` + `requirements_hf.txt`, `pip install -e .`, then runs `pytest`. Includes a guard to avoid duplicate runs on fork PRs. |
+
+## For AI Agents
+
+### Working In This Directory
+- CI installs the **`[hf]`** dependencies too, so text/NLP tests run in CI and can download models — keep them deterministic enough to pass on a clean runner.
+- If you add a Python version, a new top-level config file, or a new dependency file, update both the `matrix` and the `paths` triggers here.
+- Local pre-push parity: run `make test`, `make lint`, and `make docs` before pushing (repo `CLAUDE.md`).
+
+### Testing Requirements
+- The workflow's job is to run `pytest`. It does not currently run flake8 or the docs build — do those locally.
+
+## Dependencies
+
+### External
+- GitHub Actions: `actions/checkout@v4`, `actions/setup-python@v4`
+
+<!-- MANUAL: Any manually added notes below this line are preserved on regeneration -->

.gitignore

@@ -135,4 +135,6 @@ dmypy.json
 */.idea/*
 *.idea/*
 
-.vscode/*
+.vscode/*
+# oh-my-claudecode tooling state
+.omc/

AGENTS.md

@@ -0,0 +1,60 @@
+<!-- Generated: 2026-07-03 | Updated: 2026-07-03 -->
+
+# data-wrangler
+
+## Purpose
+`data-wrangler` (published on PyPI as `pydata-wrangler`, imported as `datawrangler` / `dw`) is a Python package that turns messy data into clean `DataFrame` objects (or lists of them). It auto-detects data types (arrays, text, DataFrames, files, URLs, null/empty) and converts them into a consistent DataFrame format, with a special emphasis on text/NLP embedding via scikit-learn and sentence-transformers. As of v0.4.0 it supports **dual backends**: pandas (default) and Polars (2-100x faster on large data) selectable with `backend='polars'`.
+
+## Key Files
+| File | Description |
+|-|-|
+| `setup.py` | Package metadata; name `pydata-wrangler`, version `0.4.0`, extras `[hf]` (HuggingFace) and `[dev]` |
+| `requirements.txt` | Core deps: pandas, polars>=0.20, numpy, scipy, scikit-learn, matplotlib, requests, six, dill, Pillow, tqdm |
+| `requirements_hf.txt` | Optional NLP deps: torch, transformers, sentence-transformers, datasets, tokenizers, sentencepiece |
+| `requirements_dev.txt` | Dev/test deps |
+| `Makefile` | Dev tasks: `make test`, `make lint`, `make coverage`, `make docs`, `make dist`, `make install` |
+| `tox.ini` | Multi-version test matrix (Python 3.9-3.12) |
+| `setup.cfg` | flake8 / tooling config |
+| `dev.yaml` | Conda environment for full ML setup (`conda env create -f dev.yaml`) |
+| `HISTORY.rst` | Changelog |
+| `README.rst` | Overview and quick-start examples |
+| `CLAUDE.md` | Repo-specific guidance for AI agents |
+| `.readthedocs.yaml` | ReadTheDocs build config |
+
+## Subdirectories
+| Directory | Purpose |
+|-|-|
+| `datawrangler/` | The Python package source (see `datawrangler/AGENTS.md`) |
+| `tests/` | pytest suite + sample resources (see `tests/AGENTS.md`) |
+| `docs/` | Sphinx documentation and Jupyter tutorials (see `docs/AGENTS.md`) |
+| `benchmarks/` | Standalone pandas-vs-Polars and import-time benchmarks (see `benchmarks/AGENTS.md`) |
+| `.github/` | GitHub Actions CI configuration (see `.github/AGENTS.md`) |
+| `notes/` | Session/handoff notes (working documents, not part of the package) |
+| `build/`, `dist/`, `pydata_wrangler.egg-info/` | Build artifacts — do not edit by hand |
+
+## For AI Agents
+
+### Working In This Directory
+- The public API surface is intentionally tiny: `dw.wrangle`, `dw.funnel`, `dw.stack`, `dw.unstack`, `dw.__version__`. Preserve these entry points.
+- Version lives in `datawrangler/core/configurator.py` (`__version__`) **and** `setup.py`. Keep them in sync on release.
+- Model/vectorizer/imputer defaults live in `datawrangler/core/config.ini`, not in code. Change behavior there when possible.
+- Per repo `CLAUDE.md`: update docs when changing tests/examples, update `requirements*.txt` when changing deps, and re-run the full check suite (tests + lint + docs) after any fix.
+
+### Testing Requirements
+- Run `make test` (pytest). Tests use **real** models, files, and network calls — never mocks. Many tests are parameterized over both `pandas` and `polars` backends via the `backend` fixture.
+- Run `make lint` (flake8) and `make docs` (Sphinx build) before pushing. If any check triggers a change, re-run all checks.
+
+### Common Patterns
+- **Plugin/priority dispatch**: `zoo/format.py` iterates `format_checkers = ['dataframe', 'text', 'array', 'null']` and calls the first matching `is_<type>` / `wrangle_<type>` pair.
+- **Config-driven defaults**: `core/config.ini` + `apply_defaults` inject defaults by function/class name.
+- **Lazy imports**: heavy deps (torch, transformers, sklearn submodules, polars) load on first use via `util/lazy_imports.py` to keep import time low.
+
+## Dependencies
+
+### External
+- pandas, numpy, scipy, scikit-learn, matplotlib (core)
+- polars>=0.20.0 (required; high-performance backend)
+- requests, dill, Pillow, six, tqdm (I/O and utilities)
+- torch, transformers, sentence-transformers, datasets (optional, via `[hf]` extra)
+
+<!-- MANUAL: Any manually added notes below this line are preserved on regeneration -->

CLAUDE.md

@@ -3,55 +3,52 @@
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
 ## Project Overview
-**data-wrangler** is a Python package that transforms messy data into clean pandas DataFrames, with emphasis on text data and NLP. Published as `pydata-wrangler` on PyPI.
+**data-wrangler** turns messy data (arrays, text, DataFrames, files, URLs, null/empty, and nested lists of these) into clean `DataFrame` objects, with a special emphasis on text/NLP embedding. Published on PyPI as `pydata-wrangler`, imported as `datawrangler` / `dw`. As of v0.4.0 every operation supports **two backends**: pandas (default) and Polars (`backend='polars'`, 2-100x faster on large data).
 
-## Development Commands
+Per-directory `AGENTS.md` files exist throughout the tree with deeper, localized notes — consult the one nearest the code you're editing.
 
-### Core Development Tasks
-- `make test` - Run pytest test suite
-- `make lint` - Run flake8 code style checks  
-- `make coverage` - Generate test coverage reports
-- `make docs` - Build Sphinx documentation
-- `pytest tests/wrangler/test_specific.py::test_function` - Run specific test
+## Development Commands
+- `make test` — run the pytest suite (default Python)
+- `pytest tests/wrangler/test_zoo.py::test_name` — run a single test
+- `make test-all` — tox across Python 3.9-3.12
+- `make lint` — flake8
+- `make coverage` — coverage report
+- `make docs` — build Sphinx docs (also runs the tutorial notebooks)
+- `make dist` / `make install` / `make clean` — build / install / clean artifacts
 
-### Build and Release
-- `make clean` - Clean all build artifacts
-- `make dist` - Build source and wheel packages
-- `make install` - Install package locally
+Full dev install (text/NLP tests need the `[hf]` extra): `pip install -e ".[hf]"`, or `conda env create -f dev.yaml`.
 
-### Testing
-- `make test-all` - Run tox across Python 3.9-3.12
-- Tests located in `/tests/wrangler/` with sample data in `/tests/resources/`
+**Tests use real resources, not mocks**: they download sklearn corpora and sentence-transformers models and fetch remote fixtures over HTTP, so a full run needs network access. Most DataFrame-producing tests are parameterized over both backends via the `backend` fixture in `tests/wrangler/conftest.py` (use its `assert_backend_type` / `assert_dataframes_equivalent` helpers).
 
 ## Architecture
 
-### Core Package Structure
-- `datawrangler/core/` - Configuration management via config.ini
-- `datawrangler/decorate/` - Function decorators (especially `@funnel`)
-- `datawrangler/io/` - File/URL loading with format auto-detection  
-- `datawrangler/util/` - Helper utilities
-- `datawrangler/zoo/` - Data type handlers (array, dataframe, text, null)
+The package is essentially **one priority-ordered dispatch loop** wrapped in decorators and fed by an I/O layer.
 
-### Key Patterns
-1. **Plugin Architecture**: `zoo/format.py` orchestrates priority-based data type detection using `is_<type>` and `wrangle_<type>` functions
-2. **Decorator Pattern**: `@funnel` automatically converts inputs to DataFrames for function compatibility
-3. **Configuration-Driven**: Uses `core/config.ini` for ML model defaults and processing options
-4. **Extensible I/O**: Supports files, URLs, multiple formats with automatic detection
+- **Dispatch (`zoo/format.py`)**: `wrangle(x, backend=None, **kwargs)` reads the ordered list `format_checkers = ['dataframe','text','array','null']` from `config.ini`, then calls the first matching `is_<type>(x)` and runs `wrangle_<type>(x, ...)`. **Order is priority** — earlier checkers win. Per-type options are passed as `<type>_kwargs` (e.g. `text_kwargs={...}`). For list inputs, a fitted model from the first element is reused across the rest.
 
-### Main Entry Points
-- `datawrangler.wrangle()` - Primary data transformation API
-- `datawrangler.decorate.funnel` - Function decorator
-- `datawrangler.io.load()` and `datawrangler.io.save()` - I/O operations
+- **The wrangler contract** (shared by every `wrangle_<type>` in `zoo/`, and relied on by `decorate/`): each accepts `return_model=False` and `backend=None`. When `return_model=True` it returns `(df, model)` where `model` is a reusable `{'model', 'args', 'kwargs'}` dict — pass it back later (e.g. `text_kwargs={'model': fitted}`) to apply the same fitted transform to new data.
 
-## Dependencies
-- **Core**: pandas, numpy, scipy, scikit-learn, matplotlib
-- **Optional ML**: PyTorch, HuggingFace transformers, sentence-transformers (in `requirements_hf.txt`)
-- **Dev**: pytest, Sphinx, flake8 (in `requirements_dev.txt`)
+- **Dual backend**: the `backend` argument threads from `wrangle` into every wrangler. pandas↔Polars conversion and Polars detection/handling live in `zoo/polars_dataframe.py` (`pandas_to_polars`, `polars_to_pandas`, `is_polars_dataframe`, `create_polars_dataframe`). A process-global default lives in `core/configurator.py` (`set_dataframe_backend` / `get_dataframe_backend` / `reset_dataframe_backend`). Predicate checks in `util/helpers.py` (`array_like`, `dataframe_like`) are **duck-typed by attribute**, not `isinstance`, so pandas- and Polars-like objects both pass.
+
+- **Config-as-code (`core/config.ini` + `core/configurator.py`)**: model/vectorizer/imputer defaults, the format-checker order, and the data-cache path all live in `config.ini`. **Its values are Python expressions that get `eval()`-ed** (e.g. `np.nan`, list literals, `os.getenv('HOME')`). `apply_defaults` injects a section's defaults into a function/class **by its `__name__`**; keys prefixed with `__` become positional args, others become keyword args.
+
+- **Lazy imports (`util/lazy_imports.py`)**: heavy deps (torch, transformers, sentence-transformers, sklearn submodules, polars) load on first use via cached `get_*` importers, keeping `import datawrangler` fast (see `benchmarks/import_time.py`). Add new heavy dependencies as `get_*` lazy importers here — never as top-level imports.
+
+- **I/O (`io/`)**: `load`/`save` auto-detect format by extension; remote URLs are content-hashed (`blake2b`) and cached under `~/.datawrangler/data/` (created on import). `panda_handler.load_dataframe` maps extensions to `pandas.read_*`.
+
+- **Decorators (`decorate/decorate.py`)**: `@funnel` runs `zoo.wrangle` on a function's input so the function can assume clean DataFrame input; `interpolate` fills missing values using imputer/interpolation defaults from `config.ini`; `pandas_stack`/`pandas_unstack` (exported as `dw.stack`/`dw.unstack`) concat/split lists of DataFrames via a MultiIndex.
 
 ## Adding New Data Types
-Implement two functions in the appropriate `zoo/` module:
-- `is_<datatype>(obj)` - Type detection function
-- `wrangle_<datatype>(obj, **kwargs)` - Conversion to DataFrame
+1. Implement `is_<type>(obj)` and `wrangle_<type>(obj, return_model=False, backend=None, **kwargs)` in a `zoo/` module (honor the wrangler contract above).
+2. Export both in `zoo/__init__.py`.
+3. Register `<type>` in `supported_formats.types` in `core/config.ini` at the correct **priority position**.
 
-## Development Environment
-Use `conda env create -f dev.yaml` for full ML environment setup.
+## Gotchas
+- `__version__` lives in **both** `core/configurator.py` and `setup.py` — keep them in sync on release.
+- HuggingFace corpora in `zoo/text.py`/`get_corpus` must use full `namespace/name` ids (e.g. `wikimedia/wikipedia`, `cam-cst/cbt`); bare legacy names (`wikipedia`, `cbt`) break on `datasets` >= 4.
+- `array_like('some string')` will try to `load()` it as a file/URL (network/FS side-effect) unless `force_literal=True`.
+
+## Dependencies
+- **Core**: pandas, numpy, scipy, scikit-learn, matplotlib, polars>=0.20 (required), requests, dill, Pillow, six, tqdm
+- **Optional NLP (`[hf]` extra / `requirements_hf.txt`)**: torch, transformers, sentence-transformers, datasets, tokenizers
+- **Dev (`requirements_dev.txt`)**: pytest, Sphinx, flake8

HISTORY.rst

@@ -2,6 +2,38 @@
 History
 =======
 
+0.5.0 (2026-07-03)
+------------------
+
+**Bug-fix release: robust remote caching + correctness fixes**
+
+**Fixed: remote file caching (the headline fix):**
+
+* ``get_extension`` now strips URL query strings (``?dl=1``) and fragments (``#...``) before
+  detecting the file type. Previously, downloading a URL like ``https://.../data.npz?dl=1``
+  (Dropbox / Google Drive share links, including the built-in text corpora) cached the file under a
+  polluted name (``<hash>.npz?dl=1``). That cached copy could not be re-read by extension
+  (``ValueError: Unknown datatype: npz?dl=1``), so the dataset was effectively re-downloaded /
+  re-cached instead of reused. Remote files now cache under a clean, stable name and re-loads reuse
+  the cached copy with no additional download.
+
+**Other correctness fixes:**
+
+* ``core.configurator.apply_defaults`` no longer raises ``KeyError`` for functions/classes that have
+  no section in ``config.ini`` (this had broken sklearn models such as ``TSNE``, ``MDS``, ``Isomap``).
+* ``decorate.apply_stacked`` no longer crashes with ``TypeError`` when the wrapped function returns a
+  ``(data, model)`` tuple (``return_model=True``) on unstacked input.
+* ``zoo.is_multiindex_dataframe`` no longer raises ``AttributeError`` on Polars DataFrames/LazyFrames
+  (which have no ``.index``).
+* ``io.load`` now correctly unwraps a single-array ``.npz`` file to the underlying array.
+* Fixed missing f-string prefixes in two ``zoo.text`` error messages.
+
+**Tests & docs:**
+
+* Added regression tests for the caching fix and deeper tests for the ``return_model`` contract, the
+  Polars helpers, backend configuration, and ``io.save`` round-trips.
+* Corrected stale version strings and API-reference stubs (backend functions, the Polars module).
+
 0.4.0 (2025-06-14)
 ------------------
 
@@ -100,7 +132,7 @@ This release brings full compatibility with NumPy 2.0+ and pandas 2.0+ while mod
 * Enhanced error handling for missing dependencies
 
 **Bug Fixes:**
-* Fixed numpy.str_ deprecation that broke in NumPy 2.0+
+* Fixed ``numpy.str_`` deprecation that broke in NumPy 2.0+
 * Updated HuggingFace datasets import for API changes
 * Fixed sklearn IterativeImputer experimental import compatibility
 * Replaced deprecated matplotlib.pyplot.imread

README.rst

@@ -15,9 +15,13 @@ list of ``DataFrame`` objects.  The package provides code for easily wrangling d
 ``DataFrame`` objects, manipulating ``DataFrame`` objects in useful ways (that can be tricky to implement, but that
 apply to many analysis scenarios), and decorating Python functions to make them more flexible and/or easier to write.
 
-🚀 **New**: ``data-wrangler`` now supports **high-performance Polars DataFrames** alongside pandas, delivering 2-100x speedups 
+🚀 **New**: ``data-wrangler`` now supports **high-performance Polars DataFrames** alongside pandas, delivering 2-100x speedups
 for large datasets with zero code changes. Simply add ``backend='polars'`` to any operation!
 
+.. image:: https://raw.githubusercontent.com/ContextLab/data-wrangler/main/docs/images/demo.gif
+   :alt: data-wrangler live demo
+   :align: center
+
 The ``data-wrangler`` package supports a variety of datatypes.  There is a special emphasis on text data, whereby
 ``data-wrangler`` provides a simple API for interacting with natural language processing tools and datasets provided by
 ``scikit-learn`` and ``hugging-face`` (via sentence-transformers).  The package is designed to provide sensible defaults, but also
@@ -103,6 +107,7 @@ Currently supported datatypes are limited to:
   - ``array``-like objects (including images)
   - ``DataFrame``-like or ``Series``-like objects (pandas and Polars)
   - text data (text is embedded using natural language processing models)
+
 or lists of mixtures of the above.
 
 **Backend Support**: All operations support both ``pandas`` (default) and ``Polars`` (high-performance) backends. Choose the backend that best fits your performance requirements and workflow preferences.

benchmarks/AGENTS.md

@@ -0,0 +1,35 @@
+<!-- Parent: ../AGENTS.md -->
+<!-- Generated: 2026-07-03 | Updated: 2026-07-03 -->
+
+# benchmarks
+
+## Purpose
+Standalone performance scripts that quantify the two headline claims of v0.4.0: the pandas-vs-Polars speedup and the reduced import time from lazy loading. These are runnable scripts, not part of the installed package or the pytest suite.
+
+## Key Files
+| File | Description |
+|-|-|
+| `dataframe_performance.py` | Times `dw.wrangle` (and DataFrame ops) across the pandas and Polars backends over varying data sizes to demonstrate the 2-100x speedup |
+| `import_time.py` | Measures `import datawrangler` startup time (via subprocess + `statistics`) to validate the lazy-import optimization |
+
+## For AI Agents
+
+### Working In This Directory
+- Run directly, e.g. `python benchmarks/dataframe_performance.py`. Each script inserts the repo root on `sys.path`, so it runs against the working-tree source without installation.
+- These scripts print human-readable timing tables; they are not asserted in CI. If you change import structure or backend dispatch, re-run them to confirm the performance claims still hold.
+
+### Testing Requirements
+- No automated assertions. Correctness here means the scripts run cleanly and report sensible numbers on the current machine.
+
+### Common Patterns
+- Timing via `time` / `statistics`; import-time measured in a fresh subprocess to avoid caching effects.
+
+## Dependencies
+
+### Internal
+- `datawrangler` (imported for benchmarking)
+
+### External
+- pandas, polars, numpy
+
+<!-- MANUAL: Any manually added notes below this line are preserved on regeneration -->