OCHA-DAP · mcarans · Jun 11, 2026 · Jun 11, 2026
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,5 +1,5 @@
 default_language_version:
-  python: python3.13
+  python: python3
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,81 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+**hdx-python-utilities** is a general-purpose Python utility library (not HDX-specific) providing helpers for: downloading files, retrieving tabular data (CSV/JSON/XLSX), date parsing, JSON/YAML I/O, dict/list manipulation, HTML parsing, file hashing, text processing, error handling, and more.
+
+## Commands
+
+```bash
+# Install dependencies
+uv sync
+
+# Run all tests with coverage
+uv run pytest
+
+# Run a single test file
+uv run pytest tests/hdx/utilities/test_downloader.py
+
+# Run a single test by name
+uv run pytest tests/hdx/utilities/test_downloader.py -k "test_download_file"
+
+# Lint and format
+uv run ruff check
+pre-commit run --all-files
+
+# Build
+uv build
+```
+
+Coverage is written to `coverage.lcov` and JUnit XML to `test-results.xml`.
+
+## Source Layout
+
+All source lives under `src/hdx/utilities/`. Tests mirror the layout under `tests/hdx/utilities/` with fixtures in `tests/fixtures/`.
+
+Key modules:
+- `base_downloader.py` — `BaseDownload` ABC defining the interface (`download_file`, `download_tabular_rows`, etc.) and `DownloadError`
+- `downloader.py` — `Download(BaseDownload)`: HTTP downloading via `requests` + `frictionless` for tabular parsing; supports auth (basic, bearer, extra params), rate limiting, retries, and streaming rows
+- `retriever.py` — `Retrieve(BaseDownload)`: wraps a `Download`, adding save/load-from-disk and static fallback directories; registered instances in `Retrieve.retrievers` dict
+- `session.py` — `get_session()` builds a `requests.Session` with retry logic and auth headers
+- `frictionless_wrapper.py` — thin wrapper around `frictionless.TableResource` for tabular reads
+- `loader.py` / `saver.py` — JSON and YAML I/O (YAML uses `ruamel.yaml` to preserve order/comments)
+- `dateparse.py` — date/time parsing and timezone utilities
+- `dictandlist.py` — dict/list helpers (merge, flatten, key renaming, etc.)
+- `path.py` — `script_dir()`, temp dir helpers, `NotFoundError`
+- `error_handler.py` / `errors_onexit.py` — structured error accumulation and on-exit reporting
+- `text.py` — text processing including `smart_split`
+- `matching.py` — fuzzy/phonetic matching utilities
+- `state.py` — lightweight on-disk state persistence
+- `useragent.py` — user-agent string construction (reads `~/.useragent.yaml` by default)
+
+## Key Patterns
+
+**Download / Retrieve layering** — `Download` handles raw HTTP; `Retrieve` adds a save/use_saved/fallback layer on top. Callers typically use `Retrieve` rather than `Download` directly. Both implement `BaseDownload` so they're interchangeable.
+
+**Tabular row iteration** — `download_tabular_rows()` (on both `Download` and `Retrieve`) yields dicts via `frictionless.TableResource`. The `frictionless_wrapper.py` module handles the `FrictionlessException` boundary. Supports CSV, JSON, XLSX, and XLS.
+
+**User-agent requirement** — `Download` requires a user agent. Either set a global user agent via `UserAgent.set_global(...)` or pass `user_agent=` directly. Missing user agent raises at construction time.
+
+**Context manager usage** — `Download` and `Retrieve` are designed as context managers (`with Download(...) as downloader:`). `__exit__` closes the session.
+
+**Auth precedence** — environment variables `EXTRA_PARAMS`, `BASIC_AUTH`, `BEARER_TOKEN` override constructor arguments. Among constructor args: `auth` > `basic_auth`/`basic_auth_file` > `bearer_token`/`bearer_token_file` > `extra_params_dict` > `extra_params_json`/`extra_params_yaml`.
+
+## Code Style
+
+- Python ≥ 3.10; type hints throughout using `X | Y` union syntax (PEP 604)
+- Google-style docstrings with `Args:` and `Returns:` sections
+- Formatted and linted with `ruff` (rules: E, F, I, UP; E501 ignored)
+- No inline comments unless the *why* is non-obvious
+
+## Collaboration Style
+
+- Be objective, not agreeable. Push back when you disagree, flag tradeoffs honestly.
+- Keep explanations brief and to the point.
+- Don't rely on recalled knowledge for facts that could be stale. Read the actual source first.
+
+## Scope of Changes
+
+When fixing a bug or addressing PR feedback, change only what is necessary to resolve the specific issue. Do not refactor surrounding code, rename variables, adjust formatting, or make improvements in the same commit unless they are directly required by the fix.
diff --git a/README.md b/README.md
@@ -10,7 +10,6 @@ Note that these are not specific to HDX.
 1. Retrieval of data from url with saving to file or from data previously saved
 1. Date utilities
 1. Loading and saving JSON and YAML (maintaining order)
-1. Loading and saving HXLated csv and/or JSON
 1. Dictionary and list utilities
 1. HTML utilities (inc. BeautifulSoup helper)
 1. Compare files (eg. for testing)

diff --git a/documentation/index.md b/documentation/index.md
@@ -9,7 +9,6 @@ Python developers. Note that these are not specific to HDX.
 1. [Retrieval of data from url with saving to file or from data previously saved](#retrieving-files)
 1. [Date utilities](#date-utilities)
 1. [Loading and saving JSON and YAML (maintaining order)](#loading-and-saving-json-and-yaml)
-1. [Loading and saving HXLated csv and/or JSON](#loading-and-saving-hxlated-csv-andor-json)
 1. [Dictionary and list utilities](#dictionary-and-list-utilities)
 1. [HTML utilities (inc. BeautifulSoup helper)](#html-utilities)
 1. [Compare files (eg. for testing)](#comparing-files)
@@ -34,6 +33,9 @@ The code for the library is [here](https://github.com/OCHA-DAP/hdx-python-utilit
 The library has detailed API documentation which can be found in the menu at the top.
 
 ## Breaking Changes
+From 4.1.0, save_hxlated_output and Download.hxl_row removed. Parameter use_hxl removed
+from all calls in Download and Retrieve.
+
 From 4.0.4, get_filename_from_url, get_filename_extension_from_url, get_path_for_url,
 get_url_for_get, get_url_params_for_post moved to module url
 
@@ -185,9 +187,6 @@ Other useful functions:
     headers, iterator = downloader.get_tabular_rows_as_list(url)
     for row in iterator:
         ...
-    # Get hxl row
-    assert Download.hxl_row(["a", "b", "c"], {"b": "#b", "c": "#c"}, dict_form=True)
-    # == {"a": "", "b": "#b", "c": "#c"}
     # Build get url from url and dictionary of parameters
     Download.get_url_for_get("http://www.lala.com/hdfa?a=3&b=4",
                              OrderedDict([("c", "e"), ("d", "f")]))
@@ -461,93 +460,6 @@ as a csv or Excel.
 
     save_iterable(xlfilepath, list_of_lists, headers=["h1", "h2", "h3", "h4"], format="xlsx")
 
-## Loading and saving HXLated csv and/or JSON
-
-`save_hxlated_output` is a utility to save HXLated output (currently JSON and/or csv are
-supported) based on a given configuration. Here is an example YAML configuration:
-
-    input:
-      headers:
-        - "Col1"
-        - "Col2"
-        - "Col3"
-      hxltags:
-        - "#tag1"
-        - "#tag2"
-        - "#tag3"
-    process:
-      - header: "tag4"
-        hxltag: "#tag4"
-        expression: "#tag1 * 10"
-    output:
-      csv:
-        filename: "out.csv"
-        hxltags:
-          - "#tag2"
-          - "#tag3"
-      json:
-        filename: "out.json"
-        data: "results"
-        metadata:
-          "#date": "{{today}}"
-          "#mytag": 123
-        hxltags:
-          - "#tag1"
-          - "#tag2"
-
-The `input` section is needed if the rows of data that are passed in are missing either
-headers or HXL hashtags. The `output` section defines what files will be created. If
-`hxltags` are specified, then only those columns are output. CSV output would look like
-this:
-
-    Col2,Col3,tag4
-    #tag2,#tag3,#tag4
-    2,3,10
-    5,6,40
-
-
-For JSON output, if no `metadata` or `data` is specified, the output will look like
-this:
-
-    [
-    {"#tag1":1,"#tag2":"2","#tag4":10},
-    {"#tag1":4,"#tag2":"5","#tag4":40}
-    ]
-
-If only `metadata` was specified, not `data`, then output is like this:
-
-    {"metadata":{"#date":"today!","#mytag":123},"data":[
-    {"#tag1":1,"#tag2":"2","#tag4":10},
-    {"#tag1":4,"#tag2":"5","#tag4":40}
-    ]}
-
-Otherwise, the result is like this:
-
-    {"metadata":{"#date":"today!","#mytag":123},"results":[
-    {"#tag1":1,"#tag2":"2","#tag4":10},
-    {"#tag1":4,"#tag2":"5","#tag4":40}
-    ]}
-
-The utility is called as follows:
-
-    save_hxlated_output(
-        configuration,
-        rows,
-        includes_header=True,
-        includes_hxltags=True,
-        output_dir=output_dir,
-        today="today!",
-    )
-
-The first parameter is the configuration which can come from a YAML file for example.
-The second parameter, `rows` is the data. That data can be a list of lists, tuples or
-dictionaries. If `includes_header` is `True`, headers are taken from `rows`, otherwise
-they must be given by the configuration. If `includes_hxltags` is `True`, HXL hashtags
-are taken from `rows`, otherwise they must be given by the configuration. `output_dir`
-specifies where the output should go and defaults to "". Any other parameters (such as
-`today` in the example above) are used to populate template variables given in the
-configuration for the metadata.
-
 ## Dictionary and list utilities
 
 Examples:

diff --git a/pyproject.toml b/pyproject.toml
@@ -39,14 +39,14 @@ requires-python = ">=3.10"
 # Extras for frictionless[excel,json] added explicitly
 # for conda-forge compatibility
 dependencies = [
-    "frictionless>=5.18.0",
+    "frictionless>=5.19.0",
     # frictionless[excel]
     "openpyxl>=3.1.2",
     "tableschema-to-template>=0.0.13",
     "xlrd>=2.0.1",
     "xlwt>=1.3.0",
     # frictionless[json]
-    "ijson>=3.2.3",
+    "ijson>=3.5.0",
     "jsonlines>=4.0.0",
     # /end frictionless extras
     "typing_extensions",
@@ -56,7 +56,7 @@ dependencies = [
     "ratelimit",
     "requests-file",
     "ruamel.yaml",
-    "xlsx2csv",
+    "xlsx2csv==0.8.4",
 ]
 
 [project.optional-dependencies]

diff --git a/src/hdx/utilities/base_downloader.py b/src/hdx/utilities/base_downloader.py
@@ -90,7 +90,6 @@ def download_json(self, url: Path | str, *args: Any, **kwargs: Any) -> Any:
     def get_tabular_rows(
         self,
         url: Path | str | Sequence[str],
-        has_hxl: bool = False,
         headers: int | Sequence[int] | Sequence[str] = 1,
         dict_form: bool = False,
         *args: Any,
@@ -100,8 +99,6 @@ def get_tabular_rows(
         where each row is returned as a list or dictionary depending on the
         dict_rows argument.
 
-        When a list of urls is supplied (in url), then the has_hxl flag indicates if the
-        files are HXLated so that the HXL row is only included from the first file.
         The headers argument is either a row number or list of row numbers (in case of
         multi-line headers) to be considered as headers (rows start counting at 1), or
         the actual headers defined as a list of strings. It defaults to 1.
@@ -110,7 +107,6 @@ def get_tabular_rows(
 
         Args:
             url: A single or list of URLs or paths to read from
-            has_hxl: Whether files have HXL hashtags. Ignored for single url. Defaults to False.
             headers: Number of row(s) containing headers or list of headers. Defaults to 1.
             dict_form: Return dict or list for each row. Defaults to False (list)
             *args (Any): Positional arguments