Apply a declarative chain of raster operations across a folder of imagery — in parallel, and resumably, so an interrupted run picks up where it left off instead of starting over.
Batch raster processing fails in two predictable ways.
The first is the four-hour run that dies at file 3,214. Maybe a scene was truncated on download, maybe the machine ran out of memory, maybe someone hit Ctrl-C. Without durable per-file state you have no idea which outputs are complete, which are half-written, and which never started — so you delete everything and run it again overnight.
The second is the typo. You write a shell loop over gdalwarp, mistype a CRS in the fourth
command, and find out forty minutes in when the first three stages have already rewritten
2,000 files.
raster-batch-runner addresses both. A job is a YAML file validated up front — a bad spec fails
in milliseconds with a line number, before a single pixel is read. Execution is tracked in a
crash-safe ledger recording every file's status, so re-running the same job skips completed work,
--retry-failed re-runs only what broke, and a kill -9 mid-write can neither corrupt the ledger
nor leave a truncated GeoTIFF that a later run mistakes for finished.
Python 3.12 or newer (rasterio 1.5 ships no wheels for 3.11 and earlier).
git clone https://github.com/python-remote-sensing/raster-batch-runner.git
cd raster-batch-runner
pip install -r requirements.txtThere is no install step — the package runs from the repo root:
python -m raster_batch --helpWrite a job spec:
# job.yml
name: clip-to-cog
input:
glob: "data/raw/*.tif"
output:
dir: "data/cog"
template: "{stem}_cog.tif"
steps:
- op: clip
bbox: [600100, 5100100, 600800, 5100800]
- op: convert-to-cog
compress: deflateCheck it before committing to a long run:
$ python -m raster_batch validate job.yml
job.yml: valid
job clip-to-cog
steps clip -> convert-to-cog
inputs 13 file(s)
units 13 unit(s) of work
output data/cogSee exactly what would happen, without writing anything:
$ python -m raster_batch run job.yml --dry-run
clip-to-cog: clip -> convert-to-cog
13 unit(s) to run, 0 already done (of 13 total)
data/raw/S2A_T32TPS_20240100_B00.tif -> data/cog/S2A_T32TPS_20240100_B00_cog.tif
data/raw/S2A_T32TPS_20240100_B01.tif -> data/cog/S2A_T32TPS_20240100_B01_cog.tif
data/raw/S2A_T32TPS_20240101_B00.tif -> data/cog/S2A_T32TPS_20240101_B00_cog.tif
... and 10 more
nothing was written (--dry-run).Then run it. One corrupt scene fails that scene only — the other twelve still finish:
$ python -m raster_batch run job.yml --workers 4
[############################] 13/13 (100.0%) 126.77/s eta 0s failed 1
12 completed, 0 skipped, 1 failed in 0.1s (113.94 files/s)
failures:
data/raw/S2A_T32TPS_corrupt_B01.tif: cannot open S2A_T32TPS_corrupt_B01.tif: 'data/raw/S2A_T32TPS_corrupt_B01.tif' not recognized as being in a supported file format.
re-run with --retry-failed to retry only these.Run the same command again and the completed work is skipped:
$ python -m raster_batch run job.yml --workers 4
0 completed, 12 skipped, 1 failed in 0.0s (0.00 files/s)Check on a job at any time — including one still running in another terminal:
$ python -m raster_batch status job.yml --failures
clip-to-cog (data/cog/.raster-batch-clip-to-cog.jsonl)
done 12/13
failed 1
outstanding 1
failures:
data/raw/S2A_T32TPS_corrupt_B01.tif: cannot open S2A_T32TPS_corrupt_B01.tif: 'data/raw/S2A_T32TPS_corrupt_B01.tif' not recognized as being in a supported file format.Fix the input, then retry just the failure:
python -m raster_batch run job.yml --retry-failedValidation errors carry the line they came from and the offending source line:
$ python -m raster_batch validate broken.yml
broken.yml:10: steps[1] (reproject): 'crs' is not a CRS rasterio understands: invalid literal for int() with base 10: 'NOPE'
| crs: EPSG:NOPEUnknown operations get a suggestion rather than a stack trace:
$ python -m raster_batch validate typo.yml
typo.yml:7: unknown operation 'reprojct' (did you mean 'reproject'?); available: clip, compress, convert-to-cog, mask, mosaic, reproject, resample, stats
| - op: reprojct--json on any command makes the tool composable:
$ python -m raster_batch run job.yml --json | jq '{completed, failed, throughput_per_second}'
{
"completed": 12,
"failed": 1,
"throughput_per_second": 101.335
}from raster_batch import load_spec, run_job
spec = load_spec("job.yml")
summary = run_job(spec, workers=8, timeout=600)
print(f"{summary.completed} done, {summary.failed} failed")
for failure in summary.failures:
print(failure["unit"], failure["error"])The repo ships a composite GitHub Action. Pointing state-file at a cached path makes a job
resume across workflow runs — useful when a scheduled job processes more imagery than fits in one
job's time limit:
- uses: actions/cache@v4
with:
path: .raster-batch-state
key: raster-batch-${{ github.ref_name }}
- uses: python-remote-sensing/raster-batch-runner@main
id: batch
with:
job: jobs/nightly-cog.yml
workers: 4
timeout: 900
state-file: .raster-batch-state/nightly.jsonl
- run: echo "processed ${{ steps.batch.outputs.completed }} scenes"Job state lives in a JSON Lines file, by default next to the outputs. Each state change for a file
is one line, written with flush() followed by os.fsync() before the writer returns.
That specific design buys three things:
- A hard kill cannot corrupt it. There is no in-place mutation to interrupt. A process killed mid-write leaves at most one truncated trailing line; the reader discards it and every earlier record is already durable. The file the killed process was appending to is still a valid ledger.
- Last write wins. A file's state is simply the last record naming it, so
running → doneneeds no seeking, locking, or rewriting. - You can read it. When an overnight run goes wrong, the state file is plain text you can pipe
through
jq, not a binary blob that needs the tool that wrote it.
The fsync is the part people skip, and skipping it is why naive resume logic loses work: without
it, a kill -9 discards buffered records and the next run redoes files it had already finished.
Compaction (collapsing superseded records) writes a fresh file and os.replaces it over the old
one. Replace is atomic, so a crash during compaction leaves the previous complete ledger intact.
A file is skipped only when all three conditions hold:
- It is recorded as
done. - Its input signature is unchanged —
mtime+sizeby default, SHA-256 under--hash. Hashing thousands of scenes to decide what to skip would defeat the point of skipping, so it is opt-in for the case that actually needs it: inputs rewritten in place with a preserved timestamp. - Its output still exists on disk. Deleting an output is therefore a valid way to redo one file.
A file recorded as running means the process died mid-file, so it is re-run. Changing the job
spec changes its fingerprint, which invalidates the whole ledger — old outputs no longer match
what the job now asks for.
Every output goes to a hidden sibling temp file and is then os.replaced into place. Within a
filesystem that is atomic, so a crash leaves either the previous output or no output — never a
truncated GeoTIFF that condition 3 above would happily accept as complete. A SIGKILL can strand a
temp file; because temps are hidden dotfiles they can never be mistaken for outputs, and the next
run sweeps them.
clip is the one operation that can avoid reading a whole scene, and it does so when it runs
first — before anything has loaded pixels. It computes the read window from the bbox and asks GDAL
for only the blocks that intersect it. Clipping a 200 MB scene to a 1 km box costs a couple of
blocks, not 200 MB of RAM. Put clip first in a pipeline whenever you can; it is the single
biggest memory lever available.
Work is distributed over a ProcessPoolExecutor (GDAL work is not meaningfully threaded, and
separate processes contain segfaults from bad files). Each file is a unit; every exception inside
a unit is converted to a result record rather than propagating, so one unreadable scene can never
take down the pool.
Results are recorded in the parent process only, which means the ledger has exactly one writer and
needs no cross-process locking. --workers 1 runs in-process, which is deterministic and much
easier to attach a debugger to.
Per-file timeouts use signal.setitimer, which interrupts only the file that overran and leaves
the worker and the rest of the pool healthy.
convert-to-cog writes the file tiled with an overview pyramid, then re-copies it with
COPY_SRC_OVERVIEWS=YES so the overview IFDs land near the header. That header layout is the
property that lets a remote reader fetch a low-resolution view in one or two HTTP range requests
rather than crawling the whole file.
The compression predictor defaults by dtype — 2 (horizontal differencing) for integers, 3 for floats. This matters: applying the integer predictor to float data makes files larger.
| Flag | Description |
|---|---|
--workers N, -w N |
Worker processes. Default: the job's setting, else CPU count. 1 runs in-process. |
--force |
Redo every file, discarding recorded state. |
--retry-failed |
Run only the files that failed in a previous run. |
--fail-fast |
Stop scheduling after the first failure. Off by default. |
--timeout SECONDS |
Abort any single file taking longer than this. |
--limit N |
Process at most N files. Useful for trialling a job on a subset. |
--hash |
Detect input changes by SHA-256 instead of mtime+size. |
--state PATH |
Override the ledger location. |
--dry-run, -n |
Show the planned work and exit. |
--json |
Emit the run summary as JSON. |
--quiet, -q |
Suppress the progress display. |
Other commands: validate JOB (check a spec), status JOB (report recorded progress, --failures
to list them), ops (list operations and their parameters).
Exit codes: 0 success, 1 failure or invalid spec, 2 usage error.
| Key | Description |
|---|---|
name |
Job name; also names the default ledger file. |
input.glob |
Glob pattern, relative to the job file. |
input.files |
Explicit list of paths, merged with glob and deduplicated. |
input.root |
Base directory for relative patterns. Defaults to the job file's directory. |
output.dir |
Output directory, created if missing. |
output.template |
Output name. Placeholders: {stem} {name} {ext} {parent} {group} {index}. |
output.stats_file |
Where to write collected statistics as JSON. |
options.workers |
Default worker count. |
options.timeout |
Default per-file timeout in seconds. |
options.fail_fast |
Default fail-fast behaviour. |
options.hash_inputs |
Use SHA-256 signatures by default. |
options.state_file |
Default ledger location. |
| Operation | What it does | Key parameters |
|---|---|---|
clip |
Crop to a bbox or GeoJSON geometry via a windowed read. | bbox, geometry, crs, all_touched, mask_outside, pad |
reproject |
Warp to a target CRS. | crs (required), resolution, resampling, nodata |
resample |
Change pixel size within the same CRS. | scale or resolution, resampling |
mosaic |
Merge a group of inputs into their union. Must be the first step. | group_by, method, resampling, nodata |
compress |
Set compression, tiling and predictor for the output. | compress, predictor, level, tiled, blocksize, bigtiff, num_threads |
convert-to-cog |
As compress, with the defaults a valid COG needs. |
Same, plus overviews |
mask |
Set pixels to nodata by value and/or geometry. Never crops. | values, geometry, invert, all_touched, nodata, bands |
stats |
Per-band statistics, excluding nodata. | percentiles, histogram_bins |
resample's scale multiplies the pixel count per axis, so resampling Sentinel-2's 20 m bands
to 10 m is scale: 2.
mosaic's group_by is either all or a regex whose first capture group, matched against the
filename, forms the group key — group_by: "(T\\d{2}[A-Z]{3})" produces one mosaic per MGRS tile.
Files matching nothing land in a group called ungrouped rather than being silently dropped.
See examples/ for complete job specs covering clipping to COG, grid harmonisation,
and per-tile mosaicking.
- Vector inputs must be GeoJSON.
clipandmaskread geometries with the standard library rather than pulling in Fiona or GeoPandas. Convert a Shapefile or GeoPackage first (ogr2ogr aoi.geojson aoi.shp). - Operations after the first hold one raster in memory. Only the leading
clipstreams. A pipeline whose first step isreprojectreads the whole scene. Putclipfirst where you can, and lower--workersfor large scenes. - Per-file timeouts need SIGALRM, so they are a no-op on Windows. Everything else is portable.
mosaicmust be the first step and loads its whole group at once — a group covering a large area needs memory proportional to the union, not to one input.- One run per output directory at a time. The ledger assumes a single writer; two concurrent runs of the same job will interleave records and the stale-temp sweep may remove a file the other run is writing.
- No cross-file operations beyond
mosaic. Temporal compositing and band math across scenes are out of scope; this tool applies a per-file (or per-group) pipeline. statsalone writes no rasters — it is a reporting pipeline, and copying every input verbatim would be waste. Combine it with another operation if you want both.
Background guides on the techniques this tool automates:
- Building a YAML-driven multi-index pipeline — the case for declaring pipelines as data rather than as code.
- Clipping a raster to a bounding box with windowed reads
— the mechanism behind the
clipoperation. - Optimizing rasterio window reads for memory efficiency — how block size and read patterns decide your memory ceiling.
- Understanding Cloud-Optimized GeoTIFF structure
— what
convert-to-cogis actually producing, and why IFD ordering matters. - Choosing COG compression: ZSTD vs DEFLATE
— picking the
compressandlevelvalues for your data. - Retrying failed raster tasks in a Prefect pipeline — the same resume problem, solved at orchestrator level.
Issues and pull requests are welcome. To work on the code:
pip install -r requirements-dev.txt
python -m pytest
ruff check . && ruff format --check .Tests are hermetic — fixtures generate synthetic GeoTIFFs with rasterio, so nothing touches the
network. New operations go in raster_batch/ops/ and register in raster_batch/ops/__init__.py;
please include tests asserting real geospatial correctness (bounds, CRS, pixel values), not just
that a file appeared.
MIT — see LICENSE.
Built and maintained alongside Python Remote Sensing & Raster Processing Pipelines.