Skip to content

Refactor for hpc batch api pull and outbreak extraction#13

Open
javierps wants to merge 65 commits into
dev_outbreak_def_updatesfrom
parallel_refactor
Open

Refactor for hpc batch api pull and outbreak extraction#13
javierps wants to merge 65 commits into
dev_outbreak_def_updatesfrom
parallel_refactor

Conversation

@javierps

@javierps javierps commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

javier and others added 30 commits June 8, 2026 14:24
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ntries × time windows

Implements the plan in snuggly-crunching-kernighan.md:
- analysis/config_defaults.yml + utils.R: shared config/helpers
- analysis/00_make_configs.R: generates pull_set / detection_set / test YAML configs
- analysis/01_pull_data.R (Batch 1): taxdat API pull → normalize → stage1 GeoParquet/flat Parquet
- analysis/02_run_outbreak_detection.R (Batch 2): identify_outbreaks() + trigger_alert() per country
- analysis/03_aggregate_results.R: parallel-load stage2 files → combined_outbreaks.{parquet,csv}
- analysis/bash/submit_0{1,2}*.sh: SLURM array scripts with credential validation and bounds checks
- R/get_shp.R: add output_parquet param + on.exit disconnect
- DESCRIPTION: add DBI/RPostgres/glue/sf to Imports; sfarrow/arrow/yaml/optparse/furrr/future/here/taxdat to Suggests
- CLAUDE.md: document analysis/ layer architecture and quick-start

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Default is now GeoJSON + RDS (no sfarrow/arrow required). Set
use_geoparquet: true in any config to restore GeoParquet + Parquet output.

- config_defaults.yml: add use_geoparquet: false
- utils.R: make filename helpers format-aware (.geojson/.rds vs .parquet);
  add write_tabular(), read_tabular(), write_spatial() I/O helpers
- 01_pull_data.R: replace sfarrow/arrow calls with helpers
- 02_run_outbreak_detection.R: remove library(arrow); dynamic glob extension;
  update run_id suffix regex; replace arrow calls with helpers
- 03_aggregate_results.R: remove library(arrow); add --format flag
  (default geojson); dynamic discovery pattern; inline conditional read;
  write parquet only in geoparquet mode, always write CSV

Existing generated configs lacking the key default to GeoJSON mode via
isTRUE(NULL) == FALSE — no regeneration required.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- R/add_population.R: new exported add_population() function; groups LPs
  by year and loads each WorldPop raster once, running two vectorized
  exact_extract calls (adj-factor + all LPs) rather than 2N per-LP loads
  that would result from calling get_pop() in a loop
- analysis/01_pull_data.R: call add_population() after fill_missing_lps(),
  before writing to parquet; fixes missing pop column consumed by
  get_outbreak_threshold() and identify_epidemic_start()
- analysis/config_defaults.yml: add raster_dir key for WorldPop cache path
- analysis/bash/install_r_packages.sh: update module load to GCC/12.3.0 +
  R/4.3.2, add rgeoboundaries, ISOcodes, readr, reshape2; fix taxdat GitHub
  path and pin Matrix version
- analysis/utils.R: whitespace cleanup; fix length(locations == 1) ->
  length(locations) == 1 parenthesis bug in read_taxonomy_data_database()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- analysis/01_pull_data.R: qualify pull_taxonomy_data() and
  rename_database_fields() with taxdat:: so they resolve correctly
  without any re-exports in utils.R

- R/clean_psql_data.R: add column normalization block at entry to
  bridge taxdat API naming (is_primary, locationPeriod_id, OC_UID,
  location_name, attributes.fields.*) to the OutbreakExtractR
  convention (primary, location_period_id, observation_collection_id,
  location, sCh, cCh, deaths). Also handle already-logical primary
  values from the API so filter(primary) is not silently vacuous.

- R/add_population.R: accept locationPeriod_id (taxdat camelCase) as
  a third fallback for the LP geometry ID column, alongside
  location_period_id and lctn_pr.

- analysis/utils.R: remove taxdat duplicate functions (pull_taxonomy_data,
  rename_database_fields, get_shp, and all helpers); these are now
  called via taxdat:: directly.

- analysis/bash/install_r_packages.sh: fix taxdat source repo to
  HopkinsIDD/cholera-mapping-pipeline (subdir packages/taxdat, branch
  dev); add missing runtime deps igraph, geodata, geojsonsf, rjson,
  httr, jsonlite.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The hardcoded URL "http://cholera-taxonomy.middle-distance.com/" was
wrong on two counts: it used http instead of https, and was missing
the api. subdomain. The correct URL is already in config_defaults.yml
as api_website. Use opt\$api_website so the value flows from the config.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The server's TLS certificate does not list api.cholera-taxonomy.middle-distance.com
as an alternative name, so curl rejects the connection. Wrap the
pull_taxonomy_data() call in httr::with_config(ssl_verifypeer = FALSE)
to bypass peer verification for just this request until the server
cert is updated.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
httr::with_config() has a promise-evaluation timing issue when the
HTTP call happens inside a nested function (pull_taxonomy_data ->
read_taxonomy_data_api -> httr::POST): the curl handle is created
before with_config has applied the options to the global httr state.

httr::set_config() modifies the global config synchronously before
the call, which is what httr::POST reads when building the handle.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The error is 'no alternative certificate subject name matches target
host name' — this is a hostname verification failure controlled by
CURLOPT_SSL_VERIFYHOST (ssl_verifyhost), not CURLOPT_SSL_VERIFYPEER.
ssl_verifyhost must be 0L (integer zero), not FALSE, for curl to
accept it. Also keep ssl_verifypeer = 0L for completeness.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pper)

pull_taxonomy_data is a source-routing wrapper; since we always use
the API path, call read_taxonomy_data_api directly.

Changes vs the wrapper call:
- password= renamed to api_key= (inner function's actual param name)
- source= dropped (wrapper-only routing param)
- time_left/time_right wrapped in as.character() — the wrapper did
  this coercion internally before forwarding; the inner function does not

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HPC 500: the taxdat dev branch changed the API base URL from
https://api.cholera-taxonomy.middle-distance.com/ to
https://cholera-taxonomy.middle-distance.com (no api. subdomain,
no trailing slash). Update config_defaults.yml to match, and restore
website = opt\$api_website in the call so both local and HPC
explicitly use the configured URL rather than each relying on the
package default.

clean_columns error: taxdat::flatten_json_result calls
jsonlite::flatten() which fails on list columns whose elements are
nested data frames or raw vectors. Patch the function in the taxdat
namespace via assignInNamespace() before calling read_taxonomy_data_api,
dropping such columns before flattening. They carry no information
used downstream.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
javier and others added 28 commits June 9, 2026 16:56
Error 1 (103 jobs): guard optional API column renames in 01_pull_data.R —
rename if present, else mutate to NA. Affects observation_collection_id,
sCh, cCh, deaths, location_period_id which are absent from some API
responses. Consolidates the existing cCh guard into the same loop.

Error 2 (27 jobs): fix fill_phantom_zeroes() — add early return for empty
input (0 rows after filtering trips the start_weekday guard), narrow the
guard to na.omit() to tolerate NA dates, and replace error() with stop()
(error() is not an R function).

Error 3 (22 jobs): guard empty geometries in add_population() — API returns
GEOMETRYCOLLECTION EMPTY for some LPs; sf::st_dimension() returns NA for
these, causing exactextractr to throw "missing value where TRUE/FALSE needed".
Filter them out after st_transform, before the exact_extract call.

Also adds analysis/pull_set_errors_2026-06-09.md summarising all six error
types from the 315-job array run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Errors fixed (from analysis/pull_set_errors_2026-06-09.md):

Error 5 — empty lp_years early return (add_population.R):
  purrr::list_rbind() on an empty map returns a 0-column tibble, breaking
  the downstream left_join on location_period_id. Added early return with
  pop = NA when no valid location_period_ids are present.

Error 6a — corrupted WorldPop download (get_pop.R):
  A partial/interrupted download of a large raster produces a truncated LZW
  stream; GDAL reports "code not yet in table" / TIFFReadEncodedTile at read
  time. download_worldpop_constrained() now reads one block after download to
  verify the file is intact; if corrupt, it deletes the cached file and retries
  with the next release (R2024B). Both releases use identical LZW+PREDICTOR=2
  compression, confirmed by gdalinfo on both R2025A and R2024B files.

Error 6b — non-polygon geometry passed to exact_extract (add_population.R):
  POINT geometries (centroid-only API responses) caused exact_extract to fail
  because it requires 2-D polygon input. Previously only empty geometries were
  filtered; now a single st_dimension() pass covers both empty (NA) and
  non-polygon (dim != 2) cases, replacing the two-step st_is_empty +
  st_dimension pattern with one combined filter.

Also: add .gitignore entries for logs/, analysis/generated_data/,
analysis/worldpop/, and *.Rcheck/; add .Rbuildignore and .Rproj.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- 01_pull_data.R: move empty-API guard before select/rename so zero-row
  sf objects (no attributes.time_left) exit cleanly instead of crashing
- add_population.R: apply st_make_valid() on unique LP geometries at
  deduplication time so all downstream uses (fallback st_union, exact_extract)
  receive topologically valid polygons
- add_population.R: extract bare ISO3 with regexpr("[A-Z]{3}") before
  gb_adm0() so sub-national codes like "TZA::Mainland" resolve correctly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hreshold()

customized_TL and customized_TR were passed without names, so R matched
them positionally to fixed_outbreak_threshold and customized_TL
respectively, leaving customized_TR=NULL inside get_outbreak_threshold().
The subsequent subset(TL >= customized_TL & TR <= customized_TR) then
evaluated NULL on the right-hand side, producing logical(0) and crashing
the tibble row subscript. Discovered during smoke tests on AGO and COD.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the last detected epidemic start had already been assigned an outbreak
number (via the multi-outbreak loop), the code called min() on an empty set
to find the tail end, returning Inf. The subsequent row subscript [X:Inf]
crashed with "result would be too long a vector".

Fix: guard the min() call with any(epidemic_tail); fall back to nrow() when
the outbreak extends to the end of the time window without a detected tail.
Also cap the end index at nrow() to prevent out-of-bounds subscripting.

This bug caused identify_outbreaks() to error on virtually every non-empty
window for countries with real data (e.g. 45/46 ETH windows crashed before
this fix), producing near-empty Stage 2 output.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When add_population() cannot find a WorldPop raster match for a location,
pop = NA. This caused get_outbreak_threshold() to produce threshold = NA and
subsequently risk = NA via ifelse(NA >= threshold, "high", "low").

Downstream, NA risk values crashed identify_epidemic_start() and
identify_epidemic_tail() with "missing value where TRUE/FALSE needed" because
if(all(NA)) and if(any(NA)) evaluate to NA rather than FALSE.

Fixes:
- get_outbreak_threshold(): use case_when to map NA pop/threshold to "low"
  risk rather than propagating NA (conservative: unknown incidence = not high)
- identify_epidemic_start(): wrap consecutive-high check with isTRUE() to
  treat NA comparisons as FALSE
- identify_epidemic_tail(): wrap rle any() check with isTRUE() for same reason

Also adds:
- analysis/parse_detect_logs.R: SLURM log parser for Stage 2 detection jobs,
  with per-batch breakdowns and per-window failure classification
- analysis/scratch_coverage.R: Stage 1 coverage heatmaps (presence, obs rows,
  suspected cases) with 3-level status (has data / zero cases / no data)
- analysis/scratch_api_vs_stage1.R: systematic API cache vs Stage 1 comparison

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…stats

Four-panel figure (PDF) loading all stage2_*.rds outputs:
- Panel A: outbreak burden by country (% of span-weeks), gaps = 0
- Panel B: temporal heatmap with explicit data-gap cells (grey)
- Panel C: outbreak prevalence by spatial scale (all loc-weeks as denom)
- Panel D: outbreak sCh per span-week normalised for observation period

Fix: Panel A denominator is now country-scale calendar-weeks to avoid
inflating pct_outbreak 10-28× when counting all-admin location-week pairs
against a calendar-week span (COD was showing 2800%, IRQ 1949%).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nzibar)

add_population() was passing the raw country_iso3 (e.g. "TZA::MAINLAND") to
download_worldpop_constrained() and the WPP2024 lookup, both of which require a
plain 3-letter ISO3 code. The raster download failed silently, leaving pop = NA
for all TZA rows and causing identify_outbreaks() to mark all weeks as low-risk
(no outbreaks detected). Fix reuses the already-computed iso3_for_boundary
variable (line 105) in both callsites.

Also adds facet_grid(who_region ~ .) to Panel B of the stage2 summary figure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- shiny/app.R: full single-file Shiny app (map, stats table, heatmap,
  weekly time series) with bslib layout and reactive sidebar filters
- shiny/data_prep.R: centroid build script; now samples one GeoJSON per
  4-year window per country (captures admin boundary changes over time),
  applies st_make_valid() before st_centroid, and back-propagates parent
  centroids for aggregate location strings (e.g. AFR::TZA::Mainland)
- shiny/data/centroids.rds: rebuilt lookup (5320 locations, up from 2850)
- Fix TZA scale labelling: Mainland/Zanzibar are ADM0 analogs; shift
  admin1→Country, admin2→Admin 1, admin3→Admin 2 in new_ob_admin
- Fix choropleth CFR: was summing per-location ratios; now aggregates
  deaths/cases before dividing
- Add Admin 3 to SCALE_LEVELS (25K admin3 rows in data)
- Fix completion detection in parse_detect_logs.R: fall back to
  "Stage 2 complete." when "Batch 2 end:" sentinel is absent

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ggplotly() converts Date axes to type "linear" (numeric days since
epoch), so ISO date strings passed as plotly shape coordinates silently
collapsed to x=0, making all bands invisible. Switch the time series
to native plot_ly() so the x-axis is explicitly type "date" and shape
coordinates work correctly.

Additional fixes:
- xmax = max(TL) + 7 days so single-week outbreaks have non-zero width
- Aggregate admin-level ts_raw by TL before band computation (multiple
  run_ids could produce duplicate TL rows, corrupting the diff() logic)
- Fix x-axis range to 2010-01-01 -- 2024-12-31 (fixed across all locations)
- Replace ggplotly-based empty placeholder with native plot_ly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Convert combined_outbreaks_cholera.csv to parquet (snappy): 66 MB → 1 MB
- Replace read.csv + here() with arrow::read_parquet() using paths
  relative to the app directory (required for shinyapps.io; Shiny sets
  working directory to the app dir on both local and hosted runs)
- Replace here("shiny/data/centroids.rds") with "data/centroids.rds"
- Swap library(here) for library(arrow); add "arrow" to dep check
- Add shiny/manifest.json (rsconnect::writeManifest) capturing 101
  package dependencies for one-command deploy:
    rsconnect::deployApp("shiny/")

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
arrow's bit dependency (4.0.5) fails to compile on shinyapps.io R 4.5.2.
Switch to native RDS format: no extra packages, no compilation, and
slightly smaller on disk (0.7 MB vs 1.0 MB parquet).

- Convert data to combined_outbreaks_cholera.rds (xz-compressed)
- Replace arrow::read_parquet() with readRDS()
- Remove arrow from library list and dependency check
- Regenerate manifest.json (97 deps, down from 101)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New exported function that accepts identify_outbreaks() output and
verifies it is consistent with the definition parameters used to
produce it. Runs 12 checks across location and outbreak levels:
risk classification, epidemic start validity (consecutive and
dual-window modes), cumulative cases at start, tail structure,
within-outbreak continuity, and inter-outbreak spacing.

Includes 44 testthat tests covering PASS, FAIL, SKIP, and edge cases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
01_pull_data.R (Batch 1): reduced to pull + clean_psql_data() only.
stage1_flat_* now holds per-window cleaned observations (geometry
dropped) rather than normalized weekly data. Normalization, population
attachment, and outbreak detection have moved to Batch 2.

02_run_outbreak_detection.R (Batch 2): restructured from per-window
detection to full-series per-country processing, matching the reference
pipeline (GenevaIDD/global-cholera-surveillance-timeseries):
  - Concatenates all stage1_flat_* + stage1_geo_* files per country
  - Applies reference normalization: fill_missing_lps ×3,
    average_duplicate_observations, set_uniform_wday_start,
    filter(n_obs > 1), fill_phantom_zeroes
  - add_population() over the full assembled series
  - identify_outbreaks() with no customized_TL/TR so the mean weekly
    incidence threshold is computed over all years (not per 4-month
    window)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New batch 44869016 runs a redesigned Stage 2 that processes all windows
together rather than per-window, producing different log output.

Key changes:
- Detect log format via "Linking to GEOS" / "Loaded N observations" signal
- Fix no_results detection for new messages: "No Stage 1 observations for:",
  "No observations after filtering for:", "No data after normalization for:",
  "No outbreaks detected for:"
- Add outbreak_rows, n_obs_loaded, no_results_reason columns
- Add error_class column with exactextractr_error / purrr_map_error classes
- Add success_with_warnings / success_no_results_with_warnings outcomes
  (distinguishes clean completions from those with per-window failures)
- Add --latest-only flag to restrict parsing to the most recent batch
- Retain backward-compatible per-window fields (NA for new-format logs)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
exactextractr::exact_extract() throws "Mixed-type geometries not supported"
when the valid_sfc passed to it contains a mix of POLYGON and MULTIPOLYGON
features — as occurs in COD (DRC) which has 784 LPs spanning multiple
admin levels and eras.

Fix: after the existing empty/non-polygon filter, cast valid_sfc to a
uniform MULTIPOLYGON type before the exact_extract call.  Conversion from
POLYGON → MULTIPOLYGON is always safe after the dimension==2 guard;
GEOMETRYCOLLECTION features are handled by sf::st_cast (first polygon part
retained, with an sf warning).

Also wraps the country-boundary adj-factor extraction in tryCatch so a
geometry error there falls through to adj_factor = 1.0 rather than halting
the job.

Verified locally: COD now completes with 158 424 rows / 25 487 outbreak-
period rows and 233 661 / 287 092 rows with population estimates.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…; fix pre-existing test failures

## New feature: filter_outbreaks_by_size
Add `filter_outbreaks_by_size` (default FALSE) to `identify_outbreaks()`. When TRUE,
drops any detected outbreak whose total cases (summed over the full outbreak window)
fall below `cumulative_min_cases`. This makes the existing `cumulative_min_cases = 50`
config value act as a genuine size floor rather than just gating one side of the
dual_window cumulative alert trigger. Enabled by default in `config_defaults.yml`
(`filter_outbreaks_by_size: true`). Wired through `02_run_outbreak_detection.R`.
Implemented via an internal `filter_small_outbreaks()` helper inserted before the
Time Period labelling step so dropped outbreaks are automatically reclassified as
non-outbreak period. Docs and `man/identify_outbreaks.Rd` updated.

## Bug fix: hardcoded +1 tail extension
Replace hardcoded `+2-1` with `+tail_period-1` in the outbreak window assignment
loop in `identify_outbreaks()`. The hardcoded extension forced a 4-week minimum
duration regardless of `tail_period = 6`; 87% of those minimum-duration events
had < 50 total cases (documented in `analysis/data_issue_short_outbreak_spike.md`).

## Test fixes (pre-existing failures)
- `R/fill_missing_lps.R`: add missing `dplyr::` prefix to `arrange`, `group_by`,
  `mutate`, `case_when`, and `ungroup` calls (NAMESPACE only imports magrittr).
- `R/clean_psql_data.R`: fix `case_when` type mismatch — `is.logical(primary) ~ primary`
  produced a <character> RHS when the primary column was character; changed to
  `~ as.logical(primary)` so all arms return <logical>, compatible with dplyr >= 1.1.

Test suite: 103 pass, 0 fail.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…:: countries

The geo-file discovery regex applied gsub("::", "_", country_iso3), but
stage1_geo_*.geojson filenames retain the :: separator (e.g.
stage1_geo_AFR_TZA::Mainland_...). The substituted pattern never matched,
so geo_files was empty, raw_sf was NULL, pop was set to NA, every week
classified as low risk, and no outbreaks were detected (TZA returned 0).
Use country_iso3 as-is to match the actual filenames.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- All 68 detection_set configs: add filter_outbreaks_by_size: yes so that
  outbreaks with < 50 cumulative cases are zeroed post-detection (cumulative_min_cases
  already set to 50; flag was missing from all but TZA configs, allowing sub-50
  outbreaks to propagate into the combined CSV)
- detection_set_40 (TZA::Mainland) + detection_set_46 (TZA::Zanzibar):
  extend time_upper_bound from 2015-12-31 to 2024-12-31 to capture full available
  API data range (database coverage confirmed through 2019 for both locations)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…g shape IDs

Some API responses omit shapes from the `included` list for certain location
periods.  match() returns NA/NULL for the missing shape_id, and the subsequent
[[NA]] index crashes with "attempt to select less than one element in
get1index" before the existing is.null(unformatted_geojson) guard can fire.

Adds an assignInNamespace patch alongside the existing flatten_json_result
patch, guarding against NA/NULL this_shape_index and falling back to an empty
point geometry.  Also qualifies flatten_json_result calls inside the patch
body as taxdat:::flatten_json_result so they resolve correctly from the global
environment.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
…es fallback

When rgeoboundaries::gb_adm0() fails (e.g. TZA::Mainland not a valid name),
the fallback unions all LP geometries as the country boundary.  GEOS 3.12.0
on Yggdrasil throws a TopologyException on the TZA polygon at this step even
though individual LP geometries are already validated.  Applying st_make_valid()
to the transformed sfc before st_union() prevents the crash.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Composite locations ("|"-joined admin names, NA location_period_id) were
silently dropped during outbreak detection because add_population() skips
NA LPs → pop = NA → threshold forced to "low" → no epidemic start.

Reproduces the handling from GenevaIDD Step2_Extract_outbreak.R:
- New R/build_composite_locations.R (exported): de-composites "|"-joined
  location names into child admin units, maps each child to its atomic
  location_period_id + WorldPop population (already attached by
  add_population()), assigns composite_loc_<ISO3>_<n> pseudo-LP ids with
  summed child population and unioned child geometries from raw_sf.
- Fallback when children are not observed atomically (e.g. BDI sanitary
  districts): use the parent admin location's population and geometry as
  an approximation (denominator covers the full parent area).
- loc_lookup_extended strips ".CountryName" dot-suffixes on country-level
  location strings so that admin1-level composites (parent = "AFR::BDI")
  match the "AFR::BDI.Burundi" entry in normalized.
- 02_run_outbreak_detection.R: wire build_composite_locations() between
  add_population and identify_outbreaks; write a composite-geometry sidecar
  stage1_geo_<region>_<iso3>_composite.geojson so the downstream converter
  picks up composite LP geometries via its existing glob.
- Exclude *_composite.geojson sidecars from the raw_sf geo-file load to
  prevent schema-mismatch rbind errors on re-runs.

Verified on BDI: 0 → 4 composite LPs in stage2 output, 50 outbreak rows,
6-feature sidecar with valid WGS84 geometries. AGO (no composites)
unchanged (3691 rows, 253 outbreak rows). Countries still needing --redo:
ETH, SOM, SDN, CMR, COD, NGA, SSD, TCD, TZA.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Batch 2 was refactored to stitch all pull windows into one continuous
per-country series (commit 5f3151e), dropping the per-window run_id
column in favor of a single time_lower_bound/time_upper_bound pair.
The Shiny app still grouped by run_id, causing a group_by() error on
load since the column no longer exists in current pipeline output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@javierps javierps force-pushed the dev_outbreak_def_updates branch from 1352be6 to bb093aa Compare July 10, 2026 08:03
@javierps javierps force-pushed the parallel_refactor branch from bb3c7d9 to 537d6ba Compare July 10, 2026 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant