feat: US EV charging coverage via OpenChargeMap with bbox tiling#86
Conversation
The OCM scraper issued a single request capped at 5000 results, silently truncating any country above the cap (DE/GB/FR/NL are affected today). Register EV_US and fix the truncation: when the root query returns at the cap, quadtree-tile the bounding box (still countrycode-filtered), recurse on capped tiles, dedupe by POI ID, all under a hard request budget with a politeness delay. - ocm.ts: BFS bbox tiling (MAX_REQUESTS=120, MAX_TILE_DEPTH=9, 300ms delay via PUMPERLY_OCM_TILE_DELAY_MS), per-run request/POI logging, partial- coverage warning instead of silent truncation - instrumentation.ts: EV_US registration (24h interval like other EV scrapers); enabled-countries now derives EV_XX from raw tokens so a bare "US" (EV-only country, no fuel scraper) enables EV_US - tests: single-request fast path, quadrant tiling + cross-tile dedupe, request-budget termination on pathological capped responses - README/.env.example: document US EV-only coverage bbox param format validated against the live OCM API (Portland box returns only in-box POIs; control query without bbox returns arbitrary US POIs). Closes #85. Refs #78.
|
Warning Review limit reached
Next review available in: 38 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughChangesUS EV coverage
Open Charge Map tiling
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
src/scrapers/ocm.test.ts (1)
223-240: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAdd a maximum-depth termination test.
This only exercises request-budget exhaustion. Add a case where one branch remains capped through depth 9 while sibling tiles return empty, then assert termination and the partial-coverage warning. That protects the distinct Lines 166-170 depth-limit path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/scrapers/ocm.test.ts` around lines 223 - 240, Add a test alongside the existing request-budget test that mocks one tile branch to remain capped through depth 9 while sibling tile requests return empty responses. Assert that scraper.fetch() terminates and emits the partial-coverage warning, specifically covering the maximum-depth handling in OCMScraper rather than request-budget exhaustion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/scrapers/ocm.ts`:
- Line 23: Update the scraper loop around MAX_REQUESTS and the request handling
near the referenced request block to enforce a total run deadline, limiting each
request timeout to the remaining duration rather than a fixed 120 seconds. When
the deadline expires, stop issuing requests, preserve the POIs accumulated so
far, and emit a warning indicating that coverage is partial due to the time
limit.
- Line 25: Validate the value assigned to TILE_DELAY_MS after parsing
PUMPERLY_OCM_TILE_DELAY_MS, falling back to 300 ms when it is NaN, infinite, or
negative. Preserve configured finite, non-negative values so throttling remains
enabled as intended.
- Line 126: Update fetchPage around the res.json() return to parse the response
through the existing or appropriate Zod schema before returning it. Ensure
malformed OCM POI data is rejected at this boundary, while valid parsed data
continues into the tiling and deduping flow.
---
Nitpick comments:
In `@src/scrapers/ocm.test.ts`:
- Around line 223-240: Add a test alongside the existing request-budget test
that mocks one tile branch to remain capped through depth 9 while sibling tile
requests return empty responses. Assert that scraper.fetch() terminates and
emits the partial-coverage warning, specifically covering the maximum-depth
handling in OCMScraper rather than request-budget exhaustion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 62ccccc4-92ae-40f0-9a31-089b52acd96a
📒 Files selected for processing (5)
.env.exampleREADME.mdsrc/instrumentation.tssrc/scrapers/ocm.test.tssrc/scrapers/ocm.ts
cli.test.ts guards that instrumentation and the manual-scrape CLI expose the identical EV_* key set; EV_US was added to instrumentation only.
…Zod boundary validation - Bound the tiling loop by elapsed time (15min) as well as request count; 120 slow responses could otherwise stall a scrape cycle for hours. - Sanitize PUMPERLY_OCM_TILE_DELAY_MS (NaN/negative falls back to 300ms instead of silently disabling throttling). - Validate the OCM payload with a minimal Zod schema at the response boundary: reject non-array payloads with a clear error, drop malformed POIs with a warning; replaces the hand-written interfaces.
|
All three findings addressed in the latest commit — thanks @coderabbitai:
|
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
fd ocm.ts src/scrapers
echo "---"
rg -n "MAX_ELAPSED|TILE_DELAY_MS|z\.|zod|safeParse|parse\(" src/scrapers/ocm.tsLength of output: 1136 🏁 Script executed: #!/bin/bash
rg -n "Date.now|startTime|elapsed|MAX_.*MS|MAX_.*TIME|15 \* 60|900000" src/scrapers/ocm.tsLength of output: 717
Looks good — thanks for addressing these. 🐇✨ |
1 participant
What
Adds US EV charging coverage via the existing OpenChargeMap scraper (
EV_US), and fixes the silent truncation that made large countries impossible: the scraper issued a single request capped atmaxresults=5000, quietly dropping everything beyond it. Germany, GB, France and the Netherlands are already being truncated by this today; the US (~50k+ operational POIs) was unservable.How
src/scrapers/ocm.ts): when the root country query returns at the cap, split the bounding box into 4 and refetch each tile (stillcountrycode-filtered), recursing on tiles that hit the cap again. BFS order so the request budget refines coverage evenly.MAX_REQUESTS=120per run,MAX_TILE_DEPTH=9, 300 ms politeness delay between tile requests (PUMPERLY_OCM_TILE_DELAY_MS, tests set 0).coverage may be partialwarning, never silent loss.EV_USregistration (src/instrumentation.ts), 24 h interval like every other EV scraper.PUMPERLY_ENABLED_COUNTRIESderivedEV_XXonly from tokens that matched a fuel scraper, so a bareUS(EV-only country) could never enableEV_US. EV codes now derive from the raw token list.boundingboxparam), quadrant tiling with cross-tile dedupe, and request-budget termination on pathological always-capped responses..env.examplenote (US is EV-only — no national fuel-price API exists).Validation
boundingbox=(lat1,lon1),(lat2,lon2)format verified against the live OCM API: a Portland-box query returns only in-box POIs (lat 45.54–45.60, lon −122.94…−123.0), while the same query without bbox returns arbitrary US POIs (OK/CA). Fail-safe by construction: even a regressed bbox filter would only re-fetch country-wide pages, which dedupe collapses — coverage falls back to today's behavior, never worse.Scope notes
ev_charger, connector metadata from OCM). US route planning is not included — prod Valhalla/Photon only carry European extracts; that's a separate infra decision.PUMPERLY_ENABLED_COUNTRIESis pinned in the compose env — needs,USappended for pumperly.com to pick this up after release.Closes #85. Refs #78 (community static-data path remains the fallback for regions OCM covers poorly).
Summary by CodeRabbit
New Features
Documentation