Fix: Wigan Borough Council - updated Search.aspx endpoint and date div class name (#2136)

[SteVader]

Wigan's My Neighbourhood site changed in two ways that broke the scraper:

• The postcode-search and address-selection postbacks now target Search.aspx instead of posting back to MyArea.aspx itself.
• The "next collection" date div was renamed from dateWrap-next to dateWrapper-next (contents are now nested spans rather than a flat text node, but get_text(strip=True) still concatenates them into the same "Friday17thJul2026"-style string the existing regex and strptime already handle, so no parsing changes were needed there).

Updates both POST endpoints to Search.aspx, and updates the date lookup to use dateWrapper-next with a fallback to the old class name plus a clearer ValueError if neither is found, so a future markup change fails loudly instead of raising an opaque AttributeError.

Summary by CodeRabbit

• Bug Fixes
  • Updated postcode lookup and collection-date retrieval to match the council’s latest web pages.
  • Improved fallback handling for bin collection date sections, so results are more resilient when the page layout changes.
  • Added a clearer error when the expected date information cannot be found.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR updates the Wigan Borough Council scraper to use new endpoint URLs (MyArea.aspx and Search.aspx) instead of the legacy MyNeighbourhood path, and adds a fallback lookup for the bin collection date container class, raising a clearer ValueError if neither class is found.

Changes

Wigan Borough Council scraper update

Layer / File(s)	Summary
Updated request endpoints uk_bin_collection/uk_bin_collection/councils/WiganBoroughCouncil.py	Initial session GET and both postcode-search POST requests now target MyArea.aspx and Search.aspx instead of the previous MyNeighbourhood/ root path.
Date container lookup with fallback uk_bin_collection/uk_bin_collection/councils/WiganBoroughCouncil.py	Bin collection date extraction now checks dateWrapper-next first, falls back to dateWrap-next, and raises a ValueError with a clearer message if neither class is present.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: dp247

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the scraper endpoint and date-class updates in the PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

uk_bin_collection/uk_bin_collection/councils/WiganBoroughCouncil.py (1)

55-55: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider a shared constant for the repeated Search.aspx URL.

The same URL string is duplicated across the two POST calls; extracting it avoids drift if it changes again.

Also applies to: 77-77

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@uk_bin_collection/uk_bin_collection/councils/WiganBoroughCouncil.py` at line
55, The Search.aspx endpoint URL is duplicated in the WiganBoroughCouncil POST
calls, so extract it into a shared constant and reuse it in the relevant request
methods (including the repeated call in the same class) to keep the URL
consistent and avoid future drift.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@uk_bin_collection/uk_bin_collection/councils/WiganBoroughCouncil.py`:
- Line 34: The session requests in WiganBoroughCouncil are missing timeouts, so
the MyNeighbourhood fetch and follow-up POST can hang indefinitely on a slow
endpoint. Update the request flow in WiganBoroughCouncil to pass a reasonable
timeout to every s.get and s.post call, including the initial area lookup and
the later POST used in the same flow. Keep the same request logic otherwise,
just ensure all session calls in this file are protected with explicit timeout
values.

---

Nitpick comments:
In `@uk_bin_collection/uk_bin_collection/councils/WiganBoroughCouncil.py`:
- Line 55: The Search.aspx endpoint URL is duplicated in the WiganBoroughCouncil
POST calls, so extract it into a shared constant and reuse it in the relevant
request methods (including the repeated call in the same class) to keep the URL
consistent and avoid future drift.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 62b9f7d8-ea69-4ebe-a3b8-cfcb929e5910

📥 Commits

Reviewing files that changed from the base of the PR and between b65502c and 9198f42.

📒 Files selected for processing (1)

• uk_bin_collection/uk_bin_collection/councils/WiganBoroughCouncil.py

[coderabbitai]

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add request timeouts on the session calls.

None of the s.get/s.post calls specify a timeout, so a slow/unresponsive Wigan endpoint (a real risk right after their site changed underlying paths) will block indefinitely.

🌐 Proposed fix

-        response = s.get("https://apps.wigan.gov.uk/MyNeighbourhood/MyArea.aspx")
+        response = s.get("https://apps.wigan.gov.uk/MyNeighbourhood/MyArea.aspx", timeout=30)

-        response = s.post("https://apps.wigan.gov.uk/MyNeighbourhood/Search.aspx", payload)
+        response = s.post("https://apps.wigan.gov.uk/MyNeighbourhood/Search.aspx", payload, timeout=30)

(apply the same to the second s.post call at line 77)

Also applies to: 52-55, 76-77

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@uk_bin_collection/uk_bin_collection/councils/WiganBoroughCouncil.py` at line
34, The session requests in WiganBoroughCouncil are missing timeouts, so the
MyNeighbourhood fetch and follow-up POST can hang indefinitely on a slow
endpoint. Update the request flow in WiganBoroughCouncil to pass a reasonable
timeout to every s.get and s.post call, including the initial area lookup and
the later POST used in the same flow. Keep the same request logic otherwise,
just ensure all session calls in this file are protected with explicit timeout
values.