Generate a robots.txt (#1068)

[dduugg]

Fixes #1068.

https://www.shift2bikes.org/robots.txt returns a 404. Hugo only emits the file when enableRobotsTXT is set, and it never has been. There is no site/static/robots.txt or template filling the gap either.

Changes

• site/config/_default/hugo.toml: set enableRobotsTXT = true.
• site/layouts/robots.txt: new template, so we ship a considered policy rather than hugo's permissive built-in default (which is just User-agent: *).

The issue asks to "allow search indexing but disallow any nefarious behaviour." The template keeps indexing open for all real content, event images, event detail pages, and the calendar, and turns crawlers away from three things:

Rule	Why
Disallow: /addevent/edit-	netlify.toml rewrites /addevent/* to the add-event page with a 200, so the emailed management links (/addevent/edit-<series>-<secret>) are real, fetchable URLs. They are unlisted, but should never be indexed if one leaks. robots.txt matching is prefix-based, so this fences off the secret links while leaving the public /addevent/ page indexable.
Disallow: /api/ and /socialapi/	Both are netlify 200-rewrites straight through to our own api server. Crawling them is load on the backend with no indexable value.
Disallow: /404.html	A real page that hugo would otherwise offer up for indexing. (Path per #1083; see the dependency note below.)

Verification

Built with the same command netlify runs for production (npm run deploy calls hugo -s site) and confirmed the file lands at site/public/robots.txt, which is the publish directory declared in netlify.toml. None of the redirect rules in netlify.toml shadow /robots.txt, and netlify serves real files ahead of redirects regardless.

The design rests on prefix matching, so the built file was checked against a real parser (Python's urllib.robotparser) rather than by inspection:

allow  /                                    home page
allow  /pages/bike-summer/                  content page
allow  /calendar/                           calendar page
allow  /calendar/event-201                  event detail
allow  /events/                             vue SPA
allow  /addevent/                           public add-event page
allow  /eventimages/201-1.jpg               event image
allow  /sitemap.xml                         sitemap
BLOCK  /addevent/edit-3-s3cr3t              SECRET management link
BLOCK  /api/events.php?startdate=...        api proxy
BLOCK  /api/search.php?q=ride               api proxy
BLOCK  /socialapi/                          mastodon rss proxy
BLOCK  /404.html                            404 page

Worth noting that enableRobotsTXT does take effect even though [outputs] home is explicitly overridden in hugo.toml. Hugo treats robotstxt as its own page kind, not as a home output format, so the two do not interact. Confirmed against the pinned hugo 0.152.2.

No backend or frontend code is touched, so there is nothing for the app test suite to cover here.

Dependency on #1083

The 404 rule is written as Disallow: /404.html to match #1083 (clean up 404 page), which moves the page from hugo's default /404/index.html to /404.html. On main today the page is also reachable at /404.html (netlify serves it there for unmatched URLs, and a postdeploy copy places it there), so this rule is already correct against production and stays correct once #1083 lands. Whichever of the two PRs merges second, no further change to this rule is needed.

Known limitation: the sitemap is not yet valid

This PR stops the 404. It does not make the advertised sitemap work, and reviewers should not assume otherwise.

Because baseURL = "/", every entry in the production sitemap is a relative path:

<loc>/pages/bike-summer/</loc>

The sitemap protocol requires absolute URLs, so search engines will reject these. The Sitemap: line is included anyway: it costs nothing, it is correct the moment baseURL is fixed, and the template carries an inline comment saying as much. Happy to drop the line if reviewers would rather not advertise the sitemap until it is valid.

The fix is not a one-line baseURL change. The theme leans on absURL throughout, including for the calendar's own API calls:

cal/scripts.html:140:      url: '{{ absURL "api/events.php" }}',
head-content.html:29:  <link rel="icon" href="{{ absURL "favicon.ico" }}">

Setting baseURL to the production URL would make deploy previews and local dev load their assets from, and fetch their event data from, www.shift2bikes.org. Doing it properly means a per-context HUGO_BASEURL in netlify.toml (production gets the canonical URL, previews get $DEPLOY_PRIME_URL). That is its own change with its own blast radius, and is better tracked separately. The production sitemap also lists /404/, which this PR does not address.