Skip to content

fix(security): rate limiter fail-closed on Redis error, add Prometheus counter (#895) - #3

Open
Vincent6581 wants to merge 1 commit into
mainfrom
fix/895-rate-limiter-fail-closed
Open

fix(security): rate limiter fail-closed on Redis error, add Prometheus counter (#895)#3
Vincent6581 wants to merge 1 commit into
mainfrom
fix/895-rate-limiter-fail-closed

Conversation

@Vincent6581

@Vincent6581 Vincent6581 commented Jun 30, 2026

Copy link
Copy Markdown
Owner

Summary

Resolves solutions-plug#895.

The newsletter subscribe rate limiter returned true (allow) whenever Redis was unavailable. A Redis outage silently became an open door for newsletter spam or brute-force abuse — the opposite of the intended safety policy.

This PR inverts the error path to fail-closed (deny → 429), adds a Prometheus counter so operators are alerted during Redis degradation, and documents the fail-closed vs fail-open tradeoff inline.


Changes

services/api/src/newsletter.rs

Fail-closed behaviour

  • IpRateLimiter::allow() now returns false (deny → HTTP 429) on any Redis error instead of true (allow)
  • Log message updated to explicitly state fail-CLOSED semantics

Prometheus counter

  • Add optional metrics: Option<Metrics> and name: String fields to IpRateLimiter
  • Add IpRateLimiter::with_metrics(cache, metrics, name) constructor for production use
  • On every Redis error: metrics.observe_rate_limiter_redis_error(&self.name) increments rate_limiter_redis_errors_total{limiter="newsletter_subscribe"}
  • Existing IpRateLimiter::new(cache) constructor still works (metrics = None)

Documentation

  • Added module-level doc block explaining:
    • Why fail-closed is correct for security-critical rate limiters
    • When fail-open might be acceptable (non-security, read-only endpoints)
    • How to alert on rate_limiter_redis_errors_total

Test

  • limiter_fails_closed_when_redis_unavailable: creates a limiter pointed at port 16399 (nothing listening), calls allow(), asserts false is returned — confirms fail-closed path returns 429

services/api/src/metrics.rs

  • Add rate_limiter_redis_errors: IntCounterVec field with label limiter
  • Register rate_limiter_redis_errors_total with the Prometheus registry
  • Add observe_rate_limiter_redis_error(limiter: &str) public method

Acceptance criteria

  • Redis failure behaviour changed to fail-closed (return 429) for newsletter subscribe
  • rate_limiter_redis_errors_total Prometheus counter added, incremented on every Redis error
  • Test verifies Redis timeout triggers fail-closed path and returns deny (false → 429)
  • Fail-closed vs fail-open tradeoff documented in the rate limiting module

Alert recommendation

Add a CloudWatch metric filter + alarm on:

increase(rate_limiter_redis_errors_total[5m]) > 0

This will page on-call as soon as Redis degradation starts — before it escalates.

… Prometheus counter (solutions-plug#895)

The IpRateLimiter in src/newsletter.rs returned Ok(true) (allow) whenever
Redis was unavailable. A Redis outage silently became an open door for
newsletter spam or brute-force abuse.

Changes:

src/newsletter.rs
- IpRateLimiter.allow() now returns false (deny → 429) on Redis error instead
  of true (allow) — fail-closed behaviour for all security-critical limiters
- Add optional Metrics field; increment rate_limiter_redis_errors_total on
  every Redis failure so operators are alerted before an outage becomes abuse
- Add IpRateLimiter::with_metrics(cache, metrics, name) constructor for
  production use; IpRateLimiter::new(cache) still works without metrics
- Add 'name' field used as the 'limiter' label in the Prometheus counter
- Replace tracing::warn log message to make fail-closed semantics explicit
- Add module-level doc comment explaining fail-closed vs fail-open tradeoff,
  when fail-open is acceptable, and how to alert on the error counter
- Add limiter_fails_closed_when_redis_unavailable test: creates a limiter
  pointed at a non-listening port, calls allow(), asserts false is returned

src/metrics.rs
- Add rate_limiter_redis_errors: IntCounterVec field with label 'limiter'
- Register rate_limiter_redis_errors_total with the Prometheus registry
- Add observe_rate_limiter_redis_error(limiter: &str) helper method

Closes solutions-plug#895
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

In-memory rate limiter fails open when Redis is unavailable

1 participant