Skip to content

closes #429: Add timeout and circuit breaker for external blockchain RPC calls - #830

Open
sherifatolanike wants to merge 1 commit into
Smartdevs17:mainfrom
sherifatolanike:feature/rpc-timeout-circuit-breaker
Open

closes #429: Add timeout and circuit breaker for external blockchain RPC calls#830
sherifatolanike wants to merge 1 commit into
Smartdevs17:mainfrom
sherifatolanike:feature/rpc-timeout-circuit-breaker

Conversation

@sherifatolanike

Copy link
Copy Markdown

Description

Implements configurable timeout and circuit breaker pattern for external blockchain RPC calls to prevent cascading failures.

Changes

New files: backend/services/rpc/

  • rpcConfig.ts — Configuration types & defaults for RPC endpoints with timeout, circuit breaker settings, and provider fallback ordering per chain
  • circuitBreaker.ts — Full circuit breaker state machine (CLOSED → OPEN → HALF_OPEN → CLOSED) with events and typed errors
  • rpcProviderFallback.ts — RPC provider with automatic fallback, AbortController-based timeout, and graceful degradation (cache_stale / fail_fast)
  • rpcMonitorService.ts — Central monitoring dashboard, Prometheus metrics, per-chain health summaries, recent event log
  • index.ts — Module exports

Modified files

  • backend/services/shared/apiResponse.ts — Added 4 new RPC error codes
  • backend/services/index.ts — Exported all RPC module types and classes
  • backend/server.ts — Added 5 dashboard endpoints + provider initialization at startup

New API Endpoints

Endpoint Description
GET /rpc/dashboard Full circuit state dashboard with chain/endpoint filtering
GET /rpc/health/:chainId Per-chain health summary
POST /rpc/reset Manual circuit reset/open/reset_all/reset_metrics
GET /rpc/events Recent circuit state change events
GET /metrics/rpc Prometheus-formatted circuit metrics

Error Codes

  • RPC_TIMEOUT (504) — Request exceeded timeout
  • RPC_CIRCUIT_OPEN (503) — Circuit breaker is open, request blocked
  • RPC_ALL_PROVIDERS_FAILED (503) — All RPC providers exhausted
  • RPC_PROVIDER_NOT_FOUND (404) — No provider configured for chain

Technical Details

  • Configurable timeout per RPC endpoint (default: 10s)
  • Circuit breaker: 5 consecutive failures → OPEN, 30s recovery timeout → HALF_OPEN, 3 max probe requests
  • Provider fallback with ordered endpoint list per chain
  • Graceful degradation via stale cache responses when all providers fail
  • Manual circuit reset via API for operations intervention
  • Prometheus metrics integrated into existing monitoring infrastructure

Closes #429

…in calls

- Configurable timeout per RPC endpoint (rpcConfig.ts with per-chain defaults)
- Circuit breaker state machine (CLOSED/OPEN/HALF_OPEN) with failure threshold & recovery timeout
- RPC provider fallback across multiple endpoints per chain
- Circuit state monitoring dashboard (GET /rpc/dashboard, GET /rpc/health/:chainId)
- Graceful degradation on circuit open (cache_stale / fail_fast modes)
- Manual circuit reset (POST /rpc/reset with reset/open/reset_all/reset_metrics)
- Prometheus metrics at GET /metrics/rpc
- New error codes: RPC_TIMEOUT, RPC_CIRCUIT_OPEN, RPC_ALL_PROVIDERS_FAILED
@drips-wave

drips-wave Bot commented Jul 28, 2026

Copy link
Copy Markdown

@sherifatolanike Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add timeout and circuit breaker for external blockchain RPC calls

1 participant