Implement API response streaming for large datasets

[attyolu]

##closes #768

---

Commit Message

feat: implement API response streaming for large datasets (#768)

- Adds AsyncCursorStream and base64url cursor codec in backend/services/shared/streaming.ts
- Implements MemoryMonitor to check RSS and heap usage against thresholds
- Builds SseEmitter with event frames, heartbeat pinging, and client disconnect handlers
- Adds async generator streaming to accountingExportService.ts and subscriptionEventStore.ts
- Mounts routes for NDJSON list streaming, SSE export progress, streaming downloads, and memory metrics
- Adds Hermes-compatible streamNdjson and subscribeToSse client utilities in streamingService.ts
- Adds useStreamingList (with synchronous double-fetch protection) and useExportStream hooks
- Integrates streaming batch creation in batchTransactionService.ts
- Adds comprehensive unit/integration tests and architectural/API documentation

---

Pull Request Description

# Pull Request: API Response Streaming for Large Datasets (#768)

## 📌 Summary

Resolves issue **#768**. Large datasets previously loaded all data into memory at once, causing high memory consumption and potential out-of-memory errors. 

This PR introduces response streaming via NDJSON line-delimited streams, cursor-based pagination, Server-Sent Events (SSE) for progress indicators, server memory monitoring, streaming export downloads, and React client hooks built for mobile/Hermes compatibility.

---

## 🚀 Key Changes

### 1. Backend Infrastructure & Utilities
- **`backend/services/shared/streaming.ts`**:
  - Implemented URL-safe, opaque `base64url` cursor encoders (`encodeOpaqueCursor`, `decodeOpaqueCursor`).
  - Created `createCursorStream` async generator for yielding records on demand.
  - Implemented `MemoryMonitor` to track RSS and heap memory against defined performance budgets.
  - Built NDJSON parser and generator helpers (`toNdjsonLine`, `parseNdjsonBuffer`).
- **`backend/services/shared/sseEmitter.ts`**:
  - Implemented `SseEmitter` wrapping HTTP `ServerResponse` with SSE event framing (`progress`, `chunk`, `complete`, `error`), automatic 15s keep-alive heartbeats, and client disconnect cleanup.

### 2. Backend Services & Streaming Routes
- **`backend/services/billing/accountingExportService.ts`**: Added `streamExportAsync`, `streamExportNdjson`, and `streamExportWithProgress` for progress-driven SSE streams.
- **`backend/services/subscription/subscriptionEventStore.ts`**: Added `queryStream` generator for cursor-based event iteration.
- **`backend/server.ts`**: Added 4 streaming routes:
  - `GET /subscriptions/stream` — NDJSON list streaming endpoint.
  - `GET /exports/stream/:exportId` — Real-time progress SSE stream endpoint.
  - `GET /exports/download/:token` — Streaming export file downloader.
  - `GET /metrics/memory` — Real-time process memory metrics.

### 3. Client Services & React Hooks
- **`app/services/streamingService.ts`**:
  - Implemented `fetchCursorPage`, `collectAllPages`, `streamNdjson`, and `subscribeToSse`.
  - Handled relative URL parsing robustly for test/browser/mobile environments.
- **`app/services/hooks/useStreamingList.ts`**: React hook for cursor-paginated list rendering with synchronous `loadingRef` protection against duplicate concurrent fetches.
- **`app/services/hooks/useExportStream.ts`**: React hook for tracking multi-stage SSE progress and export downloads.
- **`app/services/batchTransactionService.ts`**: Added `executeBatchCreateStream` and `memoryPressure()` estimation helpers.

### 4. Documentation
- Created **`developer-portal/docs/streaming-api.md`** and updated **`developer-portal/docs/api-reference.md`**.
- Created **`docs/streaming-architecture.md`**.

---

## 🧪 Verification & Testing

All **49 unit & integration tests** across 4 test suites passed:

```bash
npm run test -- --testPathPattern=streaming

PASS app/services/hooks/__tests__/useStreamingList.test.ts
PASS app/services/__tests__/streamingService.test.ts
PASS backend/services/billing/__tests__/accountingExportService.streaming.test.ts
PASS backend/services/shared/__tests__/streaming.test.ts

Test Suites: 4 passed, 4 total
Tests:       49 passed, 49 total

[drips-wave]

@attyolu Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits