Skip to content

feat(observability): add Prometheus alerting rules and Grafana dashboard (closes #349) - #494

Open
beulah7717108-eng wants to merge 2 commits into
XStreamRollz:mainfrom
beulah7717108-eng:feat/349-observability-monitoring
Open

feat(observability): add Prometheus alerting rules and Grafana dashboard (closes #349)#494
beulah7717108-eng wants to merge 2 commits into
XStreamRollz:mainfrom
beulah7717108-eng:feat/349-observability-monitoring

Conversation

@beulah7717108-eng

Copy link
Copy Markdown

Summary

Closes #349 — No alerting rules or Grafana dashboard shipped with the project.

Changes

New: monitoring/ directory

prometheus-rules.yaml — Production-ready alerting rules:

Alert Severity Trigger
HighErrorRate critical 5xx > 5% for 5 min
HighLatency warning API p99 > 1s for 5 min
WebSocketConnectionsZero warning No active WS for 5 min
DatabasePoolSaturation warning Waiting > idle for 2 min
WorkerRestarts warning Uptime counter reset
HighQueueDepth warning Queue > 200 for 10 min
CriticalQueueDepth critical Queue > 500 for 5 min
WorkerHighErrorRate warning Errors > 10/s for 5 min
LockRenewalFailures warning Renewal failures

Also includes recording rules for job:http_request_duration_seconds:p50/p90/p99.

grafana-dashboard.json — Pre-built dashboard covering:

  • API: request rate by path, latency percentiles, error rate gauge, WebSocket connections, DB pool
  • Worker: queue depth, throughput (messages vs errors), worker status, lock activity, active locks
  • Database: connection pool (active/idle/waiting)

Updated: k8s/README.md

  • Comprehensive Monitoring section with:
    • Alert rules table with severities
    • promtool check rules validation instructions
    • Installation for kube-prometheus-stack and standalone Prometheus
    • Grafana dashboard import (UI + ConfigMap/GitOps)
    • Prometheus scrape configuration examples
    • NetworkPolicy considerations for cross-namespace monitoring
    • Datasource UID customization note

Validation

promtool check rules monitoring/prometheus-rules.yaml
# Expected: SUCCESS: ... rules found, 0 invalid

peaceshallom37-rgb and others added 2 commits July 30, 2026 00:28
The api Docker build was failing because the build context (api/) could
not resolve file: workspace dependencies (`@xstreamroll/types`,
`@xstreamroll/contract-tests`, and the root `xstreamroll` package).

Changes:
- Rewrite api/Dockerfile to use repo root as build context, copying
  workspace packages (packages/types, tests/contracts, root) and
  database/migrations into the builder stage
- Add `rxjs` as a direct dependency in api/package.json (imported by
  source code but previously only available transitively via NestJS)
- Regenerate api/pnpm-lock.yaml (was out of sync with package.json,
  causing ERR_PNPM_OUTDATED_LOCKFILE)
- Create root .dockerignore to exclude node_modules, dist, .next, etc.
  from the Docker build context
- Update docker-compose.yml api service: context changed from ./api to
  ., with dockerfile: api/Dockerfile
- Update security-scan.yml, release.yml, docker-scan.yml: api matrix
  entries now use context: . with dockerfile: api/Dockerfile

Closes the api Docker build failure in the security-scan workflow.

Co-Authored-By: Freebuff <noreply@freebuff.com>
The app Docker build was failing because the build context (app/) could
not resolve the file: workspace dependency @xstreamroll/types
(file:../packages/types).

Changes:
- Rewrite app/Dockerfile to use repo root as build context, copying
  packages/types into the builder stage so npm can resolve the
  file: dependency. Merge deps+builder stages to avoid cross-stage
  symlink breakage.
- Add RUN mkdir -p public to ensure the optional Next.js public/
  directory exists for the runner stage.
- Update docker-compose.yml app service: context changed from ./app to
  ., with dockerfile: app/Dockerfile
- Update security-scan.yml, release.yml, docker-scan.yml: app matrix
  entries now use context: . with dockerfile: app/Dockerfile

Closes the app Docker build failure in the security-scan workflow.
@beulah7717108-eng
beulah7717108-eng force-pushed the feat/349-observability-monitoring branch from 1489bb8 to 521c328 Compare July 30, 2026 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

observability: No alerting rules or Grafana dashboard shipped with the project

2 participants