Allow modules to provision alert rules

[Amygos]

Description

NS8 modules can already publish Prometheus scrape targets, Grafana datasources, and Grafana dashboards through Redis-backed keys that are consumed by the metrics module. They cannot publish their own Prometheus alert rules in the same supported way.

Add a supported integration that lets a module instance define alerting rules for the metrics stack. For example, a PostgreSQL module should be able to publish a PostgresqlDown rule together with its exporter target, then notify the metrics module so Prometheus loads or reloads the rule.

The feature benefits module authors and administrators because application-specific alerts can be shipped by the module that owns the service, instead of requiring manual changes in the generated rules.d directory.

Expected behavior:

• modules publish alert rules under a Redis hash such as module/<module_id>/metrics_alert_rules;
• a new metrics-alert-rules-changed event causes the metrics module to reprovision and reload alert rules;
• both a full Prometheus rule file with groups: and a single alert rule payload are accepted;
• only alerting rules are accepted; recording rules are rejected;
• invalid updates do not break Prometheus and keep the previous valid generated file;
• missing metric checks generate warnings only;
• the metrics module injects labels such as source_module_id and module_id;
• duplicate alert names generate warnings but are not rejected, with documentation recommending module/application-prefixed names such as PostgresqlDown;
• module-provided alerts are forwarded to Alertmanager, alert-proxy, Mimir, and email using the current default flow;
• existing experimental custom_alerts support is extended to use the same parser and validator, without strict backward compatibility for old malformed entries.

The planned implementation should add a generated file layout such as:

rules.d/provision_<module_id>_<name>.yml

and a new event handler:

imageroot/events/metrics-alert-rules-changed/15handler

Prometheus target updates currently rely on file_sd_configs and do not need a reload. Alert rule files are different, so the event should reload Prometheus with SIGHUP and fall back to systemctl --user try-restart prometheus.service if needed.

Alternative solutions

• Keep only the existing experimental custom_alerts mechanism. This is not enough because it is local to the metrics module and does not provide a clear cross-module contract.
• Require administrators to manage rules.d files manually. This conflicts with the generated-state model because files under rules.d can be overwritten by provisioning.
• Accept only full Prometheus rule files. Supporting single-rule payloads as well makes migration from the current experimental examples easier.
• Reject duplicate alert names. For the first implementation, warning is less disruptive; documentation should recommend unique module/application-prefixed names.

References

• Prometheus alerting rules: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
• Prometheus configuration reload: https://prometheus.io/docs/prometheus/latest/management_api/