Skip to content

fix(healthcheck): probe domain nodes by node domain, not resolved ip#13743

Merged
AlinsRan merged 1 commit into
apache:masterfrom
AlinsRan:fix/healthcheck-node-domain-host
Jul 24, 2026
Merged

fix(healthcheck): probe domain nodes by node domain, not resolved ip#13743
AlinsRan merged 1 commit into
apache:masterfrom
AlinsRan:fix/healthcheck-node-domain-host

Conversation

@AlinsRan

@AlinsRan AlinsRan commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Description

With the default pass_host, an active health check on a domain-based HTTPS upstream always marks the node unhealthy, so the whole upstream is taken out of service and never recovers. The active probe sends the TLS SNI as the resolved ip, so the handshake fails against a backend whose certificate is issued for the domain. The same root cause affects plain HTTP: the probe sends Host: <resolved-ip>, hitting the wrong virtual host and getting a misleading result.

Root cause

An active health check runs independently of any request — there is no downstream client. The gateway itself is the client probing the node, so the probe must address the node by the node's own identity: its configured domain. pass_host is a request-forwarding rule (how a client request's Host is passed to the upstream); it has no meaning for a client-less probe.

compute_targets() in apisix/healthcheck_manager.lua nonetheless derives the probe Host header by reusing that request-path logic:

local use_node_hdr = up_conf.pass_host == "node" or nil
local host_hdr = up_hdr or (use_node_hdr and node.domain) or nil

For a domain node with the default pass_host (pass) this yields a nil Host header, and the health-check library then falls back to hostname, which is the resolved ip for a domain node. Both the probe Host header and the HTTPS SNI derive from this value, so both become the ip.

Real traffic is unaffected: for pass_host = node/rewrite the request Host/SNI is node.domain/upstream_host, and for pass the real client's Host is forwarded. Only the client-less health checker degrades to the ip.

Fix

local host_hdr = up_hdr or node.domain

Always carry node.domain as the probe Host header, except when pass_host is rewrite (which still pins upstream_host via up_hdr). The probe then addresses the node by its own identity, independent of pass_host. node.domain is nil for ip-based nodes, so ip upstreams are unaffected.

Per pass_host mode:

pass_host before after
node node.domain node.domain (unchanged)
rewrite upstream_host upstream_host (unchanged)
pass (default), domain node nil → probe uses resolved ip node.domain
any, ip node nil nil (unchanged)

Tests

t/node/healthcheck-node-domain-host.t, both with a domain node (test.com, resolved locally to 127.0.0.1) and the default pass_host:

  • TEST 2 (HTTPS SNI) — the failing case above: an https upstream + type: https active check; the TLS probe target logs $ssl_server_name. Asserts the probe's SNI is test.com, not 127.0.0.1.
  • TEST 1 (HTTP Host): the probe target logs the received Host header. Asserts the probe sends Host: test.com, not Host: 127.0.0.1.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change (no user-facing config or documented behavior change; this is an internal probe-addressing fix)
  • I have verified that the change follows the existing code style

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Jul 24, 2026
@AlinsRan
AlinsRan force-pushed the fix/healthcheck-node-domain-host branch from 95d3fd7 to 010c740 Compare July 24, 2026 03:27
@AlinsRan AlinsRan changed the title fix(healthcheck): probe domain nodes by the node domain, not the resolved ip fix(healthcheck): HTTPS domain upstreams marked unhealthy — probe with node domain, not resolved ip Jul 24, 2026
@AlinsRan AlinsRan changed the title fix(healthcheck): HTTPS domain upstreams marked unhealthy — probe with node domain, not resolved ip fix(healthcheck): probe domain nodes by node domain, not resolved ip Jul 24, 2026
…lved ip

An active health check has no downstream client -- the gateway itself is the
client probing the node, so it must address the node by the node's own identity
(its configured domain). pass_host only decides how a *client* request's Host is
forwarded, which is undefined here (there is no client Host).

compute_targets gated the probe Host header on pass_host == "node", so a
domain-based upstream with the default pass_host got a nil Host header. The
health-check library then falls back to the resolved ip, so the probe sends
Host: <ip> (hitting the wrong virtual host) and, over HTTPS, SNI: <ip> (the TLS
handshake fails), marking otherwise-healthy domain upstreams as unhealthy.

Always carry node.domain as the probe Host header (nil for ip nodes, so they are
unaffected), except when pass_host is "rewrite", which still pins upstream_host.
This makes the probe honor the documented default of checks.active.host being the
node host.

Tests cover both the HTTP Host header and the HTTPS SNI of the probe.
@AlinsRan
AlinsRan force-pushed the fix/healthcheck-node-domain-host branch from 010c740 to 550f96b Compare July 24, 2026 03:43

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@AlinsRan AlinsRan self-assigned this Jul 24, 2026
@AlinsRan
AlinsRan merged commit fcefe6d into apache:master Jul 24, 2026
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants