No mDNS records are published

[PAI5REECHO]

In my external-dns Pod logs it is correctly reporting/publishing the mDNS records I want for the Ingresses I have configured in my cluster, but no mDNS records are ever actually published over the network. I captured all the IGMP & UDP port 5353 traffic from my K8s node running external-mdns with tcpdump -i any 'udp port 5353 or igmp' -w /tmp/mdns.pcap and all I see are some IGMP membership report packets being sent out but nothing else (nothing on mDNS ports).

[nlflint]

Does your external-mDNS deployment/pod have hostNetwork: true? It needs access to the nodes host interface for the multicast packets to leave the cluster.

[tdrozdowski]

I've run into this same thing using microk8s in strict mode - I see its creating the entry and I have hostNetwork: true but no dice.

2024/12/15 20:25:26 Added 1.0.0.127.in-addr.arpa. 120 IN PTR grafana.lab1.local.

[rr4444]

Same ( I think) on k3s. Avahi publishing, but that doesn't result in getent/ping/browser/etc resolving fine.

1. Stock k3s and nothing installed yet except traefik, certman and kubevip
2. Stock raspbian on nodes
3. Node where e-mdns pod is runs don't show in tcpdump on that node
4. Pod logs show detection of traefik service e.g. showing traefik.traefik2.local
5. Am a confused why the pod logs show some attempt to access a kube config file, which seems odd

$ ping traefik.traefik2.local
ping: traefik.traefik2.local: Name or service not known
$ getent hosts traefik.trafeik2.local
$ avahi-resolve --name traefik.traefik2.local
traefik.traefik2.local  172.16.3.201

2025-04-07T20:38:34.866733865+01:00 2025/04/07 19:38:34 app.config [default-namespace:"default" expose-ipv4:"true" expose-ipv6:"false" kubeconfig:"/nonexistent/.kube/config" master:"" namespace:"" publish-internal-services:"false" record-ttl:"120" source:"[ingress service]" test:"false" without-namespace:"false"]
2025-04-07T20:38:34.936084678+01:00 2025/04/07 19:38:34 Added traefik.traefik2.local. 120 IN A 172.16.3.201
2025-04-07T20:38:34.936338508+01:00 2025/04/07 19:38:34 Added 201.3.16.172.in-addr.arpa. 120 IN PTR traefik.traefik2.local.
2025-04-07T20:38:34.936394748+01:00 2025/04/07 19:38:34 Added traefik-traefik2.local. 120 IN A 172.16.3.201
2025-04-07T20:38:34.936420618+01:00 2025/04/07 19:38:34 Added 201.3.16.172.in-addr.arpa. 120 IN PTR traefik-traefik2.local.
2025-04-07T20:38:34.866733865+01:00 2025/04/07 19:38:34 app.config [default-namespace:"default" expose-ipv4:"true" expose-ipv6:"false" kubeconfig:"/nonexistent/.kube/config" master:"" namespace:"" publish-internal-services:"false" record-ttl:"120" source:"[ingress service]" test:"false" without-namespace:"false"]
2025-04-07T20:38:34.936084678+01:00 2025/04/07 19:38:34 Added traefik.traefik2.local. 120 IN A 172.16.3.201
2025-04-07T20:38:34.936338508+01:00 2025/04/07 19:38:34 Added 201.3.16.172.in-addr.arpa. 120 IN PTR traefik.traefik2.local.
2025-04-07T20:38:34.936394748+01:00 2025/04/07 19:38:34 Added traefik-traefik2.local. 120 IN A 172.16.3.201
2025-04-07T20:38:34.936420618+01:00 2025/04/07 19:38:34 Added 201.3.16.172.in-addr.arpa. 120 IN PTR traefik-traefik2.local.

[rr4444]

(fixed comment code blocks)

I should have also said that those command line results are identical on a random laptop on the LAN, and on the node host the pod is running on. And mdns4 is in nsswitch.conf

Proper summary with help of Gemini summarising my logs:

I'm facing a persistent issue where standard Linux resolvers (getent, ping) fail to resolve mDNS .local names advertised by external-mdns on Raspbian nodes, even though direct mDNS queries (avahi-resolve) work perfectly for the same names, and resolution works for names advertised by native Avahi daemons.

Environment:

• Cluster: K3s v1.31.6+k3s1 on Raspberry Pi 4 nodes (ARM64)
• Node OS: Raspberry Pi OS (Debian Bookworm based)
• Networking: Kube-VIP providing LoadBalancer IPs (e.g., 172.16.3.201 for Traefik Service), likely ARP mode. Nodes have wired (eth0 at 172.16.3.x) and wireless (wlan0 at 172.16.0.x) interfaces.
• mDNS Publisher: external-mdns (from git clone of blake/external-mdns master/recent version), deployed as a Kubernetes Deployment.
• Node mDNS Setup: Native avahi-daemon running, libnss-mdns installed (v0.15.1-3), /etc/nsswitch.conf hosts: line is files mdns4_minimal [NOTFOUND=return] dns. systemd-resolved and nscd are not active or installed.

Configuration:

• external-mdns Deployment uses hostNetwork: true.
• Target Service (Traefik LoadBalancer in traefik2 namespace, VIP 172.16.3.201) is annotated: external-mdns.alpha.kubernetes.io/hostname: traefik.traefik2.local (this is the primary name I am testing resolution for).

Problem Summary & Diagnostics:

1. Direct mDNS Query Success: avahi-resolve --name traefik.traefik2.local works correctly when run on the K3s nodes (e.g., rpinode3) and on other LAN devices (e.g., Linux laptop), returning the correct VIP (172.16.3.201). This confirms external-mdns is successfully publishing the A record onto the LAN via the host node's network interface, and the mDNS packets are reachable and parsable by avahi-resolve.
2. Standard NSS Resolver Failure: getent hosts traefik.traefik2.local (and subsequently ping) fails (returns nothing) when run on the K3s nodes themselves (e.g., rpinode3).
3. Native Avahi Records Work via NSS: getent hosts rpinode1.local (where rpinode1.local is advertised by the native avahi-daemon on another Pi node) works correctly using the same getent command on the same K3s nodes (e.g., rpinode3).
4. NSS Config Appears Correct: As noted, /etc/nsswitch.conf seems standard and correct for using libnss-mdns (mdns4_minimal appears before dns), and the libnss-mdns package is installed. No other known conflicting resolvers (systemd-resolved, nscd) are active.
5. IPv4/IPv6 Difference: The working native Avahi records (rpinode1.local) include both A (IPv4) and AAAA (IPv6) records. The failing external-mdns record (traefik.traefik2.local) includes only an A record (IPv4), as IPv6 publishing is disabled in external-mdns.
6. Packet Capture (tcpdump): Confirmed mDNS announcement packets for traefik.traefik2.local with the correct A record are visible leaving the host node's interface when external-mdns starts/advertises (correcting an earlier misinterpretation - the packets are indeed sent, aligning with avahi-resolve working).
7. Attempted Fixes (No Effect on getent failure for external-mdns name):
   • Restarting node avahi-daemon.
   • Changing nsswitch.conf from mdns4_minimal to mdns_minimal.
   • Reconfiguring external-mdns deployment with --publish-ptr=false.

Core Question & Hypothesis:
Why does libnss-mdns (v0.15.1-3 on Raspbian/Debian Bookworm), when invoked via the standard NSS interface (getent, ping, etc.), successfully resolve .local hostnames advertised by a native avahi-daemon (containing A+AAAA records), but consistently fail to resolve .local hostnames advertised by external-mdns (containing only an A record), even though direct mDNS query tools (avahi-resolve) confirm the latter record is valid, present on the network, and resolvable at the mDNS protocol level?

This points strongly towards a potential incompatibility, bug, or specific behavioral quirk within this version of libnss-mdns where it mishandles or ignores mDNS A records via the NSS path when they originate from external-mdns or perhaps when they lack a corresponding AAAA record, despite nsswitch.conf specifying mdns4_minimal.

Request:
I am looking for a solution that allows standard system resolvers on the Raspbian nodes (and other diverse LAN clients) to resolve the .local names published by external-mdns, without resorting to manual node/client configuration changes (nsswitch.conf, hosts files).

Could you provide insights into this behavior?

• Is this a known issue or limitation with libnss-mdns v0.15.1-3 on Debian Bookworm/Raspbian (arm64)?
• Are there specific flags, configurations, or workarounds for external-mdns (blake/external-mdns) known to improve compatibility with libnss-mdns in such scenarios (e.g., related to record formatting, TTL, specific TXT records, interface binding nuances)?
• Would attempting to publish a dummy IPv6 record (if feasible with Kube-VIP) potentially work around this suspected libnss-mdns issue?
• Are there alternative Kubernetes mDNS publishers known to be more compatible with libnss-mdns on this platform?

Any guidance on resolving this discrepancy between direct mDNS resolution (avahi-resolve) and NSS-based resolution (getent) for records published by external-mdns on these Debian-based systems would be greatly appreciated. Thank you!

[rr4444]

Updated a few silly hallucinations and rudeness by Gemini

[rr4444]

Ok, I worked it out..... mdns4_minimal for linux host resolution does not allow for service.namespace.local. There are too many levels, so service-namespace.local does work. This is an important consideration for out of the box resolution on clients. I think this will affect android too.