Skip to content

Service debugging methodology

In Kubernetes, the Service debugging methodology is a systematic approach to troubleshooting connectivity issues within the cluster. It relies on understanding the relationship between DNS records, iptables/IPVS rules, and Endpoints.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

DNS Verification

When a Service cannot be accessed via its DNS name, the first step is to isolate whether the issue lies with the Service configuration or the cluster DNS. A recommended approach is to query the Kubernetes Master Node's internal Service DNS record.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

For example, checking kubernetes.default should return the API Server's Service IP (e.g., 10.0.0.1). If this fails, the issue lies with kube-dns; if it succeeds, the issue likely lies with the specific Service's configuration.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

Endpoint Inspection

If DNS resolution is successful, the next step is to verify the Service's Endpoints. Endpoints are the Pods selected by the Service, and only Pods in the Running state with a successful readinessProbe are included in this list.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

You can check Endpoints using kubectl get endpoints <service-name>. If the list is empty or missing expected Pods, it indicates that the Pods are either not matching the Service's selector or are failing their readiness checks.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

Proxy and Rule Analysis

If Endpoints are populated correctly but connectivity is still failing, the issue lies with the network rules controlled by kube-proxy.

Kube-proxy Status

Verify that kube-proxy is running correctly. In a healthy cluster (e.g., one deployed via kubeadm), logs should indicate it is "Setting endpoints" for various services and syncing iptables rules.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

iptables Verification

For clusters using the iptables proxy mode, debugging involves inspecting the specific chains on the host. A functioning Service corresponds to a specific set of rules:^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

  1. KUBE-SERVICES / KUBE-NODEPORTS: These serve as the entry chains and should correspond to the Service's VIP (ClusterIP) and port.
  2. KUBE-SVC-(hash): This chain handles load balancing. The number of rules in this chain should match the number of Endpoints.
  3. KUBE-SEP-(hash): These are DNAT chains that forward traffic to specific Pod IPs; they should correspond one-to-one with the Endpoints.
  4. NodePort SNAT: If using NodePort, there should be a KUBE-POSTROUTING rule handling MASQUERADE for Source NAT.

IPVS Verification

In large-scale clusters, IPVS mode is preferred to avoid the CPU overhead of refreshing thousands of iptables rules.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md] Debugging here involves checking the kube-ipvs0 virtual interface and using ipvsadm -ln to ensure the VIP and Pod backends are correctly registered in the kernel's IPVS table.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

Hairpin Mode (Self-Access)

A specific issue occurs when a Pod cannot access its own Service. This is often caused by the hairpin-mode setting not being configured correctly on the node.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

  • hairpin-veth: Check that the Hairpin mode is set to 1 on the VETH devices inside the CNI bridge (e.g., /sys/devices/virtual/net/cni0/brif/veth*/hairpin_mode).^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]
  • promiscuous-bridge: Check that the CNI bridge (e.g., cni0) has PROMISC (promiscuous mode) enabled.^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]

Sources

^[400-devops-06-kubernetes-k8s-paas-kubernetes-yaml.md]