K8s Cluster Deployment Troubleshooting

This guide is compiled from a real troubleshooting session on a KubeKey-deployed KubeSphere environment. It covers complete diagnostic paths and resolutions for 6 categories of issues: NFS service startup failure, NFS Provisioner scheduling failure, repeated Worker node NotReady, scripts failing to execute on noexec mounts, missing sudo permission configuration, and missing node labels.

Issue 1: nfs-server Service Startup Failure (unit is masked)

Symptoms

Starting the NFS service reports:

text

failed to restart nfs-server.service
unit nfs-server.service is masked

Cause

masked is a more thorough state than disable; it means the service was explicitly blocked by the system administrator to prevent it being started by any means. Masking is typically implemented by creating a symlink pointing to /dev/null.

Solution

bash

# 1. Unmask the service
sudo systemctl unmask nfs-server.service

# 2. Start the service
sudo systemctl restart nfs-server.service

# 3. Enable on boot (optional)
sudo systemctl enable nfs-server.service

Additional Checks

If the service still fails to start after unmasking, check whether other dependent services are also masked:

bash

# Check for symlinks pointing to /dev/null
ls -l /etc/systemd/system/ | grep nfs

# Check related dependencies
systemctl status nfs-mountd rpc-statd

The NFS service depends on components such as nfs-mountd and rpc-statd. If any of them is similarly masked, unmask it too.

Issue 2: NFS Provisioner Scheduling Failure

Symptoms

bash

kubectl describe pod -n kube-system nfs-subdir-external-provisioner-xxx

Output:

text

Conditions:
  PodScheduled  False

Events:
  Warning  FailedScheduling
  0/2 nodes are available:
  1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: },
  1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }
  preemption: 0/2 nodes are available

Node-Selectors:  rise.io/manager=true

Cause

Two independent causes are stacked:

The master node has the control-plane taint and the Pod has no matching toleration, so it cannot be scheduled.
The worker node has the unreachable taint (caused by the node being NotReady), so the Pod cannot be scheduled either.
The Pod's Node-Selector requires the rise.io/manager=true label, but the node lacks it.

Solution

Step 1: Fix the worker node NotReady (see Issue 3)

Once the node returns to Ready, the unreachable taint disappears automatically.

Step 2: Label the node

bash

# Run on the master
kubectl label node 110236p.corp.scaniagroup.cn rise.io/manager=true

# Verify the label
kubectl get node 110236p.corp.scaniagroup.cn --show-labels

# Check whether the Pod is scheduled normally
kubectl get pod -n kube-system | grep nfs

Notes

rise.io/manager=true is a label used by KubeKey or storage components to designate the management node, pinning the NFS Provisioner to a specific node.

Issue 3: Repeated NotReady on Worker Node Caused by Port 8081 Occupation

Symptoms

The status of Worker node 110236p.corp.scaniagroup.cn keeps flipping between Ready and NotReady.

kubectl describe node shows:

text

Conditions:
  Type              Status   Reason
  Ready             Unknown  NodeStatusUnknown
  MemoryPressure    Unknown  Kubelet stopped posting node status
  DiskPressure      Unknown  Kubelet stopped posting node status
  PIDPressure       Unknown  Kubelet stopped posting node status

Taints:
  node.kubernetes.io/unreachable:NoExecute
  node.kubernetes.io/unreachable:NoSchedule

Lease:
  Failed to get lease: leases.coordination.k8s.io "..." not found

Key kubelet log errors:

text

E0914 ... kubelet_node_status.go] "Unable to register node with API server"
  err="Post \"https://127.0.0.1:6443/api/v1/nodes\": dial tcp 127.0.0.1:6443: connect: connection refused"

E0914 ... controller.go:145] "Failed to ensure lease exists, will retry"
  err="Get \"https://127.0.0.1:6443/apis/coordination.k8s.io/v1/...\":
  dial tcp 127.0.0.1:6443: connect: connection refused"

Root Cause Analysis

Failure chain:

text

McAfee endpoint security software occupies port 8081
        ↓
HAProxy static Pod cannot bind 8081, crashes on startup
        ↓
Local 127.0.0.1:6443 proxy unavailable
        ↓
kubelet cannot reach API Server → node NotReady
        ↓
Node gets the unreachable taint → other Pods cannot be scheduled

One additional issue was also found:

HAProxy config file permission problem: the host config file is owned by LDAP user gqu1vx with 640 permissions. The in-container HAProxy process runs as non-root and fails to read it with Permission denied.

Diagnostics

Step 1: Observe from the cluster side

bash

kubectl get nodes
kubectl describe node 110236p.corp.scaniagroup.cn

Step 2: SSH to the node and check kubelet

bash

systemctl status kubelet
journalctl -u kubelet --no-pager -n 100

Step 3: Locate the API Server connectivity issue

bash

grep server /etc/kubernetes/kubelet.conf

Expected: in a KubeKey HA deployment, this should be https://127.0.0.1:6443.

Step 4: Check the local HAProxy status

bash

crictl ps -a | grep haproxy
crictl logs <haproxy-container-id>

Finding 1: Config file permission problem

text

[ALERT] (1) : config : Could not open configuration file
  /usr/local/etc/haproxy/haproxy.cfg : Permission denied

bash

ls -l /etc/haproxy/haproxy.cfg
# -rwxr-xr-x. 1 gqulvx ot010005-scan iausers 1096 Sep 14 15:06 haproxy.cfg

Finding 2: Port conflict

text

[ALERT] (1) : Binding [/usr/local/etc/haproxy/haproxy.cfg:24]
  for frontend healthz: cannot bind socket (Address in use) for [0.0.0.0:8081]
[ALERT] (1) : [haproxy.main()] Some protocols failed to start their listeners! Exiting.

bash

netstat -anput | grep 8081
# tcp6  0  0 :::8081  :::*  LISTEN  3384/macmnsvc

Solution

Fix 1: Config file permissions

bash

chown root:root /etc/haproxy/haproxy.cfg
chmod 644 /etc/haproxy/haproxy.cfg

Note: do NOT change permissions on /etc/kubernetes/manifests/haproxy.yaml. That is the static Pod manifest read by kubelet and is unrelated to the in-container process permission issue.

Fix 2: Resolve the port conflict

Change the health-check port in the host config file:

bash

sed -i 's/bind :8081/bind :8082/' /etc/haproxy/haproxy.cfg

Also update the probe port in /etc/kubernetes/manifests/haproxy.yaml:

yaml

containers:
- name: haproxy
  image: ...
  livenessProbe:
    httpGet:
      path: /healthz
      port: 8082   # must match haproxy.cfg
  readinessProbe:
    httpGet:
      path: /healthz
      port: 8082   # must match haproxy.cfg

Key principle: the HAProxy healthz bind port must match the kubelet probe target port. If one changes, the other must change with it.

Verification

bash

# 1. Check port listeners
ss -tlnp | grep -E '8082|6443'

# 2. Simulate a health check
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8082/healthz
# Expected output: 200

# 3. Verify the 6443 proxy
curl -k https://127.0.0.1:6443/healthz
# Expected output: ok

# 4. Check container status
crictl ps | grep haproxy
# Expected: Running

# 5. Observe node status on the master
kubectl get nodes -w

KubeKey HA Architecture Notes

In a KubeKey HA deployment, every node runs a local HAProxy static Pod:

text

kubelet → 127.0.0.1:6443 (local HAProxy) → master:6443 (real API Server)

Design intent: kubelet does not need to know the real master IP; it always uses the local loopback address, so when the master changes, only the HAProxy config needs updating.
Side effect: once the local HAProxy fails, the node's kubelet immediately loses connectivity and the node turns NotReady. Always check local HAProxy status first when troubleshooting.

Issue 4: Scripts Cannot Execute on the /tmp Partition with noexec

Symptoms

Executing ./run.sh as root reports:

text

bash: ./run.sh: Permission denied

But bash ./run.sh works fine.

Cause

Difference between ./run.sh and bash ./run.sh:

Method	Needs x permission	Needs r permission	Uses shebang
`./run.sh`	✅ required	depends	✅ required
`bash ./run.sh`	❌ not needed	✅ required	❌ not needed

Even if the file has execute permission and the shebang is correct (#!/usr/bin/env bash), the kernel still refuses to execute it if the file resides on a noexec mount (e.g. /tmp).

Diagnostics

bash

# 1. Confirm file permissions
ls -l ./run.sh

# 2. Confirm the mount options of the partition containing the file
findmnt -T ./run.sh -o TARGET,SOURCE,FSTYPE,OPTIONS

# 3. Check whether noexec is present
mount | grep -E "$(df --output=target run.sh | tail -1)"

If noexec appears in OPTIONS, this is the cause.

Solution

Option A: Temporary remount (takes effect immediately, lost on reboot)

bash

sudo mount -o remount,exec /tmp

Option B: Permanent change (edit /etc/fstab)

bash

# Back up first
sudo cp /etc/fstab /etc/fstab.bak

# Edit fstab: remove noexec or replace with exec for the /tmp line
sudo vi /etc/fstab

From:

text

/dev/vdb1  /tmp  ext4  rw,noexec,relatime  0 2

To:

text

/dev/vdb1  /tmp  ext4  rw,exec,relatime  0 2

Option C: Move the script out of /tmp (safest)

bash

cp /tmp/run.sh /root/run.sh
cd /root && ./run.sh

If compliance requires /tmp to stay noexec, this option is recommended.

Notes

Even with x permission, if the script's first line lacks a shebang, ./run.sh falls back to /bin/sh, which can cause syntax incompatibilities.

Other causes of Permission denied include: filesystems without execute support (vfat/ntfs/exfat), ACL restrictions, SELinux/AppArmor interception, and noexec mounts inside containers.

Issue 5: sudo User Missing sudoers Configuration Causes Errors

Symptoms

KubeKey deployment reports:

text

Sorry, user gqu1vx is not allowed to execute '/bin/bash -c echo greetings,kubekey' as root on master, process exited with status 1

Cause

This error means sudo itself works, but the user does not match any passwordless sudo rule.

For LDAP/AD users, the username in the sudoers file must exactly match the login name returned by id or getent passwd.

If SSSD is configured with use_fully_qualified_names = True, the recognized username may be gqu1vx@your.domain.com instead of the short name gqu1vx.

Diagnostics

bash

# 1. Confirm the username sudo actually recognizes
id
whoami

# 2. Confirm whether the sudoers rule matches
sudo -l -U gqu1vx

# 3. Check sudoers file syntax
sudo visudo -c -f /etc/sudoers.d/kubekey

Solution

Option A: Configure with the short name

bash

echo "gqu1vx ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/kubekey
sudo chmod 440 /etc/sudoers.d/kubekey

Option B: Configure with the fully qualified name (if the system resolves users via FQDN)

bash

echo "gqu1vx@your.domain.com ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/kubekey

Option C: Grant via group (more robust)

bash

# Confirm the user's groups first
id gqu1vx

# Assuming the user is in the wheel or sudo group
echo "%wheel ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/kubekey

Note: the group name must be prefixed with %.

Verification

bash

# Non-interactive test: succeeds without a password
sudo -n true && echo "passwordless sudo configured" || echo "configuration not effective"

# Validate sudoers syntax
sudo visudo -c -f /etc/sudoers.d/kubekey
# Should output parsed OK

Additional Checks

Check whether /etc/sudoers includes /etc/sudoers.d/:

bash

grep includedir /etc/sudoers

Should output something like #includedir /etc/sudoers.d. If not, use visudo to append the rule directly at the end of the main file.

Check LDAP user resolution:

bash

id gqu1vx
getent passwd gqu1vx

If both commands return empty, the problem is in the LDAP client configuration (/etc/nsswitch.conf, /etc/sssd/sssd.conf, etc.), not in sudoers itself.

Why KubeKey Requires Passwordless sudo

KubeKey executes remote commands with sudo -E over a non-interactive SSH channel. When sudo requires a password, the prompt cannot be passed through, causing sudo: 3 incorrect password attempts even with a correct password. The official and community consensus is therefore to configure passwordless sudo for the deployment user in /etc/sudoers on the target node.

Issue 6: Components Fail to Schedule Because the rise.io/manager=true Label Is Missing

Symptoms

Pods of some components (e.g. NFS Provisioner) stay Pending. kubectl describe pod shows:

text

Node-Selectors:  rise.io/manager=true
Events:
  Warning  FailedScheduling
  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector

Cause

The Pod's Node-Selector requires nodes to carry the rise.io/manager=true label, but no node in the cluster satisfies this condition.

This label is typically used by KubeKey or storage components to designate the management node, pinning specific components to one node.

Solution

Run on the master node:

bash

# Label the target node
kubectl label node 110236p.corp.scaniagroup.cn rise.io/manager=true

# If the label exists with a different value, use --overwrite
kubectl label node 110236p.corp.scaniagroup.cn rise.io/manager=true --overwrite

Verification

bash

# Show all node labels
kubectl get node 110236p.corp.scaniagroup.cn --show-labels

# Show only this label's value
kubectl get node 110236p.corp.scaniagroup.cn -o jsonpath='{.metadata.labels.rise\.io/manager}'
# Expected output: true

# Check whether the Pod is scheduled normally
kubectl get pod -n kube-system | grep nfs
# Expected: Pending → Running

Notes

If other Pods later require this label too, labeling once is enough — all Pods that need it will match. If the label is supposed to be added automatically by KubeKey at a specific stage, also investigate why it was not added (e.g. interrupted deployment flow, an addon not executed).

Appendix A: Common Pitfalls Quick Reference

Pitfall	Description
Wrong file edited	`haproxy.yaml` (static Pod manifest) and `haproxy.cfg` (HAProxy config) are two different files
Port changed in one place only	After changing HAProxy's `bind` port, the YAML probe port must change too
LDAP user permissions	LDAP user UID/GID may not match inside the container, causing file read failures
Security software interference	McAfee/Trellix etc. often fight with K8s components for ports and hook syscalls
Residual taint	After the node returns to `Ready`, the `unreachable` taint does not disappear immediately; wait for the control plane to process it
Script execute permission	`./run.sh` needs x permission, `bash ./run.sh` only needs r permission; noexec mounts refuse execution
sudo username matching	The LDAP user's short name / fully qualified name must match what the system recognizes
Missing label	If the label required by Node-Selector is missing, Pods stay Pending forever

Appendix B: Worker Node NotReady Quick Checklist

When a Worker node is NotReady, check in this order:

[ ] kubectl describe node <node> — check Conditions and Taints
[ ] systemctl status kubelet — confirm kubelet process state
[ ] journalctl -u kubelet --no-pager | grep -i error — check key errors
[ ] grep server /etc/kubernetes/kubelet.conf — confirm the API Server address
[ ] crictl ps -a | grep haproxy — check local HAProxy status
[ ] crictl logs <haproxy-id> — check HAProxy logs
[ ] ls -l /etc/haproxy/haproxy.cfg — check config file permissions
[ ] netstat -anput | grep <port> — check port occupation
[ ] ss -tlnp | grep 6443 — confirm the local proxy listener
[ ] curl -k https://127.0.0.1:6443/healthz — verify proxy connectivity

Appendix C: Complete Failure Chain

text

McAfee occupies port 8081
        ↓
HAProxy fails to bind port → startup failure
        ↓
Local 127.0.0.1:6443 proxy unavailable
        ↓
kubelet cannot reach API Server → node NotReady
        ↓
Node gets the unreachable taint
        ↓
Pods such as NFS Provisioner cannot be scheduled
        ↓
Also: node lacks the rise.io/manager=true label → scheduling fails further

Document Info

Item	Value
Document version	v2.0
Applicable environment	KubeSphere / Kubernetes cluster deployed by KubeKey
Issues covered	6 categories
Last updated	2026-09-15

K8s Cluster Deployment Troubleshooting ​

Issue 1: nfs-server Service Startup Failure (unit is masked) ​

Symptoms ​

Cause ​

Solution ​

Additional Checks ​

Issue 2: NFS Provisioner Scheduling Failure ​

Symptoms ​

Cause ​

Solution ​

Notes ​

Issue 3: Repeated NotReady on Worker Node Caused by Port 8081 Occupation ​

Symptoms ​

Root Cause Analysis ​

Diagnostics ​

Solution ​

Verification ​

KubeKey HA Architecture Notes ​

Issue 4: Scripts Cannot Execute on the /tmp Partition with noexec ​

Symptoms ​

Cause ​

Diagnostics ​

Solution ​

Notes ​

Issue 5: sudo User Missing sudoers Configuration Causes Errors ​

Symptoms ​

Cause ​

Diagnostics ​

Solution ​

Verification ​

Additional Checks ​

Why KubeKey Requires Passwordless sudo ​

Issue 6: Components Fail to Schedule Because the rise.io/manager=true Label Is Missing ​

Symptoms ​

Cause ​

Solution ​

Verification ​

Notes ​

Appendix A: Common Pitfalls Quick Reference ​

Appendix B: Worker Node NotReady Quick Checklist ​

Appendix C: Complete Failure Chain ​

Document Info ​

K8s Cluster Deployment Troubleshooting

Issue 1: nfs-server Service Startup Failure (unit is masked)

Symptoms

Cause

Solution

Additional Checks

Issue 2: NFS Provisioner Scheduling Failure

Symptoms

Cause

Solution

Notes

Issue 3: Repeated NotReady on Worker Node Caused by Port 8081 Occupation

Symptoms

Root Cause Analysis

Diagnostics

Solution

Verification

KubeKey HA Architecture Notes

Issue 4: Scripts Cannot Execute on the /tmp Partition with noexec

Symptoms

Cause

Diagnostics

Solution

Notes

Issue 5: sudo User Missing sudoers Configuration Causes Errors

Symptoms

Cause

Diagnostics

Solution

Verification

Additional Checks

Why KubeKey Requires Passwordless sudo

Issue 6: Components Fail to Schedule Because the rise.io/manager=true Label Is Missing

Symptoms

Cause

Solution

Verification

Notes

Appendix A: Common Pitfalls Quick Reference

Appendix B: Worker Node NotReady Quick Checklist

Appendix C: Complete Failure Chain

Document Info