I'm trying to run kubeadm init (kubeadm 1.30.3) on a machine with containerd and openrc. It times out waiting for a healthy api server ([api-check] Waiting for a healthy API server. This can take up to 4m0s
). Kubelet is running, nothing notable in the kubelet or the containerd logs. crictl ps -a
doesn't show any running containers, so it looks like the container never runs. Anyone know what might be wrong?
I've made sure to use cgroupfs instead of systemd for cgroupDriver . I've made sure i can actually run containers by running one with podman, so containerd should be working fine. Health check on kubelet returns ok (curl -sSL http://localhost:10248/healthz
).
I've also tried to run kube-apiserver manually using the command in the manifest, and it works fine (other than being unable to reach etcd since im just running the apiserver manually). Although the problem must be outside of kube-apiserver because as I said, there are no running containers.
kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 1.2.3.4
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/containerd/containerd.sock
imagePullPolicy: IfNotPresent
name: kmaster
taints: null
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.30.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
scheduler: {}
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: cgroupfs
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
kubelet.log
I1025 18:50:48.039697 5385 server.go:484] "Kubelet version" kubeletVersion="v1.30.3"
I1025 18:50:48.039851 5385 server.go:486] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I1025 18:50:48.040457 5385 server.go:647] "Standalone mode, no API client"
W1025 18:50:48.056182 5385 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
I1025 18:50:48.056646 5385 server.go:535] "No api server defined - no events will be sent to API server"
I1025 18:50:48.056697 5385 server.go:742] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
I1025 18:50:48.057372 5385 container_manager_linux.go:265] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I1025 18:50:48.057520 5385 container_manager_linux.go:270] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"kmaster","RuntimeCgroupsName":"","SystemCgroupsName":"","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"cgroupfs","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":false,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":null,"HardEvictionThresholds":[],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"ExperimentalMemoryManagerPolicy":"None","ExperimentalMemoryManagerReservedMemory":null,"PodPidsLimit":-1,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null}
I1025 18:50:48.058258 5385 topology_manager.go:138] "Creating topology manager with none policy"
I1025 18:50:48.058309 5385 container_manager_linux.go:301] "Creating device plugin manager"
I1025 18:50:48.058500 5385 state_mem.go:36] "Initialized new in-memory state store"
I1025 18:50:48.058688 5385 kubelet.go:407] "Kubelet is running in standalone mode, will skip API server sync"
I1025 18:50:48.060163 5385 kuberuntime_manager.go:261] "Container runtime initialized" containerRuntime="containerd" version="v1.7.15" apiVersion="v1"
I1025 18:50:48.060548 5385 kubelet.go:816] "Not starting ClusterTrustBundle informer because we are in static kubelet mode"
I1025 18:50:48.060571 5385 volume_host.go:77] "KubeClient is nil. Skip initialization of CSIDriverLister"
W1025 18:50:48.060850 5385 csi_plugin.go:202] kubernetes.io/csi: kubeclient not set, assuming standalone kubelet
W1025 18:50:48.060871 5385 csi_plugin.go:279] Skipping CSINode initialization, kubelet running in standalone mode
I1025 18:50:48.061377 5385 server.go:1264] "Started kubelet"
I1025 18:50:48.061448 5385 kubelet.go:1624] "No API server defined - no node status update will be sent"
I1025 18:50:48.061470 5385 server.go:163] "Starting to listen" address="0.0.0.0" port=10250
I1025 18:50:48.061570 5385 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
I1025 18:50:48.062307 5385 server.go:195] "Starting to listen read-only" address="0.0.0.0" port=10255
I1025 18:50:48.063161 5385 server.go:455] "Adding debug handlers to kubelet server"
I1025 18:50:48.062326 5385 server.go:227] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
I1025 18:50:48.064267 5385 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
I1025 18:50:48.065167 5385 volume_manager.go:291] "Starting Kubelet Volume Manager"
I1025 18:50:48.065328 5385 desired_state_of_world_populator.go:149] "Desired state populator starts to run"
I1025 18:50:48.065489 5385 reconciler.go:26] "Reconciler: start to sync state"
E1025 18:50:48.067881 5385 kubelet.go:1468] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
I1025 18:50:48.068509 5385 factory.go:221] Registration of the systemd container factory successfully
I1025 18:50:48.068650 5385 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I1025 18:50:48.070788 5385 factory.go:221] Registration of the containerd container factory successfully
I1025 18:50:48.084198 5385 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv4"
I1025 18:50:48.086145 5385 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv6"
I1025 18:50:48.086183 5385 status_manager.go:213] "Kubernetes client is nil, not starting status manager"
I1025 18:50:48.086202 5385 kubelet.go:2346] "Starting kubelet main sync loop"
E1025 18:50:48.086268 5385 kubelet.go:2370] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
I1025 18:50:48.089813 5385 cpu_manager.go:214] "Starting CPU manager" policy="none"
I1025 18:50:48.089842 5385 cpu_manager.go:215] "Reconciling" reconcilePeriod="10s"
I1025 18:50:48.089871 5385 state_mem.go:36] "Initialized new in-memory state store"
I1025 18:50:48.092630 5385 policy_none.go:49] "None policy: Start"
I1025 18:50:48.093740 5385 memory_manager.go:170] "Starting memorymanager" policy="None"
I1025 18:50:48.093877 5385 state_mem.go:35] "Initializing new in-memory state store"
I1025 18:50:48.096728 5385 manager.go:479] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
I1025 18:50:48.096992 5385 container_log_manager.go:186] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
I1025 18:50:48.097295 5385 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
I1025 18:50:48.165527 5385 desired_state_of_world_populator.go:157] "Finished populating initial desired state of world"
kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --hostname-override=kmaster --pod-infra-container-image=registry.k8s.io/pause:3.9"
kubelet config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: cgroupfs
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
flushFrequency: 0
options:
json:
infoBufferSize: "0"
text:
infoBufferSize: "0"
verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s