Kubernetes - Troubleshooting
... is attempting to grant RBAC permissions not currently held
Error:
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io "foo-cluster-role" is forbidden: user "[email protected]" (groups=["bar"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["nodes"], Verbs:["list"]}
Solution: use kubectl patch
to add the missing permission
$ kubectl patch clusterrole cluster-role-name \
--kubeconfig ${KUBECONFIG} \
--type='json' \
-p='[{"op": "add", "path": "/rules/0", "value":{ "apiGroups": [""], "resources": ["nodes"], "verbs": ["list"]}}]'
If kubectl patch
fails for the current user does not have the permission, so it cannot grant permission to this clusterrole.
: Check your kubeconfig, if there's another context with higher permissions, use the context:
$ kubectl config use-context admin-context
Then patch again.
Err:28
: map[DriverName:filesystem Enclosed:map[Err:28 Op:mkdir ...
Error: Err:28
map[DriverName:filesystem Enclosed:map[Err:28 Op:mkdir Path:/var/lib/registry/docker/registry/v2/repositories/<project>/<repository>]]
Root cause: not enough space.
Verification: check disk space of the harbor registry pod:
$ kubectl -n HARBOR_NAMESPACE exec HARBOR_REGISTRY_POD_NAME -- df -ah | less
Solution: resize the disk size for the registry.
# Get the pod
POD=$(kubectl get pods -n HARBOR_NAMESPACE -l goharbor.io/operator-controller=registry -o name --kubeconfig=/path/to/kubeconfig)
# Set the new size
STORAGE_SIZE=400Gi
# Patch PVC
kubectl patch Persistentvolumeclaim/harbor-registry \
--kubeconfig=/path/to/kubeconfig \
-n harbor-system --type=merge \
-p '{"spec": {"resources": {"requests": {"storage": "'$STORAGE_SIZE'"}}}}'
# Wait until the storage capacity is changed
kubectl --kubeconfig=/path/to/kubeconfig -n HARBOR_NAMESPACE exec $POD -- df -ah | grep "/var/lib/registry"
Err:30
Error: Err:30
map[DriverName:filesystem Enclosed:map[Err:30 Op:mkdir Path:/var/lib/registry/docker/registry/v2/repositories/<project>/<repository>]]
Root cause: Err 30 is -EROFS
, error due to writeback to read-only filesystem.
Verification:
# Get the pod.
POD=$(kubectl get pods -n HARBOR_NAMESPACE -l goharbor.io/operator-controller=registry -o name --kubeconfig=/path/to/kubeconfig)
kubectl --kubeconfig=/path/to/kubeconfig -n HARBOR_NAMESPACE exec $POD -- mount | grep /var/lib/registry
# Check if it is mounted as `ro`.
Solution: try to delete and recreate the pod and check if the volume is attached as rw
.
Object stuck in Terminating Status
Check the finalizers
of the object. Objects will not be removed until its metadata.finalizers
field is empty.
The target object remains in a terminating state while the control plane, or other components, take the actions defined by the finalizers.
https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/
message: 'The node was low on resource: ephemeral-storage.
Error
Pods are failing:
"message: 'The node was low on resource: ephemeral-storage."
Debug
Check disk usage
$ df -h
If the disk is indeed full, check what is taking up the disk spaces in /var/lib/kubelet
or /var/log
.