logo

K8s Troubleshooting - "Err:28"

Error: Err:28

map[DriverName:filesystem Enclosed:map[Err:28 Op:mkdir Path:/var/lib/registry/docker/registry/v2/repositories/<project>/<repository>]]

Root cause: not enough space.

Verification: check disk space of the harbor registry pod:

$ kubectl -n HARBOR_NAMESPACE exec HARBOR_REGISTRY_POD_NAME -- df -ah | less

Solution: resize the disk size for the registry.

# Get the pod
POD=$(kubectl get pods -n HARBOR_NAMESPACE -l goharbor.io/operator-controller=registry -o name --kubeconfig=/path/to/kubeconfig)

# Set the new size
STORAGE_SIZE=400Gi

# Patch PVC
kubectl patch Persistentvolumeclaim/harbor-registry \
    --kubeconfig=/path/to/kubeconfig \
    -n harbor-system --type=merge \
    -p '{"spec": {"resources": {"requests": {"storage": "'$STORAGE_SIZE'"}}}}'

# Wait until the storage capacity is changed
kubectl --kubeconfig=/path/to/kubeconfig -n HARBOR_NAMESPACE exec $POD -- df -ah | grep "/var/lib/registry"