logo
CHEATSHEETSTOOLSABOUT中文

K8s Troubleshooting - "Err:30"

Error

map[DriverName:filesystem Enclosed:map[Err:30 Op:mkdir Path:/var/lib/registry/docker/registry/v2/repositories/<project>/<repository>]]

This error commonly surfaces in Harbor registry pods running on Kubernetes.

Root Cause

Err:30 is the Linux kernel error code -EROFS — Error: Read-Only File System. The pod is attempting a write operation (mkdir) on a path that is backed by a read-only volume mount.

This typically happens when:

  • A PersistentVolume is mounted with accessModes: ReadOnlyMany instead of ReadWriteOnce or ReadWriteMany
  • The underlying storage (NFS, Ceph, etc.) remounted as read-only due to an I/O error or quota breach
  • A node-level disk issue caused the filesystem to flip to read-only mode as a self-protection measure
  • A misconfigured volumeMount in the pod spec has readOnly: true

Diagnosis

Step 1 — Find the registry pod:

POD=$(kubectl get pods -n <HARBOR_NAMESPACE> \
  -l goharbor.io/operator-controller=registry \
  -o name --kubeconfig=/path/to/kubeconfig)
echo $POD

Step 2 — Check the mount flags:

kubectl --kubeconfig=/path/to/kubeconfig \
  -n <HARBOR_NAMESPACE> exec $POD -- mount | grep /var/lib/registry

Look for ro in the mount options. A healthy mount will show rw:

/dev/sdb on /var/lib/registry type ext4 (rw,relatime)   ✅ healthy
/dev/sdb on /var/lib/registry type ext4 (ro,relatime)   ❌ read-only

Step 3 — Check PVC access mode:

kubectl get pvc -n <HARBOR_NAMESPACE> --kubeconfig=/path/to/kubeconfig

Verify ACCESS MODES is RWO or RWX, not ROX.

Step 4 — Check node-level disk health:

# SSH into the node hosting the pod
dmesg | grep -i "read-only\|remount\|I/O error" | tail -20
journalctl -k | grep -i "ext4\|xfs\|ceph" | tail -20

Resolution

Cause Fix
PVC mounted ReadOnly Update the PVC/PV to ReadWriteOnce and re-attach
Disk I/O error → auto-remount Investigate disk health (smartctl, fsck), then remount rw
Pod-level readOnly: true in volumeMount Remove readOnly: true from the container spec
Quota exceeded → forced ro Free space or increase quota, then remount

Quick fix — delete and recreate the pod (forces re-attachment of the volume):

kubectl delete $POD -n <HARBOR_NAMESPACE> --kubeconfig=/path/to/kubeconfig

The new pod will re-mount the PV. If the volume re-attaches as rw, the issue was transient. If it attaches as ro again, the root cause is in the storage layer and requires deeper investigation.

Prevention

  • Set resource quotas and storage limits to avoid disk exhaustion
  • Enable storage monitoring alerts (e.g., kubelet_volume_stats_available_bytes in Prometheus)
  • Use ReadWriteOnce for stateful single-replica workloads; avoid ReadOnlyMany for writable paths
Loading...