K8s Troubleshooting - "the object has been modified"

Error:

the object has been modified; please apply your changes to the latest version and try again

If you see this when manually applying a manifest

One possible cause is that you applied a dirty manifest (yaml), maybe previously dumped from an existing object, which contains fields such as creationTimestamp, resourceVersion, etc.

You can manually clean up the manifest, or use a command like this:

$ kubectl get KIND NAME -n NAMESPACE -o json | \
    jq "del(.status, .metadata.annotations, .metadata.creationTimestamp,
            .metadata.finalizers, .metadata.generation,
            .metadata.resourceVersion, .metadata.uid)" > object.json

If you see this in your controller

When you "Get" an object, the object has a resourceVersion inside ObjectMeta; if resourceVersion is changed before you submit your update, API server detects a diff in resourceVersion so it rejects your request.

Under the hood etcd stores a 64-bit int called revision for each object.

solution:

use RetryOnConflict

import (
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
)

err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
  if _, err := controllerutil.CreateOrUpdate(ctx, client, object, func() error {
    // ...
    return nil
  }); err != nil {
    return err
  }
  return nil
})
if err != nil {
    return fmt.Errorf("failed to update object: %w", err)
}