InvalidSpec: The Rollout "canary-release" is invalid: spec.strategy.canary.canaryService: Invalid value: "canary": Service "canary" has unmatch label "app.kubernetes.io/version" in rollout

[cam634]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

We had a issue when in the middle of the rollout we ran into middle of the rollout step 5 when the rollout went into a failed state with I validated that all labels still existed on our kubernetes canary service and they match what were on our canary pods

InvalidSpec: The Rollout "canary-release" is invalid: spec.strategy.canary.canaryService: Invalid value: "canary": Service "canary" has unmatch label "app.kubernetes.io/version" in rollout

This processed worked through the first 4 steps before the rollout started failing. We run multiple microservices all with the same configuration and the rest of the mirco-services were able to deploy with no issues

To Reproduce

Could not reproduce
Expected behavior

The canary promotes as expected since all labels are available or if the labels did not exist we would expect the rollout to fail in this manner

Screenshots

Version
v1.5.0
Logs

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

```bash
time="2024-11-14T14:24:59Z" level=info msg="Updating replica set 'canary-release-' revision from 12 to 14" namespace= prod rollout=canary-release

time="2024-11-14T14:24:59Z" level=info msg="Enqueueing parent of prod/canary-release-: Rollout prod/canary-release"

time="2024-11-14T14:24:59Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"canary-release-\": the object has been modified; please apply your changes to the latest version and try again" generation=2266 namespace=prod resourceVersion=200671007 rollout=canary-release

time="2024-11-14T14:24:59Z" level=info msg="Reconciliation completed" generation=2266 namespace=prod resourceVersion=rollout=canary-release time_ms=21.29092

time="2024-11-14T14:24:59Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"canary-release-\": the object has been modified; please apply your changes to the latest version and try again" namespace=prod rollout=canary-release

time="2024-11-14T14:24:59Z" level=info msg="rollout syncHandler queue retries: 3 : key \"prod/canary-release\"" namespace= prod rollout=canary-release
time="2024-11-14T14:24:59Z" level=error msg="Operation cannot be fulfilled on replicasets.apps \"canary-release-\": the object has been modified; please apply your changes to the latest version and try again\n" error="<nil>"

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[cam634]

Some added weirdness we see here that in the degraded state traffic was still flowing to the new version of the application. Even when we did a rollback undo at 9am, wasnt resolved until we applied a new image at around 9:28 am Output of rollback undo