fix: handle objects from cache of type DeletedFinalStateUnknown

[nilsgstrabo]

In rare situations, the object received from cache is of type DeletedFinalStateUnknown, which causes kubernetes-replicator to panic (see trace)

In this PR MustGetObject is modified to also check if obj is of type cache.DeletedFinalStateUnknown and calls itself with the Obj field, which should contain the real object.

I1012 05:51:47.296479       1 trace.go:236] Trace[2025117757]: "DeltaFIFO Pop Process" ID:myappns/mysecret,Depth:10184,Reason:slow event handlers blocking the queue (12-Oct-2024 05:51:46.898) (total time: 398ms):
Trace[2025117757]: [398.233324ms] [398.233324ms] END
E1012 05:51:47.296948       1 runtime.go:79] Observed a panic: Unknown type: cache.DeletedFinalStateUnknown (Unknown type: cache.DeletedFinalStateUnknown)
goroutine 7 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1209fa0, 0x40121d2eb8})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x7c
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4000368e00?})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x78
panic({0x1209fa0?, 0x40121d2eb8?})
        /opt/hostedtoolcache/go/1.22.6/x64/src/runtime/panic.go:770 +0x124
github.com/mittwald/kubernetes-replicator/replicate/common.MustGetObject({0x1245240, 0x40100e6800})
        /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/strings.go:57 +0x10c
github.com/mittwald/kubernetes-replicator/replicate/common.MustGetKey({0x1245240?, 0x40100e6800?})
        /home/runner/work/kubernetes-replicator/kubernetes-replicator/replicate/common/strings.go:40 +0x2c
github.com/mittwald/kubernetes-replicator/replicate/common.(*GenericReplicator).ResourceDeleted(0x40001a10a0, {0x1245240, 0x40100e6800})

[nilsgstrabo]

@martin-helmich Could you have a look at the changes in this PR?
The last release 2.10.2, which included the fix for concurrent map access, reduced number of crashes from 20 per day to 1 per week for us. The changes in this this PR should fix the remaining cause of crashes. I have tested it in one of our AKS clusters, and it works as expected.

[martin-helmich]

Apologies for the delay. LGTM -- thanks for the fix 👍

[martin-helmich]

Still wondering about that failing test case. 🤔 On a quick glance, I'd think it's unrelated to the changes in this PR. I've rebased to current master to see if it wasn't a fluke.

[nilsgstrabo]

Thanks 😊