All data gone - topics deleted themselves (I think)

[Brandt1930]

We have a Kubernetes Cluster (GitVersion 1.19.6, on OpenSuse Linux) running on 5 node (one of which is master-node) with the latest Strimzi. Inside we have mutliple Namespaces with 3-Node-Kafka-ZK-Clusters. Yesterday everything worked fine, today I entered the office and everything was gone... no Topics (only the Strimzi ones, and __custormer_offset), no data on the PV's (checked the hard drives)..
I still cant figure out why so any help is appreciated.

Some log extracts here (example Topic mqtt-raw):

The logs in /var/log/pods/ are starting with a reboot at 7:37, everything loads fine:

"log":"2021-02-24 08:06:00,488 INFO [Partition mqtt-raw-0 broker=0] Log loaded for partition mqtt-raw-0 with initial high watermark 19374 (kafka.cluster.Partition) [data-plane-kafka-request-handler-3]\n","strea m":"stdout","time":"2021-02-24T08:06:00.488242293Z"}

But some seconds later seemingly out of nowhere:

{"log":"2021-02-24 08:06:09,706 TRACE [Broker id=0] Deleted partition mqtt-raw-0 from metadata cache in response to UpdateMetadata request sent by controller 2 epoch 4 with correlation id 30 (state.change.logger) [data-plane-kafka-request-handler-1]\n","stream":"stdout","time":"2021-02-24T08:06:09.706708403Z"}
And then some seconds later it is marked as deleted and its gone..

Also, the entity-topic-operator doesnt work anymore the logs say:

021-02-24 13:32:34,49767 WARN  [OkHttp https://10.233.0.1/...] WatcherWebSocketListener:96 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2021-02-24 13:32:34,50331 INFO  [vert.x-eventloop-thread-0] TopicOperator:1249 - Starting initial  reconciliation
2021-02-24 13:32:34,50324 ERROR [vert.x-eventloop-thread-1] Main:56 - Error deploying Session
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found

	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:98) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
	Suppressed: java.lang.Throwable: waiting here
		at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:154) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
		at io.fabric8.kubernetes.client.dsl.internal.WebSocketClientRunner.waitUntilReady(WebSocketClientRunner.java:50) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
		at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.waitUntilReady(AbstractWatchManager.java:164) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:810) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:783) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:85) ~[io.fabric8.kubernetes-client-5.0.2.jar:?]
		at io.strimzi.operator.topic.Session.startWatcher(Session.java:274) ~[io.strimzi.topic-operator-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
		at io.strimzi.operator.topic.Session.lambda$start$7(Session.java:237) ~[io.strimzi.topic-operator-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
		at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
		at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
		at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.59.Final.jar:4.1.59.Final]
		at java.lang.Thread.run(Thread.java:834) [?:?]

I know my data is gone but I want to understand whats going on

[scholzj]

Well, this error:

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'

Means the KafkaTopic CRD does not exist anymore. So someone / something deleted it and that also deleted all the topics (when the CRD is deleted it normally deletes all the CRs).

[Brandt1930]

Yeah I also read your answer to this on another thread but my CRDs are all installed..

So someone / something deleted it

Can anyone outside of Kubernetes delete this? Another option would be an systemwide Docker-update, could this harm the system in that way?

[Brandt1930]

I found my issue regarding the entity operator not being successfully deployed: I had an outdated version of the CRD which had a schema error which @scholzj fixed 2 days ago. After a git pull and applying the latest version everything works fine..
But I still cant explain why my topics got deleted..

[scholzj]

Wait ... that would mean that you are using the builds from master branch. That would be absolutely unpredictable as to running for long term. Assuming we are talking about some environment where you actually care about the data, you should move to some stable release.

[Brandt1930]

Okay, thanks for your help!