Blazegraph.jnl just disappeared

[botnumberseven]

Fellas, i don't know how to put it, but on one of our nodes blazegraph just disappeared, specifically journal file - blazegraph.jnl.
It's a 3 cores, 4GB RAM + 4GB swap system with 80GB SSD, similar to others. It had enough resources while it was running.
It is running ubuntu 20 with 8.1.0+beta.12 node, node.js 16.20.1.

It was doing the same thing as other nodes.

1. I add a few paranets into the config to sync
2. let it run and monitor how it goes thru all the assets in the paranet journalctl -o cat -u otnode -f | grep "offset" | cut -c1-215
3. Once I see no getKnowledgeAssetsWithPagination messages i start to monitor paranetSyncCommand in general - journalctl -o cat -u otnode -f | grep "paranetSyncCommand"
4. at some point is says no new assets to sync, so I consider paranet to be synced, stop the node, add another paranet in the config file, start the node.

After one of the restarts i failed to see any getKnowledgeAssetsWithPagination messages for some time, so i went to check things, and discovered that blazegraph.jnl is just not there anymore (.jar is still there).

This could explain why on all the nodes I saw #3344, but that was an issue with remote node. Now my node is this remote node.

it looks like my otnode logs do not go back far enough, as at the beginning of available logs i can see all these blazegraph errors.
I'm not sure how to determine what caused blazegraph to disappear, appreciate you suggestions.
Node is stopped at the moment.
I'm pretty confident I did not delete it manually.

I see all this even at the very beginning of my logs.

Oct 25 23:46:59 v8test7 node[179015]: [2024-10-25 23:46:59] WARN: {Operation ID: 1f561ff4-c997-43de-be1b-8f5ddfd17ad6} [v1_0_0GetRequestCommand] (Command ID: d3b95a4b-a592-4652-9e41-4181b2056692)  Error in command: v1_0_0GetRequestCommand, error: Invalid SPARQL endpoint response from http://localhost:9999/blazegraph/namespace/did-dkg-base-84532-0xb8b904c73d2fb4d8c173298a51c27fab70222c32-5603732/sparql (HTTP status 500):
Oct 25 23:46:59 v8test7 node[179015]: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /root/ot-node/blazegraph.jnl
Oct 25 23:46:59 v8test7 node[179015]: Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /root/ot-node/blazegraph.jnl
Oct 25 23:46:59 v8test7 node[179015]: Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /root/ot-node/blazegraph.jnl
Oct 25 23:46:59 v8test7 node[179015]: Caused by: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /root/ot-node/blazegraph.jnl
Oct 25 23:46:59 v8test7 node[179015]: Caused by: java.nio.file.NoSuchFileException: /root/ot-node/blazegraph.jnl
Oct 25 23:46:59 v8test7 node[179015]: [2024-10-25 23:46:59] WARN: {Operation ID: 1f561ff4-c997-43de-be1b-8f5ddfd17ad6} [v1_0_0GetRequestCommand] (Command ID: d3b95a4b-a592-4652-9e41-4181b2056692)  Error in command: v1_0_0GetRequestCommand, error: Invalid SPARQL endpoint response from http://localhost:9999/blazegraph/namespace/did-dkg-base-84532-0xb8b904c73d2fb4d8c173298a51c27fab70222c32-5603732/sparql (HTTP status 500):```