Inconsistent genesis block hash on mekong devnet

[tersec]

https://mekong.ethpandaops.io/

Is it 0x13613aa0... or 0x2ffd69f6...

Oct 30 16:43:01.623 2024 REQUEST 
{
  "jsonrpc": "2.0",
  "method": "engine_forkchoiceUpdatedV3",
  "params": [
    {
      "headBlockHash": "0x13613aa073d32a70ebd3c9f952da2c0f956978b64c1b37c25641c6fecb024ade",
      "safeBlockHash": "0x13613aa073d32a70ebd3c9f952da2c0f956978b64c1b37c25641c6fecb024ade",
      "finalizedBlockHash": "0x13613aa073d32a70ebd3c9f952da2c0f956978b64c1b37c25641c6fecb024ade"
    },
    null
  ],
  "id": 0
}

Oct 30 16:43:01.625 2024 RESPONSE (status 200 OK) 
{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "payloadStatus": {
      "status": "SYNCING"
    }
  }
}

...

Oct 30 16:43:18.469 2024 REQUEST 
{
  "jsonrpc": "2.0",
  "method": "eth_getBlockByNumber",
  "params": [
    "0x0",
    false
  ],
  "id": 6
}

Oct 30 16:43:18.471 2024 RESPONSE (status 200 OK) 
{
  "jsonrpc": "2.0",
  "id": 6,
  "result": {
    "number": "0x0",
    "hash": "0x2ffd69f638c457944db233d22ff93dab4d8cdf0fab4dbbde4669f82f1dabba99",
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "transactionsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "stateRoot": "0x5ba6c0e2b23cb2bf598cb01ce8ebe8eebd39d17e2b531e16e2581932495c81b6",
    "receiptsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "miner": "0x0000000000000000000000000000000000000000",
    "difficulty": "0x1",
    "extraData": "0x",
    "gasLimit": "0x17d7840",
    "gasUsed": "0x0",
    "timestamp": "0x67221174",
    "nonce": "0x0000000000001234",
    "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "size": "0xfffffffffffe02e0",
    "totalDifficulty": "0x0",
    "transactions": [],
    "uncles": [],
    "baseFeePerGas": "0x3b9aca00",
    "withdrawals": [],
    "withdrawalsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "blobGasUsed": "0x0",
    "excessBlobGas": "0x0",
    "parentBeaconBlockRoot": "0x0000000000000000000000000000000000000000000000000000000000000000"
  }
}

...

Oct 31 11:00:12.515 2024 REQUEST 
{
  "jsonrpc": "2.0",
  "method": "engine_newPayloadV3",
  "params": [
    {
      "parentHash": "0x13613aa073d32a70ebd3c9f952da2c0f956978b64c1b37c25641c6fecb024ade",
      "feeRecipient": "0xf97e180c050e5ab072211ad2c213eb5aee4df134",
      "stateRoot": "0xc4e95f08b97e5e816dcd418d43760772caa684a8393302cfe01e3e363c1f53b7",
      "receiptsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
      "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
      "prevRandao": "0x13613aa073d32a70ebd3c9f952da2c0f956978b64c1b37c25641c6fecb024ade",
      "blockNumber": "0x1",
      "gasLimit": "0x17d7840",
      "gasUsed": "0x0",
      "timestamp": "0x6723633c",
      "extraData": "0x4e65746865726d696e64",
      "baseFeePerGas": "0x342770c0",
      "blockHash": "0x08bbbb17ed780e21d4999a7d36efc8a547d4a2e23c4fb7e8c806a42397357ccc",
      "transactions": [],
      "withdrawals": [],
      "blobGasUsed": "0x0",
      "excessBlobGas": "0x0"
    },
    [],
    "0x730cd6a6ff2fe403d2b0c473f071f0c8a2d60c1ea2e9bd8a6305fe4bca453370"
  ],
  "id": 6904
}

Oct 31 11:00:12.517 2024 RESPONSE (status 200 OK) 
{
  "jsonrpc": "2.0",
  "id": 6904,
  "result": {
    "status": "SYNCING"
  }
}

Oct 31 11:00:12.684 2024 REQUEST 
{
  "jsonrpc": "2.0",
  "method": "engine_forkchoiceUpdatedV3",
  "params": [
    {
      "headBlockHash": "0x08bbbb17ed780e21d4999a7d36efc8a547d4a2e23c4fb7e8c806a42397357ccc",
      "safeBlockHash": "0x13613aa073d32a70ebd3c9f952da2c0f956978b64c1b37c25641c6fecb024ade",
      "finalizedBlockHash": "0x13613aa073d32a70ebd3c9f952da2c0f956978b64c1b37c25641c6fecb024ade"
    },
    null
  ],
  "id": 6905
}

Oct 31 11:00:12.685 2024 RESPONSE (status 200 OK) 
{
  "jsonrpc": "2.0",
  "id": 6905,
  "result": {
    "payloadStatus": {
      "status": "SYNCING"
    }
  }
}

Full logs available on teku-nimbusel-1.mekong-devnet-0.ethpandaops.io via snooper-engine Docker container.

[advaita-saha]

Then we are getting a lot of inconsistent blockHashes across different devnets

[tersec]

Well, it's only running on two devnets right now.

[advaita-saha]

BlockHash difference is most likely due to the difference in ( at blockNumber=0 )

Nimbus ( doesn't match with genesis file )

"timestamp": "0x67221174"

Geth ( match with genesis file )

timestamp: "0x672362f4",

[barnabasbusa]

This issue was caused by the fact that our cleanup job did not yet include nimbus-eth1.

ethpandaops/mekong-devnets@c4c91b5

As of this commit its been fixed. Apologies for the false reporting.