tx was previously rejected

[validexis]

Summary of Bug

Recently, a recurring error has been observed on Full consensus nodes, causing them to temporarily stop functioning. As a result, the bridge node also ceases to operate and requires a restart to resume normal functionality.

7:36PM INF commit synced commit=436F6D6D697449447B5B323030203631203130312036362032343620323034203439203230312034342031383320343820313737203330203136312035312031313120313238203630203233352031383620323330203133203133342031393620343520313139203231382031383820323031203130203233372035355D3A3435383145447D
7:36PM INF committed state app_hash=C83D6542F6CC31C92CB730B11EA1336F803CEBBAE60D86C42D77DABCC90AED37 height=4555245 module=state num_txs=5
7:36PM INF indexed block events height=4555245 module=txindex
7:36PM INF Could not add tx err="tx was previously rejected" txKey=[246,145,28,11,138,79,248,70,166,223,231,40,155,57,31,72,82,235,94,72,32,213,142,70,252,242,84,185,178,32,158,10]
7:36PM INF Could not add tx err="tx was previously rejected" txKey=[42,216,169,72,95,136,36,142,253,92,239,130,82,64,125,26,211,226,214,35,233,212,13,172,246,19,101,45,27,74,248,151]
7:36PM INF Could not add tx err="tx was previously rejected" txKey=[131,114,23,254,70,179,172,112,146,195,153,83,57,182,89,179,243,37,147,58,162,147,183,4,120,27,158,172,124,43,72,213]
7:36PM INF Could not add tx err="tx was previously rejected" txKey=[229,28,105,185,162,79,58,140,70,85,134,39,4,141,63,153,242,213,35,164,187,241,129,1,11,179,124,201,42,224,18,132]
7:36PM INF Could not add tx err="tx was previously rejected" txKey=[125,238,165,104,110,196,14,44,114,51,25,94,105,73,114,206,75,157,208,85,9,16,13,161,91,4,186,32,98,53,163,125]
7:36PM INF Could not add tx err="tx was previously rejected" txKey=[132,235,36,60,197,66,218,131,57,56,192,148,103,98,119,72,144,166,182,174,229,127,189,254,118,229,171,82,27,138,242,36]

Version

celestia-appd: v3.3.1-mocha

Steps to Reproduce

sudo journalctl -u celestia-appd -f --no-hostname -o cat

[evan-forbes]

thanks for reporting

it looks like your node fell behind somehow, which can result in this error popping up in the mempool as peers gossip valid transactions that your node thinks are invalid due to it having outdated state

while certainly not impossible, it is unlikely that your mempool is contributing to consensus halts. That being said, which mempool are you using?

if you changed any configs, please post them here.

it doesn't sound like you ran out of storage if restarting fixed the issue, but its worth checking.

[validexis]

We had a new installation, in which the default mempool version was v2, with the following parameters:

version = "v2"
max_txs_bytes = 83886080
max_tx_bytes = 2097152

Reverting the mempool version back to v1 seems to have solved the problem. Manually entered:

version = "v1"
max_txs_bytes = 39485440
max_tx_bytes = 7897088