Tests: arbitary tests failing arbitrarily.

[D-Stacks]

I have noticed in my recent bulk of pull requests that tests will fail in a seemingly arbitary fashion. The tests never fail on my local branch, but on github they often do.

pushing empty commits / editing the description, will rerun the test suites, and often pass the tests again. the tests failing seem arbitary as well.

examples:

2022-07-02T09:58:50.3204143Z     ibd_test.go:93: Timeout waiting for IBD to finish.
2022-07-02T09:58:50.3204351Z 2022-07-02 09:58:17.754 [WRN] KASD: Kaspad shutting down
2022-07-02T09:58:50.3204588Z 2022-07-02 09:58:17.755 [DBG] TXMP: Disconnecting from 127.0.0.1:56262
2022-07-02T09:58:50.3204900Z 2022-07-02 09:58:17.755 [INF] NTAR: Disconnected from <5bd081d7ecf5a44831a50b859602fc8d: 127.0.0.1:56262>
2022-07-02T09:58:50.3205132Z 2022-07-02 09:58:17.755 [DBG] TXMP: Disconnected from 127.0.0.1:54321
2022-07-02T09:58:50.3205368Z 2022-07-02 09:58:17.755 [DBG] TXMP: Disconnecting from 127.0.0.1:54321
2022-07-02T09:58:50.3205682Z 2022-07-02 09:58:17.755 [INF] NTAR: Disconnected from <753798cc3f7d01fbed5dbc37c2a8d37c: 127.0.0.1:54321>
2022-07-02T09:58:50.3205909Z 2022-07-02 09:58:17.755 [DBG] TXMP: Disconnecting from 127.0.0.1:59116
2022-07-02T09:58:50.3206158Z 2022-07-02 09:58:17.755 [INF] NTAR: Disconnected from <<nil>: 127.0.0.1:59116>
2022-07-02T09:58:50.3206355Z 2022-07-02 09:58:17.756 [WRN] KASD: Kaspad shutting down
2022-07-02T09:58:50.3206583Z 2022-07-02 09:58:17.756 [DBG] TXMP: Disconnecting from 127.0.0.1:59010
2022-07-02T09:58:50.3206821Z 2022-07-02 09:58:17.756 [INF] NTAR: Disconnected from <<nil>: 127.0.0.1:59010>
2022-07-02T09:58:50.3207018Z 2022-07-02 09:58:17.757 [WRN] KASD: Kaspad shutting down
2022-07-02T09:58:50.3207240Z 2022-07-02 09:58:17.757 [DBG] TXMP: Disconnecting from 127.0.0.1:60466
2022-07-02T09:58:50.3207470Z 2022-07-02 09:58:17.757 [INF] NTAR: Disconnected from <<nil>: 127.0.0.1:60466>
2022-07-02T09:58:50.3207661Z --- FAIL: TestIBDWithPruning (30.65s)

• passed without change to codebase on 2nd try

2022-06-29T23:27:51.3221624Z     utxo_index_test.go:68: Error: Circulating supply Mismatch - Circulating Sompi: 4950000000000 Sompi Mined: 5000000000000
2022-06-29T23:27:51.3222262Z 2022-06-29 23:27:27.686 [WRN] KASD: Kaspad shutting down
2022-06-29T23:27:51.3222691Z 2022-06-29 23:27:27.686 [DBG] TXMP: Disconnecting from 127.0.0.1:36670
2022-06-29T23:27:51.3223128Z 2022-06-29 23:27:27.686 [INF] NTAR: Disconnected from <<nil>: 127.0.0.1:36670>
2022-06-29T23:27:51.3223505Z --- FAIL: TestUTXOIndex (0.71s)

• passed without change to codebase on 2nd try