Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply some basic rocksdb options #2339

Merged
merged 1 commit into from
Jun 12, 2024
Merged

Apply some basic rocksdb options #2339

merged 1 commit into from
Jun 12, 2024

Conversation

arnetheduck
Copy link
Member

These options, inspired by Nethermind and general internet wisdom, bring the database size down to 2/3 without affecting throughput. In theory, they should also bring down memory usage and/or make more efficient use of whatever memory is already assigned to rocksdb but this needs verification in a longer test at synced-mainnet sizes.

In the meantime, they make testing easier by removing some noise that the profiler says are bad, such as excessive SkipList access (countered by bloom filters).

These options, inspired by Nethermind and general internet wisdom, bring
the database size down to 2/3 without affecting throughput. In theory,
they should also bring down memory usage and/or make more efficient use
of whatever memory is already assigned to rocksdb but this needs
verification in a longer test at synced-mainnet sizes.

In the meantime, they make testing easier by removing some noise that
the profiler says are bad, such as excessive SkipList access (countered
by bloom filters).
@arnetheduck arnetheduck merged commit 54f793f into master Jun 12, 2024
26 checks passed
@arnetheduck arnetheduck deleted the rocksdb-basics branch June 12, 2024 12:52
@arnetheduck
Copy link
Member Author

Results from running this PR:

stats-20240611_1013-eb041abb.csv vs stats-20240612_1309-54f793f9.csv
                       bps_x     bps_y     tps_x        tps_y    bpsd    tpsd    timed
block_number
(498305, 990113]    2,101.05  2,113.79  3,996.04  4040.003741   1.02%   1.02%   -0.93%
(990113, 1481921]     922.49    932.96  4,614.59  4670.467487   1.37%   1.37%   -1.17%
(1481921, 1973729]    554.29    560.58  3,986.51  4035.914392   1.18%   1.18%   -1.01%
(1973729, 2465537]    373.17    408.68  2,843.89  3115.029665   9.65%   9.65%   -7.87%
(2465537, 2957345]    388.60    412.94  2,737.82  2910.144307   5.79%   5.79%   -4.37%
(2957345, 3449153]    384.91    404.97  3,450.42  3647.157591   6.76%   6.76%   -4.51%
(3449153, 3940961]    131.46    137.63  2,688.94  2812.589718   5.06%   5.06%   -3.89%
(3940961, 4432769]     25.90     28.21  1,843.68  2031.137182  10.99%  10.99%   -8.86%
(4432769, 4924577]     12.79     14.21  1,430.89  1604.376076  12.01%  12.01%  -10.34%

blocks: 4426272, baseline: 24h16m11s, contender: 21h58m6s
bpsd (mean): 5.98%
tpsd (mean): 5.98%
Time (total): -2h18m5s, -9.48%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant