Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate archive node performance #1870

Open
olehnikolaiev opened this issue Apr 16, 2024 · 3 comments · May be fixed by #2063
Open

Investigate archive node performance #1870

olehnikolaiev opened this issue Apr 16, 2024 · 3 comments · May be fixed by #2063
Assignees
@olehnikolaiev
Labels
epic:archive-node
Milestone
SKALE 4.0

Comments

@olehnikolaiev
Copy link
Contributor

archive node may catchup blocks slower then they appear on the core nodes and may have a long response time for json-rpc calls. need to investigate the reasons
@PolinaKiporenko PolinaKiporenko moved this to Ready For Pickup in SKALE Engineering 🚀 Apr 16, 2024
@PolinaKiporenko PolinaKiporenko added this to the SKALE 2.5 milestone Apr 16, 2024
@PolinaKiporenko PolinaKiporenko added epic:archive-node bug Something isn't working labels Apr 16, 2024
@OleksanderSalamatov
Copy link

Added grafana monitoring for Exorde sync node
https://skale-grafana.skaleserver.com/d/ExordeSync/exorde-sync-node-hardware-overview
For credentials DM @OleksanderSalamatov

@DmytroNazarenko DmytroNazarenko changed the title investigate archive node performance Investigate archive node performance Apr 17, 2024
@olehnikolaiev
Copy link
Contributor Author

moving from leveldb to rocks db can potentially improve performance. need to investigate

This was referenced Jun 5, 2024
Open
Open
@olehnikolaiev olehnikolaiev moved this from Ready For Pickup to In Progress in SKALE Engineering 🚀 Jul 9, 2024
olehnikolaiev added a commit that referenced this issue Jul 25, 2024
@olehnikolaiev
#1870 install rocksdb
a506d64
olehnikolaiev added a commit that referenced this issue Jul 29, 2024
@olehnikolaiev
#1870 tune rocksdb options
9e3cea0
olehnikolaiev added a commit that referenced this issue Jul 31, 2024
@olehnikolaiev
#1870 disable avx512 instructions for rocksdb
3b81360
olehnikolaiev added a commit that referenced this issue Jul 31, 2024
@olehnikolaiev
#1870 add patch timestamp for rocksdb
018d090
olehnikolaiev added a commit that referenced this issue Aug 1, 2024
@olehnikolaiev
#1870 fix rocksdb installation
e3b6d55
olehnikolaiev added a commit that referenced this issue Sep 23, 2024
@olehnikolaiev
#1870 introduce maxHistoricStateDbSize
e9614a5
olehnikolaiev added a commit that referenced this issue Sep 23, 2024
@olehnikolaiev
#1870 rotating historic state
f459b93
olehnikolaiev added a commit that referenced this issue Sep 23, 2024
@olehnikolaiev
#1870 historic state rotation
a40d63d
olehnikolaiev added a commit that referenced this issue Sep 25, 2024
@olehnikolaiev
#1870 fix initing databases
7479f7d
olehnikolaiev added a commit that referenced this issue Sep 25, 2024
@olehnikolaiev
#1870 rotate historic state dbs before block execution
d80a2d5
olehnikolaiev added a commit that referenced this issue Sep 30, 2024
@olehnikolaiev
#1870
e426ae7
@PolinaKiporenko PolinaKiporenko modified the milestones: SKALE 3.1, SKALE 3.1.1 Oct 2, 2024
@dimalit
Copy link
Contributor

dimalit commented Oct 4, 2024 •
edited
Loading

Block creation steps:

  1. m_state = m_state.createNewCopyWithLocks();
  2. m_state.mutableHistoricState().rotateDbsIfNeeded( _timestamp );
  3. Block::m_state = m_state.createStateModifyCopyAndPassLock();
  4. State stateSnapshot = Block::m_state.createStateModifyCopyAndPassLock();
  5. Run EVM
  6. stateSnapshot.commit()
  7. Repeat steps 4-6 for each transaction
  8. Block::m_state.mutableHistoricState().saveRootForBlock( m_currentBlock.number() );
  9. Block::m_state.releaseWriteLock();
  10. m_state = m_state.createNewCopyWithLocks();
  11. m_state.mutableHistoricState().setRoot( m_working.mutableState().mutableHistoricState().globalRoot() );

olehnikolaiev added a commit that referenced this issue Oct 7, 2024
@olehnikolaiev
#1870 dbs cache for historic state
4dafff3
olehnikolaiev added a commit that referenced this issue Oct 7, 2024
@olehnikolaiev
#1870 fix constructor
a4d3de7
olehnikolaiev added a commit that referenced this issue Oct 7, 2024
@olehnikolaiev
#1870 check opened dbs
1f47de9
olehnikolaiev added a commit that referenced this issue Oct 7, 2024
@olehnikolaiev
#1870 check opened dbs
1f4e7ca
olehnikolaiev added a commit that referenced this issue Oct 8, 2024
@olehnikolaiev
#1870 optimized search in hisstoric dbs
84d7671
olehnikolaiev added a commit that referenced this issue Oct 16, 2024
@olehnikolaiev
#1870
29cb50d
olehnikolaiev added a commit that referenced this issue Oct 29, 2024
@olehnikolaiev
#1870 change storage usage calculation]
2a37352
olehnikolaiev added a commit that referenced this issue Oct 29, 2024
@olehnikolaiev
Revert "#1870 change storage usage calculation]"
f19aa68 
This reverts commit 2a37352.
olehnikolaiev added a commit that referenced this issue Oct 30, 2024
@olehnikolaiev
#1870 clear historic caches after commit
e7aa0f6
olehnikolaiev added a commit that referenced this issue Nov 1, 2024
@olehnikolaiev
#1870 remove unnecessary lookup
f35d714
olehnikolaiev added a commit that referenced this issue Nov 1, 2024
@olehnikolaiev
#1870 new storage usage calculator
c9e377b
olehnikolaiev added a commit that referenced this issue Nov 4, 2024
@olehnikolaiev
#1870 fix core mode build
bf4e730
olehnikolaiev added a commit that referenced this issue Nov 4, 2024
@olehnikolaiev
#1870 fix tests
883ca08
olehnikolaiev added a commit that referenced this issue Nov 4, 2024
@olehnikolaiev
#1870 remove m_storageUsage for historic state
eebad6e
olehnikolaiev added a commit that referenced this issue Nov 4, 2024
@olehnikolaiev
#1870 rotate historic roots db
8953f74
@olehnikolaiev olehnikolaiev linked a pull request Jan 2, 2025 that will close this issue
Open
@olehnikolaiev olehnikolaiev moved this from In Progress to Code Review in SKALE Engineering 🚀 Jan 6, 2025
@olehnikolaiev olehnikolaiev linked a pull request Jan 13, 2025 that will close this issue
Feature/1870 historic state rotation #2063
Open
olehnikolaiev added a commit that referenced this issue Jan 13, 2025
@olehnikolaiev
#1870 fix build after merge
5256373
olehnikolaiev added a commit that referenced this issue Jan 13, 2025
@olehnikolaiev
#1870 fix build after merge
242b97b
olehnikolaiev added a commit that referenced this issue Jan 13, 2025
@olehnikolaiev
#1870 fix tests after merge
075bf07
@PolinaKiporenko PolinaKiporenko removed the bug Something isn't working label Jan 16, 2025
olehnikolaiev added a commit that referenced this issue Feb 5, 2025
@olehnikolaiev
#1870 improve tests
54fbcc0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@olehnikolaiev olehnikolaiev

Labels
epic:archive-node
Projects
SKALE Engineering 🚀
Status: Code Review
Milestone
SKALE 4.0
Development

Successfully merging a pull request may close this issue.

Feature/1870 historic state rotation skalenetwork/skaled
4 participants
@dimalit @olehnikolaiev @OleksanderSalamatov @PolinaKiporenko