Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High RAM usage when loaded with JSON-RPC #1629

Open
2 tasks
dimalit opened this issue Aug 28, 2023 · 6 comments · May be fixed by #1725
Open
2 tasks

High RAM usage when loaded with JSON-RPC #1629

dimalit opened this issue Aug 28, 2023 · 6 comments · May be fixed by #1725
Assignees
Labels
bug Something isn't working epic:HPD
Milestone

Comments

@dimalit
Copy link
Contributor

dimalit commented Aug 28, 2023

1 Have blocks full of 24-kb transactions
2 Enable blockscout. It will request all blocks with their transactions.
3 See RAM > 2.5 GB

Tasks

Preview Give feedback
@DmytroNazarenko DmytroNazarenko added bug Something isn't working epic:HPD labels Aug 28, 2023
@PolinaKiporenko PolinaKiporenko added this to the SKALE 2.3 milestone Aug 28, 2023
@DmytroNazarenko DmytroNazarenko moved this to Ready For Pickup in SKALE Engineering 🚀 Sep 4, 2023
@dimalit dimalit moved this from Ready For Pickup to In Progress in SKALE Engineering 🚀 Oct 2, 2023
@dimalit
Copy link
Contributor Author

dimalit commented Oct 19, 2023

HTTP request flow:

  1. Incoming connection is put in accept backlog or rejected if backlog is full
  2. Connection waits in backlog
  3. Connection is accepted
  4. Request body is read
  5. Response is generated
  6. Response is transferred back to client
  7. Connection is terminated

Problem:
When many connections are in the middle of stage 6 - it can consume a lot of RAM.

Proposed solution(s):
Count current number (or RAM size?) of requests in stage 6 - and if limit is reached:

  1. Don't accept new requests on stage 3
    or
  2. Block requests from moving from stage 4 to stage 5
    or
  3. Drop some old connections that are on stage 6 for long time

Need to decide

@DmytroNazarenko
Copy link
Collaborator

How much is it - Many connections in the middle of stage 6?

@DmytroNazarenko
Copy link
Collaborator

In p.1 clients will receive the message that server is overloaded straightaway and for them it would be clearer the root cause of the issue. That's why p.1 is better than 2 and 3

@dimalit dimalit moved this from In Progress to Blocked in SKALE Engineering 🚀 Oct 27, 2023
@dimalit dimalit moved this from Blocked to In Progress in SKALE Engineering 🚀 Nov 3, 2023
@dimalit dimalit linked a pull request Nov 7, 2023 that will close this issue
@PolinaKiporenko PolinaKiporenko moved this from In Progress to Code Review in SKALE Engineering 🚀 Nov 7, 2023
@PolinaKiporenko PolinaKiporenko moved this from Code Review to In Progress in SKALE Engineering 🚀 Nov 8, 2023
@PolinaKiporenko PolinaKiporenko moved this from In Progress to Blocked in SKALE Engineering 🚀 Nov 14, 2023
@PolinaKiporenko
Copy link
Contributor

Moved to Blocked due to lack of testing resources on DevNet, needs to recheck during Regression testing 2.3

For now timeout was decreased to 5 sec from 30 min (default timeout 2 min), still need to check if it not too short timeout and make sure that the amount of memory consumed is sufficient

RAM 1,5 GB

@PolinaKiporenko PolinaKiporenko modified the milestones: SKALE 2.3, SKALE 2.4 Jan 4, 2024
@PolinaKiporenko
Copy link
Contributor

Moved to 2.4 - need more investigate of performance with decreasing limits

@PolinaKiporenko PolinaKiporenko moved this from Blocked to Ready For Pickup in SKALE Engineering 🚀 Jan 4, 2024
@PolinaKiporenko
Copy link
Contributor

Could be postponed to 2.5 release

@PolinaKiporenko PolinaKiporenko modified the milestones: SKALE 2.4, SKALE 2.5 Apr 9, 2024
@PolinaKiporenko PolinaKiporenko modified the milestones: SKALE 2.5, SKALE 2.6 Apr 24, 2024
@PolinaKiporenko PolinaKiporenko moved this from Ready For Pickup to To Do in SKALE Engineering 🚀 Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working epic:HPD
Projects
Status: No status
Development

Successfully merging a pull request may close this issue.

3 participants