Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store: Synchronous API #244

Open
Wondertan opened this issue Jan 28, 2025 · 1 comment
Open

store: Synchronous API #244

Wondertan opened this issue Jan 28, 2025 · 1 comment

Comments

@Wondertan
Copy link
Member

Current writes and read in the Store are asynchronous. Writes via Append cannot be accessed via Get immediately after write(same as for Head after #239).

However, doing this with our requirements is not trivial. We need to keep writes IO free and return as early as possible as its currently done. With introduction of contiguous head in #239 this becomes even more problematic, as each write has to advance this head. Advancing, in its current implementation, always hits the disk by checking if the next disk is available locally and setting this new header as new contiguous head. Allowing each write to hit the disk is gonna be a major performance regression, so the solution to that must be found.

Conceptually, the solution is possible by tracking the gaps after the contiguous head within the Store. As Store has full control over what gets written into it, it can retain header ranges that didn't contribute to a contiguous chain and produces gaps. Then, whenever any new header is written, the Store is gonna check tracked header and see if it advance the contiguous head without hitting the disk. This should work well in conjuction with autoadvancing on Start implemented as #239, allowing writes to avoid disk hits in case tracked header gaps are cleared out after the restart.

@Wondertan
Copy link
Member Author

POC: #241

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant