Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

safekeeper: speed up timeline deletion #10265

Open
erikgrinaker opened this issue Jan 3, 2025 · 2 comments
Open

safekeeper: speed up timeline deletion #10265

erikgrinaker opened this issue Jan 3, 2025 · 2 comments
Labels
a/performance Area: relates to performance of the system c/storage/safekeeper Component: storage: safekeeper t/incident Issue type: incident in our service

Comments

@erikgrinaker
Copy link
Contributor

We saw a Safekeeper timeline deletion with 12 TB WAL take 30 minutes to complete. We should try to speed this up by about an order of magnitude. Some ideas:

  • Use an iterator to list objects instead of re-listing for every batch.
  • Send many (e.g. 16) concurrent batch deletion requests.
  • Respect the remote storage's max_keys_per_delete (Azure only allows 256, not 1000 like S3).
@erikgrinaker erikgrinaker added a/performance Area: relates to performance of the system c/storage/safekeeper Component: storage: safekeeper t/incident Issue type: incident in our service labels Jan 3, 2025
@jcsp
Copy link
Collaborator

jcsp commented Jan 3, 2025

Agree this could be made to run faster, but on the other hand deletions can be thought of as a very low priority operation, so it's not necessarily helpful to take more than one unit of remote storage concurrency for them 🤔

@arpad-m
Copy link
Member

arpad-m commented Jan 8, 2025

Earlier issue: #9152

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a/performance Area: relates to performance of the system c/storage/safekeeper Component: storage: safekeeper t/incident Issue type: incident in our service
Projects
None yet
Development

No branches or pull requests

3 participants