Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use generics instead of dynamic dispatch internally in tar::Builder #395

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

GabrielDertoni
Copy link

@GabrielDertoni GabrielDertoni commented Jan 12, 2025

TL;DR Using generics instead of dynamic dispatch allows for specialization and a performance win on modern Linux systems.

bench old time new time delta
cache=yes size=583M files=23k 1.9 s ± 0.1 s 1.7 s ± 0.1 s -10.08%
cache=no size=583M files=23k 7.3 s ± 0.2 s 6.7 s ± 0.3 s -8.95%
cache=yes size=284M files=1k 500 ms ± 4 ms 362 ms ± 28 ms -38.12%
cache=no size=284M files=1k 918 ms ± 25 ms 889 ms ± 29 ms -3.26%

(look here for full benchmark details)

This PR is specifically targeted to allow triggering the specialization of the Write implementations when using tar::Builder. In particular, the stdlib has a specialization on Linux for std::io::copy that allows it to use the copy_file_range syscall. Unfortunately the code as-is is not capable of performing such specialization (or potentially others) since it relies on dynamic dispatch internally. This PR changes the underlying implementation to uses generics instead of dynamic dispatch. This understandably comes at the potential cost of increased code size but I'd argue this choice is already exposed to users that should be willing to pay the code size cost when using tar::Builder<T> directly instead of tar::Builder<dyn Write>. Users looking for reduced code footprint can explicitly opt in through using tar::Builder<dyn Write> instead.

PS: I had to change one of the tests because it looks like this specialization behaves differently than the usual before. Previously since the pax_simple_write test was using a BufWriter it would try to append_file the archive inside the archive itself. That would work because the writes to the file would be buffered and thus reading the archive at that point would read nothing. But with copy_file_range that is no longer true. Anyway, I am pretty sure this was a bug in the test code, but if not, the correct fix would be to do some sort of inode tracking like tar does.

@cgwalters
Copy link
Collaborator

In particular, the stdlib has a specialization on Linux for std::io::copy that allows it to use the copy_file_range syscall.

And that would happen here when your tar output is to a file, and you're providing a File instance as input e.g. append()? OK. (Or a pipe/socket for either of those)

OK, though I do kind of question how often this occurs, because most use cases for tar end up being compressed (i.e. the output side is not a fd).

Can you at briefly explain your higher level use case and what led you to try this optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants