Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to sampling profiler for driver profiling #1399

Open
pquentin opened this issue Dec 10, 2021 · 0 comments
Open

Switch to sampling profiler for driver profiling #1399

pquentin opened this issue Dec 10, 2021 · 0 comments
Labels
enhancement Improves the status quo help wanted We'd be happy about a community contribution :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc.
Milestone

Comments

@pquentin
Copy link
Member

Profiling the load driver is important to make sure there's no accidental client bottleneck. Rally currently uses yappi for this: https://esrally.readthedocs.io/en/latest/command_line_reference.html#clr-enable-driver-profiling.

My understanding is that yappi is basically cProfile with asyncio support. It's a deterministic tracing profiler, which means that every single function call is recorded. In practice, just like cProfile, it's going to tell us functions are slow because the profiler is making them slow. This is why cProfile is basically a bad practice in Python.

We should use a sampling profiler instead, like vmprof or py-spy. vmprof appears to be useful for async/await code: python-trio/trio#943. It would be nice if profiling stayed as easy as it is now, but producing useful results is even more important!

@pquentin pquentin added enhancement Improves the status quo :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc. labels Dec 10, 2021
@pquentin pquentin added this to the Backlog milestone Dec 10, 2021
@pquentin pquentin added the help wanted We'd be happy about a community contribution label May 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improves the status quo help wanted We'd be happy about a community contribution :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc.
Projects
None yet
Development

No branches or pull requests

1 participant