Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run benchmarks on push to default branch #67

Open
LilithHafner opened this issue Jan 28, 2025 · 3 comments
Open

Run benchmarks on push to default branch #67

LilithHafner opened this issue Jan 28, 2025 · 3 comments

Comments

@LilithHafner
Copy link
Contributor

It is sometimes useful to run benchmarks on every commit to the default branch (posting comments on commits) in addition to on pull requests. The results may differ from those posted to the pull requests if the pull request head is out of date.

@MilesCranmer
Copy link
Owner

One tricky part about this is that the variability in performance across runners tends to be larger than variability over a single action. The action runner is some small virtual machine and while even a single action is pretty noisy, multiple actions will be noisier since it might be a completely different machine. But worth thinking about, and maybe it's not as big a deal as I think.

@LilithHafner
Copy link
Contributor Author

Good to know. Whether that noise is an issue likely depends on the benchmarks being tracked.

FYI I've implemented this here: LilithHafner/DynamicDiscreteSamplers.jl#50 (Example comment: LilithHafner/DynamicDiscreteSamplers.jl@79e5728#commitcomment-151878662). We'll see if it's actually useful.

@LilithHafner
Copy link
Contributor Author

It is very low-noise which is nice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants