-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enhance: reduce stats task cost by skipping ser/de #39568
base: master
Are you sure you want to change the base?
Conversation
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tedxu The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
@tedxu go-sdk check failed, comment |
@tedxu E2e jenkins job failed, comment |
@tedxu cpp-unit-test check failed, comment |
Signed-off-by: Ted Xu <[email protected]>
a12ccda
to
a80189f
Compare
@tedxu E2e jenkins job failed, comment |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #39568 +/- ##
===========================================
- Coverage 80.97% 69.40% -11.57%
===========================================
Files 1408 302 -1106
Lines 198881 27065 -171816
===========================================
- Hits 161041 18785 -142256
+ Misses 32151 8280 -23871
+ Partials 5689 0 -5689
|
@tedxu go-sdk check failed, comment |
Signed-off-by: Ted Xu <[email protected]>
Signed-off-by: Ted Xu <[email protected]>
@tedxu E2e jenkins job failed, comment |
In this PR the sort cost fairly depends on the partial order of the original dataset. If the original dataset is ordered, the CPU and memory cost can be negligible. I've also tested the worst case, for 1M rows the cost is as follows (2seconds & 4.5GB):
The worst cost is still an improvement compares to current implementation. The primary cost comes from the |
@tedxu go-sdk check failed, comment |
Signed-off-by: Ted Xu <[email protected]>
@tedxu go-sdk check failed, comment |
@tedxu E2e jenkins job failed, comment |
@tedxu cpp-unit-test check failed, comment |
Signed-off-by: Ted Xu <[email protected]>
@tedxu go-sdk check failed, comment |
@tedxu E2e jenkins job failed, comment |
See #37234