Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: Optimize the task scheduling logic in DataCoord #39084

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

enhance: Optimize the task scheduling logic in DataCoord #39084

wants to merge 2 commits into from
+821 −469

Conversation

xiaocai2333
Copy link
Contributor

@xiaocai2333 xiaocai2333 commented Jan 8, 2025 •
edited
Loading

issue: #39101

@sre-ci-robot sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Jan 8, 2025
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 5cc499d to 4fb4229 Compare January 8, 2025 09:14
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Jan 8, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 8, 2025

@xiaocai2333 Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 8, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 8, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@codecov Codecov
Copy link

codecov bot commented Jan 8, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 84.07080% with 36 lines in your changes missing coverage. Please review.

Project coverage is 80.99%. Comparing base (fdb968d) to head (2fd0e7d).
Report is 5 commits behind head on master.

Files with missing lines Patch % Lines
internal/datacoord/task_index.go 57.14% 6 Missing and 3 partials ⚠️
internal/datacoord/task_scheduler.go 80.00% 5 Missing and 2 partials ⚠️
pkg/util/merr/utils.go 0.00% 6 Missing ⚠️
internal/datacoord/session/indexnode_manager.go 85.71% 4 Missing and 1 partial ⚠️
internal/datacoord/task_analyze.go 55.55% 4 Missing ⚠️
internal/indexnode/task_scheduler.go 86.36% 2 Missing and 1 partial ⚠️
internal/indexnode/task_analyze.go 0.00% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #39084      +/-   ##
==========================================
- Coverage   81.00%   80.99%   -0.01%     
==========================================
  Files        1408     1408              
  Lines      198565   198711     +146     
==========================================
+ Hits       160840   160947     +107     
- Misses      32068    32093      +25     
- Partials     5657     5671      +14
Components Coverage Δ
Client 79.50% <ø> (ø)
Core 69.48% <ø> (ø)
Go 82.93% <84.07%> (-0.01%) ⬇️
Files with missing lines Coverage Δ
internal/datacoord/task_stats.go 92.79% <100.00%> (+0.16%) ⬆️
internal/indexnode/indexnode.go 77.82% <100.00%> (+0.09%) ⬆️
internal/indexnode/indexnode_service.go 83.83% <100.00%> (+0.12%) ⬆️
internal/indexnode/task_index.go 72.06% <100.00%> (+0.19%) ⬆️
internal/indexnode/task_stats.go 70.87% <100.00%> (+0.54%) ⬆️
internal/indexnode/util.go 61.29% <100.00%> (+18.43%) ⬆️
pkg/util/merr/errors.go 92.18% <ø> (ø)
pkg/util/paramtable/component_param.go 98.41% <100.00%> (+0.01%) ⬆️
internal/indexnode/task_analyze.go 78.72% <0.00%> (-1.14%) ⬇️
internal/indexnode/task_scheduler.go 87.26% <86.36%> (+0.08%) ⬆️
... and 5 more

... and 39 files with indirect coverage changes

@mergify mergify bot added the ci-passed label Jan 8, 2025
@xiaocai2333
Copy link
Contributor Author

rerun go-sdk

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 9, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from a69d6ee to 242bff0 Compare January 9, 2025 03:22
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 9, 2025

@xiaocai2333 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 9, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 9, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333
Copy link
Contributor Author

/run-cpu-e2e

@mergify mergify bot added the ci-passed label Jan 9, 2025
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 9d0bd08 to 30fbbf6 Compare January 13, 2025 11:49
@sre-ci-robot sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Jan 13, 2025
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 30fbbf6 to 4b4d61f Compare January 13, 2025 11:49
@mergify mergify bot removed the ci-passed label Jan 13, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 13, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 13, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 4b4d61f to 5585e34 Compare January 14, 2025 12:20
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xiaocai2333
To complete the pull request process, please assign liliu-z after the PR has been reviewed.
You can assign the PR to them by writing /assign @liliu-z in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Jan 14, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 15, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 1bb49db to 757ba41 Compare January 15, 2025 02:55
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 15, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 757ba41 to 2ed32cb Compare January 15, 2025 06:19
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 15, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@czs007
Copy link
Collaborator

czs007 commented Jan 15, 2025

/hold

@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 2ed32cb to 5e299a6 Compare January 15, 2025 07:48
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 15, 2025

@xiaocai2333 go-sdk check failed, comment rerun go-sdk can trigger the job again.

@xiaocai2333 xiaocai2333 mentioned this pull request Jan 17, 2025
Open
1 task
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from 5e299a6 to 50b9bf6 Compare January 17, 2025 06:23
@xiaocai2333
Copy link
Contributor Author

rerun go-sdk

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 17, 2025

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@xiaocai2333
Refine index task scheduler policy
67dcec4 
Signed-off-by: Cai Zhang <[email protected]>
@xiaocai2333
fix ut
2fd0e7d 
Signed-off-by: Cai Zhang <[email protected]>
@xiaocai2333 xiaocai2333 force-pushed the refine_index_scheduler branch from c78c46d to 2fd0e7d Compare January 20, 2025 02:22
@mergify mergify bot added the ci-passed label Jan 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@congqixia congqixia Awaiting requested review from congqixia

Assignees
No one assigned
Labels
area/test ci-passed dco-passed DCO check passed. do-not-merge/hold kind/enhancement Issues or changes related to enhancement sig/testing size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xiaocai2333 @sre-ci-robot @czs007