Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] clean stale column stats periodically (backport #45839) #46152

Merged
merged 2 commits into from
May 24, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented May 23, 2024

Why I'm doing:

Truncate operation didn't clear the stale column statistics.

What I'm doing:

  • The record drop partition step in OlapTable.dropPartition() interface leads it will be processed in follower replay operation. So move it to the outer process to ensure it only be processed in leader.
  • add a periodic task to clear the stale column statistics. The strategy is:
    • do the clear task once every 12 hours.
    • After the last cleanup, if a table has successfully undergone a statistics collection, and the collection completion time is after the last cleanup time (or it is before the cleanup time in 30 minutes), then during the next cleanup process, the stale column statistics would be cleared.

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

This is an automatic backport of pull request #45839 done by [Mergify](https://mergify.com). ## Why I'm doing: Truncate operation didn't clear the stale column statistics.

What I'm doing:

  • The record drop partition step in OlapTable.dropPartition() interface leads it will be processed in follower replay operation. So move it to the outer process to ensure it only be processed in leader.
  • add a periodic task to clear the stale column statistics. The strategy is:
    • do the clear task once every 12 hours.
    • After the last cleanup, if a table has successfully undergone a statistics collection, and the collection completion time is after the last cleanup time (or it is before the cleanup time in 30 minutes), then during the next cleanup process, the stale column statistics would be cleared.

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

@mergify mergify bot added the conflicts label May 23, 2024
Copy link
Contributor Author

mergify bot commented May 23, 2024

Cherry-pick of 1e4bd06 has failed:

On branch mergify/bp/branch-2.5/pr-45839
Your branch is up to date with 'origin/branch-2.5'.

You are currently cherry-picking commit 1e4bd06191.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   fe/fe-core/src/main/java/com/starrocks/common/Config.java
	new file:   test/sql/test_refresh_statistics/R/test_clear_stats
	new file:   test/sql/test_refresh_statistics/T/test_clear_stats

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/statistic/AnalyzeManager.java
	deleted by us:   fe/fe-core/src/main/java/com/starrocks/statistic/ExternalAnalyzeJob.java
	deleted by us:   fe/fe-core/src/main/java/com/starrocks/statistic/NativeAnalyzeJob.java
	both modified:   fe/fe-core/src/test/java/com/starrocks/statistic/StatisticsCollectJobTest.java

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@wanpengfei-git wanpengfei-git enabled auto-merge (squash) May 23, 2024 11:28
@mergify mergify bot closed this May 23, 2024
auto-merge was automatically disabled May 23, 2024 11:28

Pull request was closed

Copy link
Contributor Author

mergify bot commented May 23, 2024

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

@mergify mergify bot deleted the mergify/bp/branch-2.5/pr-45839 branch May 23, 2024 11:29
@packy92 packy92 restored the mergify/bp/branch-2.5/pr-45839 branch May 24, 2024 02:08
@packy92 packy92 reopened this May 24, 2024
@wanpengfei-git wanpengfei-git enabled auto-merge (squash) May 24, 2024 02:09
@packy92 packy92 force-pushed the mergify/bp/branch-2.5/pr-45839 branch from bc48608 to 1a412b2 Compare May 24, 2024 05:50
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ packy92
❌ mergify[bot]
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
B Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

@wanpengfei-git wanpengfei-git merged commit 411c8b8 into branch-2.5 May 24, 2024
30 of 32 checks passed
@wanpengfei-git wanpengfei-git deleted the mergify/bp/branch-2.5/pr-45839 branch May 24, 2024 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants