Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] Add storage-related profile for cloud-native #54007

Merged
merged 7 commits into from
Dec 30, 2024
Merged

[Enhancement] Add storage-related profile for cloud-native #54007

merged 7 commits into from
Dec 30, 2024

Conversation

banmoy
Copy link
Contributor

@banmoy banmoy commented Dec 17, 2024 •
edited
Loading

Why I'm doing:

The follow-up work of #45560. This pr is for cloud-native

What I'm doing:

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.4
    • 3.3
    • 3.2
    • 3.1
    • 3.0

@banmoy banmoy requested review from a team as code owners December 17, 2024 02:14
@mergify mergify bot assigned banmoy Dec 17, 2024
murphyatwork
murphyatwork reviewed Dec 17, 2024
View reviewed changes
be/src/runtime/lake_tablets_channel.cpp
ADD_AND_UPDATE_COUNTER(profile, "MemoryExceedCount", TUnit::UNIT, writer_stat.memory_exceed_count);
ADD_AND_UPDATE_TIMER(profile, "WriteWaitFlushTime", writer_stat.write_wait_flush_tims_ns);
ADD_AND_UPDATE_TIMER(profile, "CloseTime", writer_stat.write_wait_flush_tims_ns);
ADD_AND_UPDATE_TIMER(profile, "FinishTime", writer_stat.finish_time_ns);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's better to add a threshold for these counters, otherwise they can be pretty noisy

@banmoy banmoy requested a review from a team as a code owner December 24, 2024 11:56
@banmoy banmoy changed the title [Enhancement][WIP] Add storage profile for cloud native [Enhancement] Add storage profile for cloud native Dec 24, 2024
@banmoy banmoy requested a review from a team as a code owner December 24, 2024 12:10
@banmoy banmoy changed the title [Enhancement] Add storage profile for cloud native [Enhancement] Add storage-related profile for cloud native Dec 25, 2024
@banmoy banmoy changed the title [Enhancement] Add storage-related profile for cloud native [Enhancement] Add storage-related profile for cloud-native Dec 25, 2024
@banmoy
Add cloud-native storage profile
35f5fc4 
Signed-off-by: PengFei Li <[email protected]>
@banmoy
Fix typo
576d18c 
Signed-off-by: PengFei Li <[email protected]>
@banmoy
Use atomic
5e5e2b1 
Signed-off-by: PengFei Li <[email protected]>
@banmoy
Fix compile problem
4a4fd6b 
Signed-off-by: PengFei Li <[email protected]>
@banmoy
Fix tests
7e6110f 
Signed-off-by: PengFei Li <[email protected]>
@banmoy
Add io stat for StarletOutputStream
f345b5e 
Signed-off-by: PengFei Li <[email protected]>
wyb
wyb reviewed Dec 26, 2024
View reviewed changes
be/src/runtime/lake_tablets_channel.cpp Outdated Show resolved Hide resolved
be/src/runtime/local_tablets_channel.cpp Outdated Show resolved Hide resolved
@banmoy
Fix CloseTime metric
fce5b79 
Signed-off-by: PengFei Li <[email protected]>
@github-actions GitHub Actions
Copy link

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@github-actions GitHub Actions
Copy link

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@github-actions GitHub Actions
Copy link

[BE Incremental Coverage Report]

✅ pass : 165 / 169 (97.63%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 be/src/storage/delta_writer.cpp 18 20 90.00% [645, 646]
🔵 be/src/runtime/lake_tablets_channel.cpp 71 73 97.26% [795, 815]
🔵 be/src/io/s3_output_stream.cpp 3 3 100.00% []
🔵 be/src/fs/fs_starlet.cpp 3 3 100.00% []
🔵 be/src/io/direct_s3_output_stream.cpp 3 3 100.00% []
🔵 be/src/storage/delta_writer.h 2 2 100.00% []
🔵 be/src/storage/lake/delta_writer.h 1 1 100.00% []
🔵 be/src/storage/memtable.cpp 14 14 100.00% []
🔵 be/src/storage/lake/delta_writer.cpp 38 38 100.00% []
🔵 be/src/storage/lake/async_delta_writer.cpp 9 9 100.00% []
🔵 be/src/runtime/local_tablets_channel.cpp 3 3 100.00% []

@trueeyu trueeyu merged commit f73d626 into StarRocks:main Dec 30, 2024
53 checks passed
maggie-zhu pushed a commit to maggie-zhu/starrocks that referenced this pull request Jan 6, 2025
@banmoy
[Enhancement] Add storage-related profile for cloud-native (StarRocks…
f9fdebb 
…#54007)

Signed-off-by: PengFei Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@murphyatwork murphyatwork murphyatwork left review comments

@starrocks-cr starrocks-cr[bot] starrocks-cr[bot] left review comments

@wyb wyb wyb approved these changes

@trueeyu trueeyu trueeyu approved these changes

Assignees

@banmoy banmoy

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@banmoy @wyb @trueeyu @murphyatwork