Track the total number of compaction input iterators #13320

archang19 · 2025-01-21T20:13:02Z

Summary

This PR adds a new statistic to track the total number of input iterators for running compactions.

Context: I am currently working on a separate project, where I am trying to tune the read request sizes made by FilePrefetchBuffer to the storage backend. In this particular case, FilePrefetchBuffer will issue larger reads and have to buffer larger read responses. This means we expect to see higher memory utilization. At least for the initial rollout, we only want to enable this optimization for compaction reads.

I want some way to get a sense of what the memory usage impact will be if the prefetch read request size is increased from (for instance) 8MB to 64MB.

If I know the number of files that compactions are actively reading from (i.e. the number of input iterators), I can determine how much the memory usage will increase if I bump up the readahead size inside FilePrefetchBuffer.

Alternatives considered:

Add histogram statistic for number of compaction input iterators #13299 gives you a histogram for the number of input iterators for a single compaction. While this statistic is interested and in the direction of what we want, we are going to be assessing the memory impact across all compactions that are currently running. Thus, this statistic does not give us all the information we need.
Add statistic to track compaction prefetching memory usage #13302 gives you the total prefetch buffer memory usage, but it doesn't tell you what happens when the readahead size is increased. Furthermore, the code change is error prone and very "invasive" -- look at how many places in the code had to be updated.

Test Plan

I updated one unit test to confirm that num_running_compaction_input_iterators starts and ends at 0 (all the additions and subtractions cancel out). I added plenty of asserts to make sure that my new statistic was in the expected state. When I added fprintf manually, I confirmed that my statistics updating code was being exercised numerous times inside db_compaction_test.

We will also monitor the generated statistics after this PR is merged.

We also have the crash tests which will be able to detect if any of my asserts fail.

archang19 · 2025-01-22T02:08:21Z

db/db_impl/db_impl.h

@@ -2963,6 +2966,10 @@ class DBImpl : public DB {
  // stores the number of compactions are currently running
  int num_running_compactions_;

+  // stores the number of input iterators required for currently running
+  // compactions


uint64_t or size_t makes more sense to me since num_running_compaction_input_iterators_ should not be negative. However, I wanted to follow the convention here since everything else is using int

archang19 · 2025-01-22T02:19:12Z

db/db_impl/db_impl.h

@@ -2963,6 +2966,10 @@ class DBImpl : public DB {
  // stores the number of compactions are currently running
  int num_running_compactions_;

+  // stores the number of input iterators required for currently running
+  // compactions
+  int num_running_compaction_input_iterators_;


I see we have InstrumentedMutexLock l(&mutex_); protecting num_running_compactions_ as well as num_running_flushes_.

I guess that means we do not need std::atomic<int> for num_running_compaction_input_iterators_

facebook-github-bot · 2025-01-22T02:40:35Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-01-22T02:56:02Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-22T02:56:24Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

archang19 · 2025-01-22T02:57:35Z

db/db_impl/db_impl.h

@@ -957,6 +957,13 @@ class DBImpl : public DB {
    return num_running_compactions_;
  }

+  // Returns the number of input iterators for currently running compactions.
+  // REQUIREMENT: mutex_ must be held when calling this function.
+  int num_running_compaction_input_iterators() {


I noticed num_running_compactions did not follow the example of num_running_flushes, where db->num_running_flushes() gets called inside HandleNumRunningFlushes. HandleNumRunningCompactions was just accessing num_running_compactions_ directly.

facebook-github-bot · 2025-01-22T17:11:17Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-22T17:12:05Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

archang19 · 2025-01-22T17:15:02Z

@cbi42 I added you since this touches compaction code. Perhaps you know a better way of doing this or even another way to obtain the same information

anand1976 · 2025-01-22T17:43:42Z

db/db_impl/db_impl_compaction_flush.cc

@@ -61,6 +61,18 @@ bool DBImpl::EnoughRoomForCompaction(
  return enough_room;
 }

+size_t DBImpl::GetNumberCompactionInputIterators(Compaction* c) {


We should check for and skip trivial moves and deletion compactions since they won't have any input iterators

anand1976 · 2025-01-22T17:49:36Z

db/internal_stats.cc

@@ -1261,7 +1269,13 @@ bool InternalStats::HandleCompactionPending(uint64_t* value, DBImpl* /*db*/,

 bool InternalStats::HandleNumRunningCompactions(uint64_t* value, DBImpl* db,
                                                Version* /*version*/) {
-  *value = db->num_running_compactions_;
+  *value = db->num_running_compactions();


Doesn't num_running_compactions() assert that DB mutex is held?

yes, it does. I think (?) that is what we want. num_running_flushes() also checks mutex_.AssertHeld(); and is called from HandleNumRunningFlushes

Ok I saw your other comment. I will update this PR to just read off the values.

anand1976 · 2025-01-22T17:50:58Z

db/internal_stats.cc

+
+bool InternalStats::HandleNumRunningCompactionInputIterators(
+    uint64_t* value, DBImpl* db, Version* /*version*/) {
+  *value = db->num_running_compaction_input_iterators();


We're not holding the mutex here. Since these are stats, it should be ok to directly read the counter even if its not 100% accurate.

Should I also get rid of num_running_flushes(), which asserts the mutex is held?

anand1976 · 2025-01-22T17:52:21Z

db/db_impl/db_impl.h

-  Status BackgroundCompaction(bool* madeProgress, JobContext* job_context,
-                              LogBuffer* log_buffer,
+  Status BackgroundCompaction(bool* madeProgress,
+                              int& num_compaction_iterators_added,


C++ style guide suggests that output parameters go at the end

facebook-github-bot added the CLA Signed label Jan 21, 2025

archang19 changed the title ~~Add num-running-compaction-iterators stat~~ Add stat for total number of compaction input iterators Jan 21, 2025

archang19 mentioned this pull request Jan 21, 2025

Add histogram statistic for number of compaction input iterators #13299

Draft

archang19 changed the title ~~Add stat for total number of compaction input iterators~~ Track the total number of compaction input iterators Jan 21, 2025

archang19 added 6 commits January 21, 2025 15:59

Add num-running-compaction-iterators stat

87399e4

Constructor inside dbimpl

2c2ba95

input iterators

1e15c3d

Fix naming

e14d51a

Comment

010fdd4

Plenty of asserts

6a71c06

archang19 force-pushed the internal-stats-total-compaction branch from 5e67c21 to 6a71c06 Compare January 21, 2025 23:59

archang19 commented Jan 22, 2025

View reviewed changes

Update one compaction stats unit test

00f5c9a

archang19 marked this pull request as ready for review January 22, 2025 02:40

Assert mutex is held

9fec093

archang19 commented Jan 22, 2025

View reviewed changes

Add comment on precondition and another assertion

f73c9eb

archang19 requested review from anand1976 and cbi42 January 22, 2025 17:12

anand1976 reviewed Jan 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track the total number of compaction input iterators #13320

Track the total number of compaction input iterators #13320

archang19 commented Jan 21, 2025 •

edited

Loading

archang19 Jan 22, 2025

archang19 Jan 22, 2025 •

edited

Loading

facebook-github-bot commented Jan 22, 2025

facebook-github-bot commented Jan 22, 2025

facebook-github-bot commented Jan 22, 2025

archang19 Jan 22, 2025

facebook-github-bot commented Jan 22, 2025

facebook-github-bot commented Jan 22, 2025

archang19 commented Jan 22, 2025

anand1976 Jan 22, 2025

anand1976 Jan 22, 2025

archang19 Jan 22, 2025

archang19 Jan 22, 2025

anand1976 Jan 22, 2025

archang19 Jan 22, 2025

anand1976 Jan 22, 2025

Track the total number of compaction input iterators #13320

Are you sure you want to change the base?

Track the total number of compaction input iterators #13320

Conversation

archang19 commented Jan 21, 2025 • edited Loading

Summary

Test Plan

Choose a reason for hiding this comment

archang19 Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

facebook-github-bot commented Jan 22, 2025

facebook-github-bot commented Jan 22, 2025

facebook-github-bot commented Jan 22, 2025

Choose a reason for hiding this comment

facebook-github-bot commented Jan 22, 2025

facebook-github-bot commented Jan 22, 2025

archang19 commented Jan 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

archang19 commented Jan 21, 2025 •

edited

Loading

archang19 Jan 22, 2025 •

edited

Loading