Skip to content

Commit

Permalink
fix: Reuse vector in LocalPartition (#122)
Browse files Browse the repository at this point in the history
Summary:
X-link: facebookincubator/velox#12002

Pull Request resolved: #122

More than 10% of the CPU are spent on the destruction of local partition output when the load is high.

Also add some optimizations for serialization.  Optimization on `ByteOutputStream::appendBool` does not show significant gain in the query in example (because they are a lot small batches), but it is net gain and would be significant in large batches, so I leave it in the code.

Reviewed By: xiaoxmeng

Differential Revision: D67742489

fbshipit-source-id: 8e70dd128f31caa7909ed7c1e2b4ac1e59d7c87d
  • Loading branch information
Yuhta authored and facebook-github-bot committed Jan 10, 2025
1 parent e3344c6 commit f858d12
Showing 1 changed file with 9 additions and 8 deletions.
17 changes: 9 additions & 8 deletions dwio/nimble/velox/FieldReader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -965,9 +965,11 @@ class ArrayWithOffsetsFieldReader final : public MultiValueFieldReader {
bool cachedLocally = rowCount > 0 && cached_ && (baseIndex == cachedIndex_);

// Initializes sizes and offsets in the vector.
auto& dictionaryValues =
const_cast<velox::VectorPtr&>(dictionaryVector->valueVector());
auto childrenRows = loadOffsets<velox::ArrayVector>(
dedupCount - cachedLocally,
dictionaryVector->valueVector(),
dictionaryValues,
/* scatterBitmap */ nullptr,
dedupCount);

Expand All @@ -985,13 +987,11 @@ class ArrayWithOffsetsFieldReader final : public MultiValueFieldReader {

elementsReader_->next(
childrenRows,
static_cast<velox::ArrayVector&>(*dictionaryVector->valueVector())
.elements(),
static_cast<velox::ArrayVector&>(*dictionaryValues).elements(),
/* scatterBitmap */ nullptr);

if (cachedLocally) {
auto vector = static_cast<velox::ArrayVector*>(
dictionaryVector->valueVector().get());
auto vector = static_cast<velox::ArrayVector*>(dictionaryValues.get());

// Copy elements from cache
const auto cacheIdx = static_cast<int64_t>(dedupCount) - 1;
Expand Down Expand Up @@ -1309,13 +1309,14 @@ class SlidingWindowMapFieldReader final : public FieldReader {
dictionaryVector->resize(rowCount);
dictionaryVector->resetNulls();
resetIfNotWritable(output, dictionaryVector->indices());
auto child =
verifyVectorState<velox::MapVector>(dictionaryVector->valueVector());
auto& dictionaryValues =
const_cast<velox::VectorPtr&>(dictionaryVector->valueVector());
auto child = verifyVectorState<velox::MapVector>(dictionaryValues);
if (child) {
child->resize(rowCount);
} else {
VectorInitializer<velox::MapVector>::initialize(
&pool_, dictionaryVector->valueVector(), type_, rowCount);
&pool_, dictionaryValues, type_, rowCount);
}
} else {
velox::VectorPtr child;
Expand Down

0 comments on commit f858d12

Please sign in to comment.