enhance: add workaround for concurrent tantivy reload and directory removal

[sunby]

issue: #39428

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sunby
To complete the pull request process, please assign wxyucs after the PR has been reviewed.
You can assign the PR to them by writing /assign @wxyucs in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• internal/core/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

[codecov]

Codecov Report

Attention: Patch coverage is 57.14286% with 3 lines in your changes missing coverage. Please review.

Project coverage is 80.99%. Comparing base (cb959cd) to head (6fb0751).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/core/src/storage/LocalChunkManager.cpp	57.14%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #39440      +/-   ##
==========================================
+ Coverage   80.97%   80.99%   +0.02%     
==========================================
  Files        1408     1408              
  Lines      198569   198575       +6     
==========================================
+ Hits       160797   160843      +46     
+ Misses      32100    32066      -34     
+ Partials     5672     5666       -6     

Components	Coverage Δ
Client	79.50% <ø> (ø)
Core	69.47% <57.14%> (-0.01%)	⬇️
Go	82.93% <ø> (+0.02%)	⬆️

Files with missing lines	Coverage Δ
internal/core/src/storage/LocalChunkManager.cpp	65.38% <57.14%> (-0.62%)	⬇️

... and 36 files with indirect coverage changes