Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] Fix transaction stream load TXN_IN_PROCESSING error (backport #54959) #55736

Closed
wants to merge 1 commit into from

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Feb 10, 2025

Why I'm doing:

Users occasionally encounter the exception TXN_IN_PROCESSING when using transaction stream load. TXN_IN_PROCESSING means someone is holding the StreamLoadContext lock and do something, and the new request can't be processed concurrently. This mechanism is designed to prevent the client from sending concurrent requests. But the exception also happens even the client sends requests sequentially which does not meet expectations. There are two possible reasons:

  1. there is a background thread _clean_stream_context to check and clean the timeout load periodically. It will hold the StreamLoadContext lock before each check. So it can conflict with the load request.
  2. when the last load finishes, it first sends the response to the client, and then releases the lock. It is possible for the client to receive the response before the server releases the lock and then send the next request. When the next request reaches the server, the lock may still not have been released, leading to this issue.

The second is introduced by #53564 recently, but this problem has been reported before it. So the first should be the primary reason, and I have verified it in a user's environment.

What I'm doing:

  1. for cause 1, reduce the conflict by removing the lock when checking the timeout. only hold the lock when aborting it because of timeout
  2. for cause 2, [BugFix] Fix transaction stream load lock leak #53564 is to solve the lock leak when there is no normal response, but if the request is normal, should release the lock before response.

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Signed-off-by: PengFei Li <[email protected]>
(cherry picked from commit 91fecd3)

# Conflicts:
#	be/src/runtime/stream_load/stream_load_context.cpp
#	be/src/runtime/stream_load/stream_load_context.h
#	be/test/http/transaction_stream_load_test.cpp
Copy link
Contributor Author

mergify bot commented Feb 10, 2025

Cherry-pick of 91fecd3 has failed:

On branch mergify/bp/branch-3.2/pr-54959
Your branch is up to date with 'origin/branch-3.2'.

You are currently cherry-picking commit 91fecd370b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/http/action/transaction_stream_load.cpp
	modified:   be/src/http/action/transaction_stream_load.h
	modified:   be/src/runtime/stream_load/transaction_mgr.cpp

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   be/src/runtime/stream_load/stream_load_context.cpp
	both modified:   be/src/runtime/stream_load/stream_load_context.h
	both modified:   be/test/http/transaction_stream_load_test.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify mergify bot added the conflicts label Feb 10, 2025
@wanpengfei-git wanpengfei-git enabled auto-merge (squash) February 10, 2025 03:21
@mergify mergify bot closed this Feb 10, 2025
auto-merge was automatically disabled February 10, 2025 03:21

Pull request was closed

Copy link
Contributor Author

mergify bot commented Feb 10, 2025

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant