Hotfix for MLFlow validator spans during async execution #1164
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses #1163
To avoid orphaned spans being spawned from the
Validator.validate
calls executed in the ThreadPoolExecutor, we check if there is a current span and do not wrap the validate call in a new span if there isn't.In conjunction with this, we now also wrap
Validator.async_validate
this does not experience the same issue as listed above because it itself is not run in the executor, rather it is the one running the executor.This is not a long term solution for the root cause, but it does avoid the behaviour we see in the issue.
Other potential issues from the ThreadPoolExecutor are any validators that attempt to retrieve information from the ContextStore as it may not have the context that existed in the main thread. This would need to be tested and verified.