Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hotfix for MLFlow validator spans during async execution #1164

Merged
merged 1 commit into from
Nov 14, 2024

Conversation

CalebCourier
Copy link
Collaborator

This PR addresses #1163

To avoid orphaned spans being spawned from the Validator.validate calls executed in the ThreadPoolExecutor, we check if there is a current span and do not wrap the validate call in a new span if there isn't.
In conjunction with this, we now also wrap Validator.async_validate this does not experience the same issue as listed above because it itself is not run in the executor, rather it is the one running the executor.

This is not a long term solution for the root cause, but it does avoid the behaviour we see in the issue.

Other potential issues from the ThreadPoolExecutor are any validators that attempt to retrieve information from the ContextStore as it may not have the context that existed in the main thread. This would need to be tested and verified.

@CalebCourier CalebCourier requested a review from a team November 14, 2024 17:49
Copy link

@dtam dtam merged commit 93a6a36 into main Nov 14, 2024
12 checks passed
@dtam dtam deleted the temp-fix-async-mlflow branch November 14, 2024 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants