Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] AutoTuner/Bootstrapper: Auto-optimize shuffle should be disabled in Databricks #1529

Open
parthosa opened this issue Feb 5, 2025 · 2 comments
Labels
? - Needs Triage bug Something isn't working core_tools Scope the core module (scala)

Comments

@parthosa
Copy link
Collaborator

parthosa commented Feb 5, 2025

AutoTuner/Bootstrapper should recommend disabling spark.databricks.adaptive.autoOptimizeShuffle.enabled in Databricks environment.

@parthosa parthosa added ? - Needs Triage bug Something isn't working core_tools Scope the core module (scala) labels Feb 5, 2025
@amahussein
Copy link
Collaborator

The autoTuner will disable it, if it is enabled.
Do we want to disable it anyway regardless of the user settings?

val aqeAutoShuffle = getPropertyValue("spark.databricks.adaptive.autoOptimizeShuffle.enabled")
if (!aqeAutoShuffle.isEmpty) {
appendRecommendation("spark.databricks.adaptive.autoOptimizeShuffle.enabled", "false")
}
appendRecommendation("spark.sql.shuffle.partitions", s"$shufflePartitions")
}

@amahussein
Copy link
Collaborator

Lets keep more context in the issue description. The purpose is that being self explained for someone to pickup the task if necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
? - Needs Triage bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Development

No branches or pull requests

2 participants