AutoTuner/Bootstrapper should recommend Dataproc Spark performance enhancements #1539
+222
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Partho Sarthi <[email protected]>
parthosa
added
feature request
amahussein
reviewed
Feb 7, 2025
| "spark.dataproc.enhanced.optimizer.enabled" -> "true", | ||
| "spark.dataproc.enhanced.execution.enabled" -> "true" | ||
| ) | ||
|
|
There was a problem hiding this comment.
should we add those to the yaml file?
There was a problem hiding this comment.
Added to tuning yaml file.
| "--conf spark.dataproc.enhanced.optimizer.enabled=true", | ||
| "--conf spark.dataproc.enhanced.execution.enabled=true" | ||
| ) | ||
| assert(expectedResults.forall(autoTunerOutput.contains)) |
There was a problem hiding this comment.
The assertion is not enough because it is possible that autotuner put two different entries for the same property.
for example, -conf spark.dataproc.enhanced.optimizer.enabled=true and another one with false..
then we need to check that each property exists exactly once.
There was a problem hiding this comment.
Added the check for comparing the complete AutoTuner output.
Signed-off-by: Partho Sarthi <[email protected]>
Signed-off-by: Partho Sarthi <[email protected]>
Signed-off-by: Partho Sarthi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1538.
This PR updates AutoTuner/Bootstrapper to recommend the following Dataproc Spark performance enhancements
Reference - https://cloud.google.com/dataproc/docs/guides/performance-enhancements