v24.06.1

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.06.1/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.06.1/

Changes

User Tools

• Fix Python runtime error caused by numpy 2.0.0 release (#1130)
• Disable the spark_rapids bootstrap command (#1114)

Core

• Handle different exception thrown by incomplete eventlogs (#1124)
• Include number of executors per node in cluster information (#1119)

v24.06.0

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.06.0/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.06.0/

Changes

User Tools

• Add support to Python 3.12 (#1111)
• user-tools: Update log messages (#1110)
• Enable xgboost prediction model by default (#1108)
• Add support to Python3.11 (#1105)
• Fix nan label issue in training (#1104)
• Fix qualx app metrics (#1102)
• clip appDuration to at least Duration (#1096)
• Fix missing assignment to savings_recommendations (#1098)
• Handle QualX behaviour when Qual Tool does not generate any outputs (#1095)
• Fix internal predict CLI and remove preprocessed argument (#1093)
• Update QualX to return default speedups and fix App Duration for incomplete apps (#1089)
• fix signature error from overlapping merges (#1084)
• sync w/ internal repo; update models (#1083)
• Reduce the maximum number of Java threads in CLI (#1082)
• Remove using Profiler metrics for QualX and Heuristics (#1080)
• Port QualX repo and add CLI for train (#1076)
• User tools fallback to default zone/region (#1054)
• Handle missing pricing info for user qual tool on Databricks platforms (#1053)
• Split job and stage level aggregated metrics into different files (#1050)
• Skip Cluster Inference when CSP CLIs are missing or not configured (#1035)
• Store Cluster Shape Recommendation in User Tools Qualification Output (#1005)
• Fix calculation of unsupported operators stage duration percentage (#1006)
• Update Databricks Azure qual tool to set env variable for ABFS paths (#1016)
• Add heuristics using stage spill metrics to skip apps (#1002)
• Fix failure in github workflow's pylint (#1015)
• Updating qual validation script to directly use top candidate view recommendation (#1001)

Core

• Fix typo in Profiler class using qual instead of prof (#1113)
• Fix missing appEndTime in raw_metrics folder (#1092)
• Sync tools with plugin newly supported operators (#1066)
• Fix java Qual tool Autotuner output when GPU device is missing (#1085)
• Update the Qual tool AutoTuner Heuristics against CPU event logs (#1069)
• Handling FileNotFound exception in AutoTuner (#1065)
• Handle metric names from legacy spark (#1052)
• Split job and stage level aggregated metrics into different files (#1050)
• Refactor ProfileResult classes to implement new interface design and add CSV output to Qual Tool (#1043)
• Hook up the auto tuner in the qualification tool (#1039)
• Profiler should identify the delta log ops and generate views for non-delta logs (#1031)
• Qualification tool - Handle cancelled jobs and stages better and don't skip the app (#1033)
• [FEA] Generate Status Report for Profiling Tool (#1012)
• Fix calculation of unsupported operators stage duration percentage (#1006)
• Fix potential problems and AQE updates in Qual tool (#1021)
• Sync supported operators with plugin changes and update default score (#1020)
• Refactor TaskEnd to be accessible by Q/P tools (#1000)

Miscellaneous

• Bump requests from 2.31.0 to 2.32.2 in /data_validation (#1077)

v24.04.0

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.04.0/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.04.0/

Changes

User Tools

• [FEA] Add CLI to run prediction on estimation_model (#961)
• Adding SHAP predict values as new output file (#982)
• Update docs for building to clarify to build in a virtual environment (#976)

Core

• [BUG] Catch Profiler error when app info is empty (#994)
• Get stages from sqlId for collecting info for output writer functions (#996)
• Account for joboverhead time in qualification tool estimation (#992)
• [Followup] Fix handling of clusterTags and SparkVersion in Q/P Tools (#993)
• Fix handling of clusterTags and SparkVersion in Q/P Tools (#991)
• Refactor AppBase to use common AppMetaData between Q/P tools (#983)
• Refactor Stage info code between Q/P tools (#971)

v24.02.4

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.4/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.4/

Changes

User Tools

• Fix Hadoop Azure version to be compatibe with Spark-3.5.0 (#975)
• Add speedup categories in qualification summary output (#958)
• Improve cluster node initialisation for CSPs (#964)

Core

• Remove databricks profiling recommendation for dynamicFilePruning (#972)
• Add AQEShuffleRead WriteFiles execs to the supportedOps and score files (#963)
• [FEA] Automate appending new operators to the platform score sheets (#954)
• Add support for InSubqueryExec Expression (#960)

Miscellaneous

• Bump dev version to 24.02.4 (#968)
• Revert versions back to 24.02.3 (#967)

v24.02.3

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.3/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.3/

Changes

User Tools

• Cache CLI calls for node instance description (#952)
• Improve error handling in prediction code (#950)
• Support dynamic calculation of JVM resources in CLI cmd (#944)
• Syncup estimation model prediction logic updates (#946)
• Cluster inference should not run for unsupported platform (#941)
• Fix invalid values in cluster creation script (#935)
• Fix core tool doc links and user qualification tool default argument values (#931)
• Fix gpu cluster recommendation in user tools (#930)
• Bump idna from 3.4 to 3.7 in /data_validation (#932)
• Add cluster details in qualification summary output (#921)
• Refactor find_matches_for_node return values (#920)
• [FEA] Add and use g5 AWS instances as default for qualification tool output (#898)
• Add jar argument to spark_rapids CLI (#902)
• Support driverlog argument in profiler CLI (#897)

Core

• Followups on handling Photon eventlogs (#953)
• Sync operators support timestamped 24-04-16 (#951)
• Add CheckOverflowInTableInsert support: verify absence from physical plan (#942)
• Fix Notes column in the supported ops CSV files (#933)
• Improve sync plugin supported CSV python script (#919)
• Add cluster details in qualification summary output (#921)
• Add support for unsupported expressions reasons per Exec (#923)
• Adding more metrics and options for qual validation (#926)
• Generate cluster details in JSON output (#912)
• Add Divide and multiple interval expressions as supported (#917)
• Add support for PythonMapInArrowExec and MapInArrowExec (#913)
• Re-enable support for GetJsonObject by default (#916)
• Add support for WindowGroupLimitExec (#906)
• [FEA] Skip Spark Structured Streaming event logs for Qualification tool (#905)
• [FEA] Add and use g5 AWS instances as default for qualification tool output (#898)
• Initial version of qual tool validation script for classification metrics (#903)
• Fix Delta-core dependency for Spark35+ (#904)
• Add support for AtomicCreateTableAsSelectExec (#895)
• Add support for KnownNullable and EphemeralSubstring expressions (#894)
• Add Support for BloomFilterAggregate and BloomFilterMightContain exprs (#891)
• [DOC] Update README for sync plugin supported ops script (#893)
• Add operators to ignore list and update WindowExpr parser (#890)
• Add support to RoundCeil and RoundFloor expressions (#889)

v24.02.2

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.2/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.2/

Changes

User Tools

• Override estimated speedups when estimation model is enabled (#885)
• [FEA] Make top candidates view as the default view in user-tools (#879)
• Introduce new csv file containing output for all apps before grouping (#875)
• Fix calculation of unsupported operators stages duration and update output row (#874)
• Implement top candidate filter for user tools CLI output (#866)

Core

• [FEA] Skip Databricks Photon jobs at app level in Qualification tool (#886)
• [FEA] Add Estimation Model to Qualification CLI (#870)
• Add rootExecutionID to output csv files (#871)
• [FEA] Generate updated supported CSV files from plugin repo (#847)
• Add action column to qual execs output (#859)
• Extend supportLevels in PluginTypeChecker (#863)
• Propagate Reason/Notes for operators disabled by default from plugin to Qualification tool unsupported operators csv file (#850)

Miscellaneous

• Bump default Spark-version to 3.5.0 (#877)
• Update Github actions version (#876)

v24.02.1

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.1/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.1/

Changes

User Tools

• Remove redundant initialization scripts from user tools output (#830)
• [DOC] Update Databricks Azure user tool setup instructions for output format (#826)
• Estimate cluster instances and generate cost savings (#803)

Core

• Fix implementation of processSQLPlanMetrics in Profiler (#853)
• Deduplicate SQL duration wallclock time for databricks eventlog (#810)
• Consider additional factors in spark.sql.shuffle.partitions recommendation in Autotuner (#722)
• Fix case matching error In AutoTuner (#828)
• Fix ReadSchema in Qualification tool and NPE in Profiling tool (#825)
• AutoTuner does not process arguments skipList and limitedLogic (#812)

v24.02.0

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.0/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.0/

Changes

User Tools

• Fix missing config file for Dataproc GKE (#778)
• [FEA] Qualification user_tools runs AutoTuner by default (#771)
• [BUG] Fix databricks-aws user profiling tool error with --gpu_cluster argument (#707)

Core

• [FEA] Qualification tool should mark WriteIntoDeltaCommand as supported (#801)
• Qualification tool should mark SubqueryExec as IgnoreNoPerf (#798)
• Generate cluster information from event logs in Qualification tool (#789)
• Sync up supported ops for 24.02 plugin release (#796)
• Qualification should mark empty2null as supported (#791)
• Incorrect parsing of aggregates in DB queries (#790)
• Qualification should mark WriteFiles as supported (#784)
• Introduce GpuDevice abstraction and refactor AutoTuner (#740)
• Consolidate unsupportedOperators into a single view (#766)
• Speedup generator script fails after adding runtime_properties (#776)
• Tools fail on DB10.4 clusters with IllegalArgException (#768)
• Fix SparkPlanGraphCluster constructor for DB Platforms (#765)
• Amendment to PR-763 (#764)
• Fix SQLPLanMetric constructor for DB Platforms (#763)
• Fix node constructor for DB platforms (#761)
• Add penalty for stages with UDF's (#757)
• Add support to appendDataExecV1 and overwriteByExprExecV1 (#756)
• Qualification fails to detect sortMergeJoin with arguments (#754)
• Fix Qualification crash during aggregation of stats (#753)
• [FEA] Extend the list of operators to be ignored in Qualification (#745)
• Remove ReusedSubquery from SparkPlanGraph construction (#741)
• Update unsupported operator csv file's app duration column (#748)
• [FEA] Qualification tool triggers the AutoTuner module (#739)
• Disable support of GetJsonObject in Qualification tool (#737)
• [FEA] AutoTuner warns that non-utf8 may not support some GPU expressions (#736)
• [FEA] AutoTuner should not skip non-gpu eventlogs (#728)

Miscellaneous

• Add auto-copyright for precommits (#732)

v23.12.3

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/23.12.3/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/23.12.3/

Changes

Core

• Add support of HiveTableScan and InsertIntoHive text-format (#723)
• Fix compilation error with JDK11 (#720)
• Generate an output file with runtime and build information (#705)
• AutoTuner should poll maven-meta to retrieve the latest jar version (#711)
• Profiling tool : Profiling tool throws NPE when appInfo is null and unchecked (#640)
• Add support to parse_url host and protocol (#708)
• [FEA] Profiling tool auto-tuner should consider spark.databricks.adaptive.autoOptimizeShuffle.enabled (#710)
• [FEA] Profiler autotuner should only specify standard Spark versions for shuffle manager setting (#662)

Miscellaneous

• [FEA] Enable AQE related recommendations in Profiler Auto-tuner (#688)

v23.12.2

Packages

• Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/23.12.2/
• PyPI Package: https://pypi.org/project/spark-rapids-user-tools/23.12.2/

Changes

User Tools

• Polling maven-metadata.xml to pull the latest tools jar (#703)

Core

• Update pom to fail on warnings (#701)