-
- Downloads
[SPARK-10710] Remove ability to disable spilling in core and SQL
It does not make much sense to set `spark.shuffle.spill` or `spark.sql.planner.externalSort` to false: I believe that these configurations were initially added as "escape hatches" to guard against bugs in the external operators, but these operators are now mature and well-tested. In addition, these configurations are not handled in a consistent way anymore: SQL's Tungsten codepath ignores these configurations and will continue to use spilling operators. Similarly, Spark Core's `tungsten-sort` shuffle manager does not respect `spark.shuffle.spill=false`. This pull request removes these configurations, adds warnings at the appropriate places, and deletes a large amount of code which was only used in code paths that did not support spilling. Author: Josh Rosen <joshrosen@databricks.com> Closes #8831 from JoshRosen/remove-ability-to-disable-spilling.
Showing
- core/src/main/scala/org/apache/spark/Aggregator.scala 15 additions, 44 deletionscore/src/main/scala/org/apache/spark/Aggregator.scala
- core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala 10 additions, 30 deletionscore/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala
- core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleManager.scala 7 additions, 1 deletion...la/org/apache/spark/shuffle/hash/HashShuffleManager.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala 8 additions, 2 deletions...la/org/apache/spark/shuffle/sort/SortShuffleManager.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 0 additions, 6 deletions...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala 11 additions, 11 deletions...test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
- docs/configuration.md 3 additions, 11 deletionsdocs/configuration.md
- docs/sql-programming-guide.md 0 additions, 7 deletionsdocs/sql-programming-guide.md
- python/pyspark/rdd.py 7 additions, 18 deletionspython/pyspark/rdd.py
- python/pyspark/shuffle.py 0 additions, 30 deletionspython/pyspark/shuffle.py
- python/pyspark/tests.py 1 addition, 12 deletionspython/pyspark/tests.py
- sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala 1 addition, 7 deletionssql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 0 additions, 2 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala 9 additions, 0 deletions.../main/scala/org/apache/spark/sql/execution/commands.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/sort.scala 2 additions, 28 deletions.../src/main/scala/org/apache/spark/sql/execution/sort.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 4 additions, 22 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/RowFormatConvertersSuite.scala 1 addition, 1 deletion...apache/spark/sql/execution/RowFormatConvertersSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/SortSuite.scala 2 additions, 2 deletions...test/scala/org/apache/spark/sql/execution/SortSuite.scala
Loading
Please register or sign in to comment