-
- Downloads
[SPARK-19650] Commands should not trigger a Spark job
Spark executes SQL commands eagerly. It does this by creating an RDD which contains the command's results. The downside to this is that any action on this RDD triggers a Spark job which is expensive and is unnecessary. This PR fixes this by avoiding the materialization of an `RDD` for `Command`s; it just materializes the result and puts them in a `LocalRelation`. Added a regression test to `SQLQuerySuite`. Author: Herman van Hovell <hvanhovell@databricks.com> Closes #17027 from hvanhovell/no-job-command.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 7 additions, 13 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala 0 additions, 2 deletions...scala/org/apache/spark/sql/execution/QueryExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 1 addition, 2 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/test/resources/sql-tests/results/change-column.sql.out 2 additions, 2 deletions...rc/test/resources/sql-tests/results/change-column.sql.out
- sql/core/src/test/resources/sql-tests/results/group-by-ordinal.sql.out 1 addition, 1 deletion...test/resources/sql-tests/results/group-by-ordinal.sql.out
- sql/core/src/test/resources/sql-tests/results/order-by-ordinal.sql.out 1 addition, 1 deletion...test/resources/sql-tests/results/order-by-ordinal.sql.out
- sql/core/src/test/resources/sql-tests/results/outer-join.sql.out 2 additions, 2 deletions...e/src/test/resources/sql-tests/results/outer-join.sql.out
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 25 additions, 0 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
Loading
Please register or sign in to comment