-
- Downloads
[SPARK-16189][SQL] Add ExternalRDD logical plan for input with RDD to have a...
[SPARK-16189][SQL] Add ExternalRDD logical plan for input with RDD to have a chance to eliminate serialize/deserialize. ## What changes were proposed in this pull request? Currently the input `RDD` of `Dataset` is always serialized to `RDD[InternalRow]` prior to being as `Dataset`, but there is a case that we use `map` or `mapPartitions` just after converted to `Dataset`. In this case, serialize and then deserialize happens but it would not be needed. This pr adds `ExistingRDD` logical plan for input with `RDD` to have a chance to eliminate serialize/deserialize. ## How was this patch tested? Existing tests. Author: Takuya UESHIN <ueshin@happy-camper.st> Closes #13890 from ueshin/issues/SPARK-16189.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 3 additions, 9 deletions...re/src/main/scala/org/apache/spark/sql/SparkSession.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala 64 additions, 8 deletions...in/scala/org/apache/spark/sql/execution/ExistingRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScanExec.scala 6 additions, 2 deletions...a/org/apache/spark/sql/execution/LocalTableScanExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 1 addition, 0 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala 7 additions, 1 deletionsql/core/src/test/scala/org/apache/spark/sql/QueryTest.scala
Loading
Please register or sign in to comment