-
- Downloads
[SPARK-18009][SQL] Fix ClassCastException while calling toLocalIterator() on...
[SPARK-18009][SQL] Fix ClassCastException while calling toLocalIterator() on dataframe produced by RunnableCommand ## What changes were proposed in this pull request? A short code snippet that uses toLocalIterator() on a dataframe produced by a RunnableCommand reproduces the problem. toLocalIterator() is called by thriftserver when `spark.sql.thriftServer.incrementalCollect`is set to handle queries producing large result set. **Before** ```SQL scala> spark.sql("show databases") res0: org.apache.spark.sql.DataFrame = [databaseName: string] scala> res0.toLocalIterator() 16/10/26 03:00:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow ``` **After** ```SQL scala> spark.sql("drop database databases") res30: org.apache.spark.sql.DataFrame = [] scala> spark.sql("show databases") res31: org.apache.spark.sql.DataFrame = [databaseName: string] scala> res31.toLocalIterator().asScala foreach println [default] [parquet] ``` ## How was this patch tested? Added a test in DDLSuite Author: Dilip Biswal <dbiswal@us.ibm.com> Closes #15642 from dilipbiswal/SPARK-18009.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 2 additions, 0 deletions...ala/org/apache/spark/sql/execution/command/commands.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala 7 additions, 0 deletions...ala/org/apache/spark/sql/execution/command/DDLSuite.scala
Please register or sign in to comment