Skip to content
Snippets Groups Projects
Commit e5d0928e authored by Dongjoon Hyun's avatar Dongjoon Hyun Committed by Davies Liu
Browse files

[SPARK-16173] [SQL] Can't join describe() of DataFrame in Scala 2.10

## What changes were proposed in this pull request?

This PR fixes `DataFrame.describe()` by forcing materialization to make the `Seq` serializable. Currently, `describe()` of DataFrame throws `Task not serializable` Spark exceptions when joining in Scala 2.10.

## How was this patch tested?

Manual. (After building with Scala 2.10, test on `bin/spark-shell` and `bin/pyspark`.)

Author: Dongjoon Hyun <dongjoon@apache.org>

Closes #13900 from dongjoon-hyun/SPARK-16173.
parent 20768dad
No related branches found
No related tags found
No related merge requests found
......@@ -1908,7 +1908,8 @@ class Dataset[T] private[sql](
// All columns are string type
val schema = StructType(
StructField("summary", StringType) :: outputCols.map(StructField(_, StringType))).toAttributes
LocalRelation.fromExternalRows(schema, ret)
// `toArray` forces materialization to make the seq serializable
LocalRelation.fromExternalRows(schema, ret.toArray.toSeq)
}
/**
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment