-
- Downloads
[SPARK-11251] Fix page size calculation in local mode
``` // My machine only has 8 cores $ bin/spark-shell --master local[32] scala> val df = sc.parallelize(Seq((1, 1), (2, 2))).toDF("a", "b") scala> df.as("x").join(df.as("y"), $"x.a" === $"y.a").count() Caused by: java.io.IOException: Unable to acquire 2097152 bytes of memory at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPage(UnsafeExternalSorter.java:351) ``` Author: Andrew Or <andrew@databricks.com> Closes #9209 from andrewor14/fix-local-page-size.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 35 additions, 13 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/SparkEnv.scala 3 additions, 1 deletioncore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala 2 additions, 1 deletion...apache/spark/scheduler/OutputCommitCoordinatorSuite.scala
Loading
Please register or sign in to comment