-
- Downloads
[SPARK-9244] Increase some memory defaults
There are a few memory limits that people hit often and that we could make higher, especially now that memory sizes have grown. - spark.akka.frameSize: This defaults at 10 but is often hit for map output statuses in large shuffles. This memory is not fully allocated up-front, so we can just make this larger and still not affect jobs that never sent a status that large. We increase it to 128. - spark.executor.memory: Defaults at 512m, which is really small. We increase it to 1g. Author: Matei Zaharia <matei@databricks.com> Closes #7586 from mateiz/configs and squashes the following commits: ce0038a [Matei Zaharia] [SPARK-9244] Increase some memory defaults
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/util/AkkaUtils.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/util/AkkaUtils.scala
- core/src/test/java/org/apache/spark/JavaAPISuite.java 1 addition, 1 deletioncore/src/test/java/org/apache/spark/JavaAPISuite.java
- core/src/test/scala/org/apache/spark/ContextCleanerSuite.scala 2 additions, 2 deletions...src/test/scala/org/apache/spark/ContextCleanerSuite.scala
- core/src/test/scala/org/apache/spark/DistributedSuite.scala 8 additions, 8 deletionscore/src/test/scala/org/apache/spark/DistributedSuite.scala
- core/src/test/scala/org/apache/spark/DriverSuite.scala 1 addition, 1 deletioncore/src/test/scala/org/apache/spark/DriverSuite.scala
- core/src/test/scala/org/apache/spark/ExternalShuffleServiceSuite.scala 1 addition, 1 deletion.../scala/org/apache/spark/ExternalShuffleServiceSuite.scala
- core/src/test/scala/org/apache/spark/FileServerSuite.scala 3 additions, 3 deletionscore/src/test/scala/org/apache/spark/FileServerSuite.scala
- core/src/test/scala/org/apache/spark/JobCancellationSuite.scala 2 additions, 2 deletions...rc/test/scala/org/apache/spark/JobCancellationSuite.scala
- core/src/test/scala/org/apache/spark/ShuffleSuite.scala 10 additions, 10 deletionscore/src/test/scala/org/apache/spark/ShuffleSuite.scala
- core/src/test/scala/org/apache/spark/SparkContextSchedulerCreationSuite.scala 1 addition, 1 deletion...org/apache/spark/SparkContextSchedulerCreationSuite.scala
- core/src/test/scala/org/apache/spark/broadcast/BroadcastSuite.scala 4 additions, 4 deletions...est/scala/org/apache/spark/broadcast/BroadcastSuite.scala
- core/src/test/scala/org/apache/spark/deploy/LogUrlsStandaloneSuite.scala 2 additions, 2 deletions...cala/org/apache/spark/deploy/LogUrlsStandaloneSuite.scala
- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala 2 additions, 2 deletions...test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala 1 addition, 1 deletion.../spark/scheduler/CoarseGrainedSchedulerBackendSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala 1 addition, 1 deletion...rg/apache/spark/scheduler/EventLoggingListenerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala 1 addition, 1 deletion...cala/org/apache/spark/scheduler/ReplayListenerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/SparkListenerWithClusterSuite.scala 1 addition, 1 deletion...pache/spark/scheduler/SparkListenerWithClusterSuite.scala
- core/src/test/scala/org/apache/spark/serializer/KryoSerializerDistributedSuite.scala 1 addition, 1 deletion...che/spark/serializer/KryoSerializerDistributedSuite.scala
- core/src/test/scala/org/apache/spark/util/collection/ExternalAppendOnlyMapSuite.scala 5 additions, 5 deletions...he/spark/util/collection/ExternalAppendOnlyMapSuite.scala
Loading
Please register or sign in to comment