Skip to content
Snippets Groups Projects
Commit 28b1af74 authored by Marcelo Vanzin's avatar Marcelo Vanzin Committed by Andrew Or
Browse files

[MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #5751 from vanzin/cached-rdd-warning and squashes the following commits:

554cc07 [Marcelo Vanzin] Change message.
9efb9da [Marcelo Vanzin] [minor] [core] Warn users who try to cache RDDs with dynamic allocation on.
parent 53befacc
No related branches found
No related tags found
No related merge requests found
...@@ -1396,6 +1396,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli ...@@ -1396,6 +1396,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
* Register an RDD to be persisted in memory and/or disk storage * Register an RDD to be persisted in memory and/or disk storage
*/ */
private[spark] def persistRDD(rdd: RDD[_]) { private[spark] def persistRDD(rdd: RDD[_]) {
_executorAllocationManager.foreach { _ =>
logWarning(
s"Dynamic allocation currently does not support cached RDDs. Cached data for RDD " +
s"${rdd.id} will be lost when executors are removed.")
}
persistentRdds(rdd.id) = rdd persistentRdds(rdd.id) = rdd
} }
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment