-
- Downloads
[SPARK-14475] Propagate user-defined context from driver to executors
## What changes were proposed in this pull request? This adds a new API call `TaskContext.getLocalProperty` for getting properties set in the driver from executors. These local properties are automatically propagated from the driver to executors. For streaming, the context for streaming tasks will be the initial driver context when ssc.start() is called. ## How was this patch tested? Unit tests. cc JoshRosen Author: Eric Liang <ekl@databricks.com> Closes #12248 from ericl/sc-2813.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 4 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/TaskContext.scala 8 additions, 1 deletioncore/src/main/scala/org/apache/spark/TaskContext.scala
- core/src/main/scala/org/apache/spark/TaskContextImpl.scala 5 additions, 0 deletionscore/src/main/scala/org/apache/spark/TaskContextImpl.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 16 additions, 1 deletioncore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 2 additions, 2 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/ResultTask.scala 4 additions, 1 deletion...rc/main/scala/org/apache/spark/scheduler/ResultTask.scala
- core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala 6 additions, 3 deletions...ain/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
- core/src/main/scala/org/apache/spark/scheduler/Task.scala 17 additions, 3 deletionscore/src/main/scala/org/apache/spark/scheduler/Task.scala
- core/src/test/scala/org/apache/spark/AccumulatorSuite.scala 3 additions, 2 deletionscore/src/test/scala/org/apache/spark/AccumulatorSuite.scala
- core/src/test/scala/org/apache/spark/ShuffleSuite.scala 4 additions, 3 deletionscore/src/test/scala/org/apache/spark/ShuffleSuite.scala
- core/src/test/scala/org/apache/spark/memory/MemoryTestingUtils.scala 3 additions, 0 deletions...st/scala/org/apache/spark/memory/MemoryTestingUtils.scala
- core/src/test/scala/org/apache/spark/scheduler/FakeTask.scala 3 additions, 1 deletion.../src/test/scala/org/apache/spark/scheduler/FakeTask.scala
- core/src/test/scala/org/apache/spark/scheduler/NotSerializableFakeTask.scala 2 additions, 1 deletion.../org/apache/spark/scheduler/NotSerializableFakeTask.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala 20 additions, 4 deletions...t/scala/org/apache/spark/scheduler/TaskContextSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 2 additions, 2 deletions...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala 4 additions, 1 deletion...cala/org/apache/spark/storage/BlockInfoManagerSuite.scala
- project/MimaExcludes.scala 3 additions, 0 deletionsproject/MimaExcludes.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala 3 additions, 0 deletions...k/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeKVExternalSorterSuite.scala 3 additions, 0 deletions...che/spark/sql/execution/UnsafeKVExternalSorterSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeRowSerializerSuite.scala 2 additions, 1 deletion...apache/spark/sql/execution/UnsafeRowSerializerSuite.scala
Loading
Please register or sign in to comment