-
- Downloads
SPARK-1128: set hadoop task properties when constructing HadoopRDD
https://spark-project.atlassian.net/browse/SPARK-1128 The task properties are not set when constructing HadoopRDD in current implementation, this may limit the implementation based on ``` mapred.tip.id mapred.task.id mapred.task.is.map mapred.task.partition mapred.job.id ``` This patch also contains a small fix in createJobID (SparkHadoopWriter.scala), where the current implementation actually is not using time parameter Author: CodingCat <zhunansjtu@gmail.com> Author: Nan Zhu <CodingCat@users.noreply.github.com> Closes #101 from CodingCat/SPARK-1128 and squashes the following commits: ed0980f [CodingCat] make SparkHiveHadoopWriter belongs to spark package 5b1ad7d [CodingCat] move SparkHiveHadoopWriter to org.apache.spark package 258f92c [CodingCat] code cleanup af88939 [CodingCat] update the comments and permission of SparkHadoopWriter 9bd1fe3 [CodingCat] move configuration for jobConf to HadoopRDD b7bdfa5 [Nan Zhu] style fix a3153a8 [Nan Zhu] style fix c3258d2 [CodingCat] set hadoop task properties while using InputFormat
Showing
- core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala 10 additions, 19 deletionscore/src/main/scala/org/apache/spark/SparkHadoopWriter.scala
- core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala 24 additions, 1 deletioncore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala 3 additions, 4 deletions...rc/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
- sql/hive/src/main/scala/org/apache/spark/SparkHadoopWriter.scala 3 additions, 5 deletions...e/src/main/scala/org/apache/spark/SparkHadoopWriter.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveOperators.scala 1 addition, 1 deletion.../main/scala/org/apache/spark/sql/hive/hiveOperators.scala
Loading
Please register or sign in to comment