-
- Downloads
[SPARK-1198] Allow pipes tasks to run in different sub-directories
This works as is on Linux/Mac/etc but doesn't cover working on Windows. In here I use ln -sf for symlinks. Putting this up for comments on that. Do we want to create perhaps some classes for doing shell commands - Linux vs Windows. Is there some other way we want to do this? I assume we are still supporting jdk1.6? Also should I update the Java API for pipes to allow this parameter? Author: Thomas Graves <tgraves@apache.org> Closes #128 from tgravescs/SPARK1198 and squashes the following commits: abc1289 [Thomas Graves] remove extra tag in pom file ba23fc0 [Thomas Graves] Add support for symlink on windows, remove commons-io usage da4b221 [Thomas Graves] Merge branch 'master' of https://github.com/tgravescs/spark into SPARK1198 61be271 [Thomas Graves] Fix file name filter 6b783bd [Thomas Graves] style fixes 1ab49ca [Thomas Graves] Add support for running pipe tasks is separate directories
Showing
- core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala 61 additions, 3 deletionscore/src/main/scala/org/apache/spark/rdd/PipedRDD.scala
- core/src/main/scala/org/apache/spark/rdd/RDD.scala 5 additions, 2 deletionscore/src/main/scala/org/apache/spark/rdd/RDD.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 44 additions, 1 deletioncore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/PipedRDDSuite.scala 27 additions, 1 deletioncore/src/test/scala/org/apache/spark/PipedRDDSuite.scala
Loading
Please register or sign in to comment