-
- Downloads
[SPARK-17585][PYSPARK][CORE] PySpark SparkContext.addFile supports adding files recursively
## What changes were proposed in this pull request? Users would like to add a directory as dependency in some cases, they can use ```SparkContext.addFile``` with argument ```recursive=true``` to recursively add all files under the directory by using Scala. But Python users can only add file not directory, we should also make it supported. ## How was this patch tested? Unit test. Author: Yanbo Liang <ybliang8@gmail.com> Closes #15140 from yanboliang/spark-17585.
Showing
- core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala 13 additions, 0 deletions...in/scala/org/apache/spark/api/java/JavaSparkContext.scala
- python/pyspark/context.py 5 additions, 2 deletionspython/pyspark/context.py
- python/pyspark/tests.py 15 additions, 5 deletionspython/pyspark/tests.py
- python/test_support/hello/hello.txt 0 additions, 0 deletionspython/test_support/hello/hello.txt
- python/test_support/hello/sub_hello/sub_hello.txt 1 addition, 0 deletionspython/test_support/hello/sub_hello/sub_hello.txt
Loading
Please register or sign in to comment