Skip to content
Snippets Groups Projects
  • hyukjinkwon's avatar
    1c403733
    [SPARK-8603][SPARKR] Use shell() instead of system2() for SparkR on Windows · 1c403733
    hyukjinkwon authored
    ## What changes were proposed in this pull request?
    
    This PR corrects SparkR to use `shell()` instead of `system2()` on Windows.
    
    Using `system2(...)` on Windows does not process windows file separator `\`. `shell(tralsate = TRUE, ...)` can treat this problem. So, this was changed to be chosen according to OS.
    
    Existing tests were failed on Windows due to this problem. For example, those were failed.
    
      ```
    8. Failure: sparkJars tag in SparkContext (test_includeJAR.R#34)
    9. Failure: sparkJars tag in SparkContext (test_includeJAR.R#36)
    ```
    
    The cases above were due to using of `system2`.
    
    In addition, this PR also fixes some tests failed on Windows.
    
      ```
    5. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#128)
    6. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#131)
    7. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#134)
    ```
    
      The cases above were due to a weird behaviour of `normalizePath()`. On Linux, if the path does not exist, it just prints out the input but it prints out including the current path on Windows.
    
      ```r
    # On Linus
    path <- normalizePath("aa")
    print(path)
    [1] "aa"
    
    # On Windows
    path <- normalizePath("aa")
    print(path)
    [1] "C:\\Users\\aa"
    ```
    
    ## How was this patch tested?
    
    Jenkins tests and manually tested in a Window machine as below:
    
    Here is the [stdout](https://gist.github.com/HyukjinKwon/4bf35184f3a30f3bce987a58ec2bbbab) of testing.
    
    Closes #7025
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    Author: Hyukjin Kwon <gurwls223@gmail.com>
    Author: Prakash PC <prakash.chinnu@gmail.com>
    
    Closes #13165 from HyukjinKwon/pr/7025.
    1c403733
    History
    [SPARK-8603][SPARKR] Use shell() instead of system2() for SparkR on Windows
    hyukjinkwon authored
    ## What changes were proposed in this pull request?
    
    This PR corrects SparkR to use `shell()` instead of `system2()` on Windows.
    
    Using `system2(...)` on Windows does not process windows file separator `\`. `shell(tralsate = TRUE, ...)` can treat this problem. So, this was changed to be chosen according to OS.
    
    Existing tests were failed on Windows due to this problem. For example, those were failed.
    
      ```
    8. Failure: sparkJars tag in SparkContext (test_includeJAR.R#34)
    9. Failure: sparkJars tag in SparkContext (test_includeJAR.R#36)
    ```
    
    The cases above were due to using of `system2`.
    
    In addition, this PR also fixes some tests failed on Windows.
    
      ```
    5. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#128)
    6. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#131)
    7. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#134)
    ```
    
      The cases above were due to a weird behaviour of `normalizePath()`. On Linux, if the path does not exist, it just prints out the input but it prints out including the current path on Windows.
    
      ```r
    # On Linus
    path <- normalizePath("aa")
    print(path)
    [1] "aa"
    
    # On Windows
    path <- normalizePath("aa")
    print(path)
    [1] "C:\\Users\\aa"
    ```
    
    ## How was this patch tested?
    
    Jenkins tests and manually tested in a Window machine as below:
    
    Here is the [stdout](https://gist.github.com/HyukjinKwon/4bf35184f3a30f3bce987a58ec2bbbab) of testing.
    
    Closes #7025
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    Author: Hyukjin Kwon <gurwls223@gmail.com>
    Author: Prakash PC <prakash.chinnu@gmail.com>
    
    Closes #13165 from HyukjinKwon/pr/7025.