Skip to content
  • Felix Cheung's avatar
    a7ab6f9a
    [SPARK-19324][SPARKR] Spark VJM stdout output is getting dropped in SparkR · a7ab6f9a
    Felix Cheung authored
    ## What changes were proposed in this pull request?
    
    This affects mostly running job from the driver in client mode when results are expected to be through stdout (which should be somewhat rare, but possible)
    
    Before:
    ```
    > a <- as.DataFrame(cars)
    > b <- group_by(a, "dist")
    > c <- count(b)
    > sparkR.callJMethod(c$countjc, "explain", TRUE)
    NULL
    ```
    
    After:
    ```
    > a <- as.DataFrame(cars)
    > b <- group_by(a, "dist")
    > c <- count(b)
    > sparkR.callJMethod(c$countjc, "explain", TRUE)
    count#11L
    NULL
    ```
    
    Now, `column.explain()` doesn't seem very useful (we can get more extensive output with `DataFrame.explain()`) but there are other more complex examples with calls of `println` in Scala/JVM side, that are getting dropped.
    
    ## How was this patch tested?
    
    manual
    
    Author: Felix Cheung <felixcheung_m@hotmail.com>
    
    Closes #16670 from felixcheung/rjvmstdout.
    a7ab6f9a
    [SPARK-19324][SPARKR] Spark VJM stdout output is getting dropped in SparkR
    Felix Cheung authored
    ## What changes were proposed in this pull request?
    
    This affects mostly running job from the driver in client mode when results are expected to be through stdout (which should be somewhat rare, but possible)
    
    Before:
    ```
    > a <- as.DataFrame(cars)
    > b <- group_by(a, "dist")
    > c <- count(b)
    > sparkR.callJMethod(c$countjc, "explain", TRUE)
    NULL
    ```
    
    After:
    ```
    > a <- as.DataFrame(cars)
    > b <- group_by(a, "dist")
    > c <- count(b)
    > sparkR.callJMethod(c$countjc, "explain", TRUE)
    count#11L
    NULL
    ```
    
    Now, `column.explain()` doesn't seem very useful (we can get more extensive output with `DataFrame.explain()`) but there are other more complex examples with calls of `println` in Scala/JVM side, that are getting dropped.
    
    ## How was this patch tested?
    
    manual
    
    Author: Felix Cheung <felixcheung_m@hotmail.com>
    
    Closes #16670 from felixcheung/rjvmstdout.
Loading