Skip to content
Snippets Groups Projects
  • hyukjinkwon's avatar
    b6a7aa4f
    [SPARK-19221][PROJECT INFRA][R] Add winutils binaries to the path in AppVeyor... · b6a7aa4f
    hyukjinkwon authored
    [SPARK-19221][PROJECT INFRA][R] Add winutils binaries to the path in AppVeyor tests for Hadoop libraries to call native codes properly
    
    ## What changes were proposed in this pull request?
    
    It seems Hadoop libraries need winutils binaries for native libraries in the path.
    
    It is not a problem in tests for now because we are only testing SparkR on Windows via AppVeyor but it can be a problem if we run Scala tests via AppVeyor as below:
    
    ```
     - SPARK-18220: read Hive orc table with varchar column *** FAILED *** (3 seconds, 937 milliseconds)
       org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
       at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:625)
       at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:609)
       at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283)
       ...
    ```
    
    This PR proposes to add it to the `Path` for AppVeyor tests.
    
    ## How was this patch tested?
    
    Manually via AppVeyor.
    
    **Before**
    https://ci.appveyor.com/project/spark-test/spark/build/549-windows-complete/job/gc8a1pjua2bc4i8m
    
    **After**
    https://ci.appveyor.com/project/spark-test/spark/build/572-windows-complete/job/c4vrysr5uvj2hgu7
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #16584 from HyukjinKwon/set-path-appveyor.
    b6a7aa4f
    History
    [SPARK-19221][PROJECT INFRA][R] Add winutils binaries to the path in AppVeyor...
    hyukjinkwon authored
    [SPARK-19221][PROJECT INFRA][R] Add winutils binaries to the path in AppVeyor tests for Hadoop libraries to call native codes properly
    
    ## What changes were proposed in this pull request?
    
    It seems Hadoop libraries need winutils binaries for native libraries in the path.
    
    It is not a problem in tests for now because we are only testing SparkR on Windows via AppVeyor but it can be a problem if we run Scala tests via AppVeyor as below:
    
    ```
     - SPARK-18220: read Hive orc table with varchar column *** FAILED *** (3 seconds, 937 milliseconds)
       org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
       at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:625)
       at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:609)
       at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283)
       ...
    ```
    
    This PR proposes to add it to the `Path` for AppVeyor tests.
    
    ## How was this patch tested?
    
    Manually via AppVeyor.
    
    **Before**
    https://ci.appveyor.com/project/spark-test/spark/build/549-windows-complete/job/gc8a1pjua2bc4i8m
    
    **After**
    https://ci.appveyor.com/project/spark-test/spark/build/572-windows-complete/job/c4vrysr5uvj2hgu7
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #16584 from HyukjinKwon/set-path-appveyor.