-
- Downloads
[SPARK-21764][TESTS] Fix tests failures on Windows: resources not being closed and incorrect paths
## What changes were proposed in this pull request? `org.apache.spark.deploy.RPackageUtilsSuite` ``` - jars without manifest return false *** FAILED *** (109 milliseconds) java.io.IOException: Unable to delete file: C:\projects\spark\target\tmp\1500266936418-0\dep1-c.jar ``` `org.apache.spark.deploy.SparkSubmitSuite` ``` - download one file to local *** FAILED *** (16 milliseconds) java.net.URISyntaxException: Illegal character in authority at index 6: s3a://C:\projects\spark\target\tmp\test2630198944759847458.jar - download list of files to local *** FAILED *** (0 milliseconds) java.net.URISyntaxException: Illegal character in authority at index 6: s3a://C:\projects\spark\target\tmp\test2783551769392880031.jar ``` `org.apache.spark.scheduler.ReplayListenerSuite` ``` - Replay compressed inprogress log file succeeding on partial read (156 milliseconds) Exception encountered when attempting to run a suite with class name: org.apache.spark.scheduler.ReplayListenerSuite *** ABORTED *** (1 second, 391 milliseconds) java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-8f3cacd6-faad-4121-b901-ba1bba8025a0 - End-to-end replay *** FAILED *** (62 milliseconds) java.io.IOException: No FileSystem for scheme: C - End-to-end replay with compression *** FAILED *** (110 milliseconds) java.io.IOException: No FileSystem for scheme: C ``` `org.apache.spark.sql.hive.StatisticsSuite` ``` - SPARK-21079 - analyze table with location different than that of individual partitions *** FAILED *** (875 milliseconds) org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string); - SPARK-21079 - analyze partitioned table with only a subset of partitions visible *** FAILED *** (47 milliseconds) org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string); ``` **Note:** this PR does not fix: `org.apache.spark.deploy.SparkSubmitSuite` ``` - launch simple application with spark-submit with redaction *** FAILED *** (172 milliseconds) java.util.NoSuchElementException: next on empty iterator ``` I can't reproduce this on my Windows machine but looks appearntly consistently failed on AppVeyor. This one is unclear to me yet and hard to debug so I did not include this one for now. **Note:** it looks there are more instances but it is hard to identify them partly due to flakiness and partly due to swarming logs and errors. Will probably go one more time if it is fine. ## How was this patch tested? Manually via AppVeyor: **Before** - `org.apache.spark.deploy.RPackageUtilsSuite`: https://ci.appveyor.com/project/spark-test/spark/build/771-windows-fix/job/8t8ra3lrljuir7q4 - `org.apache.spark.deploy.SparkSubmitSuite`: https://ci.appveyor.com/project/spark-test/spark/build/771-windows-fix/job/taquy84yudjjen64 - `org.apache.spark.scheduler.ReplayListenerSuite`: https://ci.appveyor.com/project/spark-test/spark/build/771-windows-fix/job/24omrfn2k0xfa9xq - `org.apache.spark.sql.hive.StatisticsSuite`: https://ci.appveyor.com/project/spark-test/spark/build/771-windows-fix/job/2079y1plgj76dc9l **After** - `org.apache.spark.deploy.RPackageUtilsSuite`: https://ci.appveyor.com/project/spark-test/spark/build/775-windows-fix/job/3803dbfn89ne1164 - `org.apache.spark.deploy.SparkSubmitSuite`: https://ci.appveyor.com/project/spark-test/spark/build/775-windows-fix/job/m5l350dp7u9a4xjr - `org.apache.spark.scheduler.ReplayListenerSuite`: https://ci.appveyor.com/project/spark-test/spark/build/775-windows-fix/job/565vf74pp6bfdk18 - `org.apache.spark.sql.hive.StatisticsSuite`: https://ci.appveyor.com/project/spark-test/spark/build/775-windows-fix/job/qm78tsk8c37jb6s4 Jenkins tests are required and AppVeyor tests will be triggered. Author: hyukjinkwon <gurwls223@gmail.com> Closes #18971 from HyukjinKwon/windows-fixes.
Showing
- core/src/test/scala/org/apache/spark/deploy/RPackageUtilsSuite.scala 4 additions, 3 deletions...st/scala/org/apache/spark/deploy/RPackageUtilsSuite.scala
- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala 2 additions, 2 deletions...test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala 30 additions, 23 deletions...cala/org/apache/spark/scheduler/ReplayListenerSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala 3 additions, 3 deletions...est/scala/org/apache/spark/sql/hive/StatisticsSuite.scala
Loading
Please register or sign in to comment