-
- Downloads
[SPARK-11647] Attempt to reduce time/flakiness of Thriftserver CLI and SparkSubmit tests
This patch aims to reduce the test time and flakiness of HiveSparkSubmitSuite, SparkSubmitSuite, and CliSuite. Key changes: - Disable IO synchronization calls for Derby writes, since durability doesn't matter for tests. This was done for HiveCompatibilitySuite in #6651 and resulted in huge test speedups. - Add a few missing `--conf`s to disable various Spark UIs. The CliSuite, in particular, never disabled these UIs, leaving it prone to port-contention-related flakiness. - Fix two instances where tests defined `beforeAll()` methods which were never called because the appropriate traits were not mixed in. I updated these tests suites to extend `BeforeAndAfterEach` so that they play nicely with our `ResetSystemProperties` trait. Author: Josh Rosen <joshrosen@databricks.com> Closes #9623 from JoshRosen/SPARK-11647.
Showing
- core/src/test/scala/org/apache/spark/deploy/RPackageUtilsSuite.scala 7 additions, 5 deletions...st/scala/org/apache/spark/deploy/RPackageUtilsSuite.scala
- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala 4 additions, 2 deletions...test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
- sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala 14 additions, 7 deletions...ala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala 13 additions, 4 deletions...cala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala
Loading
Please register or sign in to comment