-
- Downloads
[SPARK-13053][TEST] Unignore tests in InternalAccumulatorSuite
These were ignored because they are incorrectly written; they don't actually trigger stage retries, which is what the tests are testing. These tests are now rewritten to induce stage retries through fetch failures. Note: there were 2 tests before and now there's only 1. What happened? It turns out that the case where we only resubmit a subset of of the original missing partitions is very difficult to simulate in tests without potentially introducing flakiness. This is because the `DAGScheduler` removes all map outputs associated with a given executor when this happens, and we will need multiple executors to trigger this case, and sometimes the scheduler still removes map outputs from all executors. Author: Andrew Or <andrew@databricks.com> Closes #10969 from andrewor14/unignore-accum-test.
Showing
- core/src/test/scala/org/apache/spark/AccumulatorSuite.scala 39 additions, 13 deletionscore/src/test/scala/org/apache/spark/AccumulatorSuite.scala
- core/src/test/scala/org/apache/spark/InternalAccumulatorSuite.scala 63 additions, 65 deletions...est/scala/org/apache/spark/InternalAccumulatorSuite.scala
Loading
Please register or sign in to comment