Skip to content
Snippets Groups Projects
Commit cafc696d authored by Imran Rashid's avatar Imran Rashid
Browse files

[HOTFIX][CORE] fix flaky BasicSchedulerIntegrationTest

## What changes were proposed in this pull request?

SPARK-15927 exacerbated a race in BasicSchedulerIntegrationTest, so it went from very unlikely to fairly frequent.  The issue is that stage numbering is not completely deterministic, but these tests treated it like it was.  So turn off the tests.

## How was this patch tested?

on my laptop the test failed abotu 10% of the time before this change, and didn't fail in 500 runs after the change.

Author: Imran Rashid <irashid@cloudera.com>

Closes #13688 from squito/hotfix_basic_scheduler.
parent 9bd80ad6
No related branches found
No related tags found
No related merge requests found
......@@ -518,10 +518,11 @@ class BasicSchedulerIntegrationSuite extends SchedulerIntegrationSuite[SingleCor
// make sure the required map output is available
task.stageId match {
case 1 => assertMapOutputAvailable(b)
case 3 => assertMapOutputAvailable(c)
case 4 => assertMapOutputAvailable(d)
case _ => // no shuffle map input, nothing to check
case _ =>
// we can't check for the output for the two intermediate stages, unfortunately,
// b/c the stage numbering is non-deterministic, so stage number alone doesn't tell
// us what to check
}
(task.stageId, task.stageAttemptId, task.partitionId) match {
......@@ -557,11 +558,9 @@ class BasicSchedulerIntegrationSuite extends SchedulerIntegrationSuite[SingleCor
val (taskDescription, task) = backend.beginTask()
stageToAttempts.getOrElseUpdate(task.stageId, new HashSet()) += task.stageAttemptId
// make sure the required map output is available
task.stageId match {
case 1 => assertMapOutputAvailable(shuffledRdd)
case _ => // no shuffle map input, nothing to check
}
// We cannot check if shuffle output is available, because the failed fetch will clear the
// shuffle output. Then we'd have a race, between the already-started task from the first
// attempt, and when the failure clears out the map output status.
(task.stageId, task.stageAttemptId, task.partitionId) match {
case (0, _, _) =>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment