Skip to content
Snippets Groups Projects
Commit 4746674a authored by Shixiong Zhu's avatar Shixiong Zhu
Browse files

[SPARK-18617][SPARK-18560][TESTS] Fix flaky test: StreamingContextSuite....

[SPARK-18617][SPARK-18560][TESTS] Fix flaky test: StreamingContextSuite. Receiver data should be deserialized properly

## What changes were proposed in this pull request?

Avoid to create multiple threads to stop StreamingContext. Otherwise, the latch added in #16091 can be passed too early.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #16105 from zsxwing/SPARK-18617-2.

(cherry picked from commit 086b0c8f)
Signed-off-by: default avatarShixiong Zhu <shixiong@databricks.com>
parent 4c673c65
No related branches found
No related tags found
No related merge requests found
......@@ -814,10 +814,12 @@ class StreamingContextSuite extends SparkFunSuite with BeforeAndAfter with Timeo
ssc = new StreamingContext(conf, Milliseconds(100))
val input = ssc.receiverStream(new TestReceiver)
val latch = new CountDownLatch(1)
@volatile var stopping = false
input.count().foreachRDD { rdd =>
// Make sure we can read from BlockRDD
if (rdd.collect().headOption.getOrElse(0L) > 0) {
if (rdd.collect().headOption.getOrElse(0L) > 0 && !stopping) {
// Stop StreamingContext to unblock "awaitTerminationOrTimeout"
stopping = true
new Thread() {
setDaemon(true)
override def run(): Unit = {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment