Skip to content
Snippets Groups Projects
Commit 75b9e351 authored by Shixiong Zhu's avatar Shixiong Zhu Committed by Tathagata Das
Browse files

[SPARK-17346][SQL][TESTS] Fix the flaky topic deletion in KafkaSourceStressSuite

## What changes were proposed in this pull request?

A follow up Pr for SPARK-17346 to fix flaky `org.apache.spark.sql.kafka010.KafkaSourceStressSuite`.

Test log: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.4/1855/testReport/junit/org.apache.spark.sql.kafka010/KafkaSourceStressSuite/_It_is_not_a_test_/

Looks like deleting the Kafka internal topic `__consumer_offsets` is flaky. This PR just simply ignores internal topics.

## How was this patch tested?

Existing tests.

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #15384 from zsxwing/SPARK-17346-flaky-test.
parent c8c09064
No related branches found
No related tags found
No related merge requests found
......@@ -22,7 +22,6 @@ import java.util.concurrent.atomic.AtomicInteger
import scala.util.Random
import org.apache.kafka.clients.producer.RecordMetadata
import org.scalatest.BeforeAndAfter
import org.scalatest.time.SpanSugar._
import org.apache.spark.sql.execution.streaming._
......@@ -344,7 +343,7 @@ class KafkaSourceSuite extends KafkaSourceTest {
}
class KafkaSourceStressSuite extends KafkaSourceTest with BeforeAndAfter {
class KafkaSourceStressSuite extends KafkaSourceTest {
import testImplicits._
......@@ -358,12 +357,6 @@ class KafkaSourceStressSuite extends KafkaSourceTest with BeforeAndAfter {
start + Random.nextInt(start + end - 1)
}
after {
for (topic <- testUtils.getAllTopicsAndPartitionSize().toMap.keys) {
testUtils.deleteTopic(topic)
}
}
test("stress test with multiple topics and partitions") {
topics.foreach { topic =>
testUtils.createTopic(topic, partitions = nextInt(1, 6))
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment