Skip to content
Snippets Groups Projects
Commit fde1340c authored by Liwei Lin's avatar Liwei Lin Committed by Shixiong Zhu
Browse files

[SPARK-14701][STREAMING] First stop the event loop, then stop the checkpoint writer in JobGenerator

Currently if we call `streamingContext.stop` (e.g. in a `StreamingListener.onBatchCompleted` callback) when a batch is about to complete, a `rejectedException` may get thrown from `checkPointWriter.executor`, since the `eventLoop` will try to process `DoCheckpoint` events even after the `checkPointWriter.executor` was stopped.

Please see [SPARK-14701](https://issues.apache.org/jira/browse/SPARK-14701) for details and stack traces.

## What changes were proposed in this pull request?

Reversed the stopping order of `event loop` and `checkpoint writer`.

## How was this patch tested?

Existing test suits.
(no dedicated test suits were added because the change is simple to reason about)

Author: Liwei Lin <lwlin7@gmail.com>

Closes #12489 from lw-lin/spark-14701.
parent 3647120a
No related branches found
No related tags found
No related merge requests found
......@@ -154,9 +154,9 @@ class JobGenerator(jobScheduler: JobScheduler) extends Logging {
graph.stop()
}
// Stop the event loop and checkpoint writer
if (shouldCheckpoint) checkpointWriter.stop()
// First stop the event loop, then stop the checkpoint writer; see SPARK-14701
eventLoop.stop()
if (shouldCheckpoint) checkpointWriter.stop()
logInfo("Stopped JobGenerator")
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment