Skip to content
Snippets Groups Projects
Commit b929537b authored by Josh Rosen's avatar Josh Rosen Committed by Reynold Xin
Browse files

[SPARK-18182] Expose ReplayListenerBus.read() overload which takes string iterator

The `ReplayListenerBus.read()` method is used when implementing a custom `ApplicationHistoryProvider`. The current interface only exposes a `read()` method which takes an `InputStream` and performs stream-to-lines conversion itself, but it would also be useful to expose an overloaded method which accepts an iterator of strings, thereby enabling events to be provided from non-`InputStream` sources.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #15698 from JoshRosen/replay-listener-bus-interface.
parent 6e629815
No related branches found
No related tags found
No related merge requests found
......@@ -53,13 +53,24 @@ private[spark] class ReplayListenerBus extends SparkListenerBus with Logging {
sourceName: String,
maybeTruncated: Boolean = false,
eventsFilter: ReplayEventsFilter = SELECT_ALL_FILTER): Unit = {
val lines = Source.fromInputStream(logData).getLines()
replay(lines, sourceName, maybeTruncated, eventsFilter)
}
/**
* Overloaded variant of [[replay()]] which accepts an iterator of lines instead of an
* [[InputStream]]. Exposed for use by custom ApplicationHistoryProvider implementations.
*/
def replay(
lines: Iterator[String],
sourceName: String,
maybeTruncated: Boolean,
eventsFilter: ReplayEventsFilter): Unit = {
var currentLine: String = null
var lineNumber: Int = 0
try {
val lineEntries = Source.fromInputStream(logData)
.getLines()
val lineEntries = lines
.zipWithIndex
.filter { case (line, _) => eventsFilter(line) }
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment