-
- Downloads
[SPARK-21873][SS] - Avoid using `return` inside `CachedKafkaConsumer.get`
During profiling of a structured streaming application with Kafka as the source, I came across this exception:  This is a 1 minute sample, which caused 106K `NonLocalReturnControl` exceptions to be thrown. This happens because `CachedKafkaConsumer.get` is ran inside: `private def runUninterruptiblyIfPossible[T](body: => T): T` Where `body: => T` is the `get` method. Turning the method into a function means that in order to escape the `while` loop defined in `get` the runtime has to do dirty tricks which involve throwing the above exception. ## What changes were proposed in this pull request? Instead of using `return` (which is generally not recommended in Scala), we place the result of the `fetchData` method inside a local variable and use a boolean flag to indicate the status of fetching data, which we monitor as our predicate to the `while` loop. ## How was this patch tested? I've ran the `KafkaSourceSuite` to make sure regression passes. Since the exception isn't visible from user code, there is no way (at least that I could think of) to add this as a test to the existing suite. Author: Yuval Itzchakov <yuval.itzchakov@clicktale.com> Closes #19059 from YuvalItzchakov/master.
Please register or sign in to comment