-
- Downloads
[SPARK-17650] malformed url's throw exceptions before bricking Executors
## What changes were proposed in this pull request? When a malformed URL was sent to Executors through `sc.addJar` and `sc.addFile`, the executors become unusable, because they constantly throw `MalformedURLException`s and can never acknowledge that the file or jar is just bad input. This PR tries to fix that problem by making sure MalformedURLs can never be submitted through `sc.addJar` and `sc.addFile`. Another solution would be to blacklist bad files and jars on Executors. Maybe fail the first time, and then ignore the second time (but print a warning message). ## How was this patch tested? Unit tests in SparkContextSuite Author: Burak Yavuz <brkyvz@gmail.com> Closes #15224 from brkyvz/SPARK-17650.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 9 additions, 7 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 20 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/SparkContextSuite.scala 22 additions, 0 deletionscore/src/test/scala/org/apache/spark/SparkContextSuite.scala
Loading
Please register or sign in to comment