-
- Downloads
[SPARK-6299][CORE] ClassNotFoundException in standalone mode when running...
[SPARK-6299][CORE] ClassNotFoundException in standalone mode when running groupByKey with class defined in REPL ``` case class ClassA(value: String) val rdd = sc.parallelize(List(("k1", ClassA("v1")), ("k1", ClassA("v2")) )) rdd.groupByKey.collect ``` This code used to be throw exception in spark-shell, because while shuffling ```JavaSerializer```uses ```defaultClassLoader``` which was defined like ```env.serializer.setDefaultClassLoader(urlClassLoader)```. It should be ```env.serializer.setDefaultClassLoader(replClassLoader)```, like ``` override def run() { val deserializeStartTime = System.currentTimeMillis() Thread.currentThread.setContextClassLoader(replClassLoader) ``` in TaskRunner. When ```replClassLoader``` cannot be defined, it's identical with ```urlClassLoader``` Author: Kevin (Sangwoo) Kim <sangwookim.me@gmail.com> Closes #5046 from swkimme/master and squashes the following commits: fa2b9ee [Kevin (Sangwoo) Kim] stylish test codes ( collect -> collect() ) 6e9620b [Kevin (Sangwoo) Kim] stylish test codes ( collect -> collect() ) d23e4e2 [Kevin (Sangwoo) Kim] stylish test codes ( collect -> collect() ) a4a3c8a [Kevin (Sangwoo) Kim] add 'class defined in repl - shuffle' test to ReplSuite bd00da5 [Kevin (Sangwoo) Kim] add 'class defined in repl - shuffle' test to ReplSuite c1b1fc7 [Kevin (Sangwoo) Kim] use REPL class loader for executor's serializer
Showing
- core/src/main/scala/org/apache/spark/executor/Executor.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/executor/Executor.scala
- repl/scala-2.10/src/test/scala/org/apache/spark/repl/ReplSuite.scala 31 additions, 19 deletions...2.10/src/test/scala/org/apache/spark/repl/ReplSuite.scala
- repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala 31 additions, 19 deletions...2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
Loading
Please register or sign in to comment