-
- Downloads
[SPARK-6152] Use shaded ASM5 to support closure cleaning of Java 8 compiled classes
This patch modifies Spark's closure cleaner (and a few other places) to use ASM 5, which is necessary in order to support cleaning of closures that were compiled by Java 8. In order to avoid ASM dependency conflicts, Spark excludes ASM from all of its dependencies and uses a shaded version of ASM 4 that comes from `reflectasm` (see [SPARK-782](https://issues.apache.org/jira/browse/SPARK-782) and #232). This patch updates Spark to use a shaded version of ASM 5.0.4 that was published by the Apache XBean project; the POM used to create the shaded artifact can be found at https://github.com/apache/geronimo-xbean/blob/xbean-4.4/xbean-asm5-shaded/pom.xml. http://movingfulcrum.tumblr.com/post/80826553604/asm-framework-50-the-missing-migration-guide was a useful resource while upgrading the code to use the new ASM5 opcodes. I also added a new regression tests in the `java8-tests` subproject; the existing tests were insufficient to catch this bug, which only affected Scala 2.11 user code which was compiled targeting Java 8. Author: Josh Rosen <joshrosen@databricks.com> Closes #9512 from JoshRosen/SPARK-6152.
Showing
- core/pom.xml 4 additions, 0 deletionscore/pom.xml
- core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala 13 additions, 12 deletions...src/main/scala/org/apache/spark/util/ClosureCleaner.scala
- docs/building-spark.md 4 additions, 0 deletionsdocs/building-spark.md
- extras/java8-tests/src/test/scala/org/apache/spark/JDK8ScalaSuite.scala 27 additions, 0 deletions...ests/src/test/scala/org/apache/spark/JDK8ScalaSuite.scala
- graphx/pom.xml 4 additions, 0 deletionsgraphx/pom.xml
- graphx/src/main/scala/org/apache/spark/graphx/util/BytecodeUtils.scala 8 additions, 8 deletions...in/scala/org/apache/spark/graphx/util/BytecodeUtils.scala
- pom.xml 8 additions, 0 deletionspom.xml
- project/SparkBuild.scala 24 additions, 2 deletionsproject/SparkBuild.scala
- repl/pom.xml 4 additions, 0 deletionsrepl/pom.xml
- repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala 4 additions, 5 deletions...ain/scala/org/apache/spark/repl/ExecutorClassLoader.scala
- sql/core/pom.xml 5 additions, 0 deletionssql/core/pom.xml
- sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala 20 additions, 30 deletions...g/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
Loading
Please register or sign in to comment