-
- Downloads
[SPARK-19372][SQL] Fix throwing a Java exception at df.fliter() due to 64KB bytecode size limit
## What changes were proposed in this pull request? When an expression for `df.filter()` has many nodes (e.g. 400), the size of Java bytecode for the generated Java code is more than 64KB. It produces an Java exception. As a result, the execution fails. This PR continues to execute by calling `Expression.eval()` disabling code generation if an exception has been caught. ## How was this patch tested? Add a test suite into `DataFrameSuite` Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes #17087 from kiszk/SPARK-19372.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala 1 addition, 1 deletion...che/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala 17 additions, 4 deletions...park/sql/catalyst/expressions/codegen/CodeGenerator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala 6 additions, 4 deletions...rg/apache/spark/sql/catalyst/expressions/predicates.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 22 additions, 1 deletion...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala 1 addition, 1 deletion...ql/execution/datasources/PartitioningAwareFileIndex.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 11 additions, 0 deletions.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/sources/SimpleTextRelation.scala
Loading
Please register or sign in to comment