-
- Downloads
Revert "[SPARK-8579] [SQL] support arbitrary object in UnsafeRow"
Reverts ObjectPool. As it stands, it has a few problems: 1. ObjectPool doesn't work with spilling and memory accounting. 2. I don't think in the long run the idea of an object pool is what we want to support, since it essentially goes back to unmanaged memory, and creates pressure on GC, and is hard to account for the total in memory size. 3. The ObjectPool patch removed the specialized getters for strings and binary, and as a result, actually introduced branches when reading non primitive data types. If we do want to support arbitrary user defined types in the future, I think we can just add an object array in UnsafeRow, rather than relying on indirect memory addressing through a pool. We also need to pick execution strategies that are optimized for those, rather than keeping a lot of unserialized JVM objects in memory during aggregation. This is probably the hardest thing I had to revert in Spark, due to recent patches that also change the same part of the code. Would be great to get a careful look. Author: Reynold Xin <rxin@databricks.com> Closes #7591 from rxin/revert-object-pool and squashes the following commits: 01db0bc [Reynold Xin] Scala style. eda89fc [Reynold Xin] Fixed describe. 2967118 [Reynold Xin] Fixed accessor for JoinedRow. e3294eb [Reynold Xin] Merge branch 'master' into revert-object-pool 657855f [Reynold Xin] Temp commit. c20f2c8 [Reynold Xin] Style fix. fe37079 [Reynold Xin] Revert "[SPARK-8579] [SQL] support arbitrary object in UnsafeRow"
Showing
- project/SparkBuild.scala 1 addition, 1 deletionproject/SparkBuild.scala
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMap.java 71 additions, 79 deletions.../catalyst/expressions/UnsafeFixedWidthAggregationMap.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java 95 additions, 134 deletions.../org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/ObjectPool.java 0 additions, 78 deletions...n/java/org/apache/spark/sql/catalyst/util/ObjectPool.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/UniqueObjectPool.java 0 additions, 59 deletions.../org/apache/spark/sql/catalyst/util/UniqueObjectPool.java
- sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java 6 additions, 10 deletions...g/apache/spark/sql/execution/UnsafeExternalRowSorter.java
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala 2 additions, 1 deletion...rg/apache/spark/sql/catalyst/CatalystTypeConverters.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala 5 additions, 4 deletions...ain/scala/org/apache/spark/sql/catalyst/InternalRow.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala 2 additions, 0 deletions...pache/spark/sql/catalyst/expressions/BoundAttribute.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala 53 additions, 0 deletions...rg/apache/spark/sql/catalyst/expressions/Projection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/UnsafeRowConverter.scala 14 additions, 28 deletions...e/spark/sql/catalyst/expressions/UnsafeRowConverter.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala 5 additions, 4 deletions...park/sql/catalyst/expressions/codegen/CodeGenerator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala 2 additions, 2 deletions...talyst/expressions/codegen/GenerateUnsafeProjection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala 7 additions, 0 deletions...ache/spark/sql/catalyst/plans/logical/LocalRelation.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMapSuite.scala 22 additions, 43 deletions...yst/expressions/UnsafeFixedWidthAggregationMapSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeRowConverterSuite.scala 38 additions, 99 deletions...rk/sql/catalyst/expressions/UnsafeRowConverterSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/ObjectPoolSuite.scala 0 additions, 57 deletions.../org/apache/spark/sql/catalyst/util/ObjectPoolSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 7 additions, 6 deletionssql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala 13 additions, 4 deletions...a/org/apache/spark/sql/execution/GeneratedAggregate.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScan.scala 0 additions, 2 deletions...scala/org/apache/spark/sql/execution/LocalTableScan.scala
Loading
Please register or sign in to comment