-
- Downloads
[SPARK-3904] [SQL] add constant objectinspector support for udfs
In HQL, we convert all of the data type into normal `ObjectInspector`s for UDFs, most of cases it works, however, some of the UDF actually requires its children `ObjectInspector` to be the `ConstantObjectInspector`, which will cause exception. e.g. select named_struct("x", "str") from src limit 1; I updated the method `wrap` by adding the one more parameter `ObjectInspector`(to describe what it expects to wrap to, for example: java.lang.Integer or IntWritable). As well as the `unwrap` method by providing the input `ObjectInspector`. Author: Cheng Hao <hao.cheng@intel.com> Closes #2762 from chenghao-intel/udf_coi and squashes the following commits: bcacfd7 [Cheng Hao] Shim for both Hive 0.12 & 0.13.1 2416e5d [Cheng Hao] revert to hive 0.12 5793c01 [Cheng Hao] add space before while 4e56e1b [Cheng Hao] style issue 683d3fd [Cheng Hao] Add golden files fe591e4 [Cheng Hao] update HiveGenericUdf for set the ObjectInspector while constructing the DeferredObject f6740fe [Cheng Hao] Support Constant ObjectInspector for Map & List 8814c3a [Cheng Hao] Passing ContantObjectInspector(when necessary) for UDF initializing
Showing
- sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala 7 additions, 1 deletion...che/spark/sql/hive/execution/HiveCompatibilitySuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala 122 additions, 63 deletions...main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/sql/hive/TableReader.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala 37 additions, 25 deletions...e/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
- sql/hive/src/test/resources/golden/constant array-0-761ef205b10ac4a10122c8b4ce10ada 1 addition, 0 deletions...s/golden/constant array-0-761ef205b10ac4a10122c8b4ce10ada
- sql/hive/src/test/resources/golden/udf_named_struct-0-8f0ea83364b78634fbb3752c5a5c725 1 addition, 0 deletions...golden/udf_named_struct-0-8f0ea83364b78634fbb3752c5a5c725
- sql/hive/src/test/resources/golden/udf_named_struct-1-380c9638cc6ea8ea42f187bf0cedf350 1 addition, 0 deletions...olden/udf_named_struct-1-380c9638cc6ea8ea42f187bf0cedf350
- sql/hive/src/test/resources/golden/udf_named_struct-2-22a79ac608b1249306f82f4bdc669b17 0 additions, 0 deletions...olden/udf_named_struct-2-22a79ac608b1249306f82f4bdc669b17
- sql/hive/src/test/resources/golden/udf_named_struct-3-d7e4a555934307155784904ff9df188b 1 addition, 0 deletions...olden/udf_named_struct-3-d7e4a555934307155784904ff9df188b
- sql/hive/src/test/resources/golden/udf_sort_array-0-e86d559aeb84a4cc017a103182c22bfb 0 additions, 0 deletions.../golden/udf_sort_array-0-e86d559aeb84a4cc017a103182c22bfb
- sql/hive/src/test/resources/golden/udf_sort_array-1-976cd8b6b50a2748bbc768aa5e11cf82 1 addition, 0 deletions.../golden/udf_sort_array-1-976cd8b6b50a2748bbc768aa5e11cf82
- sql/hive/src/test/resources/golden/udf_sort_array-10-9e047718e5fea6ea79124f1e899f1c13 1 addition, 0 deletions...golden/udf_sort_array-10-9e047718e5fea6ea79124f1e899f1c13
- sql/hive/src/test/resources/golden/udf_sort_array-2-c429ec85a6da60ebd4bc6f0f266e8b93 4 additions, 0 deletions.../golden/udf_sort_array-2-c429ec85a6da60ebd4bc6f0f266e8b93
- sql/hive/src/test/resources/golden/udf_sort_array-3-55c4cdaf8438b06675d60848d68f35de 0 additions, 0 deletions.../golden/udf_sort_array-3-55c4cdaf8438b06675d60848d68f35de
- sql/hive/src/test/resources/golden/udf_struct-0-f41043b7d9f14fa5e998c90454c7bdb1 1 addition, 0 deletions...rces/golden/udf_struct-0-f41043b7d9f14fa5e998c90454c7bdb1
- sql/hive/src/test/resources/golden/udf_struct-1-8ccdb20153debdab789ea8ad0228e2eb 1 addition, 0 deletions...rces/golden/udf_struct-1-8ccdb20153debdab789ea8ad0228e2eb
- sql/hive/src/test/resources/golden/udf_struct-2-4a62774a6de7571c8d2bcb77da63f8f3 0 additions, 0 deletions...rces/golden/udf_struct-2-4a62774a6de7571c8d2bcb77da63f8f3
- sql/hive/src/test/resources/golden/udf_struct-3-abffdaacb0c7076ab538fbeec072daa2 1 addition, 0 deletions...rces/golden/udf_struct-3-abffdaacb0c7076ab538fbeec072daa2
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala 8 additions, 0 deletions.../org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
- sql/hive/v0.12.0/src/main/scala/org/apache/spark/sql/hive/Shim12.scala 57 additions, 0 deletions...2.0/src/main/scala/org/apache/spark/sql/hive/Shim12.scala
Loading
Please register or sign in to comment