-
- Downloads
[SPARK-2177][SQL] describe table result contains only one column
``` scala> hql("describe src").collect().foreach(println) [key string None ] [value string None ] ``` The result should contain 3 columns instead of one. This screws up JDBC or even the downstream consumer of the Scala/Java/Python APIs. I am providing a workaround. We handle a subset of describe commands in Spark SQL, which are defined by ... ``` DESCRIBE [EXTENDED] [db_name.]table_name ``` All other cases are treated as Hive native commands. Also, if we upgrade Hive to 0.13, we need to check the results of context.sessionState.isHiveServerQuery() to determine how to split the result. This method is introduced by https://issues.apache.org/jira/browse/HIVE-4545. We may want to set Hive to use JsonMetaDataFormatter for the output of a DDL statement (`set hive.ddl.output.format=json` introduced by https://issues.apache.org/jira/browse/HIVE-2822). The link to JIRA: https://issues.apache.org/jira/browse/SPARK-2177 Author: Yin Huai <huai@cse.ohio-state.edu> Closes #1118 from yhuai/SPARK-2177 and squashes the following commits: fd2534c [Yin Huai] Merge remote-tracking branch 'upstream/master' into SPARK-2177 b9b9aa5 [Yin Huai] rxin's comments. e7c4e72 [Yin Huai] Fix unit test. 656b068 [Yin Huai] 100 characters. 6387217 [Yin Huai] Merge remote-tracking branch 'upstream/master' into SPARK-2177 8003cf3 [Yin Huai] Generate strings with the format like Hive for unit tests. 9787fff [Yin Huai] Merge remote-tracking branch 'upstream/master' into SPARK-2177 440c5af [Yin Huai] rxin's comments. f1a417e [Yin Huai] Update doc. 83adb2f [Yin Huai] Merge remote-tracking branch 'upstream/master' into SPARK-2177 366f891 [Yin Huai] Add describe command. 74bd1d4 [Yin Huai] Merge remote-tracking branch 'upstream/master' into SPARK-2177 342fdf7 [Yin Huai] Split to up to 3 parts. 725e88c [Yin Huai] Merge remote-tracking branch 'upstream/master' into SPARK-2177 bb8bbef [Yin Huai] Split every string in the result of a describe command.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/commands.scala 16 additions, 0 deletions...rg/apache/spark/sql/catalyst/plans/logical/commands.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala 21 additions, 0 deletions.../main/scala/org/apache/spark/sql/execution/commands.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 5 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 54 additions, 16 deletions...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 10 additions, 0 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/hiveOperators.scala 60 additions, 0 deletions...a/org/apache/spark/sql/hive/execution/hiveOperators.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala 23 additions, 1 deletion.../apache/spark/sql/hive/execution/HiveComparisonTest.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala 11 additions, 6 deletions...che/spark/sql/hive/execution/HiveCompatibilitySuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala 94 additions, 8 deletions.../org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
Loading
Please register or sign in to comment