-
- Downloads
[SPARK-2784][SQL] Deprecate hql() method in favor of a config option, 'spark.sql.dialect'
Many users have reported being confused by the distinction between the `sql` and `hql` methods. Specifically, many users think that `sql(...)` cannot be used to read hive tables. In this PR I introduce a new configuration option `spark.sql.dialect` that picks which dialect with be used for parsing. For SQLContext this must be set to `sql`. In `HiveContext` it defaults to `hiveql` but can also be set to `sql`. The `hql` and `hiveql` methods continue to act the same but are now marked as deprecated. **This is a possibly breaking change for some users unless they set the dialect manually, though this is unlikely.** For example: `hiveContex.sql("SELECT 1")` will now throw a parsing exception by default. Author: Michael Armbrust <michael@databricks.com> Closes #1746 from marmbrus/sqlLanguageConf and squashes the following commits: ad375cc [Michael Armbrust] Merge remote-tracking branch 'apache/master' into sqlLanguageConf 20c43f8 [Michael Armbrust] override function instead of just setting the value 7e4ae93 [Michael Armbrust] Deprecate hql() method in favor of a config option, 'spark.sql.dialect'
Showing
- dev/audit-release/sbt_app_hive/src/main/scala/HiveApp.scala 4 additions, 4 deletionsdev/audit-release/sbt_app_hive/src/main/scala/HiveApp.scala
- docs/sql-programming-guide.md 9 additions, 9 deletionsdocs/sql-programming-guide.md
- examples/src/main/scala/org/apache/spark/examples/sql/hive/HiveFromSpark.scala 6 additions, 6 deletions...la/org/apache/spark/examples/sql/hive/HiveFromSpark.scala
- python/pyspark/sql.py 12 additions, 8 deletionspython/pyspark/sql.py
- sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala 16 additions, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 9 additions, 2 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/api/java/JavaSQLContext.scala 11 additions, 3 deletions.../scala/org/apache/spark/sql/api/java/JavaSQLContext.scala
- sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala 1 addition, 1 deletion...g/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
- sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala 1 addition, 1 deletion...l/hive/thriftserver/server/SparkSQLOperationManager.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 20 additions, 6 deletions...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/api/java/JavaHiveContext.scala 14 additions, 1 deletion.../org/apache/spark/sql/hive/api/java/JavaHiveContext.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/CachedTableSuite.scala 7 additions, 7 deletions...st/scala/org/apache/spark/sql/hive/CachedTableSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala 5 additions, 5 deletions...est/scala/org/apache/spark/sql/hive/StatisticsSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/api/java/JavaHiveQLSuite.scala 10 additions, 9 deletions.../org/apache/spark/sql/hive/api/java/JavaHiveQLSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala 2 additions, 2 deletions.../apache/spark/sql/hive/execution/HiveComparisonTest.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala 48 additions, 45 deletions.../org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveResolutionSuite.scala 3 additions, 3 deletions...apache/spark/sql/hive/execution/HiveResolutionSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveTypeCoercionSuite.scala 1 addition, 1 deletion...ache/spark/sql/hive/execution/HiveTypeCoercionSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUdfSuite.scala 5 additions, 5 deletions...la/org/apache/spark/sql/hive/execution/HiveUdfSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruningSuite.scala 1 addition, 1 deletion...la/org/apache/spark/sql/hive/execution/PruningSuite.scala
Loading
Please register or sign in to comment