-
- Downloads
[SPARK-5595][SPARK-5603][SQL] Add a rule to do PreInsert type casting and...
[SPARK-5595][SPARK-5603][SQL] Add a rule to do PreInsert type casting and field renaming and invalidating in memory cache after INSERT This PR adds a rule to Analyzer that will add preinsert data type casting and field renaming to the select clause in an `INSERT INTO/OVERWRITE` statement. Also, with the change of this PR, we always invalidate our in memory data cache after inserting into a BaseRelation. cc marmbrus liancheng Author: Yin Huai <yhuai@databricks.com> Closes #4373 from yhuai/insertFollowUp and squashes the following commits: 08237a7 [Yin Huai] Merge remote-tracking branch 'upstream/master' into insertFollowUp 316542e [Yin Huai] Doc update. c9ccfeb [Yin Huai] Revert a unnecessary change. 84aecc4 [Yin Huai] Address comments. 1951fe1 [Yin Huai] Merge remote-tracking branch 'upstream/master' c18da34 [Yin Huai] Invalidate cache after insert. 727f21a [Yin Huai] Preinsert casting and renaming.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 5 additions, 1 deletion...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala 11 additions, 1 deletion...c/main/scala/org/apache/spark/sql/json/JSONRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/sources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/commands.scala 7 additions, 3 deletions...rc/main/scala/org/apache/spark/sql/sources/commands.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala 16 additions, 0 deletions.../main/scala/org/apache/spark/sql/sources/interfaces.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/rules.scala 76 additions, 0 deletions...e/src/main/scala/org/apache/spark/sql/sources/rules.scala
- sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala 25 additions, 0 deletions.../src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/DataSourceTest.scala 5 additions, 1 deletion...t/scala/org/apache/spark/sql/sources/DataSourceTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/InsertIntoSuite.scala 80 additions, 0 deletions.../scala/org/apache/spark/sql/sources/InsertIntoSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 1 addition, 0 deletions...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
Loading
Please register or sign in to comment