-
- Downloads
[SPARK-14913][SQL] Simplify configuration API
## What changes were proposed in this pull request? We currently expose both Hadoop configuration and Spark SQL configuration in RuntimeConfig. I think we can remove the Hadoop configuration part, and simply generate Hadoop Configuration on the fly by passing all the SQL configurations into it. This way, there is a single interface (in Java/Scala/Python/SQL) for end-users. As part of this patch, I also removed some config options deprecated in Spark 1.x. ## How was this patch tested? Updated relevant tests. Author: Reynold Xin <rxin@databricks.com> Closes #12689 from rxin/SPARK-14913.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 1 addition, 1 deletion...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala 27 additions, 38 deletions...e/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala
- sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 2 additions, 4 deletions...re/src/main/scala/org/apache/spark/sql/SparkSession.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTable.scala 1 addition, 1 deletion...org/apache/spark/sql/execution/command/AnalyzeTable.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala 0 additions, 64 deletions...a/org/apache/spark/sql/execution/command/SetCommand.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/execution/command/tables.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 4 additions, 4 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala 1 addition, 5 deletions.../spark/sql/execution/datasources/FileSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelation.scala 1 addition, 5 deletions...ql/execution/datasources/InsertIntoHadoopFsRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/fileSourceInterfaces.scala 1 addition, 1 deletion...park/sql/execution/datasources/fileSourceInterfaces.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONRelation.scala 1 addition, 1 deletion...e/spark/sql/execution/datasources/json/JSONRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala 1 addition, 1 deletion...k/sql/execution/datasources/parquet/ParquetRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala 1 addition, 1 deletion...apache/spark/sql/execution/streaming/FileStreamSink.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala 1 addition, 1 deletion...ache/spark/sql/execution/streaming/FileStreamSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala 1 addition, 1 deletion...pache/spark/sql/execution/streaming/HDFSMetadataLog.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamFileCatalog.scala 1 addition, 1 deletion...che/spark/sql/execution/streaming/StreamFileCatalog.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala 1 addition, 1 deletion...e/spark/sql/execution/streaming/state/StateStoreRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/RuntimeConfigImpl.scala 0 additions, 73 deletions...ala/org/apache/spark/sql/internal/RuntimeConfigImpl.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 1 addition, 8 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala 15 additions, 2 deletions...in/scala/org/apache/spark/sql/internal/SessionState.scala
Loading
Please register or sign in to comment