-
- Downloads
[SPARK-19085][SQL] cleanup OutputWriterFactory and OutputWriter
## What changes were proposed in this pull request? `OutputWriterFactory`/`OutputWriter` are internal interfaces and we can remove some unnecessary APIs: 1. `OutputWriterFactory.newWriter(path: String)`: no one calls it and no one implements it. 2. `OutputWriter.write(row: Row)`: during execution we only call `writeInternal`, which is weird as `OutputWriter` is already an internal interface. We should rename `writeInternal` to `write` and remove `def write(row: Row)` and it's related converter code. All implementations should just implement `def write(row: InternalRow)` ## How was this patch tested? existing tests. Author: Wenchen Fan <wenchen@databricks.com> Closes #16479 from cloud-fan/hive-writer.
Showing
- mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala 7 additions, 3 deletions...la/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
- mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala 3 additions, 3 deletions...g/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 2 additions, 2 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala 10 additions, 12 deletions...he/spark/sql/execution/datasources/FileFormatWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala 1 addition, 1 deletion...ution/datasources/InsertIntoHadoopFsRelationCommand.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/OutputWriter.scala 1 addition, 25 deletions...apache/spark/sql/execution/datasources/OutputWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVRelation.scala 1 addition, 3 deletions...che/spark/sql/execution/datasources/csv/CSVRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala 1 addition, 3 deletions...spark/sql/execution/datasources/json/JsonFileFormat.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetOutputWriter.scala 1 addition, 3 deletions...l/execution/datasources/parquet/ParquetOutputWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala 1 addition, 3 deletions...spark/sql/execution/datasources/text/TextFileFormat.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala 1 addition, 4 deletions...n/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/CommitFailureTestSource.scala 4 additions, 3 deletions...rg/apache/spark/sql/sources/CommitFailureTestSource.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala 4 additions, 4 deletions...ala/org/apache/spark/sql/sources/SimpleTextRelation.scala
Loading
Please register or sign in to comment