Skip to content
  • Reynold Xin's avatar
    9ca0760d
    [SPARK-10063][SQL] Remove DirectParquetOutputCommitter · 9ca0760d
    Reynold Xin authored
    ## What changes were proposed in this pull request?
    This patch removes DirectParquetOutputCommitter. This was initially created by Databricks as a faster way to write Parquet data to S3. However, given how the underlying S3 Hadoop implementation works, this committer only works when there are no failures. If there are multiple attempts of the same task (e.g. speculation or task failures or node failures), the output data can be corrupted. I don't think this performance optimization outweighs the correctness issue.
    
    ## How was this patch tested?
    Removed the related tests also.
    
    Author: Reynold Xin <rxin@databricks.com>
    
    Closes #12229 from rxin/SPARK-10063.
    9ca0760d
    [SPARK-10063][SQL] Remove DirectParquetOutputCommitter
    Reynold Xin authored
    ## What changes were proposed in this pull request?
    This patch removes DirectParquetOutputCommitter. This was initially created by Databricks as a faster way to write Parquet data to S3. However, given how the underlying S3 Hadoop implementation works, this committer only works when there are no failures. If there are multiple attempts of the same task (e.g. speculation or task failures or node failures), the output data can be corrupted. I don't think this performance optimization outweighs the correctness issue.
    
    ## How was this patch tested?
    Removed the related tests also.
    
    Author: Reynold Xin <rxin@databricks.com>
    
    Closes #12229 from rxin/SPARK-10063.
Loading