Skip to content
Snippets Groups Projects
Commit 8861cdf1 authored by Raymond Liu's avatar Raymond Liu Committed by Josh Rosen
Browse files

Clean unused code in SortShuffleWriter

Just clean unused code which have been moved into ExternalSorter.

Author: Raymond Liu <raymond.liu@intel.com>

Closes #1882 from colorant/sortShuffleWriter and squashes the following commits:

e6337be [Raymond Liu] Clean unused code in SortShuffleWriter
parent 8df4dad4
No related branches found
No related tags found
No related merge requests found
......@@ -17,12 +17,11 @@
package org.apache.spark.shuffle.sort
import java.io.{BufferedOutputStream, File, FileOutputStream, DataOutputStream}
import java.io.File
import org.apache.spark.{MapOutputTracker, SparkEnv, Logging, TaskContext}
import org.apache.spark.executor.ShuffleWriteMetrics
import org.apache.spark.scheduler.MapStatus
import org.apache.spark.serializer.Serializer
import org.apache.spark.shuffle.{ShuffleWriter, BaseShuffleHandle}
import org.apache.spark.storage.ShuffleBlockId
import org.apache.spark.util.collection.ExternalSorter
......@@ -37,10 +36,6 @@ private[spark] class SortShuffleWriter[K, V, C](
private val numPartitions = dep.partitioner.numPartitions
private val blockManager = SparkEnv.get.blockManager
private val ser = Serializer.getSerializer(dep.serializer.orNull)
private val conf = SparkEnv.get.conf
private val fileBufferSize = conf.getInt("spark.shuffle.file.buffer.kb", 32) * 1024
private var sorter: ExternalSorter[K, V, _] = null
private var outputFile: File = null
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment