Skip to content
Snippets Groups Projects
Commit 88c9c467 authored by dding3's avatar dding3 Committed by Sean Owen
Browse files

[SPARK-15562][ML] Delete temp directory after program exit in DataFrameExample

## What changes were proposed in this pull request?
Temp directory used to save records is not deleted after program exit in DataFrameExample. Although it called deleteOnExit, it doesn't work as the directory is not empty. Similar things happend in ContextCleanerSuite. Update the code to make sure temp directory is deleted after program exit.

## How was this patch tested?

unit tests and local build.

Author: dding3 <ding.ding@intel.com>

Closes #13328 from dding3/master.
parent 5d4dafe8
No related branches found
No related tags found
No related merge requests found
......@@ -32,6 +32,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.rdd.{RDD, ReliableRDDCheckpointData}
import org.apache.spark.shuffle.sort.SortShuffleManager
import org.apache.spark.storage._
import org.apache.spark.util.Utils
/**
* An abstract base class for context cleaner tests, which sets up a context with a config
......@@ -206,8 +207,7 @@ class ContextCleanerSuite extends ContextCleanerSuiteBase {
}
test("automatically cleanup normal checkpoint") {
val checkpointDir = java.io.File.createTempFile("temp", "")
checkpointDir.deleteOnExit()
val checkpointDir = Utils.createTempDir()
checkpointDir.delete()
var rdd = newPairRDD()
sc.setCheckpointDir(checkpointDir.toString)
......
......@@ -28,6 +28,7 @@ import org.apache.spark.ml.linalg.Vector
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
import org.apache.spark.sql.{DataFrame, Row, SparkSession}
import org.apache.spark.util.Utils
/**
* An example of how to use [[org.apache.spark.sql.DataFrame]] for ML. Run with
......@@ -86,8 +87,7 @@ object DataFrameExample {
println(s"Selected features column with average values:\n ${featureSummary.mean.toString}")
// Save the records in a parquet file.
val tmpDir = Files.createTempDir()
tmpDir.deleteOnExit()
val tmpDir = Utils.createTempDir()
val outputDir = new File(tmpDir, "dataframe").toString
println(s"Saving to $outputDir as Parquet file.")
df.write.parquet(outputDir)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment