Skip to content
Snippets Groups Projects
Commit ecc6eb50 authored by Tathagata Das's avatar Tathagata Das
Browse files

[SPARK-7315] [STREAMING] [TEST] Fix flaky WALBackedBlockRDDSuite

`FileUtils.getTempDirectoryPath()` path may or may not exist. We want to make sure that it does not exist.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #5853 from tdas/SPARK-7315 and squashes the following commits:

141afd5 [Tathagata Das] Removed use of FileUtils
b08d4f1 [Tathagata Das] Fix flaky WALBackedBlockRDDSuite
parent 7394e7ad
No related branches found
No related tags found
No related merge requests found
......@@ -16,13 +16,13 @@
*/
package org.apache.spark.streaming.rdd
import java.io.File
import java.nio.ByteBuffer
import java.util.UUID
import scala.reflect.ClassTag
import scala.util.control.NonFatal
import org.apache.commons.io.FileUtils
import org.apache.spark._
import org.apache.spark.rdd.BlockRDD
import org.apache.spark.storage.{BlockId, StorageLevel}
......@@ -108,9 +108,13 @@ class WriteAheadLogBackedBlockRDD[T: ClassTag](
// writing log data. However, the directory is not needed if data needs to be read, hence
// a dummy path is provided to satisfy the method parameter requirements.
// FileBasedWriteAheadLog will not create any file or directory at that path.
val dummyDirectory = FileUtils.getTempDirectoryPath()
// FileBasedWriteAheadLog will not create any file or directory at that path. Also,
// this dummy directory should not already exist otherwise the WAL will try to recover
// past events from the directory and throw errors.
val nonExistentDirectory = new File(
System.getProperty("java.io.tmpdir"), UUID.randomUUID().toString).getAbsolutePath
writeAheadLog = WriteAheadLogUtils.createLogForReceiver(
SparkEnv.get.conf, dummyDirectory, hadoopConf)
SparkEnv.get.conf, nonExistentDirectory, hadoopConf)
dataRead = writeAheadLog.read(partition.walRecordHandle)
} catch {
case NonFatal(e) =>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment