Skip to content
Snippets Groups Projects
Commit 4399b7b0 authored by Marcelo Vanzin's avatar Marcelo Vanzin Committed by Reynold Xin
Browse files

[SPARK-9651] Fix UnsafeExternalSorterSuite.

First, it's probably a bad idea to call generated Scala methods
from Java. In this case, the method being called wasn't actually
"Utils.createTempDir()", but actually the method that returns the
first default argument to the actual createTempDir method, which
is just the location of java.io.tmpdir; meaning that all tests in
the class were using the same temp dir, and thus affecting each
other.

Second, spillingOccursInResponseToMemoryPressure was not writing
enough records to actually cause a spill.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #7970 from vanzin/SPARK-9651 and squashes the following commits:

74d357f [Marcelo Vanzin] Clean up temp dir on test tear down.
a64f36a [Marcelo Vanzin] [SPARK-9651] Fix UnsafeExternalSorterSuite.
parent 8c320e45
No related branches found
No related tags found
No related merge requests found
...@@ -101,7 +101,7 @@ public class UnsafeExternalSorterSuite { ...@@ -101,7 +101,7 @@ public class UnsafeExternalSorterSuite {
public void setUp() { public void setUp() {
MockitoAnnotations.initMocks(this); MockitoAnnotations.initMocks(this);
sparkConf = new SparkConf(); sparkConf = new SparkConf();
tempDir = new File(Utils.createTempDir$default$1()); tempDir = Utils.createTempDir(System.getProperty("java.io.tmpdir"), "unsafe-test");
shuffleMemoryManager = new ShuffleMemoryManager(Long.MAX_VALUE); shuffleMemoryManager = new ShuffleMemoryManager(Long.MAX_VALUE);
spillFilesCreated.clear(); spillFilesCreated.clear();
taskContext = mock(TaskContext.class); taskContext = mock(TaskContext.class);
...@@ -143,13 +143,18 @@ public class UnsafeExternalSorterSuite { ...@@ -143,13 +143,18 @@ public class UnsafeExternalSorterSuite {
@After @After
public void tearDown() { public void tearDown() {
long leakedUnsafeMemory = taskMemoryManager.cleanUpAllAllocatedMemory(); try {
if (shuffleMemoryManager != null) { long leakedUnsafeMemory = taskMemoryManager.cleanUpAllAllocatedMemory();
long leakedShuffleMemory = shuffleMemoryManager.getMemoryConsumptionForThisTask(); if (shuffleMemoryManager != null) {
shuffleMemoryManager = null; long leakedShuffleMemory = shuffleMemoryManager.getMemoryConsumptionForThisTask();
assertEquals(0L, leakedShuffleMemory); shuffleMemoryManager = null;
assertEquals(0L, leakedShuffleMemory);
}
assertEquals(0, leakedUnsafeMemory);
} finally {
Utils.deleteRecursively(tempDir);
tempDir = null;
} }
assertEquals(0, leakedUnsafeMemory);
} }
private void assertSpillFilesWereCleanedUp() { private void assertSpillFilesWereCleanedUp() {
...@@ -234,7 +239,7 @@ public class UnsafeExternalSorterSuite { ...@@ -234,7 +239,7 @@ public class UnsafeExternalSorterSuite {
public void spillingOccursInResponseToMemoryPressure() throws Exception { public void spillingOccursInResponseToMemoryPressure() throws Exception {
shuffleMemoryManager = new ShuffleMemoryManager(pageSizeBytes * 2); shuffleMemoryManager = new ShuffleMemoryManager(pageSizeBytes * 2);
final UnsafeExternalSorter sorter = newSorter(); final UnsafeExternalSorter sorter = newSorter();
final int numRecords = 100000; final int numRecords = (int) pageSizeBytes / 4;
for (int i = 0; i <= numRecords; i++) { for (int i = 0; i <= numRecords; i++) {
insertNumber(sorter, numRecords - i); insertNumber(sorter, numRecords - i);
} }
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment