Skip to content
Snippets Groups Projects
Commit f893e262 authored by Adam Roberts's avatar Adam Roberts Committed by Sean Owen
Browse files

[SPARK-17524][TESTS] Use specified spark.buffer.pageSize

## What changes were proposed in this pull request?

This PR has the appendRowUntilExceedingPageSize test in RowBasedKeyValueBatchSuite use whatever spark.buffer.pageSize value a user has specified to prevent a test failure for anyone testing Apache Spark on a box with a reduced page size. The test is currently hardcoded to use the default page size which is 64 MB so this minor PR is a test improvement

## How was this patch tested?
Existing unit tests with 1 MB page size and with 64 MB (the default) page size

Author: Adam Roberts <aroberts@uk.ibm.com>

Closes #15079 from a-roberts/patch-5.
parent d15b4f90
No related branches found
No related tags found
No related merge requests found
......@@ -338,15 +338,17 @@ public class RowBasedKeyValueBatchSuite {
@Test
public void appendRowUntilExceedingPageSize() throws Exception {
// Use default size or spark.buffer.pageSize if specified
int pageSizeToUse = (int) memoryManager.pageSizeBytes();
RowBasedKeyValueBatch batch = RowBasedKeyValueBatch.allocate(keySchema,
valueSchema, taskMemoryManager, 64 * 1024 * 1024); //enough capacity
valueSchema, taskMemoryManager, pageSizeToUse); //enough capacity
try {
UnsafeRow key = makeKeyRow(1, "A");
UnsafeRow value = makeValueRow(1, 1);
int recordLength = 8 + key.getSizeInBytes() + value.getSizeInBytes() + 8;
int totalSize = 4;
int numRows = 0;
while (totalSize + recordLength < 64 * 1024 * 1024) { // default page size
while (totalSize + recordLength < pageSizeToUse) {
appendRow(batch, key, value);
totalSize += recordLength;
numRows++;
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment