Skip to content
Snippets Groups Projects
Commit 0c00391f authored by Thomas Graves's avatar Thomas Graves Committed by Andrew Or
Browse files

[SPARK-15121] Improve logging of external shuffle handler

## What changes were proposed in this pull request?

Add more informative logging in the external shuffle service to aid in debugging who is connecting to the YARN Nodemanager when the external shuffle service runs under it.

## How was this patch tested?

Ran and saw logs coming out in log file.

Author: Thomas Graves <tgraves@staydecay.corp.gq1.yahoo.com>

Closes #12900 from tgravescs/SPARK-15121.
parent 6ae9fc00
No related branches found
No related tags found
No related merge requests found
......@@ -35,6 +35,7 @@ import org.apache.spark.network.server.RpcHandler;
import org.apache.spark.network.server.StreamManager;
import org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.AppExecId;
import org.apache.spark.network.shuffle.protocol.*;
import org.apache.spark.network.util.NettyUtils;
import org.apache.spark.network.util.TransportConf;
......@@ -86,7 +87,8 @@ public class ExternalShuffleBlockHandler extends RpcHandler {
blocks.add(blockManager.getBlockData(msg.appId, msg.execId, blockId));
}
long streamId = streamManager.registerStream(client.getClientId(), blocks.iterator());
logger.trace("Registered streamId {} with {} buffers", streamId, msg.blockIds.length);
logger.trace("Registered streamId {} with {} buffers for client {} from host {}", streamId,
msg.blockIds.length, client.getClientId(), NettyUtils.getRemoteAddress(client.getChannel()));
callback.onSuccess(new StreamHandle(streamId, msg.blockIds.length).toByteBuffer());
} else if (msgObj instanceof RegisterExecutor) {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment