-
- Downloads
[SPARK-4307] Initialize FileDescriptor lazily in FileRegion.
Netty's DefaultFileRegion requires a FileDescriptor in its constructor, which means we need to have a opened file handle. In super large workloads, this could lead to too many open files due to the way these file descriptors are cleaned. This pull request creates a new LazyFileRegion that initializes the FileDescriptor when we are sending data for the first time. Author: Reynold Xin <rxin@databricks.com> Author: Reynold Xin <rxin@apache.org> Closes #3172 from rxin/lazyFD and squashes the following commits: 0bdcdc6 [Reynold Xin] Added reference to Netty's DefaultFileRegion d4564ae [Reynold Xin] Added SparkConf to the ctor argument of IndexShuffleBlockManager. 6ed369e [Reynold Xin] Code review feedback. 04cddc8 [Reynold Xin] [SPARK-4307] Initialize FileDescriptor lazily in FileRegion.
Showing
- core/src/main/scala/org/apache/spark/deploy/worker/StandaloneWorkerShuffleService.scala 1 addition, 1 deletion.../spark/deploy/worker/StandaloneWorkerShuffleService.scala
- core/src/main/scala/org/apache/spark/shuffle/FileShuffleBlockManager.scala 6 additions, 2 deletions...la/org/apache/spark/shuffle/FileShuffleBlockManager.scala
- core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala 6 additions, 2 deletions...a/org/apache/spark/shuffle/IndexShuffleBlockManager.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala 1 addition, 1 deletion...la/org/apache/spark/shuffle/sort/SortShuffleManager.scala
- core/src/test/scala/org/apache/spark/ExternalShuffleServiceSuite.scala 1 addition, 1 deletion.../scala/org/apache/spark/ExternalShuffleServiceSuite.scala
- network/common/src/main/java/org/apache/spark/network/buffer/FileSegmentManagedBuffer.java 11 additions, 12 deletions...apache/spark/network/buffer/FileSegmentManagedBuffer.java
- network/common/src/main/java/org/apache/spark/network/buffer/LazyFileRegion.java 111 additions, 0 deletions.../java/org/apache/spark/network/buffer/LazyFileRegion.java
- network/common/src/main/java/org/apache/spark/network/util/TransportConf.java 17 additions, 0 deletions...ain/java/org/apache/spark/network/util/TransportConf.java
- network/common/src/test/java/org/apache/spark/network/ChunkFetchIntegrationSuite.java 6 additions, 3 deletions.../org/apache/spark/network/ChunkFetchIntegrationSuite.java
- network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java 3 additions, 2 deletions...he/spark/network/shuffle/ExternalShuffleBlockHandler.java
- network/shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockManager.java 9 additions, 4 deletions...he/spark/network/shuffle/ExternalShuffleBlockManager.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockManagerSuite.java 7 additions, 3 deletions...ark/network/shuffle/ExternalShuffleBlockManagerSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleCleanupSuite.java 8 additions, 5 deletions...he/spark/network/shuffle/ExternalShuffleCleanupSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleIntegrationSuite.java 1 addition, 1 deletion...park/network/shuffle/ExternalShuffleIntegrationSuite.java
- network/shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleSecuritySuite.java 1 addition, 1 deletion...e/spark/network/shuffle/ExternalShuffleSecuritySuite.java
- network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java 2 additions, 2 deletions...ava/org/apache/spark/network/yarn/YarnShuffleService.java
Loading
Please register or sign in to comment