-
- Downloads
[SPARK-6229] Add SASL encryption to network library.
There are two main parts of this change: - Extending the bootstrap mechanism in the network library to add a server-side bootstrap (which works a little bit differently than the client-side bootstrap), and to allow the bootstraps to modify the underlying channel. - Use SASL to encrypt data going through the RPC channel. The second item requires some non-optimal code to be able to work around the fact that the outbound path in netty is not thread-safe, and ordering is very important when encryption is in the picture. A lot of the changes outside the network/common library are just to adjust to the changed API for initializing the RPC server. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #5377 from vanzin/SPARK-6229 and squashes the following commits: ff01966 [Marcelo Vanzin] Use fancy new size config style. be53f32 [Marcelo Vanzin] Merge branch 'master' into SPARK-6229 47d4aff [Marcelo Vanzin] Merge branch 'master' into SPARK-6229 7a2a805 [Marcelo Vanzin] Clean up some unneeded changes. 2f92237 [Marcelo Vanzin] Add comment. 67bb0c6 [Marcelo Vanzin] Revert "Avoid exposing ByteArrayWritableChannel outside of test code." 065f684 [Marcelo Vanzin] Add test to verify chunking. 3d1695d [Marcelo Vanzin] Minor cleanups. 73cff0e [Marcelo Vanzin] Skip bytes in decode path too. 318ad23 [Marcelo Vanzin] Avoid exposing ByteArrayWritableChannel outside of test code. 346f829 [Marcelo Vanzin] Avoid trip through channel selector by not reporting 0 bytes written. a4a5938 [Marcelo Vanzin] Review feedback. 4797519 [Marcelo Vanzin] Remove unused import. 9908ada [Marcelo Vanzin] Fix test, SASL backend disposal. 7fe1489 [Marcelo Vanzin] Add a test that makes sure encryption is actually enabled. adb6f9d [Marcelo Vanzin] Review feedback. cf2a605 [Marcelo Vanzin] Clean up some code. 8584323 [Marcelo Vanzin] Fix a comment. e98bc55 [Marcelo Vanzin] Add option to only allow encrypted connections to the server. dad42fc [Marcelo Vanzin] Make encryption thread-safe, less memory-intensive. b00999a [Marcelo Vanzin] Consolidate ByteArrayWritableChannel, fix SASL code to match master changes. b923cae [Marcelo Vanzin] Make SASL encryption handler thread-safe, handle FileRegion messages. 39539a7 [Marcelo Vanzin] Add config option to enable SASL encryption. 351a86f [Marcelo Vanzin] Add SASL encryption to network library. fbe6ccb [Marcelo Vanzin] Add TransportServerBootstrap, make SASL code use it.
Showing
- core/src/main/scala/org/apache/spark/SecurityManager.scala 15 additions, 2 deletionscore/src/main/scala/org/apache/spark/SecurityManager.scala
- core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala 11 additions, 6 deletions...cala/org/apache/spark/deploy/ExternalShuffleService.scala
- core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala 11 additions, 11 deletions...pache/spark/network/netty/NettyBlockTransferService.scala
- core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala 2 additions, 2 deletions...cala/org/apache/spark/network/nio/ConnectionManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 2 additions, 1 deletion...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- network/common/src/main/java/org/apache/spark/network/TransportContext.java 20 additions, 6 deletions.../main/java/org/apache/spark/network/TransportContext.java
- network/common/src/main/java/org/apache/spark/network/client/TransportClientBootstrap.java 3 additions, 1 deletion...apache/spark/network/client/TransportClientBootstrap.java
- network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java 4 additions, 1 deletion...g/apache/spark/network/client/TransportClientFactory.java
- network/common/src/main/java/org/apache/spark/network/sasl/SaslClientBootstrap.java 34 additions, 7 deletions...va/org/apache/spark/network/sasl/SaslClientBootstrap.java
- network/common/src/main/java/org/apache/spark/network/sasl/SaslEncryption.java 291 additions, 0 deletions...in/java/org/apache/spark/network/sasl/SaslEncryption.java
- network/common/src/main/java/org/apache/spark/network/sasl/SaslEncryptionBackend.java 33 additions, 0 deletions.../org/apache/spark/network/sasl/SaslEncryptionBackend.java
- network/common/src/main/java/org/apache/spark/network/sasl/SaslRpcHandler.java 42 additions, 14 deletions...in/java/org/apache/spark/network/sasl/SaslRpcHandler.java
- network/common/src/main/java/org/apache/spark/network/sasl/SaslServerBootstrap.java 49 additions, 0 deletions...va/org/apache/spark/network/sasl/SaslServerBootstrap.java
- network/common/src/main/java/org/apache/spark/network/sasl/SparkSaslClient.java 29 additions, 4 deletions...n/java/org/apache/spark/network/sasl/SparkSaslClient.java
- network/common/src/main/java/org/apache/spark/network/sasl/SparkSaslServer.java 38 additions, 11 deletions...n/java/org/apache/spark/network/sasl/SparkSaslServer.java
- network/common/src/main/java/org/apache/spark/network/server/TransportServer.java 17 additions, 2 deletions...java/org/apache/spark/network/server/TransportServer.java
- network/common/src/main/java/org/apache/spark/network/server/TransportServerBootstrap.java 36 additions, 0 deletions...apache/spark/network/server/TransportServerBootstrap.java
- network/common/src/main/java/org/apache/spark/network/util/ByteArrayWritableChannel.java 20 additions, 6 deletions...g/apache/spark/network/util/ByteArrayWritableChannel.java
- network/common/src/main/java/org/apache/spark/network/util/TransportConf.java 18 additions, 0 deletions...ain/java/org/apache/spark/network/util/TransportConf.java
- network/common/src/test/java/org/apache/spark/network/ProtocolSuite.java 1 addition, 0 deletions...src/test/java/org/apache/spark/network/ProtocolSuite.java
Loading
Please register or sign in to comment