-
- Downloads
[SPARK-21490][CORE] Make sure SparkLauncher redirects needed streams.
The code was failing to account for some cases when setting up log redirection. For example, if a user redirected only stdout to a file, the launcher code would leave stderr without redirection, which could lead to child processes getting stuck because stderr wasn't being read. So detect cases where only one of the streams is redirected, and redirect the other stream to the log as appropriate. For the old "launch()" API, redirection of the unconfigured stream only happens if the user has explicitly requested for log redirection. Log redirection is on by default with "startApplication()". Most of the change is actually adding new unit tests to make sure the different cases work as expected. As part of that, I moved some tests that were in the core/ module to the launcher/ module instead, since they don't depend on spark-submit. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #18696 from vanzin/SPARK-21490.
Showing
- core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java 1 addition, 60 deletions...st/java/org/apache/spark/launcher/SparkLauncherSuite.java
- launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java 3 additions, 0 deletions...ava/org/apache/spark/launcher/AbstractCommandBuilder.java
- launcher/src/main/java/org/apache/spark/launcher/ChildProcAppHandle.java 11 additions, 3 deletions...in/java/org/apache/spark/launcher/ChildProcAppHandle.java
- launcher/src/main/java/org/apache/spark/launcher/OutputRedirector.java 4 additions, 0 deletions...main/java/org/apache/spark/launcher/OutputRedirector.java
- launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java 58 additions, 29 deletions...rc/main/java/org/apache/spark/launcher/SparkLauncher.java
- launcher/src/test/java/org/apache/spark/launcher/OutputRedirectionSuite.java 226 additions, 0 deletions...ava/org/apache/spark/launcher/OutputRedirectionSuite.java
- launcher/src/test/resources/log4j.properties 4 additions, 0 deletionslauncher/src/test/resources/log4j.properties
Loading
Please register or sign in to comment